This is an excerpt from my recent book, Building Nostr. You can download and read the whole book for free at building-nostr.coracle.social.

Dealing with other people’s signed events is always going to require some adversarial thinking. Events can be malformed, either from a buggy implementation or from someone trying to attack software applications. A p tag might have an invalid or missing value, or a relay hint that is trying to spy on users. Event content might include HTML injection attacks, or illegal content.

Distinguishing between when to discard an event and when to try to handle it anyway is difficult. The most important thing when handling events is to protect the user from attacks. Everything else should be handled in a fault-tolerant way, assuming incompetence rather than hostility.

It’s important to note that Postel’s “be liberal in what you accept” has two different possible interpretations here. Fault tolerance does not mean repairing data that contradicts established conventions. A common example of this is when clients publish events with bech32-encoded data in tags instead of hex-encoded data.

This is of course a difficult judgment call, since as mentioned above the vocabulary of nostr is idiomatic and evolving. The important thing is not to allow a minority, non-standard data format to hijack a well-established content type. This can go some way toward taming the chaos of a decentralized system, as well as protect against organized attacks that would attempt to “embrace, extend, extinguish” the protocol.

Here are a few examples of malformed data and how it should be handled:

• If an event’s signature is invalid, discard it — there’s no way to prove it isn’t forged. There are of course exceptions to this, like with NIP 59 wrapped events, which have no signature, but are still authenticated cryptographically.
• If an event contains code intended to crash, denial-of-service, or hijack the client, the client should protect itself using conventional defensive programming techniques — by sanitizing HTML before displaying it, not using regular expressions with user provided data, and by handling errors using null checks and fallbacks.
• If an event contains an adversarial relay hint or suspicious media URL, clients should have a policy in place, informed by user preferences, which blocks connections to unknown or untrustworthy hosts. This can be accomplished through domain white/black listing, or through trust assessments based on proof-of-work or the reputation of the event author.
• If an event contains data that doesn’t adhere to the relevant specification, for example missing start or end times on a calendar event, clients might choose to show a fallback value, e.g. “not specified”, or an error indicator.

One reason to prefer fault-tolerant handling over aggressive validation is that strict validation reduces interoperability unnecessarily by converting minor, recoverable errors into catastrophic exceptions that ultimately result in a poor UX. Care has to be taken to balance user protection, user experience, and long-term stewardship of the protocol when making these decisions.

An additional wrinkle to this problem involves writing rather than reading events. A number of bugs have emerged where data published by one client will be unwittingly deleted by another client. Here are a few examples:

• Some clients at one point started adding JSON-encoded relay selections to the content field of kind 3 follower list events. Other clients would drop these selections when updating user follows.
• Some clients started adding muted words to kind 10000 mute lists, in addition to muted pubkeys. Clients that weren’t expecting this ended up dropping muted words when updating muted pubkeys.
• Similarly, some clients began adding private mutes to kind 100000 mute lists by JSON-encoding the tags, encrypting the result, and placing it in the event’s content field. This resulted in mutes being ignored or overwritten on update.

In every case, these bugs are a result of poor design — overloading a single kind for multiple purposes is always a recipe for disaster. But in cases where poor designs become conventional, it can be helpful to handle events in such a way that unanticipated data doesn’t get dropped.

One way I’ve found to do this is to write parsers for an event in order to turn it into a standard data structure that can be used elsewhere in my application with the original event attached. Then, when saving a new version of the event, I update the event directly, keeping tags and content intact as much as possible before re-publishing. Here’s an example:

readProfile = e => {...decodeJson(e.content), e}
writeProfile = ({e, ...p}) => {kind: 0, content: encodeJson(p), tags: e.tags}

You can see that the original event is passed through to writeProfile, allowing us to avoid dropping any tags that may have been included.

This adds a certain amount of additional effort to implementations, but is important for avoiding disruption of user experience.

Backwards Compatibility

Backwards incompatibility is one of the big problems of spec design. When breaking backwards compatibility, not only do you break other existing implementations, but in a system where events can’t be migrated to the new format you also break all historical data, even if it was published by your own app.

At the same time, breaking backwards compatibility can free you up to improve the protocol in important ways. In many cases, it also won’t have much of an impact. Supporting historical data is important for some applications, like archival services, but many applications have a strong recency bias such that they don’t necessarily mind throwing away old data.

I think there’s a balance to this. I don’t think we can be backwards compatibility maximalists, but we also shouldn’t be careless about throwing away old formats for no reason. There is a perverse impulse among software developers to refactor for dumb reasons which should absolutely be avoided.

When introducing new backwards-compatible functionality to Nostr, it’s usually best to enhance existing data formats as long as it doesn’t result in overloading a single kind with multiple use cases.

When breaking backwards compatibility however, it’s best to create an entirely new kind and evangelize for migrating to it. This is much harder than modifying the behavior of existing event kinds, but it’s far more polite. The downside is it breaks network effects - and can still result in a poor UX for clients that don’t adopt the new format if it results in missing content.

Over time, Nostr protocol development has gotten increasingly conservative. As use cases proliferate, it becomes more difficult to get feedback on new data formats. As implementations proliferate, it becomes more difficult to advocate for breaking changes. Nostr specifications are not sacred, but their effectiveness relies almost entirely on interoperability. Maintaining compatibility requires conscientiousness, communication, and contributions to other projects - there’s no better way to get people to listen to you than writing code to improve their implementation.