Content-addressable storage gives an AI record an identity that is a hash of its own bytes, computed at the trust boundary before the model is called. Canonicalise the encoding (RFC 8785 JCS for JSON), never the content. Store payloads by digest; keep occurrences in an append-only event log. Content addressing proves two records are identical. It proves nothing about tampering.
Ross Jones — Founder, The Hopium Lab. Last modified 22 July 2026.
Two definitions decide everything below. Encoding canonicalisation is a value-preserving byte transform — key order, number formatting, string escaping, character encoding — that changes bytes and never meaning. Semantic normalisation is a lossy transform of the content itself — lowercasing, trimming, rounding, unit conversion, redaction, schema migration — and is a function of your current build rather than the value. One runs before the hash. The other runs after.
Why hash the content instead of minting a UUID?
A digest answers a question no UUID can: is this the same input, byte for byte, as something already recorded — no coordinator, no lookup table, no trust in whoever minted the identifier. Two services that never speak reach one identity for one payload.
Digests do not replace UUIDs. Idempotent agent tool calls pins attempt identity to a tuple — run id, step index, tool name, argument digest — because a hash cannot tell a retry from a person who genuinely wants the second reminder sent. The digest identifies the payload; the tuple identifies the occurrence.
Content addressing answers whether two things are the same. It says nothing about how many times something happened. An audit trail needs both, and they are different columns.
Deduplication arrives free, and free deduplication is a hazard for audit records. Key an event on the payload digest and the second identical prompt overwrites the first, so the log under-reports the number a regulator asks for. Build two stores: a content-addressed payload store keyed on the digest, and an append-only event log carrying that digest plus sequence, timestamp and actor.
Where does normalisation belong, before or after the hash?
Encoding canonicalisation belongs before the hash, semantic normalisation after it, and collapsing the two is how an audit trail rots. A digest over unsorted keys is a digest over an accident of your serialiser. A digest taken after a normaliser deploy stops identifying the input and starts identifying the input as processed by whatever build was live — because normalisers are code, and code ships.
Canonicalise the encoding. Never canonicalise the content. The first is a function of the value. The second is a function of your current build.
Version the normaliser and record the version in the event row. A transform with no version string is an unlogged input.
What does RFC 8785 actually pin down?
RFC 8785 pins key order, string escaping and UTF-8 encoding, and defines no number formatting of its own — delegating that entirely to ECMAScript. Less than the pages citing it assume, in other words. RFC 8785 (June 2020) is Informational, not Standards Track, so calling it the IETF standard for canonical JSON discredits everything downstream. Key sorting runs over UTF-16 code units as unsigned integers: neither alphabetical nor UTF-8 byte order. Number formatting lands on ECMA-262 10th Edition (ES2019) section 7.1.12.1 — now section 6.1.6.1.20 in the tc39 living specification, so citing the old number without the edition sends readers to a section that no longer exists.
Five consequences that "just sort your keys" never reaches:
- I-JSON applies: numbers must fit IEEE 754 doubles, so money and high-precision decimals belong in strings. Safe integers stop at 9007199254740991 (2^53 − 1); 9007199254740992 is already indistinguishable from its successor.
- Negative zero serialises to zero. A confidence of -0.0 and one of 0.0 are one record.
- NaN and Infinity must abort a compliant implementation, so a −Infinity logprob arriving as a native float crashes the canonicaliser rather than logging.
- Unicode normalisation is out of scope. NFC café and NFD café are two records, and macOS filesystems and iOS keyboards hand you both forms of one user's input.
- Surrogate pairs sort by surrogate code units, placing astral-plane keys before U+E000–U+FFFF, so a Rust implementation sorting UTF-8 bytes disagrees with a JavaScript one.
The security boundary sits in front of the canonicaliser, in the parser. Bishop Fox's survey of 49 JSON parsers found seven taking the first duplicate key — Go's jsonparser and gojay, rapidjson, json-iterator, Elixir's Jason and Poison, Erlang's jsone — while most take the last. JCS forbids duplicate property names and never sees one, because the parser already resolved it. Reject at parse. Never resolve.
Should you canonicalise at all, or hash the bytes you received?
Hash the bytes you received, whenever you own the boundary and can store them verbatim. JWS compact serialisation base64url-encodes the raw payload for that reason, and the parser-differential attack class disappears with it. RFC 8785's own introduction points at the XML signature era as the cautionary tale.
Canonicalisation earns its place only where identity must be re-derived from a structured object whose original bytes are gone: cross-service reconciliation, re-hashing after a storage migration, checking a record rebuilt from a database. Deterministic encoding in RFC 8949 (STD 94) section 4.2 is Standards Track where JCS is not, its map-key ordering differs from the Canonical CBOR of the obsoleted RFC 7049, and draft-ietf-cbor-cde (-13) is a draft, not an RFC.
| Identity strategy | Same input, same id? | Cost | Where it breaks |
|---|---|---|---|
| Digest of received bytes | Yes | Originals kept verbatim, forever | You never held the bytes |
| JCS, then digest | Within I-JSON | Parser hardening, doubles only | Duplicate keys resolved upstream; NFC/NFD; -0 |
| Deterministic CBOR (RFC 8949 §4.2) | Yes | Binary payloads, thinner tooling | "Canonical CBOR" cited without an ordering |
| Digest after semantic normalisation | No | None visible, which is the trap | The deploy that changes the normaliser |
| UUID minted at the call site | No | Coordination | Regenerated on the retry path |
| UUID checkpointed before the effect | Per occurrence | Durable state written first | It never tells you two payloads were identical |
Why compute the digest before the model call?
Because nothing downstream of the request is reproducible, and no client-side setting makes it so. Non-determinism in hosted inference is server-side: load moves batch size, and batch size moves reduction order inside normalisation, matmul and attention. Temperature zero and a fixed seed do not touch it — mechanism and measurements in reproducible LLM systems. Pin identity at the boundary or lose it.
What do Git and IPFS already know about this?
Git and IPFS both learned that content addressing gives a stable identity for a fixed serialisation and nothing above it for free. IPFS says so in its own documentation: "Two identical files can produce different CIDs" (IPFS docs) — chunk size, DAG layout, codec and CID version each change the address. JSON key order is that problem one layer down.
Git's object id is not a hash of your file either. Git prefixes a type-and-length header, blob <size>\0, hashes header plus content, and compresses afterwards, so sixteen bytes of what is up, doc? become bd9dbf5aae1a3862dd1526723246b20206e5fc37 — a digest no plain sha1sum reproduces. That header is domain separation, the mechanism behind the leaf and node prefixes in Merkle domain separation. Content addressing is a protocol, not a hash call.
Plain SHA-1 left Git in 2.13.0 (2017) for SHA-1DC, which detects the SHAttered disturbance vectors and errors out; the colliding PDFs do not collide inside a repository. The SHA-256 object format lost its experimental label in 2.42 (August 2023) and gained initial SHA-1/SHA-256 interoperability in 2.45 (April 2024).
Hash agility belongs inside the identifier. A multihash prefixes a registry code for the function, so a CID describes its own algorithm. A bare sha256: in a schema describes nothing, and Diablo, my production-readiness auditor, treats an unversioned digest field as a finding.
Does content addressing make records tamper-proof?
No: anyone who can rewrite stored content can recompute the digest and rewrite the index. The OCI image specification encodes the correct direction of trust — content is verified against a digest already held, and trust never flows back from content to the digest naming it.
A digest binds content only for a verifier who already holds that digest by a path the adversary cannot rewrite. Content addressing relocates trust. It does not remove it.
Four properties content addressing does not supply: ordering and completeness need an append-only Merkle log; authorship needs a signature over the digest; existence at a time needs an RFC 3161 timestamp token; non-equivocation needs a witness you do not control. All four are the primitives in tamper-evident without a blockchain, where an evidence layer begins rather than ends.
How do you erase from a store that cannot delete?
Encrypt payloads under a per-subject key and compute every digest, inclusion proof and anchor over the ciphertext. Destroying the key satisfies an erasure request without invalidating a proof already issued, at the price of deduplication scoping to one subject.
The "it is only a hash" defence needs care. The CJEU held in C-413/23 P (EDPS v SRB, 4 September 2025), an EU-institutions data protection case whose reasoning reads across, that pseudonymised data is not automatically personal data for every party — but the test is contextual re-identification by that recipient, not a categorical exemption. A SHA-256 over a low-entropy input such as an email address or an order id falls to brute force in seconds. Record-keeping duties under Article 12 of the EU AI Act do not suspend erasure rights, and architecture supports a conformity case rather than conferring one.
Where does content addressing fail in production?
Six failures recur, named here because unnamed failure modes never get budget.
- Encoding drift. One logical record, two digests, because a client changed key order, number formatting or Unicode form.
- Normaliser drift. A semantic transform crept in front of the hash, so identity tracks your build rather than the input.
- Duplicate-key resolution. The parser picked a winner silently and canonicalisation certified the winner.
- Occurrence collapse. The payload digest became the event's primary key, the second identical run vanished, and the log under-reports.
- Hash lock-in. No algorithm identifier in the digest field, so moving off SHA-256 means rewriting every record.
- Digest re-identification. A hash over a low-entropy identifier was treated as anonymous, published, and reversed.
What do you actually build?
Two stores, one strict parser, and a digest taken before anything non-deterministic runs.
CONTENT-ADDRESSED AI RECORDS — identity rules — v1.0 (22 July 2026)
The Hopium Lab. Supports a conformity case. Does not confer one.
ENCODING CANONICALISATION is a value-preserving byte transform — key order,
number formatting, string escaping, character encoding — that changes bytes
and never meaning. It runs BEFORE the hash.
SEMANTIC NORMALISATION is a lossy transform of the content itself —
lowercasing, trimming, rounding, unit conversion, redaction, schema
migration — and is a function of your current build. It runs AFTER the hash.
def ingest(raw_bytes, ctx):
obj = strict_parse(raw_bytes) # rejects duplicate keys, non-double
# numbers, NaN, Inf. Never repairs.
if ctx.own_the_boundary:
payload, profile = raw_bytes, "raw" # hash what actually arrived
else:
payload, profile = jcs(obj), "jcs-2020" # RFC 8785, re-derivable later
stored = seal(payload, subject_key(ctx)) # AES-SIV, deterministic: same
# payload, same ciphertext,
# within one subject
body = b"airec-v1\x00" + stored # domain separation, as Git does
digest = f"sha256-{profile}:" + sha256(body).hex() # algorithm AND profile
blobs.put_if_absent(digest, stored) # dedup lives here, only here
events.append({ # append-only. NEVER keyed on digest.
"seq": events.next_seq(), "payload": digest,
"normaliser": ctx.normaliser_version, # which build, if one ran
"run_id": ctx.run_id, "actor": ctx.actor, "at": ctx.timestamp,
})
return digest # computed BEFORE the provider call
CHECKS
[ ] Digest computed at the trust boundary, before any provider call
[ ] Parser rejects duplicate keys and out-of-range numbers, never repairs
[ ] Digest field carries hash function AND canonicalisation profile
[ ] Nothing lossy before the hash; normaliser version recorded if after
[ ] Event log keyed on sequence, never on payload digest
[ ] Unicode form fixed at ingest, or documented as unfixed
[ ] Digests over ciphertext wherever payloads carry personal data
[ ] A test asserts two byte-different, logically identical inputs agree — AND
that two identical payloads still produce two event rows
A test that fails when two identical payloads produce one event row is the check that survives you. Conventions get deleted in a refactor. A red test does not.
A record whose identity is a hash of itself can be checked by a hostile reader months later, holding nothing but the digest and the bytes. Every other identity scheme asks that reader to trust your database.
Ross Jones, Founder, The Hopium Lab. Last modified 22 July 2026. Engineering guidance, not legal advice.