Skip to content

The token vault

Section 13 · From docs/MCP_PRIVACY.md on GitHub

TokenVault (cloudg/mcp/transforms/vault.py) turns values into pseudonyms and back. Each Policy owns one, built from vault:, unless one is passed in (Policy.load(..., vault=v) or derive()), which lets several layers share pseudonyms.

13.1 Keys and determinism

The raw key comes from vault.key in the policy, else the environment variable named by vault.key_env (CLOUDG_MCP_VAULT_KEY), else 32 random bytes. stats() reports which (config, env, random) as key_source, and a 12-hex key_id that identifies the key without revealing it. Six sub-keys are derived with HMAC-SHA256 under fixed labels: tokenising, prefix-preserving IP permutation, the hash strategy, file encryption, file authentication, and the key id.

A pseudonym is derived from HMAC(key, namespace | entity type | attempt | value) and formatted to look like the value it replaces. The same key, namespace, entity type and value give the same pseudonym on every call, in every process. That is what makes pseudonyms useful across a conversation and across restarts.

With a random key every restart issues new pseudonyms, and pseudonyms from an earlier conversation stop resolving. Set CLOUDG_MCP_VAULT_KEY (from a secret store, not a policy file in git) whenever pseudonyms must survive a restart.

If a new pseudonym collides with one already issued in the namespace for a different value, the vault re-derives with the next attempt number (up to 64; after 48 it falls back to the generic prefix-hex format). An alternative pseudonym therefore depends on which value was seen first, so it is not reproducible across processes unless the vault file is persisted. With 12-digit and 10-hex formats collisions are rare, and the vault counts them in stats()["counters"]["collisions"].

13.2 Formats

Measured with key docs-example-key:

Entity Real value Pseudonym
aws_account_id 123456789012 192604312079
aws_arn arn:aws:ec2:us-east-1:123456789012:instance/i-0abc1234def567890 arn:aws:ec2:us-east-1:192604312079:instance/i-ccc5e66b39a18abd0
aws_arn arn:aws:iam::123456789012:role/service-role/deploy arn:aws:iam::192604312079:role/res-fffbcad49c/res-24015814c5
aws_arn arn:aws:s3:::prod-data arn:aws:s3:::res-01c3f9edb0
aws_arn arn:aws:lambda:us-east-1:123456789012:function:api:$LATEST arn:aws:lambda:us-east-1:192604312079:function:res-d5c846d2b7:$LATEST
azure_resource_id /subscriptions/1b2c3d4e-1111-2222-3333-444455556666/resourceGroups/prod-rg/providers/Microsoft.Compute/virtualMachines/web-vm-01 /subscriptions/1ab68968-cc4d-87d1-8e82-b2d5f7f95a11/resourceGroups/rg-e29e001059/providers/Microsoft.Compute/virtualMachines/res-e6d7502940
azure_subscription_id 1b2c3d4e-1111-2222-3333-444455556666 1ab68968-cc4d-87d1-8e82-b2d5f7f95a11
gcp_resource_name //compute.googleapis.com/projects/acme-prod-1/zones/us-central1-a/instances/web-1 //compute.googleapis.com/projects/proj-05215474/zones/us-central1-a/instances/res-3c1f1e6b50
gcp_resource_name projects/123456789012/secrets/db projects/988508927314/secrets/res-373ed2f827
gcp_project_id acme-prod-1 proj-05215474
private_ip 10.0.1.5, 10.0.1.77 10.54.186.36, 10.54.186.99
private_cidr 10.0.1.0/24 10.54.186.0/24
private_cidr (interface) 10.0.1.5/24 10.54.186.36/24
private_ip 10.0.2.5 10.54.184.29
private_ip 192.168.4.20, 172.20.3.4, 100.64.1.1 192.168.83.226, 172.21.75.180, 100.72.64.136
public_ip 54.12.33.4 198.19.25.195
public_cidr 54.12.33.0/24 198.18.245.0/24
public_cidr 52.0.0.0/8 247.0.0.0/8
public_ip (v6) 2600:1f18:abcd::5 2001:db8:6512:bb37:80c6:8b30:9ec5:c732
private_ip (v6) fd12:3456:789a::1 fc24:210:8aab:6532:70a3:d9b1:c872:1498
special 0.0.0.0/0, 169.254.169.254, 10.0.0.0/8 unchanged
email alice@corp.com user-f53165f9@d-a9aaafa8bb.example
email (GCP service account) ci@acme-prod-1.iam.gserviceaccount.com sa-3777b2e6@proj-05215474.iam.gserviceaccount.com
email (numeric service account) 123456789-compute@developer.gserviceaccount.com 451309491-compute@developer.gserviceaccount.com
hostname (cloud) my-lb-1.us-east-1.elb.amazonaws.com h-c4293743bd.us-east-1.elb.amazonaws.com
hostname (other) db.corp.com host-ae8a86b8.example
hostname (only service labels) s3.amazonaws.com unchanged
resource_name (AWS id) i-0abc1234def567890 i-ccc5e66b39a18abd0
resource_name web-1 res-3c1f1e6b50
aws_access_key_id AKIAIOSFODNN7EXAMPLE AKIA plus 16 pseudonymous upper-case letters and digits (same shape, not a usable key)
mac_address 00:1A:2B:3C:4D:5E 7A:21:08:5D:DD:CF
person, tag_value alice, payments person-97eed0ceb8, tag-3f91de21dd
cloudg_uri cloudg://assets/web-1 cloudg://assets/res-3c1f1e6b50
cloudg_uri cloudg://assets/arn:aws:ec2:us-east-1:123456789012:instance/i-0abc1234def567890/neighbors cloudg://assets/arn:aws:ec2:us-east-1:192604312079:instance/i-ccc5e66b39a18abd0/neighbors
cloudg_uri cloudg://datasets/prod/summary cloudg://datasets/ds-65d25a1594/summary
dataset_name prod ds-65d25a1594
url_username alice user-e2d4d2de32
cloud_account prod acct-b7955d8854
any other entity whatever (entity custom_thing) customthin-dcd402240d

The rules behind the table:

  • Digits (aws_account_id, gcp_project_number, number) keep their length; the first digit is non-zero unless the original's was. A non-numeric value of a digit type gets the generic format (awsaccount-...).
  • GUID types (azure_subscription_id, azure_tenant_id, uuid) become RFC 9562 version 8 UUIDs (the 8 in the third group marks them as synthetic); upper case is preserved.
  • ARNs are rebuilt component by component. Partition, service and region stay. A 12-digit account becomes the account's own pseudonym, so the ARN and the account_id field agree. The resource part is split on / and :; the first segment (the resource type: instance, role, function) stays unless the service is S3 or there is only one segment; empty, *, $... segments and short numbers after the first stay; every other segment becomes a resource_name pseudonym. Each component is registered in the vault on its own, so a pseudonymised account can be looked up by itself.
  • Azure IDs: values after subscriptions, tenants and resourceGroups are pseudonymised (as rg-... for groups), the provider namespace and types stay, resource names become res-....
  • GCP names: collection names stay; values after zones, regions and locations, and -, stay; projects become project pseudonyms, numeric organizations, folders and billing accounts become same-length numbers, everything else res-....
  • E-mails: the domain is pseudonymised on its own (d-...) and .example appended, so all users of one domain share it. GCP service accounts keep their shape.
  • Hostnames: under a known cloud suffix (amazonaws.com, cloudfront.net, azurewebsites.net, windows.net, googleapis.com, run.app, internal, compute.internal and others) region labels and service labels such as elb, rds, s3, blob, vault stay and the other labels become h-.... Other hostnames become host-<8 hex>.example.
  • cloudg URIs keep their scheme, collection and suffix (/neighbors, /findings, /summary); the reference inside is pseudonymised as the entity it looks like (an ARN as an ARN, an IP as an IP, anything else as a resource_name), so it matches the pseudonym the same value gets under id. Dataset names become ds-<10 hex>.
  • Resource names in the AWS prefix-hex shape (i-, sg-, vpc-, subnet- with 8 to 40 hex digits) keep their prefix and length; other names become res-<10 hex>; *, empty and $... values are left alone.
  • Access key IDs keep their 4-letter prefix (AKIA, ASIA, AROA...) followed by 16 base32 characters.
  • MAC addresses become locally administered unicast addresses with the same separator and case.
  • Private IPv4 and IPv6 addresses stay inside their private block (10/8, 172.16/12, 192.168/16, 100.64/10, fc00::/7) and are permuted with a keyed, prefix-preserving permutation in the style of Crypto-PAn. Two addresses that share an n-bit prefix inside the block have pseudonyms sharing an n-bit prefix, so subnets stay subnets: 10.0.1.5 and 10.0.1.77 land in 10.54.186.0/24, which is the pseudonym of 10.0.1.0/24, and 10.0.2.5 lands outside it. An interface address such as 10.0.1.5/24 keeps its host part. A CIDR as wide as its block or wider (10.0.0.0/8) is unchanged.
  • Public IPv4 addresses are mapped into 198.18.0.0/15, the RFC 2544 benchmarking range; public CIDRs wider than /15 go to 240.0.0.0/4 (and /0 to /3 stay as they are). Public IPv6 goes to the documentation range 2001:db8::/32 (prefixes shorter than /32 stay). Public mappings are random within the pool, not prefix-preserving.
  • Special addresses and /0 routes are never pseudonymised.

13.3 Namespaces

With scope: global (the default) all callers share one namespace. With scope: principal the namespace is p:<principal id>: each caller gets different pseudonyms for the same value, and cannot reverse anyone else's, because the namespace is part of the HMAC input and of the lookup. Measured with {"extends": "strict", "vault": {"scope": "principal"}}: web-1's ARN came out as arn:aws:ec2:us-east-1:560207693111:instance/res-fb264e8d29 for principal a and arn:aws:ec2:us-east-1:118326746920:instance/res-c6428e008a for b. When b sent a's ARN, nothing was reversed, the handler found no such asset, and the error quoted the ARN pseudonymised once more in b's namespace. Use principal scope when callers should not be able to correlate their results, and global scope when they share findings with each other.

13.4 TTL

ttl_seconds forgets a mapping that has not been used for that long (the clock restarts on every use). An expired pseudonym no longer reverses; it is dropped when someone tries, and purge_expired() drops the rest. Because pseudonyms are deterministic, the same value gets the same pseudonym again the next time it is seen, so a TTL limits how long a pseudonym can be reversed, not what it looks like.

13.5 Persistence

With path set, the vault loads the file when it is created, and it is saved in three situations: by policy.save_vault() (or vault.save()), which cloudg mcp serve calls when the server stops; by an atexit hook that saves every vault with a path that changed since its last save; and, with autosave: true, after every transform pass that issued new pseudonyms. policy.save_vault(path) can also write a copy elsewhere, and returns None when no path is configured. Measured: a process that pseudonymised one ARN under a strict-based policy with a path and then simply exited left the file behind.

A process killed with SIGKILL, or one that crashes hard, saves nothing; use autosave if every pseudonym must survive that.

Writes are atomic (a temporary file in the same directory, then os.replace) and the file is created with mode 0600. Measured with {"extends": "strict", "vault": {"path": ".../vault.json", "autosave": true}} after one get_asset:

vault file mode 0o600 size 1473
{"format": "cloudg-vault/1", "encrypted": true, "key_id": "07fcbfa77970",
 "nonce": "4oJIZusjiKRLeXJdxtNJCA==", "data": "OJDO+Ms7QqrMFrkAERMJenaLnzkZPU91fItohveT...",
 "tag": "19b0a683a2b0a8a0b00005dae83f461777597161..."}

A second layer built from the same policy and key loaded the 8 entries and resolved the pseudonymised ARN to arn:aws:ec2:us-east-1:111111111111:instance/i-0bast.

The encrypted format:

Field Content
format cloudg-vault/1
encrypted true
key_id Identifies the key, reveals nothing about it
nonce 16 random bytes, base64
data The plaintext XORed with a keystream of HMAC-SHA256(file key, nonce + 8-byte block counter) blocks, base64
tag HMAC-SHA256(MAC key, nonce + ciphertext), hex

The tag is checked before decryption; a wrong key or a modified file raises ValueError("Vault file authentication failed (wrong key or tampered file)"). The plaintext is the export() document. The construction is encrypt-then-MAC with separate keys, built from the standard library so the layer needs no crypto dependency; it has not been reviewed the way a library AEAD has. If that matters, keep the file on an encrypted volume as well. encrypt: false writes the plaintext export (fast, readable, and full of real identifiers).

13.6 Export, import and reversing text

vault.export(namespace=None) returns every mapping, real values included:

{"format": "cloudg-vault/1", "key_id": "07fcbfa77970",
 "entries": [["p:a", "resource_name", "web-1", "res-02025c5207", 1791470271.735539, 1791470271.735539],
             ["p:a", "aws_account_id", "111111111111", "560207693111", 1791470271.7356002, 1791470271.7356646]]}

(Taken from the principal-scoped vault above.)

The entry fields are namespace, entity type, real value, pseudonym, created and last used (Unix time). Treat an export as you would the dataset itself. import_mappings(data, overwrite=False) loads one and returns the number added; it accepts exports made under another key (the old pseudonyms reverse fine, new ones will differ). clear(namespace=None) forgets mappings.

detokenize(token) reverses one exact pseudonym; entry(token) also returns its entity type and creation time; lookup(value, entity) returns an existing pseudonym without creating one. detokenize_text(text) replaces every pseudonym it can find in free text. It first tries the whole string, then scans for pseudonym shapes: e-mail addresses, IPv4 and IPv6 addresses and CIDRs (a CIDR whose address part is a pseudonym is rebuilt with the same prefix length), GUIDs, MAC addresses, hostnames, access key IDs, prefix-hex tokens and runs of 6 to 20 digits. Inside e-mails and hostnames it replaces the embedded h-..., d-... and digit tokens. An ARN is reversed component by component, which also works for ARNs that were never issued whole:

acct = vault.lookup("123456789012", "aws_account_id")
vault.detokenize_text(f"arn:aws:sts::{acct}:assumed-role/x")
# 'arn:aws:sts::123456789012:assumed-role/x'

13.7 Limitations

  • Public CIDRs and the public addresses inside them are mapped independently. Measured: 54.12.33.4 became 198.19.25.195, 54.12.33.0/24 became 198.18.245.0/24, and the first is not inside the second. Private ranges do not have this problem.
  • A real address can collide with a pseudonym. With 10.0.1.5 pseudonymised as 10.54.186.36, a user who types the real address 10.54.186.36 (another host in your network) has it read back as 10.0.1.5 on the way in, because the input side cannot tell a real value from a pseudonym. The real 10.54.186.36 gets its own pseudonym (10.19.234.158) on the way out, so outputs stay consistent; only inputs are ambiguous. Under strict the model only ever sees pseudonyms, so this needs a human typing real private IPs into the conversation.
  • Pseudonymisation preserves format, not meaning. tag-3f91de21dd says nothing about the team, and res-... names hide naming conventions the model might have used.
  • Only values the detectors and key rules find are pseudonymised. Section 18 lists keys that hold names but are not covered.
  • A random key (no CLOUDG_MCP_VAULT_KEY) issues new pseudonyms at every restart.