Detectors

Section 12 · From docs/MCP_PRIVACY.md on GitHub

The detection model follows Microsoft Presidio and Google Cloud DLP: a detector only reports spans of an entity type with a confidence; the strategy is the redactor's business. Detectors are regular expressions with an optional validator; each also carries lower-case "hint" substrings, and its regex only runs on strings that contain one of them. Matches are taken leftmost first and a match that overlaps an earlier one is discarded, so an ARN swallows the account number inside it; when two matches start at the same position, the detector listed first wins.

12.1 Content detectors

The first detector in the list is cloudg_uri (category identifier, confidence 0.99). It matches cloudg's own resource URIs that name an asset or a dataset (cloudg://assets/web-1/neighbors, cloudg://datasets/prod/summary) and pseudonymises only the reference inside, so a URI in a result points at the same pseudonym as the asset's id. cloudg://findings/... and other URIs are not matched.

Secrets (category secret):

Detector	Entity	Conf.	Matches	Example that matches	Example that does not
private_key	private_key	1.0	PEM, OpenSSH and PGP private key blocks, to the END line or end of text	-----BEGIN OPENSSH PRIVATE KEY-----...	a PUBLIC KEY block
jwt	jwt	0.95	eyJ<header>.eyJ<payload>.<signature>	a complete JWT	eyJhbGciOiJIUzI1NiJ9 alone
url_credentials	password	0.95	The password in scheme://user:password@host (value only); placeholder check	redis://default:pa55word99@cache.internal reports pa55word99	https://alice@git.example.com (no password; see url_userinfo)
connection_string_secret	password	0.95	Value of Password=, Pwd=, AccountKey=, SharedAccessKey=, SharedSecret=, ClientSecret=; placeholder check	...;Password=S3cr3t!x; reports S3cr3t!x	Password=;
aws_secret_access_key	aws_secret_access_key	0.95	40 base64 characters within 20 characters of an aws ... secret or sk label	aws_secret_access_key = wJalrXUt...	the same 40 characters with no label (the high-entropy detector may still catch it)
secret_assignment	secret_value	0.85	Value after password, secret, client_secret, api_key, access_token, token, bearer, private_key, passphrase... followed by : or =; at least 4 characters; placeholder check	{"client_secret": "zq81Lm2Xp0"}, api_key: zq81Lm2Xp0	password = null, token: ${TOKEN}, api_key="<redacted>"
authorization_header	secret_value	0.9	Credential after Bearer or Basic, 16+ characters	Authorization: Bearer abcdefghijklmnop1234	Bearer abc123
github_token	api_token	0.99	ghp_, gho_, ghu_, ghs_, ghr_ plus 36+, github_pat_ plus 22+	ghp_ followed by 36 characters	ghp_short123
slack_token	api_token	0.99	xoxa-, xoxb-, xoxp-, xoxo-, xoxs-, xoxr- plus 10+	xoxb-1234567890-abcdefghij	xoxb-123
google_api_key	api_token	0.99	AIza plus 35	AIza plus 35 characters	AIzaShort
stripe_key	api_token	0.99	sk_, rk_, pk_ + live_ or test_ + 16+	sk_live_x1x1x1x1x1x1x1x1x1x1	sk_live_short
azure_sas_signature	secret_value	0.95	The sig= value of a SAS URL, 20+ characters	...&sig=AbCdEf0123456789%2BxyzQQ	sig=short
azure_storage_key	secret_value	0.8	Exactly 86 base64 characters plus ==	an 88-character storage key	85 characters plus ==
high_entropy_secret	secret_value	0.5	Opaque tokens of 32+ characters: mixed case and digits, not hex, fewer than three 4-letter word runs, at least 4 bits of entropy per character	Qm9vb3RzdHJhcEtleTEyMzQ1Njc4OTBhYmNkZWZn	40 hex digits, Zx9 repeated 12 times, my_password_is_long_and_wordy_value_here_ok

The placeholder check rejects empty values, null, none, true, false, undefined, redacted, n/a, empty, string, required, optional, runs of *, x, #, - or ., and anything starting with [redacted, ${, {{ or <.

Credentials and identifiers:

Detector	Entity	Category	Conf.	Matches	Matches	Does not match
aws_access_key_id	aws_access_key_id	credential	0.99	AKIA or ASIA plus 16 upper-case letters or digits	AKIAIOSFODNN7EXAMPLE	AKIA1234567890ABCDE (15)
aws_unique_id	aws_unique_id	identifier	0.95	AIDA, AROA, AGPA, AIPA, ANPA, ANVA, APKA, ABIA, ACCA plus 16 or 17	AROAJ2UCCR6DPCEXAMPLE	AIDA123456789012345 (15)
aws_arn	aws_arn	identifier	0.99	arn:<aws partition>:<service>:<region>:<12 digits, aws or empty>:<resource>	arn:aws:s3:::my-bucket, arn:aws-us-gov:...	arn:partition:ec2:x
azure_resource_id	azure_resource_id	identifier	0.99	/subscriptions/<GUID> and the path after it	/subscriptions/1b2c.../resourceGroups/prod-rg	/subscriptions/not-a-guid/...
gcp_resource_name	gcp_resource_name	identifier	0.95	//<svc>.googleapis.com/... and relative projects/, organizations/, folders/ names	projects/acme-prod-1/topics/orders	projects/ab/x (id too short), projects-list/zones
azure_subscription_ref	azure_subscription_id	identifier	0.9	A GUID after subscription id or tenant id and : or =	tenant_id: 9f86d081-...	a bare GUID
url_userinfo	url_username	pii	0.9	The user name in scheme://user@host when there is no password	https://alice@git.example.com/repo reports alice (pseudonym user-...)	https://git.example.com/repo
email	email	pii	0.95	local@domain.tld	alice@example.com	user@localhost, the user info of a URL
aws_account_id	aws_account_id	identifier	0.7	A bare 12-digit number not inside a longer token; all-same-digit placeholders rejected	account 123456789012	1234567890123, 000000000000, /aws/lambda/fn-123456789012, 123456789012.5
uuid	uuid	identifier	0.6	Any GUID	9f86d081-884c-4d63-9a2b-1c5e3f0a7b21

Network (category network):

Detector	Entity	Conf.	Matches	Matches	Does not match
ipv6	refined (below)	0.9	IPv6 addresses and CIDRs with real structure, validated with ipaddress	fe80::1, 2600:1f18::/32, ::1, ::/0	std::vector, a::b, ::
ipv4	refined (below)	0.9	Dotted quads and CIDRs, validated	10.0.1.5, 54.12.33.4/32, 0.0.0.0/0	999.1.1.1, 10.0.0.256, 1.2.3.4.5
mac_address	mac_address	0.8	Six hex pairs separated by : or -	00:1a:2b:3c:4d:5e	five pairs
hostname	hostname	0.7	Fully qualified names ending in a common TLD or internal, local, localdomain, lan, corp, intra	db1.prod.corp.com, ip-10-0-1-5.ec2.internal, my-bucket.s3.amazonaws.com	config.yaml, service.example.invalid

Temporal: timestamp (category temporal, 0.9) matches ISO 8601 date-times such as 2024-05-01T12:30:00Z, not bare dates.

In https://alice@git.example.com/repo, url_userinfo reports alice and hostname reports git.example.com; the email detector does not fire.

12.2 IP classification

The IP detectors refine their entity with classify_ip_text:

Class	Addresses	Entity
special	unspecified (0.0.0.0, ::), loopback, link-local (including the 169.254.169.254 metadata endpoint and fe80::/10), multicast, IPv4 reserved (240.0.0.0/4), 255.255.255.255; any /0 route	special_ip, special_cidr
private	10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 100.64.0.0/10 (CGNAT), IPv6 ULA fc00::/7	private_ip, private_cidr
public	everything else, including the documentation ranges	public_ip, public_cidr

A CIDR is classified by its network address. This is why strict can say special_ip: keep and special_cidr: keep: "SSH open to 0.0.0.0/0" and "IMDS at 169.254.169.254" stay readable, and they identify nobody.

12.3 Key rules

Keys are normalised before matching: camelCase is split and every run of non-alphanumerics becomes _, so SecretAccessKey, secret-access-key and secretAccessKey all become secret_access_key.

Rule	Entity	Category	Applies to keys	Extra
sensitive_key	sensitive_field	secret	Containing a word such as password, passwd, pwd, passphrase, secret(s), client_secret, api_key, access_key, secret_key, private_key, token(s), access_token, refresh_token, id_token, session_token, auth_token, bearer, credential(s), authorization, auth, cookie(s), connection_string, conn_str, sas_token, sas, signature, user_data, userdata, primary_key, secondary_key, account_key, shared_key, master_key, admin_password, kubeconfig, pem	Whole subtree; numbers too; safe-key exclusions below
account_id_key	cloud_account (typed by shape, section 11.2)	identifier	account_id, owner_id, account, account_ids, accounts, each optionally prefixed aws_ or cloud_	defers to detectors
subscription_key	azure_subscription_id	identifier	subscription, subscription_id, azure_subscription_id	defers
tenant_key	azure_tenant_id	identifier	tenant, tenant_id, azure_tenant_id	defers
gcp_project_key	gcp_project_id	identifier	project, project_id, gcp_project_id	defers
gcp_project_number_key	gcp_project_number	identifier	project_number	numbers too; defers
resource_name_key	resource_name	identifier	resource_name, display_name, bucket_name, function_name, instance_name, cluster_name, db_name, database_name, computer_name, vm_name, role_name, user_name, username, group_name, key_name, table_name, queue_name, topic_name, repository_name, asset_name, source_name, target_name, resource_names, asset_names	defers
resource_ref_key	resource_name	identifier	ref, refs, asset_ref(s), asset_id(s), resource_id(s), resource, asset, source, target, source_id, target_id, source_ref, target_ref, node, node_id, nodes, from, to, start, end, via, members, neighbors, neighbours, path	defers; lists of references too
dataset_key	dataset_name	identifier	dataset, dataset_name, datasets, active_dataset, base, base_dataset, target_dataset	defers; pseudonym ds-...
dataset_listing_name_key	dataset_name	identifier	name	Only in a dict that also has loaded_at or compliance_results (dataset listings); defers
asset_name_key	resource_name	identifier	name	Only in a dict that also has asset_type, arn, provider, resource_type, account_id or region, so tool and prompt names in catalog listings and policy descriptions are left alone; defers
asset_id_key	resource_name	identifier	id	Same sibling condition as asset_name_key; defers
hostname_key	hostname	network	hostname, host_name, dns_name, fqdn, private_dns_name, public_dns_name, domain_name, endpoint_address	defers

A key rule only does something when its strategy is not keep, so under standard only sensitive_key is active. "Defers" means the content detectors decide first when one of them matches the whole value (section 11.2). Custom key rules (extra_key_rules) can use requires_sibling, but defer cannot be set from configuration, so a custom rule always applies its own entity.

Safe-key exclusions stop sensitive_key from firing on keys that describe a secret instead of holding one. A key is excluded if it ends in _id, _ids, _arn, _name, _type, _state, _status, _usage, _enabled, _last_used, _last_used_date, _last_changed, _last_rotated, _age, _length, _count, _policy, _rotation..., _expir..., _created..., _date, _time, _version, _present, _required, _set, _configured, _manager, _source, _format, _algorithm, _kind, _ref, _location, _uri, _url, _endpoint, _header, _scheme, _method, _mode, _provider, _issuer, _audience, _ttl, _lifetime, _reset_required, _exists, _level, _size, _fingerprint, _thumbprint, _hint (and a few plurals), or starts with has_, is_, num_, require(s)_, allow(s)_, enable(d)_, use(s)_, max_, min_. So password_last_used, kms_key_id, has_password and token_expiry are left alone, while password, db_password and client_secret are replaced.

12.4 Inspecting detectors

list_detectors (and cloudg://privacy/detectors) shows each detector, the strategy the caller's generic pipeline gives it, the strategies of its refined entities and whether it is active. Under strict with category: network, abridged:

{
  "policy": "strict",
  "detectors": [
    {"name": "ipv4", "entity": "ip_address", "category": "network", "confidence": 0.9,
     "strategy": "pseudonymize",
     "refined": {"private_ip": "pseudonymize", "public_ip": "pseudonymize", "special_ip": "keep"},
     "active": true},
    {"name": "hostname", "entity": "hostname", "category": "network", "confidence": 0.7,
     "strategy": "pseudonymize", "active": true}
  ],
  "key_rules": [
    {"name": "hostname_key", "entity": "hostname", "category": "network",
     "key_pattern": "^(?:hostname|host_name|dns_name|fqdn|private_dns_name|public_dns_name|domain_name|endpoint_address)$",
     "strategy": "pseudonymize"}
  ]
}

12.5 Custom detectors

Declare detectors in the policy; they are added to every redactor and secret guard the policy builds.

detectors:
  - name: jira_ticket            # required, unique; replaces a built-in of the same name
    entity: ticket_id            # defaults to the name
    category: internal           # defaults to identifier
    pattern: '\b(?:SECOPS|INFRA)-\d{2,6}\b'   # required; numbered groups only
    group: 0                     # which group is the reported span (0 = whole match)
    confidence: 0.9              # default 0.8
    hints: ["secops-", "infra-"] # lower-case pre-filter substrings; empty = always run
    flags: [ignorecase]          # i, m, s, x or their long names; or ignore_case: true
    validator: null              # ip | secret | not_placeholder | entropy[:bits] | high_entropy[:bits]
    min_length: 1
    enabled: true
transform_options:
  redact:
    strategies:
      ticket_id: hash

Bad patterns fail when the policy loads. Without hints the regex runs on every string, which you will notice on large results. The validator names mean: ip validates and refines like the IP detectors, secret and not_placeholder apply the placeholder check, entropy:N requires N bits per character, and high_entropy:N applies the full opaque-token heuristic.

disabled_detectors: [email, hostname] turns built-ins off for the whole policy. Section 17.5 has a measured example.