Skip to content

How pipelines are assembled

Section 10 · From docs/MCP_PRIVACY.md on GitHub

10.1 Transform specs

A transform is named by a string or described by a mapping:

transforms:
  - sanitize                                       # name only
  - type: redact                                   # type plus options
    id: secrets-only                               # optional, defaults to the type
    options:
      strategies: {secret: redact}
  - {type: project, max_list: 50}                  # options may also be inline

type may also be spelled transform or name. Names are lower-cased and hyphens become _. The registered types and their aliases:

Type Class Aliases
sanitize UntrustedTextGuard sanitise, untrusted, untrusted_text
redact Redactor redaction, redactor, dlp
project Projection projection, shape
annotate Annotator annotation, annotations, annotator
substitute Substitution substitution
alias AliasMap aliases
regex_replace RegexReplace none
rename_keys KeyRename rename
template TemplateField templates
depseudonymize Depseudonymizer depseudonymise, depseudonymizer
guard_secrets SecretArgumentGuard secret_guard, reject_secrets

build_transform(spec, **context) creates the transform, passing the options as keyword arguments. The policy supplies context: its vault to redact and depseudonymize, its custom detectors to redact and guard_secrets, and its name to annotate as profile. Unknown options raise ValueError("Bad options for transform ...").

Your own transform types plug in with register_transform:

from cloudg.mcp.transforms import register_transform

class Upper:
    name = "upper"
    def apply(self, value, ctx):          # must not mutate value
        return value.upper() if isinstance(value, str) else value

register_transform("upper", Upper)        # policies can now say "- upper"

A transform is anything with a name and apply(value, ctx). ctx is a TransformContext holding the principal, the spec, the kind (tool, resource, prompt, completion), the direction (output or input), the vault, and the report dictionary that becomes _meta["cloudg/transforms"]. Built transforms are cached per policy and shared between pipelines with the same spec, so they must be stateless across calls (caches are fine).

10.2 The output pipeline

Policy.output_pipeline(spec, principal):

  1. Start from the policy's transforms.
  2. For each matching rule, in order: combine its transforms according to transforms_mode (append adds at the end, prepend at the start, replace throws away what was built so far), and deep-merge its transform_options into the running options. A rule's honor_hints, if set, overrides the policy's.
  3. Move every alias or substitute step that comes before the last redact step to just after it, keeping their order. Aliases are presentation: the redactor has to see the real values (an aliased account inside an ARN would hide the ARN from its detector), and an alias applied afterwards still labels the pseudonym of the value it names. This is why the soc-analyst alias rule can simply append.
  4. Apply the merged options. A transform receives the options stored under its type and then those under its id; canonical names work as keys (transform_options: {redaction: ...} reaches redact).
  5. Apply the tool's hints (10.4).
  6. Build every remaining spec into a transform; if nothing is left, the result is the shared IDENTITY pipeline.

Pipelines are cached per primitive and per role set.

10.3 The input pipeline

Policy.input_pipeline(spec, principal) is the policy's input_transforms, followed by each matching rule's input_transforms, minus anything the hints skip. transform_options do not apply to input transforms.

A Depseudonymizer is appended automatically when the caller's output pipeline can produce something that needs reversing:

  • a redact transform with at least one pseudonymize strategy (and pseudonymisation not disabled by a hint),
  • a substitute transform with aliases, or an alias transform,
  • a sanitize transform in fence mode, so fenced strings the model copies back lose their fence markers.

standard's input pipeline is therefore guard_secrets then depseudonymize; open's is empty. The depseudonymiser is skipped when a hint sets depseudonymize: false or lists it in skip_input.

10.4 Tool transform hints

A tool can ask for different treatment through ToolSpec.transform_hints (the transform_hints= argument of @registry.tool):

Hint Effect Kind
skip: [names] Remove these output transforms (by type or id; canonical names accepted). "*" or "all" removes every one. relaxing
pseudonymize: false (or pseudonymise) Every redactor turns pseudonymize into keep for this tool. relaxing
project: {...} (or projection) Deep-merged into the project transform's options. either
transforms: [specs] (or extra) Appended to the output pipeline. restrictive
skip_input: [names] Remove these input transforms. relaxing
input_guard: false Remove guard_secrets from the input pipeline. relaxing
depseudonymize: false Do not add the depseudonymiser. relaxing

With honor_hints: false (policy-wide, or in a rule) only transforms and extra survive, so a tool can add protection but not remove it. Two catalog tools use hints. preview_transform skips every output transform (its result is already the policy's output) and opts out of input guarding and depseudonymisation (so you can preview secrets, and so a pseudonym you paste is not quietly reversed). reveal_token turns off pseudonymisation of its own output, skips the substitute and alias steps, and opts out of both input transforms, because it has to see the pseudonym and return the real value untouched.

10.5 The generic pipeline

preview_transform without a tool, privacy_status and list_detectors use the policy's generic pipeline: the policy's transforms with the policy's transform_options and the options of rules that apply without a primitive (rules matching only on roles or principals). Rule transforms are not included, so the soc-analyst alias rule does not show up in a generic preview. The reordering of step 3 in section 10.2 is not applied either, so a policy whose own transforms list puts an alias before redact previews differently from what tools return (section 18). Pass tool= to preview the exact pipeline of a tool.

10.6 The transform report

Each transform adds what it did to ctx.report. The layer returns the report as _meta["cloudg/transforms"] on tool results and on resource contents, and AuditLogMiddleware copies it into the audit line. Keys you can see:

Key Written by Content
redacted, masked, hashed, pseudonymized, generalized, dropped redact {entity: count}; names replaced inside free text count as resource_name_mention
redacted_arguments guard_secrets in redact mode {entity: count}
untrusted sanitize stripped_chars, sanitized_fields, suspicious, truncated, paths, action, notice
projection project excluded, keys_dropped, lists_truncated, strings_truncated, depth_limited, budget
truncated project true when the budget could not be met
aliased, regex_replaced, keys_renamed, templated substitute and friends counts
depseudonymized depseudonymize (input side, not returned) count
annotations, content_annotations annotate section 11.6
truncated_chars layer length of the text after the max_output_chars cut