Skip to content

Live operations and the MCP layer

Section 7 · From docs/RESILIENCE.md on GitHub

7.1 The guard

LiveOperationGuard (resilience/guard.py) is pure asyncio with no MCP or provider imports. It decides whether a live operation may start:

  • Single-flight: a call whose key equals an operation already running joins it and gets the same result (or exception) instead of starting a second collection.
  • Concurrency caps: at most max_concurrent_per_scope live operations per scope and max_concurrent_total overall. Extra callers get LiveOperationBusy, or wait for a slot with wait=True.
  • Cooldown: after an operation touching a scope finishes, successfully or not, a new one for that scope is refused for the cooldown with CooldownActive. A failure is often throttling, which is why failures cool down too (cooldown_after_failure=True).
  • Caller quotas: at most caller_max_operations new live operations per caller per window (CallerQuotaExceeded). Off by default.
  • Operation timeout: an operation still running after operation_timeout_seconds is cancelled, and every caller waiting for it gets LiveOperationTimeout. A running operation is also cancelled when every caller waiting for it has gone (each was cancelled, by a client or a tool timeout), so an abandoned collection cannot keep its scopes busy.

Every rejection subclasses LiveOperationRejected and has reason (cooldown, busy, quota or timeout), retry_after and to_dict(): {"reason", "message"} plus retry_after_seconds, scopes and caller when they are set. A scope is any string; the provider for a per-provider cooldown is the part before the first / or :. The limits can be passed as keyword arguments or as one GuardSettings object (LiveOperationGuard(GuardSettings(...), clock=...), with keyword arguments overriding it; an unknown keyword raises TypeError). LiveOperationGuard.from_config(config) maps the ratelimit.live_* keys, live_operation_timeout_seconds (default 3600, null for no limit) included, and each provider's live_cooldown_seconds. guard.status() reports operations in flight, active counts per scope, the seconds left on each cooling scope, and how many operations started and joined.

7.2 In the MCP layer

The workspace owns one guard, workspace.live_guard, built from config.ratelimit on first use. The live tools map_inventory, collect_assets, run_pipeline and run_scanners run like this:

  1. arguments and the result dataset name are validated;
  2. force=true is checked: only principals with the admin or operator role may use it, others get an access error;
  3. the credential preflight runs (MCP guide, section 15); for run_scanners only when Prowler or ScoutSuite is enabled, since the other scanners need no cloud access;
  4. the collection runs inside guard.run(...). run_scanners always runs under the guard, so identical scans share one run; it uses the cloud scopes for cooldowns and caps only when a cloud scanner is enabled.

The guard scopes are derived from the config: aws/<account> for each configured account, else aws/profile:<name>, else aws; azure/<subscription> or azure; gcp/<project>, else gcp/org:<id>, else gcp. The single-flight key covers the operation, the providers, the scopes and the tool's arguments, so only truly identical calls share a run. A call that joined another gets "joined": true and a note in its result.

A refusal is a tool error (isError: true) with error code -31029, the code of RateLimitedError. Its data is the guard's to_dict() plus use_dataset: the newest live dataset whose providers include every provider of the refused operation, so an agent refused an AWS map is never pointed at a GCP-only dataset. When there is one, the message tells the agent to query it instead of collecting again; otherwise it suggests list_datasets. For a cooldown it also says that an admin or operator can pass force=true. force only skips the cooldown; the concurrency caps and caller quotas still apply.

Two read-only views show the state without calling any cloud API: the tool rate_limit_status and the resource cloudg://ratelimit. Both return {"live_guard": {...}, "throttling": {...}}: the guard's status plus its cooldown, concurrency and quota settings, and the process governor's lifetime summary, including tripped breakers and slowed buckets. All four live tools run their collection inside its own stats_scope(). When the run was throttled, the answer includes a throttling block with the totals, up to 20 messages and the skipped scopes, taken from InventoryResult.throttling for map_inventory and from the run's stats for the others. map_inventory, collect_assets and run_pipeline also keep the full summary in the new dataset's metadata. The per-tool reference is in MCP_TOOLS.md.

This is separate from the MCP policy's own rate limits, which count calls per principal and tool (see MCP_PRIVACY.md). The policy decides how often someone may call a tool; the guard decides whether the cloud can take another collection right now.