How one call is paced¶

Section 2 · From docs/RESILIENCE.md on GitHub

call_with_resilience(fn, scope)
  1. circuit breaker      open?  -> CircuitOpenError, no request sent
  2. deadline             spent? -> DeadlineExceededError
  3. rate limiter         provider bucket, account bucket, leaf bucket (adaptive)
  4. bulkhead             in-flight cap per provider x account
  5. fn(...)              the API call
  6. on error             FATAL      -> re-raise at once
                          THROTTLED  -> halve the scope's rate, pause for Retry-After
                          THROTTLED or TRANSIENT -> sleep (decorrelated jitter,
                            never less than Retry-After) and go to 1, while
                            retries, deadline and retry budget allow
                          otherwise  -> breaker counts a failure, re-raise
  every step is counted in the run's stats; give-ups also go to the task's
  throttle ledger, which turns its coverage record PARTIAL

That is the full sequence of call_with_resilience. The SDK integrations in section 3 reuse the same parts but let the SDK do the retrying, so not every step applies to every provider. The table in section 5.4 says which.

2.1 Scopes¶

Every call is attributed to a Scope(provider, account, region, service, operation). Its string form is what you see in telemetry: aws/123456789012/us-east-1/ec2, with * for a missing part and the operation appended when there is one (aws/123456789012/us-east-1/ec2/DescribeSubnets).

Provider	account	region	service	operation
AWS	the account id the session was instrumented with; `*` for the base session before role assumption	the client's region	botocore service name (`ec2`, `iam`, `sts`)	API name (`DescribeSubnets`)
Azure	subscription id, lowercased; none for Resource Graph	none	resource-provider namespace without `Microsoft.` (`network`, `compute`, `storage`); `resources` for paths without one; `resourcegraph`	none
GCP	`projects/<id>`, `organizations/<id>` or `folders/<id>`	none	`cloudasset`	RPC name (`ListAssets`, `SearchAllResources`, `SearchAllIamPolicies`)

Telemetry is aggregated per service scope (operation removed). Rate-limiter leaf buckets, and the circuit breakers with them, go down to the operation where the service is configured that way (section 2.3).

2.2 Classifying errors¶

classify(exc) (in errors.py) sorts any exception into one of three kinds. It imports no SDK; it reads the shapes by duck typing.

Kind	Meaning	Examples
`THROTTLED`	the provider asked cloudg to slow down	AWS `Throttling`, `ThrottlingException`, `RequestLimitExceeded`, `TooManyRequestsException`, `SlowDown`, `RateExceeded` and the rest of botocore's standard-mode list; Azure `RateLimiting`, `SubscriptionRequestsThrottled`, `TenantRequestsThrottled`; GCP `ResourceExhausted`, `TooManyRequests`, and `ServiceUnavailable`, which carries HTTP 503; any other HTTP 429 or 503 without a more specific code
`TRANSIENT`	worth retrying, not a rate signal	HTTP 408, 500, 502, 504; AWS `InternalError`, `ServiceUnavailable` (the error code wins over its 503 status), `RequestTimeout`; GCP `DeadlineExceeded`, `InternalServerError`; connection resets and timeouts from botocore, urllib3, aiohttp, httpx and azure-core
`FATAL`	retrying will not help	`AccessDenied`, validation errors, anything else with a provider error code

Where it looks, in order: the botocore response["Error"]["Code"] and HTTP status; azure-core status_code, error.code and response.headers; google-api-core exception class names (along the MRO) and .code; urllib HTTPError.code and .headers; aiohttp .status. For wrapped errors it follows exc.cause (api_core RetryError) and explicit raise ... from chains, up to four levels, and never the implicit __context__. Only when nothing else matched does it search the message text for phrases such as "rate exceeded", "throttl" or "too many requests".

Two codes need context. AWS LimitExceededException is also used for "too many resources" quotas, so it counts as throttling only when the message mentions a rate (rate, throttl, too many, per second, tps); otherwise it is FATAL. Azure RetryableErrorDueToAnotherOperation arrives as a 429 but is a lock conflict, so classify() calls it TRANSIENT. The Azure SDK policy (3.2) sees only the response status, though, and treats every 429 as throttling.

retry_after(exc) returns the delay the server asked for, in seconds, from the first of these it finds: retry-after-ms or x-ms-retry-after-ms (milliseconds), Retry-After as delta-seconds or an HTTP-date, Resource Graph's x-ms-user-quota-resets-after as hh:mm:ss (used when x-ms-user-quota-remaining is 0 or missing, or the status is 429), and gRPC RetryInfo.retry_delay. describe_error(exc) is str(exc) with a throttled: prefix for throttling errors; every coverage record in the collectors goes through it.

classify_strict(exc) is classify() without the message-text search: it returns None when no error code, exception class or HTTP status says anything. is_provider_answer(exc) is true when the error carries a provider status or error code, that is, the request reached the service and it answered; cloudg's own errors and local failures are not answers. cloudg.retry.with_retry uses the two together, so it retries throttling only when the provider itself said so.

cloudg's own errors (CircuitOpenError, RetryBudgetExhaustedError, DeadlineExceededError) carry retry_after and a botocore-shaped .response, so existing error-code helpers report them. The first two classify as THROTTLED, the deadline error as TRANSIENT. An instrumented AWS client raises them as AWSCircuitOpenError and AWSRetryBudgetExhaustedError (in resilience/aws.py), which are also botocore ClientErrors with the code Throttling, so code that wraps its calls in except ClientError keeps catching them.

2.3 Token buckets and levels¶

RateLimiter keeps one token bucket per key. A call takes a token from up to three buckets, most general first, and waits for the slowest:

a provider-wide bucket (global_max_rps; off for every provider by default);
an account bucket (account_max_rps; on by default only for Azure, 20 rps per subscription, because ARM's limit is per subscription);
a leaf bucket per account x region x service. The leaf goes down to the operation when the service is marked per_operation (built in for AWS ec2, which throttles each API action separately) or when an override exists for "<service>.<Operation>".

A bucket refills at rate tokens per second up to capacity (the burst, or twice the rate when no burst is given, never below 1). Buckets work by reservation: taking a token never blocks while the limiter's lock is held. The balance may go negative, and the caller gets back the number of seconds to wait before using its token. That queues callers fairly, in arrival order, and lets one limiter serve asyncio code (await limiter.acquire(scope)) and worker threads (limiter.acquire_sync(scope)) at the same time.

2.4 Adaptive rate (AIMD)¶

Every leaf bucket adjusts its own rate the way TCP congestion control does, additive increase and multiplicative decrease:

When a call in the scope is throttled, the bucket's rate is halved (DECREASE_FACTOR = 0.5), never below the provider's min_rps, and its stored burst is drained so queued callers cannot all fire at once. Throttles that arrive within one second of the last decrease (DECREASE_INTERVAL) count as the same congestion event, so forty simultaneous 429s halve the rate once, not forty times.
When the provider sent a Retry-After hint, the bucket is also paused for that long (capped at 300 seconds, MAX_PAUSE_SECONDS, to survive bogus headers). Every caller of the scope waits.
After five seconds without a throttle (INCREASE_INTERVAL), each success adds 5% of the configured ceiling back (INCREASE_STEP, at least 0.01 rps), at most once per five seconds, until the configured rate is reached. Capacity scales with the rate.

In numbers: an EC2 action at its 20 rps ceiling that gets throttled twice, a second apart, runs at 5 rps. Steady success then adds 1 rps every five seconds, so it is back at 20 rps after about 75 seconds without another throttle. botocore's adaptive retry mode does something similar per client; the difference is that this state is shared by every client of the process.

The provider-wide and account buckets do not adapt; only leaves do. adaptive: false keeps a provider's rates fixed.

2.5 Bulkhead¶

Bulkhead caps the requests in flight per provider x account (max_concurrency: AWS 128, Azure 16, GCP 16). It is one slot pool shared by asyncio tasks and threads: async callers await a free slot without blocking the loop, threads block. What holds a slot depends on the integration:

Integration	One slot is held for
AWS hooks	one API call, from `before-call` (after the rate token) until `after-call` or `after-call-error`, so botocore's retries of that call reuse the slot
Azure policy	one HTTP attempt, from `on_request` until `on_response` or `on_exception`
GCP	one whole Cloud Asset listing, every page included
`call_with_resilience`	one attempt

The SDK hooks keep the slot (a Slot object) in the request context. If neither release event fires, for example because the task was cancelled between the SDK's events, the slot is released when the request context is garbage collected. A blocking boto3 or Azure call that runs on an event-loop thread takes a slot only if one is free and otherwise goes ahead without one, so it never blocks the loop.

2.6 Circuit breakers¶

BreakerRegistry holds one breaker per breaker scope, which has the same granularity as the scope's leaf bucket: per operation where buckets are per operation (built in for EC2, or any service with per_operation: true or an operation key), per service otherwise. A throttled DescribeSubnets no longer blocks DescribeVpcs in the same account and region. The breaker of a scope moves through three states:

Closed: calls flow. Each call that still failed with throttling or a transient error after its retries counts as one failure; any success resets the count. Calls the open circuit rejected were never sent and do not count, and neither does a call that stopped because the retry budget ran out: the budget stopped it, not the service.
Open: after breaker_threshold (default 5) consecutive failures, calls are rejected at once with CircuitOpenError, whose message is circuit open for <scope> after repeated throttling; retry in <n>s, for breaker_cooldown_seconds (default 60).
Half-open: after the cooldown one probe call goes through. Success closes the breaker and restores the base cooldown; failure reopens it with the cooldown doubled, up to 600 seconds. The probe holds a lease of one cooldown: if it never reports back (it was cancelled, or ended before it was sent), the lease runs out and the next call becomes the probe, so a breaker cannot stay half-open with nobody probing.

A FATAL answer from the provider (AccessDenied, a 404, a validation error) never counts as a failure, so an AccessDenied storm never opens a circuit. It does prove that the service is reachable, so it counts like a success: it resets the failure count and closes a half-open breaker. FATAL errors raised locally, without a provider status or code, leave the breaker alone.

2.7 Retries¶

call_with_resilience retries THROTTLED and TRANSIENT errors. The sleep before each retry is decorrelated jitter, min(cap, uniform(base, 3 x previous)) with base = 0.5 seconds and cap = max_backoff_seconds (60), as described in the AWS Architecture Blog post "Exponential Backoff And Jitter". It is never shorter than the server's Retry-After. Retrying stops at the first of:

max_retries retries (8 by default);
the per-call deadline (deadline_seconds, 900): a backoff or a rate-limit wait that would overrun it ends the call with the original error or DeadlineExceededError;
the provider's retry budget (retry_budget, 500): every retry after the first two of a call costs one token, every success refunds 0.1, and when a whole provider is degraded the budget runs dry and cloudg stops retrying instead of multiplying the load. The first two retries of every call are free, so a long run that spent the budget does not leave every later call without a single retry. This is the "retry quota" of the AWS SDKs' standard mode. The budget is shared by every integration of the provider: the SDK hooks draw from it too (section 3), and an empty budget ends the SDK's own retry loop with RetryBudgetExhaustedError, recorded as a throttled give-up.

When retrying stops, the original exception is re-raised, the breaker counts a failure, and for throttling the scope is recorded as given up.

2.8 Telemetry¶

ResilienceStats counts per service scope: calls, throttled, transient_errors, retries, gave_up, rejected (never attempted because a circuit was open), breaker_trips, waits, wait_seconds, max_wait_seconds, the current and lowest adaptive rate, and the last error. summary() returns totals, one human-readable line per eventful scope in messages, the abandoned scopes with their reason in skipped, and the full per-scope counters in scopes.

The governor writes every event twice: into its lifetime stats, and into the stats object bound to the current context by stats_scope(). The binding is a context variable, so asyncio tasks and asyncio.to_thread workers started inside the block inherit it, and two runs in one process (two MCP-triggered collections, say) keep separate numbers.

What calls counts depends on the integration: one per API call for AWS (botocore retries happen inside the call), one per HTTP attempt for Azure, one per first request and per further page for GCP, and one per attempt for call_with_resilience.

2.9 Coverage¶

ThrottleLedger is a per-task variant of the stats. The AWS collector binds one around every service task. Many collectors catch per-item errors so one bad resource does not stop the sweep; before 0.6.0 that meant a collector whose calls were all throttled reported SUCCESS with fewer assets. Now the ledger counts two kinds of lost calls: gave_up (sent, retried, still throttled) and skipped (rejected by an open circuit without being sent). If either is non-zero (ledger.degraded), the task's coverage record becomes PARTIAL with a message such as throttled: 1 call(s) gave up after retries, 2 call(s) skipped (circuit open) (<up to five reasons>).

2.10 The governor¶

Governor (governor.py) owns the limiter, the breakers, the per-provider retry settings and budgets, and the lifetime stats. There is one per process, returned by get_governor(). That is deliberate: rates learned while one collection was throttled keep protecting the next one, which is the point when an agent triggers live maps back to back.

configure(config) applies a CloudGConfig, a RateLimitConfig or a plain dict, and is idempotent: buckets and breakers are rebuilt only for providers whose settings changed, and budgets are reset only when retry settings changed. A provider whose settings did not change keeps its learned rates, its breakers, its budget and the concurrency slots its calls hold, so two runs that configure the same governor do not reset each other. MultiAccountCollector and InventoryMapper both call it (and InventoryMapper.map_inventory() reuses a stats scope its caller already bound, so an outer stats_scope() sees the whole run), so every entry point picks up config.ratelimit, aws.max_retries and aws.retry_mode. Anything else passed to configure (a test double, None aside) is ignored. reset_governor() replaces the process governor with a fresh default one and forgets everything learned; it exists for tests.