Provider wiring

Section 3 · From docs/RESILIENCE.md on GitHub

3.1 AWS

install_aws_hooks(session, account_id, region, default_config=...) (resilience/aws.py) registers four handlers on a boto3 or aioboto3 session's event emitter. Every client created from that session afterwards is covered, paginators included, without touching the collectors:

Event	What the hook does
before-call	Resolves the scope, fails fast with AWSCircuitOpenError if the scope's breaker is open, counts the call, takes a limiter token, then a slot of the account's bulkhead
needs-retry	A throttled attempt halves the scope's rate and honours Retry-After; transient errors are counted. Every retry botocore is about to make, throttled or transient, costs one token of the retry budget, except the first two retries of the call; when the budget is empty the hook raises AWSRetryBudgetExhaustedError, which ends botocore's retry loop. A throttled retry also waits for a fresh rate token, on top of botocore's own backoff
after-call	Releases the bulkhead slot. 2xx/3xx feeds additive recovery and closes the breaker. An error that survived botocore's retries goes to the governor's on_final_error: throttling counts toward the breaker and is recorded as given up (stats and the task's ledger), a transient error counts toward the breaker, and a FATAL provider answer counts as proof that the service is reachable
after-call-error	The same for exceptions (connection errors and the like). A spent retry budget is recorded as given up without a breaker failure, and a call the open circuit refused is not recorded again

aiobotocore awaits coroutine handlers, so the aioboto3 hooks never block the event loop. Plain boto3 sessions get blocking handlers; when such a call runs on an event-loop thread, the token wait is capped at one second (MAX_INLINE_WAIT) so the loop is not stalled. The token is still consumed.

Installation is idempotent (a marker on the botocore session) and never breaks authentication: any error while installing is logged at DEBUG and the session is used as is. Where it is installed:

• AsyncAWSCollector._get_aio_session(): the aioboto3 session behind every core and deep collector (all 133 deep collector tasks), the Cloud Control and tagging sweeps and the per-item fan-out, scoped to the collector's account and region.
• credentials.build_aws_session(): the base boto3 session (account *, since the account is not known yet) and, after STS AssumeRole, the assumed session (with the target account id). That covers STS, Organizations and Control Tower discovery and the per-account sessions of the organization fan-out.

default_config becomes the session's default client config when it has none, so clients created without config= still get the configured retries.

botocore keeps doing the per-request retrying. Its settings now come from aws.retry_mode and aws.max_retries, which earlier releases accepted in the config but ignored (cloudg hard-coded adaptive mode and 10). The defaults are unchanged. botocore reads max_attempts in the retries dict as the number of retries after the first attempt, so the default max_retries: 10 allows up to 11 attempts per call; the integration tests check exactly that (with max_retries: 3, a throttled operation is sent four times). ratelimit.aws.max_retries, max_backoff_seconds and deadline_seconds are different settings: they bound cloudg-level retries in call_with_resilience, not botocore's. Per-call time limits on AWS stay botocore's job (its connect and read timeouts together with the retry count).

3.2 Azure Resource Manager

Every azure-mgmt client cloudg builds goes through build_azure_client(cls, ...) (resilience/azure.py): the collector's per-kind clients, resource_management_client, subscription_client and the Resource Graph client. It adds two keyword arguments to the constructor:

• retry_policy: an explicit azure-core RetryPolicy(retry_total=8, retry_status=8, retry_backoff_max=60, timeout=900), taken from ratelimit.azure.max_retries, max_backoff_seconds and deadline_seconds. azure-core's default allows only 3 status retries; its RetryPolicy already honours Retry-After on 429 and 503. The values were checked on a real ARM client;
• per_retry_policies=[AzureThrottlePolicy], which runs around every HTTP attempt, retries included.

The policy resolves the scope from the request URL (subscription and the last Microsoft.<Namespace> segment), fails fast on an open breaker, counts the call, takes a token (subscription bucket, then the resource-provider leaf) and a slot of the subscription's bulkhead. On the response, or on an exception, the slot is released, and then:

• 429 or 503: the scope's rate is halved and paused for Retry-After, or for x-ms-user-quota-resets-after when that is the only hint;
• 429, 503, 408, 500, 502 or 504: the response costs one token of the retry budget, because azure-core is about to retry it (the first two retries of a request are free). When the budget is empty the policy raises RetryBudgetExhaustedError, which ends azure-core's retry loop and is reported as throttling. The token is taken even on the last attempt, which azure-core would not retry;
• other 4xx: nothing;
• success: additive recovery and breaker reset, then two proactive checks. x-ms-user-quota-remaining: 0 pauses the Resource Graph scope until x-ms-user-quota-resets-after. x-ms-ratelimit-remaining-subscription-reads below 25 pauses the subscription's aggregate bucket for one second, which holds every call of that subscription, whatever the resource provider, because ARM refills about 25 tokens a second. With no account limit configured, the pause falls back to the resource provider's bucket.

If a client class rejects the extra keyword arguments (an old SDK, a test double), build_azure_client logs at DEBUG and builds it plainly. When a service still fails, the collector records service_errors[name] = describe_error(exc); for throttling it also reports the scope as given up so the breaker counts it. MultiAccountCollector turns each service error into an azure_<service> coverage record with status FAILED and marks azure_full PARTIAL.

3.3 Azure Resource Graph

Resource Graph allows 15 queries per 5-second window per user, so it has its own scope, azure/*/*/resourcegraph (no subscription: the quota follows the caller, not the subscription), at 3 rps with a burst of 15.

_fetch_page in cloudg/inventory/azure_graph/query.py handles one page:

• Clients from default_graph_client_factory are built with build_azure_client, so the throttle policy paces every HTTP attempt inside the SDK pipeline and reads the quota headers. _fetch_page notices that (_cloudg_throttle_policy on the client) and does not take a second token.
• Clients from a custom factory are paced by _fetch_page itself: breaker check and one token per page.
• A client built with the throttle policy already retries 429 inside its azure-core pipeline, so a 429 that reaches _fetch_page from it is final: it is recorded (on_final_error) and raised, not retried a second time.
• For a client from a custom factory, a 429 slows the scope down and is retried up to QueryLimits.max_retries (5) times, with a back-off of 2, 4, 8, 16 then 30 seconds. The limiter, which is paused for the server's hint, waits the hint out; with the limiter switched off the back-off itself is stretched to the hint, capped at 300 seconds. After the last retry the query is recorded as given up and the error is raised.

3.4 GCP Cloud Asset Inventory

GCPCollector._iterate drains one paged call (ListAssets, SearchAllResources or SearchAllIamPolicies) in a worker thread:

1. breaker check, one token and one counted call for gcp/<parent>/*/cloudasset/<Rpc>, then one bulkhead slot for the whole listing (max_concurrency concurrent listings per project or organization);
2. the call itself, with retry=gcp_retry(scope, counter=...): a google-api-core Retry on ResourceExhausted, ServiceUnavailable, DeadlineExceeded and InternalServerError, starting at 1 second, doubling, capped at ratelimit.gcp.max_backoff_seconds (60), with an overall timeout of deadline_seconds (900). Its on_error reports each retried error to the governor, so throttling halves the rate and RetryInfo delays pause it. It also enforces two limits api_core does not have: each retry after the first two of a page costs a retry-budget token, and at most ratelimit.gcp.max_retries (8) consecutive retries are made per page (a RetryCounter shared with paced resets on every page that arrives). Past either limit, on_error raises RetryBudgetExhaustedError, chained to the provider error, which ends api_core's retry loop;
3. paced(...) around the pager takes a token every page_size items, just before the pager fetches the next page. It does not check the breaker again: the breaker was checked once for the whole listing in step 1, so a listing that is a half-open breaker's probe is not rejected by its own breaker on page 2. Pages are fetched lazily while iterating, so this keeps org-wide listings under the per-minute quotas. With short pages it is approximate, never off by more than one page per short page.

If the listing still fails, throttling is recorded as given up and transient errors count toward the breaker. _iterate returns what it collected so far plus the error, and the caller records listing truncated: throttled: ... (PARTIAL), or FAILED when nothing came back, and falls back to SearchAllResources when ListAssets is unavailable. GCPDeepInventoryCollector labels its IAM policy search the same way.