What you see when a provider throttles

Section 6 · From docs/RESILIENCE.md on GitHub

Throttling never fails a map. A service that stayed throttled after its retries is recorded in coverage, the rest of the run continues, and the run's numbers end up in the result.

To produce the output below, a test-style script ran the AWS collector's vpc and subnets tasks against moto, with every DescribeSubnets request answered by RequestLimitExceeded and botocore's sleep removed. It used the default aws.max_retries: 10 and retry_mode: standard, wrapped the run in stats_scope(), and built an InventoryResult from the coverage and get_governor().summary(stats).

botocore sent DescribeSubnets 11 times (one attempt plus 10 retries). The subnet collector catches its own errors and returned nothing, and the ledger turned that into PARTIAL:

{"service": "vpc", "status": "SUCCESS", "asset_count": 2, "error": null}
{"service": "subnets", "status": "PARTIAL", "asset_count": 0, "error": "throttled: 1 call(s) gave up after retries (throttled: DescribeSubnets (retries exhausted): RequestLimitExceeded: Request limit exceeded.)"}

A collector that lets the error escape is recorded FAILED with describe_error(exc), for example throttled: An error occurred (RequestLimitExceeded) when calling the DescribeSubnets operation (reached max retries: 10): Request limit exceeded.. The Azure equivalent is an azure_<service> record with status FAILED and azure_full PARTIAL; on GCP it is listing truncated: throttled: ... on gcp_list_assets.

InventoryResult.throttling holds the governor summary. It is None unless something eventful happened: a throttle, a transient error, a give-up, a rejection, a breaker trip, or at least one second spent waiting for tokens in some scope. result.summary["throttling"] carries the short form:

{
  "totals": {
    "calls": 2,
    "throttled": 11,
    "transient_errors": 0,
    "retries": 10,
    "gave_up": 1,
    "rejected": 0,
    "breaker_trips": 0,
    "wait_seconds": 0.949
  },
  "messages": [
    "aws/123456789012/us-east-1/ec2: throttled 11x, slowed to 5.0 rps, 10 retries, 1 calls gave up"
  ],
  "skipped": {
    "aws/123456789012/us-east-1/ec2": "throttled: DescribeSubnets (retries exhausted): RequestLimitExceeded: Request limit exceeded."
  }
}

inventory-map.json gets the whole summary under a top-level throttling key, which InventoryResult.load() reads back. The same run, with the part shown above trimmed:

"throttling": {
  "totals": {...},
  "messages": [...],
  "skipped": {...},
  "scopes": {
    "aws/123456789012/us-east-1/ec2": {
      "calls": 2,
      "throttled": 11,
      "transient_errors": 0,
      "retries": 10,
      "gave_up": 1,
      "rejected": 0,
      "breaker_trips": 0,
      "waits": 10,
      "wait_seconds": 0.949,
      "max_wait_seconds": 0.1,
      "rate_rps": 5.0,
      "min_rate_rps": 5.0,
      "last_error": "throttled: DescribeSubnets (retries exhausted): RequestLimitExceeded: Request limit exceeded."
    }
  },
  "breakers": {},
  "slowed_buckets": {
    "aws/123456789012/us-east-1/ec2/DescribeSubnets": {
      "rate_rps": 5.0,
      "ceiling_rps": 20.0,
      "tokens": 0.173,
      "capacity": 25.0,
      "paused_for": 0.0,
      "throttles": 11
    }
  }
}

The rate dropped from 20 to 5 rps, not lower, because the 11 throttles arrived within about two seconds and decreases are at most one per second. slowed_buckets is keyed per operation because EC2 buckets are per action, and so are EC2 breakers; scopes is per service.

breakers and slowed_buckets come from the process-wide governor at the moment the run ended, not from the run alone. In a process that runs several collections at once they can include other runs' scopes.

Running the same script with ratelimit.aws.breaker_threshold: 1, then a second subnets task and a second vpc task, shows the breaker. It is per EC2 action, so it opened for DescribeSubnets only. The second subnets task never reached the API and was recorded as skipped, while DescribeVpcs went through:

{"service": "subnets_again", "status": "PARTIAL", "asset_count": 0, "error": "throttled: 1 call(s) skipped (circuit open) (throttled: circuit open for aws/123456789012/us-east-1/ec2/DescribeSubnets after repeated throttling; retry in 60s)"}
{"service": "vpc_again", "status": "SUCCESS", "asset_count": 2, "error": null}

aws/123456789012/us-east-1/ec2: throttled 11x, slowed to 5.0 rps, 10 retries, circuit opened 1x, 1 calls skipped, 1 calls gave up

cloudg map logs every message line at WARNING as Throttling: <line>. MultiAccountCollector.resilience_stats holds the stats object of its last collect_all() run (the caller's, when one was already bound).