Skip to content

Tuning

Section 9 · From docs/RESILIENCE.md on GitHub

Start from the defaults and read the run's throttling block before changing anything. A few patterns:

  • Throttling on one service only: override that service, not the provider default. For EC2, prefer an operation key (ec2.DescribeImageAttribute) over lowering ec2, which slows every action.
  • Throttling spread over many services of the same account: set account_max_rps. For AWS it is off by default, and an organization run with many regions can otherwise send 20 rps per service per region into one account.
  • Shared accounts: lower max_rps so cloudg leaves headroom for the pipelines and people using the same quota. The cost is a longer run; the adaptive limiter only reacts after the provider has already throttled someone.
  • Breakers opening on a flaky service: raise breaker_threshold rather than the cooldown. A breaker covers what its bucket covers: one EC2 action, one operation with its own override, or one whole service elsewhere. Set per_operation: true on a service to give each of its operations its own breaker.
  • Slow maps of AMI-heavy or bucket-heavy accounts: that is the per-item fan-out being held to the documented rates. 1,000 DescribeImageAttribute calls in one region take about 50 seconds at 20 rps. Raise the operation's rate only if the account's quota has been increased.
  • Agents collecting too often: raise live_cooldown_seconds, or set live_caller_max_operations and live_caller_window_seconds to give each principal a budget. Tell the agent to read cloudg://ratelimit before calling a live tool.
  • Many accounts mapped from one host: global_max_rps caps everything the process sends to one provider, whatever the account. max_concurrency limits requests in flight per account; lower it when an account's calls start to time out without any throttling error.
  • A provider degraded as a whole: the retry budget stops every integration from retrying once 500 retries have been spent without enough successes to refill it. Lower retry_budget to give up sooner.
  • Benchmarks or a mock cloud: ratelimit.enabled: false, or raise the rates (see section 11).