cloudg inventory internals

From docs/INVENTORY_INTERNALS.md on GitHub

A developer guide to the inventory mapper (cloudg map, cloudg deps, cloudg.inventory): how a mapping run flows, what every module does, how collectors declare relationships, how the linker resolves them, how dependencies are analysed, and how to extend each part. It is written for people changing cloudg or building on its internals.

Companion documents:

• INVENTORY_REFERENCE.md: the exact structure of every return value and exported file.
• INVENTORY_CATALOG.md: per-asset-type metadata keys, declared relations, and the relationship matrix.
• DOCUMENTATION.md: user-facing CLI, configuration and Python API.

Sections

• 1. Design principles

  1. Declare, don't resolve. A collector knows only the identifier of what an asset talks to (a role ARN, a queue URL, an image URI, a subnet ID). It...

• 2. Architecture and data flow

• 3. Code map

  GCP parity modules:

• 4. A mapping run, step by step

  InventoryMapper(config, tagging_sweep=None).map_inventory():

• 5. The AWS deep collector

• 6. AWS Organizations and Control Tower

  discover_organization(session, control_tower=True, home_region=None) (blocking, read-only):

• 7. The relationship linker

  RelationshipLinker(assets, materialize_external=True) subclasses IdentifierIndex.

• 8. Azure

  AzureDeepInventoryCollector(credential, subscription_id, graph_client_factory=None, use_resource_graph=True):

• 9. GCP

  GCPCollector(project_id, credentials=None, options) (GCPCollectorOptions: organization_id, scope, project_filter, skip_asset_types, page_size, include_iam,...

• 10. Hierarchies beyond AWS

• 11. Post-processing

• 12. Dependency analysis

  DependencyGraph turns each edge into dependent → dependency with DEPENDENCY_DIRECTION (reference table). The rule of thumb: for "uses" edges (references,...

• 13. Reference catalogs

  Reference data lives in YAML under cloudg/inventory/catalogs/, loaded by load_catalog(name) (cached per process):

• 14. Graph, ontology and viewer integration

• 15. Recipes

  1. Pick the package by domain (application, containers, data_ml, governance, network_ext, platform, security, serverless) and add a module, or extend one...

• 16. Security rules for collectors

  These rules are enforced in review and by CodeQL; follow them in every collector:

• 17. Code quality constraints

  The repository is checked by Codacy and CodeQL on every pull request. Code that passes them:

• 18. Testing

  tests/conftest.py has three autouse fixtures that make moto work with the async collectors:

• 19. Public API and compatibility

  Kept stable across 0.6.x (breaking them needs a minor version bump and a changelog entry):