Skip to content

fix(aws): paginate resource inventory using boto3 paginators#49

Merged
bencehezso merged 2 commits into
mainfrom
task/paginate_aws_resource_inventory
Jul 23, 2026
Merged

fix(aws): paginate resource inventory using boto3 paginators#49
bencehezso merged 2 commits into
mainfrom
task/paginate_aws_resource_inventory

Conversation

@bencehezso

Copy link
Copy Markdown
Contributor

Summary

  • build_aws_resource_inventory in core/utils_aws.py used a hand-rolled while "NextToken" in response loop, which only paginated services that use NextToken. Services paginating by Marker, NextMarker, ContinuationToken, ExclusiveStartTableName, or lowercase nextToken were silently truncated to their first page — e.g. DynamoDB list_tables capped at 100, S3 buckets past the first page missing.
  • Replaced the loop with boto3's built-in paginators via a shared paginate() / paginate_or_call() helper in core/utils_aws.py, and refactored core/utils_egress_aws.py to use the same helper (removing its duplicate _paginate). Both inventory and egress now paginate identically, so DynamoDB (and every other multi-page resource type) counts match across the two paths.
  • Retry with exponential backoff on Throttling / RequestLimitExceeded / BotoCoreError now applies per page, not just per initial call.

@bencehezso bencehezso self-assigned this Jul 22, 2026
@bencehezso bencehezso added the bug Something isn't working label Jul 22, 2026
@bencehezso
bencehezso merged commit 0fa09d4 into main Jul 23, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant