mbox series

[0/1] fetch/s3: use boto3 natively for S3 fetches

Message ID 20261105012522.878275-1-kpoduval@lab126.com
Headers show
Series fetch/s3: use boto3 natively for S3 fetches | expand

Message

Poduval, Karthik Nov. 5, 2026, 1:25 a.m. UTC 
From: Karthik Poduval <kpoduval@lab126.com>

Hi,

This reworks the S3 fetcher (lib/bb/fetch/s3.py) to fetch objects in-process
with the boto3 SDK instead of shelling out to the "aws" CLI once per object,
with an automatic fallback to the CLI when boto3 is not importable.

Disclosure (per the project's AI-assisted contribution policy): this change
was developed with AI assistance. The commit carries an "AI-Generated: Kiro,
Claude Opus 4.8" tag. I have reviewed, built and tested the change and take
responsibility for it.

The motivation is sstate mirrors hosted on S3. The current fetcher runs
"aws s3 ls" / "aws s3 cp" as a subprocess for every object; each call pays the
Python interpreter + botocore import startup, re-resolves credentials and
configuration, and opens a fresh TLS connection. For a mirror of many small
objects (thousands of setscene HEAD+GET lookups) that per-object startup cost
dominates the build wall time rather than the transfer itself.

Fetching with boto3 in-process and caching one warm S3 client per worker
process removes that overhead. In my testing an "aws s3 ls" costing ~0.9s per
object dropped to ~0.08s warm (~11x), and a core-image-minimal build backed by
an S3 sstate mirror went from ~20 minutes to ~9 minutes with full sstate reuse
and no fetch errors.

The series also fixes a real bug that the SDK path exposes: bitbake's mirror
rewriting percent-encodes ':' in sstate object names as %3A. The aws CLI
tolerated this, but boto3 re-encodes the key at signing time, so without
unquoting the key first S3 returns 400 Bad Request on every lookup.

Points I would appreciate maintainer guidance on:

- boto3 is imported lazily and only used if importable; otherwise the existing
  aws-CLI path runs unchanged, so this adds no new hard dependency. I am happy
  to change how the optional dependency is gated if you prefer.
- Region/profile are resolved from the datastore and then the standard AWS
  environment variables, with ~/.aws/config honoured by boto3. There is a
  small credential-retrieval backoff for transient credential_process rate
  limits; I can drop or simplify that if it is considered out of scope for
  bitbake.

Tested on bitbake master (parent 741285343): the offline fetcher selftests
(bb.tests.fetch, 218 tests) pass, and the boto3 download/head_object,
key-decoding (%3A), 404 handling and CLI-fallback paths were exercised
directly. The fetcher was also exercised end-to-end against a real S3 sstate
mirror, restoring ~1400 sstate objects with no fetch errors during a full
core-image-minimal build.

Thanks,
Karthik

Karthik Poduval (1):
  fetch/s3: fetch via boto3 by default, falling back to the AWS CLI

 lib/bb/fetch/s3.py | 233 ++++++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 206 insertions(+), 27 deletions(-)