mbox series

[0/1] crates.io data access policy

Message ID 20260427135122.1504143-1-johanam@axis.com
Headers show
Series crates.io data access policy | expand

Message

Johan Anderholm April 27, 2026, 1:51 p.m. UTC 
It appears that the crate fetcher does not adhere to the data access policy
of crates.io as is described here:

https://crates.io/data-access#api

Namely the part about adding a user-agent identifying the application and
information on how to reach the project in case of issues.

Also users are requested to use the CDN instead of the download API, as that
does not have a 1 request per second limit:

https://github.com/rust-lang/crates.io/issues/13482#issuecomment-4304855751

This patch addresses the request to use a CDN and seems to fix a recent
breakage of the crate fetcher. It does not address the part about a user-agent
since I do not know how to do that, and I do not know what to put in it anyway.

Johan Anderholm (1):
  fetch2/crate: use CDN for fetching crates

 lib/bb/fetch2/crate.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

Comments

Quentin Schulz April 27, 2026, 5:35 p.m. UTC | #1 
Hi Johan,

On 4/27/26 3:51 PM, Johan Anderholm via lists.openembedded.org wrote:
> [You don't often get email from johan.anderholm=axis.com@lists.openembedded.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
> 
> It appears that the crate fetcher does not adhere to the data access policy
> of crates.io as is described here:
> 
> https://crates.io/data-access#api
> 
> Namely the part about adding a user-agent identifying the application and
> information on how to reach the project in case of issues.
> 
> Also users are requested to use the CDN instead of the download API, as that
> does not have a 1 request per second limit:
> 
> https://github.com/rust-lang/crates.io/issues/13482#issuecomment-4304855751
> 
> This patch addresses the request to use a CDN and seems to fix a recent
> breakage of the crate fetcher. It does not address the part about a user-agent
> since I do not know how to do that, and I do not know what to put in it anyway.
> 

The crate fetcher is derived from the wget fetch method which does seem 
to implement a UA reporting bitbake and its version, see 234f9e810494 
("fetch2/wget: set User-Agent to 'bitbake/version' in checkstatus()"). 
The best thing would be to try yourself (I'm assuming having a webserver 
you control and change the fetcher to use your local webserver (or 
possibly even setting up a PREMIRRORS for crates)).

Cheers,
Quentin
Ross Burton April 27, 2026, 8:25 p.m. UTC | #2 
On 27 Apr 2026, at 18:35, Quentin Schulz via lists.openembedded.org <quentin.schulz=cherry.de@lists.openembedded.org> wrote:
> 
> The crate fetcher is derived from the wget fetch method which does seem to implement a UA reporting bitbake and its version, see 234f9e810494 ("fetch2/wget: set User-Agent to 'bitbake/version' in checkstatus()"). The best thing would be to try yourself (I'm assuming having a webserver you control and change the fetcher to use your local webserver (or possibly even setting up a PREMIRRORS for crates)).

That’s checkstatus, not the actual fetch.

Put this on the list of reasons why I dislike the wget fetcher: two entirely distinct codepaths.

Ross