| Message ID | 20220121153905.1701521-1-ptsneves@gmail.com |
|---|---|
| State | New |
| Headers | show |
| Series | ] checksum: Support uri formatted file list | expand |
On Fri, 2022-01-21 at 16:39 +0100, Paulo Neves wrote: > Before this commit, if the file list for checksumming had > files names with spaces there would be a crash. This happened > due to filelist.split breaking on the file names instead of > on k:v boundaries. Now we validate this case and emit a fatal > error if such case is found. This needs to be fatal as the split > will generate broken k:v from then on. > > Instead of putting literal spaces in the file list the user > should urlencode the file names and if they contain coded spaces > they will be decoded. This is consistent with the current > practice where file names are urlencoded. A reproducer of the > issue this commit fixes, was to pass a do_compile[file-checksums] > list with files containing spaces in their names, urlencoded or > literal. > > Change-Id: I6ac4f1cffbb86e913883491d46e8cc69a028e992 > Signed-off-by: Paulo Neves <ptsneves@gmail.com> > --- > bitbake/lib/bb/checksum.py | 9 +++++++-- > 1 file changed, 7 insertions(+), 2 deletions(-) > > diff --git a/bitbake/lib/bb/checksum.py b/bitbake/lib/bb/checksum.py > index fb8a77f6ab..97cf10825e 100644 > --- a/bitbake/lib/bb/checksum.py > +++ b/bitbake/lib/bb/checksum.py > @@ -8,6 +8,7 @@ > import glob > import operator > import os > +import urllib.parse > import stat > import bb.utils > import logging > @@ -110,10 +111,14 @@ class FileChecksumCache(MultiProcessCache): > > checksums = [] > for pth in filelist.split(): > - exist = pth.split(":")[1] > + spl = pth.split(':') > + if len(spl) != 2: > + bb.fatal("found unformatted path in filelist " + pth) > + > + exist = spl[1] > if exist == "False": > continue > - pth = pth.split(":")[0] > + pth = urllib.parse.unquote(spl[0]) > if '*' in pth: > # Handle globs > for f in glob.glob(pth): Does any code currently urlencode the filepaths into this flag? I think we may also need to add some tests around this to bitbake-selftest... Cheers, Richard
I am not aware of it directly being done anywhere but files declared in the recipe are in URI form as far as i know. Where in the self test should i add such tests? Paulo Neves On 1/25/22 1:20 PM, Richard Purdie wrote: > On Fri, 2022-01-21 at 16:39 +0100, Paulo Neves wrote: >> Before this commit, if the file list for checksumming had >> files names with spaces there would be a crash. This happened >> due to filelist.split breaking on the file names instead of >> on k:v boundaries. Now we validate this case and emit a fatal >> error if such case is found. This needs to be fatal as the split >> will generate broken k:v from then on. >> >> Instead of putting literal spaces in the file list the user >> should urlencode the file names and if they contain coded spaces >> they will be decoded. This is consistent with the current >> practice where file names are urlencoded. A reproducer of the >> issue this commit fixes, was to pass a do_compile[file-checksums] >> list with files containing spaces in their names, urlencoded or >> literal. >> >> Change-Id: I6ac4f1cffbb86e913883491d46e8cc69a028e992 >> Signed-off-by: Paulo Neves <ptsneves@gmail.com> >> --- >> bitbake/lib/bb/checksum.py | 9 +++++++-- >> 1 file changed, 7 insertions(+), 2 deletions(-) >> >> diff --git a/bitbake/lib/bb/checksum.py b/bitbake/lib/bb/checksum.py >> index fb8a77f6ab..97cf10825e 100644 >> --- a/bitbake/lib/bb/checksum.py >> +++ b/bitbake/lib/bb/checksum.py >> @@ -8,6 +8,7 @@ >> import glob >> import operator >> import os >> +import urllib.parse >> import stat >> import bb.utils >> import logging >> @@ -110,10 +111,14 @@ class FileChecksumCache(MultiProcessCache): >> >> checksums = [] >> for pth in filelist.split(): >> - exist = pth.split(":")[1] >> + spl = pth.split(':') >> + if len(spl) != 2: >> + bb.fatal("found unformatted path in filelist " + pth) >> + >> + exist = spl[1] >> if exist == "False": >> continue >> - pth = pth.split(":")[0] >> + pth = urllib.parse.unquote(spl[0]) >> if '*' in pth: >> # Handle globs >> for f in glob.glob(pth): > Does any code currently urlencode the filepaths into this flag? > > I think we may also need to add some tests around this to bitbake-selftest... > > Cheers, > > Richard >
Hi all, On 1/25/22 13:55, Paulo Neves wrote: > I am not aware of it directly being done anywhere but files declared in > the recipe are in URI form as far as i know. Where in the self test > should i add such tests? > > Paulo Neves > > On 1/25/22 1:20 PM, Richard Purdie wrote: >> On Fri, 2022-01-21 at 16:39 +0100, Paulo Neves wrote: >>> Before this commit, if the file list for checksumming had >>> files names with spaces there would be a crash. This happened >>> due to filelist.split breaking on the file names instead of >>> on k:v boundaries. Now we validate this case and emit a fatal >>> error if such case is found. This needs to be fatal as the split >>> will generate broken k:v from then on. >>> >>> Instead of putting literal spaces in the file list the user >>> should urlencode the file names and if they contain coded spaces >>> they will be decoded. This is consistent with the current >>> practice where file names are urlencoded. A reproducer of the >>> issue this commit fixes, was to pass a do_compile[file-checksums] >>> list with files containing spaces in their names, urlencoded or >>> literal. >>> >>> Change-Id: I6ac4f1cffbb86e913883491d46e8cc69a028e992 >>> Signed-off-by: Paulo Neves <ptsneves@gmail.com> >>> --- >>> bitbake/lib/bb/checksum.py | 9 +++++++-- >>> 1 file changed, 7 insertions(+), 2 deletions(-) >>> >>> diff --git a/bitbake/lib/bb/checksum.py b/bitbake/lib/bb/checksum.py >>> index fb8a77f6ab..97cf10825e 100644 >>> --- a/bitbake/lib/bb/checksum.py >>> +++ b/bitbake/lib/bb/checksum.py >>> @@ -8,6 +8,7 @@ >>> import glob >>> import operator >>> import os >>> +import urllib.parse >>> import stat >>> import bb.utils >>> import logging >>> @@ -110,10 +111,14 @@ class FileChecksumCache(MultiProcessCache): >>> checksums = [] >>> for pth in filelist.split(): >>> - exist = pth.split(":")[1] >>> + spl = pth.split(':') >>> + if len(spl) != 2: >>> + bb.fatal("found unformatted path in filelist " + pth) >>> + >>> + exist = spl[1] >>> if exist == "False": >>> continue >>> - pth = pth.split(":")[0] >>> + pth = urllib.parse.unquote(spl[0]) >>> if '*' in pth: >>> # Handle globs >>> for f in glob.glob(pth): >> Does any code currently urlencode the filepaths into this flag? >> Not sure it's related in any way but I've had issues with spaces in filenames in externalsrc recipes in thud. Wanted to fix it instead of opening a bug and I forgot. Been a few years already :/ May be a way to provide a reproducer for this and check this patch fixes it. Cheers, Quentin >> I think we may also need to add some tests around this to >> bitbake-selftest... >> >> Cheers, >> >> Richard >> > > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#13280): https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_bitbake-2Ddevel_message_13280&d=DwIFaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=pncegqWmjAxc9HxxMkfqtewvKngB2W_pcdgejWkzwLKvGFKmfr6uVcaphu-SiuwN&s=mLKHnmnxJkR7swGcar1oHE6xSVtMIr693Gcm7foIKTo&e= > Mute This Topic: https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_88585130_6293953&d=DwIFaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=pncegqWmjAxc9HxxMkfqtewvKngB2W_pcdgejWkzwLKvGFKmfr6uVcaphu-SiuwN&s=h2HL5U0BZSx3nzTr6VH_57MEWt3w5l1rMfSOQem3A1Q&e= > Group Owner: bitbake-devel+owner@lists.openembedded.org > Unsubscribe: https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_bitbake-2Ddevel_unsub&d=DwIFaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=pncegqWmjAxc9HxxMkfqtewvKngB2W_pcdgejWkzwLKvGFKmfr6uVcaphu-SiuwN&s=uFEWtQO2c-QNbOVQsu7jvNWbudDEpbbuWtZHwy_ZFJ8&e= [quentin.schulz@theobroma-systems.com] > -=-=-=-=-=-=-=-=-=-=-=- >
diff --git a/bitbake/lib/bb/checksum.py b/bitbake/lib/bb/checksum.py index fb8a77f6ab..97cf10825e 100644 --- a/bitbake/lib/bb/checksum.py +++ b/bitbake/lib/bb/checksum.py @@ -8,6 +8,7 @@ import glob import operator import os +import urllib.parse import stat import bb.utils import logging @@ -110,10 +111,14 @@ class FileChecksumCache(MultiProcessCache): checksums = [] for pth in filelist.split(): - exist = pth.split(":")[1] + spl = pth.split(':') + if len(spl) != 2: + bb.fatal("found unformatted path in filelist " + pth) + + exist = spl[1] if exist == "False": continue - pth = pth.split(":")[0] + pth = urllib.parse.unquote(spl[0]) if '*' in pth: # Handle globs for f in glob.glob(pth):
Before this commit, if the file list for checksumming had files names with spaces there would be a crash. This happened due to filelist.split breaking on the file names instead of on k:v boundaries. Now we validate this case and emit a fatal error if such case is found. This needs to be fatal as the split will generate broken k:v from then on. Instead of putting literal spaces in the file list the user should urlencode the file names and if they contain coded spaces they will be decoded. This is consistent with the current practice where file names are urlencoded. A reproducer of the issue this commit fixes, was to pass a do_compile[file-checksums] list with files containing spaces in their names, urlencoded or literal. Change-Id: I6ac4f1cffbb86e913883491d46e8cc69a028e992 Signed-off-by: Paulo Neves <ptsneves@gmail.com> --- bitbake/lib/bb/checksum.py | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)