| Message ID | 20230111223729.3472181-1-yoann.congal@smile.fr |
|---|---|
| State | Accepted, archived |
| Commit | bf681d173263cd42ffc143655f61abe0573fd83c |
| Headers | show |
| Series | persist_data: Handle sqlite error when cachefile path is too long | expand |
On 2023-01-11 17:37, Yoann Congal via lists.openembedded.org wrote: > Sqlite can't open the cachefile when its path is too long (>= 505 char by > my tests). > > We do have a path length check in sanity.bbclass but this code is called > before sanity checks. > > Treat the error in the exception instead of checking before hand in case > sqlite eventually fixes this. Nice. Did you re-open the bug upstream to see if a longer max path is acceptable this decade && got a link to that? ../Randy > > Fixes [YOCTO #12374] > > Signed-off-by: Yoann Congal<yoann.congal@smile.fr> > --- > lib/bb/persist_data.py | 18 +++++++++++++++++- > 1 file changed, 17 insertions(+), 1 deletion(-) > > diff --git a/lib/bb/persist_data.py b/lib/bb/persist_data.py > index ce84a158..a027f912 100644 > --- a/lib/bb/persist_data.py > +++ b/lib/bb/persist_data.py > @@ -249,4 +249,20 @@ def persist(domain, d): > > bb.utils.mkdirhier(cachedir) > cachefile = os.path.join(cachedir, "bb_persist_data.sqlite3") > - return SQLTable(cachefile, domain) > + > + try: > + return SQLTable(cachefile, domain) > + except sqlite3.OperationalError: > + # Sqlite fails to open database when its path is too long. > + # After testing, 504 is the biggest path length that can be opened by > + # sqlite. > + # Note: This code is called before sanity.bbclass and its path length > + # check > + max_len = 504 > + if len(cachefile) > max_len: > + logger.critical("The path of the cache file is too long " > + f"({len(cachefile)} chars > {max_len}) to be opened by sqlite! " > + f"Your cache file is \"{cachefile}\"") > + sys.exit(1) > + else: > + raise > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#14305):https://lists.openembedded.org/g/bitbake-devel/message/14305 > Mute This Topic:https://lists.openembedded.org/mt/96211174/3616765 > Group Owner:bitbake-devel+owner@lists.openembedded.org > Unsubscribe:https://lists.openembedded.org/g/bitbake-devel/unsub [randy.macleod@windriver.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On 1/12/23 03:41, Randy MacLeod wrote: > On 2023-01-11 17:37, Yoann Congal via lists.openembedded.org wrote: >> Sqlite can't open the cachefile when its path is too long (>= 505 char by >> my tests). >> >> We do have a path length check in sanity.bbclass but this code is called >> before sanity checks. >> >> Treat the error in the exception instead of checking before hand in case >> sqlite eventually fixes this. > > Nice. > > Did you re-open the bug upstream to see if a > longer max path is acceptable this decade > && > got a link to that? Nope, I did not re-open the bug. I was somewhat calmed down by the sheer list of supported systems by the code which include the sqlite limit : > This file contains the VFS implementation for unix-like operating systems include Linux, MacOSX, *BSD, QNX, VxWorks, AIX, HPUX, and others. My guess is: one of those impose this limit. I'll look into this. > ../Randy
On 1/12/23 08:44, Yoann Congal via lists.openembedded.org wrote: > On 1/12/23 03:41, Randy MacLeod wrote: >> On 2023-01-11 17:37, Yoann Congal via lists.openembedded.org wrote: >>> Sqlite can't open the cachefile when its path is too long (>= 505 >>> char by >>> my tests). >>> >>> We do have a path length check in sanity.bbclass but this code is called >>> before sanity checks. >>> >>> Treat the error in the exception instead of checking before hand in case >>> sqlite eventually fixes this. >> >> Nice. >> >> Did you re-open the bug upstream to see if a >> longer max path is acceptable this decade >> && >> got a link to that? > > Nope, I did not re-open the bug. I was somewhat calmed down by the sheer > list of supported systems by the code which include the sqlite limit : > > This file contains the VFS implementation for unix-like operating > systems include Linux, MacOSX, *BSD, QNX, VxWorks, AIX, HPUX, and others. > My guess is: one of those impose this limit. > > I'll look into this. Here the "ticket" I opened with sqlite (it's a forum post): https://sqlite.org/forum/forumpost/0b1b8b5116 Regards,
Hi Yoann, Yoann Congal <yoann.congal@smile.fr> escreveu no dia quarta, 11/01/2023 à(s) 22:38: > Sqlite can't open the cachefile when its path is too long (>= 505 char by > my tests). > > We do have a path length check in sanity.bbclass but this code is called > before sanity checks. > > Treat the error in the exception instead of checking before hand in case > sqlite eventually fixes this. > > Fixes [YOCTO #12374] > > Signed-off-by: Yoann Congal <yoann.congal@smile.fr> > --- > lib/bb/persist_data.py | 18 +++++++++++++++++- > 1 file changed, 17 insertions(+), 1 deletion(-) > > diff --git a/lib/bb/persist_data.py b/lib/bb/persist_data.py > index ce84a158..a027f912 100644 > --- a/lib/bb/persist_data.py > +++ b/lib/bb/persist_data.py > @@ -249,4 +249,20 @@ def persist(domain, d): > > bb.utils.mkdirhier(cachedir) > cachefile = os.path.join(cachedir, "bb_persist_data.sqlite3") > - return SQLTable(cachefile, domain) > + > + try: > + return SQLTable(cachefile, domain) > + except sqlite3.OperationalError: > + # Sqlite fails to open database when its path is too long. > + # After testing, 504 is the biggest path length that can be > opened by > + # sqlite. > + # Note: This code is called before sanity.bbclass and its path > length > + # check > + max_len = 504 > + if len(cachefile) > max_len: > + logger.critical("The path of the cache file is too long " > + f"({len(cachefile)} chars > {max_len}) to be opened > by sqlite! " > + f"Your cache file is \"{cachefile}\"") > As fair as I know bitbake don't use python fstrings anywhere grep --include=\*.py -rnw bitbake/lib/ -e 'f".*"' bitbake/lib/bb/data_smart.py:345: """Get the line where a operation is made on a variable in file f""" bitbake/lib/bb/main.py:135: parser.add_option("-f", "--force", action="store_true", dest="force", default=False, bitbake/lib/bb/tests/runqueue.py:305: tasks = self.run_bitbakecmd(["bitbake", "mc:mc_2:a1", "-c", "compile", "-f"], tempdir, "", extraenv=extraenv, cleanup=True) bitbake/lib/pyinotify.py:2255: parser.add_option("-f", "--raw-format", action="store_true", Jose > + sys.exit(1) > + else: > + raise > -- > 2.30.2 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#14305): > https://lists.openembedded.org/g/bitbake-devel/message/14305 > Mute This Topic: https://lists.openembedded.org/mt/96211174/5052612 > Group Owner: bitbake-devel+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [ > quaresma.jose@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
On Thu, 2023-01-12 at 11:43 +0000, Jose Quaresma wrote: > Hi Yoann, > > > Yoann Congal <yoann.congal@smile.fr> escreveu no dia quarta, 11/01/2023 à(s) 22:38: > > Sqlite can't open the cachefile when its path is too long (>= 505 char by > > my tests). > > > > We do have a path length check in sanity.bbclass but this code is called > > before sanity checks. > > > > Treat the error in the exception instead of checking before hand in case > > sqlite eventually fixes this. > > > > Fixes [YOCTO #12374] > > > > Signed-off-by: Yoann Congal <yoann.congal@smile.fr> > > --- > > lib/bb/persist_data.py | 18 +++++++++++++++++- > > 1 file changed, 17 insertions(+), 1 deletion(-) > > > > diff --git a/lib/bb/persist_data.py b/lib/bb/persist_data.py > > index ce84a158..a027f912 100644 > > --- a/lib/bb/persist_data.py > > +++ b/lib/bb/persist_data.py > > @@ -249,4 +249,20 @@ def persist(domain, d): > > > > bb.utils.mkdirhier(cachedir) > > cachefile = os.path.join(cachedir, "bb_persist_data.sqlite3") > > - return SQLTable(cachefile, domain) > > + > > + try: > > + return SQLTable(cachefile, domain) > > + except sqlite3.OperationalError: > > + # Sqlite fails to open database when its path is too long. > > + # After testing, 504 is the biggest path length that can be opened by > > + # sqlite. > > + # Note: This code is called before sanity.bbclass and its path length > > + # check > > + max_len = 504 > > + if len(cachefile) > max_len: > > + logger.critical("The path of the cache file is too long " > > + f"({len(cachefile)} chars > {max_len}) to be opened by sqlite! " > > + f"Your cache file is \"{cachefile}\"") > > > > > As fair as I know bitbake don't use python fstrings anywhere > > grep --include=\*.py -rnw bitbake/lib/ -e 'f".*"' > bitbake/lib/bb/data_smart.py:345: """Get the line where a operation is made on a variable in file f""" > bitbake/lib/bb/main.py:135: parser.add_option("-f", "--force", action="store_true", dest="force", default=False, > bitbake/lib/bb/tests/runqueue.py:305: tasks = self.run_bitbakecmd(["bitbake", "mc:mc_2:a1", "-c", "compile", "-f"], tempdir, "", extraenv=extraenv, cleanup=True) > bitbake/lib/pyinotify.py:2255: parser.add_option("-f", "--raw-format", action="store_true", We don't, no. We do now meet the minimum python requirements so in theory we could but it would mean the code can't be backported. A fix like this could be useful in older releases so it is perhaps sticking to the older syntax but I know others would prefer to be able to use it. I'm torn on it... Cheers, Richard
On 1/12/23 13:50, Richard Purdie wrote: > On Thu, 2023-01-12 at 11:43 +0000, Jose Quaresma wrote: >> Hi Yoann, >> >> >> Yoann Congal <yoann.congal@smile.fr> escreveu no dia quarta, 11/01/2023 à(s) 22:38: >>> Sqlite can't open the cachefile when its path is too long (>= 505 char by >>> my tests). >>> >>> We do have a path length check in sanity.bbclass but this code is called >>> before sanity checks. >>> >>> Treat the error in the exception instead of checking before hand in case >>> sqlite eventually fixes this. >>> >>> Fixes [YOCTO #12374] >>> >>> Signed-off-by: Yoann Congal <yoann.congal@smile.fr> >>> --- >>> lib/bb/persist_data.py | 18 +++++++++++++++++- >>> 1 file changed, 17 insertions(+), 1 deletion(-) >>> >>> diff --git a/lib/bb/persist_data.py b/lib/bb/persist_data.py >>> index ce84a158..a027f912 100644 >>> --- a/lib/bb/persist_data.py >>> +++ b/lib/bb/persist_data.py >>> @@ -249,4 +249,20 @@ def persist(domain, d): >>> >>> bb.utils.mkdirhier(cachedir) >>> cachefile = os.path.join(cachedir, "bb_persist_data.sqlite3") >>> - return SQLTable(cachefile, domain) >>> + >>> + try: >>> + return SQLTable(cachefile, domain) >>> + except sqlite3.OperationalError: >>> + # Sqlite fails to open database when its path is too long. >>> + # After testing, 504 is the biggest path length that can be opened by >>> + # sqlite. >>> + # Note: This code is called before sanity.bbclass and its path length >>> + # check >>> + max_len = 504 >>> + if len(cachefile) > max_len: >>> + logger.critical("The path of the cache file is too long " >>> + f"({len(cachefile)} chars > {max_len}) to be opened by sqlite! " >>> + f"Your cache file is \"{cachefile}\"") >>> >> >> >> As fair as I know bitbake don't use python fstrings anywhere >> >> grep --include=\*.py -rnw bitbake/lib/ -e 'f".*"' >> bitbake/lib/bb/data_smart.py:345: """Get the line where a operation is made on a variable in file f""" >> bitbake/lib/bb/main.py:135: parser.add_option("-f", "--force", action="store_true", dest="force", default=False, >> bitbake/lib/bb/tests/runqueue.py:305: tasks = self.run_bitbakecmd(["bitbake", "mc:mc_2:a1", "-c", "compile", "-f"], tempdir, "", extraenv=extraenv, cleanup=True) >> bitbake/lib/pyinotify.py:2255: parser.add_option("-f", "--raw-format", action="store_true", > > We don't, no. > > We do now meet the minimum python requirements so in theory we could > but it would mean the code can't be backported. A fix like this could > be useful in older releases so it is perhaps sticking to the older > syntax but I know others would prefer to be able to use it. I'm torn on > it... I really thought I saw some f-strings near this patch... But I guess I mixed my codebases... sorry. I'm okay with sending a V2 based on "".format().
diff --git a/lib/bb/persist_data.py b/lib/bb/persist_data.py index ce84a158..a027f912 100644 --- a/lib/bb/persist_data.py +++ b/lib/bb/persist_data.py @@ -249,4 +249,20 @@ def persist(domain, d): bb.utils.mkdirhier(cachedir) cachefile = os.path.join(cachedir, "bb_persist_data.sqlite3") - return SQLTable(cachefile, domain) + + try: + return SQLTable(cachefile, domain) + except sqlite3.OperationalError: + # Sqlite fails to open database when its path is too long. + # After testing, 504 is the biggest path length that can be opened by + # sqlite. + # Note: This code is called before sanity.bbclass and its path length + # check + max_len = 504 + if len(cachefile) > max_len: + logger.critical("The path of the cache file is too long " + f"({len(cachefile)} chars > {max_len}) to be opened by sqlite! " + f"Your cache file is \"{cachefile}\"") + sys.exit(1) + else: + raise
Sqlite can't open the cachefile when its path is too long (>= 505 char by my tests). We do have a path length check in sanity.bbclass but this code is called before sanity checks. Treat the error in the exception instead of checking before hand in case sqlite eventually fixes this. Fixes [YOCTO #12374] Signed-off-by: Yoann Congal <yoann.congal@smile.fr> --- lib/bb/persist_data.py | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-)