| Message ID | 20220419094616.433632-1-quaresma.jose@gmail.com |
|---|---|
| State | New |
| Headers | show |
| Series | [1/2] sstate: add a LockedSet class to be used on the mirror thread pool | expand |
On Tue, 2022-04-19 at 10:46 +0100, Jose Quaresma wrote: > The python set() is not thread safe and we use it on the ThreadedPool. > With this LockedSet python class we can call the 'add' and 'remove' > safely inside the ThreadedPool. > > This piece of code is taken from the stackoverflow > https://stackoverflow.com/questions/13610654/how-to-make-built-in-containers-sets-dicts-lists-thread-safe > > May be related with [YOCTO #14775] -- https://bugzilla.yoctoproject.org/show_bug.cgi?id=14775 > > Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> For info, https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/5083 was a failure with master-next which included the changes in this discussion. It doesn't seem to address that issue :( Cheers, Richard
Richard Purdie <richard.purdie@linuxfoundation.org> escreveu no dia quinta, 28/04/2022 à(s) 13:30: > On Tue, 2022-04-19 at 10:46 +0100, Jose Quaresma wrote: > > The python set() is not thread safe and we use it on the ThreadedPool. > > With this LockedSet python class we can call the 'add' and 'remove' > > safely inside the ThreadedPool. > > > > This piece of code is taken from the stackoverflow > > > https://stackoverflow.com/questions/13610654/how-to-make-built-in-containers-sets-dicts-lists-thread-safe > > > > May be related with [YOCTO #14775] -- > https://bugzilla.yoctoproject.org/show_bug.cgi?id=14775 > > > > Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> > > For info, > https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/5083 > was a failure with master-next which included the changes in this > discussion. It > doesn't seem to address that issue :( > > Cheers, > > Richard > > Unfortunately this solution still doesn't solve the problem. I'm going to analyze the log to try to understand what's going on, however I leave here one of the tracebacks in case anyone can see an anomaly here. The mirror thread pool is considerably different in this patch series which may imply that the problem may be on the server side but it is still too early to draw conclusions. I still haven't had time to think about how to do a test for this case. Jose ERROR: SState: cannot test file://universal/f3/69/sstate:python3-wheel-native:x86_64-linux:0.37.1:r0:x86_64:8:f36956da8ee13cebb1fb6df615d0eb548a4bf637c50021c213d8c6f0973674bf_deploy_source_date_epoch.tar.zst: TimeoutError('timed out') Traceback (most recent call last): File "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/meta/classes/sstate.bbclass", line 1023, in checkstatus fetcher.checkstatus() File "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/__init__.py", line 1829, in checkstatus ret = m.try_mirrors(self, ud, self.d, mirrors, True) File "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/__init__.py", line 1602, in try_mirrors return bool(try_mirrors(fetch, d, urldata, mirrors, check)) File "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/__init__.py", line 1128, in try_mirrors ret = try_mirror_url(fetch, origud, uds[index], ld, check) File "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/__init__.py", line 1023, in try_mirror_url found = ud.method.checkstatus(fetch, ud, ld) File "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/wget.py", line 361, in checkstatus with opener.open(r, timeout=30) as response: File "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/urllib/request.py", line 519, in open response = self._open(req, data) File "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/urllib/request.py", line 536, in _open result = self._call_chain(self.handle_open, protocol, protocol + File "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain result = func(*args) File "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/wget.py", line 166, in http_open return self.do_open(HTTPConnectionCache, req) File "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/wget.py", line 236, in do_open r = h.getresponse() File "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/http/client.py", line 1374, in getresponse response.begin() File "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/http/client.py", line 318, in begin version, status, reason = self._read_status() File "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/http/client.py", line 279, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") File "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/socket.py", line 705, in readinto return self._sock.recv_into(b) TimeoutError: timed out
Jose Quaresma via lists.openembedded.org <quaresma.jose= gmail.com@lists.openembedded.org> escreveu no dia sexta, 29/04/2022 à(s) 14:33: > > > Richard Purdie <richard.purdie@linuxfoundation.org> escreveu no dia > quinta, 28/04/2022 à(s) 13:30: > >> On Tue, 2022-04-19 at 10:46 +0100, Jose Quaresma wrote: >> > The python set() is not thread safe and we use it on the ThreadedPool. >> > With this LockedSet python class we can call the 'add' and 'remove' >> > safely inside the ThreadedPool. >> > >> > This piece of code is taken from the stackoverflow >> > >> https://stackoverflow.com/questions/13610654/how-to-make-built-in-containers-sets-dicts-lists-thread-safe >> > >> > May be related with [YOCTO #14775] -- >> https://bugzilla.yoctoproject.org/show_bug.cgi?id=14775 >> > >> > Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> >> >> For info, >> https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/5083 >> was a failure with master-next which included the changes in this >> discussion. It >> doesn't seem to address that issue :( >> >> Cheers, >> >> Richard >> >> > Unfortunately this solution still doesn't solve the problem. > > I'm going to analyze the log to try to understand what's going on, > however I leave here one of the tracebacks in case anyone can see an > anomaly here. > > The mirror thread pool is considerably different in this patch series > which may imply that the problem may be on the server side but it is still > too early to draw conclusions. > > I still haven't had time to think about how to do a test for this case. > > Jose > > ERROR: SState: cannot test > file://universal/f3/69/sstate:python3-wheel-native:x86_64-linux:0.37.1:r0:x86_64:8:f36956da8ee13cebb1fb6df615d0eb548a4bf637c50021c213d8c6f0973674bf_deploy_source_date_epoch.tar.zst: > TimeoutError('timed out') > Traceback (most recent call last): > File > "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/meta/classes/sstate.bbclass", > line 1023, in checkstatus > fetcher.checkstatus() > File > "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/__init__.py", > line 1829, in checkstatus > ret = m.try_mirrors(self, ud, self.d, mirrors, True) > File > "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/__init__.py", > line 1602, in try_mirrors > return bool(try_mirrors(fetch, d, urldata, mirrors, check)) > File > "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/__init__.py", > line 1128, in try_mirrors > ret = try_mirror_url(fetch, origud, uds[index], ld, check) > File > "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/__init__.py", > line 1023, in try_mirror_url > found = ud.method.checkstatus(fetch, ud, ld) > File > "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/wget.py", > line 361, in checkstatus > with opener.open(r, timeout=30) as response: > File > "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/urllib/request.py", > line 519, in open > response = self._open(req, data) > File > "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/urllib/request.py", > line 536, in _open > result = self._call_chain(self.handle_open, protocol, protocol + > File > "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/urllib/request.py", > line 496, in _call_chain > result = func(*args) > File > "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/wget.py", > line 166, in http_open > return self.do_open(HTTPConnectionCache, req) > File > "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/wget.py", > line 236, in do_open > r = h.getresponse() > File > "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/http/client.py", > line 1374, in getresponse > response.begin() > File > "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/http/client.py", > line 318, in begin > version, status, reason = self._read_status() > File > "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/http/client.py", > line 279, in _read_status > line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") > File > "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/socket.py", > line 705, in readinto > return self._sock.recv_into(b) > TimeoutError: timed out > > Latest tracebacks are messed up so here is a copy of it: ERROR: SState: cannot test file://universal/f3/69/sstate:python3-wheel-native:x86_64-linux:0.37.1:r0:x86_64:8:f36956da8ee13cebb1fb6df615d0eb548a4bf637c50021c213d8c6f0973674bf_deploy_source_date_epoch.tar.zst: TimeoutError('timed out') Traceback (most recent call last): File "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/meta/classes/sstate.bbclass", line 1023, in checkstatus fetcher.checkstatus() File "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/__init__.py", line 1829, in checkstatus ret = m.try_mirrors(self, ud, self.d, mirrors, True) File "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/__init__.py", line 1602, in try_mirrors return bool(try_mirrors(fetch, d, urldata, mirrors, check)) File "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/__init__.py", line 1128, in try_mirrors ret = try_mirror_url(fetch, origud, uds[index], ld, check) File "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/__init__.py", line 1023, in try_mirror_url found = ud.method.checkstatus(fetch, ud, ld) File "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/wget.py", line 361, in checkstatus with opener.open(r, timeout=30) as response: File "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/urllib/request.py", line 519, in open response = self._open(req, data) File "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/urllib/request.py", line 536, in _open result = self._call_chain(self.handle_open, protocol, protocol + File "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain result = func(*args) File "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/wget.py", line 166, in http_open return self.do_open(HTTPConnectionCache, req) File "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/wget.py", line 236, in do_open r = h.getresponse() File "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/http/client.py", line 1374, in getresponse response.begin() File "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/http/client.py", line 318, in begin version, status, reason = self._read_status() File "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/http/client.py", line 279, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") File "/home/pokybuild/yocto-worker/qemumips64/build/build/tmp/work/qemumips64-poky-linux/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/socket.py", line 705, in readinto return self._sock.recv_into(b) TimeoutError: timed out > -- > Best regards, > > José Quaresma > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#165013): > https://lists.openembedded.org/g/openembedded-core/message/165013 > Mute This Topic: https://lists.openembedded.org/mt/90558156/5052612 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > quaresma.jose@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass index 1c0cae4893..ebc336b497 100644 --- a/meta/classes/sstate.bbclass +++ b/meta/classes/sstate.bbclass @@ -1008,6 +1008,21 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, if progress: bb.event.fire(bb.event.ProcessProgress(msg, len(tasklist) - thread_worker.tasks.qsize()), d) + import threading + class LockedSet(set): + """A set where add() and remove() operator are thread-safe""" + def __init__(self, *args, **kwargs): + self._lock = threading.Lock() + super(LockedSet, self).__init__(*args, **kwargs) + + def add(self, elem): + with self._lock: + super(LockedSet, self).add(elem) + + def remove(self, elem): + with self._lock: + super(LockedSet, self).remove(elem) + tasklist = [] for tid in missed: sstatefile = d.expand(getsstatefile(tid, siginfo, d)) @@ -1016,6 +1031,10 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, if tasklist: nproc = min(int(d.getVar("BB_NUMBER_THREADS")), len(tasklist)) + # this collections will be used on the thread pool so let's do it safely + found = LockedSet(found) + missed = LockedSet(missed) + progress = len(tasklist) >= 100 if progress: msg = "Checking sstate mirror object availability"
The python set() is not thread safe and we use it on the ThreadedPool. With this LockedSet python class we can call the 'add' and 'remove' safely inside the ThreadedPool. This piece of code is taken from the stackoverflow https://stackoverflow.com/questions/13610654/how-to-make-built-in-containers-sets-dicts-lists-thread-safe May be related with [YOCTO #14775] -- https://bugzilla.yoctoproject.org/show_bug.cgi?id=14775 Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> --- meta/classes/sstate.bbclass | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)