diff mbox series

patch to fix wrong http response for sstate cache checkstatus

Message ID iqRH.1720058591230617209.hhMB@lists.openembedded.org
State New
Headers show
Series patch to fix wrong http response for sstate cache checkstatus | expand

Commit Message

y75zhang July 5, 2024, 1:42 a.m. UTC
the issue is in sstate-cache checkstatus() for http mirrors
A request expired 30s timeout and released the connection,
B request took the connection into use and get the wrong response(meant to response A request and late than 30s)
details can be found here:

* how to reproduce:
modify /usr/lib64/python3.10/http/client.py like this to fake timeout event
you will find that some sstate files with 200 response in fact do not exists in remote http server
— a/client.py
+++ b/client.py
@@ -68,6 +68,7 @@ Req-started-unread-response    _CS_REQ_STARTED    <response_class>
Req-sent-unread-response       _CS_REQ_SENT       <response_class>
"""

+import random
import email.parser
import email.message
import errno
@@ -1370,6 +1371,8 @@ class HTTPConnection:
else:
response = self.response_class(self.sock, method=self._method)

+        if random.choice([False, False, False, False, False, False, False, True]):
+            raise TimeoutError('timed out test')
try:
try:
response.begin()

patch content:

From 76bc06e823d67e4eecaf7f2c7be0b67e405b1f76 Mon Sep 17 00:00:00 2001
From: y75zhang <yang-mark.zhang@nokia-sbell.com>
Date: Wed, 3 Jul 2024 08:20:34 +0300
Subject: [PATCH] bitbake: checkstatus: drop shared connecton when catch
Timeout error
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

* to avoid wrong http response in checkstatus function:
in wget checkstatus() we are using 'HTTPConnectionCache' to share connections
1. state_file1(exists on http server) use shared connection <shared1> to send request
2. http_server recieved request of state_file1, but delayed by some reason to sent respone
3. state_file1 checkstatus() failed by timeout and drop shared connection <shared1>
4. state_file2(not exists on http server) get shared connection <shared1> and send request
5. http_server finally send 200 response for state_file1
6. state_file2 recived 200 response and thought it was exists on http_server

* how to reproduce:
modify /usr/lib64/python3.10/http/client.py like this to fake timeout event
you will find that some sstate files with 200 response in fact do not exists in remote http server
— a/client.py
+++ b/client.py
@@ -68,6 +68,7 @@ Req-started-unread-response    _CS_REQ_STARTED    <response_class>
Req-sent-unread-response       _CS_REQ_SENT       <response_class>
"""

+import random
import email.parser
import email.message
import errno
@@ -1370,6 +1371,8 @@ class HTTPConnection:
else:
response = self.response_class(self.sock, method=self._method)

+        if random.choice([False, False, False, False, False, False, False, True]):
+            raise TimeoutError('timed out test')
try:
try:
response.begin()

Signed-off-by: y75zhang <yang-mark.zhang@nokia-sbell.com>
---
bitbake/lib/bb/fetch2/wget.py | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)

-- --

*Yang zhang*

yang-mark.zhang@nokia-sbell.com
diff mbox series

Patch

diff --git a/bitbake/lib/bb/fetch2/wget.py b/bitbake/lib/bb/fetch2/wget.py
index 2e92117..56caf92 100644
--- a/bitbake/lib/bb/fetch2/wget.py
+++ b/bitbake/lib/bb/fetch2/wget.py
@@ -244,7 +244,12 @@  class Wget(FetchMethod):
fetch.connection_cache.remove_connection(h.host, h.port)
raise urllib.error.URLError(err)
else:
-                    r = h.getresponse()
+                    try:
+                        r = h.getresponse()
+                    except TimeoutError as e:
+                        if fetch.connection_cache:
+                            fetch.connection_cache.remove_connection(h.host, h.port)
+                        raise TimeoutError(e)

# Pick apart the HTTPResponse object to get the addinfourl
# object initialized properly.
@@ -496,7 +501,7 @@  class Wget(FetchMethod):
valid = 1
elif self._vercmp(version, newver) < 0:
version = newver
-
+
pupver = re.sub('_', '.', version[1])

bb.debug(3, "*** %s -> UpstreamVersion = %s (CurrentVersion = %s)" %