| Message ID | d0e6b7661912a5aeedc3e193ff76982cb0fc8dc1.1665392835.git.paul.eggleton@linux.microsoft.com |
|---|---|
| State | New |
| Headers | show |
| Series | [01/34] migration-general: add section on using buildhistory | expand |
Hi Paul, On 10/10/22 11:23, Paul Eggleton wrote: > From: Paul Eggleton <paul.eggleton@microsoft.com> > > Add BB_PRESSURE_MAX_CPU, BB_PRESSURE_MAX_IO and BB_PRESSURE_MAX_MEMORY > variables (new in 4.1), based upon the supremely detailed commit > messages from Aryaman Gupta <aryaman.gupta@windriver.com> who > implemented the functionality. There is probably enough material for a > proper BitBake manual section (particularly if we can have some good > guidance on how users should determine the values for best performance) > but I did not have time to write such a section, and the BitBake manual > probably needs some additional preamble on tasks / scheduling > beforehand. > > Signed-off-by: Paul Eggleton <paul.eggleton@microsoft.com> > --- > documentation/ref-manual/variables.rst | 42 ++++++++++++++++++++++++++++++++++ > 1 file changed, 42 insertions(+) > > diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst > index 0c6bd80..e4304d1 100644 > --- a/documentation/ref-manual/variables.rst > +++ b/documentation/ref-manual/variables.rst > @@ -530,6 +530,48 @@ system and gives an overview of their function and contents. > ":ref:`dev-manual/common-tasks:speeding up a build`" > section in the Yocto Project Development Tasks Manual. > > + :term:`BB_PRESSURE_MAX_CPU` > + Specifies a maximum CPU pressure threshold, above which BitBake's > + scheduler will not start new tasks (providing there is at least > + one active task). If no value is set, CPU pressure is not > + monitored when starting tasks. > + > + The pressure data is calculated based upon what (recent) Linux > + kernels expose under ``/proc/pressure``. The threshold represents > + the difference in "total" pressure from the previous second. The > + minimum value is 1.0 (extremely slow builds) and the maximum is > + 1000000 (a pressure value unlikely to ever be reached). > + > + :term:`BB_PRESSURE_MAX_IO` > + Specifies a maximum I/O pressure threshold, above which BitBake's > + scheduler will not start new tasks (providing there is at least > + one active task). If no value is set, I/O pressure is not > + monitored when starting tasks. > + > + The pressure data is calculated based upon what (recent) Linux > + kernels expose under ``/proc/pressure``. The threshold represents > + the difference in "total" pressure from the previous second. The > + minimum value is 1.0 (extremely slow builds) and the maximum is > + 1000000 (a pressure value unlikely to ever be reached). > + > + :term:`BB_PRESSURE_MAX_MEMORY` > + Specifies a maximum memory pressure threshold, above which BitBake's > + scheduler will not start new tasks (providing there is at least > + one active task). If no value is set, memory pressure is not > + monitored when starting tasks. > + > + The pressure data is calculated based upon what (recent) Linux > + kernels expose under ``/proc/pressure``. The threshold represents > + the difference in "total" pressure from the previous second. The > + minimum value is 1.0 (extremely slow builds) and the maximum is > + 1000000 (a pressure value unlikely to ever be reached). > + > + Memory pressure is experienced when time is spent swapping, > + refaulting pages from the page cache or performing direct reclaim. > + This is why memory pressure is rarely seen, but setting this variable > + might be useful as a last resort to prevent OOM errors if they are > + occurring during builds. > + > :term:`BB_SERVER_TIMEOUT` > Specifies the time (in seconds) after which to unload the BitBake > server due to inactivity. Set :term:`BB_SERVER_TIMEOUT` to determine how Actually, these have recently been accepted in the BitBake manual (the patches from Aryaman got accepted after he left WindRiver), though apparently with fewer details: https://git.openembedded.org/bitbake/commit/?id=c014281f72f4f54ec8e681ef2b8e1080de9ab5cf Shall we submit your updates to the Bitbake manual instead? This way, we don't have these variables documented in two different places. Cheers Michael.
On Tuesday, 11 October 2022 23:30:50 NZDT Michael Opdenacker via lists.yoctoproject.org wrote: > Actually, these have recently been accepted in the BitBake manual (the > patches from Aryaman got accepted after he left WindRiver), though > apparently with fewer details: > https://git.openembedded.org/bitbake/commit/?id=c014281f72f4f54ec8e681ef2b8e > 1080de9ab5cf > > Shall we submit your updates to the Bitbake manual instead? This way, we > don't have these variables documented in two different places. Ah, oops - I missed that. I'm OK with that for now; however, and I may have also missed if this has been discussed recently, but it's my opinion that all variables documented in the BitBake manual should also be in the Yocto reference manual, since they are two different contexts - the catch being that for some variables there is information you can't reasonably include in the bitbake manual since it relies on some definitions unique to OE so the entries can't always be copied exactly. If on the other hand some variables only appear in the BitBake manual, there will be people who just won't find them. To avoid things getting out of sync, we could potentially have a script that would check for and report on variables present in the BitBake manual that aren't in the Yocto reference manual. I'd be happy to write such a script. Cheers Paul
Hi Paul, On 10/16/22 22:38, Paul Eggleton wrote: > On Tuesday, 11 October 2022 23:30:50 NZDT Michael Opdenacker via > lists.yoctoproject.org wrote: >> Actually, these have recently been accepted in the BitBake manual (the >> patches from Aryaman got accepted after he left WindRiver), though >> apparently with fewer details: >> https://git.openembedded.org/bitbake/commit/?id=c014281f72f4f54ec8e681ef2b8e >> 1080de9ab5cf >> >> Shall we submit your updates to the Bitbake manual instead? This way, we >> don't have these variables documented in two different places. > Ah, oops - I missed that. I'm OK with that for now; however, and I may have > also missed if this has been discussed recently, but it's my opinion that all > variables documented in the BitBake manual should also be in the Yocto > reference manual, since they are two different contexts - the catch being that > for some variables there is information you can't reasonably include in the > bitbake manual since it relies on some definitions unique to OE so the entries > can't always be copied exactly. If on the other hand some variables only > appear in the BitBake manual, there will be people who just won't find them. I agree that BitBake variables should all appear in the Yocto variable index. There are already some variables which are initially documented in the BitBake manual and then the description is expanded in the Yocto manual according to the OE additions. SRC_URI is one of them: * https://docs.yoctoproject.org/bitbake/2.0/bitbake-user-manual/bitbake-user-manual-ref-variables.html#term-SRC_URI * https://docs.yoctoproject.org/ref-manual/variables.html#term-SRC_URI > > To avoid things getting out of sync, we could potentially have a script that > would check for and report on variables present in the BitBake manual that > aren't in the Yocto reference manual. I'd be happy to write such a script. This would be useful, but if I recall correctly, Quentin was rather thinking about having Sphinx include these from the BitBake manual, at least for the ones which are only defined in the BitBake manual. Wouldn't it be better than duplicating content, which always causes trouble. Any further thoughts? Cheers Michael.
Hi all, On 10/17/22 10:15, Michael Opdenacker via lists.yoctoproject.org wrote: > Hi Paul, > > On 10/16/22 22:38, Paul Eggleton wrote: >> On Tuesday, 11 October 2022 23:30:50 NZDT Michael Opdenacker via >> lists.yoctoproject.org wrote: >>> Actually, these have recently been accepted in the BitBake manual (the >>> patches from Aryaman got accepted after he left WindRiver), though >>> apparently with fewer details: >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__git.openembedded.org_bitbake_commit_-3Fid-3Dc014281f72f4f54ec8e681ef2b8e&d=DwICaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=x6GN6XCgxQmI7f-H7T3WT3-Fm0BA5lJwsTI4AYj7vqNo4vO7FONFXbwhZxNDjOg5&s=w3C2EZQkLelO9mCiJE4ey-FVbNc3zA7PBHZ2nnBc4wE&e= 1080de9ab5cf >>> >>> Shall we submit your updates to the Bitbake manual instead? This way, we >>> don't have these variables documented in two different places. >> Ah, oops - I missed that. I'm OK with that for now; however, and I may >> have >> also missed if this has been discussed recently, but it's my opinion >> that all >> variables documented in the BitBake manual should also be in the Yocto >> reference manual, since they are two different contexts - the catch >> being that >> for some variables there is information you can't reasonably include >> in the >> bitbake manual since it relies on some definitions unique to OE so the >> entries >> can't always be copied exactly. If on the other hand some variables only >> appear in the BitBake manual, there will be people who just won't find >> them. > > > I agree that BitBake variables should all appear in the Yocto variable > index. > > There are already some variables which are initially documented in the > BitBake manual and then the description is expanded in the Yocto manual > according to the OE additions. SRC_URI is one of them: > > * > https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.yoctoproject.org_bitbake_2.0_bitbake-2Duser-2Dmanual_bitbake-2Duser-2Dmanual-2Dref-2Dvariables.html-23term-2DSRC-5FURI&d=DwICaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=x6GN6XCgxQmI7f-H7T3WT3-Fm0BA5lJwsTI4AYj7vqNo4vO7FONFXbwhZxNDjOg5&s=MUG2hR2Xp_pBuwrtNxNwKRKxCuWXh8fgIcmZm7k0cTw&e= * https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.yoctoproject.org_ref-2Dmanual_variables.html-23term-2DSRC-5FURI&d=DwICaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=x6GN6XCgxQmI7f-H7T3WT3-Fm0BA5lJwsTI4AYj7vqNo4vO7FONFXbwhZxNDjOg5&s=jTk77XY6-zxpJ_a9dWSexbLvlpev2lr2-QfhnWYv74E&e= >> >> To avoid things getting out of sync, we could potentially have a >> script that >> would check for and report on variables present in the BitBake manual >> that >> aren't in the Yocto reference manual. I'd be happy to write such a >> script. > This would be a good start, though... > > This would be useful, but if I recall correctly, Quentin was rather > thinking about having Sphinx include these from the BitBake manual, at > least for the ones which are only defined in the BitBake manual. > I'd prefer having a Sphinx plugin that does this automatically (I haven't checked if this is actually possible). But we can start with this script and manually adding those to the YP glossary. Someone may eventually find the time and motivation to look into a Sphinx plugin but I for sure don't have this time at the moment. sorry. How do you think of implementing this? I had in mind to just add a variable in the glossary whose description would be: See :term:`BitBake documentation <bitbake:FOOVAR>` so we don't need to take care of keeping their description in sync, Cheers, Quentin
Hi Quentin On 10/17/22 10:26, Quentin Schulz wrote: > Hi all, > > On 10/17/22 10:15, Michael Opdenacker via lists.yoctoproject.org wrote: >> Hi Paul, >> >> On 10/16/22 22:38, Paul Eggleton wrote: >>> On Tuesday, 11 October 2022 23:30:50 NZDT Michael Opdenacker via >>> lists.yoctoproject.org wrote: >>>> Actually, these have recently been accepted in the BitBake manual (the >>>> patches from Aryaman got accepted after he left WindRiver), though >>>> apparently with fewer details: >>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__git.openembedded.org_bitbake_commit_-3Fid-3Dc014281f72f4f54ec8e681ef2b8e&d=DwICaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=x6GN6XCgxQmI7f-H7T3WT3-Fm0BA5lJwsTI4AYj7vqNo4vO7FONFXbwhZxNDjOg5&s=w3C2EZQkLelO9mCiJE4ey-FVbNc3zA7PBHZ2nnBc4wE&e= >>>> 1080de9ab5cf >>>> >>>> Shall we submit your updates to the Bitbake manual instead? This >>>> way, we >>>> don't have these variables documented in two different places. >>> Ah, oops - I missed that. I'm OK with that for now; however, and I >>> may have >>> also missed if this has been discussed recently, but it's my opinion >>> that all >>> variables documented in the BitBake manual should also be in the Yocto >>> reference manual, since they are two different contexts - the catch >>> being that >>> for some variables there is information you can't reasonably include >>> in the >>> bitbake manual since it relies on some definitions unique to OE so >>> the entries >>> can't always be copied exactly. If on the other hand some variables >>> only >>> appear in the BitBake manual, there will be people who just won't >>> find them. >> >> >> I agree that BitBake variables should all appear in the Yocto >> variable index. >> >> There are already some variables which are initially documented in >> the BitBake manual and then the description is expanded in the Yocto >> manual according to the OE additions. SRC_URI is one of them: >> >> * >> https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.yoctoproject.org_bitbake_2.0_bitbake-2Duser-2Dmanual_bitbake-2Duser-2Dmanual-2Dref-2Dvariables.html-23term-2DSRC-5FURI&d=DwICaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=x6GN6XCgxQmI7f-H7T3WT3-Fm0BA5lJwsTI4AYj7vqNo4vO7FONFXbwhZxNDjOg5&s=MUG2hR2Xp_pBuwrtNxNwKRKxCuWXh8fgIcmZm7k0cTw&e= >> * >> https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.yoctoproject.org_ref-2Dmanual_variables.html-23term-2DSRC-5FURI&d=DwICaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=x6GN6XCgxQmI7f-H7T3WT3-Fm0BA5lJwsTI4AYj7vqNo4vO7FONFXbwhZxNDjOg5&s=jTk77XY6-zxpJ_a9dWSexbLvlpev2lr2-QfhnWYv74E&e= >>> >>> To avoid things getting out of sync, we could potentially have a >>> script that >>> would check for and report on variables present in the BitBake >>> manual that >>> aren't in the Yocto reference manual. I'd be happy to write such a >>> script. >> > > This would be a good start, though... > >> >> This would be useful, but if I recall correctly, Quentin was rather >> thinking about having Sphinx include these from the BitBake manual, >> at least for the ones which are only defined in the BitBake manual. >> > > I'd prefer having a Sphinx plugin that does this automatically (I > haven't checked if this is actually possible). > > But we can start with this script and manually adding those to the YP > glossary. Someone may eventually find the time and motivation to look > into a Sphinx plugin but I for sure don't have this time at the > moment. sorry. > > How do you think of implementing this? I had in mind to just add a > variable in the glossary whose description would be: > > See :term:`BitBake documentation <bitbake:FOOVAR>` > > so we don't need to take care of keeping their description in sync, Yes, this would be perfectly acceptable as a first solution IMHO, if nobody objects this redirection. Then Paul's script would definitely help (thanks in advance!). Cheers Michael.
diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst index 0c6bd80..e4304d1 100644 --- a/documentation/ref-manual/variables.rst +++ b/documentation/ref-manual/variables.rst @@ -530,6 +530,48 @@ system and gives an overview of their function and contents. ":ref:`dev-manual/common-tasks:speeding up a build`" section in the Yocto Project Development Tasks Manual. + :term:`BB_PRESSURE_MAX_CPU` + Specifies a maximum CPU pressure threshold, above which BitBake's + scheduler will not start new tasks (providing there is at least + one active task). If no value is set, CPU pressure is not + monitored when starting tasks. + + The pressure data is calculated based upon what (recent) Linux + kernels expose under ``/proc/pressure``. The threshold represents + the difference in "total" pressure from the previous second. The + minimum value is 1.0 (extremely slow builds) and the maximum is + 1000000 (a pressure value unlikely to ever be reached). + + :term:`BB_PRESSURE_MAX_IO` + Specifies a maximum I/O pressure threshold, above which BitBake's + scheduler will not start new tasks (providing there is at least + one active task). If no value is set, I/O pressure is not + monitored when starting tasks. + + The pressure data is calculated based upon what (recent) Linux + kernels expose under ``/proc/pressure``. The threshold represents + the difference in "total" pressure from the previous second. The + minimum value is 1.0 (extremely slow builds) and the maximum is + 1000000 (a pressure value unlikely to ever be reached). + + :term:`BB_PRESSURE_MAX_MEMORY` + Specifies a maximum memory pressure threshold, above which BitBake's + scheduler will not start new tasks (providing there is at least + one active task). If no value is set, memory pressure is not + monitored when starting tasks. + + The pressure data is calculated based upon what (recent) Linux + kernels expose under ``/proc/pressure``. The threshold represents + the difference in "total" pressure from the previous second. The + minimum value is 1.0 (extremely slow builds) and the maximum is + 1000000 (a pressure value unlikely to ever be reached). + + Memory pressure is experienced when time is spent swapping, + refaulting pages from the page cache or performing direct reclaim. + This is why memory pressure is rarely seen, but setting this variable + might be useful as a last resort to prevent OOM errors if they are + occurring during builds. + :term:`BB_SERVER_TIMEOUT` Specifies the time (in seconds) after which to unload the BitBake server due to inactivity. Set :term:`BB_SERVER_TIMEOUT` to determine how