| Message ID | 20240909145530.25301-10-twoerner@gmail.com |
|---|---|
| State | New |
| Headers | show |
| Series | scarthgap update | expand |
Hi Trevor, On 9/9/24 4:55 PM, Trevor Woerner via lists.yoctoproject.org wrote: > From: Quentin Schulz <quentin.schulz@cherry.de> > > v4l2codecs is the gstreamer plugin for V4L2 stateless video hardware > decoding. The Rockchip SoCs that have a VPU all seems to be based on > Hantro, RKVDEC or RKVDECv2, all stateless encoding/decoding VPUs. > > Therefore, let's enable VPU decoding in Gstreamer whenever possible, > when the SoC supports it. > > PX30, RK3066, RK3188, RK3288, RK3328, RK3399, RK356x and RK3588(s) all > have at least one Hantro VPU. > > RK3328, RK3399, RK356x and RK3588(s) all have at least one > RKVDEC/RKVDECv2 VPU (though not necessarily supported in the upstream > kernel just yet). > > === PX30 > Tested on PX30 Ringneck with with Haikou+Haikou Video Demo adapter: > > $ gst-launch-1.0 filesrc location=$FILE ! parsebin ! v4l2slh264dec ! waylandsink > > with FILE storing the path to any h264 file, e.g. > https://download.blender.org/peach/bigbuckbunny_movies/big_buck_bunny_720p_h264.mov > https://download.blender.org/peach/bigbuckbunny_movies/big_buck_bunny_1080p_h264.mov > > Needed packages are: > - weston > - gstreamer1.0-plugins-bad (for waylandsink and v4l2slh264dec) > - gstreamer1.0-plugins-base (for parsebin) > Weston output is broken on Scarthgap. c.f. https://lore.kernel.org/all/1f84dd9c-bdef-4099-9dba-04b445a7ee0a@cherry.de/ > A few frames are dropped every other second for 1080p but otherwise > smooth. > > === RK3399 > Tested on RK3399 Puma with Haikou: > > $ gst-launch-1.0 filesrc location=$FILE ! parsebin ! v4l2slh264dec ! waylandsink > > with FILE storing the path to any h264 file, e.g. > https://download.blender.org/peach/bigbuckbunny_movies/big_buck_bunny_1080p_h264.mov > https://download.blender.org/demo/movies/BBB/bbb_sunflower_2160p_30fps_normal.mp4.zip > > Needed packages are: > - weston > - gstreamer1.0-plugins-bad (for waylandsink and v4l2codecs) > - gstreamer1.0-plugins-base (for parsebin) > Weston output is broken on Scarthgap. c.f. https://lore.kernel.org/all/1f84dd9c-bdef-4099-9dba-04b445a7ee0a@cherry.de/ > === RK3588 > > Tested on a RK3588 Tiger with Haikou+Haikou Video Demo adapter - on a > downstream v6.6 (upstream-based, not Rockchip BSP-based) with DSI > patches - : > > $ gst-launch-1.0 filesrc location=$FILE ! parsebin ! v4l2slav1dec ! fakesink > > with FILE storing the path to any AV1 file, e.g. > http://download.opencontent.netflix.com.s3.amazonaws.com/AV1/cmaf/spark-8b-59.94fps/spark_606kbps_432p.mp4 > https://woolyss.com/f/av1-nosound-chimera.mp4 > https://woolyss.com/f/av1-opus-sita.webm > > Needed packages are: > - gstreamer1.0-plugins-bad (for fakesink and v4l2slav1dec) > - gstreamer1.0-plugins-base (for parsebin) > Stateless AV1 decoding is not available in the mesa release used on Scarthgap. So... RK3588 seems to have gained upstream support for VDPU121 which should allow h264, mpeg2 and vp8 decoding. Building a linux-next kernel after tag next-20240829 should have the support AFAICT. So, I would retest RK3588 support for those video formats and drop the AV1 mention from the commit log as it for sure doesn't work because of the version of mesa in Scarthgap. We could backport later releases into meta-rockchip to gain support for it. I think this **may** be an idea once there's support for 24.1 and later merged into OE-Core because it brings Panthor (open-source GPU drivers) support to RK3588 which would be a shame not to have in an LTS. But let's see when that happens. As for the issues with display on PX30 and RK3399, it seems those are supposed to be patched in newer releases of gstreamer, c.f. https://gitlab.freedesktop.org/wayland/weston/-/issues/839. Note that Robert said it would be fixed in gstreamer 1.22 but the linked patch only made it to 1.23 (Scarthgap has 1.22, Styhead 1.24). Considering what Robert said in the link above, I assume people using another compositor than weston may be fine on Scarthgap, so I would suggest to backport the patches I submitted in this series to scarthgap anyway? One is not forced to use weston to do VPU decoding. FYI, I gave ~30min at backporting the gstreamer patches to 1.22.11 but this is above my skill set today. For those, I would just mention the issue in the README and commit log and maybe switch to fakesink in the commit log instead of waylandsink (and drop weston from the needed packages). What do you think? Cheers, Quentin
On Mon 2024-09-09 @ 06:01:30 PM, Quentin Schulz via lists.yoctoproject.org wrote: > Hi Trevor, > > On 9/9/24 4:55 PM, Trevor Woerner via lists.yoctoproject.org wrote: > > From: Quentin Schulz <quentin.schulz@cherry.de> > > > > v4l2codecs is the gstreamer plugin for V4L2 stateless video hardware > > decoding. The Rockchip SoCs that have a VPU all seems to be based on > > Hantro, RKVDEC or RKVDECv2, all stateless encoding/decoding VPUs. > > > > Therefore, let's enable VPU decoding in Gstreamer whenever possible, > > when the SoC supports it. > > > > PX30, RK3066, RK3188, RK3288, RK3328, RK3399, RK356x and RK3588(s) all > > have at least one Hantro VPU. > > > > RK3328, RK3399, RK356x and RK3588(s) all have at least one > > RKVDEC/RKVDECv2 VPU (though not necessarily supported in the upstream > > kernel just yet). > > > > === PX30 > > Tested on PX30 Ringneck with with Haikou+Haikou Video Demo adapter: > > > > $ gst-launch-1.0 filesrc location=$FILE ! parsebin ! v4l2slh264dec ! waylandsink > > > > with FILE storing the path to any h264 file, e.g. > > https://download.blender.org/peach/bigbuckbunny_movies/big_buck_bunny_720p_h264.mov > > https://download.blender.org/peach/bigbuckbunny_movies/big_buck_bunny_1080p_h264.mov > > > > Needed packages are: > > - weston > > - gstreamer1.0-plugins-bad (for waylandsink and v4l2slh264dec) > > - gstreamer1.0-plugins-base (for parsebin) > > > > Weston output is broken on Scarthgap. c.f. > https://lore.kernel.org/all/1f84dd9c-bdef-4099-9dba-04b445a7ee0a@cherry.de/ > > > A few frames are dropped every other second for 1080p but otherwise > > smooth. > > > > === RK3399 > > Tested on RK3399 Puma with Haikou: > > > > $ gst-launch-1.0 filesrc location=$FILE ! parsebin ! v4l2slh264dec ! waylandsink > > > > with FILE storing the path to any h264 file, e.g. > > https://download.blender.org/peach/bigbuckbunny_movies/big_buck_bunny_1080p_h264.mov > > https://download.blender.org/demo/movies/BBB/bbb_sunflower_2160p_30fps_normal.mp4.zip > > > > Needed packages are: > > - weston > > - gstreamer1.0-plugins-bad (for waylandsink and v4l2codecs) > > - gstreamer1.0-plugins-base (for parsebin) > > > > Weston output is broken on Scarthgap. c.f. > https://lore.kernel.org/all/1f84dd9c-bdef-4099-9dba-04b445a7ee0a@cherry.de/ > > > === RK3588 > > > > Tested on a RK3588 Tiger with Haikou+Haikou Video Demo adapter - on a > > downstream v6.6 (upstream-based, not Rockchip BSP-based) with DSI > > patches - : > > > > $ gst-launch-1.0 filesrc location=$FILE ! parsebin ! v4l2slav1dec ! fakesink > > > > with FILE storing the path to any AV1 file, e.g. > > http://download.opencontent.netflix.com.s3.amazonaws.com/AV1/cmaf/spark-8b-59.94fps/spark_606kbps_432p.mp4 > > https://woolyss.com/f/av1-nosound-chimera.mp4 > > https://woolyss.com/f/av1-opus-sita.webm > > > > Needed packages are: > > - gstreamer1.0-plugins-bad (for fakesink and v4l2slav1dec) > > - gstreamer1.0-plugins-base (for parsebin) > > > > Stateless AV1 decoding is not available in the mesa release used on > Scarthgap. > > So... > > RK3588 seems to have gained upstream support for VDPU121 which should allow > h264, mpeg2 and vp8 decoding. Building a linux-next kernel after tag > next-20240829 should have the support AFAICT. > > So, I would retest RK3588 support for those video formats and drop the AV1 > mention from the commit log as it for sure doesn't work because of the > version of mesa in Scarthgap. We could backport later releases into > meta-rockchip to gain support for it. I think this **may** be an idea once > there's support for 24.1 and later merged into OE-Core because it brings > Panthor (open-source GPU drivers) support to RK3588 which would be a shame > not to have in an LTS. But let's see when that happens. > > > > As for the issues with display on PX30 and RK3399, it seems those are > supposed to be patched in newer releases of gstreamer, c.f. > https://gitlab.freedesktop.org/wayland/weston/-/issues/839. Note that > Robert said it would be fixed in gstreamer 1.22 but the linked patch > only made it to 1.23 (Scarthgap has 1.22, Styhead 1.24). > > Considering what Robert said in the link above, I assume people using > another compositor than weston may be fine on Scarthgap, so I would suggest > to backport the patches I submitted in this series to scarthgap anyway? One > is not forced to use weston to do VPU decoding. > FYI, I gave ~30min at backporting the gstreamer patches to 1.22.11 but this > is above my skill set today. > > For those, I would just mention the issue in the README and commit log and > maybe switch to fakesink in the commit log instead of waylandsink (and drop > weston from the needed packages). > > What do you think? To be clear, when I say I've run-tested this build I'm not implying I've tested every aspect, feature, and functionality of each chip. I've simply flashed an SD card, applied power, watched it boot to a prompt, log in, and run something simple like "uname -a" to verify it's using the scarthgap kernel and not the latest master. I'm not really in a position to test stateless VPU decoding or anything fancy like that :-) If there aren't any issues or objections, perhaps I could start by applying the other patches in this series while considering what to do about this one? Maybe after the others are applied you'd like to have a go at making a scarthgap version of this patch (with all the proper commit messages etc)? Best regards, Trevor
Hi Trevor, On 9/10/24 6:52 AM, Trevor Woerner via lists.yoctoproject.org wrote: > On Mon 2024-09-09 @ 06:01:30 PM, Quentin Schulz via lists.yoctoproject.org wrote: >> Hi Trevor, >> >> On 9/9/24 4:55 PM, Trevor Woerner via lists.yoctoproject.org wrote: >>> From: Quentin Schulz <quentin.schulz@cherry.de> >>> >>> v4l2codecs is the gstreamer plugin for V4L2 stateless video hardware >>> decoding. The Rockchip SoCs that have a VPU all seems to be based on >>> Hantro, RKVDEC or RKVDECv2, all stateless encoding/decoding VPUs. >>> >>> Therefore, let's enable VPU decoding in Gstreamer whenever possible, >>> when the SoC supports it. >>> >>> PX30, RK3066, RK3188, RK3288, RK3328, RK3399, RK356x and RK3588(s) all >>> have at least one Hantro VPU. >>> >>> RK3328, RK3399, RK356x and RK3588(s) all have at least one >>> RKVDEC/RKVDECv2 VPU (though not necessarily supported in the upstream >>> kernel just yet). >>> >>> === PX30 >>> Tested on PX30 Ringneck with with Haikou+Haikou Video Demo adapter: >>> >>> $ gst-launch-1.0 filesrc location=$FILE ! parsebin ! v4l2slh264dec ! waylandsink >>> >>> with FILE storing the path to any h264 file, e.g. >>> https://download.blender.org/peach/bigbuckbunny_movies/big_buck_bunny_720p_h264.mov >>> https://download.blender.org/peach/bigbuckbunny_movies/big_buck_bunny_1080p_h264.mov >>> >>> Needed packages are: >>> - weston >>> - gstreamer1.0-plugins-bad (for waylandsink and v4l2slh264dec) >>> - gstreamer1.0-plugins-base (for parsebin) >>> >> >> Weston output is broken on Scarthgap. c.f. >> https://lore.kernel.org/all/1f84dd9c-bdef-4099-9dba-04b445a7ee0a@cherry.de/ >> >>> A few frames are dropped every other second for 1080p but otherwise >>> smooth. >>> >>> === RK3399 >>> Tested on RK3399 Puma with Haikou: >>> >>> $ gst-launch-1.0 filesrc location=$FILE ! parsebin ! v4l2slh264dec ! waylandsink >>> >>> with FILE storing the path to any h264 file, e.g. >>> https://download.blender.org/peach/bigbuckbunny_movies/big_buck_bunny_1080p_h264.mov >>> https://download.blender.org/demo/movies/BBB/bbb_sunflower_2160p_30fps_normal.mp4.zip >>> >>> Needed packages are: >>> - weston >>> - gstreamer1.0-plugins-bad (for waylandsink and v4l2codecs) >>> - gstreamer1.0-plugins-base (for parsebin) >>> >> >> Weston output is broken on Scarthgap. c.f. >> https://lore.kernel.org/all/1f84dd9c-bdef-4099-9dba-04b445a7ee0a@cherry.de/ >> >>> === RK3588 >>> >>> Tested on a RK3588 Tiger with Haikou+Haikou Video Demo adapter - on a >>> downstream v6.6 (upstream-based, not Rockchip BSP-based) with DSI >>> patches - : >>> >>> $ gst-launch-1.0 filesrc location=$FILE ! parsebin ! v4l2slav1dec ! fakesink >>> >>> with FILE storing the path to any AV1 file, e.g. >>> http://download.opencontent.netflix.com.s3.amazonaws.com/AV1/cmaf/spark-8b-59.94fps/spark_606kbps_432p.mp4 >>> https://woolyss.com/f/av1-nosound-chimera.mp4 >>> https://woolyss.com/f/av1-opus-sita.webm >>> >>> Needed packages are: >>> - gstreamer1.0-plugins-bad (for fakesink and v4l2slav1dec) >>> - gstreamer1.0-plugins-base (for parsebin) >>> >> >> Stateless AV1 decoding is not available in the mesa release used on >> Scarthgap. >> >> So... >> >> RK3588 seems to have gained upstream support for VDPU121 which should allow >> h264, mpeg2 and vp8 decoding. Building a linux-next kernel after tag >> next-20240829 should have the support AFAICT. >> >> So, I would retest RK3588 support for those video formats and drop the AV1 >> mention from the commit log as it for sure doesn't work because of the >> version of mesa in Scarthgap. We could backport later releases into >> meta-rockchip to gain support for it. I think this **may** be an idea once >> there's support for 24.1 and later merged into OE-Core because it brings >> Panthor (open-source GPU drivers) support to RK3588 which would be a shame >> not to have in an LTS. But let's see when that happens. >> >> >> >> As for the issues with display on PX30 and RK3399, it seems those are >> supposed to be patched in newer releases of gstreamer, c.f. >> https://gitlab.freedesktop.org/wayland/weston/-/issues/839. Note that >> Robert said it would be fixed in gstreamer 1.22 but the linked patch >> only made it to 1.23 (Scarthgap has 1.22, Styhead 1.24). >> >> Considering what Robert said in the link above, I assume people using >> another compositor than weston may be fine on Scarthgap, so I would suggest >> to backport the patches I submitted in this series to scarthgap anyway? One >> is not forced to use weston to do VPU decoding. >> FYI, I gave ~30min at backporting the gstreamer patches to 1.22.11 but this >> is above my skill set today. >> >> For those, I would just mention the issue in the README and commit log and >> maybe switch to fakesink in the commit log instead of waylandsink (and drop >> weston from the needed packages). >> >> What do you think? > > To be clear, when I say I've run-tested this build I'm not implying I've > tested every aspect, feature, and functionality of each chip. I've simply > flashed an SD card, applied power, watched it boot to a prompt, log in, and > run something simple like "uname -a" to verify it's using the scarthgap kernel > and not the latest master. I'm not really in a position to test stateless VPU > decoding or anything fancy like that :-) > > If there aren't any issues or objections, perhaps I could start by applying > the other patches in this series while considering what to do about this one? I haven't had the time to look at the other patches in depth just yet. But I agree with the suggestion of putting this one patch on hold. I currently have this enabled in our vendor layer: https://git.embedded.cherry.de/yocto-layers/meta-cherry-es.git/commit/?id=37422a2f8d2553701dfa80de2bde5537ffffa770 https://git.embedded.cherry.de/yocto-layers/meta-cherry-es.git/commit/?id=a72a8bbabd5721f25ccef369ac470ea5d27b864c https://git.embedded.cherry.de/yocto-layers/meta-cherry-es.git/commit/?id=7c575990a66490e8907376a75c784b15c484b0c7 So I clearly am interested in having this work with meta-rockchip in the Scarthgap branch. It's on my TODO list. I also would like to test h264, mpeg2 and vp8 on RK3588 on Scarthgap too :) So basically, I'm volunteering for this as I have professional interest in this, but if someone does this before I have time to handle it, I'll just review it whenever I have time :) > Maybe after the others are applied you'd like to have a go at making a > scarthgap version of this patch (with all the proper commit messages etc)? > Sounds good to me. I'll try to find time to review the other backports ASAP. Cheers, Quentin
diff --git a/README b/README index 8ff006e942b9..09f8691300f4 100644 --- a/README +++ b/README @@ -103,6 +103,28 @@ Notes: this layer, perform the same steps as above except for the step enabling RK_RAUC_DEMO. + HW video decoding with gstreamer + + Most Rockchip SoCs have some integrated VPU, either Hantro, RKVDEC or + RKVDECv2. Those are called stateless VPUs and they require tools to be + adapted so they can be used. Upstream ffmpeg doesn't support them (yet) + but gstreamer does, via the v4l2sl* plugins. However, by default those + aren't built in OE-Core. This layer enables these plugins by default for + all SoCs that have at least one VPU (at the time of writing, all SoCs + supported by this layer to the exception of RK3308). + + If you wish to NOT build those plugins anymore, you can set + + ENABLE_STATELESS_VPU_GST = "0" + + in any appropriate conf file (or in a + gstreamer1.0-plugins-bad_%.bbappend in your own layer). + + Note that this is only relevant for upstream-based Linux kernels as + Rockchip have developed their own ABI for their own kernel that isn't + compatible (see Rockchip Media Process Platform (MPP) and + downstream gstreamer-rockchip plugin for those kernels). + U-Boot Environment: ------------------ In order to configure U-Boot to be able to store its environment into the diff --git a/conf/machine/include/px30.inc b/conf/machine/include/px30.inc index d78409ee5c7e..8173cb19be2c 100644 --- a/conf/machine/include/px30.inc +++ b/conf/machine/include/px30.inc @@ -19,3 +19,5 @@ TFA_BUILD_TARGET = "bl31" UBOOT_SUFFIX ?= "itb" UBOOT_ENTRYPOINT ?= "0x06000000" + +ENABLE_STATELESS_VPU_GST ?= "1" diff --git a/conf/machine/include/rk3066.inc b/conf/machine/include/rk3066.inc index 026d5b556db7..a898309bbf88 100644 --- a/conf/machine/include/rk3066.inc +++ b/conf/machine/include/rk3066.inc @@ -12,3 +12,5 @@ SERIAL_CONSOLES = "115200;ttyS2" KBUILD_DEFCONFIG = "multi_v7_defconfig" KERNEL_FEATURES:append:rk3066 = " bsp/rockchip/remove-non-rockchip-arch-arm.scc" KERNEL_IMAGETYPE ?= "zImage" + +ENABLE_STATELESS_VPU_GST ?= "1" diff --git a/conf/machine/include/rk3188.inc b/conf/machine/include/rk3188.inc index 0120e2fd585f..554d4f1c904d 100644 --- a/conf/machine/include/rk3188.inc +++ b/conf/machine/include/rk3188.inc @@ -12,3 +12,5 @@ SERIAL_CONSOLES = "115200;ttyFIQ0" KBUILD_DEFCONFIG = "multi_v7_defconfig" KERNEL_FEATURES:append:rk3188 = " bsp/rockchip/remove-non-rockchip-arch-arm.scc" KERNEL_IMAGETYPE ?= "zImage" + +ENABLE_STATELESS_VPU_GST ?= "1" diff --git a/conf/machine/include/rk3288.inc b/conf/machine/include/rk3288.inc index 684e8645d176..06fda69a3eb7 100644 --- a/conf/machine/include/rk3288.inc +++ b/conf/machine/include/rk3288.inc @@ -14,3 +14,5 @@ KERNEL_FEATURES:append:rk3288 = " bsp/rockchip/remove-non-rockchip-arch-arm.scc" KERNEL_IMAGETYPE ?= "zImage" UBOOT_SUFFIX ?= "bin" + +ENABLE_STATELESS_VPU_GST ?= "1" diff --git a/conf/machine/include/rk3328.inc b/conf/machine/include/rk3328.inc index 4ecd6814f9ba..e6f810dcd2ca 100644 --- a/conf/machine/include/rk3328.inc +++ b/conf/machine/include/rk3328.inc @@ -20,3 +20,5 @@ TFA_BUILD_TARGET = "bl31" UBOOT_SUFFIX ?= "itb" UBOOT_ENTRYPOINT ?= "0x06000000" + +ENABLE_STATELESS_VPU_GST ?= "1" diff --git a/conf/machine/include/rk3399.inc b/conf/machine/include/rk3399.inc index 47f0560d1578..cd1be49064ed 100644 --- a/conf/machine/include/rk3399.inc +++ b/conf/machine/include/rk3399.inc @@ -20,3 +20,5 @@ TFA_BUILD_TARGET = "bl31" UBOOT_SUFFIX ?= "itb" UBOOT_ENTRYPOINT ?= "0x06000000" + +ENABLE_STATELESS_VPU_GST ?= "1" diff --git a/conf/machine/include/rk3566.inc b/conf/machine/include/rk3566.inc index 586d68806df6..2522851fa392 100644 --- a/conf/machine/include/rk3566.inc +++ b/conf/machine/include/rk3566.inc @@ -19,3 +19,5 @@ PREFERRED_PROVIDER_optee-os = "rockchip-rkbin" UBOOT_SUFFIX ?= "itb" UBOOT_ENTRYPOINT ?= "0x06000000" + +ENABLE_STATELESS_VPU_GST ?= "1" diff --git a/conf/machine/include/rk3568.inc b/conf/machine/include/rk3568.inc index 1da212a1530c..db188a311297 100644 --- a/conf/machine/include/rk3568.inc +++ b/conf/machine/include/rk3568.inc @@ -19,3 +19,5 @@ PREFERRED_PROVIDER_optee-os = "rockchip-rkbin" UBOOT_SUFFIX ?= "itb" UBOOT_ENTRYPOINT ?= "0x06000000" + +ENABLE_STATELESS_VPU_GST ?= "1" diff --git a/conf/machine/include/rk3588s.inc b/conf/machine/include/rk3588s.inc index cd84461dab41..d349108fb973 100644 --- a/conf/machine/include/rk3588s.inc +++ b/conf/machine/include/rk3588s.inc @@ -18,3 +18,5 @@ PREFERRED_PROVIDER_optee-os = "rockchip-rkbin" UBOOT_SUFFIX ?= "itb" UBOOT_ENTRYPOINT ?= "0x06000000" + +ENABLE_STATELESS_VPU_GST ?= "1" diff --git a/recipes-multimedia/gstreamer/gstreamer1.0-plugins-bad_%.bbappend b/recipes-multimedia/gstreamer/gstreamer1.0-plugins-bad_%.bbappend new file mode 100644 index 000000000000..276230411e9c --- /dev/null +++ b/recipes-multimedia/gstreamer/gstreamer1.0-plugins-bad_%.bbappend @@ -0,0 +1 @@ +PACKAGECONFIG:append:rockchip = "${@' v4l2codecs' if bb.utils.to_boolean(d.getVar('ENABLE_STATELESS_VPU_GST'), False) else ''}"