[Bug]: bgr0 -> nv12 conversion looks bad

[markg85]

Describe the bug
Linux desktop streaming. I'm using x11grab which is a pixel format of bgr0.
AMF has a mapping for that: AV_PIX_FMT_BGR0 maps to AMF_SURFACE_BGRA

To be fair, it's just a mapping. It's not wired up to HEVC. So i did add that one myself. In amfenc.c i just added AV_PIX_FMT_BGR0 in the ff_amf_pix_fmts array.

Compiling ffmpeg with this makes it work and lets bgr0 be handled by AMF. Sweet! The more that can be offloaded to AMF in the encoding pipeline the better it is for latency purposes.

The following table shows the conversion as i think they happen. I do have to put a big fat disclaimer here. I'm assuming the cpu does a pix format conversion when pix_format is supplied and that it does that conversion to a format the driver (hevc_amf) supports. I'm also assuming that no cpu conversion happens when pix_format is not supplied.

Also note that no pix format in the table means i didn't supply pix_format as argument thus it's the source format. Which is bgr0.

pix_format	CPU conversion	AMF conversion	Picture
none	none	bgr0 -> nv12
nv12	bgr0 -> nv12	none
p010	bgr0 -> p010	p010 -> nv12

By far the best one in that table is that last one, p010. The color representation is good and 10 bit (irrelevant for this issue) means less banding too, which you can see in the grey gradient behind the AMD logo if you look carefully!

You can easily see that the first row looks very off. In theory this would be the best one as AMF would do all the work.
In practice it looks like the AMF conversion from bgr0 -> nv12 is not doing the same as the cpu based one when nv12 is provided as pix_format.

At this point i'm not entirely convinced the issue is in ffmpeg (x11grab specifically) or in AMF. To try and rule out x11grab, i did modify it's code to output BGRA instead of BGR0 but the results were the same.

Is this a hardware bug?

Setup (please complete the following information):

• OS: ArchLinux
• Driver Version 1.4.35 + latest header changes from this repo
• GPU 7900 XT

[MikhailAMD]

Few things:

1. Instead of adding BGR0 to AMF encoder's list of input formats, with the latest patches you can use amf context and hwupload:
   -vf 'hwupload'. With that encoder input will be AV_PIX_FMT_AMF_SURFACE. You will also need "-hwaccel amf"
2. Most of AMD VCN encoders like your 7900 XT support BGRA(0,A)/RGB(0,A) etc. directly, meaning that color conversion happens inside VCN. If used, it will improve performance.
3. If VCN doesn't support HW conversion, it will be done by AMF color converter using Compute. Can be internal converter inside AMF encoder or new vpp_amf filter. New filter also can do scaling.
   -vf vpp_amf=1920:1080:format=rgba (or bgra or nv2 etc)
4. AMF color converter doesn't yet support proper conversion between 8 and 10 bit YUVs (NV12 <-> P010). It is coming.
5. But if you want 10-bit, you can submit to encoder R8G8B8A8, B8G8R8A8 or R10G10B10A2 or RGBA_F16 or P010 and configure encoder to HEVC 10-bit or AV1 10-bit.

[markg85]

1. Instead of adding BGR0 to AMF encoder's list of input formats, with the latest patches you can use amf context and hwupload:
   -vf 'hwupload'. With that encoder input will be AV_PIX_FMT_AMF_SURFACE. You will also need "-hwaccel amf"
2. Most of AMD VCN encoders like your 7900 XT support BGRA(0,A)/RGB(0,A) etc. directly, meaning that color conversion happens inside VCN. If used, it will improve performance.

My command is getting "a little" complicated...

export LD_LIBRARY_PATH=/home/mark/pkgbuilds/ffmpeg/upstream/build/lib
./ffmpeg -hwaccel amf -hwaccel_device /dev/dri/renderD128 -init_hw_device amf=/dev/dri/renderD128 \
    -f x11grab -framerate 60 -max_delay 0 -max_probe_packets 1 -thread_queue_size 1024 -video_size 2560x1440 -r 60 -i :0.0+0,0 \
    -vf 'hwupload' \
    -c:v hevc_amf \
    -log_to_dbg 1 \
    -usage lowlatency \
    -rc vbr_peak \
    -minrate 2M \
    -maxrate 15M \
    -quality quality \
    -preset quality \
    -vbaq 0 -enforce_hrd 0 -aud 0 -latency 1 -filler_data 0  \
    -profile:v main10 \
    -profile_tier high \
    -b:v 8M \
    -f nut -muxdelay 0 udp://10.0.3.55:5555

It took some fiddling to get that hwupload to work. Turns out the init_hw_device is required for that. At least in the case of AMF. I did undo my changes to both x11grab and amf, it's now the same as your patches + upstream.

Some logging that might be helpful:

[graph -1 input from stream 0:0 @ 0x7ed8580045c0] w:2560 h:1440 pixfmt:bgr0 tb:1/60 fr:60/1 sar:0/1 csp:unknown range:unknown
[format @ 0x7ed858004c40] Setting 'pix_fmts' to value 'nv12|yuv420p|p010le|amf'
[format @ 0x7ed858004c40] Setting 'color_ranges' to value 'tv|pc'
[AVFilterGraph @ 0x7ed858001000] query_formats: 4 queried, 9 merged, 0 already done, 0 delayed
[hwupload @ 0x7ed858004240] Surface format is bgr0.
[graph -1 input from stream 0:0 @ 0x7ed8580045c0] video frame properties congruent with link at pts_time: 0
[vost#0:0/hevc_amf @ 0x5ddca8233980] [enc:hevc_amf @ 0x5ddca8233e80] Using input frames context (format amf) with hevc_amf encoder.

I don't see any message from ffmpeg that it's converting formats so i assume it's passed on to AMF (or VCN).
Unfortunately, the color result is exactly the same as previously (the wrong washed out colors). Is there anything i'm doing wrong in my parameters?

[MikhailAMD]

try to use -color_range full
Do you see incorrect colors or washed-out text?
Can you share the original image to compare?

[markg85]

With hwupload and color_range full:

And this is how i expect it to be (provided pix_format p010 and removed hwupload)

[MikhailAMD]

If you pass pix_fmt P010 to AMF HEVC encoder, it will initialize with HEVC 10-bit instead of default 8-bit. Yes, colors will be better.
I guess, FFmpeg AMF encoder code can be improved to force RGBA input + 10-bit explicitly to use internal conversion. I will ask for a patch.

[markg85]

If you pass pix_fmt P010 to AMF HEVC encoder, it will initialize with HEVC 10-bit instead of default 8-bit. Yes, colors will be better. I guess, FFmpeg AMF encoder code can be improved to force RGBA input + 10-bit explicitly to use internal conversion. I will ask for a patch.

I'm doubtful if that will work.
These two images from my last post, both had 10 bit. You can see it in the gradient behind the AMD logo, essentially no banding. So i'm quite sure the encoder already does 10 bit internally. What i'm guessing is the A channel in RGBA (or BGR0) to be interpreted a little differently in the hardware path. Can you confirm if that's the case?

[MikhailAMD]

Two different issues:

1. Encoder bit-ness: currently the code is this:
   color_depth = AMF_COLOR_BIT_DEPTH_8;
   pix_fmt = avctx->hw_frames_ctx ? ((AVHWFramesContext*)avctx->hw_frames_ctx->data)->sw_format
   : avctx->pix_fmt;
   if (pix_fmt == AV_PIX_FMT_P010) {
   color_depth = AMF_COLOR_BIT_DEPTH_10;
   }
   AMF_ASSIGN_PROPERTY_INT64(res, ctx->encoder, AMF_VIDEO_ENCODER_HEVC_COLOR_BIT_DEPTH, color_depth);
   Unless pix_fmt == AV_PIX_FMT_P010, HEVC will be 8-bit.

2. Bit-ness and format of captured images:
   Unless you enable HDR for display, the captured image is RGBA8 (RGB0). There are two options you tried:
   AMF conversion: RGBA8->NV12->HEVC-8
   FFmpeg conversion RGBA8->P010->HEVC-10
   I assume you stream to another computer and check colors on the client. Bit-ness give different results in colors.

If you want to check bit-ness of your output for these two cases, just write few frames to a file: -frames 100 -f out.mkv. You can also check color_range in the output HEVC stream.

[markg85]

Ahh, i see what you mean!
Interesting how banding (the lack thereof in fact) fooled me in this instance.

It indeed has yuv420 as output format when passing bgr0.

I patched ffmpeg, re-added BGR0 to the formats for HEVC and I unconditionally enabled AMF_COLOR_BIT_DEPTH_10. The output now does have 10 bit (yuv420p10le) but it's completely solid green!

[MikhailAMD]

Well... Something is still wrong. Could be your changes, command line, FFmpeg integration, general AMF, Linux specific things. It would help if you get some simple FFmpeg RGBA input like -mandelbrot filter or similar, make full command line with your parameters writing to a MP4 or MKV file and share the line. Also you can enable detailed FFmpeg log which in turn will enable detailed AMF log - part of these patches and share the log.

[markg85]

Ok, will do!
I do have to modify more. Even to get logging. Because.. errr...

❯ cppgrep "log_to_dbg"
libavcodec/amfenc_hevc.c:109:    { "log_to_dbg",     "Enable AMF logging to debug output",   OFFSET(log_to_dbg), AV_OPT_TYPE_BOOL,{ .i64 = 0 }, 0, 1, VE },
libavcodec/amfenc_h264.c:143:    { "log_to_dbg",     "Enable AMF logging to debug output",   OFFSET(log_to_dbg)    , AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, VE },
libavcodec/amfenc.h:64:    int                 log_to_dbg;
libavcodec/amfenc_av1.c:132:    { "log_to_dbg",     "Enable AMF logging to debug output",   OFFSET(log_to_dbg), AV_OPT_TYPE_BOOL,{.i64 = 0 }, 0, 1, VE },

It's not being used 😒 This wasn't the case before. (same for forced_idr, haven't looked at the rest). Am i slowly becoming an AMF dev here? lol

[markg85]

Right, nope. This is a little more involved then just enabling one little thing.. We're now in AMF vtable and initialization magic. I'll happily apply a patch if you have one for me that enables log output else this is probably where i draw the line and rather wait for updated patches.

[MikhailAMD]

We would welcome you as dev :)
log_to_dbg seem obsolete with these patches. Need to remove it.
in these patches AMF logging level is connected to FFmpeg. just add -loglevel debug

[markg85]

We would welcome you as dev :)
Hah 😆

That doesn't show me much AMF logging at all..
I know how AMF logging "can" look from before this patch series. It doesn't seem like any of that is in the output.
Do i need to enable something else too?

Here's an output:

./ffmpeg \
    -f x11grab -framerate 60 -max_delay 0 -max_probe_packets 1 -thread_queue_size 1024 -video_size 2560x1440 -r 60 -i :0.0+0,0 \
    -c:v hevc_amf \
    -log_to_dbg 1 \
    -loglevel debug \
    -usage lowlatency \
    -rc vbr_peak \
    -minrate 2M \
    -maxrate 15M \
    -quality quality \
    -preset quality \
    -vbaq 0 -enforce_hrd 0 -aud 0 -latency 1 -filler_data 0  \
    -profile:v main10 \
    -profile_tier high \
    -b:v 8M \
    -pix_fmt bgr0 \
    -f nut -muxdelay 0 udp://10.0.3.55:5555
ffmpeg version N-118369-g959b799c8d Copyright (c) 2000-2025 the FFmpeg developers
  built with gcc 14.2.1 (GCC) 20240910
  configuration: --prefix=/home/mark/pkgbuilds/ffmpeg/upstream/build --disable-debug --disable-static --disable-stripping --enable-amf --enable-avisynth --enable-cuda-llvm --enable-lto --enable-fontconfig --enable-frei0r --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libdav1d --enable-libdrm --enable-libdvdnav --enable-libdvdread --enable-libfreetype --enable-libfribidi --enable-libglslang --enable-libgsm --enable-libharfbuzz --enable-libiec61883 --enable-libjack --enable-libjxl --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libplacebo --enable-libpulse --enable-librav1e --enable-librsvg --enable-librubberband --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpl --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-nvdec --enable-nvenc --enable-opencl --enable-opengl --enable-shared --enable-vapoursynth --enable-version3 --enable-vulkan
  libavutil      59. 56.100 / 59. 56.100
  libavcodec     61. 31.101 / 61. 31.101
  libavformat    61.  9.106 / 61.  9.106
  libavdevice    61.  4.100 / 61.  4.100
  libavfilter    10.  9.100 / 10.  9.100
  libswscale      8. 13.100 /  8. 13.100
  libswresample   5.  4.100 /  5.  4.100
  libpostproc    58.  4.100 / 58.  4.100
Splitting the commandline.
Reading option '-f' ... matched as option 'f' (force container format (auto-detected otherwise)) with argument 'x11grab'.
Reading option '-framerate' ... matched as AVOption 'framerate' with argument '60'.
Reading option '-max_delay' ... matched as AVOption 'max_delay' with argument '0'.
Reading option '-max_probe_packets' ... matched as AVOption 'max_probe_packets' with argument '1'.
Reading option '-thread_queue_size' ... matched as option 'thread_queue_size' (set the maximum number of queued packets from the demuxer) with argument '1024'.
Reading option '-video_size' ... matched as AVOption 'video_size' with argument '2560x1440'.
Reading option '-r' ... matched as option 'r' (override input framerate/convert to given output framerate (Hz value, fraction or abbreviation)) with argument '60'.
Reading option '-i' ... matched as input url with argument ':0.0+0,0'.
Reading option '-c:v' ... matched as option 'c' (select encoder/decoder ('copy' to copy stream without reencoding)) with argument 'hevc_amf'.
Reading option '-log_to_dbg' ... matched as AVOption 'log_to_dbg' with argument '1'.
Reading option '-loglevel' ... matched as option 'loglevel' (set logging level) with argument 'debug'.
Reading option '-usage' ... matched as AVOption 'usage' with argument 'lowlatency'.
Reading option '-rc' ... matched as AVOption 'rc' with argument 'vbr_peak'.
Reading option '-minrate' ... matched as AVOption 'minrate' with argument '2M'.
Reading option '-maxrate' ... matched as AVOption 'maxrate' with argument '15M'.
Reading option '-quality' ... matched as AVOption 'quality' with argument 'quality'.
Reading option '-preset' ... matched as AVOption 'preset' with argument 'quality'.
Reading option '-vbaq' ... matched as AVOption 'vbaq' with argument '0'.
Reading option '-enforce_hrd' ... matched as AVOption 'enforce_hrd' with argument '0'.
Reading option '-aud' ... matched as AVOption 'aud' with argument '0'.
Reading option '-latency' ...Routing option latency to both codec and muxer layer
 matched as AVOption 'latency' with argument '1'.
Reading option '-filler_data' ... matched as AVOption 'filler_data' with argument '0'.
Reading option '-profile:v' ... matched as option 'profile' (set profile) with argument 'main10'.
Reading option '-profile_tier' ... matched as AVOption 'profile_tier' with argument 'high'.
Reading option '-b:v' ... matched as option 'b' (video bitrate (please use -b:v)) with argument '8M'.
Reading option '-pix_fmt' ... matched as option 'pix_fmt' (set pixel format) with argument 'bgr0'.
Reading option '-f' ... matched as option 'f' (force container format (auto-detected otherwise)) with argument 'nut'.
Reading option '-muxdelay' ... matched as option 'muxdelay' (set the maximum demux-decode delay) with argument '0'.
Reading option 'udp://10.0.3.55:5555' ... matched as output url.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option loglevel (set logging level) with argument debug.
Successfully parsed a group of options.
Parsing a group of options: input url :0.0+0,0.
Applying option f (force container format (auto-detected otherwise)) with argument x11grab.
Applying option thread_queue_size (set the maximum number of queued packets from the demuxer) with argument 1024.
Applying option r (override input framerate/convert to given output framerate (Hz value, fraction or abbreviation)) with argument 60.
Successfully parsed a group of options.
Opening an input file: :0.0+0,0.
[x11grab @ 0x63f2824cae40] Probe buffer size limit of 5000000 bytes reached
[x11grab @ 0x63f2824cae40] Stream #0: not enough frames to estimate rate; consider increasing probesize
Input #0, x11grab, from ':0.0+0,0':
  Duration: N/A, start: 1738173398.460660, bitrate: 7077888 kb/s
  Stream #0:0, 1, 1/1000000: Video: rawvideo, 1 reference frame (BGR[0] / 0x524742), bgr0, 2560x1440, 0/1, 7077888 kb/s, 60 fps, 1000k tbr, 1000k tbn
Successfully opened the file.
Parsing a group of options: output url udp://10.0.3.55:5555.
Applying option c:v (select encoder/decoder ('copy' to copy stream without reencoding)) with argument hevc_amf.
Applying option profile:v (set profile) with argument main10.
Applying option b:v (video bitrate (please use -b:v)) with argument 8M.
Applying option pix_fmt (set pixel format) with argument bgr0.
Applying option f (force container format (auto-detected otherwise)) with argument nut.
Applying option muxdelay (set the maximum demux-decode delay) with argument 0.
Successfully parsed a group of options.
Opening an output file: udp://10.0.3.55:5555.
[out#0/nut @ 0x63f2824d3e80] No explicit maps, mapping streams automatically...
[vost#0:0/hevc_amf @ 0x63f2824d5740] Created video stream from input stream 0:0
[udp @ 0x63f2824d8cc0] No default whitelist set
Successfully opened the file.
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> hevc (hevc_amf))
[vost#0:0/hevc_amf @ 0x63f2824d5740] Starting thread...
[vf#0:0 @ 0x63f2824d6280] Starting thread...
[vist#0:0/rawvideo @ 0x63f2824d3bc0] [dec:rawvideo @ 0x63f2824d7a00] Starting thread...
[in#0/x11grab @ 0x63f2824cab80] Starting thread...
Press [q] to stop, [?] for help
[rawvideo @ 0x63f2824d7f00] PACKET SIZE: 14745600, STRIDE: 10240
detected 32 logical cores
[graph -1 input from stream 0:0 @ 0x751570003f80] w:2560 h:1440 pixfmt:bgr0 tb:1/60 fr:60/1 sar:0/1 csp:unknown range:unknown
[format @ 0x7515700045c0] Setting 'pix_fmts' to value 'bgr0'
[format @ 0x7515700045c0] Setting 'color_ranges' to value 'tv|pc'
[AVFilterGraph @ 0x751570001000] query_formats: 4 queried, 9 merged, 0 already done, 0 delayed
[graph -1 input from stream 0:0 @ 0x751570003f80] video frame properties congruent with link at pts_time: 0
[rawvideo @ 0x63f2824d7f00] PACKET SIZE: 14745600, STRIDE: 10240
    Last message repeated 2 times
[AVHWDeviceContext @ 0x7515700057c0] AMF initialisation succeeded via Vulkan.
[rawvideo @ 0x63f2824d7f00] PACKET SIZE: 14745600, STRIDE: 10240
0125-00-29 18:56:38 7D7FF6C0 [EncodeQueuePalImpl] Warning: EncodeQueueServicePalImpl::CreateEncodeQueue(): unsupported queue type 4 requested, fallback to queue type 2.
[AVHWDeviceContext @ 0x7515700057c0] EncodeQueuePalImpl: 0125-00-29 18:56:38 7D7FF6C0 [EncodeQueuePalImpl] Warning: EncodeQueueServicePalImpl::CreateEncodeQueue(): unsupported queue type 4 requested, fallback to queue type 2.
Output #0, nut, to 'udp://10.0.3.55:5555':
  Metadata:
    encoder         : Lavf61.9.106
  Stream #0:0, 0, 1/61440: Video: hevc, 1 reference frame (HEVC / 0x43564548), bgr0(pc, gbr/unknown/unknown, progressive), 2560x1440, 0/1, q=2-31, 8000 kb/s, 60 fps, 61440 tbn
    Metadata:
      encoder         : Lavc61.31.101 hevc_amf
[out#0/nut @ 0x63f2824d3e80] Starting thread...
[rawvideo @ 0x63f2824d7f00] PACKET SIZE: 14745600, STRIDE: 10240
    Last message repeated 25 times
[rawvideo @ 0x63f2824d7f00] PACKET SIZE: 14745600, STRIDE: 10240rate=  64.9kbits/s speed=0.967x    
    Last message repeated 29 times
[rawvideo @ 0x63f2824d7f00] PACKET SIZE: 14745600, STRIDE: 10240rate=  54.6kbits/s speed=0.983x    
    Last message repeated 29 times
[rawvideo @ 0x63f2824d7f00] PACKET SIZE: 14745600, STRIDE: 10240rate=  51.2kbits/s speed=0.989x    
    Last message repeated 29 times
[rawvideo @ 0x63f2824d7f00] PACKET SIZE: 14745600, STRIDE: 10240rate=  49.6kbits/s speed=0.991x    
    Last message repeated 29 times
[rawvideo @ 0x63f2824d7f00] PACKET SIZE: 14745600, STRIDE: 10240rate=  48.6kbits/s speed=0.993x    
    Last message repeated 29 times
[rawvideo @ 0x63f2824d7f00] PACKET SIZE: 14745600, STRIDE: 10240rate=  47.9kbits/s speed=0.994x    
    Last message repeated 29 times

As an aside, my ffmpeg is compiled with --disable-debug. Could it be that this logging you're talking about is only there when compiling ffmpeg in debug mode?

[MikhailAMD]

Compile option doesn't matter. AMF logs from installed SO.
AMF logging is enabled from amf_device_create. There see the check for AMF v1.4.35. It forces trace level to warnings only to avoid the memory corruption you reported before. You may remove this while you don't use PA.