[Bug]: Issues with 8-bit to 10 bit conversion and reverse. (Forcing bit depth also enforces color primary conversion).

[DimkaTsv]

Describe the bug
For some reason if you enforce bit depth and it does not align with source (aka 8bit --> 10bit and 10bit-->8bit respectively), resulting video will have distorted colors due to setting wrong color primaries (and doing conversion based on them).

For HEVC 8bit encodes BT.709+BT.709 is enforced, while for 10bit BT.2020+PQ. Mastering data is lost on encode (unless specified).
For AV1 encodes color primary data is stripped, but mastering for HDR is transferred without specification.

Weirdly TranscodeHW and VCEEncC (after update which dropped most it's defaults to AMF defaults) break colors in different ways.
TranscodeHW makes 8->10bit outputs look as low contrast,10->8bit look similar way.
While VCEEncC 8->10bit looks oversaturated, while 10->8bit looks bleak.

UPD: Oh no it's even worse than i thought... I had HEVC10 source with BT.709 color primaries lying around. Well, if i enforce 8-bit encode via TranscodeHW, it gets completely ruined.
At least VCEEncC could do HEVC 10bit BT.709 -> HEVC 8bit without completely ruining picture.

For VCEEncC:
BT.709 -> BT.709 conversion is always correct.
BT.709 -> BT.2020 causes oversaturated result.
BT.2020->BT.709 causes bleak result. Can be fixed by transferring HDR parameters.

For TranscodeHW:
BT.709->BT.709 conversion completely breaks with 8bit->10bit encode.
BT.709->BT.2020 causes image with less contrast, but colors are preserved.
BT.2020->BT.709 causes image with less contrast, but colors are preserved.

I assume that stems from source detection and what color primaries encoder is made to think source in? VCEEncC always presumes that source is BT.709
TranscodeHW always presumes that 8-bit source is BT.709, while 10-bit source is BT.2020. But why does it darken outputs?

To Reproduce
Used software:
TranscodeHW - from latest AMF repo. Just cloned and built it before testing.
VCEEncC 8.27 (also latest on moment of writing)

Used commands:

@echo off
TranscodeHW command list.
:: HEVC 10 bit
%~dp0\TranscodeHW.exe -input %1 -output "%~n1_TranscodeHW_HEVC10%~x1" -codec hevc -HevcRateControlMethod CQP -HevcQP_I 26 -HevcQP_P 26 -HevcColorBitDepth 10
pause

:: HEVC 8 bit
%~dp0\TranscodeHW.exe -input %1 -output "%~n1_TranscodeHW_HEVC8%~x1" -codec hevc -HevcRateControlMethod CQP -HevcQP_I 26 -HevcQP_P 26 -HevcColorBitDepth 8
pause

:: AV1 10bit (i only had 1080p source of 10 bit video in BT.709, hence alignment)
%~dp0\TranscodeHW.exe -input %1 -output "%~n1_TranscodeHW_AV110%~x1" -codec av1 -Av1RateControlMethod CQP  -Av1QIndex_Inter 26 -Av1QIndex_Intra 26 -Av1ColorBitDepth 10 -Av1AlignmentMode 2
pause

:: AV1 8bit (i only had 1080p source of 10 bit video in BT.709, hence alignment)
%~dp0\TranscodeHW.exe -input %1 -output "%~n1_TranscodeHW_AV18%~x1" -codec av1 -Av1RateControlMethod CQP  -Av1QIndex_Inter 26 -Av1QIndex_Intra 26 -Av1ColorBitDepth 8 -Av1AlignmentMode 2
pause

VCEEncC command list.

:: HEVC 8 bit
%~dp0\VCEEncC64.exe -i %1 -o "%~n1_processed_hevc10%~x1" --avsw --codec hevc --cqp 26 --preset slow --profile main --output-depth 10 --audio-copy
pause

:: HEVC 10 bit
%~dp0\VCEEncC64.exe -i %1 -o "%~n1_processed_hevc8%~x1" --avsw --codec hevc --cqp 26 --preset slow --profile main --output-depth 8 --audio-copy
pause

:: AV1 10bit
%~dp0\VCEEncC64.exe -i %1 -o "%~n1_processed_av110%~x1" --avsw --codec av1 --cqp 26 --preset slow --profile main --output-depth 10 --audio-copy
pause

:: AV1 8bit
%~dp0\VCEEncC64.exe -i %1 -o "%~n1_processed_av18%~x1" --avsw --codec av1 --cqp 26 --preset slow --profile main --output-depth 8 --audio-copy
pause

Setup (please complete the following information):

• OS: Windows 11, 23H2, build 22631.4602
• Driver Version: 24.12.1
• GPU: 7800XT
• Which component has the issue [e.g. Encoder, Decoder]: Encoder.

Screenshots
No debug logs, encode proceeds properly. But outputs are messed up. Here are screenshots. Names of files are visible and show respective encode parameters.

1. TranscodeHW. 8bit BT.709. Two right samples (-> 10bit enforced) look as low contrast.
   

2. TranscodeHW. 10bit BT.2020. Two left samples (-> 8bit enforced) look as low contrast.
   

3. TranscodeHW. 10bit BT.709. Two left samples (-> 8bit enforced) are broken. And two right samples (-> 10bit enforced) are oversaturated.
   

4. VCEEncC 8.27. 8bit BT.709. Two right samples (-> 10bit enforced) are oversaturated.
   

5. VCEEncC 8.27. 10bit BT.2020. Two left samples (-> 8bit enforced) are bleak
   (well, this one actually is more or less expected outcome. Source is HDR and i didn't transfer it's parameters)
   

6. VCEEncC 8.27. 10bit BT.709. Two right samples (-> 10bit enforced) are oversaturated.
   

Expected behavior
Output color primaries should not solely depend on forced bit depth.
Breaking 10-bit BT.709 sample to this extent when reencoding it to 8-bit BT.709 definitely should not be a thing.

Additional context
Additional issue:
When i use TranscodeHW on 10-bit BT.2020 HDR source fragment (cropped via ffmpeg) beginning of outputs is ALWAYS corrupted.
First frames (like 3 or 4, perhaps? At least movement is visible) is fine, then there is corrupted several frame gap up to a exactly 1 second mark.

->AV1 8-bit enforced = several frames are either stuck or skipped. Looks like pause and snap afterwards.
->AV1 10-bit enforced = several frames are either stuck or skipped AND green tinted.
->HEVC 8-bit enforced = several frames are corrupted with distorted image.
-> HEVC 10-bit enforced = several frames are corrupted with distorted image and white-grey tint.
-> AVC (8-bit only) = several frames are either stuck/skipped or rapidly switching between two static frames until threshold is passed.

Here is how it looks. Sadly it is impossible to show stuck frames there.

Frankly speaking this also should not be a thing. (VCEEncC does not have this issue), so this one may be either decoder or reader issue. FFMPEG, perhaps?

[MikhailAMD]

At this point AMF color converter doesn't support YUV-8 (NV12) to YUV-10 (P010) or opposite. We are working on this.

[DimkaTsv]

I see. Would be looking forward to.

Is enforcement of BT.709 and BT.2020 primaries on HEVC 8 and 10 bit encodes independent from source (aka based purely on bit-depth) intended though? Because from what i can see (at least in most cases) oversaturation in case of encoding BT.709 source as BT.2020 can be definitely resolved by this.

Oversaturation happens even if i encode P010 source into P010 output if source had BT.709 color primaries. Meaning no color conversion should happen in-between. In fact it happens with any BT.709 source encoded as 10-bit, frankly speaking.

I did test with older VCEEncC (8.17, but any version up to 8.23 included should work). Because i knew about it's behaviour differences.
Encoding of BT.709 8-bit source with forced 10-bit works fine for HEVC because output has BT.709 primaries set correctly, but it fails for AV1, as AV1 metadata sets primaries to BT.2020 by default (and doesn't show primaries and function in mediadata window for some reason...). As result HEVC-10 video is normal, but AV1-10 is oversaturated.

Screenshots below are from VCEEncC 8.17
On these screenshots you can see that matrix remains BT.709 for every encoded video, while primaries and function are enforced to BT.2020 on AV1 one.
BT.709 8-bit --> 10-bit. HEVC has normal colors because it has BT.709 set, while AV1 defaulted to BT.2020

But exact same thing happens with this sample. Even though source is already 10-bit. Would also argue that HEVC-10 encode was also slightly oversaturated, but it feels at least more or less reasonable as reencode processing artifact, unlike AV1 oversaturation which is completely off charts.
BT.709 10-bit --> 10-bit

[MikhailAMD]

At AMF there is no enforcement or such.
There are three related parameters in VUI header for H264 and HEVC or ColorConfig header in AV1:
color_primaries
transfer_characteristics
matrix_coefficients
They all can be set to any value via AMF H264 encoder parameters:
AMF_VIDEO_ENCODER_OUTPUT_COLOR_PROFILE
AMF_VIDEO_ENCODER_OUTPUT_TRANSFER_CHARACTERISTIC
AMF_VIDEO_ENCODER_OUTPUT_COLOR_PRIMARIES
or similar HEVC or AV1 parameters.

I did a simple experiment: set these parameters in SimpleEncoder sample app and checked results in stream analyzer. All parameters are successfully set to whatever value application requested.
The encoding of P010 input doesn't depend on these values and will be always the same.
In conclusion it is up to the app to set these values matching color conversion parameters that produced P010.

[DimkaTsv]

They all can be set to any value via AMF H264 encoder parameters:
AMF_VIDEO_ENCODER_OUTPUT_COLOR_PROFILE
AMF_VIDEO_ENCODER_OUTPUT_TRANSFER_CHARACTERISTIC
AMF_VIDEO_ENCODER_OUTPUT_COLOR_PRIMARIES
or similar HEVC or AV1 parameters.

I did a simple experiment: set these parameters in SimpleEncoder sample app and checked results in stream analyzer. All parameters are successfully set to whatever value application requested.

Well, i knew they existed. And i even used them before, to transfer HDR. I just thought that they are mostly used in case of HDR or specific conversion case and not expected to be explicitly passed each time. If anything i think it should be expected to see them being copied from source by default.
But if i understood correctly it is not AMF job, but application to have these defaults set to transfer, right?

Well, i presume it is probably hard to have dynamic defaults based on source in API, rather than interface, because AMF does not parse through source and only executes call with set parameters... So BT.709 for 8-bit with BT.2020 for 10-bit are just fallback options instead? In case no option had been specified manually?
But why they are directly tied to bit depth? Well, i guess one reasoning is probably because most 10-bit content is HDR content, while 8-bit is SDR.

Correct me if i was wrong in conclusion.

Anyways, thanks, i guess... I will refer to this for VCEEncC creator to adjust transfer of these characteristics by default.

[MikhailAMD]

These parameters are for encoder only. Yes, if not set, there are default values based on bit-ness: BT.709 for 8-bit with BT.2020 for 10-bit.
For AMF decoder there are similar parameters attached to every output frame - if they exist in the input stream. If not, defaults should be considered: AMF_VIDEO_DECODER_COLOR_TRANSFER_CHARACTERISTIC, PRIMARIES, PROFILE.
Don't confuse them with AMF_VIDEO_DECODER_OUTPUT_TRANSFER_CHARACTERISTIC, PRIMARIES, PROFILE which are needed if internal color converter in decoder if used.

[DimkaTsv]

Thanks for confirming.

Then major part of the issue has been answered.
Secondary part as well, (YUV-8 to YUV-10 and reverse conversion) which i presume should result in fixing that slight color distortion.

Now, third part, i guess... That addendum in original post, with HDR video having it's first second broken on TranscodeHW transcode for some reason. All codecs had been affects, but each in it's own way.
This one surely is not normal or expected behaviour. I noticed it before, tbh... Hmm, let me check.

Well, after check on several sources i have, i can make an assumption that all affected files had been cropped out from larger source with ffmpeg. So i guess there is some wonkyness in interaction with cropped via ffmpeg sources... Even if they had been cropped by IDR frame. Still weird, though.