SOLVED: H265 media not transcoded by certain devices

[quietwalker]

Hi people,
I'm new in town

I use jellyfin from multiple months and recently I decided to transcode with TDARR all my media library from H264 to H265 to save storage space.
This media conversion is still in progress, but I already see some issues:
- my smartphone (Android 13) works fine using the Official Jellyfin app
- both Firefox and Chrome are not able to play H265/HVEV (mkv) files: they always reports "the client is not compatible with the media file and the server is not sending a compatible media format" (I'm translating it, so it will not looks 100% in that way in english probably )
- the above message is also reported by a Xiaomi Mi Box 3 device and Firestick TV

In this last case I also set the client option to use LibVLC as video player and I see an interesting behavior:
- the Mix Box 3 is still reporting problems with the format
- the Amazon Firestick works correctly

Now, checking at the logs I see ffmpeg errors when the client reports issues reproducing a video, so I guess the issue is caused by something in the transcoding.
This seems also to be true because I also see in the logs:

Code:

[09:34:01] [INF] [28] Jellyfin.Api.Helpers.MediaInfoHelper: StreamBuilder.BuildVideoItem( Profile=Anonymous Profile, Path=/JF_SERIES/True Lies/Season 1/True Lies - S01 E11 - Unfamiliar Partnerships HDTV-1080p.mkv, AudioStreamIndex=1, SubtitleStreamIndex=4 ) => ( PlayMethod=Transcode, TranscodeReason=VideoCodecNotSupported, AudioCodecNotSupported ) media:/videos/90f39a77-6f1c-6624-d57f-af18c2eb9d21/master.m3u8?MediaSourceId=90f39a776f1c6624d57faf18c2eb9d21&VideoCodec=h264&AudioCodec=aac,mp3&AudioStreamIndex=1&VideoBitrate=139616000&AudioBitrate=384000&MaxFramerate=23.976025&api_key=<token>&TranscodingMaxAudioChannels=2&RequireAvc=false&Tag=6af93a3e3f86f49772cee0bd368516ce&SegmentContainer=ts&MinSegments=1&BreakOnNonKeyFrames=True&hevc-level=120&hevc-videobitdepth=8&hevc-profile=main&TranscodeReasons=VideoCodecNotSupported,%20AudioCodecNotSupported

Until 3 days ago I was using the intel quicksync to transcode videos in jellyfin container, but now I put the NVIDIA Quadro P600 into my server to assign it to TDARR; this made me think that maybe it is not possible to use the NVIDIA for TDARR and the intel GPU only for jellyfin, thinking that this was the cause of the problem. For this reason I decided to configure Jellyfin to use NVIDIA for video decoding/encoding but the result is exactly the same (I'll attach the logs).

Could you help me understand how I can solve my problem? If the format is not supported by a client, I expect Jellyfin to transcode correctly, rather than returning a playback error to the client: am I right? Why this is not happening?

[TheDreadPirate]

Can you share your ffmpeg logs with us? ffmpeg is failing.

And what model CPU do you have?

[quietwalker]

sure, here are the logs: https://privatebin.net/?a1da67b2eb534638...VViHZ2fprL

[TheDreadPirate]

What CPU do you have?

If you were using your Intel iGPU before, I am pretty sure you need to change which GPU Jellyfin uses. Stop jellyfin, edit /etc/jellyfin/encoding.xml. Change "VaapiDevice" to whatever render device in /dev/dri corresponds to your Nvidia GPU. Jellyfin is probably still configured to use your Intel iGPU, but you have the hardware acceleration settings set to NVENC, probably.

You should be able to use QSV for Jellyfin and NVENC for tdarr. Or the other way around.

[quietwalker]

oh sorry, I forgot to say that my CPU is an intel core i3-13100.

I reverted the changes in order to use again QSV, but even if the devices in /dev/dri/ are viewable in the docker container, it looks like the system doesn't allow me to use it:

Code:

ffmpeg version 5.1.3-Jellyfin Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 10 (Debian 10.2.1-6)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-libs=-lfftw3f --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libsvtav1 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-opencl --enable-vaapi --enable-amf --enable-libmfx --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      57. 28.100 / 57. 28.100
  libavcodec    59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter    8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample  4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
[AVHWDeviceContext @ 0x55f3ab7e7900] No VA display found for any default device.
Device creation failed: -22.
Failed to set value 'vaapi=va:,driver=iHD,kernel_driver=i915' for option 'init_hw_device': Invalid argument
Error parsing global options: Invalid argument

so it make me think like I should use the NVIDIA card if I will decide to keep it attached to my server after the transcode of my library. Does my assumption looks correct?
Anyway I'm trying again using the NVIDIA grafic card, replacing the grafic card in my /etc/jellyfin/encoding.xml.

Actually, in the container, it looks like this:

Code:

<?xml version="1.0" encoding="utf-8"?>
<EncodingOptions xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <EncodingThreadCount>-1</EncodingThreadCount>
  <TranscodingTempPath>/config/transcodes</TranscodingTempPath>
  <FallbackFontPath />
  <EnableFallbackFont>false</EnableFallbackFont>
  <DownMixAudioBoost>2</DownMixAudioBoost>
  <MaxMuxingQueueSize>2048</MaxMuxingQueueSize>
  <EnableThrottling>false</EnableThrottling>
  <ThrottleDelaySeconds>180</ThrottleDelaySeconds>
  <HardwareAccelerationType>qsv</HardwareAccelerationType>
  <EncoderAppPathDisplay>/usr/lib/jellyfin-ffmpeg/ffmpeg</EncoderAppPathDisplay>
  <VaapiDevice>/dev/dri/renderD128</VaapiDevice>
  <EnableTonemapping>false</EnableTonemapping>
  <EnableVppTonemapping>false</EnableVppTonemapping>
  <TonemappingAlgorithm>bt2390</TonemappingAlgorithm>
  <TonemappingMode>auto</TonemappingMode>
  <TonemappingRange>auto</TonemappingRange>
  <TonemappingDesat>0</TonemappingDesat>
  <TonemappingPeak>100</TonemappingPeak>
  <TonemappingParam>0</TonemappingParam>
  <VppTonemappingBrightness>16</VppTonemappingBrightness>
  <VppTonemappingContrast>1</VppTonemappingContrast>
  <H264Crf>23</H264Crf>
  <H265Crf>28</H265Crf>
  <EncoderPreset />
  <DeinterlaceDoubleRate>false</DeinterlaceDoubleRate>
  <DeinterlaceMethod>yadif</DeinterlaceMethod>
  <EnableDecodingColorDepth10Hevc>true</EnableDecodingColorDepth10Hevc>
  <EnableDecodingColorDepth10Vp9>true</EnableDecodingColorDepth10Vp9>
  <EnableEnhancedNvdecDecoder>true</EnableEnhancedNvdecDecoder>
  <PreferSystemNativeHwDecoder>true</PreferSystemNativeHwDecoder>
  <EnableIntelLowPowerH264HwEncoder>true</EnableIntelLowPowerH264HwEncoder>
  <EnableIntelLowPowerHevcHwEncoder>true</EnableIntelLowPowerHevcHwEncoder>
  <EnableHardwareEncoding>true</EnableHardwareEncoding>
  <AllowHevcEncoding>true</AllowHevcEncoding>
  <EnableSubtitleExtraction>true</EnableSubtitleExtraction>
  <HardwareDecodingCodecs>
    <string>h264</string>
    <string>hevc</string>
    <string>mpeg2video</string>
    <string>mpeg4</string>
    <string>vc1</string>
    <string>vp8</string>
    <string>vp9</string>
    <string>av1</string>
  </HardwareDecodingCodecs>
  <AllowOnDemandMetadataBasedKeyframeExtractionForExtensions>
    <string>mkv</string>
  </AllowOnDemandMetadataBasedKeyframeExtractionForExtensions>
</EncodingOptions>

[quietwalker]

Urrà! Problem solved.

The problem was fixed putting back the --runtime=nvidia to the container parameter and then I edited the encoding.xml config file in order to replace the line:

Code:

<VaapiDevice>/dev/dri/renderD128</VaapiDevice>

with:

Code:

<VaapiDevice>/dev/nvidia0</VaapiDevice>

and the transcode works correctly on devices with doesn't support DirectPlay or DirectStream.
Thanks you so much for your help @TheDreadPirate!