Subtitle burn-in won't use extracted embeded subtitles

[homelabResearcher]

Hey guys, recently I ran into issues with some subtitle streams in specific files can be very slow to load and renders burn-in virtually unusable. After checking the logs, I believe that's because these subtitles don't have specified sizes and the ffmpeg have to probe through the entire file to find them. Luckily Jellyfin has a subtitle extraction plugin that could extract embeded subtitles for faster subtitle loading.

So I just ran the subtitle extraction on my entire library and all embeded subtitles have been extracted to local /config/data folder, then I did a full scan to make sure they are correctly mapped with the files. The subtitles do popup instantly if they're not burnt-in, however if I choose to burn them in, ffmpeg would still get embeded subtitles as input, the command would be:

Code:

/usr/lib/jellyfin-ffmpeg/ffmpeg -analyzeduration 200M -init_hw_device vaapi=va:,kernel_driver=i915,driver=iHD -init_hw_device qsv=qs@va -init_hw_device opencl=ocl@va -filter_hw_device qs -hwaccel qsv -hwaccel_output_format qsv -c:v hevc_qsv -autorotate 0 -i file:"/media/Tests/TestLib_Movie/Star Wars Episode II Attack of the Clones (2002).mkv" -autoscale 0 -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:0 -codec:v:0 h264_qsv -low_power 1 -preset 7 -look_ahead 0 -b:v 60073405 -maxrate 60073405 -bufsize 120146810 -g:v:0 72 -keyint_min:v:0 72 -filter_complex "alphasrc=s=2578x1080:r=5:start='0',format=bgra,subtitles=f='/media/Tests/TestLib_Movie/Star Wars Episode II Attack of the Clones (2002).mkv':si=0:alpha=1:sub2video=1:fontsdir='/config/cache/attachments/2289313a163e3e45fc50f0bc01906bd5',hwupload=derive_device=qsv:extra_hw_frames=64[sub];[0:0]setparams=color_primaries=bt2020:color_trc=smpte2084:colorspace=bt2020nc,hwmap=derive_device=opencl,tonemap_opencl=format=nv12:p=bt709:t=bt709:m=bt709:tonemap=bt2390:peak=100:desat=0,hwmap=derive_device=qsv:reverse=1:extra_hw_frames=16,format=qsv[main];[main][sub]overlay_qsv=eof_action=endall:shortest=1:repeatlast=0:w=3840:h=1608" -start_at_zero -codec:a:0 libfdk_aac -ac 2 -ab 384000 -af "volume=2" -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type mpegts -start_number 0 -hls_segment_filename "/config/transcodes/6c7bb8e7935fa54b98a7b2201f015581%d.ts" -hls_playlist_type vod -hls_list_size 0 -y "/config/transcodes/6c7bb8e7935fa54b98a7b2201f015581.m3u8"

and ffmpeg will just probe forever:

Code:

[Parsed_subtitles_2 @ 0x56419c233500] Shaper: FriBidi 1.0.8 (SIMPLE) HarfBuzz-ng 2.7.4 (COMPLEX)
[matroska,webm @ 0x56419c31a440] Could not find codec parameters for stream 7 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
[matroska,webm @ 0x56419c31a440] Could not find codec parameters for stream 8 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
[matroska,webm @ 0x56419c31a440] Could not find codec parameters for stream 9 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
[matroska,webm @ 0x56419c31a440] Could not find codec parameters for stream 10 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
[matroska,webm @ 0x56419c31a440] Could not find codec parameters for stream 11 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
[matroska,webm @ 0x56419c31a440] Could not find codec parameters for stream 12 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
[matroska,webm @ 0x56419c31a440] Could not find codec parameters for stream 13 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
[matroska,webm @ 0x56419c31a440] Could not find codec parameters for stream 14 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options

Later I pulled all generated subtitles and library.db as well, but I don't see any entries in the db that matched the extracted subtitles' filenames, all embeded subtitles still have null as their path, and I can't tell if these subtitles are correctly linked with the media files. 


I think it would be wonderful if the plugin just have an option to save the subtitles in media folders just as the opensubtitle plugin.

[TheDreadPirate]

I see in your signatures that you are using a non-standard kernel with Ubuntu 22.04. Any particular reason for using 6.7 Zabbly over Ubuntu's 6.5 HWE kernel? I'm wondering if the non-standard kernel is the root of the problem.

[homelabResearcher]

I see in your signatures that you are using a non-standard kernel with Ubuntu 22.04.  Any particular reason for using 6.7 Zabbly over Ubuntu's 6.5 HWE kernel?  I'm wondering if the non-standard kernel is the root of the problem.

ADL-N would require kernel 6.6+ to have its iGPU working, so I went for such a kernel that do not require newer libc dependency as mainline kernels do. I can experiment on hwe kernels too, the device would losing video output and DE should not work, but Jellyfin would be fine.

[TheDreadPirate]

So this may be the difference between what is built into the kernel vs what packages are available. 6.6 probably has all the drivers built into the kernel and Ubuntu 22.04 may be missing some of the driver packages to make it work prior to that kernel. ADL-N came out quite a bit later than 22.04 so driver packages are probably not available in the Jammy repos.

[homelabResearcher]

So this may be the difference between what is built into the kernel vs what packages are available.  6.6 probably has all the drivers built into the kernel and Ubuntu 22.04 may be missing some of the driver packages to make it work prior to that kernel.  ADL-N came out quite a bit later than 22.04 so driver packages are probably not available in the Jammy repos.

Thank you for your reply. Just did a reboot on that machine with 6.5 hwe kernel, got same results though (logs attached). I think transcoding is handled by intel-media-va-driver-non-free thus it's kernel invariant. Also same file fails on my Windows backup server. 

Probably the problem is, after subtitle extraction, ffmpeg should be able to load these subtitles externally as other external subtitles (i.e. downloaded by OpenSubtitle plugin) and probing issues would be gracefully addressed on Jellyfin side (C# world, not these driver or ffmpeg stuff).

[TheDreadPirate]

jellyfin-ffmpeg comes with all the user space Intel media drivers except OpenCL, which is required for HDR tone mapping. And the log you attached is indicates the video is HDR. Because 22.04's repos don't contain a new enough OpenCL package for the N100, you will need to install it manually from Intel's git.

https://github.com/intel/compute-runtime/releases

Regarding it failing on Windows, there is a known issue with Intel's OpenCL drivers on Windows being broken. You would need to revert to a November or December 2023 driver release.

[homelabResearcher]

jellyfin-ffmpeg comes with all the user space Intel media drivers except OpenCL, which is required for HDR tone mapping.  And the log you attached is indicates the video is HDR.  Because 22.04's repos don't contain a new enough OpenCL package for the N100, you will need to install it manually from Intel's git.

https://github.com/intel/compute-runtime/releases

Regarding it failing on Windows, there is a known issue with Intel's OpenCL drivers on Windows being broken.  You would need to revert to a November or December 2023 driver release.

Tone mapping is fine for other subtitles or no subtitle burn-in, I chose another subtitle and it worked perfectly fine as log shows. Also my Windows server is using NVENC. I believe reason why play starts slow is that ffmpeg have to probe through the whole file to find requested subtitle, which inherently can be solved by subtitle extracting. 
I will try to clean the extracted subtitles and try agian to see if the plugin ultimately fails to link these subtitles to original media, since for some subtiitles ffmpeg has to probe through the entire file, this process would be long. Hope you can give me some insights into how Jellyfin handles these extracted subtitles lol.

[TheDreadPirate]

Are you using the Subtitle Extract plugin or just on-the-fly subtitle extraction? If you are relying on on-the-fly extraction, install the Subtitle Extract plugin. It will pre-extract your subtitles and store them in the Jellyfin metadata directory.

[homelabResearcher]

Are you using the Subtitle Extract plugin or just on-the-fly subtitle extraction?  If you are relying on on-the-fly extraction, install the Subtitle Extract plugin.  It will pre-extract your subtitles and store them in the Jellyfin metadata directory.

I'm using the subtitle extract plugin, and the subtitle can be found under /config/data/subtitles directory, I cant seem to find a setting to choose where to store these subtitle files. In this thread I'm just wondering why ffmpeg is not using these pre-extracted subtitles when burning-in though. If these subtitles can be found under media folders this issue would be gone.

[TheDreadPirate]

I'll have to do some testing and get back to you on that.