Scrubbing not working for transcoded HEVC

[Cyboc]

Quick technical summary:
- Jellyfin is running as a docker container (linuxserver, latest)
- Container host runs on J4205 CPU (iGPU should be used for transcoding)
- /dev/dri/renderD128 is passed through to container
- VAAPI is enabled for hardware acceleraction (but issue also present without hardware acceleration)
- Jellyfin Version: 10.8.11
- Architecture: X64

So my issue is that my hevc media cannot be scrubbed anymore, skipping to a specific timestamp gets the playback stuck and after a while the first frame is shown again, not playing back anymore. This has been tested with different clients (Android App, Browser) and it occurs when the client is not able to DirecPlay the media. The transcoding logs show that when skipping to a timestamp (e.g. in the middle of the movie), ffmpeg quits and multiple new transcoding logs are created, one after another.

Transcoding log before skipping:

Code:

...
[hevc @ 0x55ae128a9780] VPS 0 does not exist
[hevc @ 0x55ae128a9780] SPS 0 does not exist.
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> h264 (h264_vaapi))
  Stream #0:1 -> #0:1 (ac3 (native) -> aac (libfdk_aac))
Press [q] to stop, [?] for help
Output #0, hls, to '/config/data/transcodes/000594aacf89311ae8e021fefd64f141.m3u8':
  Metadata:
    encoder        : Lavf60.3.100
  Stream #0:0: Video: h264 (High), vaapi(tv, bt709, progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 9422 kb/s, 23.98 fps, 90k tbn
    Metadata:
      DURATION        : 01:57:36.758000000
      encoder        : Lavc60.3.100 h264_vaapi
  Stream #0:1: Audio: aac, 48000 Hz, stereo, s16, 384 kb/s (default)
    Metadata:
      DURATION        : 01:57:36.864000000
      encoder        : Lavc60.3.100 libfdk_aac
frame=    0 fps=0.0 q=0.0 size=N/A time=00:00:00.00 bitrate=N/A speed=  0x   
frame=  60 fps=0.0 q=-0.0 size=N/A time=00:00:02.68 bitrate=N/A speed=5.11x   
[hls @ 0x55ae12897200] Opening '/config/data/transcodes/000594aacf89311ae8e021fefd64f1410.ts' for writing
frame=  140 fps=136 q=-0.0 size=N/A time=00:00:06.05 bitrate=N/A speed=5.88x   
[hls @ 0x55ae12897200] Opening '/config/data/transcodes/000594aacf89311ae8e021fefd64f1411.ts' for writing
frame=  215 fps=140 q=-0.0 size=N/A time=00:00:09.19 bitrate=N/A speed=  6x   
[hls @ 0x55ae12897200] Opening '/config/data/transcodes/000594aacf89311ae8e021fefd64f1412.ts' for writing
[hls @ 0x55ae12897200] Opening '/config/data/transcodes/000594aacf89311ae8e021fefd64f1413.ts' for writing
frame=  292 fps=144 q=-0.0 size=N/A time=00:00:12.39 bitrate=N/A speed=6.09x   
[hls @ 0x55ae12897200] Opening '/config/data/transcodes/000594aacf89311ae8e021fefd64f1414.ts' for writing
frame=  367 fps=145 q=-0.0 size=N/A time=00:00:15.53 bitrate=N/A speed=6.12x   
[hls @ 0x55ae12897200] Opening '/config/data/transcodes/000594aacf89311ae8e021fefd64f1415.ts' for writing
frame=  448 fps=147 q=-0.0 size=N/A time=00:00:18.88 bitrate=N/A speed=6.21x   
[hls @ 0x55ae12897200] Opening '/config/data/transcodes/000594aacf89311ae8e021fefd64f1416.ts' for writing
frame=  525 fps=148 q=-0.0 size=N/A time=00:00:22.08 bitrate=N/A speed=6.24x   
[hls @ 0x55ae12897200] Opening '/config/data/transcodes/000594aacf89311ae8e021fefd64f1417.ts' for writing
frame=  603 fps=149 q=-0.0 size=N/A time=00:00:25.34 bitrate=N/A speed=6.27x   
[hls @ 0x55ae12897200] Opening '/config/data/transcodes/000594aacf89311ae8e021fefd64f1418.ts' for writing
frame=  685 fps=151 q=-0.0 size=N/A time=00:00:28.77 bitrate=N/A speed=6.34x   
[hls @ 0x55ae12897200] Opening '/config/data/transcodes/000594aacf89311ae8e021fefd64f1419.ts' for writing
frame=  768 fps=152 q=-0.0 size=N/A time=00:00:32.23 bitrate=N/A speed=6.39x   
[hls @ 0x55ae12897200] Opening '/config/data/transcodes/000594aacf89311ae8e021fefd64f14110.ts' for writing


[q] command received. Exiting.

[hls @ 0x55ae12897200] Opening '/config/data/transcodes/000594aacf89311ae8e021fefd64f14111.ts' for writing
frame=  802 fps=153 q=-0.0 Lsize=N/A time=00:00:33.45 bitrate=N/A speed=6.39x   
video:15086kB audio:1578kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown

And one file after skipping:

Code:

...
[hevc @ 0x562184ac0500] VPS 0 does not exist
[hevc @ 0x562184ac0500] SPS 0 does not exist.
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> h264 (h264_vaapi))
  Stream #0:1 -> #0:1 (ac3 (native) -> aac (libfdk_aac))
Press [q] to stop, [?] for help
[hevc @ 0x562184ac0500] PPS id out of range: 0
[hevc @ 0x562184ac0500] Error parsing NAL unit #0.
[hevc @ 0x562184ad3bc0] PPS id out of range: 0
[hevc @ 0x562184ad3bc0] Error parsing NAL unit #0.
[hevc @ 0x562184afb680] PPS id out of range: 0
[hevc @ 0x562184afb680] Error parsing NAL unit #0.
[hevc @ 0x562184fbfa40] PPS id out of range: 0
[hevc @ 0x562184fbfa40] Error parsing NAL unit #0.
Error while decoding stream #0:0: Invalid data found when processing input
[hevc @ 0x562184fd1980] PPS id out of range: 0
[hevc @ 0x562184fd1980] Error parsing NAL unit #0.
[hevc @ 0x562184ac0500] PPS id out of range: 0
[hevc @ 0x562184ac0500] Error parsing NAL unit #0.
Error while decoding stream #0:0: Invalid data found when processing input
[hevc @ 0x562184ad3bc0] PPS id out of range: 0
[hevc @ 0x562184ad3bc0] Error parsing NAL unit #0.
Error while decoding stream #0:0: Invalid data found when processing input
[hevc @ 0x562184afb680] PPS id out of range: 0
[hevc @ 0x562184afb680] Error parsing NAL unit #0.
Error while decoding stream #0:0: Invalid data found when processing input
    Last message repeated 1 times
....

Is anyone running a similar setup successfully or can help me to identify a solution? I really would appreciate any kind of help!

[TheDreadPirate]

Why are you using VAAPI instead of QSV? Does QSV not work? Also, have you tried the official jellyfin image instead of the linuxserver image? The official image includes all the Intel media drivers needed for QSV transcoding and tone mapping.

[Cyboc]

I had no luck using QSV, but the issue is present regardless of the hardware acceleration setting, even with it being disabled. I just tried running the official image, but the issue is exactly the same.

[Cyboc]

I have got an update: I found that in the movies which I transcoded with my AMD RX 7900XT I cannot skip on the timeline. CPU transcoding or just using my NVIDIA 1070 results in an output which I can skip on the timeline.

Although I now how to fix the issue, I would really appreciate any ideas which could explain this behaviour!

[TheDreadPirate]

You learned exactly the same lesson I did. But I had a different problem related to encoding using my RX 6800.

I initially re-encoded stuff with my GTX 1070. Got an RX6800, started encoding stuff with that. Noticed that the chapter image extractor and keyframe extractor tasks were failing. Noticed it was only for the media I added since using my RX6800. I don't recall if I tried to play the videos and verify it could seek.

I went back to using my GTX 1070 to re-encode stuff until I got my i3 12100.

Long story short, friends don't let friends use AMD AMF to encode videos.

[Cyboc]

Thank you for sharing your experience. Somehow I am glad that I am not the only one experiencing this issue. Unfortunately I did transcode my entire library, discarding the original files after checking if the files are playable. Any ideas on what part of my media might be broken, and maybe even a fix for that?

[toytown]

Do you have a small file i can test with? I'm hoping that perhaps just rewriting the mp4/mkv with the same audio/video will be a fix.

[Cyboc]

I have uploaded a transcoded version of big buck bunny: https://file.io/tQQ37yeYpXDg
Thank you for offering your help!

[nyanmisaka]

This is a limitation of FFmpeg. When seeking, some HW decoders (e.g. all hwaccel and QSV decoder) are unable to take advantage of the hevc_mp4toannexb bitstream filter to handle such unusual bitstream.

hevc_mp4toannexb means "Convert an HEVC/H.265 bitstream from length prefixed mode to start code prefixed mode"

As a workaround, you can use this command to pre-remux you video to fix the issue. This is not transcoding, so it usually completes quickly.

Code:

ffmpeg -y -strict -2 -i HEVC_INPUT.mp4 -map 0:v? -map 0:a? -map 0:s? -dn -map_chapters -1 -c:v copy -bsf:v hevc_mp4toannexb -c:a copy -c:s mov_text HEVC_OUTPUT.mp4

I don't want to integrate it in Jellyfin because it will be very slow for long videos, basically depending on the speed of your hard drive. This results in long lead times before transcoding.

[Cyboc]

Thank you for the explanation, things start to get clearer. Unfortunately I still don't quite understand why this is an issue for video-files encoded with an AMD GPU, and not for video files encoded with a NVIDIA GPU. Did I understood you correctly that the issue is on the decoding side, being the client playing back the hevc video?