Transcoding fails at specific point in movie

[pto199]

Not sure whats happening here.  Been using Jellyfin for years with my current setup, transcoding all sorts of stuff just fine.  Recently I was trying to watch Taken 3, and when it gets to 1.35.59 and then the movie shuts off.  Heres what the log says, and the FFMPEG log.  Watching through Firefox or Chrome.  If I watch on my cell phone with Directplay, movie runs fine.  This is a Transcoding error

Jellyfin Log:
024-02-25 15:16:59.170 -08:00] [INF] [138] Jellyfin.Api.Auth.CustomAuthenticationHandler: "CustomAuthentication" was not authenticated. Failure message: "Invalid token."
[2024-02-25 15:16:59.170 -08:00] [INF] [138] Jellyfin.Api.Auth.CustomAuthenticationHandler: "CustomAuthentication" was not authenticated. Failure message: "Invalid token."
[2024-02-25 15:16:59.170 -08:00] [INF] [138] Jellyfin.Api.Auth.CustomAuthenticationHandler: AuthenticationScheme: "CustomAuthentication" was challenged.
[2024-02-25 15:17:01.444 -08:00] [INF] [138] Jellyfin.Api.Helpers.TranscodingJobHelper: Stopping ffmpeg process with q command for "/transcode/f8936e093aec7acf86062564df2d5801.m3u8"
[2024-02-25 15:17:01.766 -08:00] [INF] [138] Jellyfin.Api.Helpers.TranscodingJobHelper: FFmpeg exited with code 0
[2024-02-25 15:17:01.784 -08:00] [INF] [138] Jellyfin.Api.Controllers.DynamicHlsController: Current HLS implementation doesn't support non-keyframe breaks but one is requested, ignoring that request
[2024-02-25 15:17:01.785 -08:00] [INF] [138] Jellyfin.Api.Helpers.TranscodingJobHelper: "/usr/lib/jellyfin-ffmpeg/ffmpeg" "-analyzeduration 200M -ss 01:35:45.000 -init_hw_device cuda=cu:0 -filter_hw_device cu -hwaccel cuda -hwaccel_output_format cuda -threads 1 -autorotate 0 -i file:\"/movies/Taken 3 (2014)/Taken 3 (2014) - [1080p.BluRay.x264.5.1 DTS].mkv\" -ss 01:35:45.000 -i file:\"/movies/Taken 3 (2014)/Taken 3 (2014) - [1080p.BluRay.x264.5.1 DTS].idx\" -autoscale 0 -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map 1:0 -sn -map -0:0 -codec:v:0 h264_nvenc -preset p1 -b:v 10848312 -maxrate 10848312 -bufsize 21696624 -profile:v:0 high -g:v:0 72 -keyint_min:v:0 72 -filter_complex \"[1:0]scale=s=1920x808:flags=fast_bilinear,format=yuva420p,hwupload=derive_device=cuda[sub];[0:0]setparams=color_primaries=bt709:color_trc=bt709:colorspace=bt709,scale_cuda=format=yuv420p[main];[main][sub]overlay_cuda=eof_action=endall:shortest=1:repeatlast=0\" -codec:a:0 libfdk_aac -ac 6 -ab 640000 -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type mpegts -start_number 1915 -hls_segment_filename \"/transcode/f8936e093aec7acf86062564df2d5801%d.ts\" -hls_playlist_type vod -hls_list_size 0 -y \"/transcode/f8936e093aec7acf86062564df2d5801.m3u8\""
[2024-02-25 15:17:02.316 -08:00] [WRN] [141] Jellyfin.Server.Middleware.ResponseTimeMiddleware: Slow HTTP Response from "https://mydomain.com/videos/52b17cda-6029-da70-71db-da8ebb341ee2/hls1/main/1915.ts?DeviceId=TW96aWxsYS81LjAgKFdpbmRvd3MgTlQgMTAuMDsgV2luNjQ7IHg2NCkgQXBwbGVXZWJLaXQvNTM3LjM2IChLSFRNTCwgbGlrZSBHZWNrbykgQ2hyb21lLzEyMS4wLjAuMCBTYWZhcmkvNTM3LjM2fDE3MDg5MDI5OTcwNjg1&MediaSourceId=52b17cda6029da7071dbda8ebb341ee2&VideoCodec=h264,h264&AudioCodec=aac,mp3&AudioStreamIndex=4&SubtitleStreamIndex=0&VideoBitrate=139360000&AudioBitrate=640000&MaxFramerate=23.975986&PlaySessionId=399e26bd22ce412d86797411b420b64b&api_key=f6ba9d4b9fae4b8fad589a62a337c491&SubtitleMethod=Encode&TranscodingMaxAudioChannels=6&RequireAvc=false&Tag=c3acc35d3f449f99308a133b98761575&SegmentContainer=ts&MinSegments=1&BreakOnNonKeyFrames=True&h264-level=41&h264-videobitdepth=8&h264-profile=high&h264-deinterlace=true&TranscodeReasons=VideoCodecNotSupported,%20AudioCodecNotSupported,%20SubtitleCodecNotSupported&runtimeTicks=57450000000&actualSegmentLengthTicks=30000000" to "192.168.1.1" in 0:00:00.8948162 with Status Code 200
[2024-02-25 15:17:05.413 -08:00] [ERR] [156] Jellyfin.Api.Helpers.TranscodingJobHelper: FFmpeg exited with code 1
[2024-02-25 15:17:11.474 -08:00] [INF] [131] Jellyfin.Api.Controllers.DynamicHlsController: Current HLS implementation doesn't support non-keyframe breaks but one is requested, ignoring that request
[2024-02-25 15:17:11.475 -08:00] [INF] [131] Jellyfin.Api.Helpers.TranscodingJobHelper: "/usr/lib/jellyfin-ffmpeg/ffmpeg" "-analyzeduration 200M -ss 01:36:00.000 -init_hw_device cuda=cu:0 -filter_hw_device cu -hwaccel cuda -hwaccel_output_format cuda -threads 1 -autorotate 0 -i file:\"/movies/Taken 3 (2014)/Taken 3 (2014) - [1080p.BluRay.x264.5.1 DTS].mkv\" -ss 01:36:00.000 -i file:\"/movies/Taken 3 (2014)/Taken 3 (2014) - [1080p.BluRay.x264.5.1 DTS].idx\" -autoscale 0 -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map 1:0 -sn -map -0:0 -codec:v:0 h264_nvenc -preset p1 -b:v 10848312 -maxrate 10848312 -bufsize 21696624 -profile:v:0 high -g:v:0 72 -keyint_min:v:0 72 -filter_complex \"[1:0]scale=s=1920x808:flags=fast_bilinear,format=yuva420p,hwupload=derive_device=cuda[sub];[0:0]setparams=color_primaries=bt709:color_trc=bt709:colorspace=bt709,scale_cuda=format=yuv420p[main];[main][sub]overlay_cuda=eof_action=endall:shortest=1:repeatlast=0\" -codec:a:0 libfdk_aac -ac 6 -ab 640000 -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type mpegts -start_number 1920 -hls_segment_filename \"/transcode/f8936e093aec7acf86062564df2d5801%d.ts\" -hls_playlist_type vod -hls_list_size 0 -y \"/transcode/f8936e093aec7acf86062564df2d5801.m3u8\""
[2024-02-25 15:17:14.174 -08:00] [INF] [131] Jellyfin.Api.Auth.CustomAuthenticationHandler: "CustomAuthentication" was not authenticated. Failure message: "Invalid token."
[2024-02-25 15:17:14.174 -08:00] [INF] [131] Jellyfin.Api.Auth.CustomAuthenticationHandler: "CustomAuthentication" was not authenticated. Failure message: "Invalid token."
[2024-02-25 15:17:14.174 -08:00] [INF] [131] Jellyfin.Api.Auth.CustomAuthenticationHandler: AuthenticationScheme: "CustomAuthentication" was challenged.
[2024-02-25 15:17:14.646 -08:00] [ERR] [145] Jellyfin.Api.Helpers.TranscodingJobHelper: FFmpeg exited with code 1
[2024-02-25 15:17:14.694 -08:00] [ERR] [145] Jellyfin.Server.Middleware.ExceptionMiddleware: Error processing request. URL "GET" "/videos/52b17cda-6029-da70-71db-da8ebb341ee2/hls1/main/1920.ts".
MediaBrowser.Common.FfmpegException: FFmpeg exited with code 1
  at Jellyfin.Api.Helpers.TranscodingJobHelper.StartFfMpeg(Stream

FFMPEG log:
      title          : English-Forced
Input #1, vobsub, from 'filemovies/Taken 3 (2014)/Taken 3 (2014) - [1080p.BluRay.x264.5.1 DTS].idx':
  Duration: N/A, bitrate: N/A
  Stream #1:0[0x0](en): Subtitle: dvd_subtitle, 1920x1080 (default)
  Stream #1:1[0x1](en): Subtitle: dvd_subtitle, 1920x1080
Stream mapping:
  Stream #0:0 (h264) -> setparams:default (graph 0)
  Stream #1:0 (dvdsub) -> scale:default (graph 0)
  overlay_cuda:default (graph 0) -> Stream #0:0 (h264_nvenc)
  Stream #0:1 -> #0:1 (dts (dca) -> aac (libfdk_aac))
Press [q] to stop, [?] for help
[vobsub @ 0x55ce7ce948c0] sub2video: using 720x576 canvas
Output #0, hls, to '/transcode/fe0ecfc6eb1e842d0c989de5c907e7ae.m3u8':
  Metadata:
    encoder        : Lavf59.27.100
  Stream #0:0: Video: h264 (High), cuda(bt709, progressive), 1920x808 [SAR 1:1 DAR 240:101], q=2-31, 10848 kb/s, 23.98 fps, 90k tbn
    Metadata:
      encoder        : Lavc59.37.100 h264_nvenc
    Side data:
      cpb: bitrate max/min/avg: 10848312/0/10848312 buffer size: 21696624 vbv_delay: N/A
  Stream #0:1: Audio: aac, 48000 Hz, 5.1, s16, 640 kb/s (default)
    Metadata:
      encoder        : Lavc59.37.100 libfdk_aac
frame=    1 fps=0.0 q=0.0 size=N/A time=00:00:00.00 bitrate=N/A speed=  0x   
frame=  58 fps=0.0 q=17.0 size=N/A time=00:00:12.09 bitrate=N/A speed=19.5x   
frame=  58 fps= 52 q=17.0 size=N/A time=00:00:22.97 bitrate=N/A speed=20.5x   
frame=  58 fps= 36 q=17.0 size=N/A time=00:00:34.83 bitrate=N/A speed=21.5x   
frame=  58 fps= 27 q=17.0 size=N/A time=00:00:44.99 bitrate=N/A speed=21.2x   
frame=  58 fps= 22 q=17.0 size=N/A time=00:00:57.85 bitrate=N/A speed=22.1x   
[AVHWDeviceContext @ 0x55ce7c4c7d40] cu->cuMemAlloc(&data, size) failed -> CUDA_ERROR_OUT_OF_MEMORY: out of memory
Error while filtering: Cannot allocate memory
Failed to inject frame into filter network: Cannot allocate memory
Error while processing the decoded data for stream #0:0
[libfdk_aac @ 0x55ce7cfe1480] 2 frames left in the queue on closing
Conversion failed!

[TheDreadPirate]

What Linux distro? What kind of Nvidia GPU? How many concurrent users do you typically have? Did you update your driver recently? Did you have the NVENC patch before, but didn't reapply after an update?

[pto199]

What Linux distro?  What kind of Nvidia GPU?  How many concurrent users do you typically have?  Did you update your driver recently?  Did you have the NVENC patch before, but didn't reapply after an update?

Unraid.  Latest Version (16.12.8 )
1660 ti
Typically have 4-10 users.  In this case I have just one. 
Yes driver is fully up to date, and yes its patched on array start in Unraid.  This is unrelated to that.

Its only this one particular file.  All other files I have work fine, multi users work fine, everything works fine.  Just this one file has this 1 particular mark.  Judging by the logs its related to a particular frame.  Not sure why jellyfin isn't just skipping it then instead of shutting down the movie and throwing an out of memory error.  If I seek past it a couple minutes ahead, movie keeps working.  Running Jellyfin 10.8.13.

Edit: File works just fine with subtitles disabled. It appears to be some sort of memory issue with this particular subtitle file. This is probably more of a bug report at this point than troubleshooting. Heres the subtitle file if a Developer wants to figure out why Jellyfin can crash on transcoding from a .sub file. Timestamp is at 1:35:59 when the crash occurs "Your daughter?" text. Keep in mind this .sub file contains 2 subtitle tracks. Only one of them causes the crash https://file.io/Jpyp1PMEEbI0

[TheDreadPirate]

Let's check the file itself for errors.

Code:

ffmpeg -v error -i "/movies/Taken 3 (2014)/Taken 3 (2014) - [1080p.BluRay.x264.5.1 DTS].mkv" -f null - 2>error.log