GPU issue?

[ovingiv]

So I'm having an interesting issue that I can't seam to pin down relating to my GPU. It wasn't happening as much on Jellyfin 10.8.* but since the latest 10.9.* release its been happening more often it seams. But I honestly couldn't give a straight answer if that's true or not. After finally fixing my last issue I did move to docker but had too many issues trying to get the GPU to work after following multiple guides and double checking all settings with Portainer to pass through the GPU to the container which did work but I would wound up having the same issues. So I did move back to bare metal installation and reset all permissions again for my directories only to again have this issue now.

I am on Ubuntu Server 22.04.5 LTS
nvidia-smi reports driver version 535.183.01
Jellyfin is running 10.9.11 

I get this error

PHP Code:

[2024-10-26 05:46:24.055 +00:00] [INF] [61] Jellyfin.Api.Helpers.MediaInfoHelper: User policy for "mc". EnablePlaybackRemuxing: True EnableVideoPlaybackTranscoding: True EnableAudioPlaybackTranscoding: True
[2024-10-26 05:46:24.113 +00:00] [INF] [61] MediaBrowser.MediaEncoding.Transcoding.TranscodeManager: Deleting partial stream file(s) "/cache/transcodes/40d6231443a4e2bd96b8348286d5b350.m3u8"
[2024-10-26 05:46:25.677 +00:00] [INF] [24] Jellyfin.Api.Controllers.DynamicHlsController: Current HLS implementation doesn't support non-keyframe breaks but one is requested, ignoring that request
[2024-10-26 05:46:25.678 +00:00] [INF] [24] MediaBrowser.MediaEncoding.Transcoding.TranscodeManager: "/usr/lib/jellyfin-ffmpeg/ffmpeg" "-analyzeduration 200M -probesize 1G -init_hw_device cuda=cu:0 -filter_hw_device cu -hwaccel cuda -hwaccel_output_format cuda -hwaccel_flags +unsafe_output -threads 1 -noautorotate -i file:\"/media/10TB/Media/Movies/Mrs. Doubtfire 1993 COMPLETE BLURAY-LEGi0N/Mrs. Doubtfire 1993 REMUX 1080p Blu-ray AVC DTS-HD MA 5 1-LEGi0N.mkv\" -noautoscale -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 h264_nvenc -preset p1 -b:v 9775214 -maxrate 9775214 -bufsize 19550428 -profile:v:0 high -g:v:0 72 -keyint_min:v:0 72 -vf \"setparams=color_primaries=bt709:color_trc=bt709:colorspace=bt709,scale_cuda=format=yuv420p\" -codec:a:0 libfdk_aac -ac 2 -ab 256000 -af \"volume=2\" -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type fmp4 -hls_fmp4_init_filename \"62f7f41c6512d66711969fc5d5b177fb-1.mp4\" -start_number 0 -hls_segment_filename \"/cache/transcodes/62f7f41c6512d66711969fc5d5b177fb%d.mp4\" -hls_playlist_type vod -hls_list_size 0 -y \"/cache/transcodes/62f7f41c6512d66711969fc5d5b177fb.m3u8\""
[2024-10-26 05:46:25.713 +00:00] [ERR] [24] MediaBrowser.MediaEncoding.Transcoding.TranscodeManager: FFmpeg exited with code 1
[2024-10-26 05:46:25.791 +00:00] [ERR] [24] Jellyfin.Api.Middleware.ExceptionMiddleware: Error processing request. URL "GET" "/videos/f1b35d6e-04b5-01eb-1ad2-998deb7a0892/hls1/main/-1.mp4".
MediaBrowser.Common.FfmpegException: FFmpeg exited with code 1
  at MediaBrowser.MediaEncoding.Transcoding.TranscodeManager.StartFfMpeg(StreamState state, String outputPath, String commandLineArguments, Guid userId, TranscodingJobType transcodingJobType, CancellationTokenSource cancellationTokenSource, String workingDirectory)
  at Jellyfin.Api.Controllers.DynamicHlsController.GetDynamicSegment(StreamingRequestDto streamingRequest, Int32 segmentId)
  at Jellyfin.Api.Controllers.DynamicHlsController.GetHlsVideoSegment(Guid itemId, String playlistId, Int32 segmentId, String container, Int64 runtimeTicks, Int64 actualSegmentLengthTicks, Nullable`1 static, String params, String tag, String deviceProfileId, String playSessionId, String segmentContainer, Nullable`1 segmentLength, Nullable`1 minSegments, String mediaSourceId, String deviceId, String audioCodec, Nullable`1 enableAutoStreamCopy, Nullable`1 allowVideoStreamCopy, Nullable`1 allowAudioStreamCopy, Nullable`1 breakOnNonKeyFrames, Nullable`1 audioSampleRate, Nullable`1 maxAudioBitDepth, Nullable`1 audioBitRate, Nullable`1 audioChannels, Nullable`1 maxAudioChannels, String profile, String level, Nullable`1 framerate, Nullable`1 maxFramerate, Nullable`1 copyTimestamps, Nullable`1 startTimeTicks, Nullable`1 width, Nullable`1 height, Nullable`1 maxWidth, Nullable`1 maxHeight, Nullable`1 videoBitRate, Nullable`1 subtitleStreamIndex, Nullable`1 subtitleMethod, Nullable`1 maxRefFrames, Nullable`1 maxVideoBitDepth, Nullable`1 requireAvc, Nullable`1 deInterlace, Nullable`1 requireNonAnamorphic, Nullable`1 transcodingMaxAudioChannels, Nullable`1 cpuCoreLimit, String liveStreamId, Nullable`1 enableMpegtsM2TsMode, String videoCodec, String subtitleCodec, String transcodeReasons, Nullable`1 audioStreamIndex, Nullable`1 videoStreamIndex, Nullable`1 context, Dictionary`2 streamOptions)
  at lambda_method2092(Closure, Object)
  at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfActionResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
  at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
  at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
  at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
  at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
  at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
  at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
  at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
  at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
  at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
  at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
  at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
  at Jellyfin.Api.Middleware.ServerStartupMessageMiddleware.Invoke(HttpContext httpContext, IServerApplicationHost serverApplicationHost, ILocalizationManager localizationManager)
  at Jellyfin.Api.Middleware.WebSocketHandlerMiddleware.Invoke(HttpContext httpContext, IWebSocketManager webSocketManager)
  at Jellyfin.Api.Middleware.IPBasedAccessValidationMiddleware.Invoke(HttpContext httpContext, INetworkManager networkManager)
  at Jellyfin.Api.Middleware.LanFilteringMiddleware.Invoke(HttpContext httpContext, INetworkManager networkManager, IServerConfigurationManager serverConfigurationManager)
  at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
  at Jellyfin.Api.Middleware.QueryStringDecodingMiddleware.Invoke(HttpContext httpContext)
  at Swashbuckle.AspNetCore.ReDoc.ReDocMiddleware.Invoke(HttpContext httpContext)
  at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
  at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
  at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
  at Jellyfin.Api.Middleware.RobotsRedirectionMiddleware.Invoke(HttpContext httpContext)
  at Jellyfin.Api.Middleware.LegacyEmbyRouteRewriteMiddleware.Invoke(HttpContext httpContext)
  at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.InvokeCore(HttpContext context)
  at Jellyfin.Api.Middleware.ResponseTimeMiddleware.Invoke(HttpContext context, IServerConfigurationManager serverConfigurationManager)
  at Jellyfin.Api.Middleware.ExceptionMiddleware.Invoke(HttpContext context)
[2024-10-26 05:46:25.893 +00:00] [INF] [21] MediaBrowser.MediaEncoding.Transcoding.TranscodeManager: Deleting partial stream file(s) "/cache/transcodes/62f7f41c6512d66711969fc5d5b177fb.m3u8"
[2024-10-26 05:46:27.393 +00:00] [INF] [61] Emby.Server.Implementations.Session.SessionManager: Playback stopped reported by app "Jellyfin Web" "10.9.11" playing "Mrs. Doubtfire". Stopped at "0" ms 


Then when I restart the system as a whole and not just the Jellyfin service, it plays normally.

PHP Code:

[2024-10-26 06:01:24.586 +00:00] [INF] [10] Jellyfin.Api.Helpers.MediaInfoHelper: User policy for "mc". EnablePlaybackRemuxing: True EnableVideoPlaybackTranscoding: True EnableAudioPlaybackTranscoding: True
[2024-10-26 06:01:25.969 +00:00] [INF] [10] Jellyfin.Api.Controllers.DynamicHlsController: Current HLS implementation doesn't support non-keyframe breaks but one is requested, ignoring that request
[2024-10-26 06:01:25.990 +00:00] [INF] [10] MediaBrowser.MediaEncoding.Transcoding.TranscodeManager: "/usr/lib/jellyfin-ffmpeg/ffmpeg" "-analyzeduration 200M -probesize 1G -init_hw_device cuda=cu:0 -filter_hw_device cu -hwaccel cuda -hwaccel_output_format cuda -hwaccel_flags +unsafe_output -threads 1 -noautorotate -i file:\"/media/10TB/Media/Movies/Mrs. Doubtfire 1993 COMPLETE BLURAY-LEGi0N/Mrs. Doubtfire 1993 REMUX 1080p Blu-ray AVC DTS-HD MA 5 1-LEGi0N.mkv\" -noautoscale -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 h264_nvenc -preset p1 -b:v 9775214 -maxrate 9775214 -bufsize 19550428 -profile:v:0 high -g:v:0 72 -keyint_min:v:0 72 -vf \"setparams=color_primaries=bt709:color_trc=bt709:colorspace=bt709,scale_cuda=format=yuv420p\" -codec:a:0 libfdk_aac -ac 2 -ab 256000 -af \"volume=2\" -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type fmp4 -hls_fmp4_init_filename \"6e10d7d7e09c55fccdb474c656aa01d6-1.mp4\" -start_number 0 -hls_segment_filename \"/cache/transcodes/6e10d7d7e09c55fccdb474c656aa01d6%d.mp4\" -hls_playlist_type vod -hls_list_size 0 -y \"/cache/transcodes/6e10d7d7e09c55fccdb474c656aa01d6.m3u8\""
[2024-10-26 06:01:32.538 +00:00] [INF] [15] MediaBrowser.Controller.MediaEncoding.TranscodingJob: Stopping ffmpeg process with q command for "/cache/transcodes/6e10d7d7e09c55fccdb474c656aa01d6.m3u8"
[2024-10-26 06:01:32.703 +00:00] [INF] [15] MediaBrowser.MediaEncoding.Transcoding.TranscodeManager: FFmpeg exited with code 0
[2024-10-26 06:01:32.706 +00:00] [INF] [15] MediaBrowser.MediaEncoding.Transcoding.TranscodeManager: Deleting partial stream file(s) "/cache/transcodes/6e10d7d7e09c55fccdb474c656aa01d6.m3u8" 


Is there something wrong with my GPU? Should I try to upgrade to a different driver? What else could I try to do?

[Fate]

Hmm is there space in your cache folder? Any software that might prevent jellyfin from writing to the cache folder?

[ovingiv]

Hmm is there space in your cache folder?

My system drive is only half full with 496gb available.

[TheDreadPirate]

Can you share your ffmpeg log? It should have a more specific error message.

[ovingiv]

Can you share your ffmpeg log?  It should have a more specific error message.

Here you go 
Before restart: Pastebin
After Restart: Pastebin

[gnattu]

See https://github.com/jellyfin/jellyfin/issues/9177 for more info

nvidia GPU in docker will have quirks and need manual workarounds.

[ovingiv]

nvidia GPU in docker will have quirks and need manual workarounds.

I am not using docker and Jellyfin is on bare metal.

[gnattu]

But I really don't know how to help you because your ffmpeg log indicates that the system lost access to GPU for some reason and I really cannot help you with that.

[TheDreadPirate]

What are you using? The transcode directory tells me this is docker or something based on docker.

Code:

/cache/transcodes/cd9bd29f26ee113a03449c4ad5ddb5fd%d.ts

If this was a bare metal apt install the transcode directory would be /var/cache/jellyfin/transcodes/.

I've read that sometimes the Nvidia container toolkit will not start before Jellyfin/docker starts and docker will not make the GPU available to the container.

[ovingiv]

So I had forgotten that I had the container set to restart always. Because of that after restoring everything from the container to bare metal and restarted my server, it kept jellyfin.service from starting as the container had priority for network ports and *that* started instead. Therefor everything I was having an issue with was with the container still... I have now fixed that, RE-restored everything from /cache and /data and let it rebuild the library.

So I apologize for the confusion on this issue. So far everything is running smoothly after borking the restore twice...

So now I will wait a week and keep an eye on it to see if it has transcoding/remuxing issues again.