RAM Usage

[natzilla]

I've been noticing that RAM usage will continue to climb until it's completely full and then never goes down. I feel like this is causing my instance to freeze up after a few days and not play anything to anyone. This doesn't feel normal for a ubuntu installed vm.

all that I mostly see in the logs is

[2023-06-19 23:29:01.852 +00:00] [WRN] As of "06/19/2023 23:28:47 +00:00", the heartbeat has been running for "00:00:14.1240388" which is longer than "00:00:01". This could be caused by thread pool starvation.

[skribe]

It would be most helpful to share the entire server log so we can take a look at it. There is not presently a known memory leak. C# can use more memory than people expect, and linux's memory management works much differently than Windows in general, so that can surprise people.

In any case, share the log. We can take a look. That error does suggest something is abnormal for your specific configuration. It is not common, to my knowledge.

[natzilla]

It would be most helpful to share the entire server log so we can take a look at it. There is not presently a known memory leak. C# can use more memory than people expect, and linux's memory management works much differently than Windows in general, so that can surprise people.

In any case, share the log. We can take a look. That error does suggest something is abnormal for your specific configuration. It is not common, to my knowledge.

I've attached the full logs to the original post.

[skribe]

Thank you for attaching those. You have a few errors here. I can't really parse them, but let me pull the errors out for the reference of others who might be able to:

Code:

[2023-06-19 04:36:24.562 +00:00] [ERR] Error sending web socket message Sessions
System.Net.WebSockets.WebSocketException (0x80004005): The WebSocket is in an invalid state ('Closed') for this operation. Valid states are: 'Open, CloseReceived'
  at System.Net.WebSockets.WebSocketValidate.ThrowIfInvalidState(WebSocketState currentState, Boolean isDisposed, WebSocketState[] validStates)
  at System.Net.WebSockets.ManagedWebSocket.SendAsync(ReadOnlyMemory`1 buffer, WebSocketMessageType messageType, WebSocketMessageFlags messageFlags, CancellationToken cancellationToken)
--- End of stack trace from previous location ---
  at MediaBrowser.Controller.Net.BasePeriodicWebSocketListener`2.SendData(Tuple`3 tuple)
[2023-06-19 04:36:25.634 +00:00] [INF] Processing playback tracker : "e62acaea3aa87ac2edd3248943aa4edab35560ad70ae21d8-edd3248943aa4edab35560ad70ae21d8-c8721bd264c89b32de695604a4562eaf"
[2023-06-19 04:36:33.092 +00:00] [ERR] Error sending web socket message Sessions
System.Net.WebSockets.WebSocketException (0x80004005): The WebSocket is in an invalid state ('Closed') for this operation. Valid states are: 'Open, CloseReceived'
  at System.Net.WebSockets.WebSocketValidate.ThrowIfInvalidState(WebSocketState currentState, Boolean isDisposed, WebSocketState[] validStates)
  at System.Net.WebSockets.ManagedWebSocket.SendAsync(ReadOnlyMemory`1 buffer, WebSocketMessageType messageType, WebSocketMessageFlags messageFlags, CancellationToken cancellationToken)
--- End of stack trace from previous location ---
  at MediaBrowser.Controller.Net.BasePeriodicWebSocketListener`2.SendData(Tuple`3 tuple)
[2023-06-19 04:36:34.964 +00:00] [ERR] Error sending web socket message Sessions
System.Net.WebSockets.WebSocketException (0x80004005): The WebSocket is in an invalid state ('Closed') for this operation. Valid states are: 'Open, CloseReceived'
  at System.Net.WebSockets.WebSocketValidate.ThrowIfInvalidState(WebSocketState currentState, Boolean isDisposed, WebSocketState[] validStates)
  at System.Net.WebSockets.ManagedWebSocket.SendAsync(ReadOnlyMemory`1 buffer, WebSocketMessageType messageType, WebSocketMessageFlags messageFlags, CancellationToken cancellationToken)
--- End of stack trace from previous location ---
  at MediaBrowser.Controller.Net.BasePeriodicWebSocketListener`2.SendData(Tuple`3 tuple)

Code:

[2023-06-19 04:42:01.604 +00:00] [INF] "/usr/lib/jellyfin-ffmpeg/ffmpeg" "-analyzeduration 50000000 -init_hw_device cuda=cu:0 -filter_hw_device cu -hwaccel cuda -hwaccel_output_format cuda -threads 1 -autorotate 0 -i file:\"/mnt/media/Shows/cartoons/Family Guy/Season 01/Family Guy - S01E01 - Death Has A Shadow.mkv\" -autoscale 0 -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 h264_nvenc -preset p1 -b:v 3425140 -maxrate 3425140 -bufsize 6850280 -g:v:0 72 -keyint_min:v:0 72 -vf \"setparams=color_primaries=bt709:color_trc=bt709:colorspace=bt709,scale_cuda=format=yuv420p\" -codec:a:0 libfdk_aac -ac 2 -ab 256000 -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type mpegts -start_number 0 -hls_segment_filename \"/var/lib/jellyfin/transcodes/47284e60e5067cf1b57949648b7cc481%d.ts\" -hls_playlist_type vod -hls_list_size 0 -y \"/var/lib/jellyfin/transcodes/47284e60e5067cf1b57949648b7cc481.m3u8\""
[2023-06-19 04:42:02.227 +00:00] [ERR] FFmpeg exited with code 1
[2023-06-19 04:42:02.293 +00:00] [ERR] Error processing request. URL "GET" "/videos/3955a055-23f7-c811-59f4-71713ee26a11/hls1/main/0.ts".
MediaBrowser.Common.FfmpegException: FFmpeg exited with code 1
  at Jellyfin.Api.Helpers.TranscodingJobHelper.StartFfMpeg(StreamState state, String outputPath, String commandLineArguments, HttpRequest request, TranscodingJobType transcodingJobType, CancellationTokenSource cancellationTokenSource, String workingDirectory)
  at Jellyfin.Api.Controllers.DynamicHlsController.GetDynamicSegment(StreamingRequestDto streamingRequest, Int32 segmentId)
  at Jellyfin.Api.Controllers.DynamicHlsController.GetHlsVideoSegment(Guid itemId, String playlistId, Int32 segmentId, String container, Int64 runtimeTicks, Int64 actualSegmentLengthTicks, Nullable`1 static, String params, String tag, String deviceProfileId, String playSessionId, String segmentContainer, Nullable`1 segmentLength, Nullable`1 minSegments, String mediaSourceId, String deviceId, String audioCodec, Nullable`1 enableAutoStreamCopy, Nullable`1 allowVideoStreamCopy, Nullable`1 allowAudioStreamCopy, Nullable`1 breakOnNonKeyFrames, Nullable`1 audioSampleRate, Nullable`1 maxAudioBitDepth, Nullable`1 audioBitRate, Nullable`1 audioChannels, Nullable`1 maxAudioChannels, String profile, String level, Nullable`1 framerate, Nullable`1 maxFramerate, Nullable`1 copyTimestamps, Nullable`1 startTimeTicks, Nullable`1 width, Nullable`1 height, Nullable`1 maxWidth, Nullable`1 maxHeight, Nullable`1 videoBitRate, Nullable`1 subtitleStreamIndex, Nullable`1 subtitleMethod, Nullable`1 maxRefFrames, Nullable`1 maxVideoBitDepth, Nullable`1 requireAvc, Nullable`1 deInterlace, Nullable`1 requireNonAnamorphic, Nullable`1 transcodingMaxAudioChannels, Nullable`1 cpuCoreLimit, String liveStreamId, Nullable`1 enableMpegtsM2TsMode, String videoCodec, String subtitleCodec, String transcodeReasons, Nullable`1 audioStreamIndex, Nullable`1 videoStreamIndex, Nullable`1 context, Dictionary`2 streamOptions)
  at lambda_method1086(Closure , Object )
  at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
  at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
  at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
  at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
  at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
  at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
  at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
  at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
  at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
  at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
  at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
  at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
  at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
  at Jellyfin.Server.Middleware.ServerStartupMessageMiddleware.Invoke(HttpContext httpContext, IServerApplicationHost serverApplicationHost, ILocalizationManager localizationManager)
  at Jellyfin.Server.Middleware.WebSocketHandlerMiddleware.Invoke(HttpContext httpContext, IWebSocketManager webSocketManager)
  at Jellyfin.Server.Middleware.IpBasedAccessValidationMiddleware.Invoke(HttpContext httpContext, INetworkManager networkManager)
  at Jellyfin.Server.Middleware.LanFilteringMiddleware.Invoke(HttpContext httpContext, INetworkManager networkManager, IServerConfigurationManager serverConfigurationManager)
  at Microsoft.AspNetCore.Authorization.Policy.AuthorizationMiddlewareResultHandler.HandleAsync(RequestDelegate next, HttpContext context, AuthorizationPolicy policy, PolicyAuthorizationResult authorizeResult)
  at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
  at Jellyfin.Server.Middleware.QueryStringDecodingMiddleware.Invoke(HttpContext httpContext)
  at Swashbuckle.AspNetCore.ReDoc.ReDocMiddleware.Invoke(HttpContext httpContext)
  at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
  at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
  at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
  at Jellyfin.Server.Middleware.RobotsRedirectionMiddleware.Invoke(HttpContext httpContext)
  at Jellyfin.Server.Middleware.LegacyEmbyRouteRewriteMiddleware.Invoke(HttpContext httpContext)
  at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.InvokeCore(HttpContext context)
  at Jellyfin.Server.Middleware.ResponseTimeMiddleware.Invoke(HttpContext context, IServerConfigurationManager serverConfigurationManager)
  at Jellyfin.Server.Middleware.ExceptionMiddleware.Invoke(HttpContext context)

One of these very well may be relevant, but I'll need someone who is better able to trace these to say if they could potentially be leading to memory issues. Hang tight.

[joshuaboniface]

Neither error seems related to memory: the first is just a client disconnecting uncleanly, and the second is a playback failure (wrong HWA perhaps). But I'm not sure either of them would really cause such a massive memory leak. For ref, my instance has been up 2 weeks and is only using ~3.3GB

Code:

● jellyfin.service - Jellyfin Media Server
    Loaded: loaded (/etc/systemd/system/jellyfin.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/jellyfin.service.d
            └─jellyfin.service.conf
    Active: active (running) since Mon 2023-06-05 21:59:44 EDT; 2 weeks 0 days ago
  Main PID: 476 (jellyfin)
      Tasks: 23 (limit: 19171)
    Memory: 3.3G
        CPU: 2d 2h 38min 29.487s
    CGroup: /system.slice/jellyfin.service
            └─476 /usr/bin/jellyfin --webdir=/usr/share/jellyfin/web --restartpath=/usr/lib/jellyfin/restart.sh --ffmpeg=/usr/local/bin/ffmpeg

---

It would also help to have more details on the specifics of your setup: what version, what OS, package format, etc.

[natzilla]

Neither error seems related to memory: the first is just a client disconnecting uncleanly, and the second is a playback failure (wrong HWA perhaps). But I'm not sure either of them would really cause such a massive memory leak. For ref, my instance has been up 2 weeks and is only using ~3.3GB

Code:

● jellyfin.service - Jellyfin Media Server
    Loaded: loaded (/etc/systemd/system/jellyfin.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/jellyfin.service.d
            └─jellyfin.service.conf
    Active: active (running) since Mon 2023-06-05 21:59:44 EDT; 2 weeks 0 days ago
  Main PID: 476 (jellyfin)
      Tasks: 23 (limit: 19171)
    Memory: 3.3G
        CPU: 2d 2h 38min 29.487s
    CGroup: /system.slice/jellyfin.service
            └─476 /usr/bin/jellyfin --webdir=/usr/share/jellyfin/web --restartpath=/usr/lib/jellyfin/restart.sh --ffmpeg=/usr/local/bin/ffmpeg

---

It would also help to have more details on the specifics of your setup: what version, what OS, package format, etc.

Sure,

No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.2 LTS
Release: 22.04
Codename: jammy

jellyfin-ffmpeg5/now 5.1.3-1-jammy amd64 [installed,upgradable to: 5.1.3-2-jammy]
jellyfin-server/unknown,now 10.8.10-1 amd64 [installed,automatic]
jellyfin-web/unknown,now 10.8.10-1 all [installed,automatic]
jellyfin/unknown,now 10.8.10-1 all [installed]

● jellyfin.service - Jellyfin Media Server
    Loaded: loaded (/lib/systemd/system/jellyfin.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/jellyfin.service.d
            └─jellyfin.service.conf
    Active: active (running) since Tue 2023-06-20 00:09:38 UTC; 5h 19min ago
  Main PID: 757 (jellyfin)
      Tasks: 36 (limit: 18652)
    Memory: 2.9G
        CPU: 13min 23.285s
    CGroup: /system.slice/jellyfin.service
            └─757 /usr/bin/jellyfin --webdir=/usr/share/jellyfin/web --restartpath=/usr/lib/jellyfin/restart.sh --ffmpeg=/usr/lib/jellyfin-ffmpeg/ffmpeg

Jun 20 05:28:54 jellyfin jellyfin[757]: [05:28:54] [INF] IntervalTrigger fired for task: Webhook Item Added Notifier
Jun 20 05:28:54 jellyfin jellyfin[757]: [05:28:54] [INF] Queuing task ItemAddedScheduledTask
Jun 20 05:28:54 jellyfin jellyfin[757]: [05:28:54] [INF] Executing Webhook Item Added Notifier
Jun 20 05:28:54 jellyfin jellyfin[757]: [05:28:54] [INF] Webhook Item Added Notifier Completed after 0 minute(s) and 0 seconds
Jun 20 05:28:54 jellyfin jellyfin[757]: [05:28:54] [INF] ExecuteQueuedTasks
Jun 20 05:29:24 jellyfin jellyfin[757]: [05:29:24] [INF] IntervalTrigger fired for task: Webhook Item Added Notifier
Jun 20 05:29:24 jellyfin jellyfin[757]: [05:29:24] [INF] Queuing task ItemAddedScheduledTask
Jun 20 05:29:24 jellyfin jellyfin[757]: [05:29:24] [INF] Executing Webhook Item Added Notifier
Jun 20 05:29:24 jellyfin jellyfin[757]: [05:29:24] [INF] Webhook Item Added Notifier Completed after 0 minute(s) and 0 seconds
Jun 20 05:29:24 jellyfin jellyfin[757]: [05:29:24] [INF] ExecuteQueuedTasks

[Venson]

Neither error seems related to memory: the first is just a client disconnecting uncleanly, and the second is a playback failure (wrong HWA perhaps). But I'm not sure either of them would really cause such a massive memory leak. For ref, my instance has been up 2 weeks and is only using ~3.3GB

Code:

● jellyfin.service - Jellyfin Media Server
    Loaded: loaded (/etc/systemd/system/jellyfin.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/jellyfin.service.d
            └─jellyfin.service.conf
    Active: active (running) since Mon 2023-06-05 21:59:44 EDT; 2 weeks 0 days ago
  Main PID: 476 (jellyfin)
      Tasks: 23 (limit: 19171)
    Memory: 3.3G
        CPU: 2d 2h 38min 29.487s
    CGroup: /system.slice/jellyfin.service
            └─476 /usr/bin/jellyfin --webdir=/usr/share/jellyfin/web --restartpath=/usr/lib/jellyfin/restart.sh --ffmpeg=/usr/local/bin/ffmpeg

---

It would also help to have more details on the specifics of your setup: what version, what OS, package format, etc.

Looking at the logfile he has a lot of errored websocket connections, maybe there is actually a memory leak there?
On the other hand the 
Webhook Item Added Notifier
message comes every few sec which could be an issue?

@natzilla i would recommend disabling all plugins and see if that helps as a troubleshooting step.

[natzilla]

Neither error seems related to memory: the first is just a client disconnecting uncleanly, and the second is a playback failure (wrong HWA perhaps). But I'm not sure either of them would really cause such a massive memory leak. For ref, my instance has been up 2 weeks and is only using ~3.3GB

Code:

● jellyfin.service - Jellyfin Media Server
    Loaded: loaded (/etc/systemd/system/jellyfin.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/jellyfin.service.d
            └─jellyfin.service.conf
    Active: active (running) since Mon 2023-06-05 21:59:44 EDT; 2 weeks 0 days ago
  Main PID: 476 (jellyfin)
      Tasks: 23 (limit: 19171)
    Memory: 3.3G
        CPU: 2d 2h 38min 29.487s
    CGroup: /system.slice/jellyfin.service
            └─476 /usr/bin/jellyfin --webdir=/usr/share/jellyfin/web --restartpath=/usr/lib/jellyfin/restart.sh --ffmpeg=/usr/local/bin/ffmpeg

---

It would also help to have more details on the specifics of your setup: what version, what OS, package format, etc.

Looking at the logfile he has a lot of errored websocket connections, maybe there is actually a memory leak there?
On the other hand the 
Webhook Item Added Notifier
message comes every few sec which could be an issue?

@natzilla i would recommend disabling all plugins and see if that helps as a troubleshooting step.

I will start with disabling the webhook. Its the only active type plugin other than infuse and open subtitles.

[natzilla]

Neither error seems related to memory: the first is just a client disconnecting uncleanly, and the second is a playback failure (wrong HWA perhaps). But I'm not sure either of them would really cause such a massive memory leak. For ref, my instance has been up 2 weeks and is only using ~3.3GB

Code:

● jellyfin.service - Jellyfin Media Server
    Loaded: loaded (/etc/systemd/system/jellyfin.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/jellyfin.service.d
            └─jellyfin.service.conf
    Active: active (running) since Mon 2023-06-05 21:59:44 EDT; 2 weeks 0 days ago
  Main PID: 476 (jellyfin)
      Tasks: 23 (limit: 19171)
    Memory: 3.3G
        CPU: 2d 2h 38min 29.487s
    CGroup: /system.slice/jellyfin.service
            └─476 /usr/bin/jellyfin --webdir=/usr/share/jellyfin/web --restartpath=/usr/lib/jellyfin/restart.sh --ffmpeg=/usr/local/bin/ffmpeg

---

It would also help to have more details on the specifics of your setup: what version, what OS, package format, etc.

Looking at the logfile he has a lot of errored websocket connections, maybe there is actually a memory leak there?
On the other hand the 
Webhook Item Added Notifier
message comes every few sec which could be an issue?

@natzilla i would recommend disabling all plugins and see if that helps as a troubleshooting step.

Making that change it did take longer for the ram to fill up. It does at least appear like it's going down when no activity is present on it, but to still get as high as 16G? that still seems much to me. I have included a new batch of logs on this reply.

[Venson]

Neither error seems related to memory: the first is just a client disconnecting uncleanly, and the second is a playback failure (wrong HWA perhaps). But I'm not sure either of them would really cause such a massive memory leak. For ref, my instance has been up 2 weeks and is only using ~3.3GB

Code:

● jellyfin.service - Jellyfin Media Server
    Loaded: loaded (/etc/systemd/system/jellyfin.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/jellyfin.service.d
            └─jellyfin.service.conf
    Active: active (running) since Mon 2023-06-05 21:59:44 EDT; 2 weeks 0 days ago
  Main PID: 476 (jellyfin)
      Tasks: 23 (limit: 19171)
    Memory: 3.3G
        CPU: 2d 2h 38min 29.487s
    CGroup: /system.slice/jellyfin.service
            └─476 /usr/bin/jellyfin --webdir=/usr/share/jellyfin/web --restartpath=/usr/lib/jellyfin/restart.sh --ffmpeg=/usr/local/bin/ffmpeg

---

It would also help to have more details on the specifics of your setup: what version, what OS, package format, etc.

Looking at the logfile he has a lot of errored websocket connections, maybe there is actually a memory leak there?
On the other hand the 
Webhook Item Added Notifier
message comes every few sec which could be an issue?

@natzilla i would recommend disabling all plugins and see if that helps as a troubleshooting step.

Making that change it did take longer for the ram to fill up. It does at least appear like it's going down when no activity is present on it, but to still get as high as 16G? that still seems much to me. I have included a new batch of logs on this reply.

Are you sure that graph only shows jellyfin memory and not "overall" system memory?