Big but old server, will it software transcode 4K?

[Strangle7068]

I am in the process of "building" a NAS/Media server/docker platform.

I have been hearing good things about Jellyfin, https://selfhosted.show/89



I do not have a GPU other than the onboard Matrox G200eW based on a design from the late 90's, https://en.wikipedia.org/wiki/Matrox_G200

but I do have a metric boat load of decade old CPU cores.

Dual socket E5-2695 v2, Ivy Bridge, 12 cores 24 threads (x2) @ 2.1-3.3 Ghz


By modern standards Individually these cores are anemic but there are a lot of them, Will Jellyfin make use of all those threads and cores to transcode?  do you think it will transcode 4k to dumb boxes like "smart" tv's and Roku's? is it going to pull down 1,000 watts to do it? Auxiliary heat for the winter?

[TheDreadPirate]

Let me put it this way. I have 12 Zen 3 cores (Ryzen 5900X) that turbo higher. If my server ran on this system, I MIGHT be able to manage two 1080p transcodes with the CPU. At a faster preset I could probably do three transcodes. But with serious quality penalties.

My actual Jellyfin server with a 10w Intel Atom class CPU (J4205) can GPU transcode 3-4 streams (1080p -> lower resolution) with shockingly good quality.

GPUs are pretty much not optional for transcoding. Even an Intel iGPU is a huge step up.

[Strangle7068]

Hmmm,

https://www.cpubenchmark.net/compare/211...en-9-5900X

For a bit over twice power consumption and core count I get about 56% of the 5900x performance. at least as measured by CPU mark.

Most of the time this will be transcoding a single stream, so a single 1080 stream should be doable, but probably not 4K?

After the computer budget has recovered a bit (9x 14TB SAS drives are being delivered today) I will look for a GPU.

[TheDreadPirate]

The problem is that video encoding does not scale linearly with core count. There is a point where adding more threads/cores will produce zero benefit. Which is why per core performance is important for encoding. This is even more true with lower resolution video as there are fewer "chunks" to spread around.

I can't say for certain how many transcodes your server will be able to handle. Just trying to manage your expectations. Direct play requires minimal resources, so try to encode your library with widely supported codecs (H264 + AAC) if you are able to. Of if you know that your playback devices support newer codecs.

[Strangle7068]

OK,  I am up and running, and it works.....almost.

I watched a few shows they ran smoothly in CPU transcode to my android TV, I then tried a 1080p H264 movie, H264 seems to take longer to get going than other formats and then about 20 min in the playback siezed for about 20 seconds,  when it came back audio was badly out of sync. I Thought it might be a problem with the file so I sought out another 1080p H264 movie for an oranges to oranges  comparison and got the same stop at about 25 min in but this time it errored out and dropped back to jellyfin browser.

This seems like a long way to make it for it to be a transcoding issue? Or is this a typical transcode issue? I also have marginal wifi in the living room, that could also be the issue.

Obviously I need to fix this to do so i need to figure out where my bottle neck is, wifi vs transcoding.

Any troubleshooting tips to figure out where my problem is?

[TheDreadPirate]

Transcodes are written to disk and are deleted once the playback stops. If the volume that the transcodes are written to fills up, that might cause what you are seeing.

How big is the volume for /var/lib/jellyfin/data/transcodes? Do you have a separate, smaller, partition for /var? Or a small / partition, in general?

[Strangle7068]

I would not think that would be the issue, I am on a 1TB ssd, other than a small efi & swap partitions, the rest is a simple large "/" partition for Debian 12 and packages like jellyfin, I should have plenty of room at least 800GB or so free. How big do these files get?

Data like the source video files are on a separate 72TB ZFS "z2" pool made from 8x 14TB SAS drives.

Sounds like this is not the typical transcode failing, let me see if temporarily stringing an ethernet cord through the house fixes it,

Unfortunayly It's a rental and my access point,  modem, router and server are in the basement at the other end of the house, I can't really tear up the walls to move the acces point closer.

[TheDreadPirate]

If that is the case, it is almost certainly not running out of space for transcodes. In /var/log/jellyfin there should be FFMPEG log files. Check those out to see if they can give us insight into the problem.

[Strangle7068]

To not leave this thread unresolved the problem just has not returned. a little unsettling to not know why the problem was there in the first place but it has been playing great in all formats up to 1080p.

To put specifically put the space question to bed:


user@host:~$ df -h
Filesystem          Size  Used Avail Use% Mounted on
# root file system SSD where Debian12 and jellyfin operate from, these numbers don't add up, I think we are not reporting numerous time shift backups of "/" which are also stored on this SSD.
/dev/sdb1            915G  20G  849G  3% /
#z2 pool where source video files live
pool/dataset          72T  17T  55T  23% /mnt/pool/dataset

having repeatedly run out of room on various NAS incarnations in the past I built this one with lots of room to grow into, I am hoping to get 5 to 10 years out of this one.


I looked through the log files, it only seems to keeps 5, and we had already rolled past the error when I looked, but here is a recent example nothing looks out of place to me but I am new here, Note file paths and tile names editted to generic replacements, and I took out a large repettative chunk in the middle.

/videos/b84c7a64-b70e-d785-ab58-56e3174f7fe5/hls1/main/0.ts

{"Protocol":0,"Id":"b84c7a64b70ed785ab5856e3174f7fe5","Path":"/mnt/pool/dataset/Kids/#LongWindedMovieTitle","EncoderPath":null,"EncoderProtocol":null,"Type":0,"Container":"avi","Size":634614208,"Name":"josephkod-disney","IsRemote":false,"ETag":"a7d5b080bf7d2cf354db79dd6fd34625","RunTimeTicks":44821905408,"ReadAtNativeFramerate":false,"IgnoreDts":false,"IgnoreIndex":false,"GenPtsInput":false,"SupportsTranscoding":true,"SupportsDirectStream":true,"SupportsDirectPlay":true,"IsInfiniteStream":false,"RequiresOpening":false,"OpenToken":null,"RequiresClosing":false,"LiveStreamId":null,"BufferMs":null,"RequiresLooping":false,"SupportsProbing":true,"VideoType":0,"IsoType":null,"Video3DFormat":null,"MediaStreams":[{"Codec":"mpeg4","CodecTag":"XVID","Language":null,"ColorRange":null,"ColorSpace":null,"ColorTransfer":null,"ColorPrimaries":null,"DvVersionMajor":null,"DvVersionMinor":null,"DvProfile":null,"DvLevel":null,"RpuPresentFlag":null,"ElPresentFlag":null,"BlPresentFlag":null,"DvBlSignalCompatibilityId":null,"Comment":null,"TimeBase":"125/2997","CodecTimeBase":null,"Title":null,"VideoRange":"SDR","VideoRangeType":"SDR","VideoDoViTitle":null,"LocalizedUndefined":null,"LocalizedDefault":null,"LocalizedForced":null,"LocalizedExternal":null,"DisplayTitle":"480p MPEG4 SDR","NalLengthSize":null,"IsInterlaced":false,"IsAVC":null,"ChannelLayout":null,"BitRate":999006,"BitDepth":8,"RefFrames":1,"PacketLength":null,"Channels":null,"SampleRate":null,"IsDefault":false,"IsForced":false,"Height":412,"Width":720,"AverageFrameRate":23.976,"RealFrameRate":23.976,"Profile":"Advanced Simple Profile","Type":1,"AspectRatio":"180:103","Index":0,"Score":null,"IsExternal":false,"DeliveryMethod":null,"DeliveryUrl":null,"IsExternalUrl":null,"IsTextSubtitleStream":false,"SupportsExternalStream":false,"Path":null,"PixelFormat":"yuv420p","Level":5,"IsAnamorphic":null},{"Codec":"mp3","CodecTag":null,"Language":null,"ColorRange":null,"ColorSpace":null,"ColorTransfer":null,"ColorPrimaries":null,"DvVersionMajor":null,"DvVersionMinor":null,"DvProfile":null,"DvLevel":null,"RpuPresentFlag":null,"ElPresentFlag":null,"BlPresentFlag":null,"DvBlSignalCompatibilityId":null,"Comment":null,"TimeBase":"1/16000","CodecTimeBase":null,"Title":null,"VideoRange":null,"VideoRangeType":null,"VideoDoViTitle":null,"LocalizedUndefined":null,"LocalizedDefault":null,"LocalizedForced":null,"LocalizedExternal":null,"DisplayTitle":"MP3 - Stereo","NalLengthSize":null,"IsInterlaced":false,"IsAVC":null,"ChannelLayout":"stereo","BitRate":128000,"BitDepth":null,"RefFrames":null,"PacketLength":null,"Channels":2,"SampleRate":48000,"IsDefault":false,"IsForced":false,"Height":null,"Width":null,"AverageFrameRate":null,"RealFrameRate":null,"Profile":null,"Type":0,"AspectRatio":null,"Index":1,"Score":null,"IsExternal":false,"DeliveryMethod":null,"DeliveryUrl":null,"IsExternalUrl":null,"IsTextSubtitleStream":false,"SupportsExternalStream":false,"Path":null,"PixelFormat":null,"Level":0,"IsAnamorphic":null}],"MediaAttachments":[],"Formats":[],"Bitrate":1132685,"Timestamp":null,"RequiredHttpHeaders":{},"TranscodingUrl":null,"TranscodingSubProtocol":null,"TranscodingContainer":null,"AnalyzeDurationMs":null,"DefaultAudioStreamIndex":null,"DefaultSubtitleStreamIndex":null}

/usr/lib/jellyfin-ffmpeg/ffmpeg -analyzeduration 200M -f avi -autorotate 0 -i file:"/mnt/pool/dataset/Kids/#LongWindedMovieTitle" -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 libx264 -preset veryfast -crf 23 -maxrate 4995030 -bufsize 9990060 -x264opts:0 subme=0:me_range=4:rc_lookahead=10:me=dia:no_chroma_me:8x8dct=0:partitions=none -force_key_frames:0 "expr:gte(t,0+n_forced*3)" -sc_threshold:v:0 0 -vf "setparams=color_primaries=bt709:color_trc=bt709:colorspace=bt709,scale=trunc(min(max(iw\,ih*a)\,min(720\,412*a))/2)*2:trunc(min(max(iw/a\,ih)\,min(720/a\,412))/2)*2,format=yuv420p" -codec:a:0 copy -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type mpegts -start_number 0 -hls_segment_filename "/var/lib/jellyfin/transcodes/30b3972f2b1ecd913b2791532614e7c4%d.ts" -hls_playlist_type vod -hls_list_size 0 -y "/var/lib/jellyfin/transcodes/30b3972f2b1ecd913b2791532614e7c4.m3u8"


ffmpeg version 5.1.3-Jellyfin Copyright © 2000-2022 the FFmpeg developers
  built with gcc 12 (Debian 12.2.0-14)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-libs=-lfftw3f --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libsvtav1 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-opencl --enable-vaapi --enable-amf --enable-libmfx --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      57. 28.100 / 57. 28.100
  libavcodec    59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter    8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample  4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
Input #0, avi, from 'filemnt/pool/dataset/Kids/#LongWindedMovieTitle':
  Duration: 01:14:42.19, start: 0.000000, bitrate: 1132 kb/s
  Stream #0:0: Video: mpeg4 (Advanced Simple Profile) (XVID / 0x4449565, yuv420p, 720x412 [SAR 1:1 DAR 180:103], 999 kb/s, 23.98 fps, 23.98 tbr, 23.98 tbn
  Stream #0:1: Audio: mp3 (U[0][0][0] / 0x0055), 48000 Hz, stereo, fltp, 128 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (mpeg4 (native) -> h264 (libx264))
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
[mpeg4 @ 0x563c2b374100] Video uses a non-standard and wasteful way to store B-frames ('packed B-frames'). Consider using the mpeg4_unpack_bframes bitstream filter without encoding but stream copy to fix it.
[libx264 @ 0x563c2b364340] using SAR=1/1
[libx264 @ 0x563c2b364340] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[libx264 @ 0x563c2b364340] profile Main, level 3.0, 4:2:0, 8-bit
[libx264 @ 0x563c2b364340] 264 - core 164 r3095 baee400 - H.264/MPEG-4 AVC codec - Copyleft 2003-2022 - http://www.videolan.org/x264.html - options: cabac=1 ref=1 deblock=1:0:0 analyse=0x1:0 me=dia subme=0 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=4 chroma_me=0 trellis=0 8x8dct=0 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=0 threads=13 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=1 keyint=250 keyint_min=23 scenecut=0 intra_refresh=0 rc_lookahead=10 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 vbv_maxrate=4995 vbv_bufsize=9990 crf_max=0.0 nal_hrd=none filler=0 ip_ratio=1.40 aq=1:1.00
Output #0, hls, to '/var/lib/jellyfin/transcodes/30b3972f2b1ecd913b2791532614e7c4.m3u8':
  Metadata:
    encoder        : Lavf59.27.100
  Stream #0:0: Video: h264, yuv420p(bt709, progressive), 720x412 [SAR 1:1 DAR 180:103], q=2-31, 23.98 fps, 90k tbn
    Metadata:
      encoder        : Lavc59.37.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 4995000/0/0 buffer size: 9990000 vbv_delay: N/A
  Stream #0:1: Audio: mp3 (U[0][0][0] / 0x0055), 48000 Hz, stereo, fltp, 128 kb/s
frame=    1 fps=0.0 q=0.0 size=N/A time=00:00:00.00 bitrate=N/A speed=  0x   
[hls @ 0x563c2b366cc0] Opening '/var/lib/jellyfin/transcodes/30b3972f2b1ecd913b2791532614e7c40.ts' for writing
[hls @ 0x563c2b366cc0] Opening '/var/lib/jellyfin/transcodes/30b3972f2b1ecd913b2791532614e7c41.ts' for writing
[hls @ 0x563c2b366cc0] Opening '/var/lib/jellyfin/transcodes/30b3972f2b1ecd913b2791532614e7c42.ts' for writing
[hls @ 0x563c2b366cc0] Opening '/var/lib/jellyfin/transcodes/30b3972f2b1ecd913b2791532614e7c43.ts' for writing
[hls @ 0x563c2b366cc0] Opening '/var/lib/jellyfin/transcodes/30b3972f2b1ecd913b2791532614e7c44.ts' for writing
[hls @ 0x563c2b366cc0] Opening '/var/lib/jellyfin/transcodes/30b3972f2b1ecd913b2791532614e7c45.ts' for writing
[hls @ 0x563c2b366cc0] Opening '/var/lib/jellyfin/transcodes/30b3972f2b1ecd913b2791532614e7c46.ts' for writing
[hls @ 0x563c2b366cc0] Opening '/var/lib/jellyfin/transcodes/30b3972f2b1ecd913b2791532614e7c47.ts' for writing
frame=  629 fps=0.0 q=28.0 size=N/A time=00:00:26.28 bitrate=N/A speed=  50x   
[hls @ 0x563c2b366cc0] Opening '/var/lib/jellyfin/transcodes/30b3972f2b1ecd913b2791532614e7c48.ts' for writing
[hls @ 0x563c2b366cc0] Opening '/var/lib/jellyfin/transcodes/30b3972f2b1ecd913b2791532614e7c49.ts' for writing
[hls @ 0x563c2b366cc0] Opening '/var/lib/jellyfin/transcodes/30b3972f2b1ecd913b2791532614e7c410.ts' for writing
frame=  882 fps=860 q=28.0 size=N/A time=00:00:36.69 bitrate=N/A speed=35.8x   
[hls @ 0x563c2b366cc0] Opening '/var/lib/jellyfin/transcodes/30b3972f2b1ecd913b2791532614e7c411.ts' for writing
[hls @ 0x563c2b366cc0] Opening '/var/lib/jellyfin/transcodes/30b3972f2b1ecd913b2791532614e7c412.ts' for writing
[hls @ 0x563c2b366cc0] Opening '/var/lib/jellyfin/transcodes/30b3972f2b1ecd913b2791532614e7c413.ts' for writing
[hls @ 0x563c2b366cc0] Opening '/var/lib/jellyfin/transcodes/30b3972f2b1ecd913b2791532614e7c414.ts' for writing
[hls @ 0x563c2b366cc0] Opening '/var/lib/jellyfin/transcodes/30b3972f2b1ecd913b2791532614e7c415.ts' for writing
frame= 1209 fps=792 q=28.0 size=N/A time=00:00:50.47 bitrate=N/A speed=33.1x   

#snip for brevity

frame=107400 fps=685 q=28.0 size=N/A time=01:14:39.48 bitrate=N/A speed=28.6x   
[hls @ 0x563c2b366cc0] Opening '/var/lib/jellyfin/transcodes/30b3972f2b1ecd913b2791532614e7c41492.ts' for writing
[hls @ 0x563c2b366cc0] Opening '/var/lib/jellyfin/transcodes/30b3972f2b1ecd913b2791532614e7c41493.ts' for writing
[hls @ 0x563c2b366cc0] Opening '/var/lib/jellyfin/transcodes/30b3972f2b1ecd913b2791532614e7c41494.ts' for writing
frame=107463 fps=685 q=-1.0 Lsize=N/A time=01:14:41.59 bitrate=N/A speed=28.6x   
video:300105kB audio:70033kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
[libx264 @ 0x563c2b364340] frame I:1495  Avg QP:17.70  size: 20907
[libx264 @ 0x563c2b364340] frame P:33558 Avg QP:21.57  size:  6316
[libx264 @ 0x563c2b364340] frame B:72410 Avg QP:24.21  size:  885
[libx264 @ 0x563c2b364340] consecutive B-frames:  5.9% 10.7%  6.2% 77.2%
[libx264 @ 0x563c2b364340] mb I  I16..4: 35.1%  0.0% 64.9%
[libx264 @ 0x563c2b364340] mb P  I16..4: 18.0%  0.0%  0.0%  P16..4: 34.8%  0.0%  0.0%  0.0%  0.0%    skip:47.2%
[libx264 @ 0x563c2b364340] mb B  I16..4:  1.3%  0.0%  0.0%  B16..8: 13.3%  0.0%  0.0%  direct: 3.8%  skip:81.6%  L0:41.8% L1:46.9% BI:11.3%
[libx264 @ 0x563c2b364340] coded y,uvDC,uvAC intra: 46.0% 58.8% 28.9% inter: 6.2% 8.5% 0.7%
[libx264 @ 0x563c2b364340] i16 v,h,dc,p: 36% 36% 15% 13%
[libx264 @ 0x563c2b364340] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 29% 25% 11%  7%  5%  5%  6%  5%  6%
[libx264 @ 0x563c2b364340] i8c dc,h,v,p: 36% 31% 23% 10%
[libx264 @ 0x563c2b364340] Weighted P-Frames: Y:5.0% UV:3.5%
[libx264 @ 0x563c2b364340] kb/s:548.50

[Strangle7068]

One other question, I did not know that trans-code file were written to disk, that's a lot of writes and I am a bit concerned about ssd write endurance here,

I have 2 ideas not sure if one or both of them would be silly or practical.

I have a scratch drive made from 2 1TB sas drives in R0 that I use as torrent cache, so that only completed files are written to the z2 pool in a nice orderly fassion, this scratch disk has plenty of room. its usually empty. would r0 spinning rust be fast enough for the transcode file?

If spinning rust is not fast enough I also have 1/4TB of EEC DDR3 ram at my disposal, ZFS is the big user here it uses it as cache space but never more than half of it, everything else barley makes the needle move, so I usually have 128GB or so of free ram, I could set up a ram drive for this trans-coding purpose? would 16Gb or 32Gb be sufficient?