Jellyfin clients capable of "unscaled" progressive output?

[voodoo5_6k]

Hi there!

I've been looking at Jellyfin for quite some time. And finally, I decided to pull the trigger and start evaluating it.

I started many years ago with an HTPC and a RAID controller, a few HDDs and a SCART adapter (yes, that was during the final years of CRT TVs), WinXP with PowerDVD. Several hardware upgrades later, it was still an HTPC (Win10), with JRiver MC + madVR and a FreeNAS and later TrueNAS CORE server. But power draw and constant issues with Win10 pushed me towards replacing the HTPC with a media player (Dune HD Pro Vision 4K Solo) running Kodi, configured for video output at the content's original resolution to feed that into a video processor (Lumagen Radiance Pro) for upscaling & DTM.

This already "hints" at why I'm asking that question... I'm doing this because the video processor's upscaling is way superior (especially for SD content like DVD) to what the usual TV or Player does. Therefore, I want the least possible upscaling on the playback device in order to provide the best possible source material quality to the video processor.

I have already migrated my TrueNAS CORE to SCALE and have Jellyfin running, with all my media imported. By the way, I really appreciate the *.nfo support. I have tons of stuff that can't be imported by other means. It baffles me that Plex still doesn't support this. Jellyfin imported all these special cases flawlessly. Great job!

Now I have two options. I could keep using the Dune media player (Jellyfin client for Android TV I'd assume). Or I could retire the Dune and do everything on the AppleTV 4K that I bought for streaming services (because of the very good rate matching capabilities). I understand there's a new client in beta called Switfin. I'd prefer the AppleTV option, because it is such a small device, and has a simple remote.

Anyhow. My question, finally. I need/want a solution that is capable of video playback at the content's original resolution, in a progressive display mode, within the HDMI specifications of course. For instance, a DVD based *.mkv containing 480i/576i video should be output at 480p/576p and a BD based *.mkv containing 1080p video should be output at 1080p. The video processor is communicating an "all progressive HDMI modes supported" EDID to the connected input devices, and then performs all the necessary actions to output towards the connected display device. Therefore, there aren't any EDID limitations that would prevent playback at e.g. 480p/576p.

Is Swiftfin capable of doing that? Or, if it can't yet, should I rephrase that as some sort of feature request for consideration? I apologize if something like this has already been addressed, but I haven't found anything defintive looking through the forum. I also do realize that this is most likely a very rare "edge case".

Thanks in advance!

[bitmap]

So...can I ask, since you are very particular about this...why haven't you found a filter and process to de-interlace and upscale these to your liking and provide those while preserving the originals? That would be the ideal -- have an "HD" copy that's pre-upscaled and de-interlaced meaning the client doesn't need to do any post-processing, you've already done it. You get to handle everything. You could run it through VapourSynth or just figure out the ffmpeg options to take care of it to your liking.

I only ask because forcing specific bounds on transcoding is not particularly feasible. A lot of this is dictated by client device profiles as well as compatibility. So whatever a client says it will work with, the server sends. If the client decides it's going to upscale after the fact, the server has no say in the matter.

[voodoo5_6k]

Thanks for your reply.

So...can I ask, since you are very particular about this...why haven't you found a filter and process to de-interlace and upscale these to your liking and provide those while preserving the originals? That would be the ideal -- have an "HD" copy that's pre-upscaled and de-interlaced meaning the client doesn't need to do any post-processing, you've already done it. You get to handle everything. You could run it through VapourSynth or just figure out the ffmpeg options to take care of it to your liking.

Well, this is not so easy. First, I've tested several upscalers, even commercial ones, but especially for SD content, the results range from not good to ridiculous. Usually, tiny issues in the source get amplified by not so intelligent AI scalers etc. Then, this needs a lot of time for a single file, even if powered by an RTX 4090 (let's ignore the huge waste of electricity for the moment). Afterwards, the result would have to be checked (i.e. real-time watching) to make sure, everything's so good, the original could be deleted. Keeping both would bloat the storage space requirements significantly (again, even more waste of resources and electricity). Doing this for the thousands of DVDs I have? At least from my point of view, this isn't feasible, and also not ecologically sound. On the other hand, I have a video processor that does better upscaling on the fly, with a <50W or something power budget. I can watch a lot of movies/TV shows that way before even reaching the electricity consumption of a single (not so good) upscale. Also, considering the amount of time saved by just doing it on the fly, while actually watching it for enjoyment, that's fascinatingly efficient from an overall perspective.

I can't say that I would agree with the "definition" of the ideal above (I know that it wasn't meant that way). I'd say, the ideal would be that the studios go and scan their negatives at at least 4K or 8K, and then re-master properly for release. Because even they can't, in almost all cases, provide a good upscale and sell it as 4K UHD (e.g. Pirates of the Caribbean 1, one of the worst 4K UHD discs ever released). And if they can't, what are our chances of coming even close?

Concluding: I have tested and found nothing but bad results, and also it would waste a lot of storage space, time and electricity. Even major studios have failed repeatedly with upscales. And on the other hand, I do have an efficient process of watching these videos at very good quality. It's just that I like the Jellyfin software as a whole and would like to use it, replacing the Dune + Kodi part of my video chain. But not at the expense of image quality. Hence my inquiry.

I only ask because forcing specific bounds on transcoding is not particularly feasible. A lot of this is dictated by client device profiles as well as compatibility. So whatever a client says it will work with, the server sends. If the client decides it's going to upscale after the fact, the server has no say in the matter.

Yeah, that's alright. I'm not asking Jellyfin (server) to do anything about it. I'm asking whether the client (i.e. Swiftfin) can be configured to act like this:

1. The user selects a video file from the server, let's say an NTSC DVD based *.mkv (i.e. 480i)
   
2. The client checks with the OS it is running on what the display device connected to the output port is capable of ("Hey, can we do 480i or 480p?")
   
3. The OS checks the EDID and responds ("Yeah, we can do 480p and we have resolution + rate matching.")
   
4. The client then requests the video from the server in original quality and de-interlaced
   
5. Finally, playback is initiated at 480p and sent to the output port
   

So basically your last sentence ("If the client decides it's going to upscale after the fact...") is what I'm after. Can the client be "taught" to realize that if the original resolution of a given file is within the display device's capabilities that then no further upscaling is required? (Again, as an option, this is not suppossed to become the standard MO.)

All these resolutions/display modes are in the HDMI specifications, so it is not like asking the client to output at some odd custom resolution/display mode. It is more like saying don't bother upscaling, there's another device in the video chain that'll take it from here. Just like saying leave the audio alone, there's an audio processor down the line and that'll take care of it. It's like resolution and rate matching, but consistently enforced for all(!) HDMI specified display modes, not just 1080p vs. 2160p, and in accordance with the actual EDID.

I sincerely hope that nothing of the above is sounding offensive in any way. It definitely isn't meant to be.

[voodoo5_6k]

I have managed to find some time for testing Swiftfin. It currently seems incapable of doing this. Everything that's below AppleTV's current video mode is being scaled up (badly). There are zero settings for scaling.

As this is no longer a general question but more or less focused on Swiftfin, I'll start a new thread in that sub-forum.

Edit: https://forum.jellyfin.org/t-scale-on-ex...han-client

[bitmap]

To answer your last question, as I haven't had time between work and a little passion project I've been doing (over in the off-topic encoding thread), I believe all of this is controlled by device profiles.

The workflow doesn't go exactly as you've described is my understanding, but a dev might have to chime in as I haven't delved into the code enough to fully understand the workflow and explain it ACCURATELY at a 30k foot view. So my understanding, based on previous threads, is this:

1. Client requests media and provides a) user and b) client information to the server.
2. Server decides how to handle the request based on a) user policies and b) device profile.
3. Server builds stream and delivers media playback information to client.

Nowhere in there do you have the option to disable upscaling, enable only de-interlacing, etc...

You can always check your logs to see what Jellyfin itself is doing as well as what ffmpeg is doing. Because it might be the case that you can utilize Jellyfin with an app + external player that will play media at original resolution (but tiny) without any upscaling. I highly doubt you will find anything that will offer to play back SD content on an HD monitor without dynamic/hardware upscaling...it's just not really a thing that displays do since it generally looks like complete trash. Here's an example from my logs of the entire process of media playback request...it's pretty underwhelming...

Code:

jellyfin  | 2023-09-20T19:41:48.588203934Z [13:41:48] [INF] [39] Jellyfin.Api.Helpers.MediaInfoHelper: User policy for office. EnablePlaybackRemuxing: True EnableVideoPlaybackTranscoding: True EnableAudioPlaybackTranscoding: True
jellyfin  | 2023-09-20T19:41:48.588323900Z [13:41:48] [INF] [39] Jellyfin.Api.Helpers.MediaInfoHelper: RemoteClientBitrateLimit: 6500000, RemoteIp: 192.168.99.99, IsInLocalNetwork: True
jellyfin  | 2023-09-20T19:41:48.588491532Z [13:41:48] [INF] [39] Jellyfin.Api.Helpers.MediaInfoHelper: StreamBuilder.BuildVideoItem( Profile=Anonymous Profile, Path=/media/4K/tv/Soft Plastic (1842) [333333]/Season 01/Soft Plastic (1842) - S01E05 - The Bendy Straw [WEBRip-2160p AV1 HDR 10bit OPUS 2.0+5.1][EN][EN]-m1CR0.mkv, AudioStreamIndex=null, SubtitleStreamIndex=null ) => ( PlayMethod=DirectPlay, TranscodeReason=0 ) media:/videos/4c237b35-c6db-5a2c-f1dd-c61f1c539c51/stream.webm?MediaSourceId=1c234b56c7rs5a5cd6aac855fasdfc555c22&Static=true&VideoCodec=av1&AudioCodec=opus&AudioStreamIndex=1&api_key=<token>

Three steps. Here's who I am, here's what I am, here's your media.

[voodoo5_6k]

Thanks for taking the time!

So my understanding, based on previous threads, is this:

1. Client requests media and provides a) user and b) client information to the server.
2. Server decides how to handle the request based on a) user policies and b) device profile.
3. Server builds stream and delivers media playback information to client.

Nowhere in there do you have the option to disable upscaling, enable only de-interlacing, etc...

I'd say that this is pretty much exactly in line with what I described. It's just that my steps 1-4 are your step 1. It's all in there. The client checking with the OS, and based on EDID information, reporting back the capabilities, that's what I'd assume is in the client information. The client information would "just" need to specifiy that the planned output mode is e.g. 480p, if that part is decided by the client based on media information. If it is the server deciding, then it needs to realize that the client has a display mode matching the content's original resolution. And then the server would build the stream accordingly, i.e. no luma scaling, just chroma, and potentially deinterlacing. And there we are, resolution + rate matching.

In other words: When the client is configured for "external scaling", the stream built by the server should target the display mode closest to the content's original resolution, based on the EDID provided by the client.

You can always check your logs to see what Jellyfin itself is doing as well as what ffmpeg is doing. Because it might be the case that you can utilize Jellyfin with an app + external player that will play media at original resolution (but tiny) without any upscaling. I highly doubt you will find anything that will offer to play back SD content on an HD monitor without dynamic/hardware upscaling...it's just not really a thing that displays do since it generally looks like complete trash. Here's an example from my logs of the entire process of media playback request...it's pretty underwhelming...

I think it didn't become clear how this currently works in my video chain. Nobody wants to watch SD material unscaled. I just want it to be sent "upstream" unscaled (well, at least not luma). It will be upscaled, upstream, by a device way more capable than the AppleTV/Swiftfin/Jellyfin combo.

All my media is on a TrueNAS SCALE server, in unchanged, original quality (SD, HD, UHD). The media is shared via SMB. On the Dune HD Pro Vision 4K Solo, Kodi is managing the library. Kodi is using Dune's internal player for playback. This player is configured for rate & resolution matching. It does output 480p for a 480i *.mkv. It also outputs 2160p for a UHD *.mkv. The "display" device connected to the Dune player is a Lumagen Radiance Pro video processor, providing the corresponding EDID (which is configurable). Connected to the video processor output is the real display device. The video processor does all the upscaling and DTM etc. This is working nicely, with great image quality for upscaled SD content (to 4K). So, in the home theater world, where video processors are usually found, playback at content's original resolution is something really sought after (often called "source direct" mode), because only then can a video processor show it's true potential (i.e. getting a video signal which has been deteriorated as little as possible by bad upscaling attempts). The output for video processor upscaled SD is definitely not "compute trash" (pretty much the opposite), but one could say that the current output of the AppleTV/Swiftfin/Jellyfin combo is, euphemistically speaking, subpar, compared to that. And unnecessarily so, because if one could configure it for external scaling, it should reach the same image quality as my Dune/Kodi combo (which has, by itself, unacceptably bad upscaling, especially for SD content).

I'm just trying to "push" for this, because I feel this is a missing feature and would really benefit the Jellyfin ecosystem (although my use case is obviously niche). I always choose open source when possible, and would love to use Jellyfin. This would allow me to have all media accessible via one source device (preferrably the AppleTV, as the Dune is not good for streaming provider platforms) instead of two (Dune + AppleTV). And Jellyfin is almost there, the import worked, all the not scrapable stuff with *.nfo files has been found, I really like what I see so far. It's just the actual playback that falls short, for me (again, niche, I know). No matter how much I'd like to use Jellyfin, no matter that it has checked all the boxes, it "disqualifies" itself when it compromises image quality, which it does currently. But maybe it is possible to implement full resolution + rate matching or "external scaling" (whatever we want to call it).

[voodoo5_6k]

One addition, as I mentioned that this is a sought after feature in the home cinema world. For example, the Strato player also has this type of feature.

"Select Video Output Behavior" --> "Allow display mode changes (use external video scaler)"

Strato player: https://www.kaleidescape.com/

Reference: https://www.avsforum.com/threads/new-lum...t-62845455

And, as mentioned in the very beginning, the Dune player can also be configured for that exact behaviour. I'd assume, that it's the same for Zidoo and Zappiti players.