Native PGS support

[Arcus]

Hello,
I'm currently working on a way to enable native PGS support without transcoding in the web-version of Jellyfin.

Problem:
I really dislike burned-in subtitles. I like my my media library to be as clean and as complete as possible with all audio tracks and all subtitles when digitize my Blue Rays. Blue Rays always have graphical subtitles (PGS - .sub) that cannot be converted into text subtitles. Graphical subtitles can only be displayed with video transcoding. I'm using a NAS without GPU support, so transcoding even a 1080p video is reaching the CPUs limit. Also the video quality drops.

I've built a tool that converts the video with all audio tracks into a MP4 using stream-friendly codecs. Then I extract all PGS subtitles, store them as .sub files next to the video. Then I run an automatic script to OCR the images and create an SRT file. The OCR only works half the time. Some images can't be read at all and are empty and every tenth word has a typo.
Yes, I'm obsessed with having a perfect media collection...

Solution:
Allow Jellyfin to stream raw .sub files to the client and let the client render the graphical subtitles on-top of the video.

Tasks / Status:

• JavaScript PGS renderer - Implemented
  
• Steam external PGS (.sup / .sub files) - Implemented
  
• Steam embedded PGS (eg. from .mkv) - Implemented
  
• Subtitle time offset - Implemented
  
• Background rendering via WebWorker - Implemented
  
• Compression - Pending
  
• Setting to disable/enable PGS streaming - Implemented
  
• Jellyfin Web - Implemented
  
• Jellyfin Vue - Implemented
  
• Jellyfin Android (web player) - Implemented by web player
  
• Jellyfin Android (exo player) - Not supported by Exo
  
• Jeyllfin Chromecast App - Implemented by web player
  
• Jeyllfin Chromecast Quick Cast - Not supported by Chromecast?
  
• Jeyllfin iOS - Missing hardware / Can't test
  
• Jeyllfin AppleTV - Missing hardware/ Can't test
  

Problems and limitations...

• Currently the subtitle must be external. However, it is possible to use ffmpeg to export PGS streams.
  
• PGS files are large ~ 20-40mb per full language per episode. Even more for movies.
  
• Downloading can take some time on slower connections. It may delay the playback or subtitles aren't ready for the first seconds. I implemented an async pgs stream reader. Early subtitles can can be displayed while the rest of the pgs file is still downloading. Big win!
  
• The player must load the whole PGS file into memory on playback. Partial reading isn't possible.
  
• The PGS renderer has to be implemented for every player: Web, Android, iOS, etc.
  
• Subtitle cropping isn't implemented yet. It is also not implemented in FFmpeg and the feature is properly not used anywhere...
  

Repositories:

• libpgs-js: https://github.com/Arcus92/libpgs-js
  
• jellyfin: https://github.com/Arcus92/jellyfin/tree/native-pgs
  
• jellyfin-web: https://github.com/Arcus92/jellyfin-web/tree/native-pgs
  
• jellyfin-vue: https://github.com/Arcus92/jellyfin-vue/tree/native-pgs
  

Is this something anyone is interested in or am I the only one who would benefit from this? Is this even in the spirit of the Jellyfin project? Is it worth to build a clean implementation from the prototype, so it could be merged some day?

Thanks for reading all of this!

[thornbill]

It sounds roughly similar to what we do for ASS/SSA subs, so I don’t see why we wouldn’t want to try it! 💜

[nyanmisaka]

Considering that PGSSUB files are relatively large, we can apply some compression algorithm while transferring.

This is the result given by 7zip fast: 182M vs 7.9M for a two hour subtitle    

BTW I saw a similar project before. https://github.com/MathNuts/SUPtitles

[Arcen]

Have you tried TrOCR or other more recent OCR models? Obviously they're more computationally intense than, say, tesseract, but curious about results.

[Arcus]

Have you tried TrOCR or other more recent OCR models? Obviously they're more computationally intense than, say, tesseract, but curious about results.

No, I haven't tried that yet. But it looks promising.

Using compression is a great idea. I'll have a look at this.


Update: I've build a working prototype:

• libpgs-js: https://github.com/Arcus92/libpgs-js
  
• jellyfin: https://github.com/Arcus92/jellyfin/tree/native-pgs
  
• jellyfin-web: https://github.com/Arcus92/jellyfin-web/tree/native-pgs
  

It's still needs a lot of polish but it already works quite nicely.

[ferferga]

@Arcus are you interested in PRing it in Vue as well?

[Arcus]

@Arcus are you interested in PRing it in Vue as well?

@ferferga Sorry, I missed your message. Yes, I've already looked at the vue project, but first I'll fix the performance concerns for the default client.

[Mats]

I rebased everything atop the latest release and tested it. Can confirm it works well with the LG webOS client (including movies).

Great work!

[Host-in-the-Shell]

This sounds like a major addition to have, particularly for us who use WebOS Jellyfin as our primary client. Is this something that later down the line could be merged into the main repos? I don't know much about how the JF code works and what its limitations are, so that's why I'm asking.