Search performance improvements?

[tomstephens89]

Hi all,

Before I go putting in a feature request, I wondered if there are already any planned improvements to how searching is handled, particularly for large libraries. I understand the index is housed in an SQLite DB which is directly queried as part of a search. I wouldn't say a couple of seconds is particularly 'slow' considering the size of my library, however it would be great to see if said index could be housed in a caching layer such as Redis or something in order to return results within ms. 

I know there's an external search proxy available, but I don't like the idea of bolting on third party completely separate services to my set up.

Thanks!

[TheDreadPirate]

With the EFCore migration, there is now the possibility of having an external DB provider for improved performance. Though that wouldn't necessarily increase performance as much as you'd hope without additional database refactoring, indexing, etc.

AFAIK the only two major database changes on the roadmap are mentioned EFCore migration for the library database and the addition of external DB providers.

There have been discussions here and there about some of the silly ways Jellyfin still does things that haven't been changed since the Emby fork. Some of them database and database optimization related. But "there are bigger fish to fry" is the current sentiment with the current level of manpower that is available to the project. So these kinds of efforts are still on the back burner slowly being worked.

[thornbill]

There are a few enhancements that are planned:

1. Reduce the number of API requests made by web. There is an open PR for this that should land in the upcoming 10.11 release.
   
2. Support for search provider plugins. This one has been discussed a good bit but likely won’t materialize for a few releases. 
   

[tomstephens89]

There are a few enhancements that are planned:

1. Reduce the number of API requests made by web. There is an open PR for this that should land in the upcoming 10.11 release.
   
2. Support for search provider plugins. This one has been discussed a good bit but likely won’t materialize for a few releases. 
   

Thanks for the detailed response. 

As I say its not slow at the moment, but I have my instance on powerful hardware and the install on NVME... But it could definitely do with being more responsive, this is most noticeable if you're using a slow or high latency internet connection which I guess confirms that there's a significant number of API calls happening to serve up the results.

[Duvel]

With the EFCore migration, there is now the possibility of having an external DB provider for improved performance.  Though that wouldn't necessarily increase performance as much as you'd hope without additional database refactoring, indexing, etc.
AFAIK the only two major database changes on the roadmap are mentioned EFCore migration for the library database and the addition of external DB providers.
There have been discussions here and there about some of the silly ways Jellyfin still does things that haven't been changed since the Emby fork.  Some of them database and database optimization related.  But "there are bigger fish to fry" is the current sentiment with the current level of manpower that is available to the project.  So these kinds of efforts are still on the back burner slowly being worked.

Would it be possible that you finally share extensive information and road map about the Postgress/MariaDB ongoing PRs ?

I opened a thread for it here : https://forum.jellyfin.org/t-the-externa...discussion

And my subject has been ignored.

I understand this one is touchy for the core team and as you mentioned it, all this is due to the emby fork legacy and all the the silly things in it.
But to not say a word about it apart "we are working on it", is unfair for the community because this is a long expected feature, and you are obviously working on it. But it has been 6 years now.
So why not just share the truths about it ?

[niels]

Information about the database efforts is available on our blog:

https://jellyfin.org/posts/efcore-refactoring-incoming/

https://jellyfin.org/posts/efcore-refactoring

[Efficient_Good_5784]

Would it be possible that you finally share extensive information and road map about the Postgress/MariaDB ongoing PRs ?

I opened a thread for it here : https://forum.jellyfin.org/t-the-externa...discussion

And my subject has been ignored.

I understand this one is touchy for the core team and as you mentioned it, all this is due to the emby fork legacy and all the the silly things in it.
But to not say a word about it apart "we are working on it", is unfair for the community because this is a long expected feature, and you are obviously working on it. But it has been 6 years now.
So why not just share the truths about it ?

Where did you get information that EFCore has been in the works for 6 years?

Sure there have been discussions in the past to switch to something else, but EFCore was only seriously worked on like half a year back only.
And maybe to your surprise, the main developer behind the EFCore rewrite was not present when Jellyfin forked from Emby.

Your assumption is that we are keeping things hidden from the community.
That's not true.

In your other post you linked, you even linked the PRs themselves that bring this functionality.

Looks like things are on the good path but unless I missed an announcement or forum thread, we dont have much official info about it.

The Jellyfin Github repos have all the code public.
It also has a timestamp of when PRs were submitted.

There's also roadmaps on the Github repos for other things if you're interested.
Like this one for example: https://github.com/orgs/jellyfin/projects/55

Now, if you're criticizing that nobody responded to your original post, it's simply just that nobody had anything to respond with, or it was missed by people that did have things to add.
It's easy to miss posts if you're not watching what is posted here regularly.

[Topomov]

With the EFCore migration, there is now the possibility of having an external DB provider for improved performance.  Though that wouldn't necessarily increase performance as much as you'd hope without additional database refactoring, indexing, etc.
AFAIK the only two major database changes on the roadmap are mentioned EFCore migration for the library database and the addition of external DB providers.
There have been discussions here and there about some of the silly ways Jellyfin still does things that haven't been changed since the Emby fork.  Some of them database and database optimization related.  But "there are bigger fish to fry" is the current sentiment with the current level of manpower that is available to the project.  So these kinds of efforts are still on the back burner slowly being worked.

Would it be possible that you finally share extensive information and road map about the Postgress/MariaDB ongoing PRs ?

I opened a thread for it here : https://forum.jellyfin.org/t-the-externa...discussion

And my subject has been ignored.

I understand this one is touchy for the core team and as you mentioned it, all this is due to the emby fork legacy and all the the silly things in it.
But to not say a word about it apart "we are working on it", is unfair for the community because this is a long expected feature, and you are obviously working on it. But it has been 6 years now.
So why not just share the truths about it ?

You're right, all of this unfairness is very unprofessional. You should ask for a refund. (or help contribute one way or another)

[Venson]

@Duvel the honest truth is that your post likely just fall though and nobody from the team read it.

But as Niels already pointed out, our official info is in the blog posts josh and i wrote plus our open PRs on that topic.
There isn't really much more i can say to this, its ongoing work but the basics are done. The PR is open but as i commented on it it will not happen on the next release but maybe the one after that. If you want to try it out, you can build a pgsql capable version for yourself though my branch but i have to warn you, it might brick your install.