Technical limits with Jellyfin ?

[vincen]

It sounds to me as if your sqlite DB is congested. This could be caused by a few things. Share your full server log and we can take a look. There is not a way to use MySQL at this time, but sqlite should also be fine for any reasonable scale. This issue is most often caused by storage congestion in general, or plugins behaving badly. The logs are the place to start.

Sorry for late answers but as notifications by email are not working I saw them late  Regarding logs can I share them without precautions ? nothing sensible in it ? I have seen there is the IP address of server that I can remove easily, anything else before I post it here ?

Just a quick thing to check, I would follow skribe's advice of posting logs, but consider looking at your scheduled tasks and how long they're taking to complete and when they're scheduled.
I had issues with scheduled tasks causing occasional playback failures and network congestion during my workday and it was because Jellyfin was just absolutely blasting the TMDB API for people images. The scheduled task was set to run on a weird schedule like every 136 hours which made it fall at odd times and it took me forever to track down. I deleted that trigger and moved it to a reasonable, low-traffic timeframe and eliminated my issue. Not saying this is the same, but the symptoms are haunting echoes of the ghosts I chased for months...

Thanks for advice about scheduled tasks and yeah I had quite a few setup like "Run every xxxx hours" so I have rescheduled all of them to run by night on different days of week and with time limit to be sure they are always running during quiet times

I have about 60% of the movie collection you do, however my interface is super snappy with page load times pretty much instant. 
The only exception is when JellyFin is scanning new media, in which case i also have the same issue and the interface is so sluggish it's not worth browsing.  I've set this to manual so i can control at least when this happens and to be fair the scan doesn't take too long (less than a minute), my guess is that SQLite DB is the cause and the eventual move to mySql will fix this.
EDIT - Just to mention my jellyfin docker and all the metadata is on a NVMe, whilst my movies are on HDD

I have fully deactivated the automatic scan of libraries because I had already noticed same problem you point out: when a scan is running the server is nearly down in term of use
Storage should not be a problem as metadata folder is on a dedicated ZFS array and an other one for media files and both have around 2.5g of data transfer rate sustained.