+- Jellyfin Forum (https://forum.jellyfin.org)
+-- Forum: Support (https://forum.jellyfin.org/f-support)
+--- Forum: Troubleshooting (https://forum.jellyfin.org/f-troubleshooting)
+---- Forum: Media Scanning & Identification (https://forum.jellyfin.org/f-media-scanning-identification)
+---- Thread: Duplicates when moving files (/t-duplicates-when-moving-files)
Duplicates when moving files - windowsdan - 2024-06-25
I am using the latest 10.9.7 JellyFin on Linux (debian based) and all works completely fine except when I move a video file from one folder in a library to another. I then get duplicates, as the file moving to the new folder is seen and scanned into the library, but the fact that it no longer exists in the previous folder is ignored by the library scan.
This persists through a full scan of all libraries or just a update new files scan. I have to manually click on the video and delete it to remove it from the library.
Surely there should be some way of Jellyfin recognising files that have disappeared and then removing their entries from the library on a scan?
An Example:
- I have the library '4K Films' and it points to /media/4k and 10 other locations across 10 hard drives /mnt/e, /mnt/f, /mnt/g ... etc
- I use the /media/4k folder to extract or rip videos to as it is on an nmve and super fast, there is then a crontab job over night to move those files to the most recent slower hard disk for storage.
- I need it to be available locally on the nmve in Jellyfin as I can be in a situation of wanting to watch straight away and won't wait for transfer to hard disk
- I don't want to have to delete the duplicates every day once the script has run over night
- I do script in a library scan after the transfer but like I said that doesn't help. It shows the library entry for the file in the new location but never removes the missing old one
RE: Duplicates when moving files - Efficient_Good_5784 - 2024-06-25
(2024-06-25, 11:38 AM)windowsdan Wrote:How slow are your HDDs that warrants this? Any ordinary HDD will perfectly play a high bitrate 4K rips. HDDs usually have no problem hitting 110MB/s read speeds, which is way more than necessary for high bitrate content.
- I need it to be available locally on the nmve in Jellyfin as I can be in a situation of wanting to watch straight away and won't wait for transfer to hard disk
As for the initial storage of the file to your server, were talking about waiting a few minutes more if you just saved directly to an HDD. Once again, if we assume the write speed of a slow HDD to be 110MB/s, that's ~100GB written every 16 minutes (though obviously as HDDs fill up, their write speeds get slower...).
Anyways, are you using something that would duplicate where your files could be found? Any symlinks or system snapshots?
RE: Duplicates when moving files - paulc - 2024-06-25
(2024-06-25, 11:38 AM)windowsdan Wrote: I move a video file from one folder in a library to another. I then get duplicates, as the file moving to the new folder is seen and scanned into the library, but the fact that it no longer exists in the previous folder is ignored by the library scan.
This persists through a full scan of all libraries or just a update new files scan. I have to manually click on the video and delete it to remove it from the library.
Surely there should be some way of Jellyfin recognising files that have disappeared and then removing their entries from the library on a scan?
I could not duplicate this issue on Jellyfin 10.9.7, on Windows 10.
I took a video, moved it to a new folder, ran a scan and the old entry was removed and the new entry appeared.
RE: Duplicates when moving files - windowsdan - 2024-06-25
(2024-06-25, 12:53 PM)Efficient_Good_5784 Wrote:(2024-06-25, 11:38 AM)windowsdan Wrote:How slow are your HDDs that warrants this? Any ordinary HDD will perfectly play a high bitrate 4K rips. HDDs usually have no problem hitting 110MB/s read speeds, which is way more than necessary for high bitrate content.
- I need it to be available locally on the nmve in Jellyfin as I can be in a situation of wanting to watch straight away and won't wait for transfer to hard disk
As for the initial storage of the file to your server, were talking about waiting a few minutes more if you just saved directly to an HDD. Once again, if we assume the write speed of a slow HDD to be 110MB/s, that's ~100GB written every 16 minutes (though obviously as HDDs fill up, their write speeds get slower...).
Anyways, are you using something that would duplicate where your files could be found? Any symlinks or system snapshots?
I don't want to download to a hard drive as it is about 4x slower than my download speed. I'm downloading at 460MB/s and most of the time it is 70 or 80Gb files. Normally I want to watch something straight away and not wait another quarter of an hour to watch from another hard drive.
Not using symlinks or snapshots etc. The host Jellyfin server is running native linux lite 7.
(2024-06-25, 12:57 PM)paulc Wrote:(2024-06-25, 11:38 AM)windowsdan Wrote: I move a video file from one folder in a library to another. I then get duplicates, as the file moving to the new folder is seen and scanned into the library, but the fact that it no longer exists in the previous folder is ignored by the library scan.
This persists through a full scan of all libraries or just a update new files scan. I have to manually click on the video and delete it to remove it from the library.
Surely there should be some way of Jellyfin recognising files that have disappeared and then removing their entries from the library on a scan?
I could not duplicate this issue on Jellyfin 10.9.7, on Windows 10.
I took a video, moved it to a new folder, ran a scan and the old entry was removed and the new entry appeared.
I didn't have this issue on Windows 11 host machine before moving to using a Linux host for JellyFin server
RE: Duplicates when moving files - Efficient_Good_5784 - 2024-06-25
Every time a file is moved, Jellyfin will see it as a new file. My guess is an issue with the database. From the duplicates, can both be played? If not, can you open the metadata from Jellyfin's GUI to see which path it originates from on your server?
Could you cause the issue again so that Jellyfin writes down what happens in the logs. Then could you share the logs here by using pastebin?
Also in case you didn't know, since moved files are treated as new files, you will lose watched & favorite statuses for them. The files will also be lost from any collections or playlists you saved them in since they're at a new path from what they were originally. And finally, currently, their chapters and trickplay files (if you enabled them) will have to be regenerated too as those will be lost.
If you want to keep that type of data intact, you'll have to make sure to add them to collections/playlists, or mark them as watched/favorited after they've been transfered to their final resting place.
RE: Duplicates when moving files - windowsdan - 2024-06-27
I don't have an issue with keeping watched data etc as I am only using the holding drive as a place to download and extract to and it doesn't sit there for more than a day. Only the actual file is playable, case even though the interface says there are two in the two different folders there physically isn't and it will just error if I try to play the one that doesn't exist.
Strangely I have figured out a fix ... sort of ... not a neat one but for some reason it works. If I don't leave the folder I've moved files from empty and I just leave an empty text file in the folder after moving stuff, Jellyfin's scraper then recognises the file is gone and removes it from the interface. Same with if I have two or more files in the holding folder ... when the script moves them over, only 1 of the files stays as a duplicate in the interface metadata. Seems like Jellyfin does not update removed files in a library, if the folder ends up empty?! This must be linux issue only as I wasn't getting this behaviour on Windows 11.
RE: Duplicates when moving files - Efficient_Good_5784 - 2024-06-27
(2024-06-27, 08:26 PM)windowsdan Wrote: I don't have an issue with keeping watched data etc as I am only using the holding drive as a place to download and extract to and it doesn't sit there for more than a day.I'm a bit confused about this statement. Do you watch things saved onto the drive or not through Jellyfin? The part that's confusing me is specifically the part where you state "only using the holding drive as a place to download and extract to".
I tried replicating your behavior on a test server I made manually on Linux.
To explain what I did in order, I:
- Added a new path to an existing library (library A)
- Copied an entire show to the new path, then scanned library A.
- Deleted the copied show from the old path.
- Scanned library A.
I ended with no duplicates. The old copy was deleted, and library A has one copy from the correct new path on the system.
(2024-06-27, 08:26 PM)windowsdan Wrote: If I don't leave the folder I've moved files from empty ... Seems like Jellyfin does not update removed files in a library, if the folder ends up empty?!You need to remove the folder too, not just the contents inside the show or movie folder.
Are you getting any images to load on the duplicate?
Where exactly do you get the error? While opening the show or while trying to play an episode?
RE: Duplicates when moving files - windowsdan - 2024-06-28
To maybe make it a bit clearer. I possibly want to watch something that could be as big as 80gb. I can extract that on my system drive to another folder on my system drive very quickly (m.2 nvme) and I have that folder setup as a location in my JellyFin library. This is so I can go straight to the Shield and watch it immediately. If I wait to copy that size file to a mechanical disk, I'm sitting around for another 20-30 minutes before watching it. However, this is no where I want the files in the long term so I have a script that runs each night to move those files, based on folder, to the mechanical ... into a matching folder.
For example I have on the system drive:
/media/4kdv, /media/4khdr, etc..
On the mechanical:
/mnt/drive/4kdv, /mnt/drive/4khdr, etc..
The 4kdv library would then contain both /media/4kdv and /mnt/drive/4kdv
No folders are ever deleted only the files inside the folders are moved from system drive to hdd. When this happens I get the library metadata for the moved file in the new location and still have the metadata for the old file on the system drive as though I have two copies of the title in the same library but in two different places. The full artwork and metadata shows for the title on the system drive, even though the folder is empty, so obviously if I click play on that version there is an error saying it can't playback.
RE: Duplicates when moving files - TheDreadPirate - 2024-06-28
If you have folders in place, Jellyfin will create empty shows/seasons/movies, etc., even if there aren't any files in them.
IMO, what you should do is create a separate "Work in progress" library for this scenario. What you're describing is, I think, the expected behavior. Even if it is undesired. But what you're doing is atypical for movies. For shows there is a library option to merge shows when they are spread across drives that a movies library lacks.