Today, 12:34 AM
(This post was last modified: Today, 12:43 AM by canadabasin. Edited 1 time in total.)
Greetings all:
BLUF EDIT: After I had an aha moment typing out this post, I just restarted my backup and skipped the introskipper folder and the rest of the backup copied as expected with speed. I realized that there are literally over 100K tiny files in that introskipper folder (about a gig's worth) and assuming that is what was jamming up the read/write IO...hat tip AI. I'm going to leave the below discussion in case anyone finds themselves in a similar situation/setup (trying to do a backup of their server config on Windows and is using the introskipper plugin). Still welcome any discussion. And still hoping for some sweet manna.
Original post:
Been a Windows JF user for a bit now (standard install, not docker) and have a very growing home media collection. I've spent a lot of time customizing my images, collections, and various user avatar images across the users and libraries. That led me to looking into how to backup my server config. I found several posts here and elsewhere with guidance to stop the server, then make backup copies of the two JF folders in the Program Data and Program Files directories. Easy enough...I have done that a couple of times in the past without much of an issue.
Today I've been trying to get the above process accomplished and have been having a time getting the Program Data folder to copy across my gigabit network or to a USB3.0 pocket drive. Super slow transfer...like 2-5MB/sec. The Program Data folder in question is 12 GB in size (Program Files is 507MB). Currently the transfer using SyncBack is at 3 hours plus. As part of my troubleshooting, I've stopped and restarted the transfer, used SyncBack, TeraCopy, and Windoze file explorer...all with similar sluggishness. I've rebooted the computers on server and receiving side, I've tried the USB3.0 drive directly plugged into the server machine. Everything seems sluggish.
To test a theory that my network or SSDs and HDDs were the culprit, I tried copying a large 5GB file across the network from the same locations (exact paths)...and the file flew across the network in about 30 seconds or less at about 100MB/sec...so I don't think my network setup is the issue. Before posting here I've been working with AI for several hours today checking my resource monitors, inspecting hard drives, doing a lot of general network transfer troubleshooting to no avail.
When watching the files that are being copied, it seems there is a REAL hangup...been running for a while on this subfolder...it's server\data\introskipper\chromaprints\xxxxxxxxxx. There's a long string of letters and numbers for the file names here at the xxxxx. It just occurred to me while typing this that at my last backup, I had not been using the introskipper plug in. Perhaps this is a problem...? At the moment, SyncBack shows there are still 47K files to go...
General questions for the experts:
1. Is 12 GB way crazy for the Program Data folder (in other words, is my JF installation doing something there it shouldn't to generate that size...is it the introskipper??)
2. Is the fact that there are so many small files scattered throughout the Program Data folder causing the slow read/send issue? (The AI oracles I've been troubleshooting with today keep telling me that might be an issue...they keep suggesting I ZIP the folder first to get one "big" file to send, but that takes as long or longer to access).
3. Is introskipper my problem?? (EDIT: YES)
4. Anyone else run into this sort of issue trying to backup there JF server?
5. Will the "in the works" backup plugin be like manna from heaven when/if it's implemented in 10.11?
Thanks for any insights or discussion about this. CB
BLUF EDIT: After I had an aha moment typing out this post, I just restarted my backup and skipped the introskipper folder and the rest of the backup copied as expected with speed. I realized that there are literally over 100K tiny files in that introskipper folder (about a gig's worth) and assuming that is what was jamming up the read/write IO...hat tip AI. I'm going to leave the below discussion in case anyone finds themselves in a similar situation/setup (trying to do a backup of their server config on Windows and is using the introskipper plugin). Still welcome any discussion. And still hoping for some sweet manna.
Original post:
Been a Windows JF user for a bit now (standard install, not docker) and have a very growing home media collection. I've spent a lot of time customizing my images, collections, and various user avatar images across the users and libraries. That led me to looking into how to backup my server config. I found several posts here and elsewhere with guidance to stop the server, then make backup copies of the two JF folders in the Program Data and Program Files directories. Easy enough...I have done that a couple of times in the past without much of an issue.
Today I've been trying to get the above process accomplished and have been having a time getting the Program Data folder to copy across my gigabit network or to a USB3.0 pocket drive. Super slow transfer...like 2-5MB/sec. The Program Data folder in question is 12 GB in size (Program Files is 507MB). Currently the transfer using SyncBack is at 3 hours plus. As part of my troubleshooting, I've stopped and restarted the transfer, used SyncBack, TeraCopy, and Windoze file explorer...all with similar sluggishness. I've rebooted the computers on server and receiving side, I've tried the USB3.0 drive directly plugged into the server machine. Everything seems sluggish.
To test a theory that my network or SSDs and HDDs were the culprit, I tried copying a large 5GB file across the network from the same locations (exact paths)...and the file flew across the network in about 30 seconds or less at about 100MB/sec...so I don't think my network setup is the issue. Before posting here I've been working with AI for several hours today checking my resource monitors, inspecting hard drives, doing a lot of general network transfer troubleshooting to no avail.
When watching the files that are being copied, it seems there is a REAL hangup...been running for a while on this subfolder...it's server\data\introskipper\chromaprints\xxxxxxxxxx. There's a long string of letters and numbers for the file names here at the xxxxx. It just occurred to me while typing this that at my last backup, I had not been using the introskipper plug in. Perhaps this is a problem...? At the moment, SyncBack shows there are still 47K files to go...
General questions for the experts:
1. Is 12 GB way crazy for the Program Data folder (in other words, is my JF installation doing something there it shouldn't to generate that size...is it the introskipper??)
2. Is the fact that there are so many small files scattered throughout the Program Data folder causing the slow read/send issue? (The AI oracles I've been troubleshooting with today keep telling me that might be an issue...they keep suggesting I ZIP the folder first to get one "big" file to send, but that takes as long or longer to access).
3. Is introskipper my problem?? (EDIT: YES)
4. Anyone else run into this sort of issue trying to backup there JF server?
5. Will the "in the works" backup plugin be like manna from heaven when/if it's implemented in 10.11?
Thanks for any insights or discussion about this. CB