Systems for Large File Management Can Save You Hours and Maybe Even Prevent Painful Data Loss
Read Time: 5 Minutes
Welcome to Your Tech Suite's weekly newsletter where we provide news, tips, and guidance on improving the technology posture in your startup or SMB. A holistic technical C-Suite... for the rest of us.
This Week's TLDR
- Adopt a hierarchical file organization and automate backups to manage large files efficiently, preventing data loss and saving time.
- Leverage cloud storage wisely and stay informed on the latest storage management strategies to keep large file handling cost-effective and streamlined.
- Bonus - Photography storage system
- Moon Landing, AV fines, Data Quality Maturity, More Apple Fines
Principles for Large File Management Systems
The evolution of cloud-based storage has significantly changed the game for document storage. Your data has a new "forever home". This is a tremendous boon for most of us.
However, not all data types benefit equally from cloud storage. Large files, such as those produced in photography, videography, graphic design, and others are odd men out.
These file misfits don't fit neatly into the cloud storage solutions like Dropbox and Microsoft OneDrive. While the systems can handle the size, the performance for editing is a nightmare.
And the problem is only getting worse.
Creators, in particular, face a double-edged sword with technological advancements that are pushing the limits of their high-powered systems. This brings growing file sizes and a cloud connectivity that isn't keeping pace.
The balancing act between creativity and file management is something I experience firsthand as my photography evolves.
For example, switching from film to digital in 2005 meant a goodbye to the physical darkroom and very manageable digital jpeg at two-thirds of a megabyte in size.
Moving to RAW files provided additional creative flexibility but required a larger 6 megabyte file. A ninefold increase.
When I upgraded to a full-frame sensor I incurred another step increase in file size. Each RAW image was roughly 36 megabytes. The current full-frame bumps that up to 55 megabytes or so.
And if I choose to process an image; welcome to the world of 250 megabytes to 1 gigabyte sizes.
Cloud storage would never work for these large active file types.
So if you're grappling with managing large files, here are some principles to help move towards a smoother process.
Understanding Your Storage Needs
Recognizing a storage problem is always the first step towards solving it. Thankfully a quick calculation can give you general direction of when those storage woes will show their ugly head.
First, grab a single representative of the large files you generate. Is it video, photography, something else?
Now multiply that file size by the number you generate at a single given "event". Maybe its a wedding, a drone outing, or a day of shooting.
Finally, the critical piece, multiply that by the number of events you do in a year.
Let's walk through with an example:
- If you shoot RAW photography at 50 mb an image, and
- You shoot 200 pictures in a given event, then
- Shooting 12 events a year (once a month) will lead to 120 gigabytes per year, but
- Shooting an event a week would lead to 520 gigabytes per year.
Storage can balloon quickly. But now you know.
Know Your Data's Home
Lots of file storage options out there, each with their pros and cons. If your a creative though and you will need access to your files, here's a quick system for understanding cost and performance.
The closer your data is stored to your computer the faster it will be and the more expensive it will be:
- Local Drive (Your Computer): Super fast and required for efficient processes. Also super expensive.
- External Drive (Connected via "The Dongle"): Relatively fast performance wise, less expensive than your local drive.
- Network Drive (Connected via "The Big Phone Cord"): Slower than local or external drive but cheaper at scale.
- Cloud Storage (Dropbox, OneDrive, etc): Slower still but effective for mass storage at an inexpensive cost.
Pro tip... these are not mutually exclusive and a good system will involve many or even all depending on your needs.
Hierarchical File Organization
This principle can't be stressed enough. Before you start amassing large quanities of these large files. Think about how they are going to be stored and catalogued.
A well-organized file system, using clear and consistent naming conventions, allows for easier navigation and retrieval. It tends to also reduce duplicated storage.
Pro tip... if you're going to use dates in your naming schema, the US format (dd/mm/yyyy) does not sort correctly. Use (yyyy/mm/dd) as it increments correctly in the file explorer's sort algorithm.
Automating Backup and Archiving
There's a IT parable of sorts.
No one in IT was ever fired for not backing something up but plenty have been fired for not being able to restore something.
Backup should not be an afterthought. Your pouring your creative soul into whatever you are doing. Imagine the pain of losing something you created.
Automate your backups to secondary and tertiary locations. Use the rule of 3, 3 copies minimum for a created work.
Older work and files can be archived out to cold storage fairly inexpensively so newer active work can be stored as close to the computer as possible.
Leverage Cloud Storage Wisely
You thought your streaming service spend was creeping up, just wait till you stop paying attention to your cloud storage. Those costs get big quick if not managed.
Cloud is excellent for backup and archiving but make sure you are compressing and de-duplicating your data before you send it to the cloud.
Regularly review your storage costs and usage.
Stay Educated
Your technology continues to change. So does storage. Stay informed on different services and tools that are in the wild. Even strategies change and can improve as time marches on.
The truth is that navigating the complexities of managing large files isn't the "fun" work creatives were looking to go down the rabbit hole on.
I get it.
For better or worse this is an area that needs to be supported if your creative work is going to grow. Acknowledge your storage needs, do the work up front so your storage strategy is clear and implemented before it rages out of control.
Adopting the principles above is going to save your from frustration in the future and hopefully heartache as well.
Bonus! Large File Management for Photography
Large file management, I'm not only an expert, I'm a client to boot. Since this is a newsletter about systems and I have a good system for supporting large files, I thought I would share it here with some commentary.
Shooting
I shoot RAW so the files are large. I shoot on camera bodies (Nikon) that allow me to save RAW files to two cards simultaneously. So as soon as I click the shutter, that image is stored in two places.
Storage is cheap, if you are outshooting your card size, get more cards, don't overflow onto the second card.
Computer
I use a Mac for photography. I made the switch years ago and here is the honest truth why.
....
.... The trackpad on the Mac is comically better than any other laptop trackpad. Now, the Mac handle photography processing extremely well but Windows machines can do just as well.
But the trackpad is indeed glorious.
I shoot thousands and thousands of images a year. And I am a digital hoarder. So I tend to go with large storage on the Mac, as the internal drive is not upgradable.
Image Import
I use Lightroom to manage the import of files onto my computer. Why?
Because it gives me a logical view of my file structure. I organize by year, then by event, then by date. So if I shoot for a week in 2023 in the San Juans, I store my files in the year folder 2023, then in the San Juans event folder, and finally by the day the image was shot.
Additionally, Lightroom let's me do this on import:
-
File Handling
- Build Smart Previews but I set the build previews to minimal (Be careful, those previews can get voluminous quickly)
-
Make a Second Copy (auto-magically)
- The second copy is pointed at the external drive that connects to my computer. So on import of a card, I now have data in two distinct, although uncomfortably close, locations.
Automated Network Backup
Using the Mac, I have access to Time Machine, which is an economical way to automate backups. Configuring it to go to a network storage system somewhere can be a bit tricky at times BUT once configured, it works well. These backups kickoff every couple hours.
So now the image has been imported and lives:
- On the local machine
- On an external drive
- On a network drive somewhere.
Automated Cloud Backup
I use a program called iDrive. It is fairly economical and I stored compressed files there. It works seamlessly with the Synology network storage I have and the Synology auto-magically sends new data to iDrive.
Now, several years ago using this system, a surge at my home fried my workstation, an external drive and the network storage system. Years 2005 to 2014 were wiped out.
But they existed in the cloud on iDrive. It took over a month to recover the data, but all the files were recovered.
Managing Local Storage
At some point, your local drive is going to get full. Happens to the best of us. That's ok, this is another reason why using Lightroom is efficient. When local storage is near full, I use Lightroom to move the oldest year on my computer to the external drive.
It will update the catalogue in LR and physically move the files.
And you can do this with a simple drag and drop in Lightroom.
And there you have it, an almost fully automated storage strategy.
News
We're back baby! For the first time in over 50 years, the US has landed something on the moon. Intuitive Machines completed the first commercial moon landing.
Free Anti-Virus... Do we all remember the saying: If you aren't the paying customer, you're the product. Free AV maker Avast fined by the FTC $16.5 million for selling user browser data.
Data quality maturity, comes up almost every week these days. Alation created an assessment to quantify how mature a data culture you have and how it contributes. SMBs release data fart to demonstrate their maturity level.
Apple set for another giant fine from the EU, a paltry half billion dollars, for some sort of infringement. My Apple infringement bank account balance still at 0 dollars.
YTS Weekly Recommendation
Cybersecurity is getting wild out there! Not sure where to begin?
Say Hi on LinkedIn, Twitter/X, & Instagram
Did someone send you this awesome email? Subscribe Here
Need additional help? Reach out to the YTS Team!