I’ve been working with computers for over thirty years now, and one thing I’ve learned from painful experience is never, ever trust your hard drive. It will let you down; the only question is when.
Whenever I create new data on a computer from e-mail to new photographs to web content, I always think about how I’m going to recover that data when the computer that’s carrying it goes dead. It sounds harsh, but that rule has saved my bacon more times than I care to count. (I’ve had two hard drives fail already this year!)
I’ve set up a pretty decent backup system using a Mac Mini, a large external hard drive, and the Perforce source code control system. Now Perforce is typically used to manage the sharing and evolution of source code among the members of a software development team. (We use it throughout Adobe, for example.) Granted, at home, I’m not a software development team, but I find it to be a very handy way to manage the various kinds of content that I develop and maintain. For example:
- Web sites: The source code and content for all of the websites I maintain is stored in Perforce. Each site calls home nightly to push a copy of its traffic logs and database content back home. I use a couple of different web hosting providers and none of them have ever lost data on me, but if they did, I’d be out no more than 24 hours’ worth of data. (My sites don’t evolve that quickly, so this is a reasonable threshold. If I wanted, I could easily make it hourly or more frequent.)
- Software distributions: I use the same set of software to run most of my sites (usually WordPress, Drupal, or PmWiki, often combined with Mint or AWStats for traffic logs). I download to a common “distribution” folder, add the changes I like each, then branch to a special folder for each site. Whenever I decide to take a new version of that software, I just check in the new distribution on top of the original, integrate again, and poof — the site is updated, along with my patches.
- Photography: The entirety of my photography collection (currently about 55 gigabytes) is checked into my Perforce server. Perforce actually does very well at managing physically large data sets such as this. P4 is then my main mechanism for bringing my photo library from one computer to another. (Hint to Lightroom users: As long as the directory structure is the same on your computers, you can check in the Lightroom Library.aglib file on one computer, sync it to another, and your photo library moves from one computer to another. I suggest using the binary+l filetype to prevent editing the library on multiple computers at once. I haven’t tested this yet, but you should be able to move the library back and forth from Mac to Windows as well.)
- Music: Same deal. The iTunes library file lends itself to moving from one computer to another so long as the directory structure is essentially the same on both computers. You can move an iTunes library file from Mac to Windows, but don’t ever bring that library file back from a PC to a Mac. It won’t work well. Sometime later, I’ll blog about how I cut down my music library to a size that will fit on a laptop. I have about 45 GB of music on my Perforce server. I also use the Mini as an iTunes music server. A Roku SoundBridge and my mother-in-law’s computer get their music exclusively from this server.
- Disk images: Once I get all the software I like installed on a new machine, I make a disk image of the System partition on that computer. If (err… when?) the hard drive later fails on that machine, I can get it back up and running very quickly by restoring from that disk image.
Oh, and it’s a pretty easy off-site backup system as well. I have another Mac Mini in a secret undisclosed location. Every day in the middle of the afternoon, it calls home and does a p4 sync. Presto, up-to-date offsite backup!
This solution is not for the everyday user. At $500 for the Mini, $300 for the hard drive, and $750 for the Perforce license (*), it’s probably reserved for the well-to-do power geek. But if you fit that description, it’s a very nice way to go.
(*) Note: If you have fewer than five computers and only one or two users, you can use Perforce for free without a license.
Hi there,
I was just listening to Lightroom Podcast #8 and one of the speakers described how one of their colleagues uses Perforce as a digital asset management system. I had never heard of that before, and am not sure how that would work. From your description above, it sounds like you use it to synch lightroom catalogs, but is it capable of searching IPTC or other typical DAM tasks? I would like to hear more about your experiences. I’m curious, because like you, I have gigabytes of captioned photos on two servers, and am trying to figure out how to get a DAM to manage them. Many commercial options just aren’t set up for collections of this size or cost thousands of dollars. Perforce’s limited license is just right for my scale. On that same podcast, the speakers felt that even Lightroom wasn’t up to the task of handling a large collection of pictures.
Yes, that’s almost certainly referring to me. I don’t consider P4 as a digital asset management tool. Whoever used those words was overstating the case.
P4 is a great file replication and backup strategy, but it has none of the search or other DAM abilities to which you are referring.
My collection of photos numbers around 10,000 photos (maybe 150GB, haven’t specifically counted recently). LR + P4 is fine for that and maybe up into the few tens of thousands.