Lightroom, External Drives, and Filesystems

Tl;dr: Don’t use exFAT to store your Lightroom catalog.

What’s exFAT, you say? What’s a filesystem, you say? Okay, let me back up. If you don’t use external drives, you probably don’t care about filesystems. Your computer came with one that the OS is installed on, and you can’t really change it, so there’s not much reason to think about it. But filesystems are important, and there are pros and cons to each.

Ok, ok, what’s a filesystem?

A filesystem is the way a storage device organizes the stuff you put on it, and there are a surprising number of options. You might have heard of FAT32, that one’s been around a while. Possibly NTFS? The thing to know about filesystems is that each operating system generally has one or two that it prefers to live on, and a handfull that it can use out of the box. If you’re using Windows and Mac, like most people are, here are the ones you might have come across:

• NTFS This one is the default filesystem Windows uses for its system disk. It’s the most mature and capable Windows filesystem, even though it’s pretty old at this point (its lineage dates from 1993). It’s not modern by filesystem nerd standards, but it works really well and has lots of good, resilient features. The biggest disadvantage is that it only works out of the box on Windows – Mac can read it, but can’t write to it. Most Linux distros can read and write, but with a relatively slow open source driver.
• FAT32 This FS is old as time, but also shows its age. It’s from Microsoft and a few other companies, and was introduced in 1977. The advantage is that everything ever made can read and write to it. The disadvantages are numerous, but the big ones are that 1) you can’t use it on drives larger than 2TB, and 2) any individual file can’t be larger than 4GB. Those are serious limits for modern tech. Most main disks are larger than that these days, and many devices can easily generate files (like video) larger than 4GB.
• HFS+ This one is the sort of NTFS equivalent in Apple land – the main filesystem used for OSX. It’s been through a bunch of versions and bolt-on modernizations, and again, while a filesystem nerd wouldn’t laud its modernety, it does pretty well. But you REALLY can’t read OR write it on anything but a mac. Okay – again, on Linux you can, but you generally wouldn’t want to have to.
• And lastly, exFAT. This one’s relatively new. It’s also by Microsoft and doesn’t have universal device support, but Mac and Windows both natively support reading and writing it, and most other systems have a driver available that you can buy. It’s basically 64-bit FAT32. It solves the two biggies of FAT32 I mentioned above – the volume and file size limits – and does so in a package that’s relatively lightweight and easy for embedded systems like phones and cameras and wearables to process. But it holds on to a lot of the myriad other problems of FAT32, and that’s what the problem is.

exFAT sounds like the best option to use across platforms, what’s the issue?

The real problem is performance, in terms of both data robustness and raw speed. Both HFS+ and NTFS are journaling fileystems. Without digging in, that means they’re much more tolerant to real-world errors like accidentally unplugging the drive, or having it go down in a power outage, or what have you. They’re also much faster for dealing with lots of small files all over the place. And it turns out, Lightroom LOVES those. What brought this post up is that copy of 142k files above. That’s me, trying to back up my main lightroom catalog. This is probably one key reason Adobe says leave your catalog on your main internal drive, but that’s not what I do – my whole catalog and image folder lives on a (super fast) external SSD. But exFAT holds it back.

The big problem is speed

That copy above touches on the problem. To back this disk up, I have to copy 142k files totaling 436GB. About 110k of those sum to 10GB, and the rest are 30k actual raw images. The goofy part is, the images copy at the max speed of the weak link in the chain, the spinning disk I’m copying to. That runs at ~110MB/s. But the bottleneck is all the small files. I clocked them copying at about 1 minute per thousand files, from one exFAT disk to another. That’s why you see the 12MB/s rate up there instead of what the drives are actually capable of. 110 minutes is a lot of time to spend copying things before you even get to the meat of the copy, which will take another hour. And moreover, I suspect Lightroom trying to hammer those files all day long probably hits that same per-file latency issue. Who knows what speed I was sacrificing just by having my catalog on a device ill-suited to the task.

The second problem is data integrity

I’d also come to no longer trust my data in the hands of the exFAT disks. You can only plug a drive in so many times to messages saying “This drive must be repaired” before you start to worry about whether your precious catalog still exists like you expect it to. But worse, running chkdisk (“Error checking”, from the windows right-click, properties, tools menu) takes half a month on a reasonably-sized exFAT partition (ok, an hour or two, could be). The journal on an NTFS partition makes the process of detecting errors WILDLY faster – 10 seconds. That’s very reassuring when you’ve just dropped a disk in a power outage or something.

So what, then?

My biggest reason for choosing exFAT (besides that it was default) is that I’ve got both Mac and Windows machines. So what can I do? Easy. Format all my externals to NTFS, and take my pick of Paragon NTFS for Mac or Tuxera NTFS for Mac, both of which are about $30 and enable good-performance read and write of NTFS on Mac. No more worrying about data integrity, spending hours with “Find and fix errors” windows, and dealing with slow backups.

After switching my drive to NTFS, I timed the same transfer (back onto the device) that I was trying before. For the large files, maximum disk bandwidth is the bottleneck, just like on exFAT. But on NTFS, the small files copied at 6.85 seconds per thousand – way faster than a minute. The good life.

Of course, if you’ve got more questions about Lightroom and external drives, feel free to comment. I’ve learned quite a bit in this process, and haven’t even hit on LR workflows yet.