Need a linux and mac compatible external hard drive please!
November 29, 2011 7:59 AM Subscribe
I need help selecting an external hard drive that will work to backup a linux with some rather large files on it and still be usable with a mac for occasional data storage. I'd like it to be at least 2 TB and vaguely reasonable in price. I've tried just searching for a hard drive online, but there seems to be some Mac and Linux specific things (or at least used to be) that may cause me problems, so I thought I'd ask for some guidance.
katers890: "and vaguely reasonable in price."
Just a heads up that the flooding in Thailand has crippled hard drive production and there are price spikes on drives as a result.
posted by sharkfu at 8:22 AM on November 29, 2011
Just a heads up that the flooding in Thailand has crippled hard drive production and there are price spikes on drives as a result.
posted by sharkfu at 8:22 AM on November 29, 2011
Response by poster: What I was worried about was that I saw that FAT limits files to only 4GB, is this true? Some of my files will be larger than that unfortunately.
I was aware of the flooding increasing prices, that's why I'm just shooting for vaguely reasonable.
posted by katers890 at 8:27 AM on November 29, 2011
I was aware of the flooding increasing prices, that's why I'm just shooting for vaguely reasonable.
posted by katers890 at 8:27 AM on November 29, 2011
The only Mac/Linux specific thing I know of is the filesystem you choose to put on that hard drive.
And, to a slightly lesser degree, the partition table format.
FAT32 breaks the 4GB barrier, and has done for ages, so that's not an issue. But: if you're never going to require Windows access, and don't mind keeping the backup and storage partitions separate, it's probably best to repartition to two partitions, one HFS+ and one ext4. That way, you're less likely to run into permissions and ownership issues, which matters when running full-system backups. Most Linux distros have gpart frontends to do this. If you want your Mac backup to be bootable, then you need the drive to have a GUID partition table.
Any bogstandard consumer external hard drive -- i.e. one that's not a NAS device with its own firmware OS -- should be able to do this.
posted by holgate at 8:36 AM on November 29, 2011 [1 favorite]
And, to a slightly lesser degree, the partition table format.
FAT32 breaks the 4GB barrier, and has done for ages, so that's not an issue. But: if you're never going to require Windows access, and don't mind keeping the backup and storage partitions separate, it's probably best to repartition to two partitions, one HFS+ and one ext4. That way, you're less likely to run into permissions and ownership issues, which matters when running full-system backups. Most Linux distros have gpart frontends to do this. If you want your Mac backup to be bootable, then you need the drive to have a GUID partition table.
Any bogstandard consumer external hard drive -- i.e. one that's not a NAS device with its own firmware OS -- should be able to do this.
posted by holgate at 8:36 AM on November 29, 2011 [1 favorite]
FAT will get you in trouble with file size limitations. EXFAT would work, but there is only experimental support (or paid through Tuxera) for Linux at this point. Reverse is true for NTFS - Macs can read it, but without NTFS-3G or Tuxera's paid version you wouldn't be able to write to it.
There is Linux kernel support for HFS+ (Mac native format) if you install hfsprogs, but apparently this requires you to disable journaling when formatting the drive, and shouldn't be used with drives over 2 TB due to potential corruption issues under Linux. A fix is in the works according to Wikipedia but hasn't landed yet.
What I have done in the past is to keep a drive shared on a Linux box over the network then writing to it from the Mac. In this setup, the Linux box handles the writes to disk, so the Mac doesn't care what filesystem is used. But the Mac can't read directly from the disk.
So I guess HFS+, not journaled, 2 TB or less is your current best answer?
posted by caution live frogs at 8:36 AM on November 29, 2011
There is Linux kernel support for HFS+ (Mac native format) if you install hfsprogs, but apparently this requires you to disable journaling when formatting the drive, and shouldn't be used with drives over 2 TB due to potential corruption issues under Linux. A fix is in the works according to Wikipedia but hasn't landed yet.
What I have done in the past is to keep a drive shared on a Linux box over the network then writing to it from the Mac. In this setup, the Linux box handles the writes to disk, so the Mac doesn't care what filesystem is used. But the Mac can't read directly from the disk.
So I guess HFS+, not journaled, 2 TB or less is your current best answer?
posted by caution live frogs at 8:36 AM on November 29, 2011
holgate: FAT32 should break the barrier but in practice it doesn't work as well as you'd expect. I couldn't make backups to a FAT32-formatted drive from my Mac due to file size issues. Until I installed NTFS-3G and formatted the drive as NTFS, I had errors with every backup attempt. (I eventually solved the problem permanently by ditching my Windows machine.)
posted by caution live frogs at 8:38 AM on November 29, 2011
posted by caution live frogs at 8:38 AM on November 29, 2011
FAT32 does, indeed, limit files to 4 GB. holgate is referring to the partition size, I think. FAT32 is still listed by Microsoft as being limited to 4 GiB files.
You're stuck with choosing one of the following:
* NTFS: Windows-native, Linux can deal with it (though it won't allow write access if the volume is "dirty", i.e. you unplugged the drive without telling windows to "safely eject" it first). There are commercial programs (paragon?) that provide file-system drivers for Mac NTFS.
* FAT32 and just archive everything inside a file format that can split across multiple files (to avoid the 4 gig limit) and can maintain permissions and other metadata. See: tar, etc.
Aaaand that's it. Because the FUSE project for ext2/3 on Mac has been dead for a while. And Windows reading HFS+ is a very funny joke.
Is a multi-partition solution reasonable for you? One HFS+ partition (for the Mac) and one ext3 partition for everyone else (since there's a wonderful free installable file system for ext2/ext3 on Windows).
posted by introp at 8:46 AM on November 29, 2011
You're stuck with choosing one of the following:
* NTFS: Windows-native, Linux can deal with it (though it won't allow write access if the volume is "dirty", i.e. you unplugged the drive without telling windows to "safely eject" it first). There are commercial programs (paragon?) that provide file-system drivers for Mac NTFS.
* FAT32 and just archive everything inside a file format that can split across multiple files (to avoid the 4 gig limit) and can maintain permissions and other metadata. See: tar, etc.
Aaaand that's it. Because the FUSE project for ext2/3 on Mac has been dead for a while. And Windows reading HFS+ is a very funny joke.
Is a multi-partition solution reasonable for you? One HFS+ partition (for the Mac) and one ext3 partition for everyone else (since there's a wonderful free installable file system for ext2/ext3 on Windows).
posted by introp at 8:46 AM on November 29, 2011
Linux is the more flexible OS out of the two...so I would format it HFS so it'll work well on OSX. Just keep in mind, while Linux can read write many filesystems, it has historically had issues with NTFS-3g and HFS+. I think most of those issues have been ironed out as of late (eg. the corruption issues for HFS drives above 2tb), but just to be on the safe side, always back things up to another location.
posted by samsara at 8:55 AM on November 29, 2011
posted by samsara at 8:55 AM on November 29, 2011
Response by poster: Hmm. Methinks I've oversold my computer intelligence here. For now, assume I'm a step above computer dumb, but not much. And direct examples of drives that are good would be appreciated as well, as newegg (at least) isn't showing much about which drives will work with linux as well as mac.
As far as partitioning it, I often have to use it to transfer data from my mac to the linux, so I assume having separate partitions wouldn't work with that, correct?
This will primarily be a backup for the linux, I just occasionally need to pull stuff off the mac to the linux, and this stuff I don't think has the size issue (or at least my old hard drive seemed to work fine between the two), so maybe I need two separate things in the end?
posted by katers890 at 8:59 AM on November 29, 2011
As far as partitioning it, I often have to use it to transfer data from my mac to the linux, so I assume having separate partitions wouldn't work with that, correct?
This will primarily be a backup for the linux, I just occasionally need to pull stuff off the mac to the linux, and this stuff I don't think has the size issue (or at least my old hard drive seemed to work fine between the two), so maybe I need two separate things in the end?
posted by katers890 at 8:59 AM on November 29, 2011
Best answer: You'll be fine on popular brand USB drives no matter what you pick....those are designed to industry standard. It's the formatting of that drive you'll want to experiment with. FAT32 and HFS+ look like the prime candidates with their own list of perks and trade-offs as mentioned above. If you're looking at smallish files, you'll likely be fine with FAT32. Just beware that FAT corruption can occur easily on FAT32 drives if dismounted improperly....and by default I believe the cluster size for FAT32 is 32k opposed to 4k for HFS. This means if you have lots of very small files, FAT32 will take up more physical space on the drive as the remaining bits in the 32k cluster is unused per file. With HFS, the smaller cluster size is more efficient with small files, but over time will fragment on your larger files causing slower reads (ymmv, many I've talked with barely notice the difference).
I would google the differences between these file systems and read on the pros and cons, and experiences under Linux and/or OSX. There's lots of good info out there that I don't think we could really fully approach in this thread. Also consider keeping a copies of your important files on cloud storage or the host PCs themselves just in case things go south. Another factor to keep in mind with very large drives is the lifespan is still the same as it was for smaller drives about 10 years ago. Expect to get a good 3-5 years of reliable use from a portable before it starts developing bad sectors...have a contingency plan if the data is irreplaceable...and so on. When you're talking terabytes of data...it's probably best to use another 4TB drive as the standby (due to possible bandwith shortfalls). Bluray might also be a decent alternative if stored properly (expect about ~10 years in a moisture free cool environment).
I realize that last paragraph might be a bit overkill for the topic. Just these things instantly come to mind when I think of portable drives, and the risks involved when relying on them. Again, I think you'll be fine with FAT32 or HFS+ (HFS will have to be enabled on the linux side). Since NTFS is not native to either OS, I'd stay clear of that. Ext2/3 might work out too if you'd rather start with one of Linux's native-ly supported file systems...there's fuse-ext2 that could be helpful there. There's also MacFUSE, however that is no longer being developed past Snow Leopard.
posted by samsara at 11:20 AM on November 29, 2011
I would google the differences between these file systems and read on the pros and cons, and experiences under Linux and/or OSX. There's lots of good info out there that I don't think we could really fully approach in this thread. Also consider keeping a copies of your important files on cloud storage or the host PCs themselves just in case things go south. Another factor to keep in mind with very large drives is the lifespan is still the same as it was for smaller drives about 10 years ago. Expect to get a good 3-5 years of reliable use from a portable before it starts developing bad sectors...have a contingency plan if the data is irreplaceable...and so on. When you're talking terabytes of data...it's probably best to use another 4TB drive as the standby (due to possible bandwith shortfalls). Bluray might also be a decent alternative if stored properly (expect about ~10 years in a moisture free cool environment).
I realize that last paragraph might be a bit overkill for the topic. Just these things instantly come to mind when I think of portable drives, and the risks involved when relying on them. Again, I think you'll be fine with FAT32 or HFS+ (HFS will have to be enabled on the linux side). Since NTFS is not native to either OS, I'd stay clear of that. Ext2/3 might work out too if you'd rather start with one of Linux's native-ly supported file systems...there's fuse-ext2 that could be helpful there. There's also MacFUSE, however that is no longer being developed past Snow Leopard.
posted by samsara at 11:20 AM on November 29, 2011
Mea culpa, introp was right that I was confusing file and partition maximums -- it remind me of how blessedly long it's been since I had to make the FAT32 vs. NTFS decision on a Windows system.
So I'd agree with samsara that it's smart to have two partitions: a native Linux one (ext3) for the backups, and a Mac-native one (HFS+, with the caveats mentioned) for the Linux-Mac transfers.
posted by holgate at 1:28 PM on November 29, 2011
So I'd agree with samsara that it's smart to have two partitions: a native Linux one (ext3) for the backups, and a Mac-native one (HFS+, with the caveats mentioned) for the Linux-Mac transfers.
posted by holgate at 1:28 PM on November 29, 2011
« Older Our fancy dress party theme is '2011'. Any ideas? | Is it squirrels? They must use squirrels. Newer »
This thread is closed to new comments.
posted by straw at 8:02 AM on November 29, 2011