Filesystem for Mac and Linux

June 24, 2021 11:42 AM

I need to format an external HDD to transfer files from Ubuntu Linux (20.10) to Mac OSX Big Sur. I plugged in a preformatted external drive (NTFS) and seem to have been able to copy the files from the Linux machine, but when I try to copy to the Mac, Finder aborts when it encounters files with the same name in different directories. What filesystem can I use that will play well with both Mac and Linux?

Further background: I have a Lenovo laptop that I've been running Linux on since 2012, with a few memory and hard drive upgrades along the way. I currently have a 2GB SD about half full of (mostly) photos and music, along with years of documents and emails. The most recent Ubuntu update broke the OS (it won't even boot in safe mode), and I can't seem to repair this installation because my HDD is encrypted.

Meanwhile, my wife and son both use Macs, so I've had to become the in-house Mac tech support guy. Switching among their Macs, my Linux, and work Windows machines has become tedious, so I'm considering switching to Mac for my personal machine (and I'm currently using my son's old Macbook Air while my Linux laptop is out of commission).

I can boot my old laptop form a USB drive and use the encryption key to access my files on the SD, but my last backup is a little out of date. I've tried copying my Thunderbird profile to the Mac to access my emails, but the Mac chokes when copying from NTFS, apparently because of duplicate file names in separate directories.

Is there a filesystem that I can use to format my external HDD that will work with both Mac and Linux? It seems like Apple has madee their filesystem as incompatible with anyone else as possible. Am I better off just formatting the external drive EXT4, copying my files, and doing a clean Ubuntu install on my old laptop, giving up on ever switching to OSX?

posted by fogovonslack to Computers & Internet (27 answers total)

Not sure why you're running into errors of that sort. OS X may care if your filenames are case-sensitive, depending on how the OS X drive is formatted, but it shouldn't complain about copying the same file to multiple directories, unless you are working with symbolic links or something of that sort. Can you post a screenshot of the error dialog box that the Finder displays?

That said, there are always FAT32 and exFAT options as an alternative to NTFS. You can format the drive in Linux or OS X (via Disk Utility). As a bonus, you can read and write to these formats on both platforms.

On the Mac, you can't write to NTFS without first installing a third-party NTFS driver. Most options are still commercial, I believe, though there is an open-source MacFUSE driver for that format.

After backing up files from the NTFS-formatted drive, I'd suggest trying reformatting the drive as FAT and then reading/writing files in either direction that way. But the error sounds odd to me.

As an alternative to using a hard drive, if the two machines are on the same network, you can also look at using rsync to synchronize folders between the Linux and OS X hosts (examples). To use this tool, you will also need to enable SSH services on the Linux and OS X hosts.
posted by They sucked his brains out! at 11:56 AM on June 24, 2021

ugh, our backup drive is still formated in FAT32 for exactly this reason :-(

Here are the things that FAT32 with long filenames doesn't support that are relevant for external HDs. If you don't need them you should just use it:

• File permissions
• Files >4GB in size
• Hard links
• Certain special characters in filenames, notably : and |

One day this state of affairs will improve. That day has not yet arrived.
posted by goingonit at 11:57 AM on June 24, 2021

the Mac chokes when copying from NTFS, apparently because of duplicate file names in separate directories

If your Linux box has created files whose names include : (colon) characters on your NTFS-formatted drive, I would expect a Mac to have trouble handling these; the colon is a pathname separator for various layers of Mac OS and having one turn up inside a filename might well confuse the Finder into thinking that the characters before it are a nonexistent directory name and those after it are a nonexistent filename.
posted by flabdablet at 12:15 PM on June 24, 2021

NTFS might be best if you're not going to be copying things *from* the Mac.

You could also "just" share the disk from your Linux box using Samba and do the copying over the network. It's even possible to configure it as a Time Machine backup drive. I have this set up on my server and it actually works!

Samba might be your best bet, actually. It's the closest you can really get to a "universal" filesystem that works with all three OSes. Everything else has compromises. (NTFS can't write on Mac without special software, ExFAT works on Win/Mac but not Linux, FAT32 has 4GB limit, APFS only works on Mac, ext4 only works on Linux...)

"just" is in scare quotes because setting up Samba can be kind of hairy.
posted by neckro23 at 12:16 PM on June 24, 2021

ExFAT works on Win/Mac but not Linux

ExFAT support is in Linux kernels from 5.4.x onward. If you're using an earlier kernel it's available via FUSE (on Ubuntu the easiest way to make that work is to install the exfat-fuse and exfat-utils packages).
posted by flabdablet at 12:20 PM on June 24, 2021

Having just now set up a toy ExFAT filesystem to play with on my own Debian box (via in-kernel ExFAT support), it seems that even Linux can't create files on it whose names include any of the Windows-reserved characters / ? < > \ : * | "

That would stop the Mac choking on files containing : so it's probably a sound choice of filesystem to format a transfer drive with as long as you're always going to be working with reasonably up-to-date systems.

Unlike FAT32 and like NTFS, ExFAT has no problem with large files. Main things ExFAT can't do that NTFS can are access permissions, transparent file compression and transparent file encryption, none of which are typically enabled for NTFS volumes mounted in Linux anyway.
posted by flabdablet at 12:36 PM on June 24, 2021

Thanks for the quick replies.

I don't know why the Mac is choking on copying the files. I'm trying to copy a Thunderbird email user profile, so I don't think there are any illegal file names (Mozilla wouldn't use naming conventions in it's Linux version of Thunderbird that wouldn't also work in OSX, would they?), and the filename it refers to is spam.msf which shouldn't cause trouble on just about any filesystem. Apple tech support was utterly unhelpful about troubleshooting and finally just said that NTFS doesn't work with OSX and that you can't have duplicate file names even if they're in different directories.

In this case, I'm not trying to write to NTFS, just read from it.

I'm a little wary of sync functions after SpiderOak deleted nearly all of my work files going back over a decade from my shared folder, and not sure I feel like getting into the weeds of setting up a Samba share.

This is the error dialog I'm seeing: This operation cannot be completed because an item with the name "spam.msf" already exists.

I thought there were issues with FAT32 and drives greater than 2TB as well as filename limitations, but I may just be confused from reading more than I ever wanted to know about filesystems, much of it over a decade old.

I don't think any of my filenames have a : or | in them, and the file that throws the error is "spam.msf"

Ubuntu 20.10 looks like it ships with kernel 5.8, so it should support ExFAT, right?

So it sounds like either FAT32 or ExFAT would work for me, right? Since I'm trying to copy files to the Mac, it would make sense to format the drive on the Mac then use it on Ubuntu. According to Apple OSX will format FAT drives up to 32 GB, and ExFAT for drives over 32GB, so is ExFAT my only option here?
posted by fogovonslack at 1:15 PM on June 24, 2021

Apple tech support was utterly unhelpful about troubleshooting and finally just said that NTFS doesn't work with OSX and that you can't have duplicate file names even if they're in different directories.

This sounds like total bullshit to me. I suspect your unhelpful Apple tech support person was clueless and trying to hide it. In my experience Apple tech support is often clueless when it comes to dealing with issues that involve going anywhere near the walls of the garden.

is ExFAT my only option here?

If ExFAT will actually work on both the systems you use, I can't think of any reason to prefer FAT32.
posted by flabdablet at 1:22 PM on June 24, 2021

My best guess for your error is that it is based on some kind of link/symlink behavior difference between NTFS and your Mac filesystem where you end up copying the same directory onto itself twice and the OS chokes. As mentioned above, rsync is great at dealing with stuff like this -- it's possible that copying onto your Mac with rsync will solve the problem without having to ditch NTFS.
posted by goingonit at 1:31 PM on June 24, 2021

Just less than 2GB? I might try just uploading everything to a free service like Dropbox and downloading it on the Mac.
posted by General Malaise at 1:48 PM on June 24, 2021

Mac also won't copy files files with "/" character.
posted by a humble nudibranch at 2:08 PM on June 24, 2021

Linux won't ever create filenames containing the / character; that and NUL are the only ones that POSIX disallows.
posted by flabdablet at 2:10 PM on June 24, 2021

I think you're on to something, goingonit. I've reformatted another drive to ExFAT and started re-copying from my old hard drive, and I just got an error that a file can't be copied because the filesystem doesn't support symlinks. I told it to skip all, so maybe I'll have better luck this time.

To be fair, fabdablet, he wasn't trying to hide his cluelessness. He noted that I was more tech savvy than most who call in and that he didn't know. When I asked whether it might work using the command line, he said that it wasn't in the script and that the engineers were the only ones who knew how to do that, and that they wouldn't help me.

Thanks for catching my typo, General Malaise. That should be 2 TB of files.
posted by fogovonslack at 2:12 PM on June 24, 2021

Indeed! This is one good reason to use rsync to copy:

rsync -Lvr --safe-links /my-source-dir /mnt/external-hd/my-dest-dir

The -L will transform symlinks into their referents (files and directories). --safe-links ignores symlinks that point outside of the source directory tree, which is probably behavior that you want!

The other possibility is tar, which turns an entire directory tree into a single bytestream. This tends to smooth away a lot of the rough edges between systems. If this is a one-time transfer, you can just turn your entire backup into a single tarball on your ext HD:

tar cf /mnt/external-hd/my-archive.tar /dir-to-archive

which you then unarchive on your Mac:

cd /my-new-dir && tar xf /Volumes/external-hd/my-archive.tar 

This is a poor long-term backup format however since it makes updates and random access quite frustrating.
posted by goingonit at 2:18 PM on June 24, 2021

You can mount HFS+ volumes on Linux if you disable journaling. It's a little fiddly.

I use HFS+ for Time Machine backups, a separate HFS+ drive for archival and Borg backups, and FAT32 for occasional transfer between systems (but tar/zip everything up to avoid file metadata problems)
posted by RobotVoodooPower at 2:40 PM on June 24, 2021

you can just turn your entire backup into a single tarball on your ext HD

as long as the ext HD isn't formatted with FAT32. Because it doesn't take much of a tarball to blow straight past that filesystem's 4GiB maximum representable file size.
posted by flabdablet at 9:58 PM on June 24, 2021

Macs also have sparse bundles which may come in handy for transferring things like apps and photo libraries. It's like a filesystem inside your filesystem.
posted by RobotVoodooPower at 10:18 PM on June 24, 2021

I'm not sure how sparse bundles help me get data off of a Linux machine and onto a Mac.

Rsync failed pretty badly. It ran for about 20 mins and then gave me a message that said"failed: too many levels of symbolic links (40)" with a blinking cursor that froze the machine for about 5 hours with the cooling fan running like a jet engine. I finally just gave up and did a hard shutdown.

Guess I'll reformat on the Mac to make sure nothing got corrupted and try copying the files again.
posted by fogovonslack at 7:26 AM on June 25, 2021

You're not by any chance trying to copy a file tree onto the external drive that includes that drive's own mount point? If so, you'll probably want to exclude that mount point explicitly from the copy source, or use whatever "one filesystem only" options exist in your copy tool of choice.

Also interested to know what symlinks exist inside your Thunderbird profile. I can't find any at all in any of mine.

Also also interested to know whether your Thunderbird profile has managed to accumulate multiple variants of spam.msf with names cased different; for example, do spam.msf and Spam.msf both exist inside the same mail folder? OS X will probably choke on trying to create spam.msf if it's already copied Spam.msf, since its file naming API treats those names as equivalent.
posted by flabdablet at 7:45 AM on June 25, 2021

I'm trying to copy my home directory. The external drive looks like it's mounted in /media/ubuntu and my home directory is (as you would expect) in /home, so I don't think I'm trying to copy the drive's mount point.

I don't think the symlinks are in my Thunderbird profile. I feel like I'm living on borrowed time with this machine, so I'm trying to get a good copy of my whole home directory. When I copied it onto the NTFS drive, I got a number of filename errors in my music directory, so I'm afraid the copy I have on that drive isn't complete. I suppose that if this copy doesn't work I should copy just my Thunderbird profile to a USB stick and see if I can get that to work.
posted by fogovonslack at 7:56 AM on June 25, 2021

Ooh, a circular symlink! Circular symlinks are surprisingly easy to create.

One good reason to run rsync with -v is that you can look at the file it choked on, and then get rid of it. You can also get rsync to exclude links entirely (though the version that ships with MacOS is pretty old).
posted by goingonit at 8:04 AM on June 25, 2021

I know it sounds frustrating for me to harp on an option that failed for you, but getting rsync to work has a few big advantages. One is that, when things to do wrong, you can learn more about why that was and what you can do about it. Another is that rsync has many options that let you figure out how to get around the issues you find in the way that is right for you. But perhaps the most important is that it's replicable: you can run the same command with slight variations to get it to work without worrying that you didn't do exactly the same thing as you did last time.
posted by goingonit at 8:11 AM on June 25, 2021

Circular symlinks are surprisingly easy to create

as are symlinks with absolute pathnames as targets.

One source of symlink horror I found in my own home folder on the laptop I'm using right now is Wine-related: Wine profiles often contain a dosdevices subfolder with symlinks from various DOS drive letters to assorted absolute Unix paths, and some of mine have a symlink from Z: to / which is particularly toxic.

You can check whether a file tree you propose to copy from contains symlink shenanigans using find before trying to copy it:

find -L /path/to/source/tree -false 2>&1 | less

The -L option makes find follow symlinks instead of treating them as little files in their own right, and the -false test makes find do nothing at all with each file it discovers. This means that the only output it generates is error messages, including those complaining about broken or looped symlinks. 2>&1 combines the standard error stream with the standard output stream before piping the whole mess into less for inspection at your leisure.
posted by flabdablet at 8:48 AM on June 25, 2021

If you're migrating stuff to an OS with different opinions about how to lay out its filesystem namespaces, it's almost always better not to copy symlinks but to re-create them on the target system as needed, possibly even one at a time by hand, instead.

That makes ExFAT a safer option for formatting transfer media than NTFS, because ExFAT doesn't support symlinks to begin with.

On the other hand, if what you're trying to make is a backup for later restoration onto the same OS you started with, using that OS's native archiving tools is probably the soundest choice; that way you don't need to care how the filesystem the archive file goes on works, as long as it can support files that are big enough.
posted by flabdablet at 8:55 AM on June 25, 2021

Another thing to be aware of is that almost all the Gnu recursive copy tools have got some option for restricting the source for copied files to one mounted filesystem, suppressing the following of links leading out of that filesystem and the descent into any other filesystems that might be mounted inside it.

This is almost always what you want if you're trying to do something like back up the entirety of /home.

For both rsync and cp this is the -x short option or --one-file-system long option. If you want to use find for a preliminary check of what you'd encounter with such an option in effect, you can do

find -L /path/to/source/tree -xdev -false 2>&1 | less

posted by flabdablet at 9:06 AM on June 25, 2021

flabdablet nailed it. Just before rsync choked, I got a string of bad symlinks in my wine directory referencing z:

And no, goingonit, it's not frustrating that you keep going on about rsync. If that's the tool I need to use, then I need to figure out how to use it. I hope I'm not frustrating you with my limited knowledge and skills here.

I think a big part of my issue is that I've got close to a decade of cruft and maintenance debt piled up. I haven't used wine in I don't know how long, and I've also got a few virtualbox machines saved up that I haven't used in years that I've kept *just in case* I need to do something in Windows. Time to bite the bullet and get rid of them.

I'm trying to do a "copy it all and sort it out later" kind of thing, and I guess I really need to sort first and copy later. I've got about 2 hours left on this attempt to just copy my home directory. When that finishes (or crashes), I'll run flabdablet's find copy to clean things up and then give rsync another try.

Oh, and look at that: another invalid filename error in my music directory creating the folder "The Motet--"Funk is Dead" - 2012-04-27 - Baltimore Soundstage - Baltimore, MD" and another one for Rodrigo y Gabriela's "11:11"

Looks like I"ve got my work cut out for me. Is there some way to automate fixing filenames that are acceptable in EXT4 but not NTFS or APFS?

And thanks again for all your patience with me--I really do appreciate your help here.
posted by fogovonslack at 9:34 AM on June 25, 2021

Is there some way to automate fixing filenames that are acceptable in EXT4 but not NTFS or APFS?

NTFS doesn't have a problem with Linux-compatible filenames, but Mac OS really can't deal with : as I mentioned above. ExFAT disallows filenames containing / ? < > \ : * | " or anything with an ASCII code below 32 (control characters), and Windows will also crack the sads if it sees these on an NTFS-formatted volume, so for migration purposes it's probably best just to get rid of anything ExFAT doesn't like.

The following commands will replace anything ExFAT would object to with a - (minus sign) character:

bad=$'[\x01-\x1f?<>\\:*|"]'
find /path/to/target -xdev -regex ".*$bad.*" -print0 |
rename -0 -v -n "s/$bad/-/g" | tee /tmp/rename.log

If you enter that lot as-is it will emit a list of proposed renames, which will also be saved in /tmp/rename.log. If they look sane you can hit up-arrow to recall the previous command line, then edit away the -n option on the rename command, then hit Enter and the renames will happen for real.

The way this works is by building a regular expression fragment in variable bad that matches all the characters ExFAT doesn't like (leaving out NUL (\x00) and forward slash, since Linux won't ever put those in a filename to begin with). The special bash quoting syntax $'...' is used to define bad, which makes bash translate \x01 and \x1f into literal control characters and \\ into a single backslash within the variable's contents.

bad then gets expanded twice, once in a find command that spits out a list of all the pathnames from inside the specified folder that include those objectionable characters, and once in a rename command that consumes the list and renames all the files on it. Within the list, filenames are delimited by NUL characters, which can never be part of a filename; the -print0 action tells find to use those NUL delimiters instead of the usual newlines, and the -0 option tells rename to expect its input list to be formatted that way. This means that the commands will even fix pathological filenames that have embedded newline characters in them (a newline is \x0A, which is included in the \x01-\x1f range from the bracketed regex fragment in $bad).

About the only way this will go wrong is if you have several files in a single directory whose names are distinguished only by differences in ExFAT-illegal characters. Since all such characters get collapsed to -, there's a potential for collisions in that case.

If Ubuntu gives you grief about not having rename available, apt install rename should fix that.
posted by flabdablet at 12:11 PM on June 25, 2021