Gigabyte Doubling Trouble
February 11, 2011 5:55 PM Subscribe
The case of the missing Mac HD space, and symlink shenanigans.
(First, I have looked at the previous questions on the topic, and this isn't your run-of-the-mill bloated cache file from what I can tell.)
I have a 500GB hard drive in my computer, a Core Duo iMac, and after recently moving from a 160GB drive, the finder is telling me 342GB is used. I've checked into it with Disk Inventory X, and interestingly, it's initial drive selection/overview pane gives a similar figure (318GB), but once it runs its scan it shows only 184 GB accounted for, which seems proper. Any idea what could be causing this doubled figure that's not showing up in D.I.X.? From what I understand any hidden/cache files should show up there.
Important info:
I *think* this missing space can be localized to my home folder, because if I try to run a Time Machine backup with everything else excluded (external drives, Applications, Dev., other users and whatever Time Machine's definition of "System Files and Applications" is) it still errors out telling me the backup will require 327GB of space. But not knowing Time Machine's internal mechanics this may be a red herring.
It's very likely this space is the result of a botched transfer of the old drive's contents; I tried to use OS X's Migration Assistant, unsuccessfully, and resorted to transfering everything manually in the Finder. That said, I don't know where these botched files would be that they aren't showing up in the drive inventory.
And most significantly, This previous question seems relevant; on the advice given there I ran sudo du -hd1 / , and troublingly this is the result:
4.7M /.fseventsd
488M /.Spotlight-V100
0B /.Trashes
0B /.vol
11G /Applications
3.9M /bin
0B /cores
du: Can't follow symlink cycle from /dev/fd/3 to /dev/fd/3
My knowledge of Unix directory structures is fuzzy, but this is clearly very much not right. It doesn't seem like it should need to be following any symlinks to get to the rest of my top-level directories, broken or not... Can someone advise me on how to troubleshoot this? I've been doing manual backups since this problem rendered Time Machine useless, but I'm *really* hoping the answer doesn't require reformatting the drive if I can avoid it...
(First, I have looked at the previous questions on the topic, and this isn't your run-of-the-mill bloated cache file from what I can tell.)
I have a 500GB hard drive in my computer, a Core Duo iMac, and after recently moving from a 160GB drive, the finder is telling me 342GB is used. I've checked into it with Disk Inventory X, and interestingly, it's initial drive selection/overview pane gives a similar figure (318GB), but once it runs its scan it shows only 184 GB accounted for, which seems proper. Any idea what could be causing this doubled figure that's not showing up in D.I.X.? From what I understand any hidden/cache files should show up there.
Important info:
I *think* this missing space can be localized to my home folder, because if I try to run a Time Machine backup with everything else excluded (external drives, Applications, Dev., other users and whatever Time Machine's definition of "System Files and Applications" is) it still errors out telling me the backup will require 327GB of space. But not knowing Time Machine's internal mechanics this may be a red herring.
It's very likely this space is the result of a botched transfer of the old drive's contents; I tried to use OS X's Migration Assistant, unsuccessfully, and resorted to transfering everything manually in the Finder. That said, I don't know where these botched files would be that they aren't showing up in the drive inventory.
And most significantly, This previous question seems relevant; on the advice given there I ran sudo du -hd1 / , and troublingly this is the result:
4.7M /.fseventsd
488M /.Spotlight-V100
0B /.Trashes
0B /.vol
11G /Applications
3.9M /bin
0B /cores
du: Can't follow symlink cycle from /dev/fd/3 to /dev/fd/3
My knowledge of Unix directory structures is fuzzy, but this is clearly very much not right. It doesn't seem like it should need to be following any symlinks to get to the rest of my top-level directories, broken or not... Can someone advise me on how to troubleshoot this? I've been doing manual backups since this problem rendered Time Machine useless, but I'm *really* hoping the answer doesn't require reformatting the drive if I can avoid it...
You really, really want to repair this filesystem. When you say "moving from", what did you specifically do?
posted by mhoye at 6:22 PM on February 11, 2011
posted by mhoye at 6:22 PM on February 11, 2011
Response by poster: As I said in the question, I just manually copied over the various folders of my home directory, using the Finder. I didn't mess with any system files, aside from various plug-ins/fonts/etc. in my ~/Library .
posted by brightghost at 6:27 PM on February 11, 2011
posted by brightghost at 6:27 PM on February 11, 2011
Broken symbolic links inside of /dev really don't mean anything. It's not unusual. Fact is, you are asking for trouble in general running 'du' on /dev.
I'm curious what the output of 'df -h' says.
posted by sbutler at 6:37 PM on February 11, 2011
I'm curious what the output of 'df -h' says.
posted by sbutler at 6:37 PM on February 11, 2011
Best answer: Yeah, the /dev/fd/3 is a furphy (it's a virtual device that's tied up with the arguments / file descriptors to the 'sudo' command).
Further info: the first report from Disk Inventory X is based on what the system reports overall; the second is the summary total of what it can see as the user you're logged in as. Likely there's something there, probably in your user directory & left over from Migration Assistant, that has non-user (e.g. root / administrator) permissions only.
Try running Disk Inventory X as root from terminal e.g.
posted by Pinback at 6:51 PM on February 11, 2011
Further info: the first report from Disk Inventory X is based on what the system reports overall; the second is the summary total of what it can see as the user you're logged in as. Likely there's something there, probably in your user directory & left over from Migration Assistant, that has non-user (e.g. root / administrator) permissions only.
Try running Disk Inventory X as root from terminal e.g.
sudo "/Applications/Disk Inventory X.app/Contents/MacOS/Disk Inventory X"and see what happens…
posted by Pinback at 6:51 PM on February 11, 2011
Response by poster: df -h gives me this:
Filesystem Size Used Avail Capacity Mounted on
/dev/disk0s2 453Gi 321Gi 132Gi 71% /
devfs 112Ki 112Ki 0Bi 100% /dev
/dev/disk0s3 12Gi 4.3Gi 7.7Gi 36% /Volumes/E-Drive
map -hosts 0Bi 0Bi 0Bi 100% /net
map auto_home 0Bi 0Bi 0Bi 100% /home
/dev/disk2s3 298Gi 260Gi 38Gi 88% /Volumes/External HD
/dev/disk3s2 298Gi 296Gi 2.0Gi 100% /Volumes/SW PORTABLE
The last two are my externals; E-Drive is an emergency startup partition on Mac HD created by TechTools. Once my latest backup is done running I will try verifying in Disk Utility and report back.
posted by brightghost at 6:52 PM on February 11, 2011
Filesystem Size Used Avail Capacity Mounted on
/dev/disk0s2 453Gi 321Gi 132Gi 71% /
devfs 112Ki 112Ki 0Bi 100% /dev
/dev/disk0s3 12Gi 4.3Gi 7.7Gi 36% /Volumes/E-Drive
map -hosts 0Bi 0Bi 0Bi 100% /net
map auto_home 0Bi 0Bi 0Bi 100% /home
/dev/disk2s3 298Gi 260Gi 38Gi 88% /Volumes/External HD
/dev/disk3s2 298Gi 296Gi 2.0Gi 100% /Volumes/SW PORTABLE
The last two are my externals; E-Drive is an emergency startup partition on Mac HD created by TechTools. Once my latest backup is done running I will try verifying in Disk Utility and report back.
posted by brightghost at 6:52 PM on February 11, 2011
Likely there's something there, probably in your user directory & left over from Migration Assistant, that has non-user (e.g. root / administrator) permissions only.
Clever thought! I'm trying to think now what happens during migration...
posted by sbutler at 7:15 PM on February 11, 2011
Clever thought! I'm trying to think now what happens during migration...
posted by sbutler at 7:15 PM on February 11, 2011
I had in mind that it copied stuff to the current user's directory, but after reading up it seems it creates a new user & prompts you to log in as that user after migration.
Also, there's this possibility, specifically:
Since brightghost says the space seems to be taken up when everything but his home directory is excluded, it may not be either of those - but it'd still be interesting to see if there's a discrepancy between what Disk Inventory X reports when run as user vs root.
posted by Pinback at 7:57 PM on February 11, 2011
Also, there's this possibility, specifically:
"From a Finder window's menubar, select Go > Go to Folder and type /Volumes in the prompt. That should show an alias for every volume (disk or partition) connected to your Mac.That one's caught me a few times too.
If anything there is not an alias, but an actual folder, that's where it put it."
Since brightghost says the space seems to be taken up when everything but his home directory is excluded, it may not be either of those - but it'd still be interesting to see if there's a discrepancy between what Disk Inventory X reports when run as user vs root.
posted by Pinback at 7:57 PM on February 11, 2011
The 342/318 discrepancy is just the usual base-10/base-2 difference. If an app uses base-10 then it's using the definition of a gigabyte as 109, but if it uses the base-2 definition then a GB is 10243 = 230. The ratio of the two is 1.0737, so both programs are reporting the same amount of free space only with different units. This confusion is why the kibi/mebi/gibibyte prefixes were invented, and that's what GiB refers to in the df output.
posted by Rhomboid at 8:01 PM on February 11, 2011
posted by Rhomboid at 8:01 PM on February 11, 2011
Response by poster: Aha! For some reason it hadn't occurred to me that Disk Inventory X wouldn't catalog items I don't have permissions for... I guess I assumed it could still list their size even if not the contents? Anyway, it turns out there is 135GB in the Trash of the former administrator account of this machine, most of which is a corrupted transfer of my home folder. I still don't understand why Time Machine has apparently been trying to include this in my backup, as it is both inaccessible by my user and explicitly excluded from the backup in the options. I wouldn't think TM would backup .Trash anyway, especially not another user's .Trash.
Anyway. I'll delete that and see what TM has to say. I'll also verify the disk once my backup finishes (rearranged a bunch of video files so rsync is taking a while). That said, is there reason to believe there is something seriously wrong with the filesystem as mhoye suggests? The /dev symlink error is not an issue?
(Oh, and lsof | grep DEL returned nothing, sbutler.)
posted by brightghost at 8:19 PM on February 11, 2011
Anyway. I'll delete that and see what TM has to say. I'll also verify the disk once my backup finishes (rearranged a bunch of video files so rsync is taking a while). That said, is there reason to believe there is something seriously wrong with the filesystem as mhoye suggests? The /dev symlink error is not an issue?
(Oh, and lsof | grep DEL returned nothing, sbutler.)
posted by brightghost at 8:19 PM on February 11, 2011
The /dev symlink error is a red herring— /dev and especially /dev/fd has a bunch of special files in it (and technically isn't even on your disk; it's a fake file system) which are likely to confuse tools like du.
posted by hattifattener at 8:36 PM on February 11, 2011
posted by hattifattener at 8:36 PM on February 11, 2011
Best answer: "That said, is there reason to believe there is something seriously wrong with the filesystem as mhoye suggests?"
Not really, and verifying through Disk Utility will tell you if there is.
"The /dev symlink error is not an issue?"
Nope. Technical answer here, but the short version is that it's the Unix Way to expose all sorts of non-file things through the filesystem. When something that's expecting a real file looks at them, the results may unexpected. Another example would be /dev/random - not a file, but the system's random number generator. Try
posted by Pinback at 11:04 PM on February 11, 2011
Not really, and verifying through Disk Utility will tell you if there is.
"The /dev symlink error is not an issue?"
Nope. Technical answer here, but the short version is that it's the Unix Way to expose all sorts of non-file things through the filesystem. When something that's expecting a real file looks at them, the results may unexpected. Another example would be /dev/random - not a file, but the system's random number generator. Try
less -f /dev/random to see what a bunch of random-generated junk looks like.posted by Pinback at 11:04 PM on February 11, 2011
Response by poster: Update: ran Disk Utility's verify last night and it repaired one error ("Invalid directory item count: expected 34, found 33"). I'm still puzzled as to why Time Machine would be including that .Trash directory in its count when I had that user's directory excluded, but everything is working as expected now. Thanks so much for the help everyone!
posted by brightghost at 1:49 PM on February 12, 2011
posted by brightghost at 1:49 PM on February 12, 2011
brightghost - a user's trash doesn't live inside their home directory (otherwise putting something on an external disk in the trash would require copying it onto the system volume). there's an invisible folder called ".Trashes" at the root of every disk and that's where individual user trash folders go.
posted by russm at 11:06 PM on February 12, 2011
posted by russm at 11:06 PM on February 12, 2011
This thread is closed to new comments.
Also, have you tried verifying the volume in Disk Utility yet?
posted by Kalthare at 6:05 PM on February 11, 2011