My external hard drive shows a different capacity than it actually has.
February 25, 2015 2:04 PM Subscribe
Upon adding up the separate folders within my drive, I can see a large discrepancy between the two amounts. I've utilzied many of the options within disk utilty for OSX. Please help.
Indeed, the drive appears to have 1.46TB of data (out of 1.5 total capacity), when in fact it only has 825GB of data.
Items to ponder:
Most of this data is CR2 RAW image files.
I utilized the Gemini duplicate finder application on this drive.
I utilized disk permissions and disk repair.
I utilized the erase free space feature within MAC disk utility.
The format is Exfat. This was necessary because the drive was originally for PC.
I used erase/restore when copying this drive to another drive. I mention this because I did not simply drag and drop, and that the discrepancy is now on both drives.
The drive is a Seagate.
One final item to ponder. Originally, some years ago, I did not partition the drive.
All your help is appreciated. Thanks in advance.
Indeed, the drive appears to have 1.46TB of data (out of 1.5 total capacity), when in fact it only has 825GB of data.
Items to ponder:
Most of this data is CR2 RAW image files.
I utilized the Gemini duplicate finder application on this drive.
I utilized disk permissions and disk repair.
I utilized the erase free space feature within MAC disk utility.
The format is Exfat. This was necessary because the drive was originally for PC.
I used erase/restore when copying this drive to another drive. I mention this because I did not simply drag and drop, and that the discrepancy is now on both drives.
The drive is a Seagate.
One final item to ponder. Originally, some years ago, I did not partition the drive.
All your help is appreciated. Thanks in advance.
What does disk utility show you when you look at partitions? Do you see more than one partition? See the image under the section "Partitioning your hard disk" here:
http://www.belightsoft.com/products/getbackup/GBHelp/cloning_formatting.html
If you click on each partition in that view, the program will show you how large each partition is.
Also, when you did a restore, how big is the image that you were restoring from? Was it not 1.5TB?
I'm guessing that you have a large section of the drive which is unformatted. This would be completely unaffected by erasing free space. Instead, you'd have to back up your data, reformat the drive, partition it with one partition, and then copy the data back.
posted by Mo Nickels at 2:20 PM on February 25, 2015 [1 favorite]
http://www.belightsoft.com/products/getbackup/GBHelp/cloning_formatting.html
If you click on each partition in that view, the program will show you how large each partition is.
Also, when you did a restore, how big is the image that you were restoring from? Was it not 1.5TB?
I'm guessing that you have a large section of the drive which is unformatted. This would be completely unaffected by erasing free space. Instead, you'd have to back up your data, reformat the drive, partition it with one partition, and then copy the data back.
posted by Mo Nickels at 2:20 PM on February 25, 2015 [1 favorite]
The discrepancy could be because of the cluster size of the partition. Basically, data on a hard disk is stored in little containers called clusters or blocks. If the container is too small for a file multiple containers are used. Depending on the kind of data you have, there can be inefficiencies present if you choose the wrong cluster size.
Consider - on a partition with 128k cluster size, and files that are all 132-170k in size, every file will take 256k of space on the hard disk no matter how big they actually are. The extra unused space in the second cluster is wasted.
You can use a smaller cluster size - this results in less wasted space, but many more clusters, so overhead is increased and performance can diminish. Particularly in the case of disk checks, which will scan every cluster - making you cluster sizes 4 times smaller means 4 times as many clusters and 4 times as long to check them.
If the extra space is necessary to you, you will probably have to reformat the drive (and replace the data from backup). I don't think you can modify the cluster size of an exfat system without reformatting.
posted by Pogo_Fuzzybutt at 2:32 PM on February 25, 2015 [1 favorite]
Consider - on a partition with 128k cluster size, and files that are all 132-170k in size, every file will take 256k of space on the hard disk no matter how big they actually are. The extra unused space in the second cluster is wasted.
You can use a smaller cluster size - this results in less wasted space, but many more clusters, so overhead is increased and performance can diminish. Particularly in the case of disk checks, which will scan every cluster - making you cluster sizes 4 times smaller means 4 times as many clusters and 4 times as long to check them.
If the extra space is necessary to you, you will probably have to reformat the drive (and replace the data from backup). I don't think you can modify the cluster size of an exfat system without reformatting.
posted by Pogo_Fuzzybutt at 2:32 PM on February 25, 2015 [1 favorite]
The discrepancy is in the hidden folders. There are a ton of Unix plumbing and other folders/files on your hard drive that the Finder hides from you to prevent you from accidentally moving or deleting them.
You might also have a lot of cache files clogging up the system in those hidden folder.
/var/db is notorious for being used as a dumping ground for a lot of cache stuff.
If you really want to see those files/folders, you have to go in through the Terminal.app and use command line tools like ls, df, or du to see the actual file/folder count and locations.
First suggestion would be to empty the Trash and then reboot and see if the disk will auto-clean up some of that space.
posted by daq at 2:38 PM on February 25, 2015
You might also have a lot of cache files clogging up the system in those hidden folder.
/var/db is notorious for being used as a dumping ground for a lot of cache stuff.
If you really want to see those files/folders, you have to go in through the Terminal.app and use command line tools like ls, df, or du to see the actual file/folder count and locations.
First suggestion would be to empty the Trash and then reboot and see if the disk will auto-clean up some of that space.
posted by daq at 2:38 PM on February 25, 2015
What does e.g. Disk Inventory X tell you?
posted by kindall at 2:41 PM on February 25, 2015 [3 favorites]
posted by kindall at 2:41 PM on February 25, 2015 [3 favorites]
When you say you're "adding up the separate folders on your drive" how are you doing that? Are you making sure to include hidden files/folders? That would be my first guess as the where the discrepancy comes from.
On the Mac, there's a file in every directory that you've opened in the Finder called
If this were your startup disk (or a clone of an old startup disk), there's a lot of other hidden folders that would be more likely to take up this much space.
A utility like Daisy Disk might help you find out where the discrepancy is. It includes hidden folders and files, so would at least eliminate that angle.
posted by duien at 2:42 PM on February 25, 2015 [1 favorite]
On the Mac, there's a file in every directory that you've opened in the Finder called
.DS_Store that stores information such as where the Finder window was on your screen and what settings that window had. It also stores preview images for the files that can be used for Spotlight and as icons. If you have a lot of images, those files could be taking up a decent amount of space. I don't think it would be as much as the discrepancy you're seeing, but it could certainly be contributing.If this were your startup disk (or a clone of an old startup disk), there's a lot of other hidden folders that would be more likely to take up this much space.
A utility like Daisy Disk might help you find out where the discrepancy is. It includes hidden folders and files, so would at least eliminate that angle.
posted by duien at 2:42 PM on February 25, 2015 [1 favorite]
(on that same note, assuming this is an external drive)
Oh, another thing to consider, especially regarding the Trash.
If you have connected this drive to another computer, and put items in the trash, but did not empty the trash before disconnecting and reconnecting to another computer, the main Trash folder may be owned by another user, and still contain those files.
If you go in on the command line, using Terminal.app, you should be able to navigate to the root of the drive (assuming this is an external) by using the following command:
cd /Volumes/[drive name]/
Then use the ls -la command to list all the files/folders in the root of the drive. You will see a hidden folder called .Trashes, which is where the file system keeps all files that are moved to the trash. If you go into that folder (cd .Trashes) and then list the contents (ls -la) you will likely see at least one folder with a numerical value (i.e. 501 or 502). Those are the trash folders for the user account that was using the drive. If that number does not match your UID (User ID, used by the operating system to assign ownership of folders), then you will not be able to "see" those files that are hidden because you don't have permission.
This is where it gets tricky. You will have to use sudo to get inside and look at what might be in there, and then use the ever so dangerous rm command to delete or move them. You could also change the ownership and permissions of those folders, but that is still kind of bleah.
If this is the case, MeMail me and when I get time I can write up a way of trying to do this that will hopefully be easier than the other options.
posted by daq at 2:48 PM on February 25, 2015 [1 favorite]
Oh, another thing to consider, especially regarding the Trash.
If you have connected this drive to another computer, and put items in the trash, but did not empty the trash before disconnecting and reconnecting to another computer, the main Trash folder may be owned by another user, and still contain those files.
If you go in on the command line, using Terminal.app, you should be able to navigate to the root of the drive (assuming this is an external) by using the following command:
cd /Volumes/[drive name]/
Then use the ls -la command to list all the files/folders in the root of the drive. You will see a hidden folder called .Trashes, which is where the file system keeps all files that are moved to the trash. If you go into that folder (cd .Trashes) and then list the contents (ls -la) you will likely see at least one folder with a numerical value (i.e. 501 or 502). Those are the trash folders for the user account that was using the drive. If that number does not match your UID (User ID, used by the operating system to assign ownership of folders), then you will not be able to "see" those files that are hidden because you don't have permission.
This is where it gets tricky. You will have to use sudo to get inside and look at what might be in there, and then use the ever so dangerous rm command to delete or move them. You could also change the ownership and permissions of those folders, but that is still kind of bleah.
If this is the case, MeMail me and when I get time I can write up a way of trying to do this that will hopefully be easier than the other options.
posted by daq at 2:48 PM on February 25, 2015 [1 favorite]
Once you've navigated to the root level of the drive, as daq suggests, you can also run the command:
du -xh -d1 .
That will calculate the actual disk usage of each of the subfolders on the drive. You can even cd into a particular folder on the drive to check its usage if it seems like it's too big.
posted by one more dead town's last parade at 2:51 PM on February 25, 2015
du -xh -d1 .
That will calculate the actual disk usage of each of the subfolders on the drive. You can even cd into a particular folder on the drive to check its usage if it seems like it's too big.
posted by one more dead town's last parade at 2:51 PM on February 25, 2015
As suggested above, and before anything else, just check your Trash and empty it if it isn't already. It may simply be that.
Apart from that, I find it's rarely worth the time to work out what's wrong with stuff. It's generally much faster to just nuke from low orbit and start afresh. In this case, I would copy all my files to another drive, erase the external and then copy the files back.
Unless you need to use this drive on a Windows machine there is no reason to leave it formatted as ExFat. If it is only going to be used on a Mac then reformat it as 'Mac OS Extended (journaled)'. As per the first image in Mo Nickels' link, make sure you select the disk rather than the volume when you erase/reformat.
A few things perhaps worth noting:
• It is rare under normal use and even when troubleshooting to need to use the Terminal. There are almost always other ways to achieve the same outcome. There is nothing wrong with using it either if you're comfortable with it, but it is unforgiving - a simple typo can erase your hard disk or render it incapable of booting. I use it when I have to, but it's not often.
• You don't have to manually add up the sizes of all the folders on your drive. Just select them all and click on the File menu. If you then hold the option (alt) key down, you will notice that 'Get Info' changes to "Show Inspector'. If you click on that, Finder will display a window that will add them up for you. Sometimes this can take a few minutes if there are a lot of items, but it gets there in the end.
• The only Mac OS drives that are likely to have large amounts of spaced used by invisible files are startup disks, which have a whole lot of stuff hidden so it doesn't get deleted or futzed with. This issue is not likely to be anything to do with invisible files or folders unless someone has specifically added some or unless it is a startup disk, i.e., one on which you have installed Mac OS X.
• The Disk Permissions that can be checked and repaired by Disk Utility only apply to startup disks and only to certain items. Items on non-startup disks have permissions, but Disk Utility doesn't fix them if they're wrong because it has no way of knowing what's right. Incorrect file permissions can be changed in the Get Info window in Finder. Groups of files can be selected and changed at once using the same technique described above to get the size of multiple items. Files on external disks can also have their permissions ignored if necessary by checking a box in the Get Info window for the disk.
• Erasing Free Space will not change the amount of free space on your disk. What that does is to write zeros to the areas on disk that are already marked as free space, but may contain deleted files that could potentially be recovered. This is because deleting files just marks the space as free to use again; the actual file will still be there until that space is reused. Use the 'Erase Free Space' function for making sure that your deleted files can't be recovered. Emptying Trash securely has the same effect, but takes much longer than emptying it normally so I don't bother with it unless it's something sensitive.
posted by mewsic at 11:03 PM on February 25, 2015 [1 favorite]
Apart from that, I find it's rarely worth the time to work out what's wrong with stuff. It's generally much faster to just nuke from low orbit and start afresh. In this case, I would copy all my files to another drive, erase the external and then copy the files back.
Unless you need to use this drive on a Windows machine there is no reason to leave it formatted as ExFat. If it is only going to be used on a Mac then reformat it as 'Mac OS Extended (journaled)'. As per the first image in Mo Nickels' link, make sure you select the disk rather than the volume when you erase/reformat.
A few things perhaps worth noting:
• It is rare under normal use and even when troubleshooting to need to use the Terminal. There are almost always other ways to achieve the same outcome. There is nothing wrong with using it either if you're comfortable with it, but it is unforgiving - a simple typo can erase your hard disk or render it incapable of booting. I use it when I have to, but it's not often.
• You don't have to manually add up the sizes of all the folders on your drive. Just select them all and click on the File menu. If you then hold the option (alt) key down, you will notice that 'Get Info' changes to "Show Inspector'. If you click on that, Finder will display a window that will add them up for you. Sometimes this can take a few minutes if there are a lot of items, but it gets there in the end.
• The only Mac OS drives that are likely to have large amounts of spaced used by invisible files are startup disks, which have a whole lot of stuff hidden so it doesn't get deleted or futzed with. This issue is not likely to be anything to do with invisible files or folders unless someone has specifically added some or unless it is a startup disk, i.e., one on which you have installed Mac OS X.
• The Disk Permissions that can be checked and repaired by Disk Utility only apply to startup disks and only to certain items. Items on non-startup disks have permissions, but Disk Utility doesn't fix them if they're wrong because it has no way of knowing what's right. Incorrect file permissions can be changed in the Get Info window in Finder. Groups of files can be selected and changed at once using the same technique described above to get the size of multiple items. Files on external disks can also have their permissions ignored if necessary by checking a box in the Get Info window for the disk.
• Erasing Free Space will not change the amount of free space on your disk. What that does is to write zeros to the areas on disk that are already marked as free space, but may contain deleted files that could potentially be recovered. This is because deleting files just marks the space as free to use again; the actual file will still be there until that space is reused. Use the 'Erase Free Space' function for making sure that your deleted files can't be recovered. Emptying Trash securely has the same effect, but takes much longer than emptying it normally so I don't bother with it unless it's something sensitive.
posted by mewsic at 11:03 PM on February 25, 2015 [1 favorite]
« Older Am I getting screwed on my house purchase? | Creating a website and keeping my identity secret Newer »
This thread is closed to new comments.
posted by pompomtom at 2:14 PM on February 25, 2015 [1 favorite]