Help me retrieve my data
September 3, 2011 5:52 PM Subscribe
Help! Repartitioning failed, can't see the contents of my partition, and it has all sorts of important, un-backed-up things on it.
I have an external hard drive with 3 partitions, two of which are old Windows installs from when it was an internal hard drive, and one is a bigger partition with data on it. Important, un-backed-up data, of course.
I decided to remove the Windows partitions and resize the big partition to make some room. I deleted one of the partitions and told GParted to resize the data partition into the space to the left.
I left it over night and saw, staring at me, this.
It reports:
Also, I don't really know how to interpret what testdisk or ntfsresize is saying. Something went wrong while repartitioning, but at what point? Did it start moving my data around and then fail? Is all my data where it was, but the partition table's scrambled? Is my data scrambled? How can I work out what is actually physically on the disk?
I'd appreciate help with learning
a) what exactly went wrong,
b) why it went wrong,
c) what shape my data is in, and
d) what I should do that can recover data without making it worse
I have an external hard drive with 3 partitions, two of which are old Windows installs from when it was an internal hard drive, and one is a bigger partition with data on it. Important, un-backed-up data, of course.
I decided to remove the Windows partitions and resize the big partition to make some room. I deleted one of the partitions and told GParted to resize the data partition into the space to the left.
I left it over night and saw, staring at me, this.
It reports:
ntfsresize v2.0.0 (libntfs 10:0:0)I poked around in TestDisk briefly (which I've used to successfully recover partition tables in other situations, and PhotoRec to pull the files out when that hasn't worked) and it reports this:
Device name : /dev/sdd3
NTFS volume version: 3.1
Cluster size : 4096 bytes
Current volume size: 261728571904 bytes (261729 MB)
Current device size: 261728409600 bytes (261729 MB)
ERROR: Current NTFS volume size is bigger than the
device size!
Corrupt partition table or incorrect device partitioning?
Unable to read the contents of this file system!
Because of this some operations may be unavailable.
Disk /dev/sdd - 320 GB / 298 GiB - CHS 38913 255 63"Quick search" shows:
Current partition structure:
Partition Start End Size in sectors
1 * HPFS - NTFS 0 32 33 2889 254 63 46425802
Error: size boot_sector 511188619 > partition 511188300
Invalid NTFS or EXFAT boot
3 P HPFS - NTFS 7093 0 1 38912 254 63 511188300
3 P HPFS - NTFS 7093 0 1 38912 254 63 511188300
Disk /dev/sdd - 320 GB / 298 GiB - CHS 38914 255 63And then
Partition Start End Size in sectors
L HPFS - NTFS 0 32 33 2889 254 63 46425802
* HPFS - NTFS 2890 0 1 34710 254 63 511204365
Disk /dev/sdd - 320 GB / 298 GiB - CHS 38914 255 63I get this when it tries to mount:
Partition Start End Size in sectors
1 E extended LBA 0 32 1 2889 254 63 46425834
2 * HPFS - NTFS 2890 0 1 34710 254 63 511204365
5 L HPFS - NTFS 0 32 33 2889 254 63 46425802
Error mounting: mount exited with exit code 12: Failed to read last sector (511188618): Invalid argumentWindows sees an unformatted, unlabeled partition in the same place as GParted sees the Data partition. I looked at it with diskpart and Windows Explorer; I'm not sure what other tools there are on Windows for diagnosis.
HINTS: Either the volume is a RAID/LDM but it wasn't setup yet,
or it was not setup correctly (e.g. by not using mdadm --build ...),
or a wrong device is tried to be mounted,
or the partition table is corrupt (partition is smaller than NTFS),
or the NTFS boot sector is corrupt (NTFS size is not valid).
Failed to mount '/dev/sdd3': Invalid argument
The device '/dev/sdd3' doesn't seem to have a valid NTFS.
Maybe the wrong device is used? Or the whole disk instead of a
partition (e.g. /dev/sda, not /dev/sda1)? Or the other way around?
Also, I don't really know how to interpret what testdisk or ntfsresize is saying. Something went wrong while repartitioning, but at what point? Did it start moving my data around and then fail? Is all my data where it was, but the partition table's scrambled? Is my data scrambled? How can I work out what is actually physically on the disk?
I'd appreciate help with learning
a) what exactly went wrong,
b) why it went wrong,
c) what shape my data is in, and
d) what I should do that can recover data without making it worse
a) what exactly went wrong,
A GUI-based tool lulled you into a false sense of security with its comforting colored boxes, allowing you to believe that you trust it to do a dangerous thing without knowing exactly what was going on, and then screwed up your disk for you.
Expanding partitions "to the left" is the most dangerous thing you can possibly do to them, double dippity dog dangerous if you don't have a backup.
b) why it went wrong,
Because you have not yet learned that important and un-backed-up are attributes that should never apply simultaneously to data.
c) what shape my data is in, and
Unknowable.
d) what I should do that can recover data without making it worse
Spend some money. Either on professional data recovery as paulsc suggests, or on another drive of equal or preferably larger capacity onto which you make a block-for-block copy of your messed up drive using Gnu ddrescue before you touch it with any other tool.
posted by flabdablet at 6:30 PM on September 3, 2011 [2 favorites]
A GUI-based tool lulled you into a false sense of security with its comforting colored boxes, allowing you to believe that you trust it to do a dangerous thing without knowing exactly what was going on, and then screwed up your disk for you.
Expanding partitions "to the left" is the most dangerous thing you can possibly do to them, double dippity dog dangerous if you don't have a backup.
b) why it went wrong,
Because you have not yet learned that important and un-backed-up are attributes that should never apply simultaneously to data.
c) what shape my data is in, and
Unknowable.
d) what I should do that can recover data without making it worse
Spend some money. Either on professional data recovery as paulsc suggests, or on another drive of equal or preferably larger capacity onto which you make a block-for-block copy of your messed up drive using Gnu ddrescue before you touch it with any other tool.
posted by flabdablet at 6:30 PM on September 3, 2011 [2 favorites]
It looks like your boot sector/partition table is hosed. Paulsc is, of course, quite right that if it's really important, your best bet is to take it to a professional. (cf.
I think at this point before I did anything else, I'd try more or less exactly what flabdablet suggests. Go get three or four 500 GB external drives, and ddrescue the whole drive to two of them. Then I'd try ddrescuing just the data partition onto a fresh drive from one of my copies. If that didn't get a working partition, I'd try to work on fixing the boot sector and/or partition table on one of the copies, leaving the second as a backup. Maybe try ntfsfix? Although perhaps a chkdsk from windows would amount to the same thing.
I'll spare you a continuation of the lecture, except to add,
posted by ob1quixote at 6:49 PM on September 3, 2011
Don't Panic! How to Recover Data From a Dead Hard Drive, Popular Mechanics, 2008)
I think at this point before I did anything else, I'd try more or less exactly what flabdablet suggests. Go get three or four 500 GB external drives, and ddrescue the whole drive to two of them. Then I'd try ddrescuing just the data partition onto a fresh drive from one of my copies. If that didn't get a working partition, I'd try to work on fixing the boot sector and/or partition table on one of the copies, leaving the second as a backup. Maybe try ntfsfix? Although perhaps a chkdsk from windows would amount to the same thing.
I'll spare you a continuation of the lecture, except to add,
Before testing or reconfiguring, always mount a scratch monkey.
posted by ob1quixote at 6:49 PM on September 3, 2011
Honestly, if it's an external drive, I don't think you're in too much pain. You've not got a dead hard-drive in the traditional sense. A friend had a similar problem, they managed to screw up their partitions and came running to me.
I ended up using R-Studio first I made an image of the drive using it (essentially doing the ddrescue command flabdablet said) then I asked it to scan the drive for files.
His was so screwed up that the names couldn't be recovered, just the file data-- so once the big recovery was done-- I just did a file-in-files approach to find some fairly uncommon words that he thought he'd used to find those important files.
So install the program (not onto the problem drive). Plug in and create an image of the problem drive, unplug the problem drive and store it safely.
Then play with the scan functions within the program using the image that was created.
posted by Static Vagabond at 6:58 PM on September 3, 2011
I ended up using R-Studio first I made an image of the drive using it (essentially doing the ddrescue command flabdablet said) then I asked it to scan the drive for files.
His was so screwed up that the names couldn't be recovered, just the file data-- so once the big recovery was done-- I just did a file-in-files approach to find some fairly uncommon words that he thought he'd used to find those important files.
So install the program (not onto the problem drive). Plug in and create an image of the problem drive, unplug the problem drive and store it safely.
Then play with the scan functions within the program using the image that was created.
posted by Static Vagabond at 6:58 PM on September 3, 2011
To work out in detail what went wrong, it would be very helpful to know the state of your partition table before you went at it with gparted. How much do you remember?
posted by flabdablet at 7:03 PM on September 3, 2011
posted by flabdablet at 7:03 PM on September 3, 2011
Response by poster: Thanks, ob1quixote. Lecturing at this point is not really much help, as I can lecture myself perfectly well, thankyouverymuch. I even work at a tech help desk that deals with computer issues (though not data recovery, but we get people in with similar problems, the answer being "go find an expert, kthxbye").
The partitions looked much like they do in the screenshot linked above, but with another ntfs partition in the unpartitioned space in the middle (which I deleted intentionally).
I've used PhotoRec to pull things off drives with totally trashed partition tables before, like Static Vagabond mentioned; files intact, file structure not so much. 80% of what I care about are photos, and they were just in folders based on date, which I can reconstruct easily enough. It's the various things like programming projects which will be almost entirely befuddled if I can't rescue the file structure.
Off to go ddrescue.
posted by BungaDunga at 7:11 PM on September 3, 2011
The partitions looked much like they do in the screenshot linked above, but with another ntfs partition in the unpartitioned space in the middle (which I deleted intentionally).
I've used PhotoRec to pull things off drives with totally trashed partition tables before, like Static Vagabond mentioned; files intact, file structure not so much. 80% of what I care about are photos, and they were just in folders based on date, which I can reconstruct easily enough. It's the various things like programming projects which will be almost entirely befuddled if I can't rescue the file structure.
Off to go ddrescue.
posted by BungaDunga at 7:11 PM on September 3, 2011
Best answer: The lecture is of course obligatory in any situation such as this, and you've taken your lumps at least semi-graciously and said you're about to make a backup so you're clearly somebody with both Clue and Hope :-)
So here's what I think has happened:
This part is informed guesswork: to extend an NTFS partition leftward, I would expect gparted to do the following: (a) resize the NTFS filesytem to its smallest possible size, thereby consolidating all its blocks at the beginning of its partition (b) alter the partition table to make the partition cover the formerly unoccupied space to the left, thereby causing the shrunken NTFS filesystem to appear at an offset within the partition (c) copy the filesystem blocks from that offset to offset 0, starting at the left so that blocks that get overwritten have already been copied by the time the overwrite happens (d) resize the NTFS filesystem to fill the new partition.
It looks to me as if gparted didn't get as far as (b) because it failed at (a). If that's right, it's good because it means that the potentially very destructive (c) will not have happened.
So why would (a) fail, and can the result be fixed?
There's an unallocated block of 1MiB at the start of the disk, which says to me that it was originally set up by Windows Vista or Windows 7. The partitioner that comes with these systems pays no attention to the old conventions about making partitions start on "track" boundaries and end on "cylinder" boundaries, on the perfectly reasonable grounds that (a) the head and sector counts are completely meaningless on modern drives and (b) as more drives become available with internal 4KiB sectoring, forcing partitions to start on a non-4KiB boundary is totally counterproductive. Instead, Vista/7 starts partitions on 1MiB boundaries and gives them whole-MiB sizes.
The Linux kernel, unless told otherwise, will assume a disk geometry of 63 sectors per track and 255 heads. This gives rise to a logical cylinder size of 63 * 255 * 512 bytes = 8225280 bytes, which is not a multiple of 1MiB (1048576 bytes). Using command-line Linux partitioning tools against Vista-formatted drives, you'll frequently see warnings about partitions not ending on cylinder boundaries.
I suspect that gparted has silently "fixed" those "errors", and that as a result, /dev/sdd3 is now a little smaller than Vista/7 left it.
The reported device size (261728409600 bytes) is exactly 31820 * 8225280 bytes, but it's not an exact muliple of 1MiB. If we round it up to the next such multiple, we get 261728763904 bytes.
The reported filesystem size (261728571904 bytes) is 162304 bytes bigger than the partition's present size, but 192000 bytes smaller than the rounded-up size. So I'm thinking that all that will need doing is fixing the partition size to make it a multiple of 1MiB again, and the filesystem should become accessible again as long as no data recovery tool has "fixed" it too much.
The steps I'd use to do that (after making a ddrescue backup, naturally!) are:
1. Use sfdisk --dump /dev/sdd >sdd.parts.old to get an editable version of the present partition table.
2. Edit sdd.parts.old, round up all the partition sizes to the next multiple of 2048 (because 2048 512-byte sectors is 1MiB) and save it as sdd.parts.new.
3. Use sfdisk -S 32 -H 64 -n /dev/sdd <sdd.parts.new to do a dummy run at applying the edited partition table to the disk. Setting sectors per track to 32 and heads to 64 yields a logical cylinder size of 1MiB and keeps both Linux and Windows happy.
4. If the dummy run reported no errors, sfdisk -S 32 -H 64 /dev/sdd <sdd.parts.new to apply the new partitioning and sfdisk --re-read to ask the kernel to adopt it.
If you do this and it works, I'd like to hear about it.
posted by flabdablet at 8:49 PM on September 3, 2011 [7 favorites]
So here's what I think has happened:
This part is informed guesswork: to extend an NTFS partition leftward, I would expect gparted to do the following: (a) resize the NTFS filesytem to its smallest possible size, thereby consolidating all its blocks at the beginning of its partition (b) alter the partition table to make the partition cover the formerly unoccupied space to the left, thereby causing the shrunken NTFS filesystem to appear at an offset within the partition (c) copy the filesystem blocks from that offset to offset 0, starting at the left so that blocks that get overwritten have already been copied by the time the overwrite happens (d) resize the NTFS filesystem to fill the new partition.
It looks to me as if gparted didn't get as far as (b) because it failed at (a). If that's right, it's good because it means that the potentially very destructive (c) will not have happened.
So why would (a) fail, and can the result be fixed?
There's an unallocated block of 1MiB at the start of the disk, which says to me that it was originally set up by Windows Vista or Windows 7. The partitioner that comes with these systems pays no attention to the old conventions about making partitions start on "track" boundaries and end on "cylinder" boundaries, on the perfectly reasonable grounds that (a) the head and sector counts are completely meaningless on modern drives and (b) as more drives become available with internal 4KiB sectoring, forcing partitions to start on a non-4KiB boundary is totally counterproductive. Instead, Vista/7 starts partitions on 1MiB boundaries and gives them whole-MiB sizes.
The Linux kernel, unless told otherwise, will assume a disk geometry of 63 sectors per track and 255 heads. This gives rise to a logical cylinder size of 63 * 255 * 512 bytes = 8225280 bytes, which is not a multiple of 1MiB (1048576 bytes). Using command-line Linux partitioning tools against Vista-formatted drives, you'll frequently see warnings about partitions not ending on cylinder boundaries.
I suspect that gparted has silently "fixed" those "errors", and that as a result, /dev/sdd3 is now a little smaller than Vista/7 left it.
The reported device size (261728409600 bytes) is exactly 31820 * 8225280 bytes, but it's not an exact muliple of 1MiB. If we round it up to the next such multiple, we get 261728763904 bytes.
The reported filesystem size (261728571904 bytes) is 162304 bytes bigger than the partition's present size, but 192000 bytes smaller than the rounded-up size. So I'm thinking that all that will need doing is fixing the partition size to make it a multiple of 1MiB again, and the filesystem should become accessible again as long as no data recovery tool has "fixed" it too much.
The steps I'd use to do that (after making a ddrescue backup, naturally!) are:
1. Use sfdisk --dump /dev/sdd >sdd.parts.old to get an editable version of the present partition table.
2. Edit sdd.parts.old, round up all the partition sizes to the next multiple of 2048 (because 2048 512-byte sectors is 1MiB) and save it as sdd.parts.new.
3. Use sfdisk -S 32 -H 64 -n /dev/sdd <sdd.parts.new to do a dummy run at applying the edited partition table to the disk. Setting sectors per track to 32 and heads to 64 yields a logical cylinder size of 1MiB and keeps both Linux and Windows happy.
4. If the dummy run reported no errors, sfdisk -S 32 -H 64 /dev/sdd <sdd.parts.new to apply the new partitioning and sfdisk --re-read to ask the kernel to adopt it.
If you do this and it works, I'd like to hear about it.
posted by flabdablet at 8:49 PM on September 3, 2011 [7 favorites]
Off to go ddrescue.
That's good. The first thing you should do is image the entire drive onto another drive, then unplug the original and work with the copy.
I've used partimage to image partitions, but in this case you want the entire disk, including the partition table. I've used winimage to image USB sticks with Linux partitions on them. I imagine it would work with a full sized USB HD as well.
hopefully ddrescue will work for you. Good luck.
posted by delmoi at 9:26 PM on September 3, 2011 [1 favorite]
That's good. The first thing you should do is image the entire drive onto another drive, then unplug the original and work with the copy.
I've used partimage to image partitions, but in this case you want the entire disk, including the partition table. I've used winimage to image USB sticks with Linux partitions on them. I imagine it would work with a full sized USB HD as well.
hopefully ddrescue will work for you. Good luck.
posted by delmoi at 9:26 PM on September 3, 2011 [1 favorite]
The main reason ddrescue is the right tool for this particular imaging job is that it makes no attempt at all to interpret the data it's copying. It's not "smart"; it doesn't skip unused sectors as partimage or ntfsclone would do, and if what you're copying has damage to the partition tables and/or filesystems this is exactly the behavior you want. The downside is that copying a big disk takes a long time.
It does the vast majority of its copying in reasonable-sized chunks, which means that disk copying runs about as fast as the underlying data transport will let it. And what it is smart about is dealing with read errors on the source disk, and resuming interrupted copying sessions.
I wouldn't use anything else to back up a damaged and/or failing drive.
posted by flabdablet at 11:20 PM on September 3, 2011
It does the vast majority of its copying in reasonable-sized chunks, which means that disk copying runs about as fast as the underlying data transport will let it. And what it is smart about is dealing with read errors on the source disk, and resuming interrupted copying sessions.
I wouldn't use anything else to back up a damaged and/or failing drive.
posted by flabdablet at 11:20 PM on September 3, 2011
A few years ago, I got impatient during an alteration of partitions and cancelled the process. Of course, I lost all of my data. On running chkdsk, though, all of the data was recovered as .chk files. Using one of the apps from this site, I managed to get most of the data back, or at least named as [randomcharacters].jpg or [randomcharacters].mp3.
I don't know if this is an option for you, as chkdsk will write the files to the root of the damaged drive, meaning that you might overwrite your own data. I just figured I'd throw it out there.
posted by Solomon at 1:55 AM on September 4, 2011
I don't know if this is an option for you, as chkdsk will write the files to the root of the damaged drive, meaning that you might overwrite your own data. I just figured I'd throw it out there.
posted by Solomon at 1:55 AM on September 4, 2011
By the way, the thing that gives me confidence that your filesystem is basically OK is that ntfsresize does in fact report a completely plausible size for it. That means that it must have found the MFT, which means that the MFT has not been overwritten. I have also seen both Windows and Linux implementations of NTFS refuse to mount filesystems whose last cluster is not readable, once due to a bad sector and several times due to non-inclusion in a newly resized partition (my error in each case). If I recall correctly, NTFS keeps a backup of its boot sector in the last filesystem sector.
In your position I would not even begin to consider running automated filesystem recovery tools against what you have until I'd (a) backed it up and (b) made sure that the partition containing the non-mountable NTFS filesystem was in fact big enough to contain that filesystem.
posted by flabdablet at 2:06 AM on September 4, 2011
In your position I would not even begin to consider running automated filesystem recovery tools against what you have until I'd (a) backed it up and (b) made sure that the partition containing the non-mountable NTFS filesystem was in fact big enough to contain that filesystem.
posted by flabdablet at 2:06 AM on September 4, 2011
Response by poster: That... worked, flabdablet. All my stuff present and accounted for. Damn. I'm very impressed! Thank you very much for the help- you've saved me a lot of time and possibly money.
And now I'm going to go back it all up one more time.
posted by BungaDunga at 1:04 PM on September 5, 2011 [2 favorites]
And now I'm going to go back it all up one more time.
posted by BungaDunga at 1:04 PM on September 5, 2011 [2 favorites]
Always nice to get a good result. Well done you.
posted by flabdablet at 4:20 PM on September 5, 2011
posted by flabdablet at 4:20 PM on September 5, 2011
This thread is closed to new comments.
If your data is really important, power down that drive immediately, and send it off to a data recovery service. As an inexperienced person poking around with low level file system tools, you stand a 99% chance of making what data might still be recovered at this point, unrecoverable. Be prepared to pay 3 figures in USD for data recovery, and for 90% file recovery to be the top end of what you'll get.
What happened, I think, was that your NTFS Master File Table (MFT) was expanded beyond the physical device limits (possibly due to Logical Block Address - LBA - errors from your drive). Such errors can happen when an older drive with accumulated media errors is prevented from substituting spare sectors for the erroring sectors from its factory reserved spares pool, sometimes by routines in a repartitioning program. The new information written to the boot sector doesn't match the device limits reporting from BIOS when the drive is auto-dectected, so Windows thinks that NTFS MFT can't possibly be correct, and suspects that the partition size is now wrong. Without a correct MFT, Windows has no way of making sense of an NTFS file system, where individual adjacent sectors may contain information from different files (random disk writes).
Professional tools and personnel try to reconstruct the file system mostly ignoring the incorrect MFT - they try looking for an older, mostly correct MFT in the old partition space, to see when sectors were written, and/or try to assemble sectors into files or file fragments worth having, by a variety of data analysis and repair techniques.
posted by paulsc at 6:16 PM on September 3, 2011