Still hoping against hope...
June 4, 2010 6:41 PM Subscribe
How is the metadata stored for image files, and what is involved in recovering it after drive failure?
Okay, I would like to revisit a question I asked about 6 weeks ago. I didn't get the response I wanted (half-joking!).
A summary of the question (here) would run something like this:
Files which "disappeared" from WD external hard drive that was not ejected (mounted) before PowerBook crashed are recoverable with data recovery software, but metadata is not. Can anything be done to retrieve the metadata?
I would now add: Or can anybody speak to the idea that the metadata is likely still there but somehow hidden/inaccessible?
Let's, for the time being, assume I'm not so interested in the iTunes files (mostly true -- I'll get over it), and that it's really photos-with-correct-metadata-attached that I'm agonizing about.
I tried DiscWarrior and it did the same as Filesalvage -- it found the files easily enough, but came back without the original metadata.
I tried finding something like chkdsk for Mac, but came up empty -- it's been some weeks so I can't remember details, but I do remember it seemed to be a deadend street. I did try Disk Utility, but wasn't sure what I was doing and so don't know if I have exhausted its possibilities. I was able to connect to a friend's PC, and run chkdsk. I ran it with both "Automatically fix file system errors" and "Scan for and attempt recovery of bad sectors" selected. This took a few hours, but when it was done, all it said was that it was done, with a choice to click "OK". I was expecting it to come back with some data or report, but there was none. When I plugged the drive into my Mac, there was no change. Perhaps I need to run the data recovery again?
Finally, I still would love to know WHY and/or HOW this can happen "in this day and age" (as per my last post to the original question). A) I'd be very interested to know what is going on in the mounting process (or lack thereof) that would cause a failure like this (or, I guess more specifically, why would the metadata be unrecoverable, but the images be easily recovered -- is the metadata stored separately from the actual file?). And B) I assume there must be something I'm not seeing here as to why this is still even a possibility, since it should have been one of the first things to be designed out of newer drives. I guess I feel that having a better understanding of the above might help me to "come to terms" with what has happened and move on. Right now I can't let go of the idea that the metadata is there, but I just can't get to it somehow.
* Final note: the drive appears to be otherwise functioning fine. I haven't saved anything to it for fear of overwriting, but I am fairly (foolishly?) confident that I will be able to use the drive normally once I have either retrieved the data I want or given up.
Any help with any of the above would be most appreciated, including suggestions for what other forums I should try taking my question(s) to.
Thanks in advance.
Okay, I would like to revisit a question I asked about 6 weeks ago. I didn't get the response I wanted (half-joking!).
A summary of the question (here) would run something like this:
Files which "disappeared" from WD external hard drive that was not ejected (mounted) before PowerBook crashed are recoverable with data recovery software, but metadata is not. Can anything be done to retrieve the metadata?
I would now add: Or can anybody speak to the idea that the metadata is likely still there but somehow hidden/inaccessible?
Let's, for the time being, assume I'm not so interested in the iTunes files (mostly true -- I'll get over it), and that it's really photos-with-correct-metadata-attached that I'm agonizing about.
I tried DiscWarrior and it did the same as Filesalvage -- it found the files easily enough, but came back without the original metadata.
I tried finding something like chkdsk for Mac, but came up empty -- it's been some weeks so I can't remember details, but I do remember it seemed to be a deadend street. I did try Disk Utility, but wasn't sure what I was doing and so don't know if I have exhausted its possibilities. I was able to connect to a friend's PC, and run chkdsk. I ran it with both "Automatically fix file system errors" and "Scan for and attempt recovery of bad sectors" selected. This took a few hours, but when it was done, all it said was that it was done, with a choice to click "OK". I was expecting it to come back with some data or report, but there was none. When I plugged the drive into my Mac, there was no change. Perhaps I need to run the data recovery again?
Finally, I still would love to know WHY and/or HOW this can happen "in this day and age" (as per my last post to the original question). A) I'd be very interested to know what is going on in the mounting process (or lack thereof) that would cause a failure like this (or, I guess more specifically, why would the metadata be unrecoverable, but the images be easily recovered -- is the metadata stored separately from the actual file?). And B) I assume there must be something I'm not seeing here as to why this is still even a possibility, since it should have been one of the first things to be designed out of newer drives. I guess I feel that having a better understanding of the above might help me to "come to terms" with what has happened and move on. Right now I can't let go of the idea that the metadata is there, but I just can't get to it somehow.
* Final note: the drive appears to be otherwise functioning fine. I haven't saved anything to it for fear of overwriting, but I am fairly (foolishly?) confident that I will be able to use the drive normally once I have either retrieved the data I want or given up.
Any help with any of the above would be most appreciated, including suggestions for what other forums I should try taking my question(s) to.
Thanks in advance.
Response by poster: BP: I guess it's EXIF I'm after. Ultimately it is date/time the pics were taken (I have a variety of images that are very similar and only really useful if I know which was taken when). Camera settings, thumbnail, etc are ultimately not essential to me.
posted by segatakai at 6:56 PM on June 4, 2010
posted by segatakai at 6:56 PM on June 4, 2010
Finally, I still would love to know WHY and/or HOW this can happen "in this day and age" (as per my last post to the original question).
It has nothing to do with the drive. It has everything to do with the filesystem.
The filesystem is the software-defined structure that is used to save and locate files.
Most external drives comes pre-formatted with the Windows/DOS FAT filesystem. They do this so that yokels can plug in their drives and have them work on everything from Windows 95 to OSX to Ubuntu to the embedded OS of a toaster. This is because FAT is such a trivially simple and ubiquitous filesystem that every OS on the planet has a driver that'll read and write to it.
The problem with FAT, though, is that it uses a file allocation table (hence "FAT") that tracks where stuff is saved. This table is kept at a specific place on the harddrive so that it can be found without needing some other descriptor to keep track of it. In normal operation, this isn't a problem.
However, if you interrupt a write to the allocation table, it's possible to corrupt it. When you corrupt the table, the entire drive appears "blank" because the locations for all of the files on it have been corrupted and lost.
It's like if you had every page of every book in a library torn out, and put in giant stacks on the floor. Then, you have a set of index cards that tell you Moby Dick is on pages 1233-3302, 44589-45391, and 10023-10220. If you lost those index cards, your pile of could-be-books becomes just a pile of jumbled pages.
And, yes, program-specific metadata is usually stored separately from the data itself. Except for data files that specifically include a method of storing metadata (like JPEGs and EXIF).
Solution: When you get an external drive, reformat it with a better filesystem. On OSX, that'd be HFS+ (I think). On Windows, NTFS. On linux, ext3 (debatable).
posted by Netzapper at 7:04 PM on June 4, 2010 [2 favorites]
It has nothing to do with the drive. It has everything to do with the filesystem.
The filesystem is the software-defined structure that is used to save and locate files.
Most external drives comes pre-formatted with the Windows/DOS FAT filesystem. They do this so that yokels can plug in their drives and have them work on everything from Windows 95 to OSX to Ubuntu to the embedded OS of a toaster. This is because FAT is such a trivially simple and ubiquitous filesystem that every OS on the planet has a driver that'll read and write to it.
The problem with FAT, though, is that it uses a file allocation table (hence "FAT") that tracks where stuff is saved. This table is kept at a specific place on the harddrive so that it can be found without needing some other descriptor to keep track of it. In normal operation, this isn't a problem.
However, if you interrupt a write to the allocation table, it's possible to corrupt it. When you corrupt the table, the entire drive appears "blank" because the locations for all of the files on it have been corrupted and lost.
It's like if you had every page of every book in a library torn out, and put in giant stacks on the floor. Then, you have a set of index cards that tell you Moby Dick is on pages 1233-3302, 44589-45391, and 10023-10220. If you lost those index cards, your pile of could-be-books becomes just a pile of jumbled pages.
And, yes, program-specific metadata is usually stored separately from the data itself. Except for data files that specifically include a method of storing metadata (like JPEGs and EXIF).
Solution: When you get an external drive, reformat it with a better filesystem. On OSX, that'd be HFS+ (I think). On Windows, NTFS. On linux, ext3 (debatable).
posted by Netzapper at 7:04 PM on June 4, 2010 [2 favorites]
Best answer: Are you sure the EXIF data isn't still in the files?
It seems to me like you're confusing system-level metadata (icons, modification dates, maybe Finder labels) with the text that's embedded right into many image files by the camera and is also often called metadata (the camera type, the date the photo was taken, various camera settings).
That second type of metadata is part and parcel of the file, not something stored in some secret place on your hard drive away from the file.
posted by bcwinters at 7:16 PM on June 4, 2010 [1 favorite]
It seems to me like you're confusing system-level metadata (icons, modification dates, maybe Finder labels) with the text that's embedded right into many image files by the camera and is also often called metadata (the camera type, the date the photo was taken, various camera settings).
That second type of metadata is part and parcel of the file, not something stored in some secret place on your hard drive away from the file.
posted by bcwinters at 7:16 PM on June 4, 2010 [1 favorite]
Are you using iPhoto? In iPhoto 9 at least, there's the "special" iPhoto folder with all the photos, metadata, etc in it. Use the left-click/ctrl-click on the special folder and select view contents. Inside the photos there's an "originals" folder that then will have files by year/month/day (that you imported the folders) in folders. Each of these files will have the original EXIF data from when you took the photo if your camera supports it.
You could attempt to rebuild the database in iPhoto by holding the Shift+Command key while starting iPhoto.
But I'm not sure if you're using iPhoto or not.
posted by birdherder at 7:20 PM on June 4, 2010
You could attempt to rebuild the database in iPhoto by holding the Shift+Command key while starting iPhoto.
But I'm not sure if you're using iPhoto or not.
posted by birdherder at 7:20 PM on June 4, 2010
Response by poster: When you get an external drive, reformat it with a better filesystem.
Duly filed away for future reference. Thank you.
Are you sure the EXIF data isn't still in the files?
No, I'm not sure. I guess that's what I'm asking.
... how do I access EXIF data?
Are you using iPhoto?
I have iPhoto '08 (version 7.0.1), though I haven't been using it. I tried once to open the salvaged files in iPhoto, but found that all photos were dated the day of the salvage. I will try again now.
posted by segatakai at 7:39 PM on June 4, 2010
Duly filed away for future reference. Thank you.
Are you sure the EXIF data isn't still in the files?
No, I'm not sure. I guess that's what I'm asking.
... how do I access EXIF data?
Are you using iPhoto?
I have iPhoto '08 (version 7.0.1), though I haven't been using it. I tried once to open the salvaged files in iPhoto, but found that all photos were dated the day of the salvage. I will try again now.
posted by segatakai at 7:39 PM on June 4, 2010
FWIW, 'fsck' is the CHKDSK equivalent on OS X (and most Unix-alike OSs). Use with care; it's not as user-friendly as CHKDSK. Disk Utility's 'verify' function is the GUI equivalent (and probably uses the exact same code or calls the CLI fsck underneath; I've never bothered to look).
A lot of the answer, however, is going to depend on how the disk was formatted - as HFS+, FAT32, NTFS, etc. Most large external drives seem to be pre-formatted as NTFS these days (at least, the 3 - 160G, 320G, & 500G - I've bought in the last year or two have been). You probably reformatted it as HFS+ when you first installed it though, unless you've installed an NTFS filesystem driver (e.g. MacFUSE / NTFS-3G) at some point.
But yeah - IIRC, the attributes stored in the system-level metadata (e.g. image dimensions, application type, etc) are stored when the file is saved. This will probably be lost when the directory entry is rebuilt with undelete tools, but some of it (e.g. image size) is still stored in the file metadata in the file header. Opening and closing the file with the usual viewer (e.g. iPhoto, Preview) might restore that part of it; re-saving it almost certainly will. But this is something where you need to start considering how the drive is formatted, exactly what went on to cause the loss, etc, etc, before you start trying things that may write to the disk.
(Also - backups! I should really do one one day… ;-)
posted by Pinback at 7:52 PM on June 4, 2010
A lot of the answer, however, is going to depend on how the disk was formatted - as HFS+, FAT32, NTFS, etc. Most large external drives seem to be pre-formatted as NTFS these days (at least, the 3 - 160G, 320G, & 500G - I've bought in the last year or two have been). You probably reformatted it as HFS+ when you first installed it though, unless you've installed an NTFS filesystem driver (e.g. MacFUSE / NTFS-3G) at some point.
But yeah - IIRC, the attributes stored in the system-level metadata (e.g. image dimensions, application type, etc) are stored when the file is saved. This will probably be lost when the directory entry is rebuilt with undelete tools, but some of it (e.g. image size) is still stored in the file metadata in the file header. Opening and closing the file with the usual viewer (e.g. iPhoto, Preview) might restore that part of it; re-saving it almost certainly will. But this is something where you need to start considering how the drive is formatted, exactly what went on to cause the loss, etc, etc, before you start trying things that may write to the disk.
(Also - backups! I should really do one one day… ;-)
posted by Pinback at 7:52 PM on June 4, 2010
Ah, that's right - iPhoto tags files with their import date, not the date from the EXIF data. There is a way to update it, but I forget what it was - possibly this?
posted by Pinback at 7:55 PM on June 4, 2010
posted by Pinback at 7:55 PM on June 4, 2010
I just opened one of my iPhoto-imported photos, and all the EXIF data was still in the original JPEG file. In my case this includes a time stamp. You can check yours by opening them in Preview, opening Tools → Inspector, and selecting the Exif tab within the More Info tab. If the EXIF data is there—and it's hard to believe it wouldn't be, unless your camera didn't store it to begin with—then it's just a matter of getting iPhoto to reload it by trying tricks like Pinback and birdherder have suggested. I don't think it's possible to make iPhoto overwrite the EXIF data, but it wouldn't hurt make a backup of those files now anyway.
posted by serathen at 8:52 PM on June 4, 2010
posted by serathen at 8:52 PM on June 4, 2010
Response by poster: Uh... I'm very embarrassed. Indeed the EXIF data is attached to the individual file. In iPhoto, *simply* by checking "Show Photo Info" from the Photos menu, I get the Original Date (taken). I'm extremely relieved that this info is indeed intact, but probably more embarrassed that it was there all along, and so easy to find.
I have yet to try sorting the photos by the Original Date, which is what I will ultimately want to do. It doesn't seem, at first glance, that there is an easy option for doing so (again, I'm not at all familiar with iPhoto). I guess I will try the Shift/Cmd restart, and/or the Joe's Scripts...
Thanks everyone for your help; if anyone has any more to add, or thoughts about sorting by Original Date, please continue to leave them. I will report back once I've tried a few things here.
posted by segatakai at 9:40 PM on June 4, 2010
I have yet to try sorting the photos by the Original Date, which is what I will ultimately want to do. It doesn't seem, at first glance, that there is an easy option for doing so (again, I'm not at all familiar with iPhoto). I guess I will try the Shift/Cmd restart, and/or the Joe's Scripts...
Thanks everyone for your help; if anyone has any more to add, or thoughts about sorting by Original Date, please continue to leave them. I will report back once I've tried a few things here.
posted by segatakai at 9:40 PM on June 4, 2010
This thread is closed to new comments.
posted by Blazecock Pileon at 6:44 PM on June 4, 2010