Photo Storage, long term formats and other questions
July 19, 2015 10:39 PM Subscribe
I have about 25 GB and growing size of photos. These are a mix of photo shoots, family vacations, parties, festive occasions and general-on-the-spur photos. While I am somewhat clear on how to keep them named, separate etc, there are a host of things I am struggling to define and create a workflow for formats, storage, duplicates, software etc.
1. Should I consider DNG or stick with CR2 (I shoot Canon)? Pros for DNG include an open source format, no sidecar files while CR2 means SW in future will be supported (CaptureOne, for example, doesn't seem to support DNGs)
2. While Lightroom+Photoshop is my current preference, I am worried about Adobe's policies to some extent. Upgrading to Camera RAW 8, for example, means that I need to upgrade Lightroom as well. But I don't expect to buy many cameras - will stick to the 600D and 5D Mark series for now. Do I need to be concerned?
3. How do I backup RAW files? I have a local backup on an external hard drive, but I have been thinking about an online backup like crashplan. But uploading so much RAW files will take infinite time. I am not in a country where seeded backup is available.
4. There are duplicates, especially of early years, where I was not regular with my naming conventions. How do I identify these duplicates with reasonable surety?
5. I don't think I am efficient with my usage of Lightroom. What resources would you recommend to learn on how to best use it?
I have another question on lighting in tricky conditions and external flash usage, but that is for another day.
1. Should I consider DNG or stick with CR2 (I shoot Canon)? Pros for DNG include an open source format, no sidecar files while CR2 means SW in future will be supported (CaptureOne, for example, doesn't seem to support DNGs)
2. While Lightroom+Photoshop is my current preference, I am worried about Adobe's policies to some extent. Upgrading to Camera RAW 8, for example, means that I need to upgrade Lightroom as well. But I don't expect to buy many cameras - will stick to the 600D and 5D Mark series for now. Do I need to be concerned?
3. How do I backup RAW files? I have a local backup on an external hard drive, but I have been thinking about an online backup like crashplan. But uploading so much RAW files will take infinite time. I am not in a country where seeded backup is available.
4. There are duplicates, especially of early years, where I was not regular with my naming conventions. How do I identify these duplicates with reasonable surety?
5. I don't think I am efficient with my usage of Lightroom. What resources would you recommend to learn on how to best use it?
I have another question on lighting in tricky conditions and external flash usage, but that is for another day.
As for #3: I'm a huge fan and long time user of crashplan. The client will silently backup in the background and resume on its own. So it takes a while. No big deal. Just leave the computer running and at some point it's done.
posted by hz37 at 12:44 AM on July 20, 2015 [1 favorite]
posted by hz37 at 12:44 AM on July 20, 2015 [1 favorite]
Response by poster: @Freedomboy - the template folder set is a good one, so will try to implement in Lightroom itself when importing.
@hz37: Crashplan took 4 days to upload 2 GB of data, so not sure how that will impact my internet bandwidth as well. But will try out again.
posted by theobserver at 4:09 AM on July 20, 2015
@hz37: Crashplan took 4 days to upload 2 GB of data, so not sure how that will impact my internet bandwidth as well. But will try out again.
posted by theobserver at 4:09 AM on July 20, 2015
Best answer: I have over a terabyte backed up to Crashplan. It took over a month to upload, but that was years ago. Now that it's there it's backed up and on autopilot. I once corrupted my hard drive and had to back up from Crashplan - it took weeks, but now all my data is back. The best part is: if I do something stupid and corrupt my hard drive, at least there is no way to corrupt my backup because that's in Crashplan's control.
If your connection took 4 days to upload 2 GB of data, then you can expect Crashplan to take 50-ish days to upload your entire 25 GB. Assuming that you want to keep these photos for decades, 50 days is a blink of an eye. If you take a ton of photos, it will take a few days for the RAW files to upload, but it's all automated and then you're done. If you ask me me, the chances of your computer crashing in those few days while its uploading are far less than the chance that something will go wrong with a home-grown backup solution (especially user error - backups require a huge amount of diligence). Better to let an automated, online system handle it.
And if you have a data cap (no idea where you live, some places have internet data caps) you can set Crashplan to upload even more slowly so it doesn't eat up your cap rapidly.
posted by Tehhund at 7:35 AM on July 20, 2015 [2 favorites]
If your connection took 4 days to upload 2 GB of data, then you can expect Crashplan to take 50-ish days to upload your entire 25 GB. Assuming that you want to keep these photos for decades, 50 days is a blink of an eye. If you take a ton of photos, it will take a few days for the RAW files to upload, but it's all automated and then you're done. If you ask me me, the chances of your computer crashing in those few days while its uploading are far less than the chance that something will go wrong with a home-grown backup solution (especially user error - backups require a huge amount of diligence). Better to let an automated, online system handle it.
And if you have a data cap (no idea where you live, some places have internet data caps) you can set Crashplan to upload even more slowly so it doesn't eat up your cap rapidly.
posted by Tehhund at 7:35 AM on July 20, 2015 [2 favorites]
I just keep the cards I shoot onto. In my photo future I will open a card to process raw files, store a copy, processed back on the original card, and one in a side tower. As you look through your raw images on a large screen only, make sure you are in a good mood, and be rational about what you keep, and even more rational about what you process. Then the card has good images and a few total keepers. When the card is full, store it in safe physical catalogue, carefully dated and described. Your side tower should have your best images labeled to link back to the proper card. Then you can refer back to the other images from the day. Store images in the tower as tiff. Tiffs are big files, but they are stable, unless "they" have come up with something better this week.
posted by Oyéah at 8:13 AM on July 20, 2015
posted by Oyéah at 8:13 AM on July 20, 2015
Best answer:
posted by Brian Puccio at 9:08 AM on July 20, 2015 [1 favorite]
- I wouldn't worry about long term accessibility of CR2 (and I say this as a large proponent of the open-data-formats-over-open-source ideaology). There are a few implementations of CR2 readers. It's such a widely used (though proprietary) format that if opening it starts to become a problem, you can bulk convert your files before it is a problem. That being said, DNG doesn't work for you if your applications don't use it.
- I think the concern that most have is that the LR/PS ecosystem is becoming more subscription based and less you-own-the-software-you-paid-for. I'm paying $10/mo now, but I'm open to other LR-esque applications with a low barrier to migration.
- I love crashplan and have 15TB or so with them. Initial backup took about a year. If you've got a slow upload, but no cap, you can stick it out with crashplan. If you've got a data cap with overage fees, online backup services won't work for you. An alternative would be to purchase an external hard drive, back up to it monthly and store it off-site.
- I like duff on Mac (via homebrew) and Linux.
posted by Brian Puccio at 9:08 AM on July 20, 2015 [1 favorite]
Best answer: 1) 2nding lack of worry about reading big-vendor raw formats in the future.
The DNG vs vendor-raw debate is large and complex, but I've never heard of the format not being readable in the future as an argument.
2) Not sure I understand. I think using any software is a risk. Both Adobe and Apple are generally headed in directions that I don't care about/think are worthless, so... but the nondestructive editing tools of LR win me over for now.
3) I use an internal drive, and clone after import to a removable drive tower. Every once in a while I make a clone of the removable drive, and swap it with one I keep in a remote/secure location. Not like a bank or anything, just a place not in the immediate area. I'm personally wary of the online services, but hey.
4) I did a batch of de-duping a long time ago using this plugin, but I guess there are probably plenty of other duplicate finding utilities, from within LR, or at the filesystem level.
5) Depends what you mean by "best use it"
I read a few LR blogs, and I think they're mostly crap. It's pro-sumer software, and it's easier to get clicks from posting about how to process a sunset than how to structure organization and whatever, so..
The DAM book by Peter Krogh was awesome in 2008, but I haven't been keeping up with his writing.. I do recall maybe seeing a great LR post from "The LR Queen."
Honestly, it's been ages since I've seen good LR specific writing, and about one week since I've seen garbage.
posted by Jack Karaoke at 11:52 AM on July 20, 2015
The DNG vs vendor-raw debate is large and complex, but I've never heard of the format not being readable in the future as an argument.
2) Not sure I understand. I think using any software is a risk. Both Adobe and Apple are generally headed in directions that I don't care about/think are worthless, so... but the nondestructive editing tools of LR win me over for now.
3) I use an internal drive, and clone after import to a removable drive tower. Every once in a while I make a clone of the removable drive, and swap it with one I keep in a remote/secure location. Not like a bank or anything, just a place not in the immediate area. I'm personally wary of the online services, but hey.
4) I did a batch of de-duping a long time ago using this plugin, but I guess there are probably plenty of other duplicate finding utilities, from within LR, or at the filesystem level.
5) Depends what you mean by "best use it"
I read a few LR blogs, and I think they're mostly crap. It's pro-sumer software, and it's easier to get clicks from posting about how to process a sunset than how to structure organization and whatever, so..
The DAM book by Peter Krogh was awesome in 2008, but I haven't been keeping up with his writing.. I do recall maybe seeing a great LR post from "The LR Queen."
Honestly, it's been ages since I've seen good LR specific writing, and about one week since I've seen garbage.
posted by Jack Karaoke at 11:52 AM on July 20, 2015
Best answer: I have a huge pile of images backed up locally - and use Crashplan backing up to a shared computer (physically outside my home, it's a work computer) to handle the need for reliance on multiple locations.
But for long term storage? I very recently decided that the offer of a free terabyte for my photos was worth using, so I uploaded everything to Flickr. I've been using the service for over a decade anyway, so why not use it to its full capacity? It took days to upload it all, but it was only the rendered images and not the RAW files (these are only in 3 places - computer, home backup, work backup) as Flickr won't do RAW yet.
It's hard to beat free. And they promise one-click download to take back all your photos if you don't want them in the cloud in the future. I was pleased to see it handled duplicates intelligently, skipped photos I'd already uploaded, and the new "magic" feature does a fairly good job of indexing contents - so my pile of 45,000 photos got sorted, tagged or not, helpful in organizing an otherwise unwieldy collection. I mean, it isn't perfect - it thinks a photo of my kid swimming is a bird, and is convinced our old cat was really a dog - but it's better than doing it all myself.
I'm still using Lightroom 3 because I hate the idea of renting software. I am really, really hoping this isn't the future. "Pay once and free upgrades for life" is a model I like best, "pay once per version" is my 2nd choice, but pure rental is a no-win game for anyone except the software company. Especially not the Adobe version, where the "benefit" of renting is that I have access to a whole slew of software I don't currently use, don't need, and will never install - great idea for a digital publishing house, terrible idea for an end user who was perfectly happy with Design Standard + Lightroom for work and hobby use.
posted by caution live frogs at 2:37 PM on July 20, 2015
But for long term storage? I very recently decided that the offer of a free terabyte for my photos was worth using, so I uploaded everything to Flickr. I've been using the service for over a decade anyway, so why not use it to its full capacity? It took days to upload it all, but it was only the rendered images and not the RAW files (these are only in 3 places - computer, home backup, work backup) as Flickr won't do RAW yet.
It's hard to beat free. And they promise one-click download to take back all your photos if you don't want them in the cloud in the future. I was pleased to see it handled duplicates intelligently, skipped photos I'd already uploaded, and the new "magic" feature does a fairly good job of indexing contents - so my pile of 45,000 photos got sorted, tagged or not, helpful in organizing an otherwise unwieldy collection. I mean, it isn't perfect - it thinks a photo of my kid swimming is a bird, and is convinced our old cat was really a dog - but it's better than doing it all myself.
I'm still using Lightroom 3 because I hate the idea of renting software. I am really, really hoping this isn't the future. "Pay once and free upgrades for life" is a model I like best, "pay once per version" is my 2nd choice, but pure rental is a no-win game for anyone except the software company. Especially not the Adobe version, where the "benefit" of renting is that I have access to a whole slew of software I don't currently use, don't need, and will never install - great idea for a digital publishing house, terrible idea for an end user who was perfectly happy with Design Standard + Lightroom for work and hobby use.
posted by caution live frogs at 2:37 PM on July 20, 2015
Best answer: 3. If it takes that long to only back up 25GB then I don't think you should bother backing up RAW files online. If you're shooting raw then your storage will get into the hundreds of gigabyte/terabyte range fairly quickly which would make uploading impractical. I would stick with JPGs in your situation and maybe upload raw files for the really good/important photos.
At a minimum you should have one local backup which is on a separate device from your working computer and one offsite backup. So you are set for the local backup and just need to get the offsite one. The second backup is important. On my honeymoon I was storing my photos on my iPod as well as a portable hard drive. When I got home I was transferring the photos from the hard drive to my laptop when my wife bumped the table they were on. The hard drive fell off the table and died when it hit the floor. This was OK because I still had the photos on my iPod. So if you were in a similar situation where you were copying files from your computer to your external hard drive and they fell or there was some crazy power surge or a pipe in the ceiling burst and the water destroyed them both then you would still have the offsite backup (as well as the files on your memory card(s), because you didn't delete them until all the backups had taken place).
My backup workflow goes something like this:
1. My camera has 2 SD card slots. The second is set to be a backup of the first.
2. Import the photos into Lightroom on my laptop.
3. Copy the files onto my NAS device over the local network.
4. Publish all the photos onto my Flickr account (published as private so only I can see them) from Lightroom.
5. The NAS device automatically uploads the raw files onto my Google Drive account.
6. Copy photos from NAS device onto 2 portable hard drives. One stays unplugged at home and the other gets swapped with one in my desk at work every month.
7. Delete the photos from the SD cards in the camera. Because the whole process takes some time, it is good to have memory cards that are sufficiently large. You don't want to be in a situation where you feel pressure to empty the cards to take more pictures before they are all backed up. I use 32GB cards in the camera, which hold about 600 pictures.
The Flickr and Google Drive steps are more for accessibility than anything else. I don't have a ton of time at home to review my photos so being able to do it on my phone or laptop on the go is a great convenience. I know you're not supposed to think of cloud services as backups and so I still have my hard drives, but it is quite reassuring to know they're on the cloud as well.
posted by any portmanteau in a storm at 2:44 PM on July 20, 2015
At a minimum you should have one local backup which is on a separate device from your working computer and one offsite backup. So you are set for the local backup and just need to get the offsite one. The second backup is important. On my honeymoon I was storing my photos on my iPod as well as a portable hard drive. When I got home I was transferring the photos from the hard drive to my laptop when my wife bumped the table they were on. The hard drive fell off the table and died when it hit the floor. This was OK because I still had the photos on my iPod. So if you were in a similar situation where you were copying files from your computer to your external hard drive and they fell or there was some crazy power surge or a pipe in the ceiling burst and the water destroyed them both then you would still have the offsite backup (as well as the files on your memory card(s), because you didn't delete them until all the backups had taken place).
My backup workflow goes something like this:
1. My camera has 2 SD card slots. The second is set to be a backup of the first.
2. Import the photos into Lightroom on my laptop.
3. Copy the files onto my NAS device over the local network.
4. Publish all the photos onto my Flickr account (published as private so only I can see them) from Lightroom.
5. The NAS device automatically uploads the raw files onto my Google Drive account.
6. Copy photos from NAS device onto 2 portable hard drives. One stays unplugged at home and the other gets swapped with one in my desk at work every month.
7. Delete the photos from the SD cards in the camera. Because the whole process takes some time, it is good to have memory cards that are sufficiently large. You don't want to be in a situation where you feel pressure to empty the cards to take more pictures before they are all backed up. I use 32GB cards in the camera, which hold about 600 pictures.
The Flickr and Google Drive steps are more for accessibility than anything else. I don't have a ton of time at home to review my photos so being able to do it on my phone or laptop on the go is a great convenience. I know you're not supposed to think of cloud services as backups and so I still have my hard drives, but it is quite reassuring to know they're on the cloud as well.
posted by any portmanteau in a storm at 2:44 PM on July 20, 2015
Response by poster: Great answers so far! Thanks and keep them coming!
1. CR2 it is for now, but I am still confused on the sidecar files. Need to research more about it, since I am worried that I will lose their association with their other halves
2. I am planning to get a NAS set to RAID 1, but still upload RAW files of keepers to Crashplan or even Google Drive/OneDrive
3. For Lightroom, I am not sure on how to maintain catalogs (one huge catalog or multiple yearly/major shoots), import + keyword + rename all *new* photos only when importing into the catalog as well as folder structure for original files, edited ones, converted to JPGs and starred ones (separate folders or maintain in Lightroom). I will look for lessons on udacity etc on this
4. Going to continue with my local copy of Lightroom till it expires and I am forced to go to the subscription model. Capture One seems to be the alternative, but there is no guarantee they will also not move to this model once they scale
posted by theobserver at 10:02 PM on July 20, 2015
1. CR2 it is for now, but I am still confused on the sidecar files. Need to research more about it, since I am worried that I will lose their association with their other halves
2. I am planning to get a NAS set to RAID 1, but still upload RAW files of keepers to Crashplan or even Google Drive/OneDrive
3. For Lightroom, I am not sure on how to maintain catalogs (one huge catalog or multiple yearly/major shoots), import + keyword + rename all *new* photos only when importing into the catalog as well as folder structure for original files, edited ones, converted to JPGs and starred ones (separate folders or maintain in Lightroom). I will look for lessons on udacity etc on this
4. Going to continue with my local copy of Lightroom till it expires and I am forced to go to the subscription model. Capture One seems to be the alternative, but there is no guarantee they will also not move to this model once they scale
posted by theobserver at 10:02 PM on July 20, 2015
Response by poster: In case someone reads this thread looking for Lightroom5 - based photo organization, couple of youtube videos are good to watch:
https://www.youtube.com/watch?v=nF2RMd3L83w
https://www.youtube.com/watch?v=-NsE457Iq-c
posted by theobserver at 6:26 AM on July 21, 2015
https://www.youtube.com/watch?v=nF2RMd3L83w
https://www.youtube.com/watch?v=-NsE457Iq-c
posted by theobserver at 6:26 AM on July 21, 2015
Sidecars are the same file name, with a diff extension. I guess you could mess it up, but.. it seems like it would require a fair lapse in judgement to do so.
Definitely don't depend on RAID 1 as an über backup. Don't even get it set up and then let yourself slide on creating the 2nd backup. RAID is great for what it is, but it introduces another level of abstraction, and thus another level that can fail.
Definitely think over your structure a bit.. but always have your uses as your main concern. There is no single *best* way to organize. It depends on what you want to do with it.
I'm pretty sure there are other alternatives to LR. I see mention of Photo Mechanic fairly often, and I know people who do fairly well with just Adobe Bridge. There's not feature parity between all of them, but in the general "image database" space there are certainly more options than C1 and LR.
posted by Jack Karaoke at 9:53 PM on July 22, 2015 [1 favorite]
Definitely don't depend on RAID 1 as an über backup. Don't even get it set up and then let yourself slide on creating the 2nd backup. RAID is great for what it is, but it introduces another level of abstraction, and thus another level that can fail.
Definitely think over your structure a bit.. but always have your uses as your main concern. There is no single *best* way to organize. It depends on what you want to do with it.
I'm pretty sure there are other alternatives to LR. I see mention of Photo Mechanic fairly often, and I know people who do fairly well with just Adobe Bridge. There's not feature parity between all of them, but in the general "image database" space there are certainly more options than C1 and LR.
posted by Jack Karaoke at 9:53 PM on July 22, 2015 [1 favorite]
Response by poster: @Jack Karaoke - you are right about letting the second backup fall through. So, trying to set up a system where I back up to NAS and Crashplan.
The main issue with structure is - during a vacation or event, I shoot some photos that are not just about family or people, but something I want to showcase as my "portfolio" of interesting photos. Should I keep them in the same catalog/folder or copy to a portfolio catalog?
As of now, I have decided to keep them separate, so that I can have a different workflow for each of them and also upload the portfolio to Flickr or Smugmu, while my other photos go to Google Photos for family viewing.
I'll check out the LR alternatives, but I am sad about having to compromise on some features of LR
posted by theobserver at 10:25 PM on July 22, 2015
The main issue with structure is - during a vacation or event, I shoot some photos that are not just about family or people, but something I want to showcase as my "portfolio" of interesting photos. Should I keep them in the same catalog/folder or copy to a portfolio catalog?
As of now, I have decided to keep them separate, so that I can have a different workflow for each of them and also upload the portfolio to Flickr or Smugmu, while my other photos go to Google Photos for family viewing.
I'll check out the LR alternatives, but I am sad about having to compromise on some features of LR
posted by theobserver at 10:25 PM on July 22, 2015
This thread is closed to new comments.
Get some cheap T drives and look into clouds.
posted by Freedomboy at 11:26 PM on July 19, 2015