1 trillion btyes of beer on the wall...
June 25, 2007 2:02 PM Subscribe
GPG/PGP: Is it possible to verify that a file is a valid GPG/PGP file before decrypting it, without a signature?
While working for a Very Large Company, I receive a number of Very Large Files, encrypted with PGP. These files are all over 500G big, some larger than a terabyte. Decrypting these items on a shared resource takes a number of days, so if a file were to fail near the end it would cost us many days to make up. These items pass through a number of error-prone hands (tape, vanilla FTP, etc) so there have been issues of file integrity in the past.
Let's also say that due to Bureaucracy, it is impossible to request a signature from the sender.
So, given an up-to-date version of GPG and a very large encrypted file (but no signature), is it possible to run a validity check on the file before decrypting?
While working for a Very Large Company, I receive a number of Very Large Files, encrypted with PGP. These files are all over 500G big, some larger than a terabyte. Decrypting these items on a shared resource takes a number of days, so if a file were to fail near the end it would cost us many days to make up. These items pass through a number of error-prone hands (tape, vanilla FTP, etc) so there have been issues of file integrity in the past.
Let's also say that due to Bureaucracy, it is impossible to request a signature from the sender.
So, given an up-to-date version of GPG and a very large encrypted file (but no signature), is it possible to run a validity check on the file before decrypting?
Response by poster: Doubtful.
The files are coming from entrenched bureaucracy, who resisted any changes to their 1979-era delivery mechanism until forced to by privacy legislation. And even then, they waited until the last minute to enact it.
Imagine the worst sort of 'not necessary/not my job/not trained to do it' type of foot-dragging. I have zero confidence that they will ever implement something as (not) complicated as MD5 sums given the amount of yelling it took to get them to encrypt something in the first place.
You're absolutely right that an MD5 file would solve this mess instantly... but I don't ever see it coming.
posted by unixrat at 2:50 PM on June 25, 2007
The files are coming from entrenched bureaucracy, who resisted any changes to their 1979-era delivery mechanism until forced to by privacy legislation. And even then, they waited until the last minute to enact it.
Imagine the worst sort of 'not necessary/not my job/not trained to do it' type of foot-dragging. I have zero confidence that they will ever implement something as (not) complicated as MD5 sums given the amount of yelling it took to get them to encrypt something in the first place.
You're absolutely right that an MD5 file would solve this mess instantly... but I don't ever see it coming.
posted by unixrat at 2:50 PM on June 25, 2007
If the decrypting program can't tell if they are valid files, I doubt you are going to have any luck.
I'd try other popular decrypting products and see if they have more robust checks of the files.
posted by mphuie at 3:57 PM on June 25, 2007
I'd try other popular decrypting products and see if they have more robust checks of the files.
posted by mphuie at 3:57 PM on June 25, 2007
Unfortunately the entire point of gpg is for validity to be indistinguishable from invalidity without verifying a signature or attempting to decrypt. That's a positive attribute of the gpg system. Thems are the breaks and the wrong tool for the job got chosen and then implemented badly.
You have two choices: demand the file be placed in a verifiable container (a zip or rar would work if it came to it) or do out of band verification before decryption. If a hash can't be generated automatically and provided with the data from the Paleolithic source, can you at least get one of the people who has to manhandle the file/tape/card stack/clay tablet/scrolls/smoke signals to generate one and slip it to you? That would at least reduce the number of unverifiable points of transit.
posted by majick at 4:33 PM on June 25, 2007
You have two choices: demand the file be placed in a verifiable container (a zip or rar would work if it came to it) or do out of band verification before decryption. If a hash can't be generated automatically and provided with the data from the Paleolithic source, can you at least get one of the people who has to manhandle the file/tape/card stack/clay tablet/scrolls/smoke signals to generate one and slip it to you? That would at least reduce the number of unverifiable points of transit.
posted by majick at 4:33 PM on June 25, 2007
100% what majick said.
nb: hashing (or uncompressed archiving) a 500gb file would take a couple hours minimum. same for checking the hash or archive integrity, but hey that's definitely < days.
posted by dorian at 4:56 PM on June 25, 2007
nb: hashing (or uncompressed archiving) a 500gb file would take a couple hours minimum. same for checking the hash or archive integrity, but hey that's definitely < days.
posted by dorian at 4:56 PM on June 25, 2007
Actually it's possible to do a very simple verification of an encrypted file, though it won't catch much other than "this file is truncated" or "this file isn't even a pgp file to begin with". An encrypted file will normally have some number of encrypted-key packets (each of which contains the symmetric session key encrypted to one recipient's asymmetric key) followed by an encrypted-data packet (containing the actual data). For full details you can read RFC 2440. You could verify that the file at least consists of a series of well-formed pgp packets, and isn't truncated etc. If the file is on a random-access medium like a disk, this would even be fairly fast, since you wouldn't even have to touch the contents of the encrypted-data packet — just seek past it (or verify that its length exactly occupies the rest of the file).
I don't know of any utilities that do this, but it would be pretty simple to whip one up in perl (or your language of choice).
Whether this is helpful depends entirely on what kind of corruption you're getting. A few altered bytes in the middle of the file won't be detected until you actually decrypt it and do the integrity checks on the contents of the encrypyed-data packet.
posted by hattifattener at 8:02 PM on June 25, 2007
I don't know of any utilities that do this, but it would be pretty simple to whip one up in perl (or your language of choice).
Whether this is helpful depends entirely on what kind of corruption you're getting. A few altered bytes in the middle of the file won't be detected until you actually decrypt it and do the integrity checks on the contents of the encrypyed-data packet.
posted by hattifattener at 8:02 PM on June 25, 2007
This thread is closed to new comments.
posted by chunking express at 2:09 PM on June 25, 2007