How can I monitor a disc rewrite process in Ubuntu?
November 8, 2008 7:45 AM Subscribe
I'm using Terminal (Ubuntu 8.04x64) right now to overwrite a 500g external hd with random information. There's been no action aside from a blinking cursor (by itself on a line, not in front of any prompt) since I entered the command yesterday afternoon:
[
I understand it's going to take a while to finish. In the meantime, is there any way to monitor the progress of the operation?
[
sudo dd if=/dev/random of=/dev/sdc bs=8m]I understand it's going to take a while to finish. In the meantime, is there any way to monitor the progress of the operation?
sudo killall -USR1 dd (at another terminal)
That'll make it print out its progress on stderr.
Also, what rpn said. Even /dev/urandom isn't blazingly fast, and you'll probably get some speed gain from using proper disk scrubber software.
posted by fvw at 8:14 AM on November 8, 2008
That'll make it print out its progress on stderr.
Also, what rpn said. Even /dev/urandom isn't blazingly fast, and you'll probably get some speed gain from using proper disk scrubber software.
posted by fvw at 8:14 AM on November 8, 2008
Here's a page that I got when googling for "dd status ubuntu".
Ubuntu Forums: Progress of dd
The method involved is to stop the dd process and then immediately start it back up again. You need to know the PID (process id).
Where (PID) is the last PID listed by the ps command. This method is supported by the dd folk. It's listed when you type "
posted by Axle at 8:17 AM on November 8, 2008
Ubuntu Forums: Progress of dd
The method involved is to stop the dd process and then immediately start it back up again. You need to know the PID (process id).
$ ps -A | grep dd
$ sudo kill -USR1 (PID)Where (PID) is the last PID listed by the ps command. This method is supported by the dd folk. It's listed when you type "
dd --help".posted by Axle at 8:17 AM on November 8, 2008
Best answer: If you open a second terminal, you can use
to send a USR1 signal to all your running instances of dd, which will make them spit out a summary of how many blocks they've read and written so far.
As rpn says, using /dev/random is going to make this a very slow process. If I were you, I'd abort the running dd and start over. First I'd install Gnu ddrescue:
Then I'd use that instead of dd to copy pseudorandom rubbish to the hard disk:
Unlike dd, ddrescue is noisy - it keeps you informed of what it's up to, and even tells you what the peak and average transfer rates are.
By the way, this hard disk you're erasing - it wouldn't be the MyBook we just managed to rescue for you, would it?
posted by flabdablet at 8:22 AM on November 8, 2008
pkill -USR1 dd
to send a USR1 signal to all your running instances of dd, which will make them spit out a summary of how many blocks they've read and written so far.
As rpn says, using /dev/random is going to make this a very slow process. If I were you, I'd abort the running dd and start over. First I'd install Gnu ddrescue:
sudo apt-get install gddrescue
Then I'd use that instead of dd to copy pseudorandom rubbish to the hard disk:
sudo ddrescue /dev/urandom /dev/sdc ddrescue.log
Unlike dd, ddrescue is noisy - it keeps you informed of what it's up to, and even tells you what the peak and average transfer rates are.
By the way, this hard disk you're erasing - it wouldn't be the MyBook we just managed to rescue for you, would it?
posted by flabdablet at 8:22 AM on November 8, 2008
On second thought,
and then using
to deliver the USR1 signal to that process only.
posted by flabdablet at 8:31 AM on November 8, 2008
pkill -USR1 dd is a bit gung-ho. Probably better is finding the process ID for the dd you want to interrogate using
ps $(pgrep dd)
and then using
kill -USR1 pid
to deliver the USR1 signal to that process only.
posted by flabdablet at 8:31 AM on November 8, 2008
For what it's worth, I just tried
on this laptop I'm working on, and ddrescue reported an average transfer rate of 5290 kb/s. Doing the same thing with /dev/random stalled altogether, probably because /dev/random blocked before it could deliver even one of the 64KiB chunks that ddrescue copies in by default.
Personally I use /dev/zero as the source for my hard disk wiping goodness. On this same laptop, ddrescue can copy /dev/zero to /dev/null at 8000 megabytes per second.
posted by flabdablet at 8:38 AM on November 8, 2008
ddrescue /dev/urandom /dev/null
on this laptop I'm working on, and ddrescue reported an average transfer rate of 5290 kb/s. Doing the same thing with /dev/random stalled altogether, probably because /dev/random blocked before it could deliver even one of the 64KiB chunks that ddrescue copies in by default.
Personally I use /dev/zero as the source for my hard disk wiping goodness. On this same laptop, ddrescue can copy /dev/zero to /dev/null at 8000 megabytes per second.
posted by flabdablet at 8:38 AM on November 8, 2008
I've just recently used shred to wipe a disk in ubuntu (prior to sending the PC back under guarantee). Admittedly it just owerwrites with zeroes each time, but I'm happy enough with that.
I used: shred -n 5 -vz /dev/hda to overwrite the disk 5 times.
It outputs percentage done as it goes along. And it takes an absolute age, or course. But it might be an alternative.
posted by le morte de bea arthur at 8:43 AM on November 8, 2008
I used: shred -n 5 -vz /dev/hda to overwrite the disk 5 times.
It outputs percentage done as it goes along. And it takes an absolute age, or course. But it might be an alternative.
posted by le morte de bea arthur at 8:43 AM on November 8, 2008
FYI, there are a couple of disk wipe tools out packaged in Ubuntu. The reason people write these instead of using a simple script like you're doing is that dozens of things can go wrong. You could use /dev/zero because its faster, but it'd be wrong because it doesn't write enough to change levels. You could accidentally use a blocking random number generator, but it'd be too slow. (The best way is to layer writes of bit patterns, it seems). The FS likely won't sync writes by default, so you need to set that.
Two packages I know of are wipe and secure-delete. I recall wipe giving excellent output as well.
posted by pwnguin at 9:22 AM on November 8, 2008
Two packages I know of are wipe and secure-delete. I recall wipe giving excellent output as well.
posted by pwnguin at 9:22 AM on November 8, 2008
Response by poster: Excellent advice so far, thank you all. I sheepishly admit to not researching this particular question as much as I usually do before asking MeFi, and then bailing to work before trying any of the suggestions. To answer flabdablet's question, Yes! I got everything I needed off it except one stubborn folder that just won't go away. And I'd like to erase every memory (in a pathetic sense) of Western Digital and the proprietary mess comes with it from the poor drive's tiny little brain. I'll update tonight when I get home from work. Thanks again!
posted by carsonb at 9:44 AM on November 8, 2008
posted by carsonb at 9:44 AM on November 8, 2008
Replacing the /dev/random with /dev/zero with just zero the drive. There's no real reason to random the drive.
posted by damn dirty ape at 10:36 AM on November 8, 2008
posted by damn dirty ape at 10:36 AM on November 8, 2008
As crazy as it may seem, this process will probably terminate sooner if you shake your mouse and pound on your keyboard.
posted by rlk at 11:00 AM on November 8, 2008 [2 favorites]
posted by rlk at 11:00 AM on November 8, 2008 [2 favorites]
Yeah, you really want to use urandom instead of random. random blocks (stops providing data) once the pool of randomness is exhausted (see: cat /proc/sys/kernel/random/entropy_avail to see the current state of the entropy pool, in bits- at most 4,096) urandom is a PRNG that is more than sufficient for generating random data.
However, as stated /dev/zero is much faster than urandom (in my testing 6 MB/sec versus the physical limits of the drive with zero), so for your use zero may be your best bet. In my testing, random is measured in bytes/sec.
posted by theclaw at 12:37 PM on November 8, 2008
However, as stated /dev/zero is much faster than urandom (in my testing 6 MB/sec versus the physical limits of the drive with zero), so for your use zero may be your best bet. In my testing, random is measured in bytes/sec.
posted by theclaw at 12:37 PM on November 8, 2008
I'd like to erase every memory (in a pathetic sense) of Western Digital and the proprietary mess comes with it from the poor drive's tiny little brain
Well, if you're hanging onto the drive for your own re-use, there's absolutely no need to do any kind of secure erase at all. Even a whole-drive zero erase is overkill. I understand the emotional necessity for it (take that, evil proprietary programming!) but technically, all you really need to do is zero the first 63 blocks of the drive to wipe out the master boot record (which includes the partition table) and any embedded boot loader code.
If I'm going to do a whole-drive erase, I will erase with zeroes rather than randomness. For me, the point of a whole-drive erase is to avoid surprises. Every filesystem recovery utility ever written knows that unused blocks on new hard drives have zeroes in them, but there's no way of predicting how how a recovery utility will interpret blocks of random garbage. If you want to wipe a drive in order to make it marginally more useful to you, wipe it with zeroes.
The only way a full random write makes any sense is if you're interested in steganographic file systems or haven't thought carefully about the Epilogue to Gutmann's original paper on secure data erasure. If you want to destroy the data on a hard disk drive, use thermite and be done with it.
Far more important than choice of erasure method is the subsequent choice of filesystem. If this drive is going to be Linux-only, then ext3 is the right choice - it's not the fastest Linux file system, but the differences are pretty marginal for most use cases, and ext3 works harder than any of the others to avoid self-corruption in assorted failure modes.
If you're going to be moving this drive between Linux and Windows, format it NTFS. All modern Linux distros include NTFS-3G, allowing safe reading and writing of Microsoft's proprietary filesystem format. The only issue you will need to be aware of is that NTFS-3G is apparently a little less rigid than Windows about enforcing the Windows limits on what can go in a filename. Using the streams_interface=windows mount option on any Linux box would probably improve compatibility, but you can't rely on that happening for auto-mounts on boxes you don't admin.
Only if you need minimal hassles in moving between Linux, Windows and Mac systems should you consider using FAT32. The good part about FAT32 is that every system available today has inbuilt support for it. The bad part about FAT32 is everything else (no journalling, tendency to fragment, no support for big files).
Also worth knowing is that Windows will sometimes refuse to mount more than one partition on an external drive. It seems to do this more often with flash drives than hard drives, but I've seen it happen on both. So if you're going to use this drive with Windows, one big partition is probably your best bet.
posted by flabdablet at 5:19 PM on November 8, 2008 [1 favorite]
Well, if you're hanging onto the drive for your own re-use, there's absolutely no need to do any kind of secure erase at all. Even a whole-drive zero erase is overkill. I understand the emotional necessity for it (take that, evil proprietary programming!) but technically, all you really need to do is zero the first 63 blocks of the drive to wipe out the master boot record (which includes the partition table) and any embedded boot loader code.
If I'm going to do a whole-drive erase, I will erase with zeroes rather than randomness. For me, the point of a whole-drive erase is to avoid surprises. Every filesystem recovery utility ever written knows that unused blocks on new hard drives have zeroes in them, but there's no way of predicting how how a recovery utility will interpret blocks of random garbage. If you want to wipe a drive in order to make it marginally more useful to you, wipe it with zeroes.
The only way a full random write makes any sense is if you're interested in steganographic file systems or haven't thought carefully about the Epilogue to Gutmann's original paper on secure data erasure. If you want to destroy the data on a hard disk drive, use thermite and be done with it.
Far more important than choice of erasure method is the subsequent choice of filesystem. If this drive is going to be Linux-only, then ext3 is the right choice - it's not the fastest Linux file system, but the differences are pretty marginal for most use cases, and ext3 works harder than any of the others to avoid self-corruption in assorted failure modes.
If you're going to be moving this drive between Linux and Windows, format it NTFS. All modern Linux distros include NTFS-3G, allowing safe reading and writing of Microsoft's proprietary filesystem format. The only issue you will need to be aware of is that NTFS-3G is apparently a little less rigid than Windows about enforcing the Windows limits on what can go in a filename. Using the streams_interface=windows mount option on any Linux box would probably improve compatibility, but you can't rely on that happening for auto-mounts on boxes you don't admin.
Only if you need minimal hassles in moving between Linux, Windows and Mac systems should you consider using FAT32. The good part about FAT32 is that every system available today has inbuilt support for it. The bad part about FAT32 is everything else (no journalling, tendency to fragment, no support for big files).
Also worth knowing is that Windows will sometimes refuse to mount more than one partition on an external drive. It seems to do this more often with flash drives than hard drives, but I've seen it happen on both. So if you're going to use this drive with Windows, one big partition is probably your best bet.
posted by flabdablet at 5:19 PM on November 8, 2008 [1 favorite]
Response by poster: Alright. I killed the random dd process as soon as I got home. (My roommate did enjoy periodically bashing on my keyboard while I was at work, though.) Then I followed flabdablet's gentle instructions and installed gddrescue. First try I used the /dev/urandom option, and wound up with a 6085 kB/s average rate. Then, simply because gddrescue tells me how to interrupt the process, I decided to try /dev/zero instead. It's faster, by a lot. The rate is averaging about 28500 kB/s within the first five minutes, and climbing.
Anyway, as long as filesystems have come up, would ext3 be the best for a drive that would be shared over a home network between my computer and my roommate's Mac?
The online Ubuntu documentation recommends LinuxCommand.org for learning how to use the command line. Is that a solid rec, or are there better options?
posted by carsonb at 8:43 PM on November 8, 2008
Anyway, as long as filesystems have come up, would ext3 be the best for a drive that would be shared over a home network between my computer and my roommate's Mac?
The online Ubuntu documentation recommends LinuxCommand.org for learning how to use the command line. Is that a solid rec, or are there better options?
posted by carsonb at 8:43 PM on November 8, 2008
In general, ext3 is the best filesystem choice for a drive that's only ever going to be directly connected to a Linux box. Whether that box is then going to share those files over a network is pretty much irrelevant.
Network file sharing protocols don't generally expose the local filesystem over the network; in general, there's no way for a network file client to discover what filesystem exists inside the NAS. As long as the filesystem is capable of storing all the data and metadata that the network file sharing server expects it to, it will in general support whatever calls the network client makes on that server.
Western Digital's engineers* aren't complete idiots, and there's a reason they put an ext3 filesystem in their NAS in the first place. It's a good, solid, reliable and acceptably fast filesystem. Use it unless you have a specific reason not to.
*I have no information about the idiocy level within Western Digital's management and marketing arms.
posted by flabdablet at 10:20 PM on November 8, 2008
Network file sharing protocols don't generally expose the local filesystem over the network; in general, there's no way for a network file client to discover what filesystem exists inside the NAS. As long as the filesystem is capable of storing all the data and metadata that the network file sharing server expects it to, it will in general support whatever calls the network client makes on that server.
Western Digital's engineers* aren't complete idiots, and there's a reason they put an ext3 filesystem in their NAS in the first place. It's a good, solid, reliable and acceptably fast filesystem. Use it unless you have a specific reason not to.
*I have no information about the idiocy level within Western Digital's management and marketing arms.
posted by flabdablet at 10:20 PM on November 8, 2008
A quick skim reveals nothing unsound about linuxcommand.org.
posted by flabdablet at 11:20 PM on November 8, 2008
posted by flabdablet at 11:20 PM on November 8, 2008
Response by poster: OK, the zeroing finished overnight, in about 6 hours, the disc is reformatted ext3, and all my files are back on it. Well, most of my files. A couple of folders were mysteriously empty, but I chalk that up to all the drive switching and folder shuffling I've been doing over the past few weeks. Anyway, a longtime goal of mine has now been achieved: My digital music collection is all in one place, and organized! Woo!
Thank you AskMe, and thanks specifically to flabdablet and pwnguin, who have been especially helpful over the course of a few questions. Couldn't have done it without you!
No doubt, fellas, I'll be back in a week or so with some other annoyingly n00b Ubuntu query. Stay tuned.
posted by carsonb at 9:37 AM on November 9, 2008
Thank you AskMe, and thanks specifically to flabdablet and pwnguin, who have been especially helpful over the course of a few questions. Couldn't have done it without you!
No doubt, fellas, I'll be back in a week or so with some other annoyingly n00b Ubuntu query. Stay tuned.
posted by carsonb at 9:37 AM on November 9, 2008
There is absolutely nothing annoying about n00bs who want to learn how to use their computer's literate interface. Glad you had a result!
posted by flabdablet at 1:58 PM on November 10, 2008
posted by flabdablet at 1:58 PM on November 10, 2008
« Older Help me make my computer quiet again, please! | Looking for romaji script for Totoro Newer »
This thread is closed to new comments.
> dstat -d -D sdc
But, using /dev/random will take a long, long time as it gives true random numbers, using input from the mouse, keyboard, network etc as a source so the amount of data you get from it is not much.
/dev/urandom would be much better, as it uses /dev/random as a seed to a good pseudo random number generator that is good enough for your use unless you are truly paranoid (and in that case you should throw your disk in the ocean after you destroyed with a few powertools instead).
I'm not sure how fast urandom can deliver data, but it should be substantially faster than /dev/random at least.
posted by rpn at 8:00 AM on November 8, 2008