Is there any easy command or application that will randomise lines in a text file?

November 12, 2004 6:55 AM

Is there any easy command or application that will randomise lines in a text file?

Answer: yes, and as usual it turns up immediately after asking the question. For those interested:

#!/usr/bin/perl
# Program to output lines of file in random order

# Read File
while (<>) {
push(foo, $_);
}

# Shuffle lines
srand;
for (0..$#foo) {
$r = rand($#foo+1);
($foo[$r], $foo[$_]) = ($foo[$_], $foo[$r]);
}

# Print it out
for (0..$#foo) {
print $foo[$_];
}


perl [inputfile] > [outputfile]

My thanks to Steve who answered me seven and a half years ago.

posted by humuhumu to technology (11 comments total)

But if you just want one random line from a file, you don't need to copy the whole file into memory:

perl -ne '$l = $_ if rand() < 1/$.; END { print $l }' your_file
posted by nicwolff at 7:19 AM on November 12, 2004

Nice shorthand, nicwolff. I had a file of 597,208 lines that needed randomising - only took a few seconds with the above method.
posted by humuhumu at 7:50 AM on November 12, 2004

Yup, very odd that this is missing from the standard unix toolkit. A shorter version of what humuhumu posted from my shell aliases:

perl -e'@l=<>;@l[$i,$s]=@l[$s=rand($i+1),$i]while++$i<@l;print@l'


Remember you can just assign the entire input array @foo=<>; no need to iterate over it.

Also, if you use zsh (very nice, I can recommend it (and have, at length here on Ask)) and don't want the overhead of starting perl, here it is as a zsh function, don't think it'll work under bash

randlinz () {

        RANDOM=$(date +'%s') 

        _buf=() 

        i=1 

        OLDIFS=IFS

        IFS=""

        while read -r a

        do

                _buf[$[$#_buf+1]]=$a 

        done

        while [ $i -lt $#_buf ]

        do

                i=$[$i+1] 

                r=$[($RANDOM%$#_buf)+1] 

                t=$_buf[r] 

                _buf[r]=$_buf[i] 

                _buf[i]=$t 

        done

        for a in $_buf

        do

                while [ $i -gt 0 ]

                do

                        echo $_buf[$i]

                        i=$[i-1] 

                done

        done

        IFS=OLDIFS

}

As soon as you have a significant number of lines using the perl version will be a lot faster though.
posted by fvw at 9:15 AM on November 12, 2004

Ehm, OLDIFS=$IFS and IFS=$OLDIFS ofcourse. I only added them while posting after realising ommitting that would cause problems in obscure cases (as said, I don't really use the zsh version).
posted by fvw at 9:16 AM on November 12, 2004

A Python version which I think is more readable ;-):

python -c 'import fileinput, random; x = list(fileinput.input()); random.shuffle(x); print "".join(x)'
posted by grouse at 11:58 AM on November 12, 2004

<?php

$lines = file(foo.txt);
shuffle($lines);
foreach($lines as $line) {
    echo $line;
}

?>
posted by sad_otter at 11:59 AM on November 12, 2004

gah. didn't quote the filename. oh well.
posted by sad_otter at 12:02 PM on November 12, 2004

grouse: That's cheating. In that case, perl -MAcme::Randomise -e 'print randomise <>;'
posted by fvw at 7:40 PM on November 12, 2004

fvw: your answer presupposes that Acme::Randomise is preinstalled, where I only used stuff from the Python standard library. The OP was not looking for a programming tutorial but something that works, so a one-liner that doesn't require installing additional stuff is probably best.
posted by grouse at 9:29 AM on November 13, 2004

I know, merely trying to start a language war. Ignore me :-)
posted by fvw at 12:06 PM on November 13, 2004

A bit late, but it turns out perl does actually come with an Acme::Randomise installed by default, although named a bit more sensibly:

perl -MList::Util -e '@a=<>; print List::Util::shuffle(@a);'

posted by fvw at 5:36 AM on November 19, 2004