http://ask.metafilter.com/96851/Uniq-New-York-Uniq-New-York/ Comments on Ask MetaFilter post Uniq New York, Uniq New York Thu, 17 Jul 2008 12:06:40 -0800 Thu, 17 Jul 2008 12:06:40 -0800 en-us http://blogs.law.harvard.edu/tech/rss 60 http://ask.metafilter.com/96851/Uniq-New-York-Uniq-New-York Why isn't GNU <code>uniq</code> 6.12 working as I expect? <br /><br /> I am trying to use the <code>--skip-fields=n</code> argument to skip over some columns that I don't want to use for the criteria of uniqueness. <br> <br> But it seems like <code>uniq</code> is ignoring those criteria.<br> <br> Here is my sample input ("<code>chr3test</code>"). Imagine that that the spaces are really tabs:<br> <br> <code>+ aatgtaatt 12124016 12124007 aattNaatt chr3<br> + aactgaatt 37509704 37509695 aattNaatt chr3<br> + aatttaatc 43787257 43787248 aattNaatt chr3<br> + aatttaatt 81433256 81433247 aattNaatt chr3<br> + aattaattt 81433277 81433268 aattNaatt chr3<br> + aatttcatt 121944135 121944126 aattNaatt chr3<br> + aattcaatt 128374695 128374686 aattNaatt chr3<br> + aattcaatc 128374700 128374691 aattNaatt chr3<br> + aagtgaatt 151747168 151747159 aattNaatt chr3<br> + tattaaatt 175957080 175957071 aattNaatt chr3<br> + aatttaatt 178762409 178762400 aattNaatt chr3<br> - aattacatt 12124016 12124007 aattNaatt chr3<br> - aattcagtt 37509704 37509695 aattNaatt chr3<br> - gattaaatt 43787257 43787248 aattNaatt chr3<br> - aattaaatt 81433256 81433247 aattNaatt chr3<br> - aaattaatt 81433277 81433268 aattNaatt chr3<br> - aatgaaatt 121944135 121944126 aattNaatt chr3<br> - aattgaatt 128374695 128374686 aattNaatt chr3<br> - gattgaatt 128374700 128374691 aattNaatt chr3<br> - aattcactt 151747168 151747159 aattNaatt chr3<br> - aatttaata 175957080 175957071 aattNaatt chr3<br> - aattaaatt 178762409 178762400 aattNaatt chr3</code><br> <br> When I run:<br> <br> <code>uniq -f 2 chr3test</code><br> <br> I get the entire file back, which is wrong. <br> <br> If this worked, I would expect:<br> <br> <code>+ aatgtaatt 12124016 12124007 aattNaatt chr3<br> + aactgaatt 37509704 37509695 aattNaatt chr3<br> + aatttaatc 43787257 43787248 aattNaatt chr3<br> + aatttaatt 81433256 81433247 aattNaatt chr3<br> + aattaattt 81433277 81433268 aattNaatt chr3<br> + aatttcatt 121944135 121944126 aattNaatt chr3<br> + aattcaatt 128374695 128374686 aattNaatt chr3<br> + aattcaatc 128374700 128374691 aattNaatt chr3<br> + aagtgaatt 151747168 151747159 aattNaatt chr3<br> + tattaaatt 175957080 175957071 aattNaatt chr3<br> + aatttaatt 178762409 178762400 aattNaatt chr3</code><br> <br> When I run:<br> <br> <code>uniq -f 3 chr3test</code><br> <br> I also get the same results.<br> <br> But when I run:<br> <br> <code>uniq -f 4 chr3test</code><br> <br> Then I get back:<br> <br> <code>+ aatgtaatt 12124016 12124007 aattNaatt chr3</code><br> <br> It seems like those numerical entries are treated as unique, and skipping over them makes the comparison criteria "<code>aattNaatt\tchr3</code>", which eliminates all but the first row.<br> <br> I checked the metacharacters and there are \t ("tabs") correctly placed in matching rows (the matching condition being everything but the first two columns).<br> <br> Can someone point to what I'm doing incorrectly, or suggest another method (preferably command-line) to strip out duplicates without losing column data? Thanks!</tab> post:ask.metafilter.com,2008:site.96851 Thu, 17 Jul 2008 12:00:12 -0800 Blazecock Pileon gnu uniq argument commandline cli http://ask.metafilter.com/96851/Uniq-New-York-Uniq-New-York#1411775 Uniq requires sorted input. It looks like your input is not sorted on that field. comment:ask.metafilter.com,2008:site.96851-1411775 Thu, 17 Jul 2008 12:06:40 -0800 mkb http://ask.metafilter.com/96851/Uniq-New-York-Uniq-New-York#1411778 So basically what you need if <pre>sort -k3 -n chr3test | uniq -f2</pre> comment:ask.metafilter.com,2008:site.96851-1411778 Thu, 17 Jul 2008 12:07:51 -0800 mkb http://ask.metafilter.com/96851/Uniq-New-York-Uniq-New-York#1411782 I forgot about that requirement. Thanks! comment:ask.metafilter.com,2008:site.96851-1411782 Thu, 17 Jul 2008 12:11:03 -0800 Blazecock Pileon