Who's listening?

July 31, 2008 4:55 AM

I have a website on which I have mp3s. This month, according to my usage stats, I had a bunch of mp3 downloads -- way more that usual. They show up on my stats page under the heading of "Top 10 of 341 Total URLs By KBytes." Are these necessarily people downloading my mp3s, or do spiders or crawlers or whatever they're called hit these links as well?

Also, if it is some sort of webcrawling thing, what are they doing with the downloads? I've just gone over my monthly traffic limit, so if it is just useless spammy bot-things, how can I cut down on them?

I am on vicodin right now and a little woozy; I hope this makes sense.

posted by Karlos the Jackal to computers & internet (5 comments total)

I don't think most crawlers will download entire mp3s - they're generally only set up to follow links with specific mime-types - usually just html pages, pdfs, text files and so on.

You could add a robots.txt and see if that helps:

User-agent: *
Disallow: /path to mp3s/
posted by le morte de bea arthur at 5:17 AM on July 31, 2008

MetaFilter's own Brad Sucks had a problem with massive downloads from Chinese IP ranges. You could check the geolocation of the IPs that are downloading and see if the same thing is happening to you.
posted by bjrn at 5:26 AM on July 31, 2008

Perhaps Google has just indexed the folder?

Finding MP3s with Google
posted by alasdair at 6:02 AM on July 31, 2008

I've seen MP3s posted as ZIP files. It doesn't save on any disk space, but it prevents people from streaming the MP3s from your site, decreasing overall bandwidth.

Still as available as ever...
posted by unixrat at 6:05 AM on July 31, 2008

seconding le morte de bea arthur - and maybe move them so the old location that the 'bots know about is no longer valid. Also, do a link search to see who's linking to your most popular files - I found a couple of iffy "MP3 finder" sites linking to me, and streaming via Flash.
posted by scruss at 6:39 AM on July 31, 2008