URL WTF

August 11, 2009 4:04 PM

My site is getting a lot of weirdly-formed referring URLs--some with two or three URLs linked with commas--or URLs that do not appear to contain links to my site. What's going on?

Here are two examples of URLs in my hit log--the first is one referrer that shows up in my log as three URLs (including mine) strung together:

http://www.lilidaviesnolink.co.uk/457/magic-betty-sing-along-6th-june, http://www.phonogramnolink.us/blogs2/dpdc/2009/05/trek_30.html, http://www.mattdidthatnolink.com/weblog

Another referring URL without a link to my site:

http://www.mp4dunyasinolink.com/index.php

(To make the URLs work, you'll need to remove the "nolink")

What's going on? What are these referrers, and why am I getting them? If this is spam, could someone please explain what they're doing and how I disable/remove it?

posted by mattdidthat to computers & internet (6 comments total)

Referrer spam. Their purpose is to get into some sort of publicly-accessible list of referers to your site so that they'll be seen by search engine crawlers.

Why toss several in on a single line? Efficiency. They get three spams per bot access instead of only one.
posted by Chocolate Pickle at 4:37 PM on August 11

I got one the other day which had like 20 URLs all packed together like that. And all of them were ".ru".

My general policy is that when I see anyone doing that kind of crap, I block them in my firewall.
posted by Chocolate Pickle at 4:39 PM on August 11

There's all kinds of weird spam bots out there. You can drive yourself nuts trying to figure them all out.
posted by meta_eli at 5:01 PM on August 11

There was one referrer spam bot we had some problems with that would send us garbage strings instead of an honest referrer. They didn't look like well-formed URLs; they were just junk. Didn't do them any good, and because the hit rate on us was so low it didn't have the effect of a DDOS, either.

We never did figure out what that guy was after. The only two possibilites we came up with were:

1. He was using canned spambot software (i.e. he was a "script kiddie") and set it up wrong.
2. He was debugging his spambot software and didn't have a paying client yet.

Anyway, as Meta_Eli says, you can drive yourself nuts trying to figure out all the strange referrers you'll see.
posted by Chocolate Pickle at 5:35 PM on August 11

one referrer spam bot we had some problems with that would send us garbage strings instead of an honest referrer

That was probably just a normal person who was using a shoddily written software firewall. Some of those claim the ability to hide the Referer header as a privacy enhancing measure. But since that kind of software works at the packet level, actually removing the header would change the length of the packet which would also alter all the sequence numbers for the rest of the packets in that connection, which requires full flow tracking logic which is a lot of work. It's a lot easier just to scramble the contents of the header so that it remains the same length.

If you want to hide your Referer header in the name of privacy it's a bad idea to use a software firewall to do it. Use a browser plugin like Refcontrol or a proxy like Privoxy.
posted by Rhomboid at 6:02 PM on August 11

one referrer spam bot we had some problems with that would send us garbage strings instead of an honest referrer

I wouldnt rule out strange character set spam (dont they have unicode url by now?) or even more probably someone looking for a buffer overflow exploit on your webserver..
posted by 3mendo at 11:52 PM on August 11