Why do people hide their http_referer?

December 18, 2004 9:55 AM

Why do people hide their http_referer? [+MI]

afaik, privacy tools such as this NIS, will hide something as innocent as your http_referer (the last webpage you visited before landing on the next, ie; google, or metafilter that links to the new page you are visiting.) Apart from breaking old fashioned anti-leech tricks such as .htaccess rewrite, and even some very useful anti-bot functions in websites that might not want a thousand fake signups or comment spam, what advantage does hiding that information have to the users who do it? I'm just wondering why people feel like they need to hide the last page visited it since I have to code around it all the time and it's getting on my nerves.

posted by dabitch to computers & internet (30 comments total)

I guess you're trying to poll for our opinions here, in which case here's mine: because it's none of your damn business, that's why.
posted by Ryvar at 10:20 AM on December 18, 2004

I load a 3 month trial of NIS on all the computers that come into my shop for the firewall and AV feature. I think it's just a simple privacy issue. It does seem kinda silly, but I still imagine there's a programmer at Symantec with a good reason for it.
posted by daHIFI at 10:20 AM on December 18, 2004

I know a number of people who have the Norton suite installed, and I guarantee that they have no idea that they're blocking the http_referer. It may be a default setting.
posted by monju_bosatsu at 10:24 AM on December 18, 2004

Why do people hide their http_referer?

1. Misguided admins often see mod_rewrite as a panacea and use it indiscriminately. Since there is no server-side distinction between linking directly to a file ("Look at this image") and hotlinking that file within a Web page (<img src="http://www.domain.tld/image.jpg">), mod_rewrite will redirect clicks elsewhere. If your browser reports no referrer, you are far less likely to find yourself staring into the gaping hole of overzealous site administration.

2. I have several linkdump pages on my servers from which I frequently browse. These pages also contain links to administration sites, third-party services, and so on. Very few people would think to find these pages if they didn't appear in their server logs (and even if they did, I have a very narrow spectrum of IP addresses that can access them, thanks to judicious use of .htaccess.)

3. Breaking old fashioned anti-leech tricks and anti-bot functions.

I have to code around it all the time and it's getting on my nerves.

Given the fact that it is solely the Web browser's responsibility to report referrers, and many browsers do so inconsistently or incorrectly, it is bad practice to rely on referrers (or UserAgent strings, or hostname lookups) as a prerequisite for functionality.
posted by Danelope at 10:28 AM on December 18, 2004

I have a web page with loads of links on that I use instead of favourites. I have friends' blogs on there. I don't want them to know how often I look at the blogs! Plus, I don't want anyone to know the location of the page.
posted by ascullion at 10:28 AM on December 18, 2004

I don't hide HTTP_REFERER. I rewrite it on every outbound request to be the same as the HTTP_REQUEST URL. This accomplishes three things: It takes care of the fact that it's none of your business. It provides an innocuous response so that poorly written server scripts that rely on this value continue to function. It clearly reinforces to clueless site administrators that no data coming from the client should be trusted.
posted by majick at 10:36 AM on December 18, 2004

What Ryvar said.
posted by five fresh fish at 10:37 AM on December 18, 2004

I'm just wondering why people feel like they need to hide the last page visited it since I have to code around it all the time and it's getting on my nerves.

I'm just wondering why people feel like they need to know the last page visited since it's none of their business and it's getting on my nerves.
posted by grouse at 10:39 AM on December 18, 2004

Danelope's summary is excellent.

That said, hiding http_referer from everyone seems like refusing all cookies from everyone. It would be nice if there were a way to block reporting your http_referer value selectively--the way discriminating users manage cookies. Say, allow a short list of servers to see your http_referer (local library, university, company extranet, whatever) and block everyone else.
posted by gimonca at 10:44 AM on December 18, 2004

Another vote for what Ryvar said.
posted by Zonker at 10:48 AM on December 18, 2004

"since I have to code around it all the time"

Also, don't do this. If you're actually using the value of Referer: , you're screwed, because even perfectly good browsers put all sorts of stupid and useless values in there. HTTP_REFERER is tainted, untrusted data. Don't use it for anything that matters, because that's insecure and bad design from the get go.

"It would be nice if there were a way to block reporting your http_referer value selectively"

privoxy can help you here.
posted by majick at 10:51 AM on December 18, 2004

Or Pith Helmet, if you're using Apple Safari.
posted by majick at 10:53 AM on December 18, 2004

Some poorly designed web sites that are supposed to be secure put session information/security tokens/passwords in their URLs. If you follow a link from one of these supposedly secure sites to an external site, it may well be possible for someone with access to the logs for the external site to get access to your account on the secure site. This is probably the reason Norton Internet Security (NIS) removes the referrer on outbound web requests.

I've heard tales of fraud that occurred something like the following:

1. An unscrupulous web site operator posts a friendly comment to a poorly secured online bulletin board. His comment contains a link to his, apparently innocuous, site.
   
2. Members follow the link to the apparently innocuous site, leaving a record of their referrer URLs in the site's log.
   
3. The unscrupulous operator uses his access to the site's log to extract session information from the referrer URLs for the online bulletin board.
   
4. Using the session information, the operator accesses the bulletin board then copies e-mail addresses and passwords from the account information for members of the bulletin board who visited his site.
   
5. Using these e-mail addresses and password combinations, the unscrupulous web site operator tries them at popular online retailers such as Amazon.
   
6. If someone has used the same e-mail/password combo and chosen to leave their credit card on file with the retailer, our unscrupulous operator orders merchandise and has it delivered to some safe address.

posted by RichardP at 10:57 AM on December 18, 2004 [1 favorite has favorites]

do any server-side libraries that provide session management store a session id in the url if cookies are disabled? i vaguely remember that java's servlet code used to do this. if you were quick of the mark richardp's attack would give you access to an active session.
posted by andrew cooke at 11:59 AM on December 18, 2004

The fact that the administrator of a site wants to know what link I followed to get there implies that they very well may want to know for a non-technical reason. For example, they may want to know because:

a) If they know what other webites their users read, they can sell them products more easily.
b) They want to see if you came as a result of their advertising.
c) They want to keep track of everything that anyone is saying about their website.
d) There may be some 'unauthorized' links to the site out there somewhere...links from sites that the site in question doesn't 'approve of,' and doesn't want to be associated with, and they want to hunt down these evildoers and bring them to justice.

The idea of having one's personal preferences and habits used as fodder for a vast machine whose purpose it is to manipulate you into buying more stuff is troublesome for many people.

Perhaps just as significantly, though: Nobody asked. Nobody offered any compensation. There was no enrollment process, nowhere to sign up or not sign up to become part of this system. And I think that a lot of people resent that, and as soon as they realize what's going on, they decide that they're going to at the very least refuse to cough up any more personal information to websites unless they absolutely have to.
posted by bingo at 12:28 PM on December 18, 2004

yeah, about using the http_ref making me quite screwed, I've noticed. ;) Prebuilt stuff has to be taken apart to delete the strings that want to know referer and it's bugging me as I clean up lots of old things (I'l probably have to pop a follow up Q about modern bot prevention later as I had lots of oldfashioned stuff like that). That's why I was curious as to why people hide it,it seemed such innocent data to me, compared to say hiding your IP which I can understand a person might want to do. The "I'm on this page and I would like to comment *click*" way it can be used made sense to me... Until I got scared from reading RichardPs example there.
posted by dabitch at 12:37 PM on December 18, 2004

Just to be the voice on the other side, for one specific instance: if I have an image up on my site, and someone decides to use it as an img src a Slashdot thread, then it's my bandwidth that's being used, and I'm the one that has to pay for that. Using referrers to weed this out is a reasonable thing to do, I think. Similarly, I think that using referrer testing is one part of wise form-submission coding to prevent cross-site-scripting attacks.

Does NIS really block referrers, even for form submission? That's a bit nutty.
posted by delfuego at 12:38 PM on December 18, 2004

grouse, I really don't need to know what brought you to X page, but while you are on it the http_referer might want to know that you are actually visiting when you click to make a comment or look at images. It was used like that. So now you know. :)

gimonca idea of selective blocking is neat. Hide how you arrived, but while you are there let them know you are on X page. ;) Ah well, the hunt of other solutions is on.
posted by dabitch at 12:42 PM on December 18, 2004

I'm just wondering why people feel like they need to know the last page visited since it's none of their business and it's getting on my nerves.

Well, seeing where visitors came from is *very* useful for site owners, especially retail site owners or even content providers.

Some poorly designed web sites that are supposed to be secure put session information/security tokens/passwords in their URLs.

But this is in fact a pretty good reason to block http_referer... though it would be nicer to simply chop off the query string.
posted by namespan at 12:42 PM on December 18, 2004

delfuego try the smarter hotlink prevention tricks outlined at alistapart.
posted by dabitch at 12:47 PM on December 18, 2004

(and, sorry my comment got chopped up) yes it seems NIS does screw up forms and things like that as well. Also, it can't even visit my site and views it properly because the word "ad-" is in the URL, making it a near textless linkless page, if the NIS user is running around with a default install.
posted by dabitch at 12:49 PM on December 18, 2004

dabitch, you are aware that that article uses a method that's totally dependent on HTTP referrer, right? It just implements it at the level of the mod_rewrite rules, rather than in a scripting language like PHP.

HTTP being a stateless protocol, there's no way to do things like the hotlink prevention stuff or the cross-site-scripting exploitation prevention stuff without having SOME method of retaining state. One way is via referrers, which maintain the most minimal state you could imagine (the previous page that referred the viewer to the current page); another way is via cookies (which do a much better job of state, but which also make certain people cringe and install stuff to prevent them from being accepted); still another way is via session data on the server (e.g., PHP's session code), which still relies on either cookies or an un-mucked-with query string on URLs. But in any of the cases, you have to have some piece of data that persists from one browser view to the next.
posted by delfuego at 12:57 PM on December 18, 2004

argh, I didn't see that until after I posted. Drats. Still hunting for other solutions then.
:/
posted by dabitch at 1:03 PM on December 18, 2004

(tune in again next week when I ask about bot defeating and anti-image-leeching tricks that don't use the http_referer! Maybe.)
posted by dabitch at 3:57 PM on December 18, 2004

As a personal site blogger, I just like to see if anyone linked to one of my posts. Too bad such a harmless use gets trumped by the (very reasonable) nastiness prevention reasons.
posted by billsaysthis at 5:53 PM on December 18, 2004

daHIFI, unless you have a business relationship you value with Symantec, I think you and your customers would be better served using Zone Alarm for a firewall and AVG 7.0 Free for antivirus. NIS is nasty, intrusive and slow.
posted by flabdablet at 10:16 PM on December 18, 2004

Re: "none of your damn business"--It's not some global conspiracy, it's just part of the spec. I mean, I suppose (for example) it's none of Matt's damn business what address you're posting from, either, but whatdayaknow--he gets that information, too.
posted by RikiTikiTavi at 10:44 PM on December 18, 2004

I don't hide my Referer, I'm just choosing not to tell you where I came from. (Because, as previously mentioned, it's none of your business) There are a lot of things I can optionally provide to the server on the other end of my connection, like Refe[r]rer, client, and OS. The remote end knowing my IP address is necessary for the entire setup to work. Where I came from isn't.
posted by fvw at 12:52 AM on December 19, 2004

RikiTikiTavi: Referer is not a mandatory part of the spec like your IP address. Also, being able to turn Referer off is part of the spec too:

Because the source of a link might be private information or might reveal an otherwise private information source, it is strongly recommended that the user be able to select whether or not the Referer field is sent. For example, a browser client could have a toggle switch for browsing openly/anonymously, which would respectively enable/disable the sending of Referer and From information.

posted by grouse at 3:11 AM on December 19, 2004

Also, it actually is possible to hide your IP address from Matt, it's just a pain in the ass, and it's not likely to be worthwhile.
posted by bingo at 8:11 AM on December 19, 2004