Why does Firefox ignore my hosts file

February 10, 2006 12:25 PM

Can anyone help me figure out why Firefox is ignoring my HOSTS file when I use a proxy server? Win XP/Firefox 1.5

I'm a web developer. I'm required to connect to the internet through a proxy at work. I also have to make changes to my hosts file in order to view the different dev/stage/live servers.

Let's say I have the following two entries in my hosts file

192.168.0.100 example.com
192.168.0.101 example.org

If I have my box/browsers setup to connect directly to the Internet, both Firefox and IE will obey the settings in my hosts file. Of course, since I'm not going through a proxy, I can't connect to anything outside the local network.

So, I setup the box/browsers to use the proxy. Here's where things get weird.

IE will continue to obey the HOSTS file, assuming I add the domain to the "Do not use proxy server for addresses beginning with:" setting.

Firefox, by and large, will also obey settings in the HOSTS file (again, assuming I've added the domain to the "No Proxy For" setting). However, there are certain domains where it will not. So, in the above example, when I'm set to connect through a proxy, Example.com would resolve to 192.168.0.101, but example.org would ignore the setting in the hosts file, and resolve to something else (192.0.34.166).

Digging through Bugzilla, I came across this bug, which seems to be related, but I don't know enough about how proxy servers should work vs. how they do work vs. how Mozilla makes them work to fix my issue.

I assume that there's something set on the proxy server that says "hey, it's important that we look this domain up, so ignore local settings", but again my knowledge of what's really going on behind the scenes it too limited to make any use of this theroy.

Has anyone run into this problem with Firefox before? Can anyone recommend tools/procedures for further debugging of proxy server issues in general?

Standard Disclaimers

Yes I've cleared my cache.

Yes I've tried restarting Firefox between proxy changes.

I love reading RFCs. I love practical advice that explains the concepts laid out in RFCs even more.

Yes, the Help Desk/IT Department should be handling this. Their response "oh, we have limited support for Firefox, sorry", (Despite the fact it's a development priority). I'd like to diagnose the problem to the point where I can say to them "I need you to do X, Y and Z".

Yes, the editing of host files is a questionable way to deal with dev/stage/live environments. While suggestions of alternatives are appreciated, I'm months away from having the political power to recommend changes likes that.

Large companies are fun, eh?

posted by alan to technology (11 comments total)

Theoretically, if you are telling firefox to not use the proxy server for certain domains, then it should not be a problem with the proxy server. However, there may be a problem where firefox IS sending the request to the proxy server - either a genuine bug or you have not specified the domain properly.
If the request goes to the proxy server, the proxy server is going to use whatever is in DNS -- it does not have access to what is in your hosts file, and when a request is sent to a proxy, the request gets looked up by the proxy, NOT your browser. That is, firefox sends a request to the proxy that looks like
GET http://www.example.com/mypage.html
not
GET http://192.168.0.12/mypage.html

What I'd check is whether or not your host entries are correct. Try, for example, pinging both example.com and example.org, and see if they attempt to ping the addresses you're looking for. Or, say, telnet example.com 80 and telnet example.org 80 and make sure it's going to the right place.
posted by RustyBrooks at 12:40 PM on February 10, 2006

Browsers are supposed to ignore the hosts file when you use a proxy. This is because they are communicating with the proxy not the final host. Basically, the browser is telling the proxy, 'connect to www.example.com', and the proxy then uses its host file first to resolve www.example.com. IE works differently, because if a proxy returns

OTOH, if you add something to the ignore proxy list in Firefox, it should connect directly and use your hosts file. It was not immediately clear, but it sounds like you need to make sure that all the domains you're trying to use your hosts file for are also listed in the no proxy settings in Firefox.
posted by boaz at 12:42 PM on February 10, 2006

On a related note, I wrote my own proxy server for debugging stuff like this. I run it on my local machine and point my browser to it. I have the proxy server dump everything it sends and receives to a file. This is great not only for finding out whether or not firefox is going to the proxy server or not, but also to find out EXACTLY what firefox is sending, and what it's getting back. This is great for decoding complicated form submissions, esp ones that involve arcane javascript.

I don't have a windows version of the proxy server. I might could build one, but you might also be able to just find a windows proxy server online with good debugging. Apache can be used as a proxy server, so that comes to mind.
posted by RustyBrooks at 12:42 PM on February 10, 2006

err, ignore that IE works differently section; it does behave a bit differently but it's not really relevant.
posted by boaz at 12:43 PM on February 10, 2006

From the bug, it sounds like the problem is the handling of proxied addresses when an IP is used to specify the do-not-proxy locations. I would make sure that in your do-not-proxy field you specify the names, not the IPs, i.e.:

.example.org, .example.com

Another thing you might try is setting up Squid (or some other proxy server), using that as your proxy, and turning on verbose/debug mode.
posted by brool at 12:50 PM on February 10, 2006

Thanks for the overview of what's supposed to happen with proxies.

Boaz, how does IE work differently? It may not be relevant, but I'm interested in, well, knowing these things. (Plus it may shed some light on things...they're very "Microsoft is computing" tech centric around here)

1. Entries in the hosts files are correct. Pinging brings up the expected IP addresses.

2. The domains I'm using are all in the No Proxy field.

I ran through another simplified example before responding, and came up with slightly different results.

Two entries in the hosts files
#note, example.com is not the actual domain,
#but one of the problem domains
127.0.0.1 google.com
127.0.0.1 stc.example.com

Two domains in the No Proxy field
#note, example.com is not the actual domain,
#but one of the problem domains
No Proxy For: google.com,stc.example.com


When "Directly Connected to the Internet", both domains will resolve to 127.0.0.1 in Firefox. Pinging both addresses confirms they're both resolving to 127.0.0.1.

When I manually set Firefox to use the proxy, and set the proxy for my computer with the Firewall client, both domain names ignore the hosts file. Flushing the DNS cache (ipconfig /flushdns) and pinging both addresses confirms they're both still resolving to 127.0.0.1.

So, it looks like I was making up that bit about it only happening for certain domains.

Since we've ruled out the proxy server being able to "override" anything, is there anything in Windows/Firefox that could make Firefox ignore what's in the No Proxy field?

Thanks again for the help/lesson.
posted by alan at 1:20 PM on February 10, 2006

Just on the off chance, are you entering the No Proxies entries in Firefox's connection settings? FireFox ignores the settings in the Internet control panel; you have to enter the No Proxy sites in Firefox's Connection Settings dialog.

Boaz, how does IE work differently? It may not be relevant, but I'm interested in, well, knowing these things.

I was really just thinking about IE's handling of simple names (i.e. no dots, just google) vs. Firefox. IIRC, if a proxy returns not found for a simple name, IE will also try www.{simplename}.com just like it would if directly connected, but FF will just display the proxy's 404 page. Not really related at all.
posted by boaz at 1:49 PM on February 10, 2006

Yes, I'm using Firefox's connection settings.

Guess I'll be installing that test proxy server and learning more about networking than reasonable humans should have to ;)
posted by alan at 1:52 PM on February 10, 2006

Also, is 'stc.example.com' the full domain? If you want FireFox to ignore, say, 'www.stc.example.com' or 'anythingelse.stc.example.com', you need to prepend your ignore entry with a period, like '.stc.example.com'. Otherwise, it'll only ignore the exact domain name you entered.
posted by boaz at 2:13 PM on February 10, 2006

I don't have a fix for the HOSTS file problem, but I do have a workaround that doesn't involve running your own proxy server and calling down the wrath of the IT department. The "proxy auto-configuration" file in every browser these days is actually a JavaScript file, which supplies functions that the browser uses to decide whether to proxy a given request. There's a description of the functions at Netscape.

Basically, you could make a function like:

function FindProxyForURL(url, host) {

  if (dnsDomainIs(host, '.example.com')) {

    return 'DIRECT';

  } else {

    return 'PROXY my.proxy.internal';

  }

}

Of course, you'll probably want a little more flexibility - the page linked above has a ton of example files. This should give you as much control as you need over proxy setup.
posted by pocams at 8:05 PM on February 10, 2006

That's really interesting pocams, I'll definitely check it out.

Plus, reading that old school Netscape page is great. The Navigator has a posse.
posted by alan at 8:57 PM on February 10, 2006