Why Am I getting 404 errors for legitimate web pages while casually browsing?

April 12, 2008 1:58 AM

While casually surfing, why am I getting 404 errors for pages that definitely exist (like google and facebook)? How can I stop it?

While casually surfing, I get at least five Hostgator 404 errors a day, for pages that definitely exist. Pages on google, on major websites like ign.com, facebook, etc.

I'm in India, surfing on a MacBook Pro, Safari, Leopard (though I've seen it on Firefox and on OS 10.4). I think it has to do with my ISP, but I'm not sure. It could be my wireless router (Airport Express).

My annoying remedy is to change my DNS Server information to OpenDNS when a page doesn't work (208.67.222.222 and 208.67.220.220). But, I get the same 404 page errors at the same frequency.

The switching of DNS servers ALWAYS helps immediately, but never long term. That is, it'll allow me to see the page/site right away, but another legitimate page will give me a 404 some time later.

Please help me. I'm really hoping that my configurations are doing this somehow, but I don't know how.

thanks!

posted by smersh to Computers & Internet (18 answers total)

It almost certainly has to do with something dropping on your ISP. If a hop dies out and isn't very stable to your DNS, that's a plenty valid reason for the dropout, though DNS entries usually cache locally for some time.

How long has this problem been occurring? How long have you been in India? Do you live there, or have you just arrived to find this problem?

I'm experiencing something very similar right now with my connection and I'm in Phoenix. There's a signal balancing issue that causes my cable connection to shift out of range with unfortunate frequency.

I'm not sure there's something you'll be able to do on your end to ensure that it works 100% of the time, since it seems to be choking upstream. When you get these errors, are you able to connect to the Airport? Can you perform a traceroute and see where the connection is failing?
posted by disillusioned at 2:37 AM on April 12, 2008

Pro tip... full internet routing tables have grown above low-level router memory. Routes vanish on old routers. You need big routers and memory to handle full internet routes. It's hot topic, old equipment can't do current internet. Get used to it. Not going away for a while. Old stuff can't do new internet routes, not enough memory. Internet will suck for a while until people upgrade.....
posted by zengargoyle at 3:21 AM on April 12, 2008

I get at least five Hostgator 404 errors a day

You're really getting 404 pages from (the webhost) Hostgator? They should have nothing to do with it, unless all the pages you're having trouble with are hosted there (and Google, IGN, Facebook, etc aren't).

I'd be suspecting some malware, toolbar, or such which is redirecting your requests through a proxy hosted on Hostgator.

FWIW, real 404 pages are returned by the server at the remote end. Browsers often present something which looks like a 404 page on errors, but they're not. The difference is subtle, but important for diagnosing problems.
posted by Pinback at 3:27 AM on April 12, 2008

Response by poster: I've had this problem for at least a year. I live here about eight months out of the year (the other four is split between Canada and the US). The problem wasn't occurring before in India.

Can you perform a traceroute and see where the connection is failing?

I never thought to do that. I can connect to Airport with no problems.

You need big routers and memory to handle full internet routes. It's hot topic, old equipment can't do current internet. Get used to it. Not going away for a while. Old stuff can't do new internet routes, not enough memory. Internet will suck for a while until people upgrade.....

When you say "people," do you mean my ISP? Or do you mean I have to upgrade my Airport Express? It's two years old. It shouldn't be the cause.

Are you saying that a lot of people are having this problem? Why haven't I heard of it? And, why haven't I been able to find anything on google about it?
posted by smersh at 3:30 AM on April 12, 2008

Response by poster: You're really getting 404 pages from (the webhost) Hostgator?

Definitely not real 404 pages. It always has the Hostgator gator on the left hand side. I know that it isn't a real 404, because I get it when trying to access facebook and google's homepage. There's just no way one would ever get a 404 on those two pages.

I'd be suspecting some malware, toolbar, or such which is redirecting your requests through a proxy hosted on Hostgator.

This happened before and after I installed OS 10.5. I performed a clean install. If it were malware or a toolbar, it would've been wiped off. I don't mind doing a virus scan, but the chances of it being that are kind of unlikely, since the problem was happening pretty much before I reinstalled any software.

The most likely is that it's from my ISP, which might be hosted on hostgator (reliancebroadband.co.in).
posted by smersh at 3:39 AM on April 12, 2008

Hmmm... this forum post suggests there's been ongoing DNS problems in India, presumably thanks to BSNL.

Where are you changing your DNS servers - is it in your broadband router, or maybe even the Airport? Because I'm thinking maybe it works when you change to OpenDNS, but then your DNS settings are reset to your ISPs servers when the DHCP lease is renewed every few hours.

If so, try setting OpenDNS to be your DNS servers in OS X ( i.e. System Preferences - Network - <your network interface> - DNS Servers). This should stick between DHCP leases.
posted by Pinback at 4:18 AM on April 12, 2008

I've ben having remarkably similar issues down here in New Zealand. Occasional 404s on sites like BBC.co.uk. I have to wonder if it is some low level broken peering thing on the Internet at large. Of course those of us outside of the US and EU might be feeling it more because of the giant routing gateways we have to make our way through?
posted by pivotal at 4:22 AM on April 12, 2008

Can't say I've noticed anything like that from here in Aus, pivotal. Odd, though, that a traceroute to smersh's indian ISP goes out through Sydney, up through Seoul to Tokyo, across to L.A., out through NYC to London, and times out somewhere in Egypt.

I know that routing from here to the western part of Asia is weird, but that still seems like it's taking the long way around to me ;-)
posted by Pinback at 4:32 AM on April 12, 2008

Are you using Hostgator as a proxy server? If so, can you kill the proxy settings and just route request directly to other sites?
posted by mathowie at 5:50 AM on April 12, 2008

When this happens to you, try typing this in your terminal:

dscacheutil -flushcache

There's some bug in Leopard whereby if the system momentarily can't find a site, it remembers this and keeps dumping you to a server not found page. (Using openDNS, for example, it will keep sending you to the openDNS page for "I don't know where that site is", which isn't exactly a 404 page, but similar...)

The command above flushes the cache, so that the computer needs to try again to find the site. Provided that it was only a momentary glitch that made the site invisible, your computer should find it on the second try.
posted by wyzewoman at 7:20 AM on April 12, 2008

@smersh, no not your problem. It's a global thing with ISPs and old routers. I remember the day when routing tables grew bigger than old routers could handle and things got wonky. It's like old routers have 512 Mb for memory and one day last year the Internet routing table grew to 600 Mb. They update each other and sometimes drop some routes. I can't get there from here, but I can login to a machine halfway across the country and I can login from there.

Check the Widows answers, I don't do Windows. I do network around the world and the day the route tables grew above the small memory size is the day things got wonky.
posted by zengargoyle at 8:14 AM on April 12, 2008

There's some bug in Leopard whereby if the system momentarily can't find a site, it remembers this and keeps dumping you to a server not found page.

That's no bug, that's engineered into DNS. DNS records are cached to avoid having to make the same requests over and over, and those publishing DNS records specify both the TTL (time-to-live, or how long to cache the data) for each record and the "negative TTL", or the amount of time to cache answers where were "that record does not exist".

DNS caching doesn't explain the Hostgator part to me.
posted by mendel at 8:49 AM on April 12, 2008

Response by poster:

Hmmm... this forum post suggests there's been ongoing DNS problems in India, presumably thanks to BSNL.

Very interesting. Same exact problem, different ISP.

Where are you changing your DNS servers - is it in your broadband router, or maybe even the Airport?

I had nothing put into my Airport Express for DNS Servers. Reliance put their own in there.

There's no other router; at least not accessible by me. All OpenDNS server info was going straight into my laptop.

I'm going to try and see what happens when I put it into Airport Express, since the other way around doesn't work consistently.

And, to be clear, I use nothing from Hostgator. The fact that Hostgator shows up is very odd to me.

Can't say I've noticed anything like that from here in Aus, pivotal. Odd, though, that a traceroute to smersh's indian ISP goes out through Sydney, up through Seoul to Tokyo, across to L.A., out through NYC to London, and times out somewhere in Egypt.

Why am I not surprised?
posted by smersh at 9:32 AM on April 12, 2008

Response by poster: When this happens to you, try typing this in your terminal:

Thanks for this suggestion. If it works, it'll at least be faster than adding/remove DNS server info.
posted by smersh at 9:34 AM on April 12, 2008

Here's a description of the issue with Leopard (taken from this discussion):


Basically, it seems that Apple has attempted in Leopard to introduce caching of negative DNS responses per RFC2308. If you read the RFC you will see that although caching of negative results is recommended, the TTLs for some classes of failure should be short.

Unfortunately, it looks like Apple are not setting the TTLs in the cache entries so they become the default (for the directory cache) 3600 seconds.

What this means is that once a DNS lookup fails for a host, even if this is because the DNS server was slow or unreachable, any subsequent lookups will fail for the next hour. So if your network/internet connection is dodgy, or you are using an internal DNS server which you have just upgraded and so are rebooting a lot, all sorts of services will hang up or fail because of the cached negative DNS responses.
posted by wyzewoman at 10:11 AM on April 12, 2008

Useful website: downforeveryoneorjustme.com
posted by lunchbox at 11:11 AM on April 12, 2008

Response by poster: In case anyone's interested, I've done two things that seem to have worked. At least, up till now.

1. Using a program called Main Menu for OS X, I flushed the DNS Cache.

2. I also put the OpenDNS server info directly into my Airport Express.

Under normal circumstances, I would've experienced at least ten 404 errors in the past two days. I have experience ZERO.

I'm going to leave it like this for a week, and then remove the OpenDNS info to see if that makes a difference.

Thanks for all the suggestions.
posted by smersh at 11:43 AM on April 14, 2008

Response by poster: I know this is an old topic, but for anyone who finds it in a search, you'll thank me for the update.

Here's what I do now when I encounter this problem. It's far less frequent, but still happens.

1. Download MainMenu (it's free).
2. When you get the error, go to Other, Flush the DNS Cache.
3. Empty Safari's cache (or whatever browser you're using)
4. Retry the site.
5. Profit
posted by smersh at 5:31 AM on March 16, 2009