Comments on: Why Am I getting 404 errors for legitimate web pages while casually browsing?

Question: Why Am I getting 404 errors for legitimate web pages while casually browsing?

smersh — Sat, 12 Apr 2008 01:58:12 -0800

While casually surfing, why am I getting 404 errors for pages that definitely exist (like google and facebook)? How can I stop it?

While casually surfing, I get at least five Hostgator 404 errors a day, for pages that definitely exist. Pages on google, on major websites like ign.com, facebook, etc.

I'm in India, surfing on a MacBook Pro, Safari, Leopard (though I've seen it on Firefox and on OS 10.4). I think it has to do with my ISP, but I'm not sure. It could be my wireless router (Airport Express).

My annoying remedy is to change my DNS Server information to OpenDNS when a page doesn't work (208.67.222.222 and 208.67.220.220). But, I get the same 404 page errors at the same frequency.

The switching of DNS servers ALWAYS helps immediately, but never long term. That is, it'll allow me to see the page/site right away, but another legitimate page will give me a 404 some time later.

Please help me. I'm really hoping that my configurations are doing this somehow, but I don't know how.

thanks!

By: disillusioned

disillusioned — Sat, 12 Apr 2008 02:37:12 -0800

It almost certainly has to do with something dropping on your ISP. If a hop dies out and isn't very stable to your DNS, that's a plenty valid reason for the dropout, though DNS entries usually cache locally for some time.

How long has this problem been occurring? How long have you been in India? Do you live there, or have you just arrived to find this problem?

I'm experiencing something very similar right now with my connection and I'm in Phoenix. There's a signal balancing issue that causes my cable connection to shift out of range with unfortunate frequency.

I'm not sure there's something you'll be able to do on your end to ensure that it works 100% of the time, since it seems to be choking upstream. When you get these errors, are you able to connect to the Airport? Can you perform a traceroute and see where the connection is failing?

By: zengargoyle

zengargoyle — Sat, 12 Apr 2008 03:21:34 -0800

Pro tip... full internet routing tables have grown above low-level router memory. Routes vanish on old routers. You need big routers and memory to handle full internet routes. It's hot topic, old equipment can't do current internet. Get used to it. Not going away for a while. Old stuff can't do new internet routes, not enough memory. Internet will suck for a while until people upgrade.....

By: Pinback

Pinback — Sat, 12 Apr 2008 03:27:36 -0800

I get at least five Hostgator 404 errors a day

You're really getting 404 pages from (the webhost) Hostgator? They should have nothing to do with it, unless all the pages you're having trouble with are hosted there (and Google, IGN, Facebook, etc aren't).

I'd be suspecting some malware, toolbar, or such which is redirecting your requests through a proxy hosted on Hostgator.

FWIW, real 404 pages are returned by the server at the remote end. Browsers often present something which looks like a 404 page on errors, but they're not. The difference is subtle, but important for diagnosing problems.

By: smersh

smersh — Sat, 12 Apr 2008 03:30:08 -0800

I've had this problem for at least a year. I live here about eight months out of the year (the other four is split between Canada and the US). The problem wasn't occurring before in India.

Can you perform a traceroute and see where the connection is failing?

I never thought to do that. I can connect to Airport with no problems.

You need big routers and memory to handle full internet routes. It's hot topic, old equipment can't do current internet. Get used to it. Not going away for a while. Old stuff can't do new internet routes, not enough memory. Internet will suck for a while until people upgrade.....

When you say "people," do you mean my ISP? Or do you mean I have to upgrade my Airport Express? It's two years old. It shouldn't be the cause.

Are you saying that a lot of people are having this problem? Why haven't I heard of it? And, why haven't I been able to find anything on google about it?

By: smersh

smersh — Sat, 12 Apr 2008 03:39:43 -0800

You're really getting 404 pages from (the webhost) Hostgator?

Definitely not real 404 pages. It always has the Hostgator gator on the left hand side. I know that it isn't a real 404, because I get it when trying to access facebook and google's homepage. There's just no way one would ever get a 404 on those two pages.

I'd be suspecting some malware, toolbar, or such which is redirecting your requests through a proxy hosted on Hostgator.

This happened before and after I installed OS 10.5. I performed a clean install. If it were malware or a toolbar, it would've been wiped off. I don't mind doing a virus scan, but the chances of it being that are kind of unlikely, since the problem was happening pretty much before I reinstalled any software.

The most likely is that it's from my ISP, which might be hosted on hostgator (reliancebroadband.co.in).

By: Pinback

Pinback — Sat, 12 Apr 2008 04:18:52 -0800

Hmmm... this forum post suggests there's been ongoing DNS problems in India, presumably thanks to BSNL.

Where are you changing your DNS servers - is it in your broadband router, or maybe even the Airport? Because I'm thinking maybe it works when you change to OpenDNS, but then your DNS settings are reset to your ISPs servers when the DHCP lease is renewed every few hours.

If so, try setting OpenDNS to be your DNS servers in OS X ( i.e. System Preferences - Network - <your network interface> - DNS Servers). This should stick between DHCP leases.

By: pivotal

pivotal — Sat, 12 Apr 2008 04:22:08 -0800

I've ben having remarkably similar issues down here in New Zealand. Occasional 404s on sites like BBC.co.uk. I have to wonder if it is some low level broken peering thing on the Internet at large. Of course those of us outside of the US and EU might be feeling it more because of the giant routing gateways we have to make our way through?

By: Pinback

Pinback — Sat, 12 Apr 2008 04:32:48 -0800

Can't say I've noticed anything like that from here in Aus, pivotal. Odd, though, that a traceroute to smersh's indian ISP goes out through Sydney, up through Seoul to Tokyo, across to L.A., out through NYC to London, and times out somewhere in Egypt.

I know that routing from here to the western part of Asia is weird, but that still seems like it's taking the long way around to me ;-)

By: mathowie

mathowie — Sat, 12 Apr 2008 05:50:46 -0800

Are you using Hostgator as a proxy server? If so, can you kill the proxy settings and just route request directly to other sites?

By: wyzewoman

wyzewoman — Sat, 12 Apr 2008 07:20:37 -0800

When this happens to you, try typing this in your terminal:

dscacheutil -flushcache

There's some bug in Leopard whereby if the system momentarily can't find a site, it remembers this and keeps dumping you to a server not found page. (Using openDNS, for example, it will keep sending you to the openDNS page for "I don't know where that site is", which isn't exactly a 404 page, but similar...)

The command above flushes the cache, so that the computer needs to try again to find the site. Provided that it was only a momentary glitch that made the site invisible, your computer should find it on the second try.

By: zengargoyle

zengargoyle — Sat, 12 Apr 2008 08:14:51 -0800

@smersh, no not your problem. It's a global thing with ISPs and old routers. I remember the day when routing tables grew bigger than old routers could handle and things got wonky. It's like old routers have 512 Mb for memory and one day last year the Internet routing table grew to 600 Mb. They update each other and sometimes drop some routes. I can't get there from here, but I can login to a machine halfway across the country and I can login from there.

Check the Widows answers, I don't do Windows. I do network around the world and the day the route tables grew above the small memory size is the day things got wonky.

By: mendel

mendel — Sat, 12 Apr 2008 08:49:09 -0800

There's some bug in Leopard whereby if the system momentarily can't find a site, it remembers this and keeps dumping you to a server not found page.

That's no bug, that's engineered into DNS. DNS records are cached to avoid having to make the same requests over and over, and those publishing DNS records specify both the TTL (time-to-live, or how long to cache the data) for each record and the "negative TTL", or the amount of time to cache answers where were "that record does not exist".

DNS caching doesn't explain the Hostgator part to me.

By: smersh

smersh — Sat, 12 Apr 2008 09:32:31 -0800

Hmmm... this forum post suggests there's been ongoing DNS problems in India, presumably thanks to BSNL.

Very interesting. Same exact problem, different ISP.

Where are you changing your DNS servers - is it in your broadband router, or maybe even the Airport?

I had nothing put into my Airport Express for DNS Servers. Reliance put their own in there.

There's no other router; at least not accessible by me. All OpenDNS server info was going straight into my laptop.

I'm going to try and see what happens when I put it into Airport Express, since the other way around doesn't work consistently.

And, to be clear, I use nothing from Hostgator. The fact that Hostgator shows up is very odd to me.

Can't say I've noticed anything like that from here in Aus, pivotal. Odd, though, that a traceroute to smersh's indian ISP goes out through Sydney, up through Seoul to Tokyo, across to L.A., out through NYC to London, and times out somewhere in Egypt.

Why am I not surprised?

By: smersh

smersh — Sat, 12 Apr 2008 09:34:22 -0800

When this happens to you, try typing this in your terminal:

Thanks for this suggestion. If it works, it'll at least be faster than adding/remove DNS server info.

By: wyzewoman

wyzewoman — Sat, 12 Apr 2008 10:11:48 -0800

Here's a description of the issue with Leopard (taken from this discussion):

Basically, it seems that Apple has attempted in Leopard to introduce caching of negative DNS responses per RFC2308. If you read the RFC you will see that although caching of negative results is recommended, the TTLs for some classes of failure should be short.

Unfortunately, it looks like Apple are not setting the TTLs in the cache entries so they become the default (for the directory cache) 3600 seconds.

What this means is that once a DNS lookup fails for a host, even if this is because the DNS server was slow or unreachable, any subsequent lookups will fail for the next hour. So if your network/internet connection is dodgy, or you are using an internal DNS server which you have just upgraded and so are rebooting a lot, all sorts of services will hang up or fail because of the cached negative DNS responses.

By: lunchbox

lunchbox — Sat, 12 Apr 2008 11:11:50 -0800

Useful website: downforeveryoneorjustme.com

By: smersh

smersh — Mon, 14 Apr 2008 11:43:52 -0800

In case anyone's interested, I've done two things that seem to have worked. At least, up till now.

1. Using a program called Main Menu for OS X, I flushed the DNS Cache.

2. I also put the OpenDNS server info directly into my Airport Express.

Under normal circumstances, I would've experienced at least ten 404 errors in the past two days. I have experience ZERO.

I'm going to leave it like this for a week, and then remove the OpenDNS info to see if that makes a difference.

Thanks for all the suggestions.

By: smersh

smersh — Mon, 16 Mar 2009 05:31:36 -0800

I know this is an old topic, but for anyone who finds it in a search, you'll thank me for the update.

Here's what I do now when I encounter this problem. It's far less frequent, but still happens.

1. Download MainMenu (it's free).
2. When you get the error, go to Other, Flush the DNS Cache.
3. Empty Safari's cache (or whatever browser you're using)
4. Retry the site.
5. Profit