Why would my ISP routing Google traffic differently?
April 22, 2013 8:47 PM   Subscribe

Why would my ISP routing Google traffic differently? What is traffic-shaping, and why is it considered "questionable" for an ISP to do this? I'm on Vancouver Island, in Canada. My ISP is Shaw, and everyone on Shaw on Vancouver Island experienced some issues with Google domains including YouTube and G+ for most of the day. Here's a thread where users are discussing the problem. In the thread I linked above, some folks are saying that there must have been either a critical failure, or "they're traffic-shaping or otherwise filtering results from Google-based websites. That's highly questionable behaviour." What does this mean? While I had internet connectivity, I couldn't access Gmail or Google Search, and ended up using my Telus phone's WiFi hotspot to work (we all work remotely and use Google Apps).
posted by KokuRyu to Computers & Internet (13 answers total) 2 users marked this as a favorite
 
I've been reading a lot about ISP's doing local caching of youtube, but it seems insane that they'd do it for google, particularly since google is now https by default now.
posted by empath at 8:53 PM on April 22, 2013


Traffic shaping is one method of rate-limiting (deliberately slowing down) a particular flow of network traffic. It would be "questionable" for Shaw to do that because of the whole network neutrality thing -- in general, we want our ISPs to be neutral in their forwarding of our traffic, not to put special rules in place for traffic to Google.

However, it would surprise me greatly if Shaw was doing that. Without having access to information about the innards of Shaw's network, I'd speculate that what happened is that the path that traffic between Shaw and Google normally takes failed, causing the network to make use of an alternate path that had less capacity. This would result in random failures of Google-based applications while traffic with the rest of the Internet would be normal.
posted by bac at 9:13 PM on April 22, 2013 [1 favorite]


Most ISP's do everything they can to make google go faster. It uses a tiny amount of bandwidth per customer (aside from youtube). They'd be insane to try and rate limit it.
posted by empath at 9:20 PM on April 22, 2013


Not only do ISPs do everything they can to make Google go faster, but Google does everything they can too. I'm wondering if there are some wonky BGP rules in place that are sending traffic down a path that's normally optimized but is less than awesome at the moment.

You can figure out if there's a "bad" path between you and Google by comparing traceroutes between you and Google, and then between you and some site that's working "right". My guess is that they're taking different paths, and that the path is the problem, not Google or your ISP. My guess is also that you're noticing Google because you all go to Google.

We can tell when a certain upstream provider that shall remain nameless (rhymes with bevel tree) is having a bad day when the admins can't get to ESPN in a timely manner. It's so much like clockwork that we added a monitor for it--and some similar cases--into our monitoring system under the title of "Canaries." Sadly, we still usually notice it before the monitoring system...
posted by togdon at 10:00 PM on April 22, 2013 [2 favorites]


I live in Northern Idaho and we can't get on Gmail or Google search today, all day, and we have satellite Internet through HughesNet. Everything else works for me. This happened about a week ago and I used Ccleaner, Cleared my cache and unplugged my modem for a minute. Then I could get back on but no way to tell if it was coming back on anyway. And I don't feel like doing all that again so hoping someone has an answer.
posted by cda at 10:25 PM on April 22, 2013


You shouldn't notice traffic shaping, except that certain services might be slower. It happens in the router, not by routing to a different place.

My guess would be that somebody's peering connection with one of Google's datacenters went down, and its traffic is getting routed somewhere else as a backup measure.
posted by gjc at 4:34 AM on April 23, 2013


End users often confuse network problems for traffic shaping. End users tend to think of their Internet connection as a hose that goes through their ISP and connects up to a big "Lake Internet", and as such, if the ISP were to be messing with the flowing water in the hose, yes, that'd be bad.

However, this isn't how the Internet works.

Your ISP is a very large network, with many bits and pieces to make it all work, including routers, wide area network connections, peerings with other large networks, and transit connections (think: wholesale ISP's, in the business called "Tier 2" or "Tier 1" providers). In many cases, your traffic does get moved from your little DSL or cable connection to one of their transit connections, but your ISP has several or many of these, and it sometimes happens that one of them is full (resulting in slower traffic), or a router that handles that traffic is having problems (resulting in packet loss, which results in slower traffic), or any of a number of other things that could go wrong. Traffic to different destinations may go via different paths, so most of the Internet may seem quite fine.

In the specific case of a big content provider such as Google, both your ISP and Google may wish to avoid paying an intermediate third party (that "wholesale ISP") for traffic. In such cases, the two networks can agree to "peer" and run a direct connection from one net to the other, which allows them to trade traffic directly. These add complexity to networks, and yet another thing that can go wrong, but the financial incentive to avoid paying a third party for the traffic can be great. The total peering capacity between an ISP and a content provider is probably not as large as the pipes from the ISP to their transit providers, though, so sometimes a peering link will become saturated or suffer from other issues.

Anyways, an ISP's network is a very complicated thing, and ISP's do their best to monitor performance and make sure things work well, but it is extremely difficult to guarantee that nothing ever goes wrong. An incorrect route that gets into the routing database somehow, a bit of equipment getting close to failure because a fan stalled and it is now baking, a backhoe cutting through a buried fiber optic cable, failures are daily events in the life of service providers. The fact you don't notice most of them is a remarkable testament to the hard work of network operations staff and a plethora of mitigation strategies that generally work.

So, you know, do your ISP a favor and don't assume the worst. The best thing you can do is to call them and let them know you're having trouble reaching Google. It's likely that they'll already know, and may (or may not) have further details, in which case you should clear off the line because the call center staff is probably already fielding calls at a greater-than-usual rate. However, it is also possible that it is a problem that is local to your regional CMTS or DSLAM, and if no automated trouble detection system has picked it up, one of the last ditch methods ISP's use for problem detection is users calling in to complain. Be polite, let them walk through all the "reboot your modem" and "reboot your PC" garbage, and work with them. The first few people to report a problem are often met with a bit of natural skepticism, but good call centers spot unusual trends very quickly.
posted by jgreco at 4:38 AM on April 23, 2013 [2 favorites]


Well, be happy you aren't using Comcast. My brother had a new modem installed that is transparently redirecting DNS requests through Comcast (even if his computer is set to use Google DNS, requests go through Comcast). Searches don't work because everything is directed to a Comcast page that doesn't exist. He gets a 403 error (access denied) when he tries to access the Comcast site itself, so he couldn't even check his own customer settings. Some sites work if typed in manually, but most don't, and ones that work initially stop working if a link is clicked. He can't even update his web browser. The install tech changed his system to use Comcast search instead of Google, and when he noticed and changed it back, all of this happened. I connected to his computer remotely (INCOMING connections work fine!), ensured that his hosts file wasn't corrupted, did a Malware Bytes scan (that found nothing at all), and set his browser up to use an SSH SOCKS proxy through a server I knew to be reliable. And even then he had multiple sites that just didn't load; secure SSH tunneled communication was being blocked or redirected, which is something I have never seen before anywhere on a non-compromised computer. The ones that DID load confirmed his system was running through the tunnel. It just blew me away how creepy the whole thing was... They are trusting that their customers are not savvy enough to realize that Comcast has taken over their search engine, and if the end user changes things back to their desired search, they end up being held hostage to the company. Use our services or don't use our network, is what it seems to be saying. If not for the issues with the SSH tunnel I'd leave it up to incompetence, but damn. Never underestimate the evil or ill-will of a cable company.
posted by caution live frogs at 4:47 AM on April 23, 2013


caution live frogs- that's not how Comcast does it in my area, and probably not anywhere. What you are experiencing is that the modem was not provisioned properly, possibly combined with some sort of malfunction on their end.

What happens is that their network allows new modems to be connected to the network, but routes all traffic to a Comcast page telling you to register your modem. As your friend is experiencing. Once a modem is properly registered, everything goes back to normal. The setup technician did not do his job correctly.
posted by gjc at 7:25 AM on April 23, 2013


The Comcast DNS redirection thing, "Domain Helper", is real, completely idiotic, and can be easily disabled via their control panel.
posted by togdon at 8:19 AM on April 23, 2013


As several answers allude to above, Google itself may be responsible for the problem. Google is very aggressive with making deals with ISPs to privately route Google traffic. It's a great thing for users; they're typically directed to a datacenter that's nearby and fast. It's also a great thing for Google, both in flexibility and cost savings. But occasionally something may go wrong and it's not hard to imagine a situation where everything on the Internet works except Google. For a big ISP like Shaw I'd expect the problem to be noticed and fixed in under an hour; if it's longer I'd call the ISP support line.

"Traffic shaping" and "filtering" is something some evil ISPs do, where they meddle with traffic to a site like Google without the site's permission. Maybe they think they can cache Youtube videos better than Google can, maybe they're threatening Netflix with a degraded customer experience, or in some extreme historical examples maybe the ISP is injecting their ads in another site's pages. Comcast is notorious for doing this kind of crap, I don't know the history with Shaw.

If you ever want to diagnose a network problem like this, the ICSI Netalyzr is a good place to start. Note it requires Java in the browser, something that's increasingly a problem.
posted by Nelson at 8:34 AM on April 23, 2013 [1 favorite]


Not trying to derail, but it wasn't just a screwed up modem. On the same network (connected via WiFi) his iPhone worked just fine. No connectivity issues. And he had exactly zero problems with his Internet connectivity on the desktop until he noticed his default search provider had been changed; when he switched it to Google, the problem immediately began. Things worked for about a week before he noticed the search provider had been changed. And still, a secure SSH tunneled connection should not have had any issues - all traffic, including DNS requests, should be going through the SOCKS port, but it wasn't.

Transparent DNS intercepts are a thing. And Comcast has been doing this in some markets. We noticed it because they are apparently really bad at it.

posted by caution live frogs at 4:56 AM on April 24, 2013 [1 favorite]


Response by poster: Hi everyone, thanks for the responses. I appreciate it. I can't really assign a best answer or mark as resolved, as the ISP has not said what caused the problem, and likely never will. However, other people in the ISP's forum remarked about what great info this is.
posted by KokuRyu at 9:55 AM on April 25, 2013


« Older Why did I lose my faith? How do I find something...   |   Simplest video card with DVI out for PC desktop? Newer »
This thread is closed to new comments.