Gauging Bandwidth Use

May 12, 2007 3:50 AM

I'm considering downgrading my cable broadband from 512kbps-800MB/month to 256kbps-uncapped. I'd like to estimate the real-world hit in terms of casual web browsing and more intensive applications.

So, I downloaded FreeMeter, a bandwidth monitor. The baseline downstream (in absence of any web clients, even hidden ones like AV updaters) registers at 120-130kbps!! How come? When requesting a page, say metafilter.com, via Firefox, the downstream shows a few large spikes of 600-800kbps, a few small spikes of 300-400kbps and then settles down to baseline. Mostly as expected. But this does make me wonder on how exactly does an ISP throttle bandwidth. A few weeks earlier, I had downloaded a large torrent from mostly European peers, and to my surprise, the average speed amounted to ~192KB/s i.e. 1536kbps. So, if I switch to 256kbps, what's likely to be the real-world difference and how is it enforced? Three different scenarios would be regular browsing i.e. email, metafilter..etc, multimedia browsing i.e. Youtube, radio..etc and finally, large data downloads.

Note that this is in Mumbai, India, and as per government regulations (TRAI), the ISP "must" guarantee 80% of the advertised speed.

posted by Gyan to Computers & Internet (10 answers total)

Wow, is that ever a complex question.

First: in TCP, latency matters at higher bitrates. Being in India, you're probably about as far from the center of the Net as you can get. Your bandwidth is likely perfectly good, but it'll take quite awhile for your signals to go halfway around the world and then back again, when you're talking with US servers, or half that far for Europe. This impacts your TCP transmission speed. TCP has a 'window' of sent packets, meaning that only a certain number can be 'in flight' at once; if your ACK packets take a long time to get back, that throttles your available bandwidth. You're probably not fast enough to seriously notice the lag time yet in REAL use -- I'd expect you could pull 4 or 5 megabits from any server on the planet -- but bandwidth meters do funny things. It's entirely possible that you're getting bad readings because of that.

Packet loss is also critically important; if you have high packet loss, it doesn't matter what your provisioned speed is; it'll be slow. And packet loss varies per destination; it depends on your ISP's links to the outside world. Usually, some are better than others. Ones with significant packet loss will seriously impair speeds.

Basically, I just wouldn't worry too much about Speedmeter. My connection here is 10 megabits down/1 megabit up, but the SiSoft Sandra benchmarks tell me I have 90kbytes of usable bandwidth. I KNOW perfectly well that I can download 1000kbytes/sec, as I've done it many, many times. So don't pay too much attention to any one benchmark.

Bittorrent is probably finding peers that are closer based on who's giving it the most data, so it'll tend to converge on the best possible peers for your situation. If you're downloading from other machines in India, you'll get better throughput... particularly on your same ISP, because the bandwidth doesn't have to transition through the (expensive to the ISP) links to other networks.

Usually, as far as enforcement goes, they'll generally let you burst a little over your limit, and then clamp down if you keep it up for more than a few seconds. So you'll briefly go much faster than your rated speed, and then drop back to the exact speed you're allowed to have. This lets you burst in Web pages and the like; it improves interactivity. How big the burst can be is dependent on your ISP.

I'm not sure why you were getting such good results with bittorrent on a theoretically-512k connection, though. That's a little confusing to me. Normally that speed is measured and throttled right at your modem, so your provider may have mis-provisioned you, or perhaps they're doing rate limiting another way. It might be that you're only rate-limited to machines not on the same ISP. I've never seen anyone do it that way, but I've never been in India either. :)

If your ISP is really limiting you to 512, then dropping to 256 will roughly double your Youtube and big file transfer times. You probably won't notice too much impact on web pages; those usually will burst in before the rate limits hit. Big web pages with lots of data will probably be slower than what you have now.

If it were me, I'd make the switch in a heartbeat; I can download 800mb without even thinking about it. At 256k, I think it'll take about four hours of solid use to hit 800mb.... roughly two at 512. I'd personally rather be able to actually use my link to its full capacity, even if it's slower, than throttle it to 1/360th of its true speed. I'd be willing to trade burst speed for more real bandwidth; without metering, you should be able to download about 180 times more data in a month than you can now. You'll just have to wait longer for it.

One other possibility: your ISP may sell you the service at 512, but physically provision you far higher, and then just zap you for bandwidth. If that's the case, you'll notice a much larger speed hit. I have no way to tell whether or not that's true.

If you can, I'd suggest talking to a customer service rep; they might be willing to tell you what you're actually provisioned for.
posted by Malor at 4:34 AM on May 12, 2007

Response by poster: Malor: Normally that speed is measured and throttled right at your modem, so your provider may have mis-provisioned you, or perhaps they're doing rate limiting another way.

Unlike in the US, there's no cable modem in my apartment. There seems to be a single modem (& router) for the entire building, and I just get a (very long) RJ-45 to plug into my adapter (well, that's how it works here :). I use a client to log in & out.

your ISP may sell you the service at 512, but physically provision you far higher, and then just zap you for bandwidth.

I don't understand the latter part.

they might be willing to tell you what you're actually provisioned for.

Trust me, they won't know :) At least, not the people available by phone.
posted by Gyan at 9:52 AM on May 12, 2007

Also to add a little about bit torrent and throttling:

Most ISP's often implement some kind of packet based throttling through the use of traffic shaping. This allows them to try and optimize overall throughput on their networks by evenly distributing bandwidth among all types of packets. Depending on how deeply their traffic shaper inspects each packet, this can still effect encrypted bit torrent data. However, shaping (mostly) only applies to data travelling outside a particular network, so like Malor said, if you have peers on the same torrent inside the same network, your download speed will be optimal. If you are trying to connect to peers outside of your network, you will be subject to shaping.

Additionally, your ISP may cap the amount of simultaneous connections you can make outside of the network, so the speed of the torrent is limited by a lack of peers to trade with. Here in Canada, Rogers Cable has recently put in this type of infrastructure, and it makes seeding or downloading more than a few things at once difficult. To get around the shaper, many people have been using VPNs to mask their traffic, since VPN packets are strongly encrypted and not shaped. But again, since this only applies to outside traffic, if you have peers close to you, your download speed will still be optimal (in your case, really optimal).

So what I'm saying here is that bit torrent isn't the greatest measure of connection speed, because if you ask me whether it'll be good or not, the best I can say is "it depends".

Finally, although it sounds like a raw deal, please don't feel disenfranchised by p2p throttling. In all likelihood, your ISP is doing the things that it does in order to maximize the amount of throughput they can wring out of their link, and so make things better for most people. To give you a very anecdotal example, at my schools ISP (for the residences) the top 8% of power users gobble up something like 96% of available bandwidth with p2p traffic, and leave 4% for everything else. This is WITH (albeit very non aggressive) throttling. So, you know, pretty unfair. Shaping is the just a way to try to ensure good speeds for everybody else.
posted by tracert at 10:03 AM on May 12, 2007

I'm on 256K DSL in the US.

There's really no issue for general browsing, email, etc. Some sites are slower than a faster connection would be, mostly image or flash heavy, but by and large it's perfectly satisfactory for general use.

On multimedia, however, it is an issue. I can't stream a YouTube video so I typically wait until it's about 50% loaded before playing so as not to have interruptions. I don't view video content very often so I only find it mildly inconvenient but I think it would be tedious if you were a frequent user.

Large data downloads are relatively slow as well. When I have large downloads I almost always start them just before I go to bed. As an example the last time I downloaded a Linux ISO file (prob 700 MB) I think it took somewhere between 4-7 hours which would be horrible if I were trying to do other browsing at the same time. So, again, it's mildly inconvenient to have to do things this way but it makes the most sense to do large downloads when I'm not otherwise using the internets. I believe you there are download applications you can use as well to schedule downloads, download files sequentially, stuff like that. I'd look into it if I frequently downloaded a lot of data.

All that said unlimited 256K is vastly preferable to an 800MB cap at any speed.
posted by 6550 at 10:09 AM on May 12, 2007

Halving your upstream will likely have no noticeable effect for you.
posted by rhizome at 10:19 AM on May 12, 2007

your ISP may sell you the service at 512, but physically provision you far higher, and then just zap you for bandwidth.

I don't understand the latter part.

Well, to meet the local laws, they may be afraid of limiting your port speed like would be done with an American cable modem. They also may not be able to do it; it's possible the equipment you're plugging into may not be sophisticated enough.

What they might be doing, instead, is giving the whole complex full access to, say, 1.5megabits or more, and then just counting bytes used per port. In that scenario, it would be very fast, and they'd settle with you at the end of the month. So you'd probably take more of a hit dropping back to 256.
posted by Malor at 3:42 PM on May 12, 2007

Response by poster: Malor: In that scenario, it would be very fast, and they'd settle with you at the end of the month.

Are you talking about metering of data transfer? If so, that's not a point of confusion for me. I was talking about pipe thickness. If I'm provisioned, say, 1.5Mbps, then what does "settle with you at the end of the month" mean?

On another note, is there a way to measure the total size of a webpage? Page Info in Firefox seems to count only the markup proper, and not the various media like images. I ask because I intiated a test session, visiting only 3 sites, www.cnn.com, www.gmail.com (checked mail) and www.nytimes.com, and the session download was recorded as 1.62MB by my ISP. Ethereal showed a total of 3,200 TCP packets, a lot more ARP packets, and a few of others like UDP, BROWSE..etc, during the session. Does the ARP traffic count? Should it?
posted by Gyan at 9:59 PM on May 12, 2007

Sorry for being unclear... I meant that they might let you download all you like, and then count how you much you downloaded and settle your bill at the end of the month. "You downloaded 900mb, so you owe us $10!" I'm assuming they don't cut you off at 800mb, but rather charge you extra.

I don't know whether ARP traffic counts, but it definitely shouldn't; that's just local traffic between you and the cablemodem. ARP is a local Ethernet protocol; it ties specific network cards to IP addresses. (ARP stands for Address Resolution Protocol.) In an Ethernet network, you can have an arbitrary number of machines on the local net. To find the machine with, say, 192.168.0.1, your local machine sends out an ARP request: "who-has 192.168.0.1". That machine answers with its physical MAC address. Your machine should remember this for 10 minutes, and will just send any traffic bound for 192.168.0.1 to that physical card. Meanwhile, other machines on your network might ask who has your IP address... your machine then invisibly replies with a packet saying "that's me!".

It's link-specific traffic, and it doesn't cross routers; it shouldn't get even one hop into your provider's network unless they have an unusual setup. So you shouldn't be charged for ARP under any sane circumstance.
posted by Malor at 2:56 AM on May 13, 2007

Also note that ARP packets are very small... 28 bytes each. Ethernet packets can be up to 1500 bytes.
posted by Malor at 3:00 AM on May 13, 2007

Response by poster: Update:

I switched to the "256kbps"-uncapped package, and as I speak, I just downloaded two large video files at an average speed of ~1 mbps (both). Go figure.

(The source wasn't local)
posted by Gyan at 2:09 AM on May 14, 2007