HTTP transfer working poorly

October 26, 2005 6:26 PM

I'm having hit-or-miss connection trouble with a Russian topographic map website, which has ~ 4 MB JPEG maps online. It's been driving me nuts all summer. Here's the scoop...

(1) Using a direct Internet connection, my downloads will usually "reset" after 30 seconds to 3 minutes. To be exact, the map transfer just quits and goes to "Done", leaving an partial map on the screen.

(2) I have to goad it along, hitting "Reload" to get more and more of the map to load.

(3) Increasing the Firefox network.http.*.timeout values doesn't have any effect.

(4) Here's the kicker -- if I'm connected through any Russian proxy server, the downloads are much faster and don't reset. It doesn't seem to help to use European proxies; they have to be Russian.

(5) The solution seems to continue to use Russian proxy servers, however they're not easy to find. The proxies listed on Atomsoft and Samair is almost entirely garbage that doesn't work, though on occasion the latter will give me a good proxy.

(6) Sample map (large 3500 x 5000 JPEG, beware)

Any network gurus here? Please help me understand what is going on here! Is there some sort of http transfer manager that might work? Or is there a better source of proxy servers than what I'm using?

posted by zek to computers & internet (5 comments total)

I've just run a few quick tests using curl. Their server seems to be set to cut you off after exactly 2 minutes, so that's why changing the values in Firefox doesn't help. The server appears to properly support resuming downloads, so if you use the Downloads window, you can automatically resume transfer.

I can't say why it's so slow, but it's probably not something software on your computer can help you with. Looking for a reliable proxy seems the best option.
posted by cillit bang at 6:56 PM on October 26, 2005

It looks like something is throttling the bandwidth and limiting the connection lifetime. I'm consistently getting 481.8 bytes per second (yes, those are all significant digits) and connections lasting about two minutes and ten seconds. If Russian proxies give you better speeds and no resets, it is very possible that the website or network it's on is purposefully degrading transfers to non-Russian IPs. My connection to the server is good, with no packet loss (as shown by a traceroute).

So, if you're not worried about how long it takes, just use any resuming HTTP transfer agent. I'm using wget to check it out (available for windows, too), but there are any number of them out there. And on preview, I guess Firefox can handle it for you, too.
posted by whatnotever at 7:22 PM on October 26, 2005

Yes, use wget. I had this same problem, with this very same web site, trying to get certain maps, and running wget (setting it for infinite replies) solves the solution beautifully. It won't be fast, but just feed in the list of maps you want and come back the next day - presto!
posted by blindcarboncopy at 7:25 PM on October 26, 2005

Awesome -- thanks! I'll head right over and get wget.
posted by zek at 8:18 PM on October 26, 2005

It could very well be that this is site policy. For example they may have cheap or free bandwidth to locations in the same country (due to peering agreements) but international IP traffic might cost significantly more. So they might cap the bandwidth of connections to addresses outside the country, for which their transit costs are higher, in order to reduce the bandwidth bill. By using a local proxy you get around that cap.
posted by Rhomboid at 8:24 AM on October 28, 2005