Ask MetaFilter questions tagged with wget

File access to router without wget

alby — Sat, 10 Oct 2009 13:01:21 -0800

How do I copy files to a Netgear DG834G router without using wget? I'm trying to install openwrt on an old Netgear DG834Gv3 router (mainly using these instructions). The instructions recommend to telnet into the router and then copy over the new firmware from the internet using wget.

Unfortunately, wget isn't playing ball. Telnetting into the router works fine and I can execute commands but whether I use http or ftp and whether I use the hostname or IP wget cannot see the internet at all. Ping isn't successful for anything either.

So how do I copy over the firmware without using wget? Or alternatively, how do I get wget to work? The router's bootloader does not allow FTP connections.

ping equivalent for web servers that reports http latency?

zippy — Sun, 03 May 2009 21:57:16 -0800

The equivalent of ping for http connections? I want to the equivalent of ping for webservers. I want to type in a simple command that generates 1 or more 1-line reports on the transaction: no. of bytes, response code, and round-trip latency.

For example, I would like to ask the tool to 'ping' my server once a minute for five minutes:

webping http://www.example.com/foo -i 60 -c 5

and see something like

1200 bytes (200) from www.example.com: seq=1 time = 240ms
1200 bytes (200) from www.example.com: seq=2 time = 240ms
1200 bytes (200) from www.example.com: seq=3 time = 240ms
1200 bytes (200) from www.example.com: seq=4 time = 240ms
300 bytes (404) from www.example.com: seq=5 time = 1500ms

Is there such a thing for linux/os-x? If not, is there a way to get what I want out of wget (shell / python munging of wget's STDOUT perfectly acceptable).

Hitting a URL from the command line

Null Pointer and the Exceptions — Wed, 11 Jul 2007 09:13:07 -0800

Alternatives to lynx, wget and get from the unix command line? I have a scheduled job that needs to hit a localhost URL once a day. Our servers don't have lynx or wget or get installed. Are there any alternatives? I've tried using telnet and Expect, and it works but when run through our scheduling app (Tivoli Workload Scheduler) I get a message "No environment-specified terminal type." even when I specify the terminal type. So I was wondering if there are other techniques I could try.

What's the easiest way to transfer a website (including hidden files) without shell access?

Manjusri — Wed, 18 Apr 2007 16:25:32 -0800

I am trying to transfer a couple of websites from 1and1 beginner accounts to dreamhost. I usually do this by zipping or tarring the source site and then ftp. The rub is that the 1and beginner package does not allow shell access (although the target does). I'm looking at wget but I don't see a way to make it copy hidden files, which could bite me later if there are some config files buried in there somewhere. Probably missing something real obvious here. Any ideas?

I need to download google!

bigmusic — Wed, 14 Jun 2006 20:04:04 -0800

How do I download google's entire cache of a website that has 137,000 hits? Ok, so em411.com got redesigned, and em/admin (site administrator) dumped the database. There were some damn good conversations that happened over the last 6 years on that site and there is no way that I am going to be able to remember what each and every one of them was about, so I need to find a way to get the entire cache of this website. Hope me please.

If I do a site:em411.com I get 137,000 hits, but only 1000 of them are accessible via google.

I'm thinking that I could get this done via some combination of wget following only the cache links (how do I do that?) and varying the keyword in the search (not sure what the best ones to pick are)

Oh, and the internet archive has basically nothing of this site.

How do I mirror a MLB web page?

event — Sat, 27 Aug 2005 15:46:29 -0800

I'd like to use wget to make a local copy of a web page at mlb.com (specifically this one) but all my attempts have resulted in an incomplete page. What arguments should I be giving wget to make it work?

Is there an easy script-way to download 15,000 pages of a website with incremental URLs?

monju_bosatsu — Thu, 10 Jun 2004 12:57:40 -0800

I need to mass download a chunk of a website. It's about 15000 pages, with identical urls except for one portion with a sequential numerical indicator for each page. I don't need to spider any links, I just need it to work through the list of pages. I know there's got to be an easy script-y way of doing this, but you know, I'm a lawyer. Please help!

Website Mirroring Tools

namespan — Sun, 28 Mar 2004 16:27:31 -0800

Website Mirroring Tools: I usually use wget, but find that it seems to have at least one nasty flaw: it doesn't fetch stylesheets, or images named in stylesheets (it may also have other flaws I'm not aware of). Is there something better out there that I can just give a homepage URL and have it suck down an entire site?