URL Find and Replace

November 26, 2007 12:21 PM

I'm working on a quick Automator workflow and need some help replacing the local hostname in a series of URLs.

Here's the deal: the Automator workflow retrieves a list of return delimited URLs with random alphanumeric local hostnames (e.g., wrim881q.example.net, ghu2z.example.net, etc.). I need to pass these into a shell script and change them all to the same local hostname (e.g., foo.example.net) and then download all the URLs with curl.

Help?

posted by nathan_teske to computers & internet (3 comments total)

As long as you're running a shell script, you might as well run them through sed inside the script.

cmiller@zippy:~ $ sh t.sh http://ask.metafilter.com/77126/URL-Find-and-Replace
http://foo.example.net/77126/URL-Find-and-Replace
cmiller@zippy:~ $ cat t.sh
#!/bin/sh
inet_url="$1"
local_hostname="foo.example.net"
local_url=`echo "$inet_url" |sed -e "s=^\\([^:]*\\)://[^/]*\\(/.*\\)$=\1://$local_hostname\2="`

echo $local_url
posted by cmiller at 1:57 PM on November 26, 2007

cmiller I think he just wants to replace the hostname part of the FQDN. And OS X's "sed" is not GNU sed so to get extended regexes it's -E. nathan, just put this in the shell script action:

sed -E 's/^([^:]+:\/\/)[^\.]+/\1foo/'

And is there a reason you're calling curl rather than just using the "Download URLs" action? (It's under "Internet".)
posted by nicwolff at 2:49 PM on November 26, 2007

Thanks nicwolff - that worked perfectly. I need to run through curl as the server I'm hitting horks if you have more than 3-4 concurrent downloads.
posted by nathan_teske at 7:49 AM on November 27, 2007