Comments on: How do you handle authentication via cookie with CURL?

Question: How do you handle authentication via cookie with CURL?

miniape — Thu, 19 May 2005 18:57:12 -0800

how do you handle authentication via cookie with CURL? I’ve been assigned a tedious job at work. I wrote a PHP script to do it for me, but it requires CURL, which I’ve never used before. Basically I have to replace text in a file with text from various sources. I’m uploading a set of files to my webhost (company doesn’t use PHP), running the script and outputting them to a separate file. The problem is one of the sources I’m getting data from is the company website, which is password protected. I have a password and username, but I’m not sure how CURL deals with them. Once they’ve been entered, a cookie is stored to save auth status.

Do I just use http://name:passwd@machine.domain/full/path/to/file and let CURL handle everything? Or do I have to tell it how to handle the cookie? Or am I not getting this at all?

Thanks.

By: nicwolff

nicwolff — Thu, 19 May 2005 19:30:15 -0800

If you mean the username and password are entered in a form on a login page, then cURL can "submit" that form like:

curl -d "username=miniape&password=SeCrEt" http://whatever.com/login

and if you want to store the cookie that comes back you do so by specifying a cookie file:

curl -c cookies.txt -d "username=miniape&password=SeCrEt" http://whatever.com/login

and to use those cookie in later requests you do:

curl -b cookies.txt -d "username=miniape&password=SeCrEt" http://whatever.com/login

or do both if you want to both send and receive cookies:

curl -b cookies.txt -c cookies.txt -d "username=miniape&password=SeCrEt" http://whatever.com/login

By the way, I learned all this in the last couple of minutes by Googling for "curl". Just sayin'.

By: nicwolff

nicwolff — Thu, 19 May 2005 19:31:54 -0800

Sorry, here's the page I found.

By: nicwolff

nicwolff — Thu, 19 May 2005 19:33:59 -0800

Sorry again, I fucked that all up. Well not all, the first request is fine, but you don't need the CGI data once you have the cookie, so it's just:

curl -b cookies.txt http://whatever.com/content

on the later requests.

By: zsazsa

zsazsa — Thu, 19 May 2005 19:35:26 -0800

Does the browser pop up an authenticaion dialog like this one (FireFox above, IE below), or is the login integrated to the webpage with a submittable form? If it's the former, the http://name:passwd@ method will work with no problems. If it's the latter, you'll have to use a cookie, which is more complicated. curl can get cookies from a standard "cookies.txt" type file like those generated by Firefox. So you'd have to visit the site, log in, find your cookies.txt file, isolate the lines having to do with the site, and adjust the expiry time so the cookie doesn't expire. If you're using curl from inside PHP, use the CURLOPT_COOKIEJAR option to point to the cookies.txt file you made.

By: cillit bang

cillit bang — Fri, 20 May 2005 02:01:55 -0800

Login using a normal browser and use the "Show Cookies" (or whatever) option to find out what cookies were set. Then use either:

curl [URL] -H 'Cookie: cookie1=value; cookie2=value, etc'
or:
curl_setopt($curl,array('Cookie: cookie1=value; cookie2=value, etc"));

to include the cookies in the request.

By: eriko

eriko — Fri, 20 May 2005 06:53:29 -0800

My day job involves, amongst many other things, sending out insurance claims to just about every variant of BCBS, Medicare, Medicaid, and a bunch of private insurers. They all do things differently. Many of them (the Medicaids, in particular), use web connections. Hates them, I do....

The best tool for web connections is curl. But you need to read headers, to figure out what to tell curl to send. So, you need something like this -- which will let you watch the HTTP transaction while you do things by hand.

Defaults: Curl command lines can get long. Create a "COMMONSWITCH" variable for the ones you'll use in every connection, since you will almost certainly need to make more than one in a given transaction. My typical setting is
"COMMONSWITCH=-k -b $LTEMP/cookies -c $LTEMP/cookies."

Usually, with secure sites, you need to first hit the site, simply to get session cookies, then hit the logon page, feeding it the correct user/pass, then hit the upload page, feeding it whatever it wants to send a file.

Note -- you have two switches to send parameters, -d and -F, and they can't be used in the same connection. -d is for URL encoded parameters, -F is for mulitpart/form-data submissions. Watching the headers as you submit manually will tell you what you need to send, and how you need to send it.

The command lines can get very long -- make sure you understand how your system handles wrapped lines.

Curl's authors provide The Art Of Scripting HTTP Requests Using Curl, which is a big help.

Finally, realize it may not work. If the site is using things like javascript to generate on-the-fly variables, you'll need to fake that somehow. If you can't, curl wont work -- curl isn't a browser, it just sends HTTP commands and saves the output. This isn't to say curl isn't useful (lordy, my life would suck without it) but it does has limits.

By: ph00dz

ph00dz — Fri, 20 May 2005 07:58:50 -0800

Can you use PHP to control curl?

Here's how I do it, but I'm sure that other, more clever folks can provide alternate implementations...

By: miniape

miniape — Sat, 21 May 2005 04:35:25 -0800

Thanks for all the help. I was able to get half of my job done with the "dump a cookie in your folder and tell curl where it is techinique." I found a work around for the other half. Cheers.