How do you go about automating/simulating (I think) an HTTP request?
March 3, 2013 7:39 PM Subscribe
I want to write a script to automate doing a search, retrieving, and parsing the search results from a website (a booking site similar to the search on www.hilton.com ).
posted by hot soup to computers & internet (5 answers total) 5 users marked this as a favorite
My (extremely) rough understanding is that I should write a script to mimic the request the form is sending, and that I can use something like Firebug or Fiddler to capture what my browser is sending.
I am way out of my depths here but am pretty committed to doing this as an educational project, so I'd appreciate any direction getting started figuring out how all of this is done - I'm familiar with basic python scripting and have used urllib and BeautifulSoup to do basic web scraping, but I don't really understand how all these pieces fit together or how to get started - pointers to good resources would help tremendously, as I've found information on StackOverflow but am having trouble deciphering it. I'm a little more comfortable with what I need to do to parse the html once I get it back.