Datamining the public web
July 30, 2007 1:52 AM Subscribe
How do i build a data warehouse that scrapes data from public websites for my own use? Tools? Tips?
posted by ilike to computers & internet (15 answers total) 19 users marked this as a favorite
Hi. I would like to track apartments on a classifieds site and use the data for analyzing the inpact of diffrent things on price. What i need is a tool or scripting language that would make it easy for me to spider the website and put the data in a database. Preferable this would be an open source solution.
I am also looking for good tools for extracting information out of longer pieces of text. For example on the site i want to mine users can put in comments on every object. I would like to be able to decide if a comment is positive, negative och neither. I have seen this be done on one online art site that i cant remember the name of right now. The artist used blog post and decided the mood of the writer by what words were used.