I need to run a web crawler to 'screen scrape' a few websites. Ideally this crawler can:
1. Deal with forms
2. Deal with javascript
3. Deal with non-pure HTML to extract bits of data
A few years ago I had a bit of success with Mozilla Parser (http://mozillaparser.sourceforge.net/) but the project seems to have gone cold.
Can anyone recommend a web crawler they have used?