- Email, PhoneNumber, Wordpress search
- PGP keys, Certificates and 3 different cryptocurrencies (XMR,BTC,ETH) search
- PostgreSQL database, docker container for Tor and docker-compose for the whole project
- A built in exporter, currently only JSON, but possibly more.
- Exporter has support for: offset, limit, query building via the command line, raw-sql and different modules you can specify, e.g you only want emails, you will only get emails and the html webpage
- Collects title, status_code, body and headers from a webpage.
- Extremely customizable, supports multiple url's at once, based of the goColly framework which gives the crawler many options, e.g Debugger, allowed-domains or url-filters.
- Built install module, so there's no need for manually grabbing our config from github.
- Choice to remember previous parameters with our save-config flag.
- Built in queue and once reached max size, crawler will automatically exit.
That's a short overview, but there will be many more features to come as I enjoy working on this project!