Show HN: Scrape-reduce – A simple way to scrape websites

1 point

3 years ago

This is a tiny little project I made to scrape some websites. Not sure if it's a novel idea, but I've found that splitting scraping into 2 steps is a good way to iterate quickly and safely.

1. "scrape" is when you actually download the HTML file for a page. This should be done very cautiously as you might be blocked by IP or captcha's if you read a webpage too often. This project caches each page you request, so it can be cancelled and resumed later if necessary.

2. "reduce" - in this context - is simply processing the downloaded HTML to get what you want. This project provides a full DOM to query for the text on the page that you're looking for, then save it to another file, or whatever you want to do.

By splitting it up into this "pipeline" we get a very simple "scrape.ts" file for fetching the HTML. And a very simple "reduce.ts" for processing the HTML.

Let me know what you think!