<https://visawhen.com/consulates/bogota/h1b>
The code is source-available (not open source) at <https://github.com/underyx/visawhen>. It's my first time choosing a source-available license over MIT, mainly out of fear of existing immigration startups just gobbling this data and code up; frankly I didn't think the implications through though, I just threw a safe license on there.
The way the project works is:
- Use requests-html to find publicly available PDFs from government pages
- Use camelot to OCR the PDFs and extract data tables from them
- Since the previous step takes crazy long for my tastes (around 8000 pages at around 5 seconds each) I've used dask to split the work into chunks and parallel-process them across my laptop's CPUs.
- Do data cleanup and processing with pandas, and save all of it to a SQLite file.
- Take data from the SQLite file with next.js and generate a static HTML page for each possible embassy - visa type combination
- The pages use ECharts to visualize data, and Bulma as a CSS framework
- Build and host each commit via Netlify
- But proxy to Netlify from CloudFlare, which I believe has more edge locations in the free plan
- Collect any donations via Ko-Fi
- Use Google Analytics to have a general idea about visitor counts
- Use FullStory session recordings to find out about bugs – I've fixed quite a few and I think I'll probably remove this tracking after a bit of time
…and that's where I'm at now. I'm pretty happy about the results. Most pages load in less than 300ms, which is something I care about all too much. More importantly, I've shared the site with some immigration communities I'm part of, and the response has been very positive! Let me know what y'all think.