Show HN: SQLFrame – I ran PySpark without Spark on a SQL database

11 points

2 years ago

Recently I open-sourced SQLFrame, a DataFrame library that implements the PySpark DataFrame API but removes Spark as a dependency. It does this by generating the corresponding SQL for the DataFrame operations using SQLGlot. Since the output is SQL this also means that the PySpark DataFrame API can now be used directly against other databases without the Spark middleman.

I built this because of two common problems I have faced in my career: 1. I prefer to write complex pipelines in PySpark but they can be hard to read for SQL-proficient co-workers. Therefore I find myself in a tradeoff between maintainability and accessibility. 2. I really enjoy using the PySpark DataFrame API but not every project requires Spark and therefore I'm not able to use the DataFrame library I am most proficient in.

The library currently focuses on transformation pipelines (reading from and writing to tables) and data analysis as key use cases. It does offer some ability to read from files directly but they must be small although this can be improved over time if there is demand for it.

SQLFrame currently supports BigQuery, DuckDB, and Postgres with Clickhouse, Redshift, Snowflake, Spark, and Trino in development or planned. You can use the "Standalone" session to test running against any engine supported by SQLGlot but there could be issues with more advanced functions that will be resolved once officially supported by SQLFrame.

Blog post for more details: https://github.com/eakmanrq/sqlframe/blob/main/blogs/sqlfram...

Would love to answer any questions or hear any feedback you may have!

4 comments