Show HN: Typol – Static typing layer for Polars

1 point

5 hours ago

Hello! Wanted to share Typol, a thin static typing layer around Polars that lets you enforce columnar schemas. We've been hesitant in the past to go with dataframes for processing reporting data, especially with Pandas, due to the long-term maintainability burden of tooling not understanding the data we're processing, or the library itself. Polars is well typed and encourages constructing shapes up rather than modifying in-place, so adding schema typing to it seemed like a natural extension. If Polars DataFrames are dicts, then Typol's are TypedDicts.

With Typol, it's easy to define your schemas, which should feel familiar if you're moving from dataclass-style code or from Polars' own schemas, and then build well-typed Polars expressions on these that enforce: (1) valid columns are referenced, (2) column values are used in a valid way for their type, and (3) expressions generate target valid columns in resulting schemas with the correct type.

  class Account(tp.Shape):
      name = tp.dimension(str)
      website = tp.dimension(str)
      uid = tp.dimension(int)

  # Works, with the type: Expr[Account, Account, str]
  email_address = accounts.s.name.str.to_lowercase() + "@" + accounts.s.website

  # Caught statically:
  # Unsupported `+` operation: `BoundDimension[Account, int]` + `Literal["@"]`
  email_address = accounts.s.uid + "@" + accounts.s.website

These types are checked statically using ty, which supports spelling the intersection types needed to infer join results, with a little dynamic enforcement filling in where static analysis can't reach. This allows you to make use of tooling both to check and guide your code (dot completion coming in handy). Existing tools, like Pandera, do provide dynamic verification of dataframe shapes. Whilst this can be good, it bites you at runtime which is well after a problem should be caught, and doesn't provide any tooling benefit.

Typol is great for production data processing pipelines, where narrowing your data to well-defined schemas at each processing stage can be appropriate and powerful. It's not well suited to a lot of data science, where columns generally get added and dropped quite freely. It covers most core Polars expression operations (laziness, arithmetic, strings, datetimes, lists, filtering, joins, aggregations), but we'd love to extend it further, and we'd love for you to try it out!

class Account(tp.Shape): name = tp.dimension(str) website = tp.dimension(str) uid = tp.dimension(int) # Works, with the type: Expr[Account, Account, str] email_address = accounts.s.name.str.to_lowercase() + "@" + accounts.s.website # Caught statically: # Unsupported `+` operation: `BoundDimension[Account, int]` + `Literal["@"]` email_address = accounts.s.uid + "@" + accounts.s.website