In this podcast episode with Jon Krohn, I discuss my journey in data engineering and the evolution of Python data processing libraries. We talk about:
- Introduction to Polars: High-performance, declarative data frame library for Python.
- Motivation for the book: Growing popularity of Polars and need for comprehensive documentation.
- Performance comparison: Polars can outperform Pandas, reducing memory usage and computation time.
- Real-world use at Alliander: Reduced memory usage from 500 GB to 40 GB, improved processing.
- Collaboration with Nvidia and Dell: Benchmarked Polars on GPU, achieving significant performance gains.
- Data visualization: Tools like HBplot and Great Tables help create appealing outputs from Polars data frames.
- Writing process: Authors share challenges, deadlines, and switching visualization libraries.
- Book giveaway: Announcement of a giveaway for signed copies of the book.