Ibis: Bringing Optionality to Python Dataframes
Trino Developer Advocate
Starburst Data
Lester Martin is a seasoned developer advocate, trainer, blogger, data engineer, and polyglot programmer focused on data pipelines & data lake analytics using Trino, Iceberg, Hive, Spark, Flink, Kafka, NiFi, NoSQL databases, and, of course, classical RDBMSs. Learn more about Lester at https://linktr.ee/lestermartin.
Love the power of writing lazy executed Dataframe code in Python that runs on your favorite distributed data cluster? Would like some flexibility to swap out your processing engine for another? If so, you need Optionality in your Python Dataframe API.
Ibis, https://ibis-project.org/, offers a Python Dataframe API that lets your code run on nearly 20 backend data processing systems. It is THE portable Dataframe library. Imagine being able to run your Ibis code in Polars on your laptop and then moving it to PySpark in your favorite cloud provider with just changing a property. No need to imagine; you can do it today.
This presentation walks you through the features available in Ibis as well as compares it with other popular Dataframe APIs. You'll see how to mix-and-match SQL and Dataframe API transformations as desired and how to change the backend system were your code is executed.
You will see a demo of a job running in DuckDB for local testing and then with a single line of code being changed run in a Trino cluster. Step-by-step instructions will be provided to follow along on your laptop or to run the exercise yourself later.