What is Polars?
TLDR: When you have datasets above GB size, there are probably enough performance returns to using Polars that you should try it out. Hey, it's 2023, you can always write Pandas code and ask ChatGPT to translate it for you.
Polars is a Python (and Rust) library that provides functionality for working with tabular data structures in the form of dataframes. You can think of the package as overall very similar to Pandas dataframes.
Polars is implemented in Rust, which makes it designed for high performance in data processing and manipulation. It uses Apache Arrow Columnar Format as its memory model, which is a columnar memory format that also supports efficient data processing.
Powerful API: Polars provides a rich and expressive API for performing data manipulation and analysis tasks. This includes a wide range of data transformation and aggregation operations, as well as support for filtering, sorting, and joining data.
High Performance: Polars is designed for high performance and takes advantage of multi-core processors by allowing data processing operations to be parallelized and executed on multiple threads. It also uses SIMD (Single Instruction, Multiple Data) instructions for the efficient processing of large amounts of data simultaneously.
Optimized Queries: Polars includes query optimization techniques to automatically optimize data processing operations for better performance, including optimizing data access patterns, minimizing data movement, and applying other performance optimizations.
Hybrid Streaming (larger than RAM datasets): Polars supports the processing of datasets that are larger than available RAM using a hybrid streaming approach, efficiently streaming data from disk or other sources.
Lazy and Eager Execution: Polars supports both lazy and eager execution of operations on data. Lazy execution defers operations until results are needed, while eager execution executes operations immediately when called. With lazy execution, Polars' query optimizer can leverage features such as predicate pushdown, column pruning, and other optimizations to minimize data processing and improve performance.
How Does Polars Compare to Pandas? (With Examples)
Performance: Polars is designed for high performance, leveraging Rust's multi-threading, SIMD instructions, and query optimizations. In contrast, Pandas is primarily single-threaded, which may not be as efficient for large datasets or complex operations compared to Polars. Combining the ability to do streaming, as well as lazy execution, and Polars has a strong claim to being the better package for datasets above GB size.
Memory Model: Polars uses Apache Arrow Columnar Format as its memory model, optimized for data processing while Pandas, on the other hand, uses a row-based memory format. This makes Pandas less efficient for certain operations (which are quite common) like column-wise computations and aggregations.
Data Types: Polars provides a more extensive set of data types compared to Pandas, including advanced data types like Date64, DateTime, and Interval, as well as support for custom data types, providing greater flexibility for handling diverse data types.
Syntax and API: Polars has a syntax and API that is similar to Pandas, making it easy for users familiar with Pandas to transition to Polars. However, Polars also provides additional functionalities which we share a few examples below.
| Pandas | Polars | |
|---|---|---|
| Filtering | filtered_df = df[df['column'] > 10] | filtered_df = df.filter(pl.col('column') > 10) |
| Renaming Columns | df = df.rename(columns={'old_name': 'new_name'}) | df = df.alias({'old_name': 'new_name'}) |
| Grouping | grouped_df = df.groupby('column').agg({'column2': 'sum'}) grouped_df = df.groupby('column')['column2'].sum().reset_index() | grouped_df = df.groupby('column').agg({'column2': pl.sum('column2')}) |
| Sorting | sorted_df = df.sort_values(by='column', ascending=False) | sorted_df = df.sort('column', reverse=True) |
| Chaining | df = df[df['age'] > 18].groupby('gender').mean() | df = df.filter(pl.col('age') > 18) & df.groupby('gender').mean() |
| Column Selection | df['age'] | pl.col('age') |
| Casting | df['age'] = df['age'].astype(float) | df = df.with_column(df['age'].cast(pl.Float64)) |
| Rolling Average | df['rolling_average'] = df['value'].rolling(window=3).mean() | df = df.with_column(df['value'].rolling(3).mean().alias('rolling_average')) |
Some claim that the Polars syntax is easier to read and write. In my opinion only, they are about equally as easy to read and write. This means that while there's some fixed overhead in trying out Polars for the first time, it is beneficial for projects where the underlying dataset is heavy and ungainly.
About
Einblick is an agile data science platform that provides data scientists with a collaborative workflow to swiftly explore data, build predictive models, and deploy data apps. Founded in 2020, Einblick was developed based on six years of research at MIT and Brown University. Einblick customers include Cisco, DARPA, Fuji, NetApp and USDA. Einblick is funded by Amplify Partners, Flybridge, Samsung Next, Dell Technologies Capital, and Intel Capital. For more information, please visit www.einblick.ai and follow us on LinkedIn and Twitter.