Daft 0.1.0 Release Notes
Daft 0.1.0 Release Notes#
Welcome to the first “minor” version release of Daft!
We hope everyone has had a great time using our
0.0.* releases, but buckle up and grab a drink while you read these release notes, because we built so much over the past month and this new release is BIG!
A big shoutout to the contributors who made this all possible - with 21,716 added and 16,090 deleted lines of code!
@xcharleslin @clarkzinzow @jeevb @samster25 @jaychia @FelixKleineBoesing
We rebuilt all of our core execution code in Rust - giving us a 2x speedup across the board for many of our benchmarks!
Our type system just levelled up! We have a much more sophisticated type system written in Rust that can handle nested types, parametrized types and type promotion semantics.
The UDF API is much cleaner now with the introduction of the Daft
Python object columns are now much more featureful with support for casting and magic methods
The full list of changes is much too long to present in this release notes, but here it is anyways.
Rust Execution Backend#
Our execution code was refactored into Rust!
Previously Daft relied on a mix of NumPy, Polars, Pandas and PyArrow for executing logic. This was problematic for a few reasons:
Difficult to perform performance optimizations
Difficult to understand and manage memory allocation
Flaky null-handling and broadcasting semantics depending on the library we used
0.1.0, Daft is now statically linked to the wonderful Arrow2 Rust library which we use for executing all our kernels.
This has several implications:
Daft now has a Python-binded Rust execution layer, mainly comprising of the
Daft is much faster (up to 2x in many cases, especially for our default multithreaded Python runner!) as GIL contention is no longer a bottleneck
Daft’s Python dependencies have been greatly reduced, and is much more lightweight!
On the user-facing API, most of these changes are completely transparent - i.e. you just got a massive speedup for free!
Enhanced Type System#
Our type system just levelled up!
Nested types were added #802
Types are now much more granular (e.g.
Types are automatically promoted when necessary during certain operations (e.g. adding a
Nullarray and a
Int64array results in
As a result, we have much cleaner support for Null array handling since the Null type can be correctly type-promoted with our new supertype semantics.
As our first minor release, several APIs have changed substantially that you should be aware of. Moving forward, Daft APIs will maintain much stricter backward compatibility semantics.
UDFs are much cleaner in
UDFs now no longer require up-front declaration of which arguments have to be
Expressions, and what input types they are passed in as (list, numpy, arrow etc). Instead:
Inputs are always passed in as
daft.series.Seriesobjects and users can now easily convert this to the format they care about using
Which inputs are going to be
daft.series.Seriesvs Python objects is inferred at runtime by checking which arguments a user passes in are
For more information, consult: UDF User Guide
Our old typing APIs have changed - the definitive typing API is now found at
If you are declaring types (for instance as return types for UDFs), you should now use the
DataType.* constructor methods!
Creation of DataFrames has been promoted to module-level functions!
from daft import DataFrame df = DataFrame.read_csv(...)
import daft df = daft.read_csv(...)
This is a big improvement in useability (moving forward, Daft will try to make it as easy as possible to use us by just importing the top-level
For more information, please see: API Documentation for Input/Output.