daft.context.set_runner_ray

Contents

daft.context.set_runner_ray#

daft.context.set_runner_ray(address: Optional[str] = None, noop_if_initialized: bool = False, max_task_backlog: Optional[int] = None) DaftContext[source]#

Set the runner for executing Daft dataframes to a Ray cluster

Alternatively, users can set this behavior via environment variables:

  1. DAFT_RUNNER=ray

  2. Optionally, DAFT_RAY_ADDRESS=ray://…

This function will throw an error if called multiple times in the same process.

Parameters:
  • address – Address to head node of the Ray cluster. Defaults to None.

  • noop_if_initialized – If set to True, only the first call to this function will have any effect in setting the Runner. Subsequent calls will have no effect at all. Defaults to False, which throws an error if this function is called more than once per process.

Returns:

Daft context after setting the Ray runner

Return type:

DaftContext