Integrations

Extend Flyte's functionality with third-party integrations

Airflow

Airflow Provider

Trigger Flyte executions from within Airflow.

Documentation View on GitHub

Pandera

Flytekit Plugin

Validate pandas dataframes.

Documentation View on GitHub

Spark

Native Backend Plugin

Run Spark jobs on a Kubernetes cluster.

Documentation Backend Setup View on GitHub

Sagemaker

Native Backend Plugin

Train models using SageMaker.

Documentation Backend Setup View on GitHub

Great Expectations

Flytekit Plugin

Validate data with Great Expectations.

Documentation View on GitHub

Kubeflow PyTorch

Native Backend Plugin

Run distributed PyTorch training jobs w/ Kubeflow.

Documentation Backend Setup View on GitHub

Kubeflow TensorFlow

Native Backend Plugin

Run distributed TF training jobs w/ Kubeflow.

Documentation Backend Setup View on GitHub

Hive

External Service Plugin

Run Hive jobs in Flyte workflows.

Documentation View on GitHub

Kubernetes Pods

Native Backend Plugin

Run K8s pods for arbitrary workloads.

Documentation View on GitHub

SQLAlchemy

Flytekit Plugin

Execute SQL queries as Flyte tasks.

Documentation View on GitHub

Dolt

Flytekit Plugin

Version your SQL databases w/ Dolt.

Documentation View on GitHub

Papermill

Flytekit Plugin

Execute Jupyter notebooks w/ Papermill.

Documentation View on GitHub

MPI

Native Backend Plugin

Run distributed training jobs w/ MPI operator.

Documentation Backend Setup View on GitHub

Modin

Flytekit Plugin

Scale pandas workflows w/ Modin.

Documentation View on GitHub

AWS Athena

External Service Plugin

Execute queries using AWS Athena.

Documentation Backend Setup View on GitHub

Snowflake

External Service Plugin

Run Snowflake jobs in Flyte workflows.

Documentation Backend Setup View on GitHub

AWS Batch

External Service Plugin

Run Flyte tasks on AWS batch service.

Documentation Backend Setup View on GitHub

BigQuery

External Service Plugin

Run BigQuery jobs in Flyte workflows.

Documentation Backend Setup View on GitHub

Whylogs

Flytekit Plugin

Generate summaries of datasets w/ Whylogs.

Documentation View on GitHub

ONNX Scikit Learn

Flytekit Plugin

Generate ONNX models from Scikit Learn models.

Documentation View on GitHub

ONNX PyTorch

Flytekit Plugin

Generate ONNX models from PyTorch models.

Documentation View on GitHub

ONNX TensorFlow

Flytekit Plugin

Generate ONNX models from TensorFlow models.

Documentation View on GitHub

Polars

Flytekit Plugin

Support polars.DataFrame as a data type.

View on GitHub

Horovod

Tutorial

Run distributed training w/ Horovod.

Documentation

Feast

Tutorial

Manage and serve ML features w/ Feast.

Documentation