Integrations

Expand the capabilities of your Flyte workflows with a range of powerful integrations.

DataFrames

Crunch data however you want. Automatically transition data from one dataframe type to the other using Structured Dataset.

HuggingFace

Use HuggingFace dataset as a native Flyte type.

pip install flytekitplugins-huggingface

Vaex

Visualize and explore big tabular datasets.

pip install flytekitplugins-vaex

Polars

Use Polars dataframe as a native Flyte type.

pip install flytekitplugins-polars
pip install flytekitplugins-modin

Data Validation

Validate data at every step of your Flyte workflow.

Great Expectations

Validate your data with expectations.

pip install flytekitplugins-great-expectations

Pandera

Validate dataframe-like objects.

pip install flytekitplugins-pandera

Databases & Data Warehouses

Manage and connect to databases and warehouses seamlessly.

DuckDB

Run intricate analytical queries with DuckDB

pip install flytekitplugins-duckdb
pip install flytekitplugins-bigquery
pip install flytekitplugins-snowflake

DoltHub

Apply git-like versioning to your SQL databases.

pip install flytekitplugins-dolt

SQLAlchemy

Execute SQL queries as Flyte tasks.

pip install flytekitplugins-sqlalchemy
pip install flytekitplugins-hive

Data Processing

Process and analyze your data with data-crunchers.

Databricks

Schedule, monitor and orchestrate Databricks jobs.

pip install flytekitplugins-spark

DBT

Transform data in your warehouses with DBT.

pip install flytekitplugins-dbt

Apache Spark

Run Spark jobs on ephemeral clusters.

pip install flytekitplugins-spark

AWS Athena

Query an AWS Athena service.

pip install flytekitplugins-athena

Feature Stores

Store, share and manage features for ML models.

Feast

Manage and serve ML features with Feast.

Model Training

Simplify the model training process.

AWS Sagemaker

Train ML models on Sagemaker from within Flyte.

pip install flytekitplugins-awssagemaker

Distributed Model Training

Perform distributed model training to speed up the model development process.

Dask

Run Dask jobs natively on a Kubernetes cluster.

pip install flytekitplugins-dask

Kubeflow MPI

Run distributed training with an MPI operator.

pip install flytekitplugins-kfmpi

Ray

Connect to Ray cluster to perform distributed model training and hyperparameter tuning.

pip install flytekitplugins-ray

Horovod

Run distributed deep learning workflows.

Kubeflow TensorFlow

Run distributed TensorFlow training jobs.

pip install flytekitplugins-kftenorflow

Kubeflow PyTorch

Run distributed PyTorch training jobs.

pip install flytekitplugins-kfpytorch

Model Deployment

Streamline the model deployment process.

ONNX TensorFlow

Generate ONNX models from TensorFlow models.

pip install flytekitplugins-onnxtensorflow

ONNX PyTorch

Generate ONNX models from PyTorch models.

pip install flytekitplugins-onnxpytorch

ONNX Scikit Learn

Generate ONNX models from Scikit Learn models.

pip install flytekitplugins-onnxscikitlearn

Monitoring

Monitor data and models from within Flyte.

MLFlow

Track your machine learning metrics with MLFlow

pip install flytekitplugins-mlflow

Whylogs

Log any kind of data and generate summaries of datasets.

pip install flytekitplugins-whylogs

Kubernetes-Native

Exercise greater control over Kubernetes resources.

Kubernetes Pods

Configure pods for arbitrary workloads.

pip install flytekitplugins-pod

Miscellaneous

pip install flytekitplugins-awsbatch
pip install flytekitplugins-papermill

Build your own integration

Create your own integration and submit it to the Flyte repository.

Join our community to discuss and share ideas for integrations