Integrations

Expand the capabilities of your Flyte workflows with a range of powerful integrations.

DataFrames

Crunch data however you want. Automatically transition data from one dataframe type to the other using Structured Dataset.

Vaex

Visualize and explore big tabular datasets.

pip install flytekitplugins-vaex

Polars

Use Polars dataframe as a native Flyte type.

pip install flytekitplugins-polars
pip install flytekitplugins-modin

HuggingFace

Use HuggingFace dataset as a native Flyte type.

pip install flytekitplugins-huggingface

Data Validation

Validate data at every step of your Flyte workflow.

Pandera

Validate dataframe-like objects.

pip install flytekitplugins-pandera

Great Expectations

Validate your data with expectations.

pip install flytekitplugins-great-expectations

Databases & Data Warehouses

Manage and connect to databases and warehouses seamlessly.

SQLAlchemy

Execute SQL queries as Flyte tasks.

pip install flytekitplugins-sqlalchemy
pip install flytekitplugins-snowflake

DuckDB

Run intricate analytical queries with DuckDB

pip install flytekitplugins-duckdb
pip install flytekitplugins-hive

DoltHub

Apply git-like versioning to your SQL databases.

pip install flytekitplugins-dolt
pip install flytekitplugins-bigquery

Data Processing

Process and analyze your data with data-crunchers.

Apache Spark

Run Spark jobs on ephemeral clusters.

pip install flytekitplugins-spark

DBT

Transform data in your warehouses with DBT.

pip install flytekitplugins-dbt

Databricks

Schedule, monitor and orchestrate Databricks jobs.

pip install flytekitplugins-spark

AWS Athena

Query an AWS Athena service.

pip install flytekitplugins-athena

Feature Stores

Store, share and manage features for ML models.

Feast

Manage and serve ML features with Feast.

Model Training

Simplify the model training process.

PyTorch Elastic v1

Pytorch-native multi-node distributed training.

pip install flytekitplugins-kfpytorch

AWS Sagemaker

Train ML models on Sagemaker from within Flyte.

pip install flytekitplugins-awssagemaker

Distributed Model Training

Perform distributed model training to speed up the model development process.

Ray

Connect to Ray cluster to perform distributed model training and hyperparameter tuning.

pip install flytekitplugins-ray

Kubeflow TensorFlow

Run distributed TensorFlow training jobs.

pip install flytekitplugins-kftenorflow

Kubeflow MPI

Run distributed training with an MPI operator.

pip install flytekitplugins-kfmpi

Kubeflow PyTorch

Run distributed PyTorch training jobs.

pip install flytekitplugins-kfpytorch

Horovod

Run distributed deep learning workflows.

Dask

Run Dask jobs natively on a Kubernetes cluster.

pip install flytekitplugins-dask

Model Deployment

Streamline the model deployment process.

ONNX TensorFlow

Generate ONNX models from TensorFlow models.

pip install flytekitplugins-onnxtensorflow

ONNX PyTorch

Generate ONNX models from PyTorch models.

pip install flytekitplugins-onnxpytorch

ONNX Scikit Learn

Generate ONNX models from Scikit Learn models.

pip install flytekitplugins-onnxscikitlearn

Monitoring

Monitor data and models from within Flyte.

Weights & Biases v1

Best in class ML/AI experiment- and inference-time tracking.

pip install flytekitplugins-wandb

Whylogs

Log any kind of data and generate summaries of datasets.

pip install flytekitplugins-whylogs

MLFlow

Track your machine learning metrics with MLFlow

pip install flytekitplugins-mlflow

Kubernetes-Native

Exercise greater control over Kubernetes resources.

Kubernetes Pods

Configure pods for arbitrary workloads.

pip install flytekitplugins-pod

Miscellaneous

pip install flytekitplugins-papermill
pip install flytekitplugins-awsbatch

Build your own integration

Create your own integration and submit it to the Flyte repository.

Join our community to discuss and share ideas for integrations