Finally, a platform that scales as your needs grow

Flyte simplifies building data and ML workflows with its user-friendly SDK. It also supports flexible scaling with minimal infrastructure costs and effort. In contrast, Airflow does not offer an infrastructure-oriented setup, which means more effort to manage the platform. Designed for teams who want more productivity, Flyte helps you easily organize and manage your workflows from the start.

Install locally ↗

Try hosted ↗

Eliminate barriers to team collaboration

Collaborating as a team should be easy, but Airflow’s lack of multi-tenancy can make it difficult to scale horizontally and share data and ML workflows. Flyte, on the other hand, is built with multi-tenancy in mind, enabling centralized management of workflows and breaking down silos within the organization.

Gain visibility into your data

Data is the driving force behind workflows, and you need visibility into its origins, transformation and movement. Airflow isn’t data-driven and doesn’t understand data flows. In contrast, Flyte’s powerful lineage capabilities provide a view of every step in the workflow, making it easier to debug and providing greater clarity on data within a workflow process.

Wave Airflow versioning hacks goodbye

Versioning should be an integral part of the orchestration process, not an added layer. Flyte incorporates versioning as a core feature, allowing teams to experiment on a centralized infrastructure without the need for workarounds.

Modern AI orchestration

Many organizations have found that the requirements for AI pipelines are significantly more complex than traditional ETL pipelines. Modern AI orchestration meets the demands of today’s workloads, which make dynamic use of heterogenous and resource-intensive infrastructures. With efficiency, scalability, ease of use and agility, modern AI orchestration empowers organizations to optimize and automate pipeline management.

“Versioning, caching and the different domains we can have in Flyte prompted us to move from Airflow to Flyte because you don’t really need to think about them and they are … available out of the box in Flyte.”

— Stephen Batifol, Machine Learning Engineer at Wolt

Contrasting Airflow and Flyte for modern orchestration

Airflow

Flyte

Versioning

Easily experiment and iterate in isolation with versioned tasks and workflows.

Multi-tenancy

A centralized infrastructure for your team and organization, enables multiple users to share the same platform while maintaining their own distinct data and configurations.

Type checking

Strongly typed inputs and outputs can simplify data validation and highlight incompatibilities between tasks making it easier to identify and troubleshoot errors before launching the workflow.

Caching

Caching the output of task executions can accelerate subsequent executions and prevent wasted resources.

Data lineage

As a data-aware platform, it can simplify rollbacks and error tracking.

Immutability

Immutable executions help ensure reproducibility by preventing any changes to the state of an execution.

Recovery

Rerun only failed tasks in a workflow to save time, resources, and more easily debug.

Human-in-the-loop

Enable human intervention to supervise, tune and test workflows - resulting in improved accuracy and safety.

Intra-task checkpointing

Checkpoint progress within a task execution in order to save time and resources in the event of task failure.

Reproducibility

With every task versioned and every dependency set is captured, making it easy to share workflows across teams and reproduce results.

Not fully supported

Flyte vs. Airflow:
What’s right for me?

Both Flyte and Airflow serve as options for orchestrating data pipelines in Python. They provide the functionality to execute pipelines on a scheduled basis or ad-hoc. However, there are notable distinctions between the two.

Airflow is commonly used for ETL/ELT tasks, whereas Flyte is particularly well-suited for running data and ML pipelines that can be easily scaled. Flyte also excels in providing support for custom environments and ensuring compute isolation.

To help you decide between Flyte vs Airlfow, let’s dive into the details.

Please note that the mentioned features are accurate as of the time of writing, but they may be subject to change in the future.

Built-in multi-tenancy for scalability and collaboration

As organizations expand, involving multiple teams in pipeline creation and managing a growing number of pipelines, the need for an orchestration platform that effectively separates team concerns becomes crucial. Simultaneously, there is a requirement for seamless collaboration among teams to share pipelines. While Airflow is currently adopting a multi-tenant architecture, Flyte was designed with built-in multi-tenancy right from the start. Flyte empowers decentralized pipeline development on a centralized infrastructure platform, facilitating scalability, collaboration, and efficient pipeline management.

Flexible scheduler scalability

Airflow’s scheduler may experience overloading when multiple DAGRuns are executed simultaneously. On the other hand, Flyte’s native scheduler excels at handling a vast number of workloads, significantly reducing the likelihood of missed schedules or suboptimal resource utilization.

Fine-grained resource management per task

Dag where each task is highlighted with their corresponding allocated resources (can be linear and only 3-4 tasks - data ingestion, model training (this one has GPU), optimization )

In Airflow, there is no built-in capability to limit resource usage on a per-task basis, and workers are not isolated from user code. As a result, resource-intensive tasks can overpower workers and adversely affect the execution of other tasks. Additionally, when a fixed pool of workers is fully utilized, heavy tasks can halt the progress of other workflows.

In contrast, Flyte enables you to precisely specify the resources allocated to each task. You can define resource requests and limits, allowing Flyte to ensure automatic load balancing. This fine-grained resource management capability enhances performance and prevents tasks from monopolizing resources, ensuring efficient workflow execution.

Environment isolation

Airflow lacks proper library isolation, making it challenging or even impossible to accommodate specific library versions for different workflows within a team. This limitation becomes particularly significant in the context of ML workflows, where teams often develop and reuse their libraries across multiple projects, such as model training and serving. Consequently, all workflows are bound to run on the same version of those libraries. While one workaround is to employ a KubernetesPodOperator to containerize the code and overcome resource limitations, this solution adds an extra layer of complexity.

Flyte offers a significant advantage in terms of environment and dependency isolation. Code and libraries are packaged within Docker images, enabling the use of different libraries and versions per team or even for specific tasks. Additionally, Flyte organizes projects into logical domains, such as development, staging, and production. These domains facilitate a step-by-step promotion of code to production ensuring adherence to best development practices like CI/CD, unit/integration testing, and code review. With Flyte’s environment isolation, teams can maintain control over their dependencies and ensure reproducibility while adhering to robust software engineering practices throughout the project lifecycle.

Simplifying local-to-cloud interactions

When it comes to transferring or retrieving data from cloud storage in Airflow, the process typically involves using an operator. However, Flyte offers a more streamlined approach. With Flyte, you can effortlessly pass Pandas DataFrames between tasks, load DataFrames into BigQuery tables using structured datasets, seamlessly offload and download data from cloud URIs using FlyteFiles, and much more. In addition, Flyte automates interactions with S3 and GCS, supports communication between different cloud services and the local file system, and minimizes the need for writing repetitive code.

Bottom line: If your goal is to construct robust data and ML pipelines that meet production standards, while also enjoying the advantages of modern orchestration and simplified infrastructure management, we built Flyte just for you.

Why engineers choose Flyte over Airflow

Avoid extensive resource consumption

Efficient resource consumption is important to reduce costs and speed up workflow executions. Flyte lets you cache your workflow outputs to avoid re-running an execution, given there has not been any change in the inputs or assumptions. That enables you to retrieve results faster.

Make developers happy

Data engineers, scientists and ML engineers want to build a variety of workflows. While Airflow is useful for static data workflows, it requires more effort to accommodate advanced data and ML workflows. A flexible orchestration platform like Flyte can handle different workflow types and use cases.

Perform iterative, remote development

Flyte makes it just as easy to develop remotely as it is to develop locally, preventing errors and allowing iterations on a remote environment with a production-grade stack.

Join an extremely helpful community

The No. 1 reason people cite for loving Flyte? Our community. We ensure that no question goes unanswered, and we tackle a diverse set of problems with the help of power users spanning several industries.

Be a part of our community

From Airflow to Flyte

If you’re an Airflow user, migrating from Airflow to Flyte isn’t hard. In fact, you may find migration makes it easier to author and maintain data and ML workflows. Switching to Flyte can help you scale and stay agile as your workflows, team and needs grow. If you’re looking for a scalable and customizable workflow orchestration platform that’s easy to use, we built Flyte just for you.

To integrate Flyte into your current stack, use the Flyte Airflow Provider to run Flyte tasks within Airflow. It lets you take advantage of Flyte while remaining within Airflow — and it can also serve as a foundation for migrating to Flyte down the road.

See for yourself how easy it is to migrate code from Airflow to Flyte

Copied to clipboard!

from datetime import datetime
from airflow import DAG
from airflow.decorators import task

with DAG(dag_id="simple_mapping", start_date=datetime(2022, 1, 1)) as dag:

    @task
    def make_list():
        return [1, 2, 3]

    @task
    def add_one(x: int):
        return x + 1

    @task
    def sum_it(values):
        print(f"Total was {sum(values)}")

    added_values = add_one.expand(x=make_list())
    sum_it(added_values)

Copied to clipboard!

from typing import List
from flytekit import map_task, task, workflow


@task
def add_one(x: int) -> int:
    return x + 1


@task
def sum_it(values: List[int]):
    print(f"Total was {sum(values)}")


@workflow
def wf(x: List[int]):
    mapped_out = map_task(add_one)(x=x)
    sum_it(values=mapped_out)

Airflow and Flyte code samples (we intentionally picked a simple example to showcase the contrast).

“To my great surprise, the migration to Flyte was as smooth and easy as the development of our initial active learning pipeline in Airflow had been painful: It literally took just a few weeks to revamp our platform’s main pipeline entirely, to the delight of users and developers alike.”

— Jennifer Prendki, Founder and CEO of Alectio