Finally, a platform that scales as your needs grow

Flyte simplifies building data and ML workflows with its user-friendly SDK. It also supports flexible scaling with minimal infrastructure costs and effort. In contrast, Airflow does not offer an infrastructure-oriented setup, which means more effort to manage the platform. Designed for teams who want more productivity, Flyte helps you easily organize and manage your workflows from the start.

Install locally ↗

Try hosted ↗

Eliminate barriers to team collaboration

Collaborating as a team should be easy, but Airflow's lack of multi-tenancy can make it difficult to scale horizontally and share data and ML workflows. Flyte, on the other hand, is built with multi-tenancy in mind, enabling centralized management of workflows and breaking down silos within the organization.

Gain visibility into your data

Data is the driving force behind workflows, and you need visibility into its origins, transformation and movement. Airflow isn’t data-driven and doesn’t understand data flows. In contrast, Flyte’s powerful lineage capabilities provide a view of every step in the workflow, making it easier to debug and providing greater clarity on data within a workflow process.

Wave Airflow versioning hacks goodbye

Versioning should be an integral part of the orchestration process, not an added layer. Flyte incorporates versioning as a core feature, allowing teams to experiment on a centralized infrastructure without the need for workarounds.

“Versioning, caching and the different domains we can have in Flyte prompted us to move from Airflow to Flyte because you don’t really need to think about them and they are … available out of the box in Flyte.”

— Stephen Batifol, Machine Learning Engineer at Wolt

Comparing Airflow and Flyte

Airflow

Flyte

Not fully supported

Intra-task checkpointing

Why engineers choose Flyte over Airflow

Avoid extensive resource consumption

Efficient resource consumption is important to reduce costs and speed up workflow executions. Flyte lets you cache your workflow outputs to avoid re-running an execution, given there has not been any change in the inputs or assumptions. That enables you to retrieve results faster.

Make developers happy

Data engineers, scientists and ML engineers want to build a variety of workflows. While Airflow is useful for static data workflows, it requires more effort to accommodate advanced data and ML workflows. A flexible orchestration platform like Flyte can handle different workflow types and use cases.

Perform iterative, remote development

Flyte makes it just as easy to develop remotely as it is to develop locally, preventing errors and allowing iterations on a remote environment with a production-grade stack.

Join an extremely helpful community

The No. 1 reason people cite for loving Flyte? Our community. We ensure that no question goes unanswered, and we tackle a diverse set of problems with the help of power users spanning several industries.

Be a part of our community

From Airflow to Flyte

If you’re an Airflow user, migrating from Airflow to Flyte isn’t hard. In fact, you may find migration makes it easier to author and maintain data and ML workflows. Switching to Flyte can help you scale and stay agile as your workflows, team and needs grow. If you're looking for a scalable and customizable workflow orchestration platform that’s easy to use, we built Flyte just for you.

To integrate Flyte into your current stack, use the Flyte Airflow Provider to run Flyte tasks within Airflow. It lets you take advantage of Flyte while remaining within Airflow — and it can also serve as a foundation for migrating to Flyte down the road.

See for yourself how easy it is to migrate code from Airflow to Flyte

Copied to clipboard!

from datetime import datetime
from airflow import DAG
from airflow.decorators import task

with DAG(dag_id="simple_mapping", start_date=datetime(2022, 1, 1)) as dag:

    @task
    def make_list():
        return [1, 2, 3]

    @task
    def add_one(x: int):
        return x + 1

    @task
    def sum_it(values):
        print(f"Total was {sum(values)}")

    added_values = add_one.expand(x=make_list())
    sum_it(added_values)

Copied to clipboard!

from typing import List
from flytekit import map_task, task, workflow


@task
def add_one(x: int) -> int:
    return x + 1


@task
def sum_it(values: List[int]):
    print(f"Total was {sum(values)}")


@workflow
def wf(x: List[int]):
    mapped_out = map_task(add_one)(x=x)
    sum_it(values=mapped_out)

Airflow and Flyte code samples (we intentionally picked a simple example to showcase the contrast).

“To my great surprise, the migration to Flyte was as smooth and easy as the development of our initial active learning pipeline in Airflow had been painful: It literally took just a few weeks to revamp our platform’s main pipeline entirely, to the delight of users and developers alike.”

— Jennifer Prendki, Founder and CEO of Alectio