Flyte Monthly Oct 2022

This blog post is an edited, public copy of Flyte Monthly’s October 2022 newsletter issue. To subscribe, visit getrevue.co/profile/flyte.

What’s Up?

Flyte returned to Hacktoberfest — but not alone. This year, UnionML, an open-source Python framework built on top of Flyte, joined the fun!

Making contributions is super simple! For one month, Hacktoberfest contributors can choose any “hacktoberfest”-labeled issue on GitHub, create pull requests linked to the issue, and fill out this form to claim their swag. Details were featured in the Meet Flyte and UnionML at Hacktoberfest 2022 blog as well as this recording from a Flyte Community Sync event - Flyte at Hacktoberfest 2022.

Customer Success

Infinome Biosciences

Infinome Bio is a biotech startup working toward a sustainable bio-enabled future, using advancements in gene editing and lean bioengineering to unlock the potential of synthetic biology for highly scalable, safer, affordable and accessible solutions.

Data Scientist Krishna Yerramsetty talked to us about Infinome’s journey with Flyte, highlighting some challenges that the industry faced, how Flyte offers features that drastically cut processing time, and why Flyte was the right fit for them. According to Krishna:

“Biology is always the bottleneck. It’s hard, so we want to make life easier for data scientists. Even if that means sometimes custom pipelines need to be built, or we need to spend a lot of time trying to make a workflow file format easily readable by biologists, that’s definitely what we want to do. We don’t want to take shortcuts for data science and make life easy for biology, and that’s where I think Flyte really really shines for us.”

Catch the full story in the video How Infinome is Using Flyte - Krishna Yerramsetty.

Tech News

Flyte’s Latest Releases

Flyte v1.2.0 was released on Oct. 3!

The most notable feature of this release, Flyte’s Ray integration, was thanks to valuable contributions from Spotify — one of Flyte’s earliest customers — who identified a pain point, talked to us, and worked with us on feature requests and PRs that resulted in Flyte improvements with significant performance gains.

The Ray integration supports large nested and dynamic workflows so they can be shared across teams or implemented as part of even bigger workflows. It can now reduce the size of Kubernetes resources for large Flyte workflows, by having FlyteAdmin write parts of the CRD to storage before it creates the Flyte workflow resource on Kubernetes. FlytePropeller then reads the CRD parts when handling the Flyte workflow resource. Learn more on the Ray integration in the blog Ray and Flyte: Distributed Computing and Orchestration by Kevin Su.

In addition to Ray support, v1.2.0 includes:

• Perf improvements
• Two Flytekit plugins: DBT plugin and hugging face
• Improvements to CLIs, including changes to PyFlyte Run

This is How We Do it

Flyte Deployment Journey

Follow Union.ai Software Engineer Eduardo Apolinario as he takes you through performance improvements in Flyte v1.2.0.

The video Flyte Roadmap — Deployment Journey, streamlines Flyte’s Getting Started, and a typical deployment journey, from a self-contained Flyte environment, to cloud deployment; production-ready; and finally, operating core infra.

Flyte Cheat Sheet

Introducing the all-new Flyte Cheat Sheet!

What’s in it?

• Code snippets of crucial Flyte concepts
• Latest fundamental Flyte features tied to Python code
• Flytekit syntax and features

Who is it for?

• Data and ML engineers

Why use it?

• To reduce the time invested in Flytekit coding

For more information and a download link, check out the Flyte Cheat Sheet blog by Union.ai Tech Evangelist Samhita Alla.

Community News

Tweet of the Week

The Twitter thread on rebuilding Data/ML stacks using Flyte and other tools was going strong for almost a week, earning over 3.1K engagements! Check it out!

Talks / MeetUps / Community Events

ML Hangout

Flyte and Ray developers recently came together for an in-person hangout to discuss all things Kubernetes.

Anyscale Engineering Manager Richard Liaw presented the technology behind Ray, while Union.ai Software Engineer Eduardo Apolinario took a deep dive into the Flyte-Ray integration.

Keep an eye out for the next one!

Open Source Summit Europe

Merantix Momentum MLOps Lead Dr. Fabio Grätz presented at the Open Source Summit in Europe “Building Robust ML Production Systems Using OSS Tools for Continuous Delivery for ML (CD4ML).” The talk discussed how Merantix Momentum:

• Lives a DevOps and MLOps culture
• Introduces software best practices to ML, and
• Builds robust ML production systems using vendor-agnostic OSS tools such as Flyte, Mlfow, Squirrel, Hydra and Seldon.

Watch the recording on the Linux Foundation YouTube channel.

Global DevSlam

Flyte Co-founder Haytham Abuelfutuh and Union.ai ML Engineer Niels Bantilan introduced Flyte and two Flyte Ecosystem projects, UnionML and Pandera, in these talks at the GITEX conference in Dubai:

• Unveiling Flyte: The ML workflow behind Lyft, Spotify, Freenome and others
• UnionML: an MLOps framework for building and deploying ML microservices
• Deep dive: Introducing the Pandera project to make data-processing pipelines more readable and robust

KubeCon + CloudNativeCon North America 2022

Flyte will be participating at KubeCon, sponsored by Union.ai. The physical booth will feature demos and videos, and a chance to connect with Flyte developers and maintainers.

The virtual booth will feature videos and live office hours, including a demo of the latest Union-Hosted Flyte Sandbox. Try it out at sandbox.union.ai.

Flyte Quarterly Planning Meeting

To respond directly to the needs of our Flyte users, we are opening up the planning process to the community.

Join Flyte maintainers in the next Flyte Quarterly Planning Meeting in early January 2023. Take part in improving the planning process, increasing community involvement, and surfacing high-level feature requests to help drive the roadmap going forward.

In the meantime, send us suggestions on Slack in the #ask-the-community and #feature-discussions channels, or ping @Eduardo Apolinario (eapolinario) directly.

Publications

Orchestration for Data, ML & Infrastructure

Zwift Senior Data & ML Engineer Sarah Floris and Union.ai Software Engineer Samhita Alla put their heads together to formulate this long-awaited guide to data, machine learning and infrastructure-orchestration techniques. Make an informed decision about which one works best for your use case.

Thank You, Contributors

Flyte would like to recognize ✨Contributors of the Month✨ of September, for their valuable work and contributions:

• Zev Isert @zevisert, for working on a UnionML PR that checks and warns about dirty Git index during deployment.
• Arief Rahmansyah @ariefrahmansyah & Eduardo Apolinario @eapolinario, for adding a Flytekit plugin for DBT (available in the upcoming Flytekit release).

And many thanks to the whole Community of Contributors! Every PR counts!

Join Us!

Be part of the Flyte Community by joining any or all of the following:

• Slack: Ask a question, learn from a thread, follow announcements, or just say hi!
• Twitter: at @flyteorg for everything Flyte.
• Zoom:
      • “Weekly on Wednesdays” office hours: three 30-minute sessions at 7:00 a.m. PT, 1:30 p.m. PT and 9:00 p.m. PT
      • Community Sync: Join us twice a month on Tuesdays at 9 a.m. PT for presentations and open discussions.‍
• GitHub: Post a question or comment on GitHub discussions, open an issue or show us your work!