Flyte’s Kubernetes-Native Workflow Engine Propels Freenome’s Cancer Detection Research

The company

‍Freenome is a biotechnology company based in San Francisco, California. The company is made up of veteran medical and healthcare industry professionals, molecular biologists, computational biologists, machine learning and software engineers and data scientists working to develop a next-gen blood test for early cancer detection. 

The challenge

Early detection of cancer–before it metastasizes–is key to improving survival rates. But early detection is, to put it mildly, incredibly difficult due to the low levels of biomolecular ‘signal’ that would indicate cancer growth. Using machine learning to identify, amplify and track this signal using biomarkers from many different types of molecules requires lots of compute power and orchestration capabilities.  

As a bioinformatics and machine learning company, Freenome is skilled at pushing immense amounts of data from its production systems to its research teams so data scientists can work their magic. But bioinformatics involves extremely complex, large and heavy workflows from both the company’s wet lab and from external public data sources. Pre-processing molecular data into a form that allows for feature engineering and machine learning tasks like hyper-parameter optimization, training and inference is critical, and a small subset of all the work in research makes it into the company’s product pipeline, Jeev Balakrishnan, Staff Software Engineer at Freenome, explains.

The solution

All of Freenome’s workflows and processes run on Kubernetes–it’s ubiquitous for operating and monitoring the company’s large cloud workloads. Freenome needed a Kubernetes-native workflow engine, and Flyte fit the bill.

“There’s actually less than a handful of [Kubernetes-native workflow engines], and Flyte was one of them,” Balakrishnan says. “But what was also important to us was that it needed to have a sense of credibility and to be battle-tested.”

Flyte’s status as a Linux Foundation graduate project lent it even more credibility and validation. Flyte’s origins at Lyft and its ability to run more than a million workflows per month across 500 million users was even more proof that the solution could handle anything thrown its way.

“Lyft has a really rich history of using open source software including Envoy and Amundsen, so we felt very good about adopting the technology,” he said. 

“It’s not an understatement to say that Flyte is really a workhorse at Freenome!”
— Jeev Balakrishnan, Staff Software Engineer at Freenome

Flyte underpins every one of these components across Freenome’s data engineering and data science stacks. While most of Freenome’s workloads are currently running on Kubernetes, Flyte’s extensible plugin system is important to the teams, Balakrishnan says.

“This was really important to us because Flyte is execution engine-agnostic. If you wanted to run Spark workloads or Pytorch workloads or AWS batch or Google AI platform, it’s a matter of simply writing a plugin that would work,” he says.

It’s simple to write a Python function to run workloads on Flyte, which can be done locally, shipped remotely and run at scale on a Kubernetes cluster. Flyte’s immutable transformation capability enables tasks to run within containerized apps with version control so changes can be easily tracked and provides a convenient layer of abstraction for data science teams, Balakrishnan says. Because bioinformatics are resource-intensive, compute-heavy and very expensive, this feature is important to avoid running the same tasks using the same inputs repeatedly. 

“We might have a simple, ‘cheap’ task and a really expensive alignment task; both of them can run within the same workflow just by specifying the appropriate resource requirements,” Balakrishnan says. “And, as it turns out, you can also override these at runtime. So if the workflow sees a really large file, it can adjust its resource requirements on the fly for the child tasks,” he says.

What’s more, Balakrishnan says Flyte’s functionality allows data engineers and data scientists to call into reference workloads that are not part of an existing project but that are pre-registered with Flyte. This capability allows teams to experiment with featurization from within the project and share the work that’s been done with other teams. This adds a lot of space for collaboration, he says. 

In addition, basic tasks like getting pull requests approved and accessing help and support from within the Flyte community has been seamless. Another key aspect is Flyte’s accessibility for users no matter what their level of programming expertise, says Balakrishnan. Because Freenome is a cross-functional company, there’s a wide range of engineering skills.

“It allows us to grow as our needs evolve,” he says. “We can keep a very basic UX for users and then run all these different engines, each with their own pros and cons, and the Python SDK means there’s less work for us to do. It’s simple to write a Python function to run workloads and then ship that remotely,” Balakrishnan explains.