How Spotify Uses Flyte to Automate Complex Financial Forecasting

The company

Spotify is a leading digital music, podcast and video service that gives users access to millions of songs and other content from creators all over the world. With nearly 200 million subscribers in 180 markets, Spotify’s Streaming, Automation and Tooling, Data and Insights, Business Modeling and Forecasting, and Data Workflow Scheduling teams work continuously to support the business’ growth.

The challenge

Growth involves expansion into markets with complex and varied needs, so Spotify’s finance team relies on precise business casing and scenario analysis to predict revenue. In fact, every quarter as part of its compliance requirements the finance team forecasts profits and losses for two years. 

Dylan Wilder, an engineering manager on Spotify’s Vivaldi finance team, explained that eight teams from Spotify all contribute their respective workflows to produce those forecasts, which incorporate historical trends and key new business initiatives. The exercise involves as many as 25 people across those eight teams, running more than 15 models on various areas of Spotify’s business — such as premium subscribers, streams, ad inventory, music ad revenue, and royalties. 

According to Wilder, “The teams usually span different business verticals; P&L line items (such as revenue vs. royalties); and specific markets. Specific models belong to specific teams, and those teams are either responsible for specific verticals (like podcast) or horizontal needs (like royalties).”

This end-to-end process typically unfolded over three to four weeks every quarter, with distributed ownership of each segment. Multiple inputs for licensing, contracts, royalties and other factors had to be validated before a manual handoff to the next step could take place. 

Once all the models had been run, the outputs were consolidated and the resulting forecasts were delivered to FP&A and business leadership as well as the C-suite for critical business planning and financial guidance to investors, Wilder noted. But with the teams involved using multiple tools and technologies, running their models in isolation, and needing validation for each step before a manual handoff to the next step could take place, the forecasting had become rife with errors and delays.

To streamline the process, the digital media giant needed to create a financial forecasting model on a cohesive platform that would unite disparate teams, automate numerous workflows and accelerate the turnaround time for feedback.

Wilder described their main goal: “It’s an automation problem. We want to fix these manual handoffs. We want to speed up the feedback time to do this process, and we want to reduce the errors.”

The solution

Using Flyte as its model’s main runtime engine, Spotify was able to build a single forecast platform, called One Model, to crunch the data in real time. One Model unifies the disparate workflows — from teams including Automation and Tooling, Data and Insights, Business Modeling and Forecasting, and Data Workflow Scheduling — into Spotify’s financial forecast ecosystem. 

One Model’s key components are an assumptions bank, where users input the future assumptions used in the models, and the forecast engine, which model execution through Flyte, which gives Spotify increased development velocity that allows the organization greater speed for data processing and machine learning. 

Wilder said Spotify chose Flyte because it offered some key benefits to the streaming service’s data scientists and analysts:

• Greater workflow composition. Different forecast models can connect with little to no refactoring (or changing of the internal code while preserving the way the code functions externally).
• Workflow shareability. One Model workflows can include other teams’ forecast models, which are easily fetched by Flyte. 
• Parallelism. Flyte figures out what complex processes can be run simultaneously so users get their results faster.
• Caching. Forecast models don’t rerun if assumptions haven’t changed, regardless of who ran them before.
• Native cloud storage. Data scientists pass DataFrames from one task to another, and Flyte automatically captures snapshots in Google Cloud Storage.
• Native validation and tests. Flyte’s registration process has many validations and tests available natively to users.
• Versioning. Flyte supports immutable records, so critical records won’t be overwritten.

“As engineers, a lot of this might be table stakes for us, but for data scientists, being able to get up and running on Flyte and getting all of this stuff for free is a really big win for them,” Wilder said. “All of this stuff — they've been really excited about it.”

The result 

Through its partnership with Flyte, Spotify achieved its goals of reducing time-to-forecast, increasing the number of runs possible in a quarter and minimizing the number of human errors.

A second benefit to using Flyte to automate the process, Wilder said, is that business casing and scenario analysis can now happen more frequently. “The number of forecasts run went from being limited to once per quarter involving all teams to thousands [of forecasts] for analytics purposes. We still have some work to do on reducing errors related to user configuration, but we've eliminated those related to manual handoffs.” 

“The overall quarterly process has gone from four weeks to less than two.”