Cameo speeds up experimentation workflows by ~90% and centralizes experimentation reporting in Eppo
About
Cameo is the leading direct-to-consumer fan engagement platform, creating over four million magical moments for fans since its launch in 2017.
Driving new and compelling initiatives to optimize customer engagement requires fast, data-driven decision-making. Experimentation offers real-world feedback; yet data teams, execs, and stakeholders all bring their own concerns and priorities to the table. Better experimentation offers flexibility with tools for transparency, visibility, and democratized access.
The Problem
“In the past, Amplitude was used for ad hoc experimentation and it led to misguided or misinformed conclusions and decisions.”
Cameo’s small but mighty product data science team was patchworking Amplitude and Looker to perform experiment analysis; the manual lift and limited flexibility made for laborious, repetitive work with the inability to produce uniform metric definitions or statistical rigor.
Sarah Lillian leads the product data science team at Cameo. She holds a PhD and has years of experience in scientific research, and has worked on advanced experimentation at Groupon before coming to Cameo. For Sarah, upscaling Cameo’s experimentation practice was tedious but critical: “In the past, Amplitude was used for ad hoc experimentation and it led to misguided or misinformed conclusions and decisions.”
Amplitude experiments were quickly replaced with more sophisticated, templatized notebooks built by Sarah herself. Although using these notebooks allowed her team to pipe the right data and add more statistical rigor, it was an extremely manual process that lacked metrics standardization - every notebook took time to set up and required custom SQL to wrangle the data.
The Turning Point
After months of manual notebooks configuration and setup, Sarah realized that this manual approach was not scalable, and her small data team didn’t have time or resources to build their own experimentation infrastructure. When reflecting on her team, Sarah recognized, “They were stuck in a loop of one-off, bespoke analysis rather than building out a system.”
In Need of Something Better
Sarah’s manager introduced her to Eppo, an end-to-end experimentation platform for the modern data stack. Sarah reached out and started a proof-of-concept with the team at Eppo.
Because Eppo sat on top of Cameo’s data warehouse, defining Cameo’s most important metrics was easy. “I basically just gave Josh (Solutions Engineer at Eppo) a bunch of key SQL snippets, making the setup very easy. There were very few obstacles.”
Within a short working session, Cameo’s entire metric repository was defined and stored within Eppo to be used for experimentation analyses. Cameo had solved the problem of having a standardized set of metrics in about an hour of collaboration between Sarah and the team at Eppo.
Not only had metric definitions been standardized, but by moving to Eppo’s Sequential framework, Cameo could call promising tests early without worrying about inflating false-positive rates. “One of the biggest things I’d emphasize about Eppo is the ability to end experiments early, which is something we didn’t have the support and infrastructure to do previously.”
90% More Efficient
Today, Sarah estimates that data scientists on her team take close to 1 hour to complete what previously took 9 hours of work for every experiment they set up and ran.
- They no longer waste time setting up notebooks for sample size estimation and experiment configuration.
- Various stakeholders (including product managers and executives) can now self-serve the status of the experiments on an ongoing basis. This takes the pressure off the data team to be able to prioritize and manually provide regular updates.
- After an experiment is run, data scientists no longer spend hours compiling reports.
Everything stakeholders need to know about an experiment is quickly retrievable in Eppo and presented in a way that everyone can understand. “This brings a certain amount of standardization and guarantees that we are all speaking the same language and interpreting the results through the same set of rules.”
Execs and Data Teams in the Same Room
On top of the time savings across the experimentation lifecycle, Sarah is able to bring Eppo into weekly business check-in meetings where executives from various business functions review experiment results.
“Rigor, visibility, and transparency. That is Eppo. Eppo puts everyone on the same page. Everyone is using the same picture and the same numbers to understand experiment results. It is no longer a game of telephone where a data scientist analyses an experiment, writes something up, shares it with the product owner, and then the product owner crafts their narrative and shares it with executives. Now, executives have very simple, easy access to look at the result of an experiment, which all use the same standardized definitions for metrics.”
