Eppo's best-in-class diagnostics ensure that your experiments yield trustworthy, actionable results.
Trust is the underlying principle of experimentation. When you’re allowing a platform to help determine the difference between good and bad ideas, you need to ensure you’re getting trustworthy results.
When we first built Eppo, we decided to prioritize diagnostics, because we know that even the most carefully planned experiments can hit a snag. Common challenges might include underlying data quality issues, imbalanced treatment groups, or an experiment that just doesn’t run properly. 95% of the time, those cases should be identifiable via diagnostics.
In a platform built to encourage cross-team collaboration, the users checking on experiment results are often not the same ones who set up the experiment, so they need a crystal-clear view of any issue in order to find the right stakeholder.
We wanted to ensure that users can self-serve when things go wrong, figure out how to take action, and clearly understand when the issue is resolved.
Today, Eppo is delivering a new-and-improved version of diagnostics, optimized for enterprise customers.
The update includes:
In this update, we’ve delivered an improved user experience for our diagnostics that includes new charts, debug tips, detailed error messages, notifications, and a slick sidebar.
Taking a step back, there are essentially two use cases for diagnostics:
First, notifying you when things have gone wrong in your experiment.
Second, helping you understand whether you can trust your experiment results.
The first use case is crucial, but it’s the second one that really builds the trust that underwrites an effective experimentation culture. At Eppo, we aim to go the extra mile to ensure that you can trust your current results and there's nothing broken underneath.
For example, we will run tests to ensure that metric values are similar between the control and the variant for users in the pre-experiment period. If the pre-experiment metrics are imbalanced, it’s likely that randomization is broken.
We continue to check if users are assigned randomly across the entire duration of the experiment. If an issue is detected at any point in the experiment, we will notify users.
When you navigate to the side panel, we show you the exact SQL logic from top to bottom, providing a transparent window into how our diagnostics work. It’s a way of showing our work and engendering additional trust in the results.
Beyond the UI improvements, our new diagnostics architecture creates an improved experience for large-scale enterprise customers. In the new model, rather than running raw queries every time we recalculate a metric value or re-wrangle assignment data, we go directly to the tables created during the latest refresh of an experiment, read the data for a given metric or assignment, and run the diagnostic inquiry much more quickly. This new architecture will allow us to continue serving enterprise customers with peak efficiency.
Best of all, Eppo users already love it!
To learn more about running experiments in Eppo, watch our video tutorial on setting up an experiment.
Building the Modern Experimentation Stack
The Warehouse-Native Experimentation Workflow
How to Set Up an Experiment in Eppo