Introducing the Copy SQL Button

Today, we are launching a feature to help analysts reconcile and re-use experimentation calculations: the “Copy SQL” button.

***

Consider a common scenario.

Let’s say that Carla is a Data Scientist at Acme Co. A Product Manager in her org is confused about why an experiment is trending negatively, and guesses that it’s because of “premium tier” users who experienced a different UX design. He asks Carla for an explanation.

In this situation, here’s what we’d see happen:

• Carla’s experiment tool doesn’t support splitting results by a concept like “premium tier,” so she decides to calculate it herself.
• Carla starts by trying to replicate the overall experiment result, but her numbers don’t match those generated by the experiment tool.
• Carla tries to untangle why her numbers are different. Is she including different users? Different metric events? Is there some secret aggregation logic?
• When calculating the report, all of these decisions are hidden within the experiment tool. Despite producing different numbers than the SaaS tool, Carla has no choice but to share her differing analysis with the premium tier segmentation. It shows no effect by subscription tier.
• The PM doesn’t know whether to believe Carla or the enterprise SaaS tool. In the end, the PM still believes there’s a problem with the subscription tier.
• Carla starts to doubt all previous experiments run on the black-box tool. Is any of the SaaS tool’s analysis correct?

Eppo already makes this problem dramatically easier to solve by being a fully warehouse-native app. Now, Eppo’s “Copy SQL” button makes this workflow even simpler, by providing SQL snippets to use for every intermediate table in your favorite query tool like Hex or Looker.

Once you click the Copy SQL button, you can choose from three options for viewing the data: assignment data, user-level aggregations, and aggregated results.

This makes it trivially easy to:

• Reconcile any manual analysis against Eppo
• Examine individual subjects’ performance in an experiment
• Conduct bespoke analyses that aren’t available in Eppo, such as cluster-level analysis

For example, with the assignment data you can see exactly which subjects were in each treatment group, including who was thrown out of the analyses due to cross-device contamination. This can be helpful when matching your manual analysis to Eppo, or for seeing how experiments intermingle with other projects (like marketing campaigns).

In addition, you can now view user-level aggregations of metrics.

For example, if you want to check how different quantiles vary across treatment and control, you now have the tools to do that. Or you can put the datasets into a regression to do your own CUPED variance reduction analysis.

***

Eppo is making it easy for your team to understand what your experiments are doing, by simply taking a look at the relevant SQL snippet. The underlying data is yours to investigate, and you don’t have to know everything going on in the pipeline in order to trust your results.

This transparency is one of the major benefits of Eppo’s warehouse-native architecture. If you were using a non-warehouse-native tool, like Carla in the example above, this data would be inaccessible. Because Eppo sits on top of the warehouse, we can open up the black box, provide your team visibility in the form of SQL, and allow you to build off of what we’re doing.

Interested in getting started with the Copy SQL button and running more trustworthy experiments? The feature is now live for all Eppo users.

Want to know more about Eppo’s warehouse-native architecture? Book a demo today.

‍