How to Craft Compelling, Impactful Experiment Reports

As someone who has run countless growth, product, and marketing experiments at tech companies, there are few things more worthwhile than writing a great experiment report. Terrific writeups get your leadership and colleagues excited about the learnings and value that your team is delivering, and open up opportunities for expanded experimentation.

In tough economic times, it’s even more important that you as an experimentation leader consistently show the value that your experimentation team delivers. The golden rule: never pass up an opportunity to write up and share results from an experiment. Some reasons why: 

• Experiment results serve as guideposts for product and marketing teams, helping them more efficiently deliver ROI and business value.
• Reports are a way to demonstrate that your experimentation team is continuing to execute, even if actual features aren’t being shipped yet.
• If your experiment delivered important learnings but nobody knew about them, is your company really learning anything?
• Generate FOMO (fear-of-missing-out) energy. Once, my team ran a small UI experiment that demonstrated high engagement gains with a simple design change. The report was posted on our company’s data-related Slack channels. Our CEO picked up on it and called it out as a terrific example of original thinking backed by hard data – “exactly what the company needs.” Within days, we were being contacted by every product team lead asking how to run UI experiments on their features – everyone wanted to be next in line! This is the flywheel in motion.
• Reports teach people how to run experiments. Someone new to the company or the team can see the types of hypotheses being tested, how people dive deeper into metrics, and even basic logistics like how to use the underlying experimentation infrastructure.
• They provide avenues for up-and-coming data scientists, analysts, or product managers to increase their visibility within the business.  

The time that you spend writing up and evangelizing your team’s work is absolutely worth it – for you, your influence, and your team’s success. It’s also vital for performance reviews. In my conversations about my performance, simply pointing to these reports has been excellent evidence for the impact I’ve delivered to the business.  

I’ve created an experiment report template - you can find it here. I’ll go into detail about several elements of it, and some best practices around writing up experiments. 

1. Start writing early, even before the experiment ends. You can punch out half of the report while the experiment’s still running: fill in the experiment and team details, the hypothesis you’re exploring, the product change you’re testing, the user targeting and randomization approach. I find it so much easier to finish a report that’s already half-written!
2. Experiment reports discuss the “why,” not the “what.” For example, did the feature change increase conversions by 10%? That’s the “what.” The bulk of the report dives into the “why” – why did that feature change cause the 10% uplift? You’ll pull from multiple data sources including user telemetry, event funnels, qualitative evidence, findings from prior experiments, customer segment analyses – to construct a plausible explanation for this finding. This is where the most fertile learning occurs and where new hypotheses are generated for future experiments. 
3. Decide your reader persona and create a narrative. Just like product marketing teams have user personas, I like to think about who’s reading my reports. Ideally, many folks across many positions will read it but I try to hone in on one key persona. That could be the product manager I’m working with, the VP of Product, other data scientists, or even the CEO. Next, what do they care most about – the idea being tested, the magnitude of change in the primary metric, the user behavior learnings, the operational experiment details (probably not)? Craft your narrative explaining the “why” for this persona. This helps you control the scope, focus on key points, and avoid never-ending rabbit holes. 
4. Save clippings left on the cutting room floor for later. You cannot produce ten-page experiment reports, and will have to leave out some interesting findings. A common strategy is to do your deep-dive work within a notebook or BI report, and continually append new code, charts, and summaries to it throughout. The best parts of it will be copied over to the actual experiment report, but your complete data deep-dive will still be available. When people have follow-up questions, you’ll often find you’ve already made that particular chart during the deep-dive! It’s also a great pocket trick for slow weeks – post a chart from it in a data-related channel or daily standup to kick off a new discussion and generate some experiment ideas. 
5. Use real numbers and leave out the jargon. This is hard for experimentation scientists, but most readers don’t know what a p-value is (or worse, have the wrong definition of it). Use numbers the reader already is familiar with. For example, if you typically have 2,000 orders a week and an experiment showed a 10% uplift with a confidence interval between 8% and 12%, say the experiment generated between 160 and 240 incremental orders per week. Given an average order value of $15, write that your experiment has created $2,400 to $3,600 additional revenue per week! Did your company generate $100,000 overall in sales last week? You can say that 2.4% to 3.6% of last week’s total revenue was driven by the experiment – a pretty amazing claim that you can confidently make and that any leader will understand. 

It’s a good feeling when you finish an experiment report and see how rich it is in new understandings about user behavior, product experience, and how they impact key metrics. It’s an even better feeling when respected colleagues start reading them and go “Wow!” and start asking follow-up questions, quoting key phrases, and generating new experiment ideas. Capture these and put them in an experiment backlog. How do you make this magic happen? 

• Identify key channels to share findings – not only data-related channels but also user research (UX/UXR), product, and marketing channels. If there aren’t any obvious areas, create them, or ask your colleagues for their thoughts. 
• If you don’t see vigorous discussion, think about why that may be. What you’re running experiments on may not be where the company’s priorities are. Maybe people don’t understand your report, or worse, don’t trust your process or team. Maybe you’re communicating to the wrong audience. What else? Get feedback and work on these weak areas. 

In conclusion, taking the time to write a great experiment report not only showcases the value of your team's work, but also instills confidence in your audience's understanding of the business problem and your team's approach to solving it. By consistently delivering clear, insightful, and visually appealing reports, you can establish yourself and your team as trusted experts in your field and open up exciting new opportunities for experimentation and growth. Go ahead and check out the experiment report template here!