Experimentation is Not a Strategy

As part of Eppo’s new Humans of Experimentation interview series, we sat down with Hila Qu, a Growth advisor and Reforge Executive-in-Residence. In past roles, Hila has led the growth teams at GitLab, where she launched the product-led growth motion, and Acorns, where she scaled the user base from 1 million to 5 million. 

In this conversation, Hila explains: 

• Why a market downturn demands experimentation to drive growth
• Why a data analyst should be your first growth hire
• And why zero-to-one growth is both “the hardest part and the fun part.”

‍

Can you introduce yourself, what you do, and how you got there?

My name is Hila Qu. I'm currently an EIR at Reforge, teaching courses on Experimentation & Testing and Advanced Growth Strategy. I'm also advising companies on product-led growth, with a very heavy focus on data and experimentation. Prior to this, I was the Head of Growth at GitLab, where I helped them build a product-led growth motion from the ground up. I built the growth team and product analytics team, and introduced a lot of data and experimentation infrastructure.

Prior to GitLab, I was the VP of Growth at an investment app called Acorns. Again, I built the growth team from the ground up. I introduced experimentation, and managed user acquisition, marketing, product, growth, and the data analytics team.

I was actually a biology major in college. I have a masters in biochemistry as well, but I was very bad at doing experiments in the lab. At one point, I almost caused an explosion! 

The idea of scientific experimentation has always made so much sense to me, and eventually I learned that you could do experimentation in business. So that quickly became my favorite thing, and I gravitate towards that in my career. 

‍

I'm sure you have a lot of learnings from running experiments, but what have you learned from teaching Experimentation & Testing at Reforge?

I already knew a lot of the best practices of experimentation because I lived through them myself. 

What I have emphasized in teaching this program is that Experimentation and Testing is not a strategy. It’s a methodology and a very powerful tool, but it's not a replacement for strategy. You need to understand your customer, understand your business, and define a vision for where you want to go.

You cannot just say, ‘Hey, let's test everything so that we figure out what we want to do and where to go.’

‍

Who are the students who take this course?

Nowadays, with the specialization of Product Management, the Growth Product Manager or “Product Manager, Growth” role has become a very standard set up in most B2C organizations, and is becoming more popular in B2B.

With this Growth PM role, there is a lot more room to use data to run experiments, and focus on influencing business metrics as the main goal. They think about how to influence conversion, activation, and retention.

One analogy I like is that typical core product managers are responsible for putting the right pieces on the table. And then the growth PMs are responsible for building a boat so you can package all the features and experience in that journey that you’ve designed for users.

They act as an interaction layer between users and product features – they help them navigate. 

In order for this Growth PM to work efficiently, they need to have stable counterparts and collaborators. They need to have engineers who help them build experiments into the product. They need to have design counterparts who design the experience or flow. And then they need the data analyst, who helps the growth squad understand how to use data to identify opportunities, and how to define experimentation metrics.

‍

How do you start an experimentation culture in a growth org? What are the tactics that you’ve seen work?

From zero to one is the hardest part; it’s also the fun part. 

I’ll talk about my experience at Acorns. When I joined, we had some foundational elements, including an experimentation tool. But the only experiment they had run before was to turn the “withdraw” page red when you tried to withdraw funds from your investment account.

It is obvious the growth org maturity was not that high at that time. So in building the culture there, I decided it was important to emphasize quick wins. When you show results, you attract attention – people don’t immediately ask how you got the results. Instead, they say, “something’s working here; let’s go see what Hila is doing.”

Results aren’t everything. You need to constantly educate your org.

In my first few months at Acorns, I did a lot of interviewing and data analysis and was able to identify a lot of low-hanging fruit. I was fairly confident that if we made some small tweaks, we could drive some pretty significant results. Initially I didn't even have a team. But I used my research to convince our lead engineer to help me launch a series of very tiny experiments. From a simple series of copy tests, we saw a 60% improvement in the metric. 

It was surprising to everyone, and we quickly built upon it, launching other experiments and generating positive results. I took it to my boss and ended up completely rewriting my job description. I said “This is clearly working. I know you hired me to do something else, but we have a big opportunity here. Can you give me engineering resources so I can continue to drive results for you?”

The next thing I learned is that results aren’t everything. You need to constantly educate your org. Even though experimentation has been a long-standing methodology in the scientific world, it’s still not that widely adopted or understood in the business world. 

It’s not many people’s first instinct, and you’ll still get a lot of resistance from all directions. For example, some engineers will say, “Am I just wasting my time building five versions of the same thing for you? Just give me the version that you think will work.” And some designers will say, “You are just trying to trick users, and even if you help the business you will hurt the user experience.”

Your colleagues will have perspectives based on their experience – and often they’re right. But you also need to educate them to look at data, understand the results, and open themselves up to the possibility of alternatives and experimentation.

As a growth leader building an experimentation program, it’s important to be very transparent, to share all of your wins and “learns.” “Wins” are when the experiment works, and “learns” are when the experiment doesn’t work, but you learned something actionable through the process. I like to share that broadly with the entire company, to help generate interest.

At GitLab, I would have people from sales and customer success reaching out to me, saying “Oh, this is so cool. I didn’t know you could do things this way. This data confirms an insight I had from working on this account for several years.” Earning that kind of grassroots support is really important to getting an experimentation culture off the ground.

I created Slack channels at those companies, and made every Wednesday “Experiment Wednesday.” I would share something I tested, and have people guess which version won. Usually, people guessed it wrong. And that's another way to educate them. Based on your expertise or observation, you might think something worked, but actually the other thing worked. 

‍

What’s the most impactful experiment you’ve run in your career?

The impact of experiments can be seen in different ways. For example, that experiment I mentioned at Acorns led to a quick win through a series of simple text changes. 

At Acorns, we had a recurring investment feature – but our flagship feature was called Roundup. Basically, it would round up your investment with your spare change. That’s where all the emphasis was – recurring investment was seen as an old, boring approach.

Without analytics, your growth efforts have no foundation. Data is the foundation that allows you to find opportunities and measure results.

When I got there, I was told that my number one priority was improving retention. Through my data analysis, and talking to team members, I recognized that recurring investment was driving retention, but the adoption rate was low, because all attention went to Roundup as the big flagship feature. 

So I sensed a big opportunity. I designed a series of experiments. First, since we didn’t have engineering resources, I ran a series of copy tests. Then, I was able to win over a designer, who helped me in his spare time to design some small flow changes, and from there I earned some additional resources to make that change. 

In the first 90 days, we were able to almost triple the adoption rate of this recurring feature, and then we saw a pretty visible improvement in the retention rate as well. In the grand scheme of things, it may not be the most impactful experiment in terms of driving revenue, but it demonstrates how you can use experimentation to really prove something and lay the foundation for more experiments in the future.

‍

How does the world of Growth feel different in a recession-like environment? Does experimentation become more important, or less?

I look at this in a couple different ways. One thing is that in a recession environment, retention of your existing customers becomes much more important. You already have them, and you want to keep and ideally expand them. Data and experimentation can be your best friend here, because you already have their usage data, and you know what features they like. You can use that data to predict the next best feature or use case to drive them towards.

The more features they use, the higher the retention, and the higher the opportunity to cross-sell and upsell. 

The other growth lever I work on with my clients is activation. In an environment where capital is abundant, you can cheat growth. Acquisition can be faked if you have money to throw at it. But in a recession environment, you need to think about how to grow more efficiently. 

So you may have a limited budget, but you want to ensure that with the marketing budget you’re spending to drive top-of-funnel, you actually get those users, rather than just pass-through visitors. Experimentation is very important to that process. It can help you figure out how to optimize the onboarding experience, from the first 30 minutes of the user experience through the first several days.

‍

You’ve built several Growth teams from the ground up. How do you know when it’s time to make your first data hire?

Before hiring a data scientist, you should hire a data analyst. One of my clients, an early-stage startup, recently went through this exercise. They don’t have anyone in Growth, Marketing, or Data. They have one PM, a bunch of engineers, and a founder. My suggestion to them was to either hire a Growth PM who can also do analysis, or just hire an analyst instead.

Because without analytics, your growth efforts have no foundation. So you either need a Growth PM who can use product analytics tools and write basic SQL, or you need a data analyst as your first growth hire. 

This analytical expertise is the first step before experimentation, because you can’t just test randomly. You can’t just make random changes and hope they’ll work. Data is the foundation that allows you to find opportunities and measure results.

‍

How has the Modern Data Stack changed the way you work and strategize?

It definitely makes things much easier. In the early days, a lot of the analytics tools and experimentation tools didn’t exist.

Think about the Google Analytics era. You had some analytics, but not at a user level, and you had some experimentation capacity, but it's very disconnected from the data side. You had to build everything natively, and it was slow.

The next-generation product analytics tools are really helping to shorten the cycle to discover opportunities. There’s a lot of built-in analysis in many of the tools. They surface those opportunities for you.

And then a lot of the experimentation tooling allows you to act on those opportunities relatively quickly and easily, and connect those back to your business goals. 

For example, at GitLab, when we ran experiments, we actually didn’t have an experimentation tool. We built things relatively manually in the early days. Whenever we ran experiments in the front end, we could only track certain front-end events, and would need to go back to manually tie the events to the backend, and to revenue. 

Today, a lot of the modern tools will allow for end-to-end measurement, which is really powerful.

‍

What’s the most exciting thing about the data space right now?

There are so many tools that can support product-led growth. I think the data stack is becoming more and more robust, and the application layer of enabling growth is becoming more exciting.

Earlier in my career, it was tricky to run pricing experiments. Now, there are actually tools that connect to your CRM, connect to your product usage data, connect to your data warehouse, and you can easily run feature-bundling, price-point experiments on top of that. 

The data layer has also enabled full-funnel user journey mapping. There was a time when all the different functions used their own tools, and had their data stored within those tools. In a B2B org, the sales team might store everything in Salesforce, and the marketing team would use HubSpot or Marketo. Product usage data would be in the data warehouse. 

Now, the data sources can be all connected. There’s an ideal journey that you can design for a customer, where those data sources work in tandem, rather than every function in a silo trying to fire at maximum power using their own weapon. The whole system has become much more intelligent.

Reforge's Spring cohort is accepting applicants through March 27. Click here to learn more about the Experimentation and Testing program.