Experimentation Culture Awards 2020 — Top 10 Takeaways

Summary of learnings from keynotes at the #expca2020

Experimentation Culture Awards 2020

“It's not about that one experiment, it's about improving the process, structure, trustworthiness, democratization and motivation for experimentation and data-driven decision making.”

Here are some of my top takeaways (I am guessing once I look at the recording, I will find tons of things I missed in my note. A follow-up post, may be.)

  1. As per Ton Wesseling, just being Agile without experimentation and validation is like running fast with your eyes closed.
  2. Love your losers! Stephen Pavlovich (CEO & founder at Conversion.com) said that experimenters should not shy away from sharing results of losing tests. In fact, in his experience, these “saving” experiments can get 10X more exposure than winning tests. So, go ahead and tell everyone how much money you saved the organization by testing something that lost!
  3. More and more companies seem to be using painted door tactics to quickly get a gauge of customer demand/ willingness to make a decision one way or the other before investing a ton of dev resources.
  4. “Make the HiPPO the hero and you be the sidekick.” was an absolute gem from Marianne Stjervall (CRO Lead at COOP Sweden). She suggested that experimenters should attend meetings, listen humbly and understand the stakeholders to know what frustrates them and what their bottlenecks are. Early on, don’t test your hypotheses and let the HiPPO decide. Your time will come!
  5. She also suggested evangelizing experimentation and educating others in the organization about running simple tests through lunch and learns, master classes etc. Share your passion with other teams.
  6. Development resources are crucial to long term success of an experimentation program. As per the research done by Hazjier Pourkhalkhali and others, lines of code per variant significantly determine the lift in primary metrics. In fact, the lift that bolder tests can bring is 2X the smaller “button color” category of tests.
  7. Cassandra Campbell from Shopify talked about focusing on deliberate learning. By logging everything around the tests you run, you can create a super repository of your own tests that can be used for driving time sensitive decisions, improving program effectiveness and disseminating learnings across the organization.
  8. Kelly Wortham, TLC founder, talked about the importance of industry engagement in an experimenter’s journey on the slope of enlightenment. She shared several great resources and platforms to stay informed about the latest and the greatest in the space and also ways to give back to augment our collective learning.
  9. Last but not the least, optimization is not about just running a few good tests. It is about putting in place the tools, culture and people required to sustain an experimentation mindset. It is about embracing humility. It is about democratizing data driven decision making. It is about the leadership giving away their power. It was a treat to hear Lukas Vermeer briefly describe the culture at booking.com and the tools they have put in place so that everybody (right from a copywriter) feels empowered to run a test idea they have without the fear of breaking anything.

“You need to ask yourself two big questions:

How willing are you to be confronted every day by how wrong you are? And how much autonomy are you willing to give to the people who work for you?

And if the answer is that you don’t like to be proven wrong and don’t want employees to decide the future of your products, it’s not going to work.”

On the path to learning all things insights and optimization (linkedin.com/in/bithikamehra) | Foodie | Environmentalist| Loves to travel | Player of a few riffs

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store