Summary of learnings from keynotes at the #expca2020
I watched the livestream of the first ever Experimentation Culture Awards this afternoon which was brought to us by Optimizely and Online Dialogue. There were awards in a few different categories - individual, team and organization - along with an impressive line-up of keynote speakers.
That the show would focus on culture was clear from the website itself:
“It's not about that one experiment, it's about improving the process, structure, trustworthiness, democratization and motivation for experimentation and data-driven decision making.”
Here are some of my top takeaways (I am guessing once I look at the recording, I will find tons of things I missed in my note. A follow-up post, may be.)
- Randomized controlled tests or A/B tests are an essential source of learnings since the quality of evidence is high and the risk of bias is low.
- As per Ton Wesseling, just being Agile without experimentation and validation is like running fast with your eyes closed.
- Love your losers! Stephen Pavlovich (CEO & founder at Conversion.com) said that experimenters should not shy away from sharing results of losing tests. In fact, in his experience, these “saving” experiments can get 10X more exposure than winning tests. So, go ahead and tell everyone how much money you saved the organization by testing something that lost!
- More and more companies seem to be using painted door tactics to quickly get a gauge of customer demand/ willingness to make a decision one way or the other before investing a ton of dev resources.
- “Make the HiPPO the hero and you be the sidekick.” was an absolute gem from Marianne Stjervall (CRO Lead at COOP Sweden). She suggested that experimenters should attend meetings, listen humbly and understand the stakeholders to know what frustrates them and what their bottlenecks are. Early on, don’t test your hypotheses and let the HiPPO decide. Your time will come!
- She also suggested evangelizing experimentation and educating others in the organization about running simple tests through lunch and learns, master classes etc. Share your passion with other teams.
- Development resources are crucial to long term success of an experimentation program. As per the research done by Hazjier Pourkhalkhali and others, lines of code per variant significantly determine the lift in primary metrics. In fact, the lift that bolder tests can bring is 2X the smaller “button color” category of tests.
- Cassandra Campbell from Shopify talked about focusing on deliberate learning. By logging everything around the tests you run, you can create a super repository of your own tests that can be used for driving time sensitive decisions, improving program effectiveness and disseminating learnings across the organization.
- Kelly Wortham, TLC founder, talked about the importance of industry engagement in an experimenter’s journey on the slope of enlightenment. She shared several great resources and platforms to stay informed about the latest and the greatest in the space and also ways to give back to augment our collective learning.
- Last but not the least, optimization is not about just running a few good tests. It is about putting in place the tools, culture and people required to sustain an experimentation mindset. It is about embracing humility. It is about democratizing data driven decision making. It is about the leadership giving away their power. It was a treat to hear Lukas Vermeer briefly describe the culture at booking.com and the tools they have put in place so that everybody (right from a copywriter) feels empowered to run a test idea they have without the fear of breaking anything.
This quote by David Vismans (Chief Product Officer, Booking.com) shared by Hazjier Pourkhalkhali summed up the challenge for leadership:
“You need to ask yourself two big questions:
How willing are you to be confronted every day by how wrong you are? And how much autonomy are you willing to give to the people who work for you?
And if the answer is that you don’t like to be proven wrong and don’t want employees to decide the future of your products, it’s not going to work.”
All in all, it was a well spent three hours. The show was a celebration of organizations that have managed to or are making a shift to an experimentation culture. Nominees were all those who were focusing on a more sustainable and democratized experimentation practice. I sat in awe listening to the transformation that many of the nominees had achieved. Some in a matter of months and others patiently working away for years. It was both humbling and inspiring and I am definitely looking forward to future installments of the awards.
p.s. I highly encourage readers of this post to get access to the recordings. Besides the regular awards and the nominees videos, there was a lifetime achievement award given to none other than Ronny Kohavi from Microsoft. His acceptance speech had a few gems which unfortunately I had missed. So, definitely get the recording!