Data-Driven Experimentation

Use-Case new home screen for customersScope rapid experimentation in productionIndustryLogistics (RIO)

Initial Situation

RIO is a logistics platform, that offers many digital services and solutions to fleet managers, carriers, shippers and vehicle rental services. They provide a lot of value - however, empirical data shows that many customers are not very engaged after their initial registration. What new users typically see is a very generic home screen that contains some informative updates for power users as well as advertising banners from the marketing department. Essentially, it shows the same information every day for every user with no possibility to interact and no solid information on how to get started. Technically, the content is maintained by marketing people and embedded in a React SPA. One reason for that has been the product-centric organization of RIO in which almost everybody has a clear focus on one of the many RIO services - and not necessarily on the overall platform from a user's perspective. Another aspect is that most proposed changes to the home screen were neglected due to the effort to support all languages and the fear to worsen the experience for some users.

Solution Approach

Technical Excellence
rapid prototyping with UI component library, AWS CI/CD, heterogeneous analytics tools
The LESS mindset
Mindset
customer-centric perspective, ultra-short feedback cycles, minimal effort on all levels

In a joint team with RIO colleagues we set the objective to improve this situation. After defining our initial hypothesis "We can increase engagement of new users after their registration by providing step-by-step introductions instead of the current generic homepage.", we immediately started to work on the concept how that new landing page might look like. We chose a completely use-case-centric step-by-step approach instead of confusing our new users with all the different services and their dependencies. After being guided through the essential first steps, users are able to indicate what they would like to explore next. This not only enabled us to optimize the landing page based on real feedback later, it also helped us to better understand the basic needs of customers who just decided to use RIO - probably with a particular goal in mind.

To launch this experiment with reasonable effort, we took sane shortcuts wherever possible. We only showed the new home screen to users who have registered within the last 14 days and only considered German-speaking customers to avoid internationalization issues. Instead of integrating the CMS content, we just hardcoded everything in our React App. All of that helped us to focus on the hypothesis and minimize coordination with our stakeholders. We kept all relevant state in cookies to avoid a costly backend for our MVP. We intentionally skipped a few engineering best-practices such as full test automation in order to get initial results fast - "done is better than perfect". In a nutshell, we went "customer first, tech second" to make sure we know what to build before we build it properly. The only thing we never put up for discussion was full-fledged user tracking which is imperative for experimentation. For the analysis of results, we chose the most suitable tool for each question and combined data from different sources (Google Analytics, Salesforce CRM, AWS Quicksight) which were recently maintained and used by different groups of people, often without a global perspective.

Within one week, we moved from a very general problem statement to a concrete concept for our experiment. One more week later, the new home screen was live. Collecting real customer data throughout the next two weeks could not be accelerated, but after that one additional week was enough to analyze and visualize it, generate insights, and recommend next steps to pursue. That is 5 weeks of cycle time and 3 weeks of actual work for a complete ideation-to-insights lean product development iteration! This velocity was only possible because we worked collaboratively with a few key people across different teams - and only with them. Instead of discussing our plans in an early stage, we decided to only show done and tangible results to a broader audience in order to collect feedback - and the feedback we did receive was fantastic at times.

Just coming out of the Stakeholder Review. I am absolutely thrilled by what I have seen. There was so much of what we at RIO stand for and want to stand for: starting from the customer perspective, rather getting something done fast than building ages without feedback, create and measure impact to then decide how to continue. And that all with a cross-functional team. I am impressed... Congrats to the Team!
Jan Kaumanns, CEO RIO

We found that our initial hypothesis needs to be rejected because there was no significant difference in user engagement within the first week of activity on the platform. However, our data clearly indicated that we can a ctually guide our users. Almost half of them even finished their onboarding during the initial visit. With that, we could increase the proportion of new accounts that activated at least one vehicle on their first day which enables them to get started with other services.