Sustainability is more relevant nowadays than ever. Flying produces at least 25x more CO2 than travelling by international train, which drives many to look for different modes of transport to reduce their ecological impact. This shift in consumer priorities is an opportunity for international trains, which offer a more sustainable alternative to travel.
With this backdrop, NS International seeks to grow its share of the international train travelers' market, and to connect customers with the parties that drive these trains. Facing NS International, however, is a complex market that the train operator aims to simplify by unifying different sales platforms into one single simple interface.
Every carrier (a party that drives their own trains) has its own pricing system, and NS International connects these different platforms and attempts to sell their services as one product. In order to do this, NS needs to design an interface which should be able to convey all the necessary information without overwhelming the visitor. To optimize this interface for all the website visitors, it is necessary to start experimenting.
We started by working with a tool called “Visual Web Optimizer”, which allows making content changes on the website and then splits the traffic randomly between the web content variants. Because the only variable in this situation is the difference between the variants, one can reliably test hypotheses and build on those results.
The more we started to test, the more we ran into the limitations of the used tool. Especially in our own interactive booking module, VWO was not powerful enough to test changes. This is where our development team came in. Together with them, we designed our own A/B-testing code, which sent its results to Google Analytics. We then analyzed those results using Bayesian statistics programmed in our own analysis scripts. Using this form of statistical analysis allowed us to more easily interpret experiment results and communicate them to management and other stakeholders.
When starting out with weaving A/B-testing into the existing process, we ran into a big “moment of truth”. The development team had worked on a new feature that they wanted to implement. We discussed putting it to the test and eventually did test it (“meten is weten”! Or measuring is knowing).
According to our results, this hard-worked-on variant turned out to be a “loser”, a test variant which performed significantly worse than the old version of the website. This created friction between the development team who suddenly got blocked from releasing their feature, and our new working method backed by data and statistics.
After discussions about this and receiving support from the management following the realization that this is what data driven working entails, we worked together with the development team to tweak their feature according to our results. After retesting, this feature became a “winner”! The tweaking we did? We swapped the color scheme that was used to mark search results… Small changes can make a big difference!
This experience created a turnaround in the local culture at NS International. The development team started to see the value in making small iterations and testing them as soon as possible. After a few years, we’ve done tens of tests, ranging from smaller to bigger changes, from A/B test to A/B/C/D tests. Part of releasing changes became to always test the effects they have on customer behavior.
By calculating the difference in conversion, and by combining it with the traffic on that page and the average order value, one can make an estimate of the impact of an A/B test on revenue. Sometimes we also simply didn’t make a significant impact on the revenue with a variant. We’ve done tests where the A/B test validated variations that improved conversion rates in such a way that it’d yield us €160.000 per month. On other occasions it saved us from implementing a variant of the booking module which would’ve lost us €220.000 per month!
To reduce the impact of delays on the operation of KLM Royal Dutch Airlines, a Xomnia team designed and implemented one predictive model and revised and improved another, going from problem identification to production environment in under three mon...
