About a year ago, the Tokyo 2020 Summer Olympic Games took place, where over 11,000 athletes competed in 50 different disciplines. Every athlete had their own tailormade plan and goal during the years of preparation and training that precede going to the Olympics. Putting in tremendous amounts of effort and dedication for multiple years is something that can be described as the ultimate form of discipline to achieve a goal.

The first athlete had the goal of winning a gold medal, but instead won a bronze medal. When interviewed, she started crying, expressing how disappointed she was because all the effort that she put in the years leading up to the Olympics doesn't justify the outcome. She concluded that her performance was “a complete failure”.

The second athlete had changed her goal from winning a medal, to participating in an Olympic final. This adjustment in her goals made sense, following a tough year during which she endured quite some injuries. As she crossed the finish line, she thought to herself happily: “An unbelievable 4th place! This exceeds my wildest expectations.”

The two stories above demonstrate that the perception of success depends on the predefined goals that one sets for themselves. Being successful doesn't mean getting the results to a 100% or finishing in first place; it's about the accomplishment of the goal that you set, or reaching the highest achievement that your full potential can get you. This also means that the goal that you set, or what you perceive to be a success, can change over time.

We see the same in AI projects: Determining your success is based on measuring the goal that you set before the start of the project. A 2019 VentureBeat research states that 87% of data science projects never make it to production. One could argue that without making it to production, the goal set beforehand for these projects is most likely not achieved - Unless, of course, the goal is to only experiment with AI. After all, as the example above shows, what counts as failure for some, can count as success for others.

When all you have is a hammer, everything looks like a nail. By forcing AI to be a solution for a certain problem, it's likely that you will end up with an unsolved problem. Adhere to real and carefully scoped problems and match the relevant technology to the problem, not the other way around.

1. What is the basic question to be solved?
2. What is the perspective/context of the problem?
3. What is the scope of the solution space?
4. What are the criteria for success?
5. Who are the main stakeholders / decision makers?
6. What are the constraints in the solution space?
7. What are the key sources of insight of the problem?

After you’ve answered these questions with the problem owner, you can make a first assessment into what kind of solution is needed and start filling in the Use Case Canvas as discussed in the previous post in this blog series.

Now, we have established that we need to set goals in order to measure the success of our AI project. Next, we need to define what types of success we can have in our AI project, and what goals lead up to that success.

Begin by defining the kinds of success that you want to achieve, which, based on Critical Success Factors for Artificial Intelligence Projects, can be grouped into one of the following categories:

When talking about adding value, people refer mostly to earning or saving more money in comparison with the status quo. An example of a commercial goal is: “By automating the manual process of assessing insurance claims, we want to save up to 3 FTE, resulting in a yearly saving of 120K Euros”. Another example is: “By personalizing the recommendations given to our customers, we aim to upsell by 2 products per customer on average, resulting in a 10% increase in the profitability per customer.”

Contractual success can refer to achieving the intended outcomes, which are often agreed upon with a project's stakeholders beforehand. This can refer to commercial goals, but also other activities. For example: “We want to set up a complete cloud infrastructure to be able to develop and run machine learning models in a production environment.”

An example of an ethical success goal for an AI-project can be the elimination of the biased outcomes that can naturally happen when humans are involved in a given process. For example: “We want to support the social welfare allocation process with an explainable and transparent model.” This can also be measured in the form of fairness metrics to define as success factors.

This success category is as close as it comes to assessing the technical performance of AI models. For example, the operational improvements on, for example, logistics processes due to the implementation of an AI-driven product: “With the automation introduced, the quality score on this process, which is 65% when done by humans, goes up to 95% when using this AI model.”

Some view an AI project as a failure because its success can’t be measured based on any of the aforementioned categories. Others, however, see the process itself as a learning success that may tie together the organization's goals regarding the project in the first place. In fact, an increasing number of organizations have “experimenting with AI'' as a goal in their yearly strategies, under which they set goals such as: “This year, we want to experiment with AI by doing 3 Proof of Concepts in order to improve knowledge and experience within AI in our business.”

—