Reaching Production is Far from the Last Mile

The starting point of the AI automation journey is often considered to be the building of an AI strategy, with the process continuing through proof of concept (POC) and finally concluding at achieving production. However, this frame of mind in which the AI journey ends after deployment on day one of production has severe implications for the ongoing success of AI solutions. The belief that the journey ends after deployment may be partially responsible for the incredibly low success rate of AI in production, and perhaps even for some of the disappointing results from AI once it reaches production. For this reason, it is crucial that we start considering deployment as only the halfway point, with the main challenges of trying to keep a model on the rails while dealing with noisy, messy, and dynamic data still ahead. AI automation does not occur on day one and can only be achieved with human-AI interaction in production. By rethinking this journey, we can collectively increase the success rate of AI.

Set Expectations from Day 1

There are almost limitless examples of unsuccessful implementations of AI systems, but all typically share a root cause for failure: unrealistic expectations. However, with AI often being compared to electricity, since it is becoming a fundamental technology requirement to do business, it is crucial that realistic expectations are set and met. But how is this possible when an abundance of things can go wrong: bugs in the software, uncertainty of predictions (predictions with low confidence, and even wrong predictions), new data (that the algorithm is not trained on) due to changes in real-world processes, rejection by the user due to lack of change management, or user resistance from fear of the new “black box” tool. The following steps have proved useful for our team in achieving successful and on-going production in AI.

Automation is a Journey

1. Start with a shadow solution for any human process. The first batch of predictions from an AI system should always work alongside the entire human process. For example, in one big insurance company, claims approval or disapproval is a process that occurs thousands of times in a day. Each claim takes a few days to process, creating a significant amount of work for the insurer. This process can be accelerated with the  of AI. At first, the decision of the AI algorithm should run without affecting the real-world process. This is a short trial period, of a few days, in which the AI uses the real-world data to make predictions as a shadow decision to human decision making.
2. Keep a human in the loop, after the trial period is complete. At this stage all AI predictions should be integrated into the real-world, human process, but only be considered recommendations. Continuing with the insurance example, the algorithm can analyze the data and free text of the claim and provide recommendations regarding authorization. The insurer may use or discard these recommendations. One of the key objectives of this step is that the user begins developing trust in the AI recommendations, compared to the previous step which was training the algorithm on real-world data.
3. Start automating the high-confidence predictions outputted from the algorithm. The higher the confidence (degree of certainty) an algorithm is in its predictions, the more it can be counted on to be accurate and the more the user can begin to automate the consumption of the results. Back to the claims example, suppose predictions for approval have a 95% confidence rate, then the user can be sure that the algorithm is accurate 95% of the time. This would mean that for these higher-confidence predictions, the real-world process can be automated.
4. Give feedback on the low confidence. Once a user has automated the high-confidence results, then human supervision of the low-confidence ones can begin. In this step, the algorithm is given human feedback for its predictions. Whether the user has positive or negative feedback, the algorithm will have a high probability of being improved by them. Improving the model with feedback is a delicate phase. The entire pipeline should be run again with new data fed into the system. The focus at this point is mainly to reiterate with the training using some restrictions on the results that will allow the deployment of the new model.
5. Continue to automate new predictions with high confidence. In the insurance example, the expectation is that over time, more claims will be predicted above 95% confidence. At this point, the human feedback can be leveraged to improve the algorithm, and also create a higher threshold, such as 97%. The overall goal is to achieve more accurate decision-making based on automated AI results.

Back to Expectations

This is real life. Not everything is perfect on the first day – especially AI. Training AI should be approached like the training of a new employee. At first, a new employee typically requires a considerable amount of monitoring, review, and feedback. However, after the employee becomes familiar enough with the material and process, and shows improvement and understanding over time, he/she earns more autonomy and independence. Every now and then, even an experienced employee is bound to make a mistake, but this is where additional feedback can be provided. As with an employee, over a relatively brief period, the amount of required feedback also decreases as AI is a fast learner with a short learning curve. And finally, this new employee or AI solution is no longer new, but now experienced and capable of performing the job independently with minimal mistakes.

By changing the definition of the AI development process, expectations can be better aligned. When referring to deployment as the middle of the process, instead of the end, it helps to emphasize the important role of the dialogue between human and machine. This approach, particularly during production, will empower a variety of stakeholders, not only developers, to automate with baby steps and caution - not due to the fear that AI will fail, but because of the desire for it to succeed. It takes time to achieve full automation, and therefore ROI from AI is not immediate, but with the right expectations and processes, it will increase over time and fulfill its promises.

About BeyondMinds: Founded in 2018, BeyondMinds has built the first enterprise AI solution that is universally applicable and easily adaptable. We deliver hyper-customized, production-ready AI systems that enable sophisticated companies to overcome the massive failure rate in AI adoption and rapidly achieve ROI-positive transformations. The company has more than 70 employees, with the majority being AI technologists. Accelerating AI democratization around the world, we have offices in New York, Tel-Aviv and London, in addition to presence in other countries. We service Global 1000 companies, including Microsoft and Samsung. BeyondMinds is online at: beyondminds.ai