Studies show that around 60-80% of all machine learning projects don't make it to a production stage and fail. There are a multitude of reasons why these failures occur. In general, machine learning projects follow the “Anna Karenina principle". It could be stated as follows:
All successful machine learning projects look the same, failed ones fail for many different reasons.
Here, I identified the three most common reasons for failure in (supervised) machine learning projects that I encountered so far and provide a possible solution to them.
When trying to solve a problem with machine learning, often the people building the solution are not domain experts but experts in data science and machine learning. In many real-world scenarios the details of the problem can be quite hard to understand for them.
Just talking to domain experts might not solve the problem, since they often lack the knowledge of a machine learning expert to imagine what a machine learning solution might look like and therefore can have a hard time viewing and describing the details of the problem to help the machine learning expert. So beyond discussing the goals of the project and the concrete metrics to measure success, it is crucial to show real outcomes of a prototype to domain experts early and ask the main question: "Is this a possible solution to the problem in question?". This greatly improves everyone's understanding of the nature and complexity of the problem at hand.
It can be hard to tell from the beginning if the data available will be sufficient to provide a useful machine learning solution. At this stage, it is a waste of time to set up infrastructure and code to train and maintain complex machine learning models.
This issue can be tackled by building a minimal prototype as soon as possible. This prototype can then be used to get a baseline estimate on the relevant metrics and even kick-start the application. Aim for a prototype that brings you between 70 and 80% close to your desired performance. Note, that this should be a rule-based system if possible. If you can achieve this, you know that there is some signal in the data to work with. This most likely will allow you to improve upon these results. Once you have trialed the prototype, start analyzing the results and uncover data/label issues the model could not pick up. Once that is done, you can decide on the next steps and potentially move on a more complex model. If you cannot get to 70% with a simple model, then it is often hard if not impossible to solve the problem with the data at hand.
Often when applying machine learning, especially in NLP, the problem you are solving has no pure machine learning solution. The product or internal workflows you are trying to improve or enable need to be adapted to allow machine learning to be applied effectively. In many cases workflows are so manual that it's hard to just patch in machine learning without building a new user experience and interface around the whole thing. This problem is often amplified for companies just starting out with their first machine learning projects. Since the data science team hasn't shown any value yet, it can be quite hard to get the other teams in the company to trust in machine learning solutions. And hence the integration in the final product suffers.
When you start to build a machine learning solution that should improve an existing workflow or enable a new workflow, the integration in the current landscape of any application is of crucial importance. Most likely data scientists and machine learning engineers are not in charge of the product integration themselves. They need to work together with others e.g.frontend developers, product managers, UX designers and domain experts among many others. To achieve success for the machine learning project, these people need to be involved as early as possible. A simple shortcut to get there is to cut most of the machine learning complexity and deploy a basic end-to-end system as early as possible. The system should kind of solve the problem in question. Even though it can be a very basic solution that only acts as a building piece to start the end-to-end integration and start user testing as early as possible. Later, the machine learning solution can be improved step by step to meet the performance requirements.
| Failure reason | Solution summary |
| Problem not understood | Build the simplest possible solution and show it to domain experts |
| Data insufficient | Build simple machine learning model early and work with the data |
| Bad integration | Start deploying and integrating something in the product as early as possible |
As you see, it all boils down to building a reasonable baseline as early as possible. Surely, there are many more things that might go wrong. But I hope you found my tips helpful for your next machine learning project.
The author of this article, Tobias Sterbak, is a machine learning expert and part of our freelancer community. If you are in need of highly skilled data science experts, get in touch with us!