7 Reasons Your Machine Learning Project Will Fail

I always tell our consultants that before they do something for a client they should ask “why” 7 times. Why do you want that pie chart? Why does it matter? Why will it help you make a decision? Etcetera. It’s a great way to ensure our projects provide meaningful insights and drive decisions that improve the organizations we do business with.

Given that, I thought it was appropriate to list 7 reasons why your AI/machine learning will fail if you don’t ask your own “7 Whys”.

1. Poorly Scoped/Designed

There’s more to life than time series. Time series forecasts are inherently bad at picking up the impacts of other dimensions. We get better results by clustering then running a time series. Or, better yet, run a 1X, 2X, 5X, 20X demand forecast. X can be hours, days, weeks, etc. The impact of holidays, product type, etc will be much more insightful; plus we’ll be able to forecast with less historical data.

What’s worse than picking the wrong type of model? Trying to do too much with a single model AND trying to do too much, period. Again, there is no one-size-fits-all approach. Trying to forecast apples and cats, you better have different models. Use cluster analysis to group like things first. Trying to forecast sales of every SKU? That’s also a waste of resources. Find the ones with the most impact first -revenue, losses, spoilage, inventory, turnover, etc. Ask yourself why you’re forecasting that product that counts for 0.001% of revenue.

2. Unrealistic Expectations

How accurate is your forecast now? Do you think you’re going to go from 80% accuracy to 99%? You’re probably not. Figure out what it really means to have a more accurate forecast. And figure out what a percentage increase in accuracy means to the company. It’s always good to be more accurate but some people think that if they don’t get 99% accuracy then it’s not worth it; that’s almost never true. Figure out what your target level of improvement is and shoot for that.

It is important to gauge the current state of your organization’s data capabilities before planning your machine learning project, too. Machine learning does not happen overnight and sometimes it is better to tackle data preparation and BI tasks before beginning any machine learning projects. 

3. Bad Data/Lack of Data Prep

Every company has bad data. Every. Single. One. Fix it. Because the only thing worse than no data is bad data. Missing values. Dirty data. Inconsistent data. These are all problems that we can address but the wrong time to address these problems is when the model is being built. It ties back to expectations; if you give me dirty data then I will give you bad results. There are loads of tools we use to prep data – some you may have and some you may have never heard of. Tools such as Trifacta can be used to clean and transform data, as well as more traditional technologies such as SQL views and python can be used too. 

4. Not Enough Data

Are you forecasting water sales but you don’t have temperature data? Are you forecasting capital equipment but you don’t have opportunity data? Are you forecasting preventative maintenance but you haven’t had any failures? We need to find a “signal” in the noise. If you can’t give us enough data there might not be a signal. This is why it’s so important to have an understanding of the business: to make sure a potential data set isn’t being overlooked. 

Referring back to the notion of bad data, not including data because it is dirty is not a viable adjustment in many cases. Deleted data can have a drastic impact on a model and in many cases just needs some additional cleaning. 

5. Too Much Success – Can’t Keep Up With Demand

So, you’ve created a model that very accurately predicts X. Now what? People keep asking you for more. And more. There’s no way to keep up with the requests from the “business” side of the house. Unless you have unlimited time, unlimited budget, or, the right tools. We’ve found that using a tool like DataRobot can help you build lots of models quickly and keep up with that demand. And you don’t need to hire more and more data scientists to do so.

6. Never Make It Into Production

If it’s not repeatable you won’t get results. One-off predictions will help you understand you business but they won’t help your people do their job better. What you need is a system that integrates your data, makes predictions, and, gets the information into the hands of the people that can take action on it. Imagine if Netflix emailed you once a month with suggested content rather than showing you on-screen what you might like. Also imagine if your doctor had to wait a week to tell you that you were about to have a heart attack. Designing the model and determining how it can go into near real-time production is how you truly build value.

7. Not Monitored

You’ve got the right model. Into the hand of the right people. At the right time. Now your accuracy drifts. What you thought was 92% accurate is now 80% accurate. How do you know? Without a system in place to tell you that things your model is drifting, there is no way for you to know that. Platforms exist to determine model accuracy faster than a data scientist could ever dream of.