ML Engineers and Data Scientists are a Luxury that Not Every Company Can Afford

There’s a big mismatch between what many companies need and what ML practitioners would like to do

A little background on this…

I began my career in 2014 after a master’s degree in math. During my studies, I got hooked by programming. It can be really fascinating! You get to build something seemingly from nothing. You begin with a virtual blank sheet and you can end up with an empire. I think what really hooked me the most is the feeling of personal growth that I have day after day as my skills improved.

Machine Learning sits at the top of the food chain in software engineering

A lot of people reading this probably already know the pyramid was brilliantly illustrated by Monica Rogati

Image by Monica Rogati

There has been numerous variants of this pyramid, but to me the most important message is the fact that AI/Machine learning sits at the top. So if you want to do machine learning, you need solid foundations. You need robust data pipelines that collect, transform and load the data. You need to clean it and store it in well organized and documented locations. Then you need tools to query this (big) data efficiently. You also need a solid infrastructure to run your algorithms. You also need pipelines for frequent repeated training and inference. You need orchestrators. You need to keep track of your past experiments and the datasets used at the time. I will stop there because the objective is not to draw up a complete list but to show that there is huge amount of technical prerequisites.

In the same vein, an important source of information is Google’s paper on the hidden technical debt of machine learning systems. It basically shows that ML models are a tiny part of the overall system and that more attention and love should be given to other non ML parts.

Figure 1 of google’s paper showing that only a tiny fraction of the system is ml code

So there you have it, the models are a tiny part of the system. But wait? What is the specialization of data scientists/ml engineers? What is it that ml practitioners crave for when they look for a job? Well, working on that tiny piece of the system. It is the most important thing we care about!

We all want to practice machine learning

If you think about it, it makes perfect sense and for numerous reasons.

First, machine learning and data science are fascinating fields. Mostly because they sit at the crossroad of computer science, mathematics and business understanding. This means that there is way more room for personal growth. When switching from software engineering to machine learning engineering my backyard suddenly tripled in size. I could continue to craft beautiful and complex systems with my hands (and become the great engineer I wanted to be) and at the same time satisfy (way more) my intellectual curiosity. Now every day, I get to learn and sit on the shoulder of giants!

Apart from the interest into the field another main reason is a bit more practical. I have spend so much time and energy into learning the necessary topics (think probability, statistics, calculus, linear algebra, distributed computing, machine learning, deep learning…) that I want this knowledge to stick in. And we are all humans. Even if your are a genius, if you don’t practice what you learn, the knowledge goes away. So when your boss ask you (for the tenth time in a row) to create a piece of software or an analysis that has nothing to do with machine learning, what is that you think? Are you happy?

Another important factor is that the field is moving at lightning speed. It was already the case when I was in software engineering but now it is not even comparable. Not a day goes by without hearing from the latest breakthrough, the newest shiny deep learning architecture, this great new book that every ML practitioner should read, etc… When you are not practicing ML in your day job, you are left with practicing it during your free time. It is ok for a little while but it is not sustainable in the long run. We are all humans. We need time off to relax and be with our loved ones. Don’t get me wrong, I love learning new things. It can be by completing a MOOC, reading a research paper, a blog post or a book. But I just can’t spend all of my free time doing so.

Then their are all the other reasons like you can make good money, you can make a big impact in your company, AI is the future, yadi yada…

Hacking your way into machine learning

So you manage to succeed that last interview. You are now officially a data scientist. You are presented with a new project with a concrete business goal. Awesome ! So what is it that you do ? Well you find a website were you can extract the data that is needed. You quickly hack a python script on the side to scrap it all and dump it into a dataframe. You open a Jupyter notebook, start massaging and plotting the data to understand it. You run a few experiments and produce a nice model using your favorite technique and framework. Finally you dump your model into a pickle file and put it in production. In the end, this is what you get:

A hacked model in production — Image by author

Congratulation, you are now the father of a nice heavyweight complex machine built on stilts. Now pray that the wind won’t blow too hard, especially when you’ll want to add this last feature the product guys asked for. This is what I call hacking your way into machine learning. You wanted to build that shiny new model so hard that you just forgot or pretend to ignore the prerequisites!

As I said previously ML sits at the top of the pyramid. Talking about a pyramid means that the foundations, in addition to be solid, are pretty large ! Generally there is way more efforts to put in the bottom layers comparing to the cherry latest SOTA model that you put on the top.

All companies want to claim they are doing AI

I also want to take the time to explain this phenomenon that will better illustrate the point of this article. I know this for a fact because I have experienced it myself.

When I started my journey into data science I landed a data scientist internship in a consulting firm. I was quickly tasked with some basic stuff like cleaning the data and producing simple charts in R. Given what the team was doing I quickly understood that I would have a hard time applying all the fancy techniquesI learned during my master. So after I complained about it, I was given the task to understand and describe the relationship between client consumptions and the prices they paid. Basically I was to model the price elasticity of demand. Don’t get me wrong I enjoyed every last moment of it. I was now able to apply my statistical lessons in a real world business problem. So overall it was a great task for a beginner in data science like me. But I went way too far for this task. I basically tried every model I could think of (even deep learning) just to practice. Overall do you think that this task justified that they needed a data scientist intern? Does that even exist? What was really needed here is someone who can perform a linear regression. That’s it. Pretty much any analyst can do that…

After that, I landed a job as a data scientist/ml engineer in a startup. For this one experience I was extremely lucky. Even if the company has limited resources, they have a strong culture of innovation and I was given a lot of freedom to experiment. So overall I managed to tackle real world and challenging machine learning tasks. But in the end, even if I have a strong culture of managing the technical debt, I was so eager to practice machine learning that the foundations of my final solution were not sufficiently robust and I had troubles latter down the road.

Finally after that experience, I have had a quick passage in another startup where once again I was struck by false promises. This company is a great place but they clearly have been hit by young data scientist who all hacked their way up into machine learning ladder. My first task was to analyze a legacy data pipeline and come up with recommendations to revamp it. This data pipeline was basically feeding all of their smart algorithms and it was in a very very bad shape to say the least. I had huge troubles just understanding what it was doing. So not only was this task a pure data engineering task it was also not very exciting.

All in all, this also makes perfect sense. Companies are in desperate need to attract talented and educated workers. Recruiting data scientist is sexy. It makes the company look good. You can claim to the world that you are leveraging the power of AI. It is also much easier to attract funds (especially for startups).

In the end, I am still grateful for this journey because it made me realize the discrepancies between what is advertised by most companies and the state of matters. I am now way more capable of predicting the odds that a company is doing and in need of real machine learning systems.

There is a huge mismatch between what most companies need and what ML practitioners want to do

Finally, to conclude on this blog post, I want to explain what I meant in the title. This is the direct corollary of what I just explained. Most of the time, even if a company has a real concrete advantage in becoming data driven, it does not need machine learning or at least not fancy complicated machine learning. If you are in charge of recruiting that next employee ask yourself:

  • Do I have the necessary resources to study my data and produce analysis to support my decisions? If not, bring in some more analysts or maybe statisticians.
  • Do I have solid technical foundations to acquire, store, refresh, modify, document, query the data that I need? If not, bring in some data engineers.
  • Do I have the technical infrastructure and tools in place to support data scientist and ml engineers? If not, bring in some software engineers (back end, front end, full stack), devops, etc…

My profound belief is that, in the end, there is not that much need of ML resources. In fact, it is pretty obvious when you look at the data pyramid of needs. The corollary of this fact is that data scientist and ML engineers are luxury employees. Not many companies can nor should afford them. This is the sad reality.

As a final note, I want to say, as everything in life, it is not all black or white but mostly gray. So yes, you can always afford yourself a data scientist even if you are a small startup because you don’t have the fund to recruit a multi-disciplinary team and you want to attract external funds. After all, data scientists are jack of all trades! They sit at the crossroad of the 3 disciplines in the Venn diagram. Sure you can. But just be aware that when doing so, you have a high chance of ending up with:

  • a product with a high technical debt which will probably fail in the long run (because too hard and expensive to take care of).
  • an employee that will be disappointed pretty quickly if you are not able to bring in some new specialized players into the game.

Leave a comment

Your email address will not be published. Required fields are marked *