The obstacles people find within a data science career are far more serious than the ones they face while getting into it.
Often, there’s a huge mismatch between the expectations of the role and actual responsibilities. If you’re blessed to work in areas you aspire to, collaborating with other roles in a data science project can be a real struggle.
It can be easier to have your tooth extracted than to cope up with daily expectations of your project manager. Get everything correct, and you might discover your solution is untouched. “Why would people not understand or make use of something so obvious”, one might wonder.
All of these factors can cause an existential crisis, at the start of your career in data science. The potential to suffer from career stagnation is high for many data science professionals. How do you solve this problem?
I’ll share the 4 big hairy challenges in data science projects that cause many of your personal struggles. Based on learnings from our work at Gramener, we’ll discuss what they mean to you. And, how you can smash them to become indispensable in your project, and in the data science industry.
1. Sharpen your ability to handle messy data
Poor data quality is one of the top challenges in data science. Bad data costs organizations over $15 Million annually. You need clean, structured data to come up with big, useful, and surprising insights. Fancy using deep learning techniques? Then, you’ll need a lot more data and it must be neatly labeled.
In data science, 80% of the time is spent preparing data, and the other 20% on complaining about it! — Kirk Borne
You must pick up skills to discover the data that your business problem needs. Learn how to curate and transform the data for analysis. Yes, data cleaning is very much a data scientist’s job. Play with data and get your hands dirty. You’ll develop an eye to spot anomalies and patterns will start jumping out at you.
Let’s say your project’s intent is to analyze customer experience. The first task is to scout for all potential data assets like customer profiles, transactions, surveys, social activities. Any of these that don’t map to your business problem must be dropped. Inspect and clean the data and you will lose some more. Do this for weeks or months, and then you are ready for your analysis!
2. Learn the techniques and don’t worry about the tools
The data science industry is crowded with hundreds of tools. No one tool covers the entire workflow. Every week, brilliant new tools get created. And a dozen go out of business or get bought out. Companies spend millions on enterprise licenses, only to find that they aren’t as compelling anymore.
This fragmented ecosystem poses a big challenge for aspirants. A top question I often get asked is, “Should I learn Python or R? PowerBI or D3?” I always say that the tool really does not matter. Learn the technique like the back of your hand. You can always transfer your learning from one tool to another in weeks.
The tool really does not matter. It is the person’s skill with a tool that counts.
For example, to master visualization, don’t start with the tools. Learn the principles of information design, the basics of visual design, and color theory. Then get some real data and internalize the techniques by solving problems. Any visualization tool you can get your hands on will do. Don’t over-optimize.
3. Master the application of techniques to solve real-world problems
Over 80% of data science projects fail. Wonder why? There are challenges throughout the lifecycle: from picking the wrong business problem to framing an incorrect solution approach. From choosing wrong techniques to a failure in translating them to users. Every role in data science contributes to these misses. No, most of these gaps aren’t technical.
Most data science projects don’t deliver business ROI because they solve the wrong problems.
What’s the common thread here? It is the poor application of skills to business problems. For example, when data scientists just want to build great models but miss paying attention to their user’s needs, it hurts projects. Don’t stop with an intuition of a technique, or the math behind it. Find where it’s relevant and what it takes to apply it. Get invested in solving user’s problems.
Let’s say you’ve mastered a dozen forecasting techniques. Which one would you pick when your user needs tomorrow’s price to make her trade, but has just 1 past data point? Does it change if you have 100 or 10,000 points? What if she just needs to know whether to ‘hold’ or ‘sell at the market price’?
4. Go beyond data and analytics skills to succeed in data analytics!
Companies often just hire for machine learning skills. They invest in data engineering and may get some training organized in visualization and data literacy. But, this team is imbalanced and will deliver just sub-optimal results. Every data science team must have 5 skills for effective project outcomes.
If you’re playing one of these roles, should youcare about this? Absolutely. Here’s how you can increase your influence in any project. Master one skill as your core area. This is your primary role. Invest and learn a secondary skill. You should be able to step in as a backup and support on this one.
What about the other three? Pick up a broad familiarity. You must be able to relate to them, understand pain areas, and connect them back to your work. Do this and you’ll be worth your weight in gold!
You need a lot more than data and analytics to succeed in the data analytics industry
A key takeaway here is that there are 5 roles in a data science career. Not just ‘data scientist’. Let’s say you’re an ML engineer. Your secondary skill can be information design. Learn about charts and how to choose the right one. Find what users look for in visuals and what it means for the UI you’re building.