Data scientists and ML engineers are some of the most sought after employees in the industry. Since these are young fields, expectations can get misaligned, leading to high turnover rates. In this article, we’ll discuss top reasons why data scientists are leaving companies, and what can be done to keep them aboard. We’ll also share advice for how to develop your team members.
Why Data Scientists Leave Companies
There are three main cultural reasons for leaving: work is not challenging enough, data models are rarely brought into production, and employees get demoralized by office politics.
First, data scientists may become disillusioned when the work they’re doing is misaligned with the original job description. When companies get job descriptions wrong, it’s not necessarily because they want to capitalize on the popularity of buzzwords, but because there’s a distinct lack of understanding of data science in HR departments.
This can be resolved by educating HR employees about specific roles so they can set the right expectations in job descriptions. Data science is a broad field, from hypothesis generation and causal inference all the way to deploying and monitoring. Many people associate the higher level work with data science; namely training, building and deploying a model. But a lot of effort precedes and supports this type of work, what is commonly associated with data analysis.
Second, data scientists will get frustrated when they spend all day working on models that never get deployed in the real world. While it is normal that a great portion of models are discarded because of noncompliance or bad performance, your company still needs a good system for achieving a good performance so the models can be deployed.
Software engineering has been maturing for decades, with many tools and methodologies in place to ship products and services. But data science and machine learning are new fields, and companies are still experimenting with project management frameworks. We’ve compiled a few popular methodologies in this article to help your team bring models to market.
Third, company politics is an often cited reason for leaving. Many data scientists come from an academic background, where job competition and gossip play less dominant roles. They might get frustrated with an office culture that rewards interpersonal games more than analytical and computational efforts. Managers can resolve this by creating a culture that rewards efforts, not outcomes. They should ask employees directly about their expectations for job growth, and to make sure their job interests are aligned with the company’s goals.
What Motivates Data Scientists To Stay?
We’ve discussed some reasons for leaving and what can be done to prevent this, but what else can leadership do to retain talent?
Compensation is one of the strongest ways to retain talent. Be aware that data scientists are in high demand, which drives up the salaries. This might make it difficult for startups to compete with large companies who can offer higher salaries and more benefits. But startups have other unique advantages over big corporations that allows them to recruit top talent.
Some engineers have been known to accept a lower salary to switch from a bureaucratic government agency to a private startup that’s disrupting their industry. They would happily take a pay cut to contribute to a meaningful product. Startups also bring the allure for early equity compensation that could explode in value in the future. Many millionaires have been created in Silicon Valley with this process.
"Data science talent is hard to find. Sometimes, to retain the best people, you have to let them work from wherever works best for them and their families." - Pushpraj Shukla, From our event Voices of ML Leaders
Compensation momentum might be even more critical to retain talent. Employees want to see salary increases that correspond with market trajectories and company seniority. This can be quantified by analyzing market offerings and competitors’ salaries. Leadership should meet 1-to-1 to plan out career paths with their employees, which will also give them a better idea of their expected compensation trajectory.
Job titles are extremely important to retain talent. They can be used to reward seniority and to highlight the scope of responsibilities for a role. Setting the right job titles requires educating HR staff on different roles. Data analyst, data engineer, data architect and data scientist may sound similar, but each has distinct duties that come with different levels of experience and seniority.
"The key is to rebuild the job description together and spend more time telling the team's story. We created and modified different individual career paths and opportunities instead of just one data science career path for everyone." - Justin Norman, From our event Voices of ML Leaders
One last thing hiring managers need to be aware of is the difference between the academic and engineering mindsets. Data science teams might consist of software engineers who are used to shipping products, and academic researchers who are comfortable with continuous fine-tuning. They might get frustrated with each others’ approach, but you still need both on a team. The best solution is to place each “mindset” in the right role. Let academia do the proper research process, and have engineers in roles that focus on testing, deploying and monitoring models.
Training your ML Team
Training employees internally motivates people to stay on board and tends to create high returns on investment. Training is also necessary for fields like data science and machine learning, where new tools and frameworks are developed rapidly, and new research papers are posted on a weekly basis. In this part, we provide best practices for training ML talent.
Training employees may seem costly at first, because it costs time and resources that are not directly contributing to the bottom line. But when done right, training can offer great returns on investment. Teaching new skills and competencies allows people to work more efficiently, or create something they couldn’t do before. High learning curves also motivates employees to stay onboard, as opposed to simple work that quickly becomes repetitive. In fact, if your work can easily be taught to someone else, you will face job insecurity sooner or later.
Training is not just reserved for large corporations with big budgets. In fact, it’s paramount for startups whose core product is centered around AI to build and grow a team from the beginning.
How to Train Your ML Team?
Companies should match career paths and job titles into the data science lifecycle. This puts your employees on a learning trajectory that corresponds to job growth, and allows you to determine who should learn what. For example, a product data scientists should be skilled at understanding data, learning relations between them and making decisions based on them. Make sure the entire data science cycle is covered by some role, and fold competencies into that role.
Start building a portfolio of ML projects to train your ML team as a whole. Machine learning is driven by trial-and-error, and most models will fail to reach production. What matters is that you can take the lessons from failure and apply them to the next project. This knowledge can be captured and annotated, and remain accessible for future generations of employees.
One successful way to introduce new tools and techniques is to organize hackathons! This is a great way to let your team experiment freely without deadlines or constraints. Hackathons have even led to the creation of successful startups and products with millions of users. While they originated in software development, they can be perfectly applied to machine learning as well. Hackathons can be centered around a new ML models or algorithms, and can offer incentives for teams with the best performance.
Hiring Managers Externally or Promoting Internally?
A common question from data leaders is whether they should focus on promoting data scientists internally to manager roles or if they should hire external help. A good rule of thumb is this: focus on promoting internally whenever you can, but consider hiring externally if you need to bring on new domain expertise, or if you want to introduce fresh ideas into the team.
An effective data scientist doesn’t always make an effective leader. But an effective data leader will need extensive knowledge of the data science process. The best data science leaders posess a good balance of the soft skills and hard skills. One tip from a data science leader is to keep spending some personal time working with data science tools and problems. This is a great way to preserve the “data science mindset” and to stay aware of new tools and libraries.
While leaders are essential, successful companies enable all employees to be “mini-leaders”. They empower them to take initiative on ideas and provide the right resources. In this sense, leadership becomes more of a support role. This can be particularly effective in data science, since it’s an exploratory process and employees working most intimately with the data sets will have a better intuition of which models might be successful.
We've covered:
- Data scientists are in high demand, which drives up salaries.
- Startups have other unique advantages over big corporations that allow them to recruit top talent.
- The key is to rebuild the job description together and spend more time telling the team's story.
- Training employees internally motivates people to stay on board and creates high returns on investment.
How can Vectice help?
By building a portfolio of ML projects that can capture the learnings of experimentation. Vectice offers the first solution that embraces the iterative nature of data science workflows and centralizes AI assets into a searchable catalog. This means you can access previous projects, take the lessons learned from different use cases, and apply them to future projects. In Vectice, knowledge is captured and annotated and can be accessed by current and prospective employees.