Now that we’ve established how OKRs work, let’s apply them to machine learning projects.
One critical difference between traditional projects and ML projects is that machine learning is research-based. Outcomes are not rigidly defined, and projects might fail. Some ML managers won’t use OKRs because machine learning is iterative and speculative. They argue that ML projects are an exploratory process and cannot be planned with deadlines.
But OKRs are great tools to manage ML projects since it allows us to set ambitious goals while remaining flexible with quarterly reviews and variable KPIs. They create a system of progress and accountability under conditions of uncertainty. Without goals, ML teams are just trying things and hoping for the best. Let’s dive deeper into the right objectives and metrics for ML projects.
While often used interchangeably, we must distinguish between data science and machine learning. Data science has proven techniques that allow decision-making on insights from data sets. Machine learning is riskier with fewer proven solutions. While data science projects can have a predictable ROI, machine learning is more experimental and its ROI is harder to predict.
How can I set Objectives for ML projects?
Many industries operate non-deterministically on new projects. Famous examples include developing vaccines with clinical trials, recruiting star athletes, and venture capital deals. This also applies to machine learning: different solutions will need to be tested to discover which ones are most impactful. So how can you set goals when the outcome of success is uncertain?
Leadership should understand that machine learning isn’t a magical solution that will solve your problems automatically. Rather, ML algorithms can be used to optimize or automate current procedures. Ask yourself what breakthrough would set your company apart from competitors. What part of your business would benefit the most from automation or operational efficiencies? What should your team work on if they had 10% more time or resources?
Objectives can be defined locally (for teams) or globally (for organizations). An example of a local objective is improving a recommendation system relevance score to 90%. An example of a global improvement is a self-driving company that achieves near-perfect collision avoidance. While success is not guaranteed, defining the most impactful outcomes sets teams on the right path to accomplish them. Working on these projects requires a combination of metrics-informed and intuition-based decision-making.
What are good Key Results for AI/ML projects?
Metrics for AI/ML projects usually fall into two categories: technical and business performance. The former measures improvements in data infrastructure, model accuracy, ML operations etc. The latter measures how ML creates value for the business or customers.
Since metrics are just a point-in-time measurement, Key Results should always be defined as a quantitative improvement based on metrics. OKRs should be supported by 3-5 Key Results, preferably less than more so teams can focus on what matters most.
Business KR examples: increase NPS score by 20 points, resolve 99% of customer support queries with an automated chatbot, predict employee retention based on past performance, identify and segment target customers in 10 subgroups.
Technical KR examples: decrease data redundancy by 10%, reduce model deployment time by 3 weeks, reuse 15 features from past ML projects, increase classification accuracy to 98%.
Example #1 of technical OKR for machine learning:
Objective: Reduce false positives in fraud detection algorithm.
Key Result 1: Randomly investigate 1 of 200 flagged transactions.
Key Result 2: Review 5.000 detected false positives from last quarter.
Key Result 3: Retrain classification model from 99% to 99.9% accuracy.
Example #2 of business OKR for machine learning:
Objective: Provide best customer experience with food delivery app.
Key Result 1: Train delivery time model to be within 3 minutes of true value.
Key Result 2: Improve recommendation engine to achieve 60% clickthrough.
Key Result 3: Process 9/10 customer inquiries with an ML-powered chatbot.
In our next article, we’ll review common mistakes when setting OKRs for ML projects.