Are you sure you hired the right developer?
By Bill Harding at 10:31am on January 17, 2019
Here’s how you’ll know.
Hiring the right developer is a time-consuming process that can drain productive hours from your week. Recruitment takes time. Mistakes can be costly. And your competition is out there, wooing the very same software engineers. After agonizing over your decision and bringing a new developer on board, how do you know if you've made the right choice?
Measuring developer performance – old school style
For those of us without a crystal ball or the ability to travel into the future, we’ve had to rely on a hodgepodge of questions to quantify the performance of a new employee.
Will the new hire fit into the corporate culture? Will they live up to the promises implied in their resume? Will they be the kind of person that accepts and learns from their mistakes? Are they a leader or a follower? Will they inspire others? And above all: will they produce?
Traditionally, measuring an employee’s performance relies heavily on subjective information. Appraisals are based on what the manager can easily remember, which are usually the most recent – but not necessarily the most relevant – events. As a result, appraisals end up relying on unsubstantiated opinions and incomplete information. Recency bias – the tendency to assign more significance to a recent event – can also skew appraisals.
Any measurement of engineer productivity that doesn’t include quantitative data is underqualified for the task at hand. If you need to know how long your pinkie is, you don’t ask questions and speculate – you get a ruler. Cohort analysis is one of the best ways to add a quantitative component to your existing developer evaluation process.
Cohort analysis – a lens into the future, from the past
A cohort is a group of people who share common characteristics within a defined period of time. It’s a great method to compare an employee to others who are at the same stage in their career, or in a similar role. Comparing individuals to a group of their peers makes it easy to spot differences in performance or work style. It’s a good way to identify the highest (or lowest) performing developers and gain insight into the factors that influence their productivity.
But traditional cohort analysis can be a time-consuming and flawed process. First, you need to define the cohort you’ll be measuring – not an easy task if you manage multiple teams working on a disparate collection of coding assignments. What type of projects should receive the most emphasis? What data points will you use? Most often, engineer cohort analysis is based on data gleaned from ticketing systems like JIRA. But counting closed tickets is an overly simplistic way of evaluating engineer productivity. Once a manager makes all these decisions, the data still has to be imported into Excel, normalized and formatted before the first report can be created. No wonder so many managers loathe the task. It’s a perfect recipe for procrastination, followed by settling for recency bias.
My company, Static Object, uses cohort analysis to give development managers a powerful new code metrics tool built for reliability. Based on our experience, we’ve listed a few areas where code metrics can steer a new developer toward their path of maximum throughput and satisfaction:
Does your new hire excel in Python, but struggle with Ruby? What types of tasks do they excel in? Are they willing to learn others?
Is your new hire struggling to commit code? How little should you expect in weeks 1-4? Can better onboarding speed that up? How does that measure change by week ten?
Does the new hire possess the self-discipline needed to work remotely? How can you challenge them to work more efficiently, without sacrificing quality? How can you identify and award those that consistently improve their output?
Reliably identify top performers
Quantifying output is a tricky proposition. Static Object determines developer output using Line Impact. This metric combines over 10 overlapping measures of engineering output, to form a composite measurement that managers can trust. The most flattering compliment we receive is “wow, this output chart matches up with what I already thought!”
Charting a developer’s contribution over time
Here are a few ways that Static Object uses cohort analysis and other visualizations to create a more accurate measure of engineer productivity.
Cohort Comparison Chart
The best chart we have to understand how a new developer compares to their teammates at the same point in their career is the Cohort Comparison chart. Compare the Line Impact over a configurable time range. Check up on how the first 2, 4, or 6 months of the Developer’s career compare to that of their colleagues at the same point of tenure. This chart can be filtered to only show “Junior Developers,” “Devops Team,” “Software Architects,” or whatever other groups your organization identifies.
Category Comparison Table
This table paints a broad-strokes picture of your employee’s strengths. Quickly grasp how a developer’s Line Impact varies based on where they’re working within your tech stack (e.g., front-end, engine, documentation). Understanding where a developer excels can help identify tasks that will maximize their productivity and job satisfaction.
Developer Comparison by Week.
Similar to the Cohort Chart, but not anchored to employee tenure. On this chart, you’ll learn how developers of all experience levels compare to one another over your selected time range. For developers past their second year of experience, it’s often most informative to use this chart, since it normalizes shared temporal conditions that may impact productivity (like the holiday season).
How are your new hires doing? How do you know?
Re-inventing cohort analysis is just one of the ways Static Object creates a clear picture of developer productivity. As a development manager, you’ll have the confidence in knowing that you are evaluating your team fairly. An evenly-applied measurement like Line Impact can be less subject to bias than human evaluators. And that, in turn, will give your developers the assurance that their hard work is rewarded – leading to even more productivity, and more great hires.