How to Measure Developers’ Productivity

Introduction

In this blog post, I’m going to run through some topics that fall under the umbrella of measuring software developer productivity.  Whether you’re a CTO, technical lead or senior developer tasked with looking after a team, we’re sure you’ll find some valuable nuggets of information.  At a high level, here are the main points that we’ll cover:

After you’ve read this post you’ll have some ideas about what’s possible, what isn’t, tools that can help you and what’s almost impossible to measure!

A definition of “Productivity”

Before exploring the meatier topics, it’s important to get a clear definition of the word productivity within the context of software development.  How can you measure something if you don’t define it?  At the most fundamental level, productivity is to do with products.  It could then be said that a productive individual is a person who regularly ships products.

If you’re from a technical background you will (quite rightly) think that both these statements alone simply aren’t enough to gauge or define productivity and may be thinking of some of the following points:

  • What about the number of defects?!
  • What about the complexity of the class library?!
  • What about the number of integration points?!
  • What about the dependencies on external systems or other teams?!
  • I just inherited a brownfield app, I need time to get up to speed!

image001

Software development isn’t a simple production line whereby code drops of faultlessly from a developer’s laptop and is copied and pasted onto a customer’s environment.  It, therefore, requires you to abandon this traditional industrial-age thinking.

Teams that consist of software developers often involve individuals from other disciplines such as business analysis or front-end graphic designers.

Due to this multi-disciplinary skill set, diverse teams must be led with modern practices, they must be open to change and adapt to real-time situations.

Because of this dynamic, it can be difficult to universally define one single metric or KPI that signifies productivity.  What we can say however is that the customer, whether they be internal or external, will define what success is.

Success in a software project general revolves around the delivery of a product that ultimately gets signed off but a customer or user.

It can be said, therefore, that a productive team can is one that can:

  • identify progress and understand where they are at any given time in a project
  • delivers functionality in line with an expected result for a user of customer

With these points in mind, we can now start to explore what this means for developer productivity, whether it can be measured and what can affect this.

 

Which developer activities you can measure?

Whether you’re implementing a waterfall model or agile model, the software development lifecycle consists of many phases. Each phase consists of tasks or activities and it’s important to be aware of these prior to capturing any data associated with developer productivity.
From a software developer productivity perspective, you can separate these activities into two components:

  • Value-add
  • Non-value-add

image003
Value-add activities

These can be defined as activities that constitute the build process of the required functionality.  Successful completion of these activities results in “sign off” by the customer.  The speed in which these types of task are completed form one part of the “productivity equation”.

Examples of these types of task include, but are not limited to, are:

  • software design
  • writing code
  • code integration
  • testing functionality
  • daily stand up to discuss impediments

Non-value-add activities

These activities can be defined as tasks that get in the way of building the required functionality.  They still must be undertaken, but aren’t truly part of the building the required functionality.  These types of task form the second part of the “productivity equation”.

Download Our Project Specification Template

Some examples of these types of task include:

  • testing and debugging code to understand how it works (when a developer encounters new code in a system)
  • configuring and setting up an environment
  • implementing tools or processes that are only understood by a select few
  • writing unit tests that aren’t representative of functional areas
  • ranting at the water cooler about some legacy code

It’s clear that non-value-add activities are obstructions to the developer and affect his or her productivity.

Activities which fall into the value-add camp, are the ones which should be measured – but only after the total time take on none-value-add activities has been deducted!  Taking this a step further, it could be said that:

productivity = (total time spent on work  total time spent on non-value-add activities)

Based on this reasoning, it can be said that the goal is to reduce the number of non-value-add activities in the software developer’s day.

Fewer non-value-add activities result in more frequent product shipping, this results in increased revenue and therefore profit for the company and profit is a strong indicator of productivity.

Challenges when measuring developer productivity

We don’t live in an ideal world however and software projects frequently encounter challenges which can ultimately affect developer productivity. Let’s explore some of these challenges before moving onto discussing tools and processes for measuring developer productivity.
image005Ambiguous documentation

Possibly every developer’s nightmare.  The developer receives a substandard specification which results in far too many questions to let them commence the design and build phase. Depending on the level of experience the developer has, they may take the initiative and wear their business analyst hat and work to refine the requirements.

If this happens, the overall build phase of the project could appear to take longer than anticipated and therefore skew the total time taken for a given task the developer is working on.

Environmental configuration

In projects where the developer is tasked with building interfaces that exchange information between multiple environments, access must often be granted, firewalls must be configured and credentials supplied.

These setup activities affect the developer’s productivity as they’re unable to hit the ground running and must wait until environments have been configured and so on.  Again, this has an impact on overall developer productivity.

image005

International development teams

It’s not uncommon for development teams to be spread across multiple continents, the internet and online collaboration tools have made it easy to have a business model that offshores or near shores software development activities.

There are certainly benefits to implementing this business model but if you want to measure developer productivity, it’s going to be much harder when you have development teams spread across the globe that operates in a different time zone not to mention working on discrete user stories with varying complexity.

Complexity

Writing code that displays “Hello World” is clearly much simpler than writing middleware that performs sentiment analysis and runs natural language processing algorithms against incoming streams of text.  You can’t apply the same measuring stick to either developer in this sort of scenario.

Integration

A developer may have implemented the required functionality as per a customer’s requirement, it may work in their development environment and have been tested successfully but now must be integrated with another developers’ class library or functionality.

On an enterprise implementation, this can be painful.  Especially if the developer is in another time zone, cue conference calls, emails, and instant messaging sessions to clear any ambiguity and/or advice of the parameters that each developer expects to be passed to the functionality they have built.

Can you really measure developer productivity?

So far, we’ve covered the following:

  • a definition of productivity
  • types of activity to monitor
  • challenges you’ll encounter when trying to measure productivity

image007

It should be apparent that highlighting some of the challenges, that you can’t simply rely on one metric to identify developer productivity.

Developers having strengths and weaknesses, some may work on more complex areas of a software application than others, some may have better “soft skills”, others may “wear more than one hat”.  It’s not uncommon for more senior developers to act as business analysts or perform consultancy type activities on a client site.

In the world of sports, there isn’t one single measurement to determine how good a football or baseball hitter is.  Pace, strength, batting average, home runs, and other factors are all considered.

 

Considering all of this, what kinds of metric can and should you capture when trying to measure developer productivity?

 

You need to forget about the idea of having one single measurement that will allow you to accurately gauge the productivity for each of your developers, there are just too many variables to consider.

Development managers and team leaders often know who their “A Players” are and who to entrust with the high-risk components of a software project.

That said, there are certain metrics that can act as a rough gauge so let’s explore them.

What kinds of metric can or should you capture?

image009

Product Features

In most software projects, developers are building features requested by clients.  One metric to record is the number of features each developer successfully completes.

Understandably, some features are simpler to implement than others – and I’ve covered some of the challenges you might expect but it still stands that tracking how many features a developer ship is an important metric to record.

Read How We Helped a Marketing Company to Build a Back-Office Custom Ads Dashboard

Velocity

If your software project adopts Agile / Scrum practices, another measurement you can track is the velocity.  This measurement will tell you the amount of work your team can complete during a single sprint.

Providing each developer is supplying estimates for Stories and Tasks in the form of Points for the Product Backlog, the velocity can be calculated. When your team has completed several sprints, you can also use the velocity to forecast how much of your product backlog can be finished in future sprints.

Bugs

A developer may ship quickly but it’s not good if the code is bug ridden.  There are two sides to measuring bugs in software application:

1)      The Severity of the bug (Low, Medium, High)

2)      The number of bugs introduced in the application

You can set a threshold in terms of the number of “acceptable” bugs in your application and measuring the number of bugs that ship with each iteration is one metric you can record per team or developer.

Naturally, you’d prefer that your product ships – defect free, but in the real world, this often isn’t the case.  The preference is to have little to no bugs that have been tagged with a Severity of Medium or High.

By setting a threshold you can run reports in your defect tracking software and in conjunction with your source control software, can identify who introduced the bug.

Code Analysis and Quality

Code quality is a trickier metric to measure.  Coding standard and practices should be in place prior to attempting to measure the quality of code.  These standards should be enforced by the team and adhered to by developers.

Visual Studio ships with Code Analysis Tools out of the box.  With your team, you can decide on a common set of standards and practices and create them as Rules.  As a development manager or senior developer, you can then run these rules over your applications source code and record areas of the code that don’t adhere to your team’s coding standards.

Performance

Most software applications being built today live on the internet.  Applications on the internet must be built with performance in mind and as such, performance is an important metric to measure.

With the advent of rich client side web applications, the traditional “page load” metric (how long a page takes to load) whilst still important, isn’t as relevant as it once used to be.

A better approach is to measure how long key specific use cases take to execute on the front end for users.

From an API perspective, Visual Studio contains a Diagnostics Tools Window that allows each developer to identify how long a class library or method is taking to execute.  These timings can be recorded and measured.

You can then measure how timings stack up against service level agreements (SLAs) that form part of the contract between you and your customer. 

Tools that can help measure developer productivity

Now that you have an idea of key metrics that you can track, what follows are some products and tools that can help you measure them.

App Dynamics

AppDynamics is a product which can be connected to your application that gives you real-time performance monitoring.  With AppDynamics, you can monitor every click, swipe, and tap in your application.  It allows you to understand your customers journey through your digital ecosystem which is all viewable through rich visual flow maps.

image011

Code Climate

Every check-in to your source code repository has the potential to reduce or improve the quality of your source code.  Code Climate helps protect you against this, the tool allows you to measure test coverage, quality, and maintainability of your products source code.

image013

It will check for code duplication and code complexity thereby allowing you to identify problem areas within the source of your product code. 

Visual Studio Team Services

Visual Studio Team Services (VSTS) is an integrated tool set which speeds up the development and delivery of your software projects.  VSTS allows you to track everything you need to, all in place.

It’s quick and easy to setup an account and if your team is small enough, you can start for free by signing up here.

Once logged in, you can add accounts and team administrators who can then begin to add your applications source code which gets stored safely in the Microsoft cloud.  Alternatively, if your data must stay within your own network, you can opt for an on-premise TFS implementation.

VSTS features Agile tools and frameworks that allow you to track, maintain and measure:

  • Backlogs
  • Bugs
  • Sprints
  • Productivity
  • User Stories
  • Velocity

image015
With rich visualizations in the form of Kanban and configurable Dashboards, VSTS is hard to beat and integrates easily with most source control repositories and development IDEs of your choice. 
Do the tools provide a fair and accurate picture of developer productivity?
Whilst the tools aren’t a “silver bullet” by any means, they at least give you a starting point in terms of capturing and managing data related to developer productivity.

You need to be mindful of the various constraints and scenarios that developers often find themselves in, however, they often work in teams which have dependencies on business analysts, project managers, clients and third-party organizations.
Developers can fall behind on the delivery of product features or defect fixes due to poor functional specifications, missing or unknown requirements.  These factors should be considered when generating metrics connected to developer productivity.

Measuring developer productivity

Taking everything that’s been discussed into account we’ve:

  • a definition of “productivity” from a developer’s perspective.
  • identified which types of activity are important (value add-activities)
  • key metrics that can be measured
  • discussed tooling that’s available which can help measure productivity

image017

Measuring developer productivity goes beyond assigning acceptable thresholds for the number of bugs found or features delivered in each sprint.  Of course, these are important metrics to measure but they don’t provide the full picture.

Certain developer attributes aren’t measurable and can’t be assigned a number.  Consider the following:

  • is the developer committed to their craft, are they passionate?
  • are they committed to delivering software on time?
  • is the developer dedicated to the company’s vision?
  • Is the developer constantly learning or improving their skills?
  • does the developer take pride in their code?

I mention these as they are not just important but completely subjective and simply can’t be tracked using products or tools.

Regardless of this, the first step in determining performance or productivity is to have a few methods of measurement.

What follows are some points to think about and approaches you can apply to help you gauge developer productivity. Whilst they shouldn’t be used as a golden set of rules, at the very least, they will allow you to initiate discussions regarding productivity!

image019
Which stage is your company at?

  1. If you’re an early stage startup with little to no runway, you’ll probably want to track speed and how quickly your developers turn around features.
  2. If you ship mission critical software, code quality and test coverage will most likely be the key metric.

Product Features

  1. Is the developer providing initial estimates and hitting release dates when building functionality in the backlog?

Sprint Burndown

  1. How many features has the developer implemented in the backlog?
  2. How often did the developer overrun in terms of their original estimate?

Bugs

  1. How many bugs are found during development?
  2. How many bugs are found after the product has shipped?
  3. How many bugs have been found during a round of system testing?
  4. How many bugs has the developer resolved in a sprint?
  5. How quick is the developer turning around bugs?

Code Performance / Quality

  1. How many times did your code analysis software show a developer didn’t adhere to coding standards?
  2. How often does a developer’s code execution time fall out with the contractual SLA?

Capturing these metrics will help you identify problematic areas of a software project and with enough historical data, you’ll be able to make more accurate forecasts for future sprints. It’s important to look beyond the numbers however for the reasons we’ve discussed in earlier sections of this blog post.

Summary

In this post, we’ve looked at developer productivity and discussed if it’s possible to measure this accurately.

We’ve seen there are far too many variables to consider when attempting to arrive at this and that each measurement will be dependent entirely on which stage your business or organization is it.

We’ve differentiated between value-add activities and non-value-activities and identified key metrics that you can start to record in an attempt measure developer productivity. We’ve also explored some tools and frameworks that you can employ to make this job easier to manage.

Thanks for reading this blog and if you’ve enjoyed it, please feel free to share it with your colleagues, friends or anyone else that you think might be interested in. You can also subscribe to our blog to get the latest updates.

Download Our Project Specification Template

Jamie Maguire

Jamie Maguire

Software Architect | Consultant | Developer | Tech Author
Jamie Maguire

Latest posts by Jamie Maguire (see all)