Menu
decrease in <code-text>Review Wait Time<code-text> in 1 week
reduction in <code-text>Change Lead Time<code-text> within 2 weeks
Will O’Neil had only been in the Engineering Manager role at Vital for a couple of months when he started noticing recent bottlenecks that were slowing his team down. He wanted to figure out the cause of these disruptions so that they could get the work moving more smoothly.
Vital was experiencing a period of rapid growth, which meant lots of new teams and new team members. Will wanted to quickly get a sense of where the bottlenecks were and what was causing them, because the longer it took to resolve them, the higher the cost on team productivity.
“Multitudes data gives me insight into what causes spikes in the time to finish PRs and allows me to ask: What might have led to that? If something's trending up in one team and not on another, is there something that's blocking those teams?”
— Will O’Neil, Engineering Manager, Vital
In the Multitudes app Will saw that the <code-text>Time to Merge<code-text> had increased to 2-3 days. This would normally be under one day. <code-text>Time to Merge<code-text> is closely related to the <code-text>Change Lead Time<code-text> metric (which is now available in Multitudes), one of the top four indicators of team and financial performance for software development teams, based on multi-year research by DevOps Research and Assessment (DORA).
When Will looked at his teams’ data in Multitudes, he noticed that some of them had a larger increase in <code-text>Time to Merge<code-text> than others. That showed him something was different across teams, so he started to dive deeper into that.
As he looked further, Will noticed that the teams with the increase in <code-text>Time to Merge<code-text> also had much larger <code-text>PR sizes<code-text>. Studies from Google have found that as the size of PR increases, the number of useful comments decreases. What’s more, the review latency increases, so the smaller PRs are correlated with better quality and faster delivery.
Will knew that the next step was to get more context from the team and to agree on next steps together. When he shared the insights about the higher <code-text>PR Size<code-text> and <code-text>Review Wait Time<code-text>, the team discussion revealed that people had different expectations around how to write and review PRs. This was because of the increase in hiring and the diverse backgrounds of people on the team. To get everyone on the same page, the team agreed on two key changes:
(1) In their PR norms, they set a goal to write shorter PRs.
(2) In their code review documentation, they set expectations around how to have notifications configured, how quickly responses were expected and what kind of feedback should be given in order to speed up <code-text>Review Wait Time<code-text>.
“We were reviewing the apps we pay for and asking, “Are we getting value out of this?” to decide whether to keep using it. I said that we were definitely getting value out of Multitudes.”
— Will O’Neil, Engineering Manager, Vital
In the week following the team discussion, the engineering teams started to respond more quickly to review requests – <code-text>Review Wait Time<code-text> reduced by 41% during this period.
Following that, <code-text>Time to Merge<code-text> also dropped significantly across all of Vital’s teams – ranging from 74% all the way to a 97% decrease! This improvement held steady for several weeks following.
As Will said, “Multitudes data gives me insight into what causes spikes in the time to merge PRs and allows me to ask: What might have led to that? If something's trending up in one team and not on another, is there something that's going on there that is causing the team an issue?”
To continue improving how his team works, next time, Will could investigate if the <code-text>Time to Merge<code-text> varies depending on the <code-text>Types of work<code-text> the team was doing at the time. This analysis shows how much effort the team is spending on bug and tech debt work relative to feature work, so that teams can understand how different types of work impact their flow. This will allow them to make trade-offs about where it is best to spend their time.
