Problem – Our work is invisible
In software development, one of the biggest yet often overlooked issues that we experience is the invisibility of our work. All code exists as bits and bytes on a developer’s laptop or in cloud storage. Unlike non-digital industries where the work product is tangible, what happens during the software development process remains largely unseen. For example, in manufacturing, business stakeholders can easily observe a production line, witness progress, identify problems, gauge how much work is underway and how many people are involved. The physical and visible nature of the work makes it easy for stakeholders to understand the challenge of adding new work or changing direction with work that has already started.
If you are outside engineering, not directly involved in the development of software, have a minimal understanding of software development, then engineering is a black box to you. Requests go in and work emerges at some point in the future. Stakeholders are dependent on others for insights and an understanding of status for all the critical work making its way through your delivery system. This creates a communication dependency with engineering and this is where problems start to occur.
Problem – Poor Communication
Engineering teams often do a poor job of communicating what is happening within the team. I touched on some of the reasons for this in my last article – Engineering Leaderships Biggest Challenge. This lack of communication is common and problematic. Developers tend to believe their primary job is to build code, and while this is fundamentally true, this results in a lack of visibility for those outside the development process. Helping others understand our processes and work is often seen as someone else’s job.
Business stakeholders are typically interested in the work items most critical to them. Regardless of your development process, the core question to engineering remains: “When will my feature be ready?”. This involves predicting the future, which is a task fraught with uncertainty. Because of this, developers are uncomfortable making predictions and may project an unwillingness to commit or, under pressure, tell stakeholders what they want to hear. Either way, this leads to poor outcomes, disappointment, and a breakdown in trust between engineering and business stakeholders.
It’s common for business stakeholders to have a limited understanding of the process and challenges developers face when building products. This is normal. It is engineering’s responsibility to make our work and our challenges visible and understandable to stakeholders. Let me explain why and how you can do this.
Why you should fix it
Stakeholder frustration frequently leads to poor outcomes for engineering. There can be a perception that things are taking too long and that the pace needs to increase. This frustration stems from not fully understanding what is happening in engineering, leading to requests for the team to do more, even though they are already stretched thin. Of course, trying to do more will make things worse, not better. In engineering, it is easy to be triggered by this. Engineering teams may view the business as ignorant and unreasonable, which is understandable but unhelpful. It is better to take responsibility. Creating visibility and understanding of our work is an engineering problem.
In every organisation I have worked in, there is alway more work than people available. In the black box model, the unclear status of work items is a problem, but the volume of work being unclear is a bigger issue. Without an understanding of workload, stakeholders have complete freedom to draw whatever conclusions seem appropriate and they will do this based on limited data and an incomplete understanding. By failing to create visibility and clearly articulating the current workload, engineering inadvertently accepts responsibility to deliver all of it. The truth is, assuming that there is business justification for all our current work, then engineering being overloaded is a business problem, not an engineering problem. The goal of making your work visible is to ensure that this burden is being shared appropriately.
How can you fix it – a real world example
There are simple steps you can take to create visibility around ongoing work. The key is to have a systemic view of delivery, not just a focus on the most critical items. Flow Metrics can help you to achieve this. Figure 1. shows Work In Progress (WIP) data for an organisation we were assisting earlier this year. For the 7 weeks reviewed, this organisation was averaging over 150 issues in flow per week. Given that the engineering group was 30 strong, this meant that on average, every individual in engineering was working on five issues at the same time. The excessive and unrealistic demands being placed on engineering by the business were invisible and not understood until this picture emerged.
Figure 1. System WIP and WIP by Issue Type graphs from Delivery Flow
The situation in this organisation maps perfectly to everything discussed previously. Stakeholders were frustrated and wondered why some critical work was taking so long. Engineering knew the business was unhappy. They felt that the business was very demanding. The only reporting taking place focused on the most critical work, which totalled about twenty issues, creating a false understanding of workload. This is why we should always think and talk about work systemically rather than just focusing on a subset of it. The engineering team knew that the demands on the team were excessive but even they didn’t understand the extent of the problem. They had not previously generated this data for themselves, never mind the stakeholders.
When we generated the WIP data and graphs, we brought it to the attention of the executive team. That was an interesting meeting. The team immediately recognised that they were trying to do too much. They also recognised the role they had played in creating the problem by not being more ruthless with priorities and creating focus on what was truly critical to the business. This group wanted the team to work hard but they fully understood that this needed to be done in a sustainable way. Although engineering believed otherwise, this group of leaders were inherently reasonable. A lack of visibility was the real culprit.
We also discussed how an overloaded system will result in slower delivery times and less delivery. If you are doing more things at the same time, with the same people, it will take longer to deliver the items. The average cycle time and throughput trend graphs in figure 2. confirmed this to be true. Queueing Theory, and Little’s Law in particular, predicts that this will happen. Based on this, we discussed how less WIP would equal more delivery. This was intuitive to the stakeholders and they quickly grasped it.
What shocked the executive team was the Work in Progress by issue type graph shown in Figure 1. This graph highlights the fact that approximately 50% of the items being worked upon, are items other than features (Features are the green line, other lines are Bugs, Risks, Debt etc.). This realisation was stunning to them as features dominated their thinking. They knew this other work existed but not to the extent that it did.
Figure 2. Average Cycle Time and Throughput trends from Delivery Flow
Conclusion
Apart from these graphs, I also shared a detailed list of every item in flow. They still wanted to know when the key features would be ready but by making work and workload visible, they had a better understanding of the organisations delivery system and they also understood that they had an important role to play in maintaining its health. After creating this visibility, there was sympathy for engineering and even some guilt. However, engineering was not completely off the hook. The stakeholders could not understand how engineering had allowed this to happen by failing to highlight the problem and pushing back sooner.
Creating visibility and communication are not skills that come naturally to most engineers but they are so crucial and impactful to our experiences at work. When we are a black box, we silently accept responsibility for a lot of wishful thinking that can occur in an organisation and all the pressure that comes with it. Don’t accept this pressure, share it by making your work highly visible and communicating this with the business. Flow Metrics can greatly help you achieve this.