First of all… this is a really, really, really big topic. If we are lucky, we’ll get a start and maybe lay a foundation for future conversations. My goal in the next 1000 words or so is to at least introduce the foundational concepts, and frankly… help me see where I want to go with this. So… with all my pre-qualifications in place, let’s see what we can do.
Over the last few posts, we’ve talked about what it takes to do Scrum well and explored many of the anti-patterns that cause Scrum to fail. One of the biggest challenges to adopting Scrum is the ability to form complete cross-functional teams. Before we get into how to solve the problem, let’s first explore why it’s so hard to begin with.
First of all… let me share that I have NEVER worked on a small agile team. I’ve coached many of them, but my introduction to agile was in the context of large enterprise class financial systems… things like online banking and bill payment. The kinds of systems where the company makes a penny or two on every transaction and does millions every year.
Think about what it takes to build these kind of systems. Quite often you had a hundred or so people involved… maybe more. You are dealing with a very diverse technology stack. At the time, we had people building software in .NET, Java, C, and Cobol… interfacing with an Enterprise Data Warehouse, leveraging web services, PeopleSoft, and any number of other systems.
To make things more complicated, quite often the systems we were building were actually systems of systems. Not just in the Systems Engineering sense of the word, but actually products of products where the product we were building was made up of other products, each with their own customers, product management, and deadlines.
Our job was to introduce changes in these systems, not break any of the other existing functionality, make sure we delivered on time, without getting in the way of anyone else delivering on time. Needless to say, these were pretty complicated delivery efforts and required tons of analysis, architecture, and business sponsorship across the entire organisation.
All said, the core team of folks was maybe 100 people or so, but we were interfacing with an organisation that all said was at least 400 people.
Okay… so now think about the guidance that we give for the formation of a Scrum team. A cross-functional group of 6-8 people who have everything necessary to deliver a working, tested, increment of the product. How in the world would I get 6-8 people with the skills necessary, the understanding of complex workflows, and the domain knowledge to touch this many systems?
I’m not suggesting that it’s *impossible* to do this, but I’ll tell you… most products have not been built with the underlying architecture, test harnessing, or continuous integration necessary to even take a stab at shared code ownership across such a diverse stack. At some point, the lack of sub-system level ownership would very likely erode the integrity of the overall solution.
Even suggesting this would be incredibly risky proposition and likely a non-starter in most organisations.
So where does that leave us relative to forming Scrum teams? The idea of feature based teams in organisations building these kinds of systems, at this level of scale, is not practical guidance. If you can form a team like this in your environment… stop reading now and go do it. It is clearly the simplest way to go. If not, you have some work to do to figure this out.
Forming Capability Based Delivery Teams
Most organisations can be viewed as a series of capabilities. A product is effectively a capability. A feature set within a product can be a capability. Services that support multiple products can be a capability. Things like accounting, and finance, release management, and deployment can be capabilities. Performance testing and integration can be capabilities.
At the team level in any organisation, rather than getting hung up on the idea of the cross-functional feature team, the idea is to form Scrum teams around capabilities. The capability based Scrum team operates just like any other textbook Scrum team, it’s just that their output is a capability that can be consumed by other capabilities within the enterprise.
Capabilities as defined here tend to have pretty well defined inputs and outputs, contracts and service levels with the rest of the organisation. They typically rely on a less diverse technology stack and the level of domain knowledge necessary to iterate the product is less relative to the entire system you are trying to build.
The comparison we like to make here draws a parallel between refactoring a legacy architecture into a services oriented architecture. The goal is to get each of the services loosely coupled and highly cohesive. Similarly forming Scrum teams, we want each team to be loosely coupled and highly cohesive… just not necessarily responsible for an end-to-end slice of the entire product.
That said, if you think about it… we might just have a semantic argument at play. One man’s product is another man’s service. So from the point of view of the Scrum team, they are building a product… it’s just that the product is a service that can be consumed by the rest of the organisation… building blocks if you will… that the enterprise software is based upon.
Forming Products From Collections of Services
The problem with services oriented teams is that, more often than not, our customers aren’ buying services from us (although sometimes that actually does happen)… it’s our internal product organisation that is consuming the service. In this case, you may very well have a traditional Scrum team formed around a product or feature set of the product consuming the output of the services teams.
In this case, these Scrum teams operate more like a textbook Scrum team. They have a backlog, meet with the Product Owner, and produce working tested software every sprint. But get this… by definition, these teams wouldn’t have everything necessary to actually deliver the software, they are dependent on the services in order to create an end-to-end slice.
Here is an interesting insight…
The problem with this scenario, isn’t that the product team is dependent on the services team… the problem is that the product team and the services team are both delivering their backlog at different rates and potentially at different times. When an end-to-end feature requires both teams to deliver their piece… we create a planning dependency between the teams that is difficult to manage.
I’m clearly jumping into the governance conversation here a bit… but it’s these planning dependencies that are killing large scale agile adoptions. You can go the SAFe route and try to get everyone into the room to work out an integrated release plan. We’ll often do something similar but leverage a smaller group of people, running features through a Kanban system.
The idea is that we’ll manage requirements decomposition through a Kanban, but have queues to batch requirements into sprints or releases if the organisation isn’t ready for a continuous flow delivery model. Either way, the challenge remains… when you have a large, multi-team system, it’s very likely that you’ll find bottlenecks in the system. Some teams will go faster than others.
The problem with bottlenecks is that one team races ahead building services while the product is struggling to make forward progress. Quite often, it’s the products that are able to race ahead and the service creation that struggles to keep up. In both scenarios, you end up effectively with partially complete code that no one can use.
High Cohesion and Loose Coupling
To alleviate these problems between teams, we come back to the idea that teams must be highly cohesive and loosely coupled. That means that we should avoid at all costs committing to features that require more than one team to be done in the same place at the same time in order to make an integrated delivery.
Maturing agile enterprises, need to move toward a model where the product teams and the services teams operate on their own individual cadence and are able to deliver software at whatever velocity they are able to commit. In practice, this results in three rules for team-to-team interaction in a large enterprise.
1. Product teams can only commit to delivering features based on services that currently exist. This is the least risky approach.
2. Product teams can commit to delivering features based on services that are on the near-term services roadmap. This carries some risk, but is usually pretty manageable.
3. Product teams inject dependencies, by requiring a service be created while the feature is in flight. This is the highest risk scenario and should be avoided if at all possible.
Basically, Scrum tries to solve the dependency and bottleneck problems by requiring each team to have everything necessary to deliver a working tested increment, asking each of the team members to serve as specialising generalists, and having people swarm on backlog items to get done and stabilise velocity. I totally agree with all of this, it’s just not practical at scale.
We’ll explore this more when we talk about governance… but for now… let’s recap where we are.
Large organisations often can’t implement the idea of a complete cross-functional feature team. In these organisations, Scrum teams need to be organised around capabilities, products and services that must be combined to evolve delver software at enterprise scale. The more effectively we can decouple the delivery teams from each other, the less planning overhead we’ll require.
Product Owner Teams
I’ve been on record for quite a while that I believe the single product owner construct is a myth in most organisations. I get why this was created and why it should work. Without a singe voice of the business to give clear unambiguous guidance to the team, the team is going to thrash. The problem is that a single person with the necessary skills and authority to create the backlog don’t often exist in companies.
We’ve been a big fan of a construct called a Product Owner Team and implement this construct almost everywhere we coach. The idea is that to sufficiently groom a product backlog you need several points of view. We encourage the PO Team to have of course the product point of view, an architectural point of view, an analyst point of view, and a project management point of view.
I’m not talking about roles or people, I’m explicitly talking about the perspective brought by each of these disciplines, regardless of who actually does the work. When you are working in a methodology based on the idea of fixing time and cost, and varying scope to meet business goals… it requires more than just the business’s point of view to make the appropriate tradeoffs.
When we are working in a predictive-convergent organisation… remember that concept… our job is to figure out how to solve relatively fixed business problems with somewhat pre-determined solutions. By having these four perspectives working together, you can make real time trade-offs about what to build and how to build it to optimise your chances of being successful.
I kept promising to tell you guys the story about my house… that would have illuminated some of my thinking here… but that will still have to wait for another day.
Portfolio Governance Teams
Similar to the Product Owner Team structure, we find that many organisations need cross functional teams of leaders to approve the investment increments at the portfolio level of the organisation. Depending on the size of the organisation, this can be a team of directors and managers. I’ve implemented it as a team of the CIO, CTO, the VPs of Engineering and Marketing.
Most organisations have some sort of governance model in place… a phase gate model if you will. Again… we’ll talk more about this when i do a post on agile governance… but for now, just think about leaving your existing phase gate model in place, model it in a Kanban, run much smaller batches through the system, limit WIP, etc… you get the picture.
For the same reason I like cross-functional teams providing guidance on requirements decomposition, I like cross-functional groups of leaders providing guidance on how work is approved and flowing through the system. When delivery is blocked, the leadership team can see it, and help get the resources necessary to the teams to get things moving again.
That Still Isn’t Everyone
At this point we’ve talked about teams formed to help work get approved and into queue to be built, teams of people that are responsible for requirements decomposition and high-level architecture, and teams of people responsible for building products, and different teams of people responsible for delivering services. What about the rest of the organisation.
Some organisations have strategy groups, marketing, sales, integration, release management, performance testing, infrastructure, DevOps, SCM… and many, many others that I’ve failed to mention. Should we tuck these people into Scrum teams, or maybe disband these groups and let the Scrum teams do everything themselves?
That might work for smaller organisations… but again, in many of the large organisations we work with, these functions are necessary, and maybe even necessarily separate. For us, this is where the Lean part of Lean-Agile program and portfolio management comes in. It doesn’t matter so much how these teams work, if they use Scrum or Kanban or Waterfall.
What matters is that these teams agree to work with the other parts of the organisation in small batches, limiting work in process, and reducing cycle time. At scale, there are going to be upstream and downstream groups that have to work together to get the product you’re building into market. Coordinating the delivery across those teams is the current problem de jour in the scaled agile discussion. I don’t believe that Scrum is the answer in these kinds of organisations.
Summary
Okay… where does that leave us? When you are designing your agile enterprise… you’ve got to ultimately take into consideration how you are going to form teams, how the work of those teams gets coordinated, and what you are going to measure to know you’re making progress. We explored (in somewhat gory detail today) the structure side of the problem.
I tried my best to make the case that the notion of the cross-functional feature team will break down at scale. There is just too much subject matter expertise, too much domain knowledge, and too diverse a technology stack for one team of 6-8 people to build much of the large scale software that is getting built today.
If this is your reality, that means you’ll have to consider implementing Scrum teams based on products and services, or more generically, capabilities that can be reused across your organisation. Governance is largely the way that work gets coordinated across these teams, but the more you can decouple teams, and have them work at their own pace, the more agile you’ll actually be when you are done.
Product teams and services teams don’t make up the whole of the agile enterprise. We have many upstream and downstream groups that have to be able to work effectively with the Scrum delivery teams to get the product into market. IMO… this is something that Scrum doesn’t have much to say about. SAFe is making a pretty good stab at it.
More generically though… the idea is to model the rest of the organisation into a Lean/Agile value stream, run smaller batches through the organisation, limit WIP, and work to reduce cycle time and reduce time to market. At scale, business agility isn’t about how effective Scrum teams are… it’s about how effectively the entire organisation can get product into market and generate a return.
Hope this gives you guys some things to think about. IMO… this is the missing piece of the puzzle for many companies.
I usually try not to directly pimp our services when I write, but helping people solve this problem is a significant part of our practice at the moment. We use all sorts of techniques from interviews and collaborative sessions to a very structured approach called Capability Modelling that allows for structured conversation and results in a heat map of your organisation that can be used, not only for forming teams, but for having conversations about the impact of changes you might want to make to your technology platforms. It’s proving to be a pretty powerful lever for making organisational changes at scale.
Full credits and permission to reproduce this post to Mike Cottmeyer
No comments yet.