When a company grows a challenge emerges: to scale teams while allowing them to work independently from each other. Although teams can have autonomy and control over parts of the system, value is often delivered when multiple teams cooperate to accomplish a goal. Let's frame this problem from a user perspective, after all that’s the goal of any system: to deliver value to the end user.
How do we architect a system that puts the user first?
Who are the users?
That’s the very first question to ask yourself if you are a software engineer. I know it may sound strange to many, especially since we are not asked those kind of questions during interviews. Irrespective of which system we are going to work on, from a B2B platform to a B2C social media application, there is always a user.
The user has motivations and goals and we should develop a system that best satisfies them. Cooperation between different skills from user experience design, product strategy, marketing to sales helps understand those goals and motivations, but ideally teams should be talking directly to the users.
It is not that hard in the beginning. A start-up with a single application that is very focused in a few goals usually does that. After a while as the system scales we are challenged to continuously break the problem in parts, draw lines between those parts and group people to take care of them.
A good way to break the system is to group user motivations in archetypes. Motivations can be translated in actions the users can perform in the system to accomplish goals. For example a music streaming platform like Spotify:
We can refer to archetypes as hats the user can wear to accomplish certain goals. The same person can wear different hats when interacting with the same application. In the example above we could have the following archetypes: Media Player, Social User, Searcher, Browser among others.
Spotify started as a very simple application, maybe only with the Media Player and the Browser archetypes. As new ideas about motivations and goals of the user came up, new archetypes were discovered. Someone had the idea that social media was important to drive user engagement with the application, then the Social User archetype was created.
In a B2B application there is usually a relationship between archetypes and the roles and titles within the organisation the software is supposed to serve. In that case it is important to keep in mind a 1 to 1 relationship between archetypes and titles is not necessarily true, since one person can wear many hats.
When the mindset shifts towards serving the users, instead of delivering projects or features, we can do the same with the way we organize ourselves and our architecture. We can have user experience designers focused on an archetype, as well as developers. Metrics about the behaviour of the user when wearing that specific hat can be gathered to help the team experiment and measure success.
Each team must have the people necessary to satisfy the motivations of its archetypes. There are some challenges when we put together designers, developers (Mobile, Web, Backend, Infrastructure), business people and so on. Managers will say there will be waste to have one specialist for each team, and they are correct.
The way to diminish waste is to create a culture of skills, not roles. Hire developers that like pair programming and that are able to coach other developers about the technologies they are specialised in. Hire developers that like user experience and designers that like to coach developers about it. Hire developers that understand test automation and quality assurance and that can coach other developers about it. Hire developers that like infrastructure, but that also like to see it being used to serve the end user.
Let’s be honest, the industry hiring processes are broken. There is too much focus on useless hard skills (like designing algorithms on a white board) and way less focus on product development, user experience, quality assurance, business analysis and coaching skills. Look for developers with the right mindset instead of developers with great hard skills, but that can’t collaborate effectively. By applying these principles, the people described in the last paragraph won’t be that hard to find.
Also there is a lot of waste when management layers are created to coordinate teams specialised in technologies (Backend, Frontend, Infrastructure, Mobile) or divided by role (Design, Quality Assurance, Product Management, Software Development). The noise created by middle management clouds the perception of the team of its end goal which creates a roadmap disassociated from the user. When a problem is broken with no focus on the user the chances of hiring more people than needed increases, because the roadmap is developed based on forecasting instead of on data gathered from user interaction with the system. The time and money saved by avoiding that approach can be used to pay the right people very well, which helps to ease the search for talent.
Once we group people around a clear objective, the architecture will follow through. There are some strategies to increase autonomy like breaking the user interface in components, so each team is responsible for the set of components that help accomplish the goals of its archetype. Although a certain degree of autonomy is achieved with this approach there will be parts of the system that are shared between archetypes. In Clean Architecture this shared logic is called Enterprise Business Logic.
Usually each archetype interacts directly with a portion of the UI and indirectly with portions of the backend. The UI can be part of a web or mobile application or it can be dedicated frontend for that archetype. The backend has the business logic that implements the goals of the archetype when interacting with the UI. In Clean Architecture this type of business logic is called Application Business Logic. The take away here is identifying when business logic belongs exclusively to one archetype or if it is shared between two or more archetypes. In other words separating Application Business Logic from Enterprise Business Logic.
The teams should be free to change their UI and their Application Business Logic, after all they are focused on one archetype. The Enterprise Business Logic is shared by definition, therefore other teams have to be taken into consideration when changing it. It is tempting to create a platform team to take care of the Enterprise Business Logic, but by doing that the teams that are focused on the user lose their autonomy to make the necessary changes to deliver value. Worse than that, the platform team loses connection with the user which can cause over-engineering and accidental complexity.
Collaboration and Autonomy
One might say that shared ownership means chaos. This argument is very similar to the one that supports people grouped by technology instead of grouped by archetype: the belief that people cannot collaborate effectively. Collaboration has a cost, but adding layers of management or creating teams that become bottlenecks is even more expensive.
In order to keep consistency while scaling Enterprise Business Logic, developers of different teams have to talk with each other often. One way to achieve this is to have a periodic collaboration session where some members of each team meet to review the current enterprise architecture and to propose changes. Other collaboration sessions about product strategy, user experience design or quality assurance should take place to maintain consistency across teams on those topics. Another way to achieve consistency is to rotate people between different teams from time to time.
Autonomy doesn’t mean developers are free to use whatever technology they want to build applications. Teams should have autonomy to deliver value to the user, not to play with technology. One way to give autonomy but still facilitate collaboration is to use only JVM languages. Developers have options regarding programming languages (functional, object oriented, etc..), while rotation between teams is less painful. It is very common to realise that logic that once belonged to a single archetype now needs to be shared. It is easy to promote code from Application Business Logic to Enterprise Business Logic, when the code is written in languages that are interoperable.
Not only programming languages, but all technology options have to be restricted: from user experience to infrastructure. One might say people have to use the best tool for the job, which is correct, but have a wide range of options hurts collaboration and therefore hurts the end user. If the end user doesn’t have its motivations attended the business as whole fails.
The right incentives
People work based on rewards, even if they are looking for self-satisfaction or a sense of accomplishment, they are going to take action in order to get to it. Remuneration is a very important reward that most people deem important. Companies in general increase compensation based on the level of responsibility one has within the company. This approach encourages people to pursue roles with greater responsibility and responsibility is directly associated with control.
When a single person has control over software architecture, product strategy or user experience, collaboration diminishes. Collaboration is important because by having different points of view about a problem results in a better solution. Also in a collaboration session a common sense of the whole is created and everybody learns together. When a single person is responsible for a problem it tends to work in isolation and to micromanage others to execute a plan. Since every decision goes through one person, solutions are planned in advance to be approved, changes take longer to be executed and the product suffers.
One might say that a good leader always delegates and listens to their subordinates. It is true, but this is about hoping for the best case and forgetting about the worst case. If leaders delegate and listen they sound much more like coaches. In a collaboration focused company, leaders emerge naturally as coaches. These people are respected and their opinions are listened by others, but more importantly they ask the right questions and maintain an environment of cooperation.
People should get bonuses when their team collaborates well in improving the product. Also all teams get bonuses when the company as a whole collaborates well in improving the product. Since teams are focused in archetypes, clear business metrics can be associated with each team resulting in a fair evaluation.
A big challenge to avoid up-front planning and increase collaboration is to communicate on the right level of abstraction. In other words to know when to discuss problems and when to discuss solutions. Also when discussing solutions to do so in the level of detail necessary for each conversation.
In an executive level discussion where company strategy is being defined, the abstraction level has to be very high. No solutions should be discussed, but instead high level goals and how those goals impact business metrics. With those high level goals in place and metrics to mesure their success, teams can collaborate in order to deliver solutions. Examples would be "increase user session time listening to music" or "increase number of paid subscriptions".
Multiple sessions can be ran with members of each team in order to define which archetypes are involved to accomplish a specific high level goal, or if new archetypes have to be created. No graphical user interface changes or technical details should be discussed during those sessions, instead high level actions the archetype would like to accomplish in the system are discussed. For example: "share the song I am listening to with my friends" or "automatically play a new song that I might like after the current one finishes". Ideally those sessions would have the end user present, either way people have to put themselves in the shoes of the user at all times.
Once a set of actions and the respective archetypes are defined, each team can collaborate to deliver solutions by changing the architecture and the user interface. Since every team had people in the prior collaboration sessions, high level context is not lost. As described before, members of different teams collaborate regularly about user experience design, product strategy and enterprise architecture. As a result, consistency in those areas is not lost when changes cross different teams.
These principles clash with what the industry has been practicing for decades. The older the company the more ingrained command and control structures are. Middle management will be the first to resist any attempt to change the status quo, simply because their very existence relies upon it.
Top level executives are more inclined to improve results and also to communicate strategy in a high level of abstraction. The problem is they are also accustomed to long term planning and budgeting, which kills collaboration and replaces it with up-front solutions and control of progress.
In order to change that mindset, the new ideas that come from the teams (like creating a Social User archetype) should be continuously evaluated based on return of investment. A clear set metrics should be checked periodically, for example "the number of people that opted-in to share their current song with their friends". The team will focus on doing the least effort to maximize that metric, which means short cycles of putting something very simple live to the end user and then measuring the results. If the metrics don't go up after a period of time, the initiative can be killed so people can focus on something else.
For a company to be user centric a variety of factors have to be taken into account, but the most important one is culture. The technical practices that enable teams to have a short feedback loop have to be supported by a high level strategy of the company to put the user first.
People organised around archetypes results in teams that can speak in the right level of abstraction, without losing focus on providing value to the user. It is easy to draw the borders of the system, because they map to the team organisation. Those borders make up to an architecture that makes it clear when teams have to collaborate and when they have autonomy to make decisions. Finally, middle management can be reduced to a minimum or even eliminated, reducing noise and enabling collaboration, which results in happier people and a better product.