Deprojectification of agile: The new orthodoxy of long-term product teams

A key event in the agile trajectory was the publication of the Agile Manifesto (Beck et al., 2001), which established a set of principles and practices that emphasised individuals and interactions (rather than processes and tools), working products (rather than comprehensive documentation), customer collaboration (rather than contract management) and responding to change (rather than following a plan). In contrast to Waterfall projects, people were now expected to “work in cycles, using frequent, time-boxed iterations that allow regular check-ins with and feedback from their colleagues and their end-product customers” (Krehbiel et al., 2017, p. 90).

The agile way of working has had a profound influence on software development (Dyba and Dingsøyr, 2009) and as a generic management idea beyond the software domain (Madsen, 2020). Software development research has (implicitly or explicitly) emphasised agile a project logic. The evolving practices of cross-disciplinary interaction, iterations, customer collaboration and incremental change contrast with Waterfall projects, which are characterised by excessive formalisation and specifications. However, a project logic still dominates within agile approaches and research on agile software organisations and methods has to a large extent analysed the benefits and challenges of Agile projects, typically contrasting them with Waterfall projects (Foss and Klein, 2022¹; Conforto et al., 2014; Dybå and Dingsøyr 2009; Felipe et al., 2016; Hobbs and Petit, 2017; Serrador and Pinto, 2015). Studies on agile teams have tended to assume that software development proceeds according to a project logic and a project mindset, generally failing to address the permanence and stability dimensions (Grass, Backmann and Hoegl, 2021; Gren, Torkar and Feldt, 2017; Przybilla, Wieser and Krcmar, 2019; Vishnubhotla, Mendes and Lundberg, 2017). This organisational logic mirrors the ambidexterity approach based on the notion that existing organisations need to combine exploitation (efficiency and fine-tuning) with exploration (creativity and innovation) (O’Reilly and Tushman, 2016; Raisch and Birkinshaw, 2008). These two sets of activities differ and require different mindsets, goals, leadership and competencies. Software development is usually handled by time-bound, autonomous teams that ideally follow agile practices and share an agile mindset. These units are disbanded upon completion of the development project (Hobbs and Petit, 2017; Kelly, 2018; Puranam and Clement, 2018; Serrador and Pinto, 2015). As stated by Puranam and Clement (2018):

Instead of marching in bureaucratic formation, they are given freedom to “sprint,” i.e., rapidly advance in parallel towards a solution with minimal top-down oversight. Interdependencies between the various teams are managed through frequent meetings and any rework is accepted as part of the iterative process. Upon completion of the immediate task, teams either dissolve or take on a new project. (blogpost).

The project logic and (implied) temporary mindset have been criticised in the popular agile literature, since such short-sightedness hinders the development of trust and cohesion in teams, requires overly complex governance mechanisms and creates extensive handover challenges between development and operations departments (Kelly, 2018; Skelton and Pais, 2019). Instead, an alternative model has been promoted based on stable, cross-disciplinary teams that encompass development, maintenance and operations and supported by the success stories of Spotify, Netflix, Amazon and ING (Barton, Carey and Charan, 2018; Kelly, 2018; Skelton and Pais, 2019; van Gerwin, 2018).

A typical description of the new form of team is as follows:

Any organisation, when they first start working with Scrum or other agile frameworks, does so in project teams. These teams stay together for the duration of the project, meet regularly and deliver the project within a short time frame. …By stable agile teams, we refer to cross-functional teams that can carry out an assignment from end to end. This kind of team works together on the same assignment full-time and is not dissolved when the assignment is completed. (Claessen, 2018).

Thus, stable teams as an organisational solution challenge agile a project logic. Kelly (2018) argued that software development has been hindered by the excessive emphasis on a project logic and project management models, which advocate working in time-bound project teams. The problem with a project logic is partly the temporariness of teams (teams disbanded quickly and maintenance and operation outside the responsibility of those teams) and partly the complex governance mechanisms associated with project management. According to Kelly (2018), software development should be regarded as continuous, requiring organisations to be based on continuous, long-term agile teams. Teams should prioritise long-term product or business domains rather than time-bound projects (Kelly, 2018).

Similar ideas were described by Skelton and Pais (2019) and DevOps proponents. Skelton and Pais (2019) stated that teams are the basic operational delivery units in an organisation. They should be diverse and cross-disciplinary, remain stable over time and be kept small to develop trust between members (Skelton and Pais, 2019). The DevOps approach emerged due to the growing friction between software development and operations (Kim, Debois, Willis and Humble, 2016; Wiedemann, Gewald, Wiesche and Krcmar, 2019). Previously, development and operations teams had separate responsibilities, with software releases “thrown over the wall” to operations. The methods did not address the gap between development delivery speed and operational teams’ capacity to provide resources and deploy updates. Development and operations should be better coordinated, and teams may be a way to achieve this (Kim et al., 2016; Wiedemann et al., 2019).

Coordination challenges arise at a higher organisational level when several cross-disciplinary agile teams are dependent om each other. Given the focus on cross-disciplinary working in a single team, how should issues of discipline-specific development, knowledge sharing inside a specialised sphere of competence, identification with a discipline and personnel management be handled? De Smet (2018) recommended an “agile matrix” that supplements the team (product) dimension of a structure with a disciplinary or functional manager who can “build up the right capabilities and people, equip them with the skills, tools and standard approaches to deliver functional excellence and ensure that they are deployed to value-creation opportunities, not involved in the day-to-day work of squads” (De Smet, 2018, p. 3). Such an agile matrix resembles a resource matrix, as described in the organisation design literature (Nesheim, 2021).

Some studies have addressed the new “conventional wisdom” in agile software development. For example, Gerster, Brenner and Dremel (2020) emphasised product orientation and stable teams that embrace development and operations as key agile features and structures. This approach contrasts with a traditional functionally based “silo” approach to organisation. However, the contrast with and changes from an agile project approach are mentioned briefly and are not discussed by the authos. The DevOps approach prompted several empirical studies, most often regarding information systems. In a review article, Leite, Rocha, Kon, Milojicic and Meirelles (2019) identified six fundamental concepts and seven key tools for DevOps, none of which capture the product versus project dimension. However, the authors briefly discussed three organisational alternatives for DevOps deployment: collaboration across departments, cross-functional product teams and DevOps teams. Based on a qualitative study, Wiedemann, Wiesche, Gewald and Krcmar (2020) found three factors that contribute to IT and business alignment: individual componentisation, integrated responsibility and multi-disciplinary knowledge.

Studies have considered the “solutions” and challenges of agile product teams and Dev-Ops ideas and practices, but few studies have focused on the product versus project dimensions and the stability versus temporality of teams. Pulakos, Kantrowitz and Scheider (2019), exceptionally, argued that stability “provides a secure foundation that helps people stay focused on performance during disruptive change, rather than being worried and distracted about what may happen” (p. 309). They found that stability, rightsized teamwork and relentless course correction play especially important roles in fostering organisational agility and resilience, which in turn lead to substantially higher organisational financial performance. However, they did not explicitly address the stability of team membership in their study.

Before the changes (2015 in Alpha, 2017 in Beta), processes and product development in both firms were typically organised according to time-bound projects that were separate from operations. Informants across the firms claimed that this structure had many disadvantages, one of which was related to the handover of solutions from projects to operations. Participants in development projects were not involved in their operational implementation; there was often minimal communication and feedback between the two functions and a lack of long-term ownership of solutions.

Another issue was the detailed control, specification and documentation required for development projects. Respondents perceived no support for autonomy and creativity, which reduced the speed and degree of innovation. Alpha 16 provided an illustrative statement, as follows:

There was a low degree of autonomy in project teams … [In practice], development took place through specifications, templates … and decisions taken outside teams to a large extent. It was very Waterfall-oriented … ownership of the product was low and the understanding of what the teams built tended to be low.

Reorganisation into agile product teams started in 2015 in Alpha and in 2017 in Beta. In Alpha, the structure of the entire Development Division was gradually changed into an organisation based on cross-disciplinary, long-term teams as operational units. The new team-based structure was supported by a simplified governance structure, initiatives to foster a more commercial and customer-focused culture and the use of agile roles and tools, such as design thinking. In contrast, in Beta, product teams emerged more selectively and mainly as three “agile islands”: digital sales, digital service and digital claims.

In both organisations, products were the foundation of the new teams. Each team now “owned” a product or domain that, rather than being time-bound, had a measure of permanence. The organisational logic was that, if a product or domain existed, it should be supported by a team. As one of the respondents stated: “For example, we will always have payments and transactions, so we will always need clever people in these areas. When you have lasting domains, it is very sensible to have lasting teams” (Alpha 14).

When organising by product, respondents perceived that a different dynamic emerged. As one respondent said:

You must both operate it and maintain it (in the same team). Now, the logic is: If you build it, you own it, you run it, you kill it. Developers need to think about how to develop a solution that the team can operate and maintain. Employees need a more cross-disciplinary approach … and to think more in terms of the life course of a solution. You need to avoid shortcuts that accelerate development but increase the cost of operations. (Alpha 8)

Although the change from distinct project and operations units to a product-based organisation was easily recognised in retrospect, the product teams tended to emerge gradually. This incremental process was especially prominent in the case of Beta, where the changes (to three product teams) took place locally and not according to a division-level plan, in contrast to Alpha. The process was explained by a Beta informant as follows:

The change from development projects to products did not take place overnight, but more gradually. We recognised that it was more sensible to work with a long-term horizon in the teams in question … I don’t remember the exact date, but we gradually started to use the term “product team” rather than “project team” … It took a long time for parts of the organisation to recognise this change. (Beta 8)

The teams were similar in the two organisations. Typically, a team was cross-disciplinary and included developers, architects, UX designers, tech leads and product owners. With few exceptions, each employee was a member of a long-term team. Thus, the organisation of the Development Division in Alfa and the “agile islands” in Beta differed from project-based structures, temporary teams and organisations in which employees were members of several (stable) teams. A critical dilemma in team-based organisations is that of stable versus flexible team membership. In both firms, respondents emphasised the advantages of team stability, claiming that trust and relationship-building among team members are vital. As one respondent stated:

Relations are very important. If you look at teams (with membership changes) … they slow down the work … Members of a team who have worked together over time have come to know each other, have worked on processes and learned how to work smarter. They are much more productive than a team that has experienced changes [in membership]. (Alpha 14)

The other main driver of membership stability was domain competence, exemplified by the following perception:

You build knowledge of the domain. For example, the loans team needs to understand the settings in which customers must borrow, how our credit model functions, what data we need and how we may obtain that data. They develop domain knowledge and a cooperative climate that we intend to conserve as long as possible. (Alpha 6)

Thus, in a team with stable membership, employees perceived that they had a long-term, safe framework for their work, came to know their colleagues well and developed in-depth domain-specific knowledge. They were allowed to work consistently in one area without having to switch between tasks in different teams. At the team level, perceived advantages were the decreased need for onboarding and other processes due to frequent changes in team membership. Teams that experienced many such changes were perceived to be less efficient.

Although one-team-per-employee was clearly preferred, this was not always feasible. A lack of specialised technical competencies or cases in which the amount of work was less than full-time could result in some employees working across teams.

Regarding team membership, although all informants desired stability, many experienced dilemmas. They perceived it as important for teams not to get stuck in a pattern because, sometimes, the team structure was less than ideal and needed to be changed. In addition, although the product or domain might be long-term in nature, a need for capacity changes or special competencies could arise during the lifetime of the team. Respondents perceived that, in some cases, teams could become “too strong” and create barriers to necessary changes in membership.

The informants in both organizations claimed that management tended to emphasise the increased autonomy of teams and give team members more responsibility than before. However, the conditions for such autonomy differed. In Alpha, team autonomy was supported by compatible division-wide changes in governance structures, cultural initiatives and agile ways of working. In Beta, the three product teams were embedded in a more complex organisational setting, coexisting with other organisational and hierarchical structures.

One Alpha informant described the new teams as follows:

It is now more up to the team (to make decisions). An increased level of autonomy is very positive. From my point of view, the biggest effect is that each member is expected to own what he or she has built from the concept phase to production and use by the customer … They perceive that they have created something for the customer, in contrast to before, when they were given an instruction and executed it. (Alpha 17)

In both organisations, teamwork was coordinated by a product owner who was responsible for dealing with backlogs, activity coordination and relations with “neighbour” teams and other parts of the organisation. Although this was a key role, informants emphasised that the position holders should not micromanage processes or team members, but instead create conditions to foster creativity, empowerment and cooperation in their teams. Specific agile roles (such as agile coach and tech lead) and agile methods (e.g., Scrum, digital boards and design thinking) were used extensively in both organisations.

The teams were perceived to have greater autonomy and to provide better solutions through working across disciplines. A vital question in both Alpha and Beta was whether all relevant disciplines were involved. Many respondents expressed a preference for involving disciplines other than ICT to align solutions better with the relevant markets and customers. In both firms, respondents pointed out that there was still some way to go:

Simply put, we are cross-disciplinary ICT and are not close enough to the business. Some teams lack people with commercial and business knowledge. Thus, there is sometimes a distance between the teams and the core business, and we don’t really understand each other. (Alpha 19)

In each “agile” team in the two organisations, employees’ attention was directed towards one team and the interaction and problem solving within that team. This was the task dimension of the organisation related to a product or a domain. Given the focus on cross-disciplinary teams, two other questions arose during the interviews:

How was the resource dimension handled, and who was responsible for personnel management?

In Alpha, a product- and team-based structure was established in 2019. Initially, the previous discipline-based departments were maintained alongside the new teams, so, for some time, the organisation resembled a matrix structure, with employees having formal dual attachments to their teams and the departments they reported to. This structure was not considered optimal. The respondents claimed that the dual attachments tended to pull employees in different directions and result in a lack of alignment between the two entities’ interests: “You were drawn in different directions, with different priorities, different managers, different roles and expectations” (Alpha 7). There were also challenges related to identification and personal management. As one respondent explained:

Developers felt more at home in the Systems Development Department. Teams depend on psychological safety and a sense of community, but the department (not the team) was the main arena for meetings and other social contact … [In a matrix], the personal manager cannot observe your work and has limited knowledge of what you do, what tasks you work on and how to prioritise. (Alpha 6)

After a while, the departments (and, consequently, the matrix) were dissolved. Instead, coordination was introduced between the teams and the division in the hierarchical structure. At this level (the “tribe” level), which comprised three to six “neighbour” teams, strategic direction, coordination of teams in the tribe and personnel management were provided for them.

Many respondents mentioned the development and sharing of special competencies among employees across teams. They acknowledged a balancing act between cross-disciplinarity and customer focus in teams on the one hand and the development of specialised, discipline-specific competencies across teams on the other hand: “There is a potential trap if you are not conscious that it may turn out that way and don’t develop your field of expertise” (Alpha 18).

While employee development usually took place during the day-to-day activities of the product teams, learning and knowledge development also occurred outside them through, for example, monthly “learning days.” Managers encouraged employees to interact with other employees with similar competencies across teams, as well as to participate in activities and arenas that involved other competencies. Learning days were open to everyone in the division, but the emphasis was on the largest disciplines. To enable discipline knowledge sharing and development, several guilds or communities of practice were established, each of which had a coordinator with a rather “loose” job description.

As described previously, the Beta organisational context was different and raised the issue of whether secondary structures should supplement the “islands” of cross-disciplinary teams. Most team members belonged to the Technology Division, which had a matrix organisation structure, comprising resource units based on employees with similar competencies, teams, and other task units, such as project units. Thus, the members of the three teams were part of a matrix organisation and reported to both their resource units and their teams. Thus, the resource unit was responsible for knowledge sharing and development within the relevant sphere of expertise. In contrast to Alpha, which had separate supplementary structures to handle the resource and knowledge sharing dimensions, the resource units in the Beta matrix structure were responsible for both.

In both organisations, informants claimed that development projects were an established tool for organising change and reflecting stability–change–stability cycles. The interviews indicated a high level of change in the organisations, combined with a lower prevalence of projects (as temporary suborganisations) and a reduced focus on project logic and a project mindset as means to facilitate change. These perceptions were evident in both organisations. Agile teams now handled activities that were previously part of development projects. As Beta 6 explained:

[Activities] that were previously part of a project are now a part of agile teams … That’s the big picture. There are still projects alongside agile teams, but a large part of what previously would have been a project has now moved inside an agile team.

In Alpha, informants tended to downplay the importance of projects to an even greater extent. Although they recognised the project manager role and engaged in temporary tasks, these were not regarded as projects. As expressed by one informant: We don’t have projects anymore. (Alpha 16).

Given the different ranges of agile teams in Alpha and Beta, the decline in project organisation and project logic was more extensive in the former. The interviews in both organisations clearly indicated that the project manager role had become less important than the product manager role. Many previous incumbents of the project manager role now work as product managers or team leads but may draw on their experience in agile project methods.