Agile and Architecture: In Thunder, Lightning, or in Rain?

“In thunder, lightning, or in rain?“ is the continuation of the question “When shall we [three] meet again?” from the opening lines of Shakespeare‘s «Macbeth». And even though the question of when architecture will meet agile again doesn't pertain to three parties –and hopefully won’t include as much bloodshed as a Shakespearean play– it seems very much a matter of your point of view whether they ever have parted ways or whether they ever will come together again.

A look at some of the History of agile

If we look at the start of the “agile” movement in the context of its time, it becomes abundantly clear that it was a counter-movement, addressing the most pressing issues with regard to the prevalent approaches to software development at the turn of the millennium. And it was a movement rooted in an intellectual niche –software development– that took certain properties as a given.

 Even though it is now called “The Agile Manifesto” by most people, the title actually reads “Manifesto for Agile Software Development“, which illustrates this point nicely. And most of the people contributing to it were excellent software developers. 

And all of them were involved with lightweight approaches to software development which made heavy use of small and competent teams. With one of the competencies being –implicitly or explicitly– architectural knowledge. 

Take extreme programming (XP) for example and the way it addressed one (of many) of the aforementioned pressing issues: Architecture. XP came to live in a project where people like Kent Beck and Ward Cunningham were team members. So the inventor of the «x»Unit-testing frameworks and the inventor of the wiki work together on a project and come to the conclusion that it’s a good idea to put the power of architectural decisions into the hands of the people working on the project. As opposed to having people removed by several project phases from implementation to “design” the architecture up-front as for example the DoD-STD 2167A –one of the heavyweight approaches of the time– would have demanded. Seems reasonable. 

And the good thing is that the authors of the manifesto for agile software development even put those thoughts into the manifesto, by writing down that “Continuous attention to technical excellence and good design enhances agility.” This fact is often overlooked since it “hides” on the “second page” of the manifesto as the ninth of the “Twelve Principles of Agile Software.“ A part of the manifesto that is almost unknown amongst many of the people whom I meet – even though they work as “agile coaches“, “Scrum Masters“, “Agile Transformation Officers “ and the like. 

Talking about things from the manifesto for agile software development hides one sad fact: “Agile”, as I encounter it today in most large corporations, is far away from the ideals of the original manifesto. As I mentioned earlier the “Agile” movement was –from my perspective– a counter-movement at the time. A movement to counter the nonsensical heavyweight processes that dominated the software development world at that point in time. Especially in large companies. Today many of those same large companies label what they do “Agile”, but use frameworks that are heavier than anything we’d seen in the nineties. Furthermore, the term “Agile” no longer encompasses only agile software development but many people assume that “Agile“ also mandates or creates a lean organisation and thus they include lots of things pertaining to organisational matters in approaches. 

And the bigger the frameworks get, the more I see the part of “technological excellence“ shrink. And thus the implied architectural knowledge inside the teams is hardly a given anymore.  

My personal anecdotal evidence is a little poll I sometimes like to do at agile come-togethers. For a couple of years now, I asked at random agile congregations how many of the people present wrote code in the last month or so (or did anything technical for that matter), and the number is frustratingly low. Even if I fall back to more generic questions like “who knows how to code?” or “who has done any technical work for an extended period of time?” the numbers usually don’t get better. The former software engineer who now works in organizational development or similar biographies is a rare exception.

But then again the meaning of software architecture has shifted over the last couple of years, too.

Architecture has quite a history on its own

Even though I spend a huge part of my life with a significant focus on software architecture, I've always had a hard time defining what software architecture exactly means. The fact that my cousin's husband is an actual architect doesn't help to make things easier, I have to admit. 

I like to look back to the works of Edsger Dijkstra and David Parnas from the 60ies and 70ies for the first original work on software architecture as I understand it today. Mostly –if I had to put it into one sentence– I would say, it is about actively managing which part of the system under consideration has which responsibilities at which point in time. Martin Fowler takes a broader look here and describes it as: «When people in the software industry talk about “architecture”, they refer to a hazily defined notion of the most important aspects of the internal design of a software system.»  [Fowler2019] 

For an in-depth discussion of what software architecture could mean from my point of view, you might want to look at the open source project arc42 [arc42] which also provides a helpful structure on where to put which part of the discussion of the system(s) under consideration. 

On the other hand, a lot of new things and challenges have come about over the last decades and nowadays I see a mixture of understandings of the term “architecture” in the software development world that seems almost Babylonian to me. 

I see groups referring to architectures (sic!) in terms of meta-models (e.g. “we'll use a hexagonal architecture” or “we implemented this with an MVC architecture” or other things like that, which –to my understanding– refer more to types of architectures or patterns than to the core concept of architecture in sense of the likes of Dijkstra and Parnas). On the other hand, I observe people talking about architecture by stringing terms like Kafka, Docker, SPA, Elixir and k8s (current –2020– technologies and patterns for implementing some aspects of information technology systems) together without any reference to the actual domain context. Then again there are people using domain driven design (DDD) and I hear that some of the technology-centered architects regard their work as “just” doing design (while defining bounded contexts and interactions is much closer to my understanding of architecture).

And I guess on some level all of them are right. Discussion of architectural meta models is important. Infrastructure architecture is important. From the point of view of an infrastructure architect DDD might well be ‘just design.’ – even though I personally would beg to differ.

In modern endeavours the original agile ideas and architectural knowledge need to come together

Just like the technical aspects of the “Manifesto for agile software development” have become compromised in a lot of today’s cargo-cult “Agile”, software architecture –to my eye– has partially lost the focus on first principles and the original ideas of structuring the internals of the system(s) under consideration. And, to single out just two things I would like to see more of from the realm of software architecture, seldom have I seen a quality tree in an architecture discussion in the last couple of years, and rarely do I hear of architectural design records. From my point of view, the appropriate approaches for today‘s endeavours need more focus on first principles – both on the side of software architecture and on sound approaches on how to organize the work. 

I wouldn't necessarily call all those approaches “Agile” in the currently widespread sense, so I personally tend to think about it as post-agile, for want of a better word. David Anderson used the Term 2010 to describe the Kanban Method «[…] as 'post-Agile' in the sense that it introduces an incremental, evolutionary approach to change while the agile movement was a rebellion that introduced a revolution against existing methods from the 1990's.» [Anderson, 2010] and Alistair Cockburn uses the same term in his 2019 treatment of central thoughts around the "Heart of Agile" movement. He describes the term as «“Post”-something means that we are in an in-between period, where one thing has been incorporated into our culture and the next thing has not yet formed to the point of naming.» [Cockburn2019] I actually don’t see that the original ideas from the manifesto for agile software development have been incorporated in our culture, but I very much agree with the sentiment that it has been absorbed (and morphed) by the culture and we need to enter a post-state if we want to re-invigorate the original intentions of the lightweight approaches.

So my final take on when and where agile and architecture shall meet (again) is that modern endeavours need to embrace both the need for sound architectural work and the need for new approaches for the old agile –which might well, by now, be post-agile– ways.

Yours

Michael Mahlberg

References

[Anderson2010], David J. Anderson, Online, in DZone, https://dzone.com/articles/how-kanban-got-hot-david

[arc42], https://arc42.org/overview/

[Cockburn2019], Alistair Cockburn, Online, on Heart of Agile, https://heartofagile.com/post-agile-thoughts/

[Fowler2019], Martin Fowler, Online, Software Architecture Guide, https://martinfowler.com/architecture/