ºÚÁÏÃÅ

Podcast Transcript

Rebecca Parsons:

Hello everybody, welcome to the ºÚÁÏÃÅ Podcasts. Today we're talking about architectural governance. My name is Rebecca Parsons. I'm the chief technology officer for ºÚÁÏÃÅ and one of the cohosts of the ºÚÁÏÃÅ Podcasts.

Mike Mason:

My name is Mike Mason and I am a head of technology with ºÚÁÏÃÅ. I work closely with Rebecca and I'm also one of the hosts of the podcast series. I'd like to introduce our two guests today. Zhamak and Johnny. Zhamak.

Zhamak Dehghani:

Hi everyone. I'm Zhamak Dehghani, I'm one of the principal consultants in ºÚÁÏÃÅ. And joining you from our San Francisco office.

Jonny LeRoy:

Hi, and I'm Johnny LeRoy, I'm playing a head of tech role in North America with ºÚÁÏÃÅ.

Rebecca Parsons:

Thank you. Zhamak and Johnny. So we're going to be talking about architectural governance today, and I'm not going to do the dreaded thing and ask for a definition because defining architecture, defining governance, that's always complicated. But tell me what your views are on architectural governance, and why it is so important?

Zhamak Dehghani:

Well, first, thank you Rebecca for not asking the difficult question of defining architecture. I have simplified the definition of governance for myself to be the framework that guides making decisions around the architectural ecosystem in an organization. And often this framework is guided and influenced by, essentially the vision and the mission of the organization, the vision of the business. So that's simply to me the definition of the governance and it matters because it guides the decision making at all different levels of kind of the technical stack and the technical landscape in an organization. And it helps people at all different levels make daily decisions essentially.

Jonny LeRoy:

Yeah. And I can add it in there because governance is often a scary word that we often try to avoid. And I think there's a few sides to it in my mind. So one is really around risk. What's the organizational risk? And it often gets bundled up with compliance for regulated industries. But all organizations, particularly when they're creating software, whether they're in a regulated industry or not have some level of risk. And governance is really about understanding that risk and having some visibility into it. So risk management is a key part of governance.

Jonny LeRoy:

There's another angle which is really around effectiveness and efficiency, particularly as organizations get bigger. And how do you leverage organizational learning? What's happening in one part of the organization? Can that be learned somewhere else? Do you need to keep making the same mistakes in different places? And governance can really be around making sure we don't keep making the same mistakes. And you can learn from each other's good and bad results that you've had in the past. So really in an ideal world, yeah, it's not about control or stopping what you're doing, in an ideal world it's about getting big picture visibility so you can make good decisions and act sensibly as a whole organization rather than just individual parts.

Mike Mason:

You touched on this a little bit, but the G word, governance, often does have a bad rap. Is that because there's bad governance and good governance? And we're trying to move towards good governance over less good. And what is the history of that?

Jonny LeRoy:

Well, there's one framing, a lot of larger and more traditional organizations that we go into, there will be some kind of governance or compliance or oversight or enterprise architecture group who can be characterized as the department of no, right? Their job is to say no to stuff that technology and engineering teams want to do. And so it can get a bad name if governance is done poorly and they just end up saying no or mandating outdated or not particularly beneficial tools or practices. So really what we want to talk about is actually how to still get the benefits of governance without defaulting to being the department of no.

Zhamak Dehghani:

And I think that there is a shift from thinking about governance as a way of control to thinking about governance as a way of enabling people. And to me it always comes down, I'm a developer, I need to make a decision or maybe I'm a domain architect or an application architect, I need to make a decision around the architecture of my application. How can I be enabled to make the right decisions so the impact of that decision is not externalized to another team around in that ecosystem in a negative way?

Rebecca Parsons:

So Zhamak that enablement or empowerment, is there an aspect of alignment of objectives, and an alignment of understanding of goals behind that as well so you can assess what the possible scope of impact of a decision is?

Zhamak Dehghani:

Yes, absolutely. I think there is a high level alignment around principles that is probably at an organizational level, that's what are the founding principles that we like to implement in compliance or influenced by the vision of our organization? And then from then on I think those principles obviously cascade down to different domains, maybe domains themselves within an organization. For instance, if you're a retail and you're in the logistic domains, you have within your domains, you may have a set of principles that impact the practices within your domain. But at an organizational level cross domains, you have a set of principles as maybe what you need to standardize where you communicate messages or your capabilities outside of your domain. So there are principles that align different domains at the organizational level. I'm not sure I answered the question you were looking for.

Rebecca Parsons:

That's good. Thank you. So Johnny, you mentioned more traditional enterprise architecture groups and such. What kind of changes have been happening that have altered the landscape around architectural governance and what it means to govern architecture effectively now?

Jonny LeRoy:

Yeah, there have been some major changes. And I think a lot of it, there's been lots of conversations about what's driving the rate of change in technology and in the business environments that we're operating in. But I think what's becoming clear about the modern landscape that's impacted by technology and where technology companies are trying to operate, is that it's far less predictable than it used to be. So in classical industry and planning, you could analyze the situation, make a long-term multiyear plan, divide it up into little steps and then tell everyone exactly what to do in a command and control approach of directing how the workforce operates.

Jonny LeRoy:

And what's happening now as environments are getting more complex and things are moving faster, the ability to completely plan is becoming harder if not impossible as we head into more complex environments. So it's driving a need for organizations to be more adaptive rather than just relying on multiyear planning and work breakdown. And what organizations are realizing is that to be adaptive you have to drive a fair amount of autonomy and decision making down to the teams who are actually doing the work. And then that in turn changes the level of oversight and governance that you're trying to do.

Jonny LeRoy:

And there's a major management shift from the old Taylorist view of your employee is evil and lazy and need to be controlled and directed to the more modern approaches of trusting your employees and trying to empower them and set them free and give them autonomy. And what we've seen is a lot of the older school enterprise architecture and governance approaches were still based in that older world. And the more successful organizations that we work with are embracing approaches to governance, approaches to planning, approaches to direction and directing teams that still enables a large amount of autonomy.

Rebecca Parsons:

So then Zhamak, Johnny was just talking about how more effective organizations these days are switching to different guiding principles around architectural governance. What do you think are some of those critical principles that will support effective architectural governance in today's world?

Zhamak Dehghani:

Yeah, absolutely. I think for me there there are a few that are really important and they are directly related to the influencers that Johnny talked about. If you have organizations that are moving fast, they have autonomy, it's really important to have architectural expression of architectural governance that is visible and understandable by everybody. So for example, tools like architectural decision records where you record and communicate easily the decisions that you have made and accepted them, and why you have made those decisions, and what you have learned from them. And make that accessible to everybody in the organization is one of those principles, have those recorded decisions that are visible and easily understandable and clear to everybody.

Zhamak Dehghani:

The other aspect I think is the testability. Be able to actually in an automated way test the governance principles and practices that we put in place because the code is changing fast, the applications are changing fast, the experimentation's come and go. How can we actually keep up maintaining those rules and guidelines? Really without having tests in place is impossible. And the last year I think respecting that autonomy and the nature of the polyglot nature of the organization that within the scope of teams, they may have their own model of governance within the team as long as the scope of impact is within the team or within the domain. So respecting that many organizations work in a very polyglot from the technical point of view, from the practices point of view environment that need a different governance model at the micro level, at the domain or team level.

Jonny LeRoy:

I can add on a bit. There's a phrase that's been going around in my mind that we talk a fair amount in the industry about two pizza teams, reasonably autonomous teams that are small enough that can be fed by two pizzas. And I think that idea came out of Amazon. And that's great when you're trying to do small things. The big problem is what happens when you try to plan a banquet made out of two pizza teams? And I think that's some of what we're looking at in this governance discussion. And maybe maybe that'd be a good title for an article. But some of the principles, because if you just send off everyone, you end up with lots and lots of pizzas. But actually maybe you want a main course and maybe you want a dessert so you want to have some principles.

Jonny LeRoy:

And I think there's four areas that we tend to look at that effective organizations are using. And the first one is driving as much compliance as possible and governance to be automated. We can get into the platform, we can get into tooling or the build time or time, there's a huge amount in terms of quality, performance, security, and compliance with rules that can be automated. And that takes a lot of load off manual enforcement. So rule number one is lean on automation, I'm going to drive that hard. Three, the next step is moving your governance from detailed requirements and prescriptions, upper level or up two levels to vision and principles. What's the vision? What's the vision for your banquet? Do you want people to have a good time, have a full meal without micromanaging exactly how much kale goes on the pizza?

Jonny LeRoy:

So you need the vision and principals moving up from just specifying exactly which tools or which practices to use, and up one level and let some good examples of visions and principles. The third piece is taking people who were gatekeepers, the members of the department of no, and moving them into becoming collaborators with teams who are actually executing. They can still maybe reside in a central organization so they can understand organizationally what you're trying to do. But then they can be seconded as consultants into teams who are actually building software, building products. And so they can collaborate.

Jonny LeRoy:

So they're partly making the visions, principles, and constraints, but also helping teams and how to meet those. And that can be a good way to actually drive some harvesting of knowledge to share. And then the final piece is more of a mindset change, which is becoming comfortable with evolution and being comfortable with the continuous flow of products and software delivery rather than seeing it as this life cycle of analyze, design, plan, execute, deploy, and then hand over to someone else. And so it's that comfort with the continuous and evolutionary world, which is really the hard mindset change.

Rebecca Parsons:

And when you're talking about testability from an architectural governance perspective, how do you go about testing architectural principles? Some it's fairly obvious, we know what performance test we might have to meet, but are we really there yet in terms of knowing how to test some of these things or is there a mindset shift in even how we think about architectural characteristics that affect their testability?

Zhamak Dehghani:

Well, I think the movement has started, you have yourself written a book, Evolutionary Architecture, which touches on a lot of examples where principles can be tested and how they can be tested. And I can repeat some of those. I think having the foundation of continuous delivery and continuous integration with different test phases to put the right checks in the place. I think that foundation is in place. And it's a well accepted and well understood practice. It's the matter of clear articulation and clear communication of what are these access of in different... access of architectural governance or guidelines that we want to implement. And how do they compose? And how do they work together? How does security impacts freshness of the data or security impacts of performance when these kinds of guidelines get, for instance composed?

Zhamak Dehghani:

I think it's more of a mindset as Johnny mentioned than lack of tooling in support of that. And the tooling is growing as well. So their type of tests for instance for governing the application architecture, if you have a layered architecture that you want to prevent your higher levels of your application, your view, not access the data directly. There are ways of actually coding that into your code. And I hope that over the coming years we get better and better in reaching our testing framework to be able to express architectural decisions this way. But I think we have the foundation to get started.

Jonny LeRoy:

Yeah. And I think metrics are very important and it's good to be driven by metrics. But we can over obsess with two lower level metrics or mandate the same metrics everywhere across the organization. And I think it's often when you're looking organizationally it's often good to move to some of the higher level metrics. And maybe some of the business facing metrics of actually what are the outcomes you're trying to achieve in terms of what you're doing for your users, what you're doing for your revenue metrics, what are you doing for some of your organizational metrics, like ability to attract and retain people or how quickly teams can move. And so I would always be very careful in how detailed the level of metrics are that you're forcing teams to expose when you're looking for some quite high level metrics as business facing as possible, but then allow teams to drill down into much lower level metrics as they are trying to improve or trying to fix something.

Jonny LeRoy:

So there's a whole slew of different code quality metrics that you can use, they're useful for improvement and for tracking. But when your doing more organizational governance you want some really quite high level metrics about ease of change of an area of the code base or lead time for new features. If you're having problems that are inside the team then you can really drill down into your duplication, and your complexity, and code coverage. And various other things like that. But I think when you expose organization you should be at a higher level.

Jonny LeRoy:

And I think it's also useful to lean on some other types of metrics as well around slightly more qualitative ones around your teams and your people. Like how quickly people get ramped up if they join a new team. One of the slices we see around governance is trying to standardize on certain sets of tools or technologies. But one of the drivers for that is to enable people to move more quickly between teams. So if that is a key driver and you're already pushing people on their language or platform or tool selections because of that you want to be tracking actually what is the ramp up time for someone joining a new team, either a new hire or someone moving from a different team? And try to see how long it takes before they become effective.

Mike Mason:

Yeah. I think the thing that I really like about a lot of what we're doing in modern software engineering and architecture is trying to re-examine the metrics that we're using and rethink them and say, "Why are we really measuring that thing? Or why are we doing this practice?" Because to a certain extent you get what you measure. Especially if you tell someone that their compensation depends on hitting a particular kind of metric or something like that. So people will figure out ways to hit those metrics and you need to make that a constructive experience rather than someone just gaming the system. And so for me, a lot of this stuff comes down to thinking about why are we trying to pull that particular lever? And are we really getting the outcome that we want? As Johnny said, if you're helping... If you're constraining technical choices in order to produce an organizational outcome, like I can train people quicker, well, you better be achieving that organizational metric because otherwise you're just stifling innovation in order to achieve something that you're not achieving.

Zhamak Dehghani:

That's that's a really interesting point. I think the job of an effective governance should allow teams to optimize some things very locally and some things globally. It's just a matter of getting the right optimization. I have seen metrics that are collected locally, for instance, the team says, "We are moving really fast." Because they're only looking at the time to commit the code. They're not actually looking globally as, when does this code actually make it to the user and the customer? And on the other hand if you want to optimize globally we will all use the same tech stack. Then some parts of the system that needs to be developed faster with a faster technology is going to suffer. And it's all about getting the right framework in place to optimize at the right level for the right things.

Rebecca Parsons:

I like this focus on the thinking about outcomes and in particular business outcomes as well because as Mike said it. It helps frame what is it that you're trying to achieve and then you can decide if the costs that you are paying to achieve a particular organizational objective is worth the benefit that are being... is worth the benefits that are actually being realized through that strategy. So we've talked about this new way of thinking about architectural governance are relying on automation, relying on establishing business metrics, and measuring how we are doing against those metrics. Are we achieving the outcomes? Having it be more vision and principle based. So what does this mean for an organization that wants to adopt this approach to governance? What kinds of impacts are they going to see in their organization in moving to this strategy for architectural governance?

Jonny LeRoy:

So there was two different parts of the question. But one part was what approaches and how to get started. And this is what you need to be clear about where you are as an organization, where you're starting from. Because the direction you head in is going to be very different depending on where you're coming from. And of the clients in the organizations we work with, they fall into multiple camps. But there's two clear distinctions. There's older, more traditional companies who have been around for quite a while and have tended to build up a fair amount of fairly restrictive governance. And a large amount of what they need to do is find ways to safely unwind some of those restrictions and move them into automation, and into visions, and moving people into collaborators and so on. But they're really loosening up.

Jonny LeRoy:

Whereas there's also another set of organizations that are younger and newer and faster moving. And they're gradually beginning to grow up and so they've been moving super fast. Maybe they're a late stage startup and they've hit the point where they're suddenly realizing that their risks are getting out of control or their quality's suffering or their ability to move quickly as suffering as they grow. And the type of direction they need to move in is actually to add a little bit more discipline and rigor. And so you're actually moving in two different directions. So you've got to understand where you're starting from. And then also in understanding where you're starting from, you've really got to understand what your business strategy is in the area you're playing in and what level of regulatory oversight you have.

Jonny LeRoy:

But I think at least understanding where you're starting from is a key piece. And then moving back from your business strategy is trying to work into what are the visions and principles that you think are important for you as an organization? What are the defining factors of your technology organization that's driving your business? And how do you want to frame that? Are some of that drives out of your strategy. Is your goal to be as fast as possible and maybe tolerant of certain types of risks? So that might push a certain level of vision. Or is quality really the most important factor? Or is ability to experiment quickly more important? Or ability to be a good ecosystem player? And so you can shape the vision you're putting in place from that. And then hopefully the style of how those get cascaded down to the teams will flow from the types of decisions you're making.

Zhamak Dehghani:

One organizational change that I have seen being implemented in some places that I quite like and find interesting is this idea of inverted organization. So as Johnny mentioned, there are these two ends of the spectrum, traditional top down organizations. They need to make a shift. And these kind of chaotic no governance organizations also need to make this shift somewhere between. And this idea of inverted organization really challenges that traditional visualization that chief architects or this form that govern the architecture sit right at the top, make all of the decisions and then those decisions get enforced and cascaded and goes down. And the idea is the opposite of that, that you have a very small group of people sitting at the bottom and enabling the wider teams and all these teams that are running around and making their local decisions.

Zhamak Dehghani:

And by enabling them you are not making a lot of decisions. Maybe where there is a decision that impacts everybody you get involved or if there is a conflict between multiple teams you get involved. But you're playing a very supportive role at the bottom of the organization if you want to visualize it. And then for that model to work, there should be some lightweight tools where the communication can flow and information can flow easily from what I see at the top, which are all the teams that are implementing the products and at the bottom the architects that are influencing and governing the organizational governance and there are tools that they can use for communication.

Jonny LeRoy:

Yeah. And just riffing on that. I was working recently with a west coast retailer who view their organization in that same way as that inverted pyramid. So the org chart is drawn upside down. And when VPs or directors are introducing themselves, they will say, "I support this team. I support that team or this group or that group." And it's a nice mental shift of actually your role. I think in terms of where to get started or how to get started. I certainly understand where you're coming from. But there are definitely some key pieces. And one of the first major planks is really leaning heavily on automation across the board.

Jonny LeRoy:

And there's this big trend of moving as many things as possible to become as code, whether it's configurations or whether it's infrastructure or code as code driving automation across everything you're doing, and driving automation and as many governance and compliance checks as possible. There's a big new growth in what you can do in terms of security bill time in terms of static analysis or the software supply chain of your dependencies or runtime analysis of detecting anomalies that are happening or blocking unauthorized traffic across your networks. So starting with really strong automation is a bedrock foundation and that should then start relieving some of the pressure to allow your gatekeepers to start becoming collaborators.

Jonny LeRoy:

Another key technique is beginning to find lightweight ways to create cross team communication. Because we often see teams that have ended up operating in silos or whether it teams or organizational units. And that can either be as a result of very siloed organizational structures or it can be that you've given teams a lot of autonomy to go and run. And so they don't have much visibility into what's happening. And so finding a lightweight way to share what they're experimenting with, what they're learning is really important. One way that we've had some good success is using the idea of the technology radar as a way to share what teams are having success with and what they've suffered failures with. She had that in a lightweight way. There's nice ways to facilitate that sharing. It's a good way for technologists to talk to each other. But it's also a good way to expose to other interested parties what's actually going on.

Jonny LeRoy:

And then there's a couple of things you can layer on top of that. If you have a little bit of discipline about the process for moving something from assess to trial, or from trial to adopt and what it actually means to be ready to move. You can use that as a signaling factor to other teams that, "Yes we've assessed UJS. We think it's good in these contexts. We're still nervous about these. But you should feel good about now trialing it." That's a good signaling mechanism so you get good organizational learning. And then one extra trick that a company we were working with in Australia used quite well was having WIP limits, so work in progress limits on how many items you can have in assess. So you can only be assessing four JavaScript build tools at any one time as an organization.

Jonny LeRoy:

And if you want to assess the fifth one that comes along, you have to wait till you're finished assessing number four. And you've come to a decision of, "Yes. Let's proceed to trial. Or no, let's move that back out to hold." And that's just a nice lightweight way to manage the amount of experimentation and churn you're having in the organization. And that's the kind of control that as doing architectural governance you should be able to play with. You can say, "Are we experimenting enough?" Or are you experimenting too much and you should be able to turn that dial. And using this sort of regular cadence, maybe quarterly of a tech radar approach is a nice way to be able to play with those dials and see the effects that you're having.

Zhamak Dehghani:

Yeah. I want to add to the idea of using Tech Radar as a way of communicating the technology decisions that teams are making. I think it's a great tool for communication but also it's a great tool to put the right level of governance in place. So technology radar gives you a framework to introduce new technology that you want to assess or technology that you've assessed and you want to take maybe to a few applications, or you want to adopt more widely across your organization. And what governance can put in place are some checks or some gates or conditions to move technology from ring to ring.

Zhamak Dehghani:

What it takes in your organization to introduce a new technology to just assess, maybe it only needs a passionate developer who wants to try it out. And that's good enough. What it means for that passionate developer to take what he's tested with to trial, and that might be that we have actually tested this, used it and took it to production in a couple of places. And from there, what it means to use it more widely? Having the right level of documentation, having the right level of testing and developer experience support for developers to come and use it more widely. So I think it's a very nice tool to put a lightweight governance to not only communicate but also guide how we adopt or retire even technology.

Jonny LeRoy:

So one area we haven't touched on is this idea of the paved road. And I think the concept, the term came out of Netflix. But the idea there is if you're setting an architectural vision and certain boundaries and constraints. As an organization, you can put some investment into building a paved road, which is the easy path to be successful so that if you choose to take the paved road, there's a lot of benefits you get. So maybe it's service templates or certain frameworks that you can plug into. And this is a good way to redirect architectural oversight into actually supportive tooling.

Jonny LeRoy:

Well, what you're not saying is it is mandated that thou shall use the paved road. It's just if you use the paved road, you can use lighter footwear. Whereas if you've got to go off the paved road, there's more risks and responsibilities that come with that. So you have to do more work to meet some of the goals and constraints that have been put in place by the organization. But sometimes that's the right thing to do if your context of what you're trying to achieve is different. So putting more time into building paved roads, supporting tooling and infrastructure rather than longer and longer documents telling you what you can and can't do is often a really good investment.

Rebecca Parsons:

Great. Well, Zhamak, Johnny, Mike, thank you so much. Hopefully we've convinced our listeners that architectural governance isn't this dry fusty old topic, but there's actually some interesting ideas, interesting innovations that are happening in it.

Mike Mason:

So thank you to Johnny and Zhamak, our guests today. Thank you to Rebecca. And my name's Mike. And we hope you tune in again next week.