ºÚÁÏÃÅ

Podcast Transcript

Alexey Boas:

Hello and welcome to ºÚÁÏÃÅ Podcast. I'm Alexey Boas, I'm the Head of Tech for Brazil and I'm going to be one of your hosts this time, together with Zhamak Dehghani. Hello Zhamak.

Zhamak Dehghani:

Hi.

Alexey Boas:

And this time we're together with Luiz Ribeiro and Renan Martins. So Luiz, could you please introduce yourself?

Luiz Ribeiro:

Yeah. Hello everyone, my name is Luiz Ribeiro, I am a Lead Consultant in ºÚÁÏÃÅ. I am based in Brazil. I have been working with ºÚÁÏÃÅ for about eight years now and I am very excited to be here talking more about our experience.

Alexey Boas:

Cool, thanks. And Renan, could you tell us a little bit about who you are?

Renan Martins:

Absolutely. Hi everyone, my name is Renan Martins. I'm also a Lead Consultant here at ºÚÁÏÃÅ. I've been here for almost eight years as well and I'm currently based in San Francisco, California. Thanks for having me, I'm really excited to be here.

Alexey Boas:

Great, thank you for joining us.

Zhamak Dehghani:

Good. It seems like we're recording another multi-continent podcast.

Alexey Boas:

Yeah, that's true.

Zhamak Dehghani:

Because I'm in north of San Francisco, Renan Martins is in San Francisco right now, where's everybody else?

Alexey Boas:

I'm in San Paolo at the moment. And Luiz Ribeiro?

Luiz Ribeiro:

I am in Portalegre, in the Southern Brazil.

Alexey Boas:

And so, we're here this time to talk about evolving platforms and about how the technical or architectural landscape changes when we try to do that. And I must confess, I'm very excited to talk about that.

Alexey Boas:

Here at ºÚÁÏÃÅ we've been talking about how platforms have been rising and platforms and the consequences for the organization have been a theme for tech raisers in a couple of past editions. So I'm quite excited to be able to talk to you about that.

Alexey Boas:

So Luiz Ribeiro and Renan Martins, you've been working together on a client of ours, right? And I know. So we're a consultancy company so many times we're not allowed to talk about many details about who the client is, but I'm sure we'll be able to exchange lots of good ideas of what happened at this particular client's. Could you share a little bit of context of what was happening, how was the overall scenario when you got there and some of the challenges?

Zhamak Dehghani:

You know, and I wonder when you started if there was such a concept as a platform or it kind of emerged along the way. So it would be nice to see that evolution and that perspective.

Renan Martins:

Okay. Well, this client like many big enterprises had a digital department under marketing and had a CDO and they were mainly responsible for the digital interactions with the customers, including the e-commerce website and the mobile applications. This is a very common scenario in big enterprises. They had to move quicker than the overall enterprise and they almost operated as a separate company.

Renan Martins:

And we were brought in into this digital organization to help them be more nimble. And back then, there was this concept of modernizing the entire ecosystem and platform thinking helped us on the building blocks to enable them to develop new experiences with their customers.

Renan Martins:

So a little bit of context. They had a 10 year old monolith written in Pearl, that served most of the web experience. And they also had Legacy mobile applications that's scrapping the website. So as you can imagine, the cost of maintaining and evolving this products were extremely high and they were in a road where new channels were showing up every day to interact with their customers, and they had to change that.

Renan Martins:

There was this new CDO, the Chief Digital Officer. He was very courageous and he was trying to implement a set of modernization efforts across the enterprise. And he partnered with us to implement this changes. And they included like, modernizing the technology stack, break up the monolith, move to the cloud and building high performing product teams. So these are already operated years, however, they usually lead to a tangle mass of Legacy technology processes and culture if not done properly.

Renan Martins:

So I think today we can explore a little bit of how we helped them go about these investments and iteratively achieve a greater value to their customers.

Zhamak Dehghani:

Yeah, it's really interesting, the pattern that you mentioned. Over and over we come across it. Whether it's a startup within a bigger organization, like it seems like the digital part of the organization within a bigger enterprise, or is it a startup itself that they want to move quickly that when it gets something to the hand of their customers, but then you build a Pearl application or Rails application, or whatever it is that turns into a monolith that can't sustain itself. And a lot of companies just get stuck there. They don't know how to go through the phase of scaling to be able to then sustain that rapid change as well as sustain the growth of the company itself.

Zhamak Dehghani:

So it's a really interesting point in time to make that pivot. And it would be interesting to talk about the things they started doing to make that pivot and change to a more sustainable, scalable solution.

Alexey Boas:

Yeah, and it's also very interesting that you've mentioned the role of the CDO in that. Because that situation is going to lead to a lack of innovation and difficulties in evolving the customer experience and be able to develop new applications, new experiences and on top of the existing infrastructure. So it's very interesting to hear about the role that a CDO had on that.

Alexey Boas:

And I once heard you say that, development of the infrastructure is kind of foundational to all that. So is that so? And sometimes we, when we talk about delivering infrastructure we see in some companies an architecture silo trying to drive things from their perspective and not taking into account developer experience. So here in ºÚÁÏÃÅ we've been talking about developer experience as something very key and is a key differentiator. So how did that come in this particular situation that you were facing?

Renan Martins:

Well, that is a very good point. I think the CDO realized they had to move quicker and starting with a delivery infrastructure platform that allowed them to move quicker, developers to get production ready, software in production working is key, it's foundational. And there was the first, very big investments they made. And you mentioned something very important, it's an anti-pattern if you try to do that in a siloed way, in a big bang faction. So what you want to do instead, is to iteratively create this platform, adding the value to the developers as you go. So instead of having that mentality of, I will build and they will come, they will be coming as you build it, basically. So you need to understand what is the low hanging fruits that you can work on to remove the friction. Because in the end of the day, a platform main goal is to reduce friction to get something done to add value to the customers.

Renan Martins:

So here we're talking about the delivery infrastructure of platform. That includes products such as, your continuous delivery pipeline, how you get software into production and how you test it. We're also talking about products around observability. Once they are in production, how do you make sure you can monitor them? They can tell you, give you insights about what's going on as a whole in the entire ecosystem.

Renan Martins:

You are also talking about reliability. You need to be able to trust the neighbors in this entire platform. So SLAs and things like that, that are built-in in the platform. Those were very important first steps that this client of ours invested on and they quickly started seeing teams using the products and starting push code to production much, much faster, in a matter of days. So we would create a GitHub repo for example, and push it all the way to production, having all these different components that I mentioned, almost out of the box for the teams.

Renan Martins:

And they achieved that first because they focused on building the platform iteratively and also because they focused on the developer experience as a very important component of this platform. So assuming that the developer is your customer, when you were talking about building products on a platform, is extremely important. Because if you were thinking about the developer as your customers, you're going to create the necessary empathy to bring them to the developing process and think about their feedback. And similarly to what you do to build like customer facing products, you do internally, building these products to your developers.

Luiz Ribeiro:

Now, one thing that is interesting is when you mentioned silos. When we were brought in, we saw that the CDO had this desire of having greater time to market and more innovation. And some of the IT folks, they were already working on extracting some of the business logic from the monolith into microservices. And another area connected to IT, they also had people working on tools for the developers, but they were working on a very siloed way and they were also in this big bang faction that Renan Martins mentioned, right? Like, let's build everything and then we're going to bring these to the teams. So a lot of the conversations that we had with them was, hey, you really need to think of these as a product that we're building for the developers.

Luiz Ribeiro:

So just like when you're building a product for your final customer, you want to get their feedback, you want to really understand the way that they're using, you want to do analytics to improve upon your products. With this feedback, then you need to apply the same techniques when you're building these products for our developers. And then we start building this more iteratively and then we started seeing these results that Renan Martins mentioned.

Luiz Ribeiro:

And the same thing applies to this services we're building, right? Like building them as products, as a platform, with this platform mindset was a key also in having better results.

Zhamak Dehghani:

You mentioned quite a key, I think point, it's worth iterating. And maybe going to the next level of details, you mentioned iteratively build a platform and have the early features as they come out be consumed and used by developers. You mentioned platform as a product, so build these capabilities with a product mindset. So you put your developers as a customer at the center of what you build and give them the best experience. You also mentioned a kind of a change of approach. Before we were there, it was like in a silo, let's build it and then put it out there, and then we kind of changed that slightly and we started building more iteratively as you mentioned.

Zhamak Dehghani:

What I want to understand is when you started building it as a product and iteratively, I want to see how we bootstrap that. Because, I've seen multiple methods of bootstrapping the process itself, or starting the process. I've seen the methods of the people who are building the delivery infrastructure, they're part of the customer teams to start with. And let's say they are building the first pipelines, or you know, whatever the scripting for your pipeline you need for them and with them, and then extract that back into the platform. So this idea of harvesting capabilities back into your product but build the first ones kind of custom for your first one or two customers.

Zhamak Dehghani:

And I've seen kind of the other way where you are anticipating what the capabilities they need or how they would need it and then with conversations, of course they're involved, but you're building it independently. And I wonder which approach they took first, and maybe how to bootstrap this kind of platform as product building and where they went next with it. And if you can talk about like the team structure and how the teams interacted. You know, these are the details that I think the listeners might appreciate, the ones who want to go through on this journey.

Luiz Ribeiro:

Yeah. So when we started working with them, this was being done in a very siloed way, as we said. And the communication between these people who were building these tools and the actual product teams was very low. And I think that the pattern they were using at the time was more, let's try to think of what these teams are going to need and what are the tools that we think the company should be using, and then push this things onto the teams.

Luiz Ribeiro:

So when we got there, we were like, hey, maybe these things that we're working on right now is not necessarily what they think you need the most, so maybe we need to bridge this communication better and close this gap. And then we had a lot of sessions to try to understand what were the main pain points that the teams are feeling and also as you said, bringing in some of the people who were working on this product teams that had the capabilities and have them integrate some of these platform teams. I think that in the end what we had was kind of like a mix of both, where we were both working on some of the tools that are going to address some of the main pain points that the teams have, but at the same time we also need to be looking forward, right? What are the things that we believe that the teams should be paying attention to and how can we try to make it easier for the teams to be paying attention to these things?

Alexey Boas:

Is it fair to say that, and Renan Martins talked about that. So taking a developer experience approach, you're going to let the needs of the developers drive the things that you need to do. And its analogous approach to what we do, when we do agile software development. And then we see the customer needs binding different skills and different backgrounds of people, and people who were previously working on different teams but that common goal and that common feedback cycle of the things that we're trying to, the customer problems that we're trying to solve and the feedback that we get from delivering things into their hands. We see that binding together and getting across silos.

Alexey Boas:

So is it fair to say that, just creating that cycle of solving developers pains and seeing how we're helping them creates a strong force to get people working closer together? Is that what you saw happening?

Luiz Ribeiro:

Definitely. I think that this is the most important thing of all. It's like treating the development of this platform as a product and then having a closer relationship with your users, right? Getting their feedback and understanding how you can bring more value to them, I think that's the main point here.

Renan Martins:

Yeah. And what ultimately happened there is that, the CDO realized the importance of that department, right, who was taking care of the platform, that would allow his product teams to be faster. He realized the importance of that team and brought into the same team's structure that the other product teams followed. So it was official now that the platform was actually a product, it had a product ownership, product management capabilities. And because of that, of course, you need to take care of your customers, in this case the developers.

Renan Martins:

So everyone was now part of the same organization and they had metrics, and the same goal, the same mission, so the collaboration increased exponentially after that move. And I think that was crucial to their later success they had on this.

Alexey Boas:

Could you, and I am personally very interested in how these things impact the overall organizational structure in general. So could you tell a little bit more about that. What changes were necessary as far as team structure's is concerned for example, and how did communication change? How did all that impact the culture in the organization? So could you tell us a little bit about the organizational changes that were needed and that were caused by all this process.

Luiz Ribeiro:

So I think that what Renan Martins mentioned was something that was a very strategic move, bringing the platform teams to the same level as all of the other product teams, was something that, it was a very strong communication best to everybody involved in the organization, right? And then holding these teams also to the same level of standards of like KPIs, and metrics and goals, shared goals between all of the teams including the platform teams, I think was one of the way there is the impact of reorganizing your company, your team's structure in the way that the communication flows.

Renan Martins:

Yeah. Luiz Ribeiro mentioned something very important. The KPIs or how do you measure and how do you see success on each individual product line, let's put this way. I think the team for example, responsible for the shopping cart in this context, right? It was e-commerce team. They were definitely concerned about conversion and things like that, whereas the delivery infrastructure platform team, they were concerned about the developer experience, the Net Promoter Score of developers within the organization, the adoption of their products. So they had their own KPIs and they were sitting at the same table when they were taking the organizational decisions. I think having a seat at the table is extremely important and try to minimize the layers to deliver value.

Renan Martins:

So they were a very agile organization in this sense, not with the capital way, they were really nimble because everyone was on the same table, they would prioritize the low hanging fruits to get something done and delivered in production quickly, that would add the overall value. And I think the move of getting all of them together and divide your organization that way, was very important.

Zhamak Dehghani:

Renan Martins you mentioned a couple of really interesting kind of metrics. It's always kind of easier if your client facing, business facing, it's easier to come up with the metrics of success because it directly or indirectly links to your bottom line and/or organizational needs or business values. But when you are in the underlying platform, delivery infrastructure and want to treat that as a product is harder to come up with indicators of success. And you mentioned Net Promoter Score and maybe you can describe what it is and how they were measuring it? A little bit of more details around that. And if there are also other metrics that they were using for success, and if you could call them out, that would be great.

Renan Martins:

Yeah, absolutely. I think the CDO was a very tech savvy kind of person and he was aware that continuous delivery and basically the triple the amount of times you'd apply to production is actually related to the overall performance of the organization. There are actually academic research showing that you can find more information on the State of the DevOps report presented by Puppet and Andorra. So he was like, okay, I need to get adoption basically. Main goal of this platform for now is to get as many users as I possibly can within the organization. So let's make it very simple to use, extremely straightforward, a new developer, a new team could quickly get software working in production. And then the NPs becomes a no brainer kind of metric they need to track.

Renan Martins:

They used to track that using surveys internally. We helped them with a couple of continuous delivery assessment maturity reports where we have our own methodology where we help assessing the maturity in terms of continuous delivery and the engineering practices that you need to have to achieve that successfully. It's not a simple after production that will give you that. You really need to have the maturity to reduce risk while delivering the software. Future Cargo, Skaneri, Little Green and many other techniques like treating the entire infrastructure S code. And so many other things that the industry at this point is accustomed to. So he was concerned about all these things and measuring this internally on the delivery infrastructure platform.

Renan Martins:

I think they reached a point where they were actually able to deliver software quickly into production, but there was not necessarily everything they needed to quickly develop new experience to their customers. That's when he and his team realized they had to think about platform thinking not only on the delivery infrastructure side, but also on the API's, the capabilities of the business. How can I expose them in a developer friendly way, business truthful way that my other product owners can have ideas and quickly implement them and putting the hand of the customers. I think that's when the second big set of investments started happening, of building this platform of APIs now.

Renan Martins:

So Luiz Ribeiro already mentioned that, but he started treating the APIs as products within the organization and that had many implications, not only on the technology but also on the organizational design.

Renan Martins:

Yeah, I think I deviated a little bit from your question. We were talking about KPI.

Zhamak Dehghani:

No, this is actually a great segue I think to then jump up a layer. But no, that makes total sense. And sometimes as you said, like the techniques are as simple as, send a survey out and collect that information. And sometimes you build those metrics into the products and infrastructure that you build to measure things, and you mentioned The State of DevOps and the metrics that will really survey your change fail ratio, your lead time to delivery, the frequency of the Pushover action. So it is a spectrum, right? It's a spectrum of metrics and a spectrum of tooling that you use. But I think it's a great segue to the next section, or the next layer of this platform, which is as you mentioned your business capabilities. Now services and APIs that delivers the business.

Zhamak Dehghani:

Shall we talk about maybe some of the first successes and also challenges that you both experienced, probably participating in different parts of this platform. And from your point of view, what were the successes and challenges?

Luiz Ribeiro:

Can I just go back a little bit to the metrics part?

Zhamak Dehghani:

Yes, it's exciting part. Let's go.

Luiz Ribeiro:

Yeah, because there's an example that I think is something that people wouldn't think of, but in this scenario was an actual metric that we used. One of the major pain points that we saw, that many of the teams had was doing database migrations. Like it took a long time for them to do that because they needed to have touch points with a central team of DBAs and all the process was very manual. So every time they needed to do some change in production that involved database migrations, it was something that would take them days to do. After identifying this pain points, the platform team started working on automating this database migration process. And the time reduced to like, I don't know, a few minutes, from many days in a week to a few minutes, to do this process. Which meant that now this team was doing many more deploys to production than they were doing before.

Luiz Ribeiro:

So some of the metrics that some of these veteran teams had, was not only how many times teams are pushing changes to production, but also like how much time it's taking the teams to actually do this, how much time it's taking the teams to start a new project, how much time it's taking the teams to spin up a new node for their systems. So all of these metrics, they're like more fine grained but they were also very useful for us to look at and take insights of how... Kind of like analytics for the platform teams, right? How are users using the platform? And which insights can we take from these usage patterns? And what can we do to improve on that?

Zhamak Dehghani:

Yeah, absolutely. Because when you think about the value stream of, I have an idea, or if there's a feature I want to implement, or I'm asked to implement to, the feature actually got implemented and pushed to production. There were all these steps, and sometimes people don't have visibility to all of these steps and they'll think about it as one, there is really one metric. But then when you drill in into that value stream and find out those bottle necks and the low hanging fruits to kind of attack, it gives you visibility as where the problems are and where the high value and possibly low complexity wants to pick up and to start with. This is interesting.

Renan Martins:

Absolutely. I think the Value Stream Mapping is something that they recurrently did to actually figure out the low hanging fruits I mentioned before. And friction point would show up in a Value Stream Mapping and they would try to either automate the process or remove it completely if it was not truly necessary in the value stream. So that is a actual very powerful tool to be used all the time.

Zhamak Dehghani:

Yeah. It's an example of that other client that I'm in, the development teams always say, "We're really fast, we're really fast. We deliver fast." And then realized they have two weeks od system integration testing, three of user acceptance testing, so what developer things they're delivering really fast, the end customer doesn't feel that or the business because the whole value stream is much longer. And when you trace back where the change needs to happen for the whole value stream to be optimized, it goes back to how developers develop features. You know, having feature flags and continuous delivery and so on. So it's interesting.

Luiz Ribeiro:

So talking about the actual business platform, I think that, I don't know, I would say from the top of my head, I think of two main challenges that we had working on that, and I think metrics was one of them. As you already mentioned Zhamak Dehghani, if you have a team working on a platform the set of APIs they're exposing the business capabilities, for other teams to build customer facing products [compound 00:30:25], these API teams they're not customer facing, right? So how can they measure the success of the work they're doing? How can they feel like they're accomplishing something? And I think that having clear metrics to these teams, and again, that product mindset was one of the issues that we faced in the beginning. Where some of these teams they were working, or many of these teams they're working on these APIs but sometimes they really didn't have a clear understanding of why these APIs were being built, who were using these APIs and for what. So I think that was one of the problems.

Luiz Ribeiro:

And I think that is connected to the second one, which is the organization structure. We had in that scenario two separate structures where one of them was the set of teams building the customer facing implications and then the second structure was the one during these APIs. And if you think Conway's law, where the architecture we're going to build is going to follow the communication patterns, and you have this very two different structures inside of your organization, the communication between the teams in these different organization they didn't flow very well. So that added to this feeling of the teams are like, why I'm I building this? What is the purpose of that?

Renan Martins:

Yeah. At the same time the front end teams, like the teams building the customer facing products, were feeling frustrated because they were of course limited by the ability of the API teams to deliver the capabilities that they needed. The API teams, they were also frustrated because they were not necessarily directly linked to the value being added to the end customer. So that generated a lot of tension within the organization, the value of the two different product owners, there's a value that the teams are bringing. So at that point, we did a couple of experiments. We actually verticalized some teams and knowing that, that could potentially lead to a more monolith application or it would lead for example on business rules on different places and the architectural boundaries not being preserved the way we initially thought in favor of time to market.

Renan Martins:

Though I think what the listeners need to understand is, there is always a trade-off. There is the value of having your platform of APIs, exposing the capabilities in a agnostic to the experience that they will compose. That of course will at first reduce your ability to quickly deliver something, but later on, that will entangle the enterprise and allow you to create things really quickly. As quickly as, you go to the developer portal, you find the APIs that you need, they're developer friendly as I mentioned, they were developed with lean product management practices in mind, closer to the developers. You would quickly create a new experience by just consuming those APIs. And if you on the other hand just focus on the entire vertical thing, you might end up creating a monolith, which is what you're trying to run away from in the first place.

Renan Martins:

So there is always a trade-off. Understanding that is the crucial part but to Luiz Ribeiro's point, we lived the tensions that exist when you try to do that. And I don't think we have all the answers, but it's definitely a challenge to manage it.

Luiz Ribeiro:

And I think that was a very strong learning experience for us as well. Because when we started working with them, and we saw that they already had this structure that had this objective of building this API platform, and we started seeing these pains that these teams were feeling, then we started to talk with them about, hey, maybe we can compromise. Maybe we can look like verticalizing some of these work. And then they were like, hey, but you were telling me about Conway's law, and if you do that, isn't it going to be so we're not going to achieve the architecture that we want, where we have this business platform. And I think it connects to one thing that we said in the beginning about not following the anti-pattern of building this platform siloed way, as a big bang, right?

Luiz Ribeiro:

If you're building this business platform of APIs with this same idea, then there's going to be many pitfalls there. So maybe sometimes you do need to compromise, maybe sometimes you do need to start working like, hey, let's deliver value with this first. And then we keep this in mind, that we want to have this platform and we're going to deliver value, we're going to work together, we're going to collaborate, paying attention to the architecture, paying attention to the fact that we want to have this platform as a deliverable of our work as well. And then in the future, we can maybe have more clear KPIs and metrics for both products separately, for your API platform and for your front end applications.

Luiz Ribeiro:

So it really depends on where you are on your journey in creating this platform, I think. That was a strong learning for us as well. I don't know Renan Martins, what do you think about that?

Renan Martins:

Yeah, I totally agree. And I think one important thing when you are compromising and making trade-offs, which happens all the time, is to make sure that the teams, they have the necessary information to understand that they are making a trade-off and that they are maybe creating a technical debt which per se is not a bad thing, it's something that you need to actually, actively manage. They need to know that. I think we had a lot of good learnings on this because at the higher level, we had a technical vision. People understood that there would be a platform of APIs, more like a foundational set of services, experienced services and also BFFs and you know, the front end applications, everything be coupled.

Renan Martins:

And if that was not very clear for every team, they would not understand that by not relying on a foundational or implementing something that should be on a foundational service in the experienced layer, they would be going against that vision but intentionally just for the time being and then extracting that into a foundational service.

Renan Martins:

I think one last one that we learned was that, you should be actively communicating that vision to every team. Ensure that they truly understand so they can understand the trade-offs that they're making.

Zhamak Dehghani:

I think, yeah. I think that point about explicitly articulating the integrity of the architecture and be able to kind of almost measure it as you make trade-offs and to see whether you're going away from it, maybe temporarily and it's acceptable or going closer to it. I guess that's the premise of evolutionary architecture. I mean, the book that Rebecca Parsons and Neal Ford and Patrick Kua wrote, that is actually what they are suggesting is that, organizations, always architectures try to optimize for some across different dimensions. And here I think that your client, they were optimizing for speed of delivery. At some point optimizing for perhaps utilizing core capabilities through shared APIs. And if those are the factors that articulates the vision, how can we go one step further and articulate that vision as the book puts forward, the fitness function of certain parameters that we would actually capture. Setting metrics that we would capture and at different points in time, we see how are we doing in terms of moving towards or moving away from this vision.

Zhamak Dehghani:

So it's not something fuzzy and nice to talk about, but we can actually measure it. And I think in the next few years, it's another frontier towards evolutionary architecture, towards distributed systems. There are architectures that are difficult to kind of harness, but we can actually formulate and measure the integrity as we change.

Luiz Ribeiro:

That's an excellent point. Actually, the main motivator that they had at this client, when they started working on these APIs was under connecture motivator. They saw architecture/organizational, because they had these digital organizational work and client facing applications and then they had IT, right? That was like very traditional working with standard, traditional, conservative ways of developing and delivering software. And they looked at it and they said, "Hey, we can't continue having this two separate organizations. So what we're going to do is we're going to have another set of teams that are going to be building these set of APIs, business APIs and these APIs are going to be the source of truth for our business and it's going to be used by both the digital organization and by IT."

Luiz Ribeiro:

So that was the main motivator, like very architectural, very organization. And then what we were telling them is that, yes, there is like the correct move, we need to think about building a platform of APIs with the business capabilities, but first we need to find our clients. Who are our clients? They're going to be using these APIs and then build these APIs together with our clients. Getting their feedback, understanding that these APIs are actually serving their purpose and move on from there, right? So again, that product mindset, and that was part of the journey that we learned working with them.

Alexey Boas:

Yeah, I find it very impressing how powerful that product mindset can be in getting alignment. And then it brings to the table of the evolutionary architectures' technique Zhamak Dehghani just mentioned. So you get driven by the things you want to deliver to the customer and that cascades down to the things that you need to do in several parts of your technical landscape and ecosystem. And that drives alignment and that drives something that you can monitor and at the same time be mindful about the trade-off that you're making, and change your mind as time evolves. I find that really powerful and goes back to the value chain topic we're talking about and really bringing value and making things happen to the end customer.

Renan Martins:

Yeah. I personally love the concept of a fitness function. I would definitely had implemented some in this case to prevent some of the things that happened. For example, the integration between back end for front ends and the re-usability of some of the back end for front ends across different channels. I think that happened because to my earlier, previous point, the teams didn't have the vision very clear in their minds so they compromised in a thing that it was almost unacceptable. So a fitness function would have prevented that, not allowing the architecture to evolve into a place we didn't want to basically.

Renan Martins:

So in that example, they learned in the bad way. They basically had a mobile application, they were now consuming these APIs, they were providing the business capabilities. Like I mentioned before, before the APIs, they used to scrape the website and they also had the web products, right? And turns out, one these BFFs that I mentioned, they evolved and did a lot of orchestration across the foundational services. And the other team saw that and saw a potential re-usability of that instead, that is actually a BFF, I should not be talking to that, I should be you know, it's clearly assigned that another service might exist, let's think about the architecture for a moment.

Renan Martins:

They didn't do that of course. There was a Time to Market issue, so they re-used that. What happened later was a problem in a one web application turned down the mobile application. So they were supposed to be separate. And that happened because the teams building the BFFs, they were not concerned by the same things that the foundational API teams were. They were not thinking about throttling, multiple customers, reliability and corrective-ness. BFFs are usually services that you optimize for the application that you serve and that team was not even aware that another team was calling their service. So for me that was a very important lesson learned, where there are certain architectural changes that you really need to avoid. And being very explicit about not allowing that even networking segregation or fitness functions could be potential ways of preventing that.

Renan Martins:

Because it's not everyone, all the time who have the vision of the architecture in their heads. So we really need to prevent that someway.

Zhamak Dehghani:

Yeah. You mentioned BFF a few times and I know we have heard it on our tech radar, but for the listeners that may not be familiar with it, can you describe what it is and perhaps an example of a capability or an application that was implemented by BFF?

Renan Martins:

Absolutely. A BFF stands for Backend for Front end. There're other terms as well in the industry but they basically are services, server side, that serve one specific channel or application and their job is to optimize for that specific channel application. So let's say you have a BFF for the mobile app, that BFF has one concern, which is for example, [inaudible 00:46:24]. It wants to group the entire orchestration of services into a single payload so that your mobile application doesn't send multiple requests to the internet. Because that example, you are concerned about internet connecture or bandwidth, or whatnot. You also want to optimize the payload size, so you would put some compression, and that's something you'd usually do in that BFF.

Renan Martins:

And that is just an example of a service that optimize for a single channel. And they usually are developed together with that application by a team focusing on building that application and not by a foundational services team. Like I mentioned, usually cares about multiple customers and because of that they need to think about throttling, API management kind of concerns, APKs and things like that. So that's just an example.

Alexey Boas:

Well, people usually say that big challenges bring big learnings and that was definitely the case with all of this. So thanks so much for sharing.

Alexey Boas:

From a personal perspective, what do you feel was something that this client's challenge brought to you as a key learning or a nice, different experience, or a challenge that you were working?

Luiz Ribeiro:

In my case I would say that, that little change is not about technology. I mean, it does involve technology but it's much more about people and the organization structure and how that encourages people to communicate with each other. So I mentioned Conway's law a little bit and Inverse Conway's Maneuver, and I think that was, especially talking about the business platform of APIs, that was a big part of it. How finding the best organization structure so that things could communicate, aligned their roadmaps better to deliver value to their customers to feel like they are achieving something. Working together with managers to talk about this things, to bring up this concerns about how we look at people, how they're working, making their daily jobs better, more fulfilling and encouraging this communication between one another, I think that was a very strong learning for me.

Renan Martins:

Yeah. I share the same learning and one thing for me personally is that I've always been on the developers side trying to have autonomy, more responsibility, I want to take care of the entire life cycle of my product. I want to be able to push things to production as quickly as possible. And I think this is all great and we should always try to achieve that. However, I learned the importance of standards and proper governance when trying to do that at scale. For example, at this client, they adopt Docker powered platform as a service. You cannot get more autonomy than that because with Docker, you can basically deploy everything, right? So this ultimately led to a very heterogeneous set of systems that they need to work together.

Renan Martins:

So we were talking about microservices here. And they of course do not work by themselves, they're part of a chain of interactions. And if you think about a platform where you need to trust your neighbors, you can only achieve that, that lab or SLA if you hold them to a set of standards and guidelines around operations, security, after production, infrastructure and everything. Otherwise, you have chaos and chaos is dangerous. So you have the developers they are happy, but all of a sudden you cannot make sense of anything and you're not actually adding value and increase complexity extremely fast.

Renan Martins:

I think acknowledging that the standardization is important, doing it right means it should add value to the teams and their products. Not imposing any unnecessary bureaucracy of course, but you should have some standards in this set of platforms. Like different flavors the developers could use to develop the application. You could even use Docker like this client of ours used but have some basic set of standards that would allow them to evolve with the SLAs in place. I think I learned that, I developed a lot of empathy for these kind of things.

Zhamak Dehghani:

I can imagine we would record a whole other podcast, for now we talk about between autonomy and having responsibility and governance and standardization. What is it right for overall, and that's a whole other discussion we could have.

Alexey Boas:

Yeah. I mean, that was a great conversation and lots of learnings. It was amazing. Thank you very much for joining us.

Zhamak Dehghani:

Thank you both. It was awesome.

Luiz Ribeiro:

Thank you for having us.

Renan Martins:

Thank you for having us and for the opportunity.