ºÚÁÏÃÅ

Podcast Transcript

Rebecca Parsons:

Hello everyone, my name is Rebecca Parsons. I'm one of the co-hosts of the ºÚÁÏÃÅ podcast. We will be speaking with Andy Yates, who was the head of strategy for ºÚÁÏÃÅ Internal IT. We call it tech ops, and he will be describing our journey to supporting citizen developers. But first, again, I'm Rebecca Parsons, chief technology officer, and I would like my co-host Zhamak to introduce herself.

Zhamak Dehghani:

Hi everyone, I'm Zhamak Dehghani. I'm joining you from our San Francisco office and I'm acting as the technology portfolio director, and excited to have Andy here to talk about his experience with supporting students and developers today.

Rebecca Parsons:

Well, welcome Andy. Why don't you start out by telling us a little bit of what are we actually trying to accomplish here and how you got involved?

Andy Yates:

Sure. And thanks for having me here. I guess people think of ºÚÁÏÃÅ as a strong developer-led culture, a strong developer-led organization. We have a lot of work in open source. We see lots of common patterns across the clients that we work with and the projects we're working on, and we want to bring some of the lessons that we've learned from the outside world into our own internal ways of working, internal IT. We talk a good talk about having empowered individuals and teams and we walk that walk. We have an attitude of, if you can see a problem, you should be able to kind of go ahead and fix that problem. We want to make sure that our people, whether they're in an internal team or whether they're consultants who are in between project, are able to to help solve some of the problems that we see around our business.

Andy Yates:

We're a pretty sort of value or outcome-driven organization. We're always trying to deliver, and so we are quite focused on moving to solving a problem as fast as we can. If I see a way of fixing something, I'll pull out a credit card and I'll figure out the details later. That's sort of the organization that we have been and have grown up as, and so we wanted to encourage this. What we see though, is that if you have that unconstrained approach of allowing everyone to work on everything, there are problems that show up that come out of that. We've wanted to put a program in place to both enable the good side of that, but at the same time, to deal with some of the things that we see that happen.

Zhamak Dehghani:

And then you mentioned there is a good side and there are challenging side by empowering individuals to solve problems in their own way. Can you talk to some of the challenges?

Andy Yates:

Sure. If you think about sort of a traditional skunkworks, if you think about the way that people approach their IT, it thought of as a bad thing, and that's thought of as a bad thing because it doesn't work in tandem, so people aren't working alongside, necessarily, other people in the business. At the same time, you're often finding that people are solving the same problem over and over again, in very, very different ways. And so there's a lack of alignment around those things. That's not due to a sort of ill will, I think, it's much more a lack of visibility. And so one of the things that we were starting to think about was how can we provide visibility into all the work that's going on, so that people are able to understand what's going on and to contribute to those things, rather than replicating the same thing over and over again.

Andy Yates:

Visibility is a big piece of that. One of the other things that we've seen is that it's very easy to stand up services. It's very easy to build sites, to create new things, but it's also very difficult to have a longterm ownership of those assets that people are creating. And so you find that things have been stood up and then people move on. Particularly in a consulting organization, where folks are working on things between projects, then as they get staffed again, things fall into disrepair and we saw a situation where we had business critical applications that didn't really have ownership and that folks didn't know where to turn, in terms of getting help and support when they needed it.

Rebecca Parsons:

Well, and in fact this bottle has been the source of some of our consulting engagements, where people have stood things up that are very specific for one department, and somebody else stood up something very specific for another department. They're trying to rationalize those and we run into the 80/20 rule, which is, it's easy to do the simple stuff, but it's that trying to reconcile the differences in process between different departments or handling the edge cases. That's where things get complicated, as well.

Andy Yates:

Absolutely. We see that, both in a technical sense and in a business sense. People often forget that when rolling out a new application to solve a problem, it's not just about putting the application in place, but there's a sort of aspect of support and education and training that are, again, not a one off sort of, "Ta-da we've released this thing," but there's an ongoing onboarding of new people, as people join the organization. And so even when solving fairly small, fairly point problems, these considerations need to be taken. We needed to put in place some kind of model that would help bring creative ideas and great ideas that were helping solve real problems that people had, but bring them into a fold in some way where people were able to take on different levels of ownership and to be able to make sure that those downsides didn't occur so much.

Rebecca Parsons:

Tell us, big reveal. What did we do?

Andy Yates:

We set up a program that we called ºÚÁÏÃÅ Labs, and that program is what some folks would think of as a citizen developer program. A program specifically in place to allow anybody in the organization who has an itch to scratch, has a thing that they want to build, to be able to do so in a way that helps sort of foster that innovation, that creativity, but to do so that's in a way that's in line with a certain set of rules or guidelines. Certainly in this organization, much more guideline than rule, but certain guidelines and certain environments and architectures that we can provide, to allow us to help manage applications through that process.

Andy Yates:

Specifically what that is, is an offering in terms of infrastructure. As a person who has a project that they want to start working on, we will provide them with a place to host their code, a way to build that into an application, and a place to host the application. And so we did a little bit of work looking at different ways of doing that, different tools that we can use for doing that, and for our initial release of the Labs program, we settled on three things. We settled on GitHub for code hosting, we settled on CircleCI for building that code and Heroku for hosting the applications.

Zhamak Dehghani:

Andy, you mentioned that you set up some guidelines around this infrastructure and how people can use it. Can you talk to us about some of those guidelines?

Andy Yates:

In a sense, that stack that I just said, is a big part of that guideline. People can build things to solve any number of problems, but we request and require them to use that stack. That's been an interesting challenge for us. We often have developers who are wanting to be fairly unconstrained in their choice of tech stack. What we found was that we wanted to be able to make some trade offs, and so we looked at, specifically, the kinds of projects we're talking about, the kinds of things that people are building, and looked to find a stack that would support the vast majority of those, but certainly doesn't support all. And then the guidelines we put in place were beyond that use of stack, where I've really been around a very lightweight process for asking for those resources, where we have certain requirements on ownership. We want named people who sponsor those applications and are named and known to be named as those owners.

Andy Yates:

We have a very lightweight InfoSec review, where we look at what those applications do and we help work with teams to make sure that their applications are secure, and we're on a path where we're starting to open up API's from our internal systems, such that those are the standard ways for people to integrate with those other sort of more centrally owned systems that take-ups themselves manage, so that there are safe ways for people to read that data out of those systems. Certainly at the moment, what we're looking at is the Labs applications, those applications that people are building can read some of our data, but they are not yet writing back into those systems. And by doing that, we're making sure that they are extensions but they're not extensions that are going to sort of massively change our overall application landscape.

Zhamak Dehghani:

One of the challenges that you mentioned earlier was visibility. Giving visibility to what tools are out there and also perhaps what applications and what problems have been solved, what applications have been built, so I can use instead of rebuilding. Are we putting any guidelines or an infrastructure to support that visibility?

Andy Yates:

We've got some infrastructure support, and the program itself, part of that program is around having people whose role it is to guide other colleagues through the process. There's some things we put in place and there's some interesting stories around that, maybe. That stack that we give to people is automated. We have a single application where I can go to the application to request the resources that I'm looking for initially. That would open up a ticket in our ticketing system and then there would be people behind the scenes manually sort of provisioning those things and then replying back. And then we've basically been sort of tightening that loop and automating the various pieces of that. Through that front end of an application, we're then enable to show the other applications that are available already.

Andy Yates:

I can essentially say, "I want to start a new application or I want to join an existing one." Our hope, very much, was that we would see people joining in extending existing applications, rather than creating new versions of the same thing. Interestingly, I think one of the interesting stories around this is around the systems for requesting leave. We have six-and-a-half thousand people around the world in 14 countries, 40 offices, and in a variety of different situations where leave is managed in different ways. We have been pretty clear, from a central IT perspective, that this is a problem that is not worth the investment to solve at a global scale. There are so many complex rules, but the benefits for solving that problem, in one place globally, seem to not really warrant the investment it would take.

Andy Yates:

This seems like a really good place where a Labs application that can be built differently for different places make sense, and so this was one of the first problems that we saw people wanting to solve in various places. What we saw was really interesting. We saw those first couple of applications spring up by region, and this was very much viewed as being a regional problem, so we saw these application spring up by region, and then we started to see a number of clones of those applications. People were taking the application, almost wholesale, and copying it and then changing it. We thought this was kind of really interesting. Why would people not want to just start working on a single code base and add their own requirements in and try and solve both those problems?

Andy Yates:

It's not fully clear to us still why people are wanting to do that copy and change, but it's been a really interesting exercise for us to sort of dig into that and try and understand how that works and how well that supports our business. In doing so, one of the other pieces that we're able to now have or see with that Labs program is we've got some visibility into cost. We're able to go back to the various regions that are sponsoring these efforts and say, "Hey look, you're currently spending this much money on these applications and this is the benefit you're getting. There are these other applications out there. If it make sense to you, you could work together." But we're able to take a sort of fairly hands off approach to that, which has been super successful, because it helps us understand and it helps people think about the cost implications of the architecture and cost implications of building small applications.

Andy Yates:

One of our regions actually decided, when they started building applications, they were looking at microservice architectures and they were looking to use the building of an application, to manage time off requests, as a way to develop capability and to practice using these new architectures. They built using Heroku, but with a microservice architecture, and ended up creating an application that took up somewhere close to half of the dyno allocation that we had for the entire program for the leave of that one particular region.

Andy Yates:

It was, again, a really interesting experience for us. I think previously, outside of that Labs program, that would have been kind of hidden. People would have done it but wouldn't of seen the cost and wouldn't have seen what happened. And so it provided a visibility to us where the people on the program team were able to go back to that region and say, "Okay, cool. You've experimented with this architecture and you've experimented with this way of building things, but here's some stuff that we can help show you about the implications of that from a cost perspective, so that you're able to learn about those things and actually develop capability to not build that kind of application on that kind of architecture, and to then sort of ramp down the usage and to simplify things for that specific use case."

Rebecca Parsons:

Then would you consider, overall, that the Labs program has been a success? And if so, how would you characterize that success and maybe, what might you change?

Andy Yates:

That's a great question. Absolutely a success, because we do have that visibility now, at an organizational level as well as individuals. We're able to see what kinds of applications people are building. We can see where the demand is. We can then take a call as to whether we want to meet that demand in different ways. We've seen some applications progress from a small experiment into something that is actually owned and managed by one of our sort of longstanding teams, and we're starting to develop a more clear pathway for that to happen. And also, one of the interesting things I mentioned at the top of the call, that's at the top of the recording, was that we don't really do a great job, we haven't really done a great job of shutting things down.

Andy Yates:

And actually what happens is things get spun up and spun off and nobody really knows where the keys are anymore. With this program, we've got a much better sense of not just what costs are happening, but how much things are being used. From that, we're able to talk to the owner's applications and and have a pretty good discussion around how much something's being used and how much value it's producing for them and for their colleagues, and where things are, trading off and trading down to zero and where people who had set up experiments have moved on from the organization or moved on from the roles that they were in. We're actually able to gracefully shut things down.

Zhamak Dehghani:

And I think, for me, hearing about ºÚÁÏÃÅ Lab beyond just enabling people to solve problems and use the tools that the labs provide, I think there is some interesting kind of learnings and patterns. One is, as you mentioned, people kind of copy paste. That's why the fork exists. This somewhat open source behavior, or writing software on their kind of open source guidelines, that you can either go back and contribute or you can kind of the fork and go on your way on your path experiment and learn how to do microservices instead of solving leave application problems. And also, I think, related to that, you mentioned a couple of light controls or light governance that you put in place. What I really liked was your emphasis on visibility. You measured the cost and just bringing the outcome of those measurements or the output available to people, let people make wise decisions, or sharing what other applications are out there. I think there are some interesting patterns around using open source model and light governance to enable a decentralized, fairly decentralized kind of organization.

Andy Yates:

Absolutely. Absolutely. Yeah. The focus of the team right now, actually, is on creating more and better dashboards, so that they're able to expose more of that information back to the people who are working on projects, but also to other parts of the business. What we're seeing, actually, is that if we think about the way that IT works in our organization, we're very aligned with the primary functions of our business. We have a product vision, where we have teams who support functions like sales or marketing or finance, and have teams that look at the end-to-end experience across journeys, across those applications, but we haven't historically had IT teams who are focused on building applications to solve problems for other groupings within the organization. I think about sort of larger initiatives or regions as being other ways that we group our organization, we think about our organization, and those regions, in particular, have specific needs.

Andy Yates:

Prior to this program, their only way of getting those needs met was to go to a global function and prevail upon them that my specific set of regional needs are important, relative to everybody else's needs across the world. This model of Labs, which has been primarily developed for small groups and individuals within the organization, actually is starting to scale up to a place where regions are able to put their own investment into people. They can say, "We would like to form a team that is not necessarily a long lived team, but long enough to build and support and manage some applications for us."

Andy Yates:

By being able to take advantage of the program, they're able to actually create a small estate of applications of their own that, again, extend the things that we provide at a global level to meet those regional needs. We're seeing this, sort of, as a way of experimenting a little bit with the IT model itself and decentralizing some of the IT that we provide, without totally decentralizing it and running into the problems we saw at the start.

Rebecca Parsons:

In addition to the patterns that Zhamak pointed out, what kind of advice would you give to other organizations that that would want to try to spin up something like this? What do you think was critical, really, to the success, or your run of success so far?

Andy Yates:

I think there's a few things that have been helping us along here. One was around ownership and having some continuity of ownership for projects. We said, if you want to start something, you can start something, but it's recorded that you are the owner, and being a little bit strict about that. If, for whatever reason, we're no longer able to get in touch with the owner or the owners of a project, then we feel much less guilty about pulling the plug. The onus is very much on that project being owned. People sometimes hand ownership across to other people as things go, but making sure that that owner is the person who's responsible for the application is really key.

Andy Yates:

Contrast that to a previous model, where people would build stuff and try and kind of throw it over the fence to an IT team, that tends to not work, because lots of things then end up kind of languishing in a middle ground where we don't know what the current state of the application is, and whether people are using it or not and what happens if somebody needs support or not? In this way, we're able to make sure that there is some ownership from the project, and specifically where applications are being used by sections of the business, that ownership also needs to be reflected within the area of the business that is using those applications. That's the first.

Andy Yates:

The second, I think, is around defining some a clear purpose for the program and some clear boundaries around that program. Like I said earlier on, we have a very fixed stack, and that fixed stack was chosen because of the nature of the kinds of projects we were working with here. We have a number of projects, the vast majority of our projects, in fact, are people who are between client projects. They're on, what we call, the beach, where they're between projects. And so what we see in terms of those projects is a fast rotation of members, where people need to sort of get started really fast and they need to be able to deliver really quickly and they to be able to hand things over very quickly.

Andy Yates:

And it was because of that, that we put a very constrained set of resources, that we drew a pretty tight line around that and it was this or or nothing, with a strong recommendation for this, because we knew that by constraining it in that way, we were able to open up experimentation in lots of other ways and we were able to then meet those needs of high, high rotation in the teams, and we could manage the infrastructure, we could have a team who is there to support and manage that infrastructure en masse, because it's a single set of instruction that they're trying to manage.

Zhamak Dehghani:

Andy, on the technology stack, you mentioned a few things that you have put in place to enable people. For example, the barebone infrastructure, in terms of the gate, build pipeline, CircleCI and Heroku for running applications and hosting applications. You also mentioned we are opening APIs from a internal information data, like the leave data or employee information for people to use and build richer applications on top of. I wonder if there are any learnings or any other enablers that we are planning to put in place to create richer applications or solve business problems, beyond what you just described in terms of APIs and the delivery infrastructure?

Andy Yates:

One of the things we're looking at is now that we've got that one very simple stack in place, are there ways for us to start to expand that? As we're starting to expand that, what are the things that people are requesting? What are the kinds of applications or kinds of sort of technology solutions that they're trying to provide? And so we've seen some amount of people requesting the ability to build and to deliver mobile applications to our colleagues, so one thing we're starting to think about has been what's the right way for us to provide some kind of infrastructure, so we can deliver mobile applications in a way where, again, it's consistent? Some of this is around providing a level of consistency in one dimension, so that we can allow flexibility in a different dimension.

Andy Yates:

Mobile applications, we're certainly starting to see. We're seeing some teams who've been experimenting with extending out our sort of own platform to allow for IoT applications or IoT solutions to problems. We are starting to flex that offering a little bit, but we're very conscious not to try and accommodate all needs, all at once, because I think that actually is going back to the sort of advice. If you try and do too many things all at once, you end up in a place where you can't provide the management that actually makes this successful. We're looking at some extension points, some ways of flexing lists, but not trying to do too much, too soon.

Zhamak Dehghani:

Do you think security and allowing people or enabling people to build secure applications, so by default, is behind our identity management? By default it's kind of security is a foundational building block of this platform?

Andy Yates:

Yeah, that's a good point actually, and something I should have mentioned earlier. Absolutely. As part of that security review, we do make sure that everything is behind our single sign on and identity management platform as a baseline, which is a baseline for all applications in our ecosystem. Absolutely, that's part of it, and so there's then some support that we've got in place, where we know that there are specific libraries that we can recommend to people that will help them do that. There's a set of advice in terms of, here are the building blocks for your applications, that we've seen lots of people use and that we would recommend people use. That absolutely is part of it.

Zhamak Dehghani:

Perfect. And I think that's kind of established that paved path over time for new application developers, to either copy paste, or just follow the practices that others have followed. And hopefully they are good practices.

Andy Yates:

Well, hopefully yes. And actually, that comes onto the third thing that I would say, in terms of implementing a program like this, in general. This is something that we see within our organization and across our organization, but a focus on making it easier to do the right things, rather than spending our time trying and making it harder to do the wrong things, is definitely the way that we tend to approach this. If my options are, I can press this button and I get all of this infrastructure, or I've got to go through a whole bunch of requests and setting things up myself, well I'm going to press this button. And so our whole approach is one where we're saying, "Okay, there's a path. We'd like you to go on this path and we're going to make it super easy for you to go down that path."

Andy Yates:

We're not stopping people going down other paths, and there are other reasons for wanting to experiment with different infrastructure. For instance, we certainly see people who want to deploy on AWS or GCP as a learning exercise, and we certainly don't stop people doing that, but that isn't part of this program. We're being clear about the fact that there's a different thing that people are doing and that when they're doing that, they're doing that to learn, and they're not doing that for building business-facing applications that scratch an itch, that we've been talking about. We're making a clear separation there, and by doing that, it means that we're able to support people doing all the different things they want to do, but to do that in a way that allows some level of responsibility around those things.

Rebecca Parsons:

That's fantastic advice, and something that we talk a lot about in terms of building platforms. Make it easy to do the right thing, make sure the right button's there to be clicked, and you can still then, allow people to experiment and take a different path if they have to. But I really love that. Focus on making it easy to do the right thing, than harder to do the wrong thing.

Rebecca Parsons:

Well, thank you very much for your time, Andy. This has been quite interesting and I hope our listeners find this to be useful as well, and find a way to actually solve some of the problems of Shadow IT, to get the benefits of the creativity, the innovation, while solving some of the very real problems that come along with things like Shadow IT. Thank you very much, Andy. Thank you, Zhamak, and thank you listeners.

Mike Mason:

Hi, my name's Mike Mason, and on the next episode of the podcast, I'll be talking to James Lewis about the physics of scaling microservices.