Podcast

Fix Your Technical Debt: The Secrets to a Healthy Engineering Ecosystem

Written by Intevity | Nov 7, 2023 5:01:18 PM

George Jagodzinski (00:03):
Today's guest has over 25 years experience building software and leading engineering teams. He's worked at companies, large and small, startups, and enterprise. He's currently an engineering leader at Under Armour. Today we explore the relationship between organizations and their engineering teams based off of the flavor and the stage of that org, we learn about keeping engineers happy and how that just makes everything better, we learn one of the best tech debt strategies I've heard ever, and we discuss build-versus-buy in the land of software platforms. We've been working with Under Armour since 2006. It's been amazing to witness the growth and change as an organization overall, but also its relationship with software and engineering teams and how that's evolved. I'm very pleased to welcome Karim Shehadeh.

(00:45):
Welcome to Evolving Industry, a no-BS podcast about business leaders who are successfully weaving technology into their company's DNA to forge a better path forward. If you're looking to actually move the ball forward rather than spinning around in a tornado of buzzwords, you're in the right place. I'm your host, George Jagodzinski.

(01:08):
Karim, thanks so much for being here.

Karim Shehadeh (01:10):
Thanks for having me.

George Jagodzinski (01:11):
You've had the pleasure of being at some very different types of organizations, and even at the organizations that you've been at, they've had quite a bit of evolution. And so one of the topics I wanted to explore with you are, what were the different kind of flavors of organizations you stepped into? And let's maybe start with some of the differences and pull the thread from there.

Karim Shehadeh (01:32):
That sounds good. I mean, the differences in the organizations, I went from a large organization... I mean, I've had several jobs before my current one, but even within my current job at Under Armour, I think one of the things that has been interesting is the evolution of the digital organization within the company. But I've also worked at startups, and their approach to how to push productivity, how to focus on quality, how to organize projects, all of that is very different. So it's been really interesting to me to go from one to the other and sort of see the advantages and disadvantages of each.

George Jagodzinski (02:11):
So let's talk about what is it like when you go from an extremely engineering-leading organization into one that's maybe hardcore project management-leading. What's that like?

Karim Shehadeh (02:23):
I think one of the interesting things is that when you start thinking about organizations and the people in those organizations, not in terms of their positions, but in terms of the roles they fill, I think you start to see some patterns emerge. So one of the things that was interesting when I was at a startup, we had this notion of no project management. I mean, there were literally no project managers. And that's very different from the company I'm at now, which has a lot of project managers, and they're very focused on that sort of cross-team organization.

(02:52):
But what happened at the startup was it's not that project management went away as a need, it just sort of fell to other people. And so cross-team communication, organization, messaging, sort of retrospective type things, all of that fell to the engineering manager and product manager. And because it wasn't super well-defined as to what that role encompassed, sometimes you had a variety of success, right? So if it happened to be an engineering manager or a product manager who understood the value of that, they spent time on it and focused on it. And those who didn't, didn't. And so, you had poor communication. And so the same can happen even when you do have a project manager if they're not great at what they do, but at least in that circumstance, it's a very well-defined role. And I think that's what having a position gives you is you're forced to think of that role, whereas it can disappear if you don't have it.

George Jagodzinski (03:47):
Interesting. People are people regardless of their role, right? And then there's the culture, and maybe you have a culture where people are more inclined to call the ball while it's in the air and catch it, and just you grab whatever they're going to grab regardless of the role. And sometimes, even if they explicitly have that title, they might not really live up to it. I'm curious if you've seen any frameworks or common language around that. I mean, I'm always a fan of RACI. I find that people find that that's a little old school, but whatever you call it, I feel like if you just throw down a laundry list of, "Here's all the stuff that needs to get done, I don't care what the heck we call it. Someone's got to grab each one of these." Have you found any frameworks or anything that would work for you?

Karim Shehadeh (04:29):
So I'm not super knowledgeable about other options other than RACI, but RACI is what we have used at Under Armour pretty extensively. And when it is applied, it's actually really helpful. And from a digital perspective is if you're being asked to be accountable to something, then you have to have the ability to make your decisions. So I guess what I'm saying is if someone says, "You are accountable to this software product," in other words, you have to make sure that it has fewer bugs, that it works well, it has high uptime, all that. If a leader comes to you and says, "You're accountable for this," then you have to have the power. You need to be empowered in order to make significant decisions about where that's going to go.
(05:17):
So for example, a very specific example is if you're being asked to say, "For this web application, we need to keep the number of regressions down to some small percentage," let's say we're actually measuring it, and it's down to five percent or whatever, then you as a leader and who's accountable, if you know that you're accountable, then you can say, "Well, if you want that, the two things have to happen. One, we have to slow down, we have to reduce the amount of new features that are going into this, and/or we have to automate significantly the testing," right?

(05:55):
So that's a decision where you're like, "If you're saying I'm accountable and you're saying we have to increase our velocity, and we have to improve our quality, in order to keep all of those things in check, I need to be empowered to tell you, 'This is what we're going to need in order to accomplish that.'" And as a leader, if your red flags should start going up, if you start being asked for accountability, and then when you start to ask for things in order to support that, you have to say, "I can't make that happen. So I can't be accountable, or you have to reduce whatever it is that you're asking me to do."

George Jagodzinski (06:33):
Evolving Industry is brought to you by Intevity. We bring order to chaos wherever people, process, and technology converge. Our culture drives our solutions, and we are solution-obsessed. We see every challenge as an opportunity, every partner as a collaborator, and every project as a chance to share our values and commitment to excellence. Give us a shout. We'd love to hear your challenges and turn them into opportunities. Find out more at intevity.com. Now, back to the show.

Karim Shehadeh (07:01):
And that's what the RACI can help with. They can help make sure that, "All right." Well, if you're informed, you obviously have the best intentions, and you want to make sure that this project, or you're advising on this. Yes, you're an advisor, and ultimately, you're not accountable to the success. You want it to be as successful as possible, but it's not that career type of lock-in. You're just like, "I'm going to help you." And that's really good to know when you're setting up any kind of project or any kind of long-term project, even if it's the product that you're going to support in perpetuity.

George Jagodzinski (07:37):
It's amazing. You can never just get away from clear expectations, accountability, and empowerment. It's pretty basic stuff. But I feel we get kind of spun around titles, and not just the titles themselves, but maybe the baggage that different people have and the overload phrases and all that. I mean, I'm half tempted sometimes when we do kickoffs just to blank canvas everything, just get rid of all of those words and the labels, and just write down the stuff that needs to get done. I don't know.

(08:05):
I was actually thinking this idea lately for a workshop for kickoff is just literally write it down on balls and just start throwing them out into the room like, "Who's going to catch this one?" because someone will be like, "All right. You're the project manager. What does that mean? You manage the project, but what does that actually mean in real life?" I'm curious, what have you seen? Because none of these models are inherently good or bad because sometimes you see an organization where the user experience team are the gods of the team. There's others where the program and project managers are. There's other times when the engineers are. How do you figure out for your team on the stage at which your team is at what the right model is for you?

Karim Shehadeh (08:51):
I think what makes it even more complicated is that it's a moving target. So as the team, as attrition happens, and as people come in and replace, their focus no longer is the same, and you suddenly find yourself in a different situation. So I think the first thing I would say is you need to set yourself up and your team set up so that you're constantly re-reviewing where you sit in the organization to better understand how to apply your thoughts on it. I was just actually listening to a podcast about Kaizen, which I never really heard about. But anyway, the idea is that you sort of meet periodically, and it's a way of applying good work to perpetual improvement. And I think that really applies here because when you are trying to figure out whether or not to where you fall in the organization right now.

(09:40):
When I was at the startup, I was at Mapbox, and that startup, it was an amazing engineering organization. They had, I don't know, 250 engineers. It was a 450-person company at the time and just brilliant people. And it was very engineering-driven, and there was really no question about where engineering sort of sat in the organization. The product we produced was software. When you're at a company like Under Armour, the product is no longer the software. The product is physical product that's being produced, but also a brand. So an idea is part of the product. That's a completely different thing. And the place of technology in there is not set in stone. So it shifts over time. So how do you see where you sit?

(10:27):
And what I've come to realize is that in an organization like Under Armour, we lead with technology for sure. Technology is an important part of everything that we do. But at the same time, ultimately, you have to keep in mind that how you achieve that technological supremacy is going to change over time because if you need to push product more, if you need to push brand more, that's always going to take precedence over technology. Now, you might use technology in order to achieve that, and that's where opinions sort of come and go. So some people might have the opinion that, "In order to improve our brand, we want to improve the applications that we use in order to buy the product." That's where they want to focus. And with those people come investments, so they're like, "Let's invest here more in the technology group."

(11:14):
But I think what I realized at Under Armour is that it is going to fluctuate. And so when you think about the decisions that you make as an engineering leader is like, "I want to make sure that I have some consistency in the output that we produce. So I'm not going to make any assumptions about the number of engineers I'm going to have." At Mapbox, I would say, "You can't produce this product without a stable set of engineers that are at a certain level." At Under Armor, you need to think more about the product that you're producing and then making sure that you reach some foundation where it can be maintained with a flexible set of engineers who are applying their expertise on it. Sometimes you're going to have people who are more junior, and sometimes you're going to have people who are more senior, and sometimes you're not going to have anyone at all. So can it stand the test of time in that sort of fluctuating marketplace?

(12:04):
So when I think about Under Armour, I set myself up for making decisions based on that. And so when we think about just going back to project management, that fluctuates as well. It fluctuates as well because, well, for a lot of reasons, but for us, it fluctuates because the type of project manager that's going to be working on your product might have a different background. So they may not have a super technologically strong background. So how do you sort of work it into your whole process? And that's where we saw it fall back on. In my opinion, like CICD and automation, even though it's very important in any organization, it's really important in an organization that doesn't have the predictability necessarily of what six months from now your organization is going to look like. But you better be sure that the thing that your organization is built on top of in technology is able to hum, right? And it needs to be able to easy deployments, easy onboarding, all of that stuff needs to happen relatively easily.

(13:04):
In the technology space, we're not innovating in terms of new computer science models, we're not thinking in terms of new paradigms of how to compute. We're using well-established methodologies for getting software out and delivering it in a really fast way. And that's where our focus needs to be. So at Under Armour, that's different. Mapbox, the focus was, "We need to develop the best possible algorithm for navigating from point A to point B. How do we make sure we do that in a way that's better even than Google Maps?" That is a completely different problem, right? And so that's what's always interesting to me is you can't apply the same sort of approaches. There's certain things that are always true, but understanding where you sit in the organization, where you sit in the marketplace is really important as an engineering leader.

George Jagodzinski (13:57):
That makes sense to make sure you're not overinvesting in building software when that's not your core business. You talked a little bit about innovation there. I feel like some people just get hung up. And I've seen some engineers, and even myself, when I was earlier in my career, I got hung up on this that I wanted to only be doing the crazy innovative, forward-leaning, whatever software it could be. But then I found that I could get just as excited about it because when you're doing something new at an organization or new within an industry, it might not be innovative as a whole out there, but the fact that it is innovative for that industry and that client, that gets exciting to me just as much. And I've found engineering teams, if you can connect them enough with the brand and the mission, I don't know if you've seen it as well, but they get just excited about it. Do you see that?

Karim Shehadeh (14:44):
Absolutely. I went through the same sort of path that you did. I felt the same way. I felt like I needed work at Google. Now, there's ChatGPT, they're breaking ground on areas that were previously unbroken, but there's lots of little niches where there's a lot of interest. Right now, at Under Armour, we recently released a completely redone website that goes significantly faster, five or six times faster. And that was a challenge for us, right? Because we're not using new technology, it's all based on open-source technology, but it's modern, and it was really interesting to learn how to do that. And I think the team that's working on it was super-duper engaged. They understand that we're using, but we're applying it in a way that is not that commonplace.

(15:36):
In this particular area, we are on the edge of where the technology is, and I think that that really helps motivate people. And then even outside the technology, even CICD because it's such an important part of the whole process, but I'd love to be able to say at the end of the day that, "We can get out a release in under 10 minutes." Yes, Facebook is releasing thousands of releases a day. They have an army of engineers, but it's in a different problem set when you have just a few engineers in a company that's not based entirely around technology. To be able to do that is quite an accomplishment in a different way.

George Jagodzinski (16:19):
That's great because I feel like there's this misconception that engineering teams, they don't like to have a lot of change, right? "Let's not change the way they're organized, let's get them in their flow state, and let them just kind of groove through the work that they have." But my experience, and I'm curious to hear yours, is they love change. You just need to have those right foundational things in place that not everything's going to break and fall apart and turn to mayhem when it's changing. Has that been your experience? And I'd also love to hear how you get the team to rally around that type and big change the way that you're organizing your teams, the way that you're attacking things, maybe going vertically or horizontally. I'd love to hear your perspective there.

Karim Shehadeh (16:59):
I think there's this cognitive load that happens, this very unpleasant thing that happens over time, especially it's true for legacy software. If you didn't have a hand in building it and you're not empowered to improve it, what you have is the situation where the engineer is not fulfilled, right? Because they're not able to explore their creativity, their skillset, their expertise, because they have to be nervous about every single thing that goes out because there's a relatively good chance it's going to break something and they're going to be held accountable for it. And that's where the accountability comes back in. It's like, "Look, if you want me to work on this thing, you need to help. You need to let me have some space in order to improve it so I don't feel terrified every time I release it." That foundational, that technical debt, that's where understanding where the technical debt, how far you go in terms of what you fix with technical debt. It really comes down to, "I want to fix it enough so that I don't feel terrified every time I push a release."

(17:59):
I think that should be the rule of thumb. And if you don't have that, then you sort of need to revisit. You need to take 20, 30 percent of whatever resources you have and just say, "They're focused on making sure that we have a stable environment." And then the engineers, that unlocks, it's not just their happiness because I'm very much about the happiness of the engineer because I think that unlocks a bunch of other things. The engineers are happy, they're more creative, they're more productive, it just creates an amazing culture. And how do you get happiness? I mean, obviously, everyone's a little different, but ultimately, it's about feeling valued and impactful. If they can't get anything out, or it's such a struggle to get anything out, then they're not going to be happy, and they're not going to be impactful, not going to be productive. And so you need to fix that quickly. If that continues, you'll never get anywhere. You'll reach some plateau, and it'll go down every once in a while, but you'll never soar. So it's really important.

George Jagodzinski (19:05):
People just start checking out at that point. I'm going to steal that from you, though, the tech debt strategy, because everyone's trying to figure out, "How are we going to attack the technical debt?" It's like, "Well, we're going to remove the horror and the fear from the organization as we attack this." I love that.
Karim Shehadeh (19:22):
I was just going to say it comes down to predictability. So people want predictability. They want to be able to say, "The next thing I'm going to do is going to happen in two weeks." If you can't give that, what engineers are being asked all the time, "When are you going to get done, when you going to get done?" They're terrified because they're like, "Well, I mean, if everything goes fine, two weeks." But in their head, they're thinking, "I'm pretty sure nothing's going to go well here. And so I'm just going to bump it up another two weeks because who knows?"

(19:51):
And so that's what you get is from the business perspective, trust your engineers. If they're nervous about releasing it, it's because they have good reason to be. And so they're either going to tell you, "If you're going to force me to give you a number here, I am going to have to give you a month or two months because I know that there's instability here." That's where it starts impacting the business. And that's where they need to start thinking about, "Hey, listen to your engineers about this for a little bit."

George Jagodzinski (20:19):
That's how you end up with a two-year backlog that doesn't really get chipped away at any given point. In pulling the thread on the developer and the engineer happiness, early in my career, we were just building a lot of stuff bespoke because the platforms weren't really where they are today. And now we're at a point that the platforms aren't only "mature," but there's just a lot of pressure from executives, there's a lot of pressure from sales teams. And I think sometimes platforms, I love them in the right place. I find that sometimes they're used incorrectly, and engineering teams can get disenfranchised in those environments. But I'm curious, from the various places that you've been, how do you navigate that build versus buy, and how do you keep your engineering teams happy within that ecosystem of platforms that are all around you?

Karim Shehadeh (21:12):
Well, I think the key is obviously you have to make the right decisions about the products you buy. But then, once you buy them, you need to use them as they were intended to be used. So some companies, they will buy a product, and then they'll be like, "Well, now you need to customize it." So they'll work with these companies, and they'll say, "Well, we need this one feature." And depending on the size of the company, if it's either a huge one, they'll say, "No, this is how you can figure it out yourself." Or if it's a small one, this is worse. They'll make changes to the product because they're scrappy, and they need revenue. That's the worst possible thing even though it sounds good because now they've created a very customized thing for you, but they're not going to be able to support. And you're going to get mad, they're going to get mad, and then nothing good will come of it.

(22:01):
So I think the key is to make sure that when you make the decision about the product to buy, first of all is that it has the features that you want as is. So if they're saying things like, "Well, we're planning on that in six months." Absolutely not. No guarantee of that happening whatsoever. And so that needs to either make it off the list of requirements or that company, that product makes it off the list entirely. But then the other thing is once you do get it... I think there's this thing that I think people realize that just because you got something off the shelf... I mean, SAP is a great example, right? It is an off-the-shelf product that requires an enormous amount of customization. It's really just a platform. Salesforce Commerce, another example.

(22:50):
If you are a brand and you need to customize your experience for the user because you are a brand, and it's all about customization, think about that. Yes, it's providing that value, but remember that you have to do a lot of work on top of it. So just work that into your plans. I think it's really just a matter of making sure that people understand that buying a product doesn't mean set it and forget it. It almost never means that, especially for larger companies that like to customize. Now, if you go and get a Shopify account and you just set up, it's great for that sort of thing. You can just set it up in no time. They have templates, and you're in their zone. Everything you're doing is just part of the platform, and that's fantastic, but you just have to keep that in mind.

(23:39):
The one thing that I often say about build versus buy is build the things that are unique to your organization that you need to differentiate yourself with. And the front end of the website, for example, or your applications, those are areas where a brand needs to show off, that's the close, right? Or it's the body of the car, the engine. Do your best, find the product that can support all the functionality that you need in order to complete the journey for the user, but you're not going to be breaking any new ground when it comes... You're not going to be like an Amazon. It's just like, "Actually, I'm just going to completely redo the way we do cloud infrastructure." That's never going to happen, and therefore just use the functionality that's available to you. So that's number one.

(24:31):
And then the other big one is, over the last two years, I have become a big proponent of the model where you have open-source software that is supported by a company. So they're operating in a way where they are using open-source software that is available to you if you want, and has a community around it, but they're providing a ton of support. So they're providing potentially infrastructure I use for sale a lot because they're an amazing company that provides a lot of expertise around a few products like Dex.js, and they provide all this infrastructure for you. You don't have to think about it at all. They provide a CICD sort of deployment structure that you don't have to worry about it at all. Those sorts of things for a smaller company are invaluable and, in my opinion, worth the cost. So you just have to sort of do a little bit of math on that.

George Jagodzinski (25:25):
I think last time we spoke, you even talked about a company where you asked for something to be added to the roadmap, and they told you, "Absolutely not." Am I remembering that correctly?

Karim Shehadeh (25:34):
Yeah. No, I mean, so we've worked with big authentication companies, Okta, and I guess they bought Auth0, but they're doing what you should do, which is saying, "Look, these are the feature sets that we have. We'll tell you what is on our roadmap, but you got to use what you have. We're not going to build something custom for you." And that's okay. That's how it should be. Now you can make the decision in plain view of saying, let's say it's two-factor off, let's say they didn't have that or whatever, it's like, "Well, it is part of our plan to have two-factor off in six months. It's not on your roadmap. Therefore, I'm going to have to move to someone else, but can you please add it?" And then they say, "We'll promise." That's not a good plan, I don't think.

George Jagodzinski (26:27):
I feel like one of the pitfalls that teams get into is you spend a lot of money on a platform and multiple platforms, and so then the strategy just kind of becomes, "Let's push as much into this thing until somebody says no," which they typically don't, or until something breaks or becomes a big giant mess where it's like, "If we just kind of put those guidelines and the foresight in place ahead of time and come up with a framework for ourselves for our specific organization that says, 'Here's what's going to go into the platform and why, and here's what's not going to and why it's not going to.'" Then at the same time, you're not only keeping things clean, but I think you can get the engineering teams excited because now they know, "We're not only dealing with the platform. Here's all this cool stuff we're going to build." And it's usually the stuff, like you said, that's much more meaningful to that organization. And it's the secret sauce, right?

Karim Shehadeh (27:19):
That's exactly right. And you just have to establish that. You just have to make sure that's clear everyone comes in. One of the problems is that if, as a company, you keep changing your sort of methodology or don't define it very well, you hire in engineers with an idea of what they're going to be doing, and then it suddenly changes, and you have that breakdown in communication, happiness goes away, productivity goes away. So really, having a good understanding of your digital plan is really important.

George Jagodzinski (27:48):
I see a lot of passion in there for leading engineering teams. What is it about engineering teams that gets you excited?

Karim Shehadeh (27:58):
For one thing, I sort of fall into this naturally because I'm socially inept. That's kind of my thing. But also, from an engineering perspective, I definitely think the same way as a lot of engineers do. We have a lot of the same interests, and we tend to have a lot of passion for the same things. And so I've been in engineering since 98, so 25 years or so of experience, and I went into management about 12 years into my stint. And one of the reasons I did that was because I really wanted to create a better environment for engineers. And so it's like a personal passion of mine to create an environment where engineers can really thrive.

(28:49):
What I've said almost all the time is that it is really rare when you don't have an engineer whose potential is super-duper high. What they're specifically doing and how they're getting there is a question of that particular person. But I think the vast majority of people, if they actually have a passion for this stuff and you give them the runway, they can do an amazing job in whatever path you set up for them. I think they can do an amazing job. In fact, my job as a leader is to step back, just clear the path, and have them go. And I think that model of doing it, I think, is obviously the best thing that works for me, and I think it works for most amazing engineers as well to just don't always have that opportunity.

George Jagodzinski (29:34):
That's great. And on the flip side, it's so frustrating when you see that passion just unnecessarily getting squashed just due to inefficiency of the org or not prioritizing or clarifying things. Man, I think you just broke my brain because I started around the same time, and 25 years, I didn't realize it was that long now. You're making me feel old right now.
Karim Shehadeh (29:56):
I feel that every day. I have a young team.

George Jagodzinski (30:01):
Nice. Well, Karim, I really enjoyed this conversation. I always like to finish on one fun question, which is, over the years, what's the best advice you've ever received in life, work, wherever?

Karim Shehadeh (30:13):
Actually, I was at a company called Millennial Media, and I had a manager. It was my first managing job. I was moving from an IC type of role into more of a manager role, but I couldn't let go of the IC part of it. I still can't, but one of the things that always sort of pulls me back is what he said, which was, "You have a team of five at the time, right? There is no way that you're going to do five people's work by yourself, right? So the best thing that you can possibly do is empower the people on your team, and then you can do five X. At best, you can do two people's job. You're never going to be someone who's going to do five people's job, and if you are, then there's something wrong with the team."

(30:57):
And so I've always taken that back. It's just like, "I'm very interested in this particular thing that we're doing," I think all engineers are probably like this, "So I can do this best." And then I realized that does not scale. That is not a good approach. Pursue the things that you can, but keep in mind that lots of people, a team makes all the difference. So I try to remember that as well as a leader.

George Jagodzinski (31:21):
That's great, especially because it appeals to that engineering mind of logic and efficiency. That just makes a lot of sense. Karim, thank you so much. I loved it.

(31:33):
Thanks for listening to Evolving Industry. For more, subscribe and follow us on your favorite podcast platform, and pretty please, drop us a review. We'd really appreciate it. If you're watching or listening on YouTube, hit that subscribe button and smash the bell button for notifications. If you know someone who's pushing the limits to evolve their business, reach out to the show at evolvingindustry@intevity.com. Reach out to me, George Jagodzinski, on LinkedIn. I love speaking with people, getting the hard work done. The business environment's always changing, and you're either keeping up or going extinct. We'll catch you next time, and until then, keep evolving.