FinOps in Action

Tim Nolan: through, you know, creating partnerships with your procurement team is, of course good, right?

You want to get the best possible deal and the best possible rate. need to do them a favor once the deal is, is in motion. you only get one chance or two chances to, to make those kinds of, policy sort of decisions. I think if you're gonna destroy resources in production, it needs to be very well communicated.

It shouldn't be emotional,

Alon Arvatz: Hello everyone and welcome to another episode of finops in Action Today. I'm particularly intrigued and excited towards the discussion with today's guest. He actually started as an engineer, moved to IT finance, and eventually found peace at finops. It started at a big financial services enterprise and moved to a first growing tech company.

On the first four month of his role, he saved over $1 million on AWS costs and achieved over 99% tagging compliance, who can say that about himself? I'm happy to welcome Senior finops engineer at DataRobot Tim Nolan. Tim, how are you today?

Tim Nolan: I'm doing good. Appreciate that warm introduction for my

first ever podcast.

Alon Arvatz: Yes, it's great to see you and please share with us team, what was the one thing you got wrong when you started your finops career?

Tim Nolan: so great question. Um, I, I think the, the, the TLDR there is, you know, everyone wants to use less cloud and focus on your usage optimization. Uh, I think I probably didn't pay enough attention to. Uh, paying less, right? This idea of rate optimization and that whole sort of side of vendor negotiations, your cloud commitments.

And, uh, if I can go back in time, I would've probably spent a little bit more time brushing up on my vendor negotiation skills and partnership with, um, procurement and finance, which I'd love to dive into in this podcast 'cause it's, uh, it, it's exciting

topic for me. So.

Alon Arvatz: That's actually very interesting because I think most of the fitness people will say, Hey, we started with cost visibility and then we moved to commitments. But sounds like your intuition was to focus in the beginning, at least on usage optimization. Is that because of your engineering background?

Tim Nolan: Uh, absolutely right. So, you know, I'm kind of born and bred I started out my career as kind of a Linux engineer and, you know, you always want to, um, at least I've always wanted to as an engineer, uh, build solutions that are not only resilient, but efficient and can run, uh, as efficiently as possible.

And, you know, when, when, you know, the companies I've worked for have kind of migrated to the cloud, you wanna. Pay, pay as little as possible. That's natural. Um, you know, I'll say, like, my disclaimer on this podcast is that your mileage may vary and, uh, everyone is at different stages or companies are at different stages of their finops journey.

Um, and, and that's okay. And, and so from my perspective, um. know, using less cloud and, and, and optimizing your usage is of course important. And, and that's, that's great. I've listened to other guests on your podcast and I just wanted to take a little bit of a, a fun turn and maybe talk about more of the, um,

more of the rate side of things.

So.

Alon Arvatz: That's actually, um, that's fascinating. I think there is, there's a lot of sense starting from usage optimization because you don't want to commit to waste. On the other hand, sounds like you, uh, were on a journey of moving from what is more interesting to you. What, what you felt that you can make an Im, uh, I don't know.

You can use your background to make an impact to a perspective of, Hey, let's focus on where I can make more impact. This is what's important.

Tim Nolan: Yeah, absolutely. I'll, I'll never forget, um, at my previous company, uh. Underwent a really cool training and it was, um, conducted and facilitated by our procurement leader at the time who, uh, taught, um, how sales, uh, and sellers, uh, will typically interact with, um, with technology teams. So that was a very, uh, impactful experience for me and my career moving forward.

And you kind of get to understand, uh, and I hope I'm not offending any of the sales folks that I've, I've worked with and, you know, they're great and I also can, could talk the, the positives, but they have deals to make and, and, um, their own, you know, kind of revenue commitments. So I think that whole side of, uh, engineering tend, you tend to.

Uh, work with your finance counterparts. Maybe you're working on, um, a forecast or the number of licenses that your tool you're responsible for has, and the, the process sort of stops there and then finance and procurement takeover. Um, whereas in, in finops and cloud especially, I think that engineers should play more of a take, take more of a front seat with, uh, commitment tracking.

Uh, with forecast tracking and, and with renewals. So I'll, I'll, I'll start there. I kinda have like three, um, three sort of tips I would say when it comes to, uh, you know, cloud commitment renewals. I'm, I'm. We'll hopefully be going on my fourth co, uh, AWS renewal, uh, in the, in the next 12 months. So I, I feel like I've been in, you know, each renewal.

I've learned something new. Um, and the first piece is to own your, own your forecast, right? Uh, own your own your run rate. I think one of the things that the cloud providers will try to dictate, is past, past performance or past cost history will dictate your company's future. Run rate. uh, it's easy to get a little defensive about that as someone who is in finops.

You wanna think that your run rate is going to go down over time, but you really should, um, do your market research, figure out kind of, um, try to use your contacts and the finops foundation on how the cloud providers will try to structure, uh, the commitments. And I think a lot of that is. AWS for example, will likely take your trailing six months of cost history, double it, and use that as a starting point and say, here's kind of the starting point for your commitment where, um, you really, you know, as it's, it's your, it's your organization, right?

You need to be a little bit more forthright and, and say, no, actually, like we're going, um, we're going on this different trajectory. AWS like, we need to start the conversations there. So I'll kind of pause there

and see

if, if that, how that.

Alon Arvatz: So the first tip on your forecasting, how do you do that? Is that through discussion with engineers? Is that through discussion with leadership to understand if there are new products planned, how do you do that.

Tim Nolan: Yeah, it's so, it, it's with both. I'll say one of the, one of the mistakes I've made in the past is focusing too much on the conversations with engineers. Less on, 'cause frankly, I worked at a big company in my previous role and, and my company now is, is, is pretty, you know, lean by comparison and much smaller.

It's understanding your company's direction. That kind of sounds, uh, obvious, but there could be strategic things that are happening that might be above your pay grade that you kind of have to, um, uh, dig into and, and understand that strategic company. Decisions could impact obviously your cloud run rate, that could be an acquisition, um, that could be a deprecation of a major technology.

All these things need to be factored in. I think a lot of the times the, uh, the finops lifecycle and usage optimization is all great, but sometimes it's the

uh, the stroke of the pen that can have the biggest

impacts, um, that will usually last the most amount of time, right? Because commitments can be three to.

years

plus.

Alon Arvatz: Yeah,

so.

on the forecast, don't let the cloud provider dictate how much to commit. What are the other tips you have for us?

Tim Nolan: Um, you know, I, I would say just, just understand and, um, understand kind

of where sellers are coming from. I say

sellers, I

mean that the, the salespeople, um, understand the, uh, um, might be a strong word, but just the strategies that they have, um, such as maybe, uh, citing your manager, manager's manager and making it kind of.

Seem like it's, it's your leadership's priority, uh, using urgency and kind of, uh, time as, as a tactic. Um, they may wanna. Delay and engage late to, to kind of create that sense of urgency. And I think, um, what I've found, and, and look, I'm not the world's best negotiator, is, uh, help your procurement team help you, right?

That would be my tip is understand that, um, if you kind of understand how the deal mechanics works, but you don't want to, um, maybe take a, a front seat in negotiation. Have your procurement team maybe handle some of the more difficult conversations if you're not comfortable with that and educate your procurement team, know, educate them about the run rate, why it changes, what you can do to affect it.

Site efficiency projects. Our company now has a lot of efficiency projects underway. We, we wanna prioritize them right, and, and, and get our run rate to a point where. When our next renewal comes, we can kind of use that as a, you know, a springboard to, um, to hopefully have, have a more competitive,

lower leaner commitment

going

forward.

Alon Arvatz: That's really fascinating and you actually open my mind to other skills and knowledge that pheno people need to have that is not discussed. So I hear a lot about technical knowledge that you need to have around cloud, around other, uh, cloud technologies. I hear about managing stakeholders. Within the organization.

Um, we hear a lot about, uh, reporting, uh, optimization. We don't hear a a lot about, Hey, you need to understand sales. You need to be a good negotiator. So.

it's a whole set of skills that you need to have on top of what everyone's talking about.

Tim Nolan: Sure. And I think it's probably complimenting the, the technical acumen with, with some of the vendor management skills. You know, one of the things, if, if we, uh, you open the podcast by saying, um, tagging hygiene and tagging compliance, and that's, that's begin, that's becoming a renewed, uh, hot topic at my company right now with respect to Kubernetes cost allocation.

And where that relates to what we just talked about is, uh, a lot of times your, um, your non-cloud. Software, right? Those renewals and those deals start to become more essential to finops because might be running, for example, security tooling that runs, uh, in Kubernetes and it runs as a Damon set and it runs on all of your nodes.

and that particular, uh, workload is. Counted by number of agents, right? For example. So therefore, the, the efficiency projects that you work on, and, and the, um, you know, the, the number of nodes that you have running is really the one metric that the cloud bill cares about, right? It's your compute hours and, um, to the extent that you get, you, you, you can optimize that.

You of course optimize your cloud costs, but you, you then kind of have, uh, these downstream optimizations with your friends in, you know, if you work in engineering, your friends in security, uh, your friends in, in, in, um, shared enterprise services like observability and, and other kind of, um, services that your whole company will consume.

So, uh, you can be a, a, a, an asset and, and a good friend to your, um, to folks like in those kind of. Roles like SRE, security, um, observability, and you can help their, uh,

optimization projects as well.

Alon Arvatz: That's awesome. And I actually wanna talk to you about Kubernetes and tagging, especially with, uh, the success you had. But maybe for that, just because I'm really, really curious. so.

you mentioned the collaboration with, uh, procurement and vendor management. I wonder, how do you do that? And more specifically, do you feel like you need to support them with the negotiation or they are supporting you through a negotiation process?

Tim Nolan: Uh, it's, it's a, it's a two-way street, so. Um, I think you need to be self-aware of what you are good at from negotiation perspective and what you're not good at. I've personally found I'm really good at being the good cop, uh, and I know that, but through, you know, creating partnerships with your procurement team is, is of course, um, uh, of course good, right?

You want to get the best possible deal and the best possible rate. need to do them a favor once the deal is, is in motion. Um, and, and you start, you know, month, month one, all the way to month 36. You wanna be able to, um, continually keep them involved in the forecast and the run rate so they can be better around, you know, kind of at the next renewal.

um, you know, just, just a lot of knowledge sharing. You know, e every procurement team is gonna be different. Um. Not every procurement person, you know, you're not the only deal that they're working on. They may be working on, um, you know, completely separate, uh, software, even non-software deals with, you know, lots of legalese and, and things like that.

So, just because they're not technically familiar with cloud, um, do them a favor and, and spend that extra, you know, uh, five minutes on a meeting afterward or just, just spend more time with them educating on. How cloud works, um, how getting into, you know, like high commitments could put, you could put your whole savings programs or your, um, your optimization programs at risk because, you know, you never wanna have to write a check at the end of the contract anyways, even though you brought your run rate down.

So those types of, you know, vastly important facets of a cloud contract, I

think is, is super helpful.

Alon Arvatz: Yes. And I guess once you realize that you can make such a big impact with, uh, negotiation and contracting, you are a lot more open to work with procurement. I'm sure it's making that a lot easier.

Tim Nolan: Absolutely.

Alon Arvatz: And, and Tim. Okay. We have to talk about the tagging compliance piece because saying 99% is extremely impressive and I would love to know, and I'm sure our listeners would love to know, what was your journey to get to this point?

Tim Nolan: Sure, and I'll, I'll caveat by saying we, we've kind of hit the reset button on tagging compliance because, um, kind of gone from EC2 and traditional cloud resource tagging into Kubernetes labeling and annotation,

but

reminding the clock about three years ago, um, uh, I, I, uh. Managed a

tool called Cloud Custodian, which I know is, is

super

popular in the finops Foundation community.

And just generally for good kind of cloud hygiene and, and, you know, a lot of, the byproduct of tagging compliance as you eliminate waste and you achieve a lot of upfront kind of immediate savings. Um, yeah, we, we had a, as a service, you know, like automation in place in all our AWS accounts.

Um, that would take a kind of a business facing document, a cloud tagging policy, and move that into code. Um, so you can take your, the rules that your finance and your marketing and your sales leadership want around how to attribute, uh, cost of sale. know, production costs versus your r and d costs and you, you, you literally move that, you know, in, into GitHub and into your, your CICD pipelines and you execute against that tagging policy, uh, to improve your hygiene all as well, right?

And, and then, then your company moves to Kubernetes and you realize that all of the, the cloud resource tagging that you did at the EC2 instance level is sort of, um. I'm not gonna say falls on its face, but they're, they're new kind of renewed challenges. Um, our company, for example, we, we did a mass kind of migration to manage Kubernetes right, to EKS.

Not that we weren't on Kubernetes before, but this is more of the, the cloud provider, um, you know, management kind of service On top of that, we, we also decoupled our, our, um, our monolith and moved to more of like a microservices architecture. So now you have teams that manage one or more workloads that, uh, now all of a sudden they, they don't just need a cost center.

They need things like, um, team identifier, right? What, what is this sub engineering team and, and how do I attribute cost to it? So lots of new opportunities and challenges. I'll be the first to admit that we're not perfect, we're it, it's, it's nice that it's cyclical, that we, we kind of. Um, the finops Foundation, the finops Lifecycle is, is kind of like, I thought I had checked that box of, of, um, attributing all of your spend to, uh, to, to the business units.

And now we kind of have to enter that, that racetrack in that lane again where, um, we're, we're kind of finding ourselves having to do that again for, for good reasons. And it's, it's a good opportunity to, to kind of dust off the. The, tagging policy document and, and, and, and make updates, you know, I'm sure reorganization within engineering and, and, and, and the company have, you know, scary effects sometimes for things like cost center alignment, tagging policies.

So it's, it's all good. It's all positive that that 99% metric, I'm gonna have to update

on my resume.

Alon Arvatz: Absolutely. And, uh, I'm curious, Tim, do you need to enforce tagging in order to reach this, uh, this, uh, achievements?

Tim Nolan: Yeah. Yeah. So, um. We traditionally use cloud custodian as a, uh, non-prod, enforcing mechanism where, um, kind of like a three strikes, you're out sort of policy where, um, you, you know, a, a warning, uh, gets, gets sent, whether it's a Slack message or just a, a log message. The resource gets paused, you know, after, um, a couple days or a week.

It's dependent on the policy of the. The specific resource. And then in non-prod, there could be, you know, like a destructive, kind of terminating the resource, uh, especially if it's a costly resource, um, and only kind of, uh, um, you know, read only sort of mode and, and production, right. That would kind of be, be managed, um, a little bit more carefully.

Um, so yeah, it's, it's, uh, enforcing, I think is enforcing with, um, uh. Right Over communicate, especially in a, a decentralized sort of global community that, uh, and, and company that I work for, um, it's, it's not always easy to, you know, just grab someone at the water cooler. You, you're, you're, you're talking to people on the west coast, you're talking to people in Europe, you're talking to people on the East coast.

So, um, you have to

over communicate, especially stuff like that.

Alon Arvatz: For sure, and I'm really curious about the level of friction. You're willing to accept with engineers, so would you shut down a resource in production if it's not tagged for a certain period of time?

Tim Nolan: No, no, I, I would not do that. I think maybe that's my bias as, as

an engineer. I think. Um,

security, reliability, resiliency. Are probably the most important. You want your product to be running. You want it to be, uh, delightful to use as, as a customer and a user. Um, efficiency is, is, I don't wanna say a secondary priority, but it's probably like priority one A in production.

Right. now in non-production I might, might take a, a different stance there, um, because you know, you really can make the case for like office hours, automation and, and, and, um, really aggressive kind of node scaling and bin packing and Kubernetes and all that is great for, um, for non-production, but it needs to be very well tested, um, for it to

go into production.

Right.

Alon Arvatz: So

I totally get the point of the resiliency, um, and security. I wanna, if you also look at it from a perspective of relationship with the engineer organization, like if I shot things in production.

hey, how would that influence the relationship with engineers or how engineers look at the finops, uh, uh, team.

Tim Nolan: Yeah, it's a, it's, it could be a very, um, kind of brittle, you know, you only get a, a. one chance or two chances to, to make those kinds of, policy sort of decisions. I think if you're gonna destroy resources in production, it needs to be very well communicated.

It shouldn't be emotional, it shouldn't be something that you're trying to sort of, um. You know, prove a point or you're trying to change behavior. I think, um, those things should be dealt with via, you know, let's have a conversation about it. Let's, let's bring in leadership if we have to, that kind of thing.

But overall, I think, you know, I've been. Extremely impressed with all the engineers my company. Uh, everyone wants to do the right thing, especially, um, you know, we're, we're not as big as my previous company. So from that perspective, you know, we all understand that the stakes are high and, and cloud cost optimization.

Literally could mean, um, you know, extended runway, um, extended, you know, open positions, right? That, that can be funded. Because as, as you're probably aware, you know, the, the cloud bill is becoming more and more at the forefront of, of, you know, the IT bill these days. So, um, to the extent you can make, uh. You know, hundreds of thousands of dollars of, of savings optimizations.

You can start to have those, you know, pretty big conversations around. Hey, we, we just saved a million dollars in this efficiency project. Now we can fund this other efficiency project, or hey, engineering team, you know, how you've been complaining about you're, you're over-resourced and you don't have enough folks to work on this.

Like, what if we tackle this efficiency project to sort of open up a new hire for your team to do X, Y, and Z? So, um, approaching things that way I think go a long way, especially when you're trying to help the teams

help themselves.

Alon Arvatz: That's, uh, really amazing and I, I wonder how you handle, uh, optimization from that perspective. I'm talking about usage optimization more specifically. Uh, sounds like you have kind of a, a project mode where you take a project, you tackle it, you get a savings, you can do, uh, additional stuff with it. So how is it happening at, uh, DataRobot today?

Tim Nolan: Yeah. Yeah. That, that's exactly it. The, the good news is that we have more efficiency projects than we have. People to work on them. so therefore it gives us a nice backlog of, of, of projects to, um, to prioritize, to fund, to resource. Um, and, and I'll, I'll be honest, a lot of it is virtual teams, right? So, so I work.

You know, a hundred percent on, you know, we, we now kind of call it efficiency. It's, it's efficiency projects rather than cloud finops projects because a lot, a lot of this stuff spans, you know, beyond cloud. so yeah, we use Jira, um, we use Slack, you know, as, as far as like communication and workflow.

Um. And, and yeah, we, we try to fund the projects and, and sort of plan, um, quarterly and try to bucket people into like weekly, um, whether the, whether they be sprints, um, doing planning that way or just swarming a project. It is my opinion, I feel like it the more you can get one or two to three engineers to just dedicate themselves and swarm a project from soup to nuts and really, really define the.

you know, this is what we're gonna do and we know we're done when, you know, this metric changes to this state, right? Like using things like OKRs to, um, to, to drive project and, um, excuse me, progress and limit scope, are things that we try to do. And again, we're not perfect. Um, I

hopefully, uh, I'm not the

only one to say

that,

so.

Alon Arvatz: And I'm really curious how you scope and prioritize this project. so.

do you scope it around a specific inefficiency that you now fix across all the accounts, or do you focus on a specific service or you focus on specific team and take the top initiatives? How you look at it?

Tim Nolan: we try to like, take a cross section of, um, cloud costs saved, right? Like what is the savings potential, of course, along with customer value. So, you know, our, our company for example, we operate in, in the cloud. We have a multi-tenant SaaS platform. We also have customers that operate on-prem. So they, they, they take DataRobot, they install it in their own, um, you know, whether, whether it's their own cloud environment or their own kind of private data center environment.

And a lot of times efficiency projects will help them, uh, meaning that the non-cloud kind of users or the self-managed users. So the more that an efficiency project can affect both, uh, this is why I say a lot of these efficiency projects span. Outside of cloud, the more that they can help both personas, the better.

So if we can reduce our, you know, this efficiency project will reduce our cloud run rate by 10%, but it'll also help our self-manage customers reduce their hardware footprint by 10%. really helps with things like renewals and, and things just, just overall delighting our customers and saying, oh, hey, by the way, you know, if you upgrade to, to our latest version of DataRobot, this hypothetical, by the way, um, you, you know, you, you can run it on 10% less hardware or maybe that translates to, to one or five less, less servers to, to run on.

their data center or in their AWS cloud environment or whichever cloud environment they're running on. And that, that tends to have a, um, those types of value props, I think go a long way for our customers. So we wanna, we have two segments of customers that we need to, um, we need to impress, you know, one is our customers and my customers tend to be our other engineers and our, our own kind of engineering

leadership.

So I'm sort of straddled

in the middle there.

Alon Arvatz: Absolutely. And Tim, I think if I'm taking one message for you is focus on where you can have the most impact. So also be efficient on where you're putting your efforts. To make sure you focus on the things that make the biggest impact. I think that's something that goes across the board on everything you just, uh, shared with us.

Tim Nolan: Absolutely. You gotta know what you know and know what you don't know, and, um, understand there are other engineering experts out there in your company and use that for good. Right? And, and, and be a cheerleader. Make sure to, to celebrate their wins. I, I,

know that's, that's

cited a lot.

Alon Arvatz: absolutely. And Tim, we would also love to learn about you, not only from you, so first of all, you live today at Great Hartford, Connecticut. Correct.

Tim Nolan: Yeah, I'm in the greater Hartford area, so

Right in the middle of the state.

Alon Arvatz: And are you originally from there?

Tim Nolan: Yeah.

Yeah.

So I grew

up, grew up

in Connecticut, born and raised.

Alon Arvatz: Very nice. Very nice. and Is there a book or a movie you would like to recommend to us?

Tim Nolan: Um, let's see, do I,

I actually have it right here. Um,

Alon Arvatz: wow.

Tim Nolan: so this is, hopefully folks know, um, this is, uh, Google's, you know how Google runs production systems. This is the SRE book. And, um, by O'Reilly by the way, I, I enjoyed it and I, I refer back to it every now and then too. Um, akin to what we talked about about production, you know, don't, don't turn off

resources in production.

Right. Make sure production

is

super stable. 'cause I, I think that a lot of times efficiency projects and the priorities of, of running more efficiently and more cost conscious conflicts with. Production stability. And, and so I think, um, I happen to work a lot with our SRE team and I've, I've grown to appreciate, you know, that production, obviously, it sounds very obvious, must be, must be stable, must be up, but sometimes you can't, um, you can't bite off those, those efficiency projects or those savings claims or, um, you know, can't realize the value of making production more efficient.

Right away, right. Sometimes it, it takes a lot of, um, a lot of tuning, a lot of tweaks and, you know, this book I, I'll say, goes into way more detail than I would've thought it

would've went into. Um, but it's super

useful, so

I

recommend it.

Alon Arvatz: so if you wanna turn off your brain and read something easy Site. Reliability Engineering by Riley.

Tim Nolan: Yeah, yeah, I, I bring it, bring it on vacation and stuff like that. So I'm

self-admitted tech nerd.

Alon Arvatz: Okay, that's great. And why do you like to do outside of work for fun?

Tim Nolan: Uh, I, I guess I call myself a golfer when I'm not, um,

busy with my two small kids at home

and,

uh, spending time with my family. That's what I like to do. So, I'm hoping one of these days I can make it to the finops, uh, X conference. Um, you know, maybe when my kids get a little bit older and

have a little bit more flexibility to travel.

So.

Alon Arvatz: Yeah, and I'm afraid to admit that I've never played golf.

Tim Nolan: Well,

we can

change that.

Alon Arvatz: Like I played top golf. Top golf I played, but like real golf in a real field, never. So that's something I should definitely do one day.

Tim Nolan: There you go.

Alon Arvatz: Maybe on finops X next year.

Tim Nolan: Yeah. Nice.

Alon Arvatz: That's great. And Tim, I'm sure a lot of people would love to reach out to you, ask questions, learn more What is the best way to do that.

Tim Nolan: Yeah. Uh, you can find me on the

finops

Foundation, slack community.

Uh, so Tim Nolan.

Alon Arvatz: Tim Nolan, that's great. You heard him guys Slack, reach out to him and you learn a lot of, uh, good, Uh, stuff. I personally learned a lot. I can tell you that besides the focus on impact and being efficient in finops operation, I really expand my horizons into what the finops roles. Include, and it's not only, uh, cloud and spend and working with internal stakeholders, but also thinking about where can make more impact when interacting with your cloud providers and maybe finops person needs to be skilled in negotiation and understand sales. so guys, go learn sales and go learn negotiation. You'll, it'll make you a lot better. FinOps people. Tim, Thank you very, very much, for joining us today.

Tim Nolan: Thank you very much, Alon.

Alon Arvatz: And I would also like to thank our listeners for joining us once again, I'm sure you had a great time just as much as I did. Don't forget to follow us and tell your friends.

Tim, thank you again. This has been another fascinating fun episode of FinOps in Action. See you next time.