FinOps in Action

FIA - Compilation
===

[00:00:00]

Intro: welcome to FinOps in Action. I'm your host, Taylor Houck. Each week I'll sit down with FinOps experts to explore the toughest challenges between FinOps and engineering. This show is brought to you by 0.5, empowering teams to optimize cloud costs with deep detection and remediation tools that actually drive action.

Pathik Sharma: two years ago I was having this conversation about, uh, AI and FinOps and the intersection of this, and this was before, it was coined AI for FinOps and FinOps for AI. And the conversation was sort of brushed off. It's like, "What are you talking about?

We're still figuring out cloud and, you know, for AI we are like way off," right? This was like three years ago. And now when we look at the survey data from FinOps Foundation, AI for FinOps, FinOps for AI, like these two sits right there, two things out of top three priorities that everyone is talking about

Taylor Houck: I think it just comes down to the acceleration of the [00:01:00] capabilities of these tools,

Yeah

back a couple years ago, it was fun. Some companies were finding valuable use cases, but it was more so just like a very small sliver of the overall cloud footprint.

Yes

But now we're seeing it accelerate so rapidly, even if it's still, let's call it 5%, 10%, whatever it is of your total cloud spend, if you look at the growth rate and the acceleration, you're realizing, holy cow, could end up being, you know, a material portion of my total cloud spend, potentially even surpassing my total cloud spend if

Yes

over, you know, the next coming quarters or even years

Pathik Sharma: Yes, yes. I, I think there was this graph shared at some point, which is like the pace of technology overall, what it could do in terms of decades, cloud was able to accelerate in, in, in terms of, uh, you know, number of years, like 10 years, 20 years, it was able to catch up with what technology has done for like last [00:02:00] 100 years.

And then AI is now catching up in one or two years that cloud has done in decades. So i- it's, uh, it, it's the pace as you've mentioned, right? And I think with that comes so many change, right? New products, new features, how your teams are using AI. Are you using AI or not? If you're not using AI, are, are we ho- holding ourselves back?

There is, there is this race in the industry that no matter what it is, you have to use AI. And I think to your point, two years ago, people are tinkering with it, it was a fun idea, it was an experimentation thing. Now we have enterprise customers who are leveraging AI and that is impacting their top line, bottom line.

It has become a larger portion of their P&L. So I think, I think things are getting serious much more quickly than ever before.

Emily Cornock: With ai, we're trying to build all of the FinOps principles into the very DNA of, of what we're [00:03:00] doing. So a lot of the things that I spoke about just before around tagging and, and transparency, and ownership and education, we're applying exactly the same way to ai.

And we are working really, really closely with our finance teams and with the kind of AI product owners to make sure that we're very, very clear on what we want to achieve and make sure that the FinOps team are helping them get clarity on if they are achieving that. If you go onto LinkedIn or you know, any news outlet at the moment, there's.

Lots and lots of stories around how companies are not seeing the ROI in in ai. That, you know, there's about 6% of people that have actually achieved an ROI and. If you look at how much money is going to be spent in ai, I think this year it was on track to be over 600, $650 billion, which is just an insane amount of money.[00:04:00]

And so if your organization is looking at AI and you're a pH practitioner, I recommend that you roll your sleeves up and, and get in there quickly because the data and the stories that we can tell with that data will let your organization know whether or not they are achieving that. ROI. Personally, I, as I said, I do think that we need to really embed this into. The genetics of, of what we are building with ai. And I love experimentation and I love innovation, but all good scientists would tell you that with experimentation, you are controlling your variables and you're changing one thing at a time or maybe two things at a time.

And it's important to have those controls in place for the rest of your experiment. And I think the cost controls. Are going to be super, super important. Even if you say you don't mind what the cost is, at least have the data to tell you what that is so you can understand it and understand whether or not that's acceptable for your business [00:05:00] and, and maybe it is, or understand where that might start to pose a risk to your business and to your customers and and to your staff.

we've been able to build a solution called Single View of Claims or, or Stock, which lets that claims handler get a full history of everything that's happened to a claims for that customer really, really quickly. Um, which is great because it means during a natural hazard event or during a a period of time where we've got lots of claims coming through, our claims handlers can help more customers during those times than ever before because they've got all the information they can tell the customer, the customers.

Hopefully happy that they've got an update. Maybe not so happy they've been updated by a, by a natural hazard, but they know we're there and we're supporting them, and then the claims handler can move on to the next one. And. It's delivered really cool stuff for our customers and it makes our claims handlers life a lot easier as well.

But behind the scenes, what it has been is actually identifying what is the business case, what are the benefits that we are going to drive in terms of [00:06:00] productivity, in terms of our customer engagement, in terms of how we're running our business. We've been very, very clear on what that business case is, and the FinOps team then have been able to provide a very clear.

Transparent view of every single cost that has been generated by this AR use case, and that is full end to end. So what are the Databricks costs for, you know, orchestrating this process? What are our open AI costs and our bedrock costs for consuming the tokens and generating these outputs? So we've got.

Dashboards now that basically say, for this AI model across all systems that have helped generate that outcome, what is the cost? And then our finance and our product teams can actually take that and compare it back to the business case and say, yeah, cool. Actually we've achieved our ROI.

Kevin Mueller: AI from a FinOps person, You know, I'm really fascinated with it. I'm really starting to dive deep on that one. I'm still just scratching the surface, and with the unit economics, the, [00:07:00] the most people, when they do to do ai, they're trying to solve a problem. Like, I want to get more productive, I want to increase conversion rates. Um, these are things that are adding value to the company, and you have to tie your AI usage to that value. And if you can't, you're just experimenting at that point.

Taylor Houck: I mean, this is exactly it. And by the way, you were saying a. Earlier that it's like you're, you're new to it. Well, the thing is, everyone is, is new to this. There's no one with 10 years experience managing the cost of AI workloads at enterprise scale. But it's funny because in the conversations that, that I've been having with, You know, big AI spenders right now, this is the main question is what is the value I'm getting out of this investment in ai?

How do I measure. The productivity gains because at this point, it, it, it, it's kind of intuitive that it is making you more productive, it's making the team more productive. But [00:08:00] how do you actually measure that and how do you make sure that your investments in AI are going to bear fruit at the end of the day, especially now that, You know, for the past couple years it's been easy to talk about AI but then kind of toss it to the side as not a big deal.

'cause it's still less than 5%, less than 2%, maybe even less than 1% of your overall cloud spend. But the growth that we're seeing right now, man, it's, it's exponential.

Kevin Mueller: when I'm advising people on FinOps for ai, it's, you have to understand why you're doing that, You know, and it is that you're trying to increase customer engagement by 40%. Okay, well, how do you measure that today and how are you gonna measure that tomorrow? I mean, the fact that you want to increase customer engagement, um, somebody in the finance or product knows how to measure that. And so now, okay, I keep measuring that same way, and I introduce this ai. Now I can. [00:09:00] Track the cost. And because I'm allocating to these products and features of customer engagement, I can say, well, this is what that cost is. And then however they're measur measuring customer engagement, is it going up or down? And if it's not moving the needle. You've just wasted a bunch of money, right? And if it is moving the needle, okay, great. What is that worth to you to increase customer engagement by 40%.

You

Zach Johnson: sometimes finops people like to be command and control. And, and sometimes it's more about being an, an enabler in a situation like the, the race to ai, right? So every company wants to enable ai. They wanna use ai. It's, the hottest thing on, on the marker right now. So sometimes it's more about how do we en enable the right dashboards and data sources to show where we're spending our time and in money, in ai, and, and rather than. Tie, tie it to like, what are you actually getting on your return for ai?

So sometimes it's, we don't yet understand that return on investment, so [00:10:00] we just need to enable teams to kind of utilize new technology and, figure out what that return on their investment is. So sometimes it's, it's not about controlling those costs, it's about enabling people to understand the data behind the money that they're spending and the business value that's hopefully driving for their organization.

we also try to go a layer layer deeper, right? You're enabling with your development teams with maybe a AI enabled I IDs or, or code generation. So. You think that's great, but how does that attribute itself to business value? Are we seeing, more commits, more features? Are we seeing less errors in our code, less pipeline failures?

'cause our, our code is better and hopefully our, our velocity is getting better. So we try and take it a, a, a layer deep beyond just reporting. You know, we're spending X dollars per engineer trying to connect it to that next layer of business context. We spent X dollars per engineer on these, these code generation or optimization tools, but we're actually getting in [00:11:00] return, features out to our customers quicker.

Our

Henrique: FinOps for ai. Right. And it's really, it is really interesting because it's just this continuous loop that AI gets more complex. And now we have, for example, these orchestrators that can automatically split your prompt into the best model.

Given the complexity of it. So now it's not even that You know which tokens you're gonna use. You have to build a resilient and, You know, a robust enough solution to accommodate for the fact that your prompt might be directed to different tokens and to different models. So you have to think on your feet here, and every time you think you okay, now I understood it, but then AI does something different and the companies build something different and you're trying to catch up.

So I'm very, I'm very bullish. I'm very excited about the future. Taylor. I do think that it's very important to have the guardrails in place. I don't see this as, You know, a silver bullet is gonna solve all the humanities issue. No, we do need to be very aware of it. We do need to be careful with it. We need to fact [00:12:00] check.

I also see people throwing, You know, summaries and analysis then by AI left and right and yeah. Have you opened the document? You know, have you actually looked through? And people come to me when I'm a consultant, You know, they say, oh, but I saw this optimization, blah, blah, blah, blah, blah. Oh, You know, that's my reservation for SQL Serverless instance, maybe.

But you can't. I mean, you can't, it's just not possible. So being able to fact check it is also, important. Very long answer. So first topic being, be curious about it. Go look it out. You know, use the terminal. There's all different flavors of it. You know, data sovereign, big hyperscalers, local, European, American, Brazilian, so many different versions.

Second. Think about every day how your role is changing in your organization with the advent of FinOps inside it, both yourself and also your colleagues around you. And then the third topic being. Be sure if your organization is using ai, that you're building the right monitoring tools to accommodate for how fast it changes.

It's better to have something [00:13:00] half made that you can isolate quickly than spending several months on building a dashboard that it may be two weeks. The company has changed its use case and the way they use AI is completely different in your dashboard is suddenly obsolete.

Oliver Milke: So we are doing on our team right now, we are trying to take AI to enable our users to understand the build better. So what we are doing is we are using the, the power BI copilot integration to allow our users, or to work on allowing our users in a hopefully very short timeframe to interact with the data. So they don't just see it in the dashboard, but they can ask like, what's important for me? What's happening right now? What has really changed? Or is it, is there any change that I need to be aware of? a dashboard might show you certain things. Yes. But even better is if you can just ask these questions, right?

What's important in it for me? So that's one way that we are using it or we are planning to use it in a very, very short timeframe from here. But I do think it can go way beyond it even. Right. So you can eventually. [00:14:00] it, take it all the way to Agen, where you can have maybe in the beginning, an agent looking through and actively reaching out to people on opportunities to potentially act on it itself.

In the end, I don't think that's happening yet, but I, I do see that as something that can happen in the not too

distant future.

Pathik Sharma: what is a safe way, for AI to basically take action?" And, and to give you an example, one, one example is cost allocation and labeling, right? Labeling generally is, uh, non-intrusive, non-disruptive operations. And if AI agent goes in and updates labels and make me more compliant from a cost allocation perspective, that's good, right?

It goes in, checks for labels. If it's missing, update it, based on, you know, X number of rules. And then it just keeps on running in the background. This-- Then there is a world where, you know, we-- the FinOps team doesn't have to work with the app [00:15:00] team to like co- continuously badger us like, "Hey, put your labels on that," right?

Because my cost allocation is being, uh, not, not accurate. So, so I think, I think that's one of the fantastic idea of where AI can actually make, a lot of strides. Now, the other spectrum is right sizing a workload in Kubernetes. We know that this is disruptive, which means the pod needs to restart, which means any ongoing transactions will be disrupted as well.

So you do not want AI to be taking that action, let's say, Black Friday, Cyber Monday for our retail customers, right? Like that, that becomes a lot of havoc. So, so I th- I think there is a, there is a healthy balance that can be done there. For example, one of the ways we are exploring is AI agent creating a pull request against your repo, so it's changing Terraform config file.

But then it's rerouting the request back to the application lead, because application lead still owns the application and [00:16:00] is accountable for the uptime and all the wonderful features that the team have built. So now the application team reviews the change that AI agents have made and then approve it, update it, or ignore it, right?

Um, and, and that's a fantastic way to get human in the loop. so, so I think, I think, I think at the end of the day, we are trying to figure out like, hey, think about AI as that eager intern that is, is like, "All right, I can do, I can do some work," but you also need to like trust but verify, check, wait for the manager to give you a nod, like things of those nature.

Shailaja Beeram: we are already using ai, so we are using copilot and we are really excited to use that and how it is, been helpful for us in our projects. So I think AI will improve visibility and identity patterns faster. But architectural awareness and human decisions will still be, very important [00:17:00] for any of the architectural things.

It's actually really, um, starts with an awareness. When engineer understand how their design decision impact cost over time, start making better choices.

Taylor Houck: With the customers that you're working with, how are they thinking about managing their AI spend as it grows across the organization?

Pathik Sharma: I think there are multiple layers to this. Um, one of the layers that typically what most people jump onto is the token cost, right? Which is your input token, your output token, and your thinking token that models are using to achieve a task or a goal. And right now, like we announced a s- flurry of, feature updates at Google Cloud Next on how you can manage it all through Agent Registry, through Agent Platform.

It shows all of your agents in one place, [00:18:00] and how can you track and manage cost in one place as well, right? But, but what we find that customers is barely even scratching the surface with token. Think about all the other rel- related costs that goes along with it, your storage cost, your storage adaptive layer cost.

If you're building your own models, maybe TPU, GPU cost, your compute. Um, so there is, there is lot more that goes into it than just the token cost, right? So how do you think about it overall, uh, through that lens con- thinking about all of those layers, right? Not just the model cost, but the platform where the model is hosted and the infrastructure beneath it where the platform is actually hosted and so on, so forth, because that covers the entire piece of it. So a retail customer, they, used, Gemini models, one of the Pro models to augment their search experience for their customers and item descriptions [00:19:00] and item pictures and all of that. Um, and they found incredible value.

Uh, so first of all, uh, by the way, value is also one of those, those things where people doesn't even think about, right? Like 200% increase in token consumption. Okay, but what did you achieve by doing that, right? I think that... So, so this retail customer, they found tremendous value in using AI. Their AI workload cost is 370,000 a month.

And the leadership is like, "Okay, that makes sense, but if we could optimize it a little bit, how, how do we think about it differently? How can we like radically just change the way we are doing AI?" And so they started experimenting with different models. Apparently, they started testing out with new Gemini Flash models, and they built a golden data set of, here are my use cases, here's my expected outcome for those use cases.

If the model performs at the expected outcome or above, then I would say this is a good model to use. [00:20:00] If it goes below, then we don't want, you know, customers to suffer. So they ran all of the series of experiments with Flash, and to their surprise, they found that model actually surpassed in all the golden data set that they had.

Wow

are like, "All right, I'm gonna switch the model from Pro to Flash." From 340,000. Now, again, they also made context caching, which was part of this as well. They also thought about provision throughput for the AI as well. So they made two or three different changes from a layer perspective. Their AI cost came down to $17,000 a month from 340.

And still, the product does the same brilliant job that it's supposed to do in reaching customer experience.

Henrique: but I've started programming, with gene ai and it is, uh, joy, I don't think I've ever used. So many of my free time hours into something,

I really hope that all FinOps practitioners and everyone in general is trying to [00:21:00] understand ai.

I think you need to face it as something that is here, and it's fundamentally changing the way we perceive work.

And I'm saying work is a very broad term here. I'm talking about. Companies, I'm talking about the way we relate to each other inside our jobs.

I'm talking about the tasks we are executed. I'm talking about value. We bring in how we measure that in organizations. So I do think that AI and specifically, You know, gen ai, be it with coding assistance or building up presentations, or, You know, incorporating the design rules of your company into your, documents automatically is going to change how we perceive those tasks and how we define value.

Marit Hughes: I don't know how much data we're gonna be sending out of our system and into a SaaS version of Snowflake. don't know how many AI tokens we're gonna use on the next thing. do you forecast for that? It's gonna be a little bit of best guess and like for the best and not letting fear cause me to triple my estimate . Right. Like put some reasonable [00:22:00] thought into it and some best guesses, document the crap out of them and move on. Right? Don't let analysis paralysis become the FinOps practitioner's primary source of fear.

Hey, I've got a pretty dashboard with the data, right? because more and more automation's gonna take care of the data itself. Right. Um, AI is gonna take care of a lot of that, uh, low hanging fruit, or it should, um, if your teams haven't become so fearful that they've put barricades around the low hanging fruit, fruit, deer, fencing, nobody can get in here. and I think that's where having that FinOps practitioner really starting to focus on those communication planning areas and starting that engagement. Way, way to the left. You know, not to be the person who quotes herself, right. If security is job zero, FinOps should be job 0.5.

Outro: That wraps up another episode of Fit Ops in Action. Thank you for joining. For show notes [00:23:00] and more, please visit fit ops in action.com. This show is brought to you by 0.5, empowering teams to optimize cloud costs with deep detection remediation tools that actually drive action.