AI on Kubernetes Is Just a Pod With a GPU (and That's the Problem)

AI-Assisted Development in CNCF Projects: Findings from Maintainers

What the TAGs Are Building and How Maintainers Plug In

Stop Overprovisioning for Application Startup

Observability 101: Capire prima di reagire

Zombie Workloads Haunt Data Center Efficiency Efforts

Agentic AI Threat Intelligence Essentials

Getting Started With Agentic AI for SecOps

Kubernetes: 5 Questions for Graziano Casto

Why goodput matters more than throughput for LLM serving

Practical Policy-as-Code with OPA and a Managed Sidecar Container

Platform Engineering Untold Truths: Is It Just an Infrastructure Matter?

An Open-Source First Step to Greenify Your Cloud Environments

A Policy-as-Code Approach to RBAC Authorization

WORKSHOP | The Game of Platforms: A Guided Journey into the World of Digital Platforms

Challenges and Takeaways of Managing AI Workloads on Cloud

Finding Observability and DevEx Tranquility Sailing the Monitoring Data Seas

From DORA to the SPACE and Beyond: Metrics to Drive Platform Engineering Initiatives

When Platform Engineers Meet SREs: The Birth of Observability-as-a-Service Superpowers

Observability-as-a-Service: When Platform Engineers and SREs Join Forces

Platform Engineers Assemble! Create Your FaaS Platform Using WASM

The Kubernetes Compass: Navigating Space and Time for Energy Efficiency

How Platform Engineering Streamlines the End-to-End Delivery of a Self-Hosted Agentic Architecture

ROUNDTABLE | When Platform Engineers Meet SREs: The Birth of Observability-as-a-Service Superpowers

In API We Trust: Circular Design for Scalable Platforms

Another Cluster Bites the Dust... and That's Just Fine!

Another Cluster Bites the Dust… E Va Bene Così!

Can an AI Agent Give a Talk? Let's Find the Speaker in the Room!

SLMs and the Agents Revolution: Rethinking How Software Is Built

Serving LLMs on Kubernetes: A Golden Path for Platform Engineers

PANEL | The Future of Developer Portals

Cloud Native Live: Kubernetes v1.35 Release

Your Cluster is Lying to ArgoCD (And How to Catch It)

The Self-Improving Platform: Closing the Loop Between Telemetry and Tuning

Beyond Deployment: Mapping the Reality of Day 2 Operations in Platform Engineering

Making Observability Work at the Edge

The Next Chapter of Developer Experience: TAG DevEx in Action

Da Zero a Cento: Costruire un’Infrastruttura che si Auto-Ottimizza

Making Efficiency a Platform Capability

Stop Hiding Kubernetes Pod Inefficiencies With Horizontal Scaling

PANEL | When Domains Collide: AI, Security & FinOps Inside the Platform Engineering Stack

ROUNDTABLE | OpenTelemetry in Production

Beyond the Data Swamp: Building Your Semantic Observability Mesh


[Published by Data Center Knowledge](https://www.datacenterknowledge.com/cloud/zombie-workloads-haunt-data-center-efficiency-efforts)

_An article by Data Center Knowledge featuring Graziano Casto on the growing problem of zombie workloads — idle or abandoned computational jobs that silently consume GPU capacity and power, undermining data center efficiency efforts at a time when AI infrastructure investment is at an all-time high._

Companies generally don’t post jobs for “Zombie Workload Hunter.” But the need is there. With each new technology generation, remnants of abandoned libraries, programs, services, and storage volumes continue to consume resources. Good sense eventually says it’s time to find and pare them down. What you don’t turn off will cost you. But finding them is a hunt.

These days, Cloud FinOps engineers, cost optimization engineers, and inventory managers do the seek-and-destroy work. This new engineering cohort uses observability, CloudOps, and FinOps automation to track resources that go by many names – orphans, strays, and, yes, zombies. They plan for new capacity and automate the decommissioning of idle assets – ideally before those assets show up on cloud bills.

This search for efficiency has never mattered more. A new AI wave has put the data center industry in the national spotlight, and GPU-based generative and agentic AI has focused public scrutiny on resource use. Enterprise teams, meanwhile, are putting new emphasis on the cost of computing.

## Why Zombie Workloads Persist in Hybrid and Multicloud

Zombie workloads can quietly bleed power, rack space, and budget across multicloud and on-premises environments. They often follow a common pattern, Roger Strukhoff, chief research officer at IDCA, told Data Center Knowledge. Cloud zombie workloads frequently trace back to “individuals or departments that stop using apps but never delete them,” he said. “They appear when internal organizations are consolidated, or companies are acquired, and no one is tasked with cleaning up unused cloud instances and applications.”

According to Strukhoff, IDCA research indicates that as much as 13% of US cloud usage comes from such zombie workloads.

FinOps tool providers like Broadcom (VMware Aria Cost, formerly CloudHealth) and AWS (e.g., Cost Explorer, Compute Optimizer) – often enlisted to assist with zombie cleanup – estimate the overall cloud waste even higher, commonly at 25% to 30% or more, with orphaned volumes and forgotten compute instances frequently in the mix.

The problem is stubborn, and ongoing shifts in application architecture don’t make it easier.

## From Scale-to-Zero and Beyond

Today’s most common solutions to cloud clutter trace back to the famed Unix kill process. In serverless architectures, scale-to-zero configurations enforce runtime limits so that idle services stop consuming resources and incurring costs. But there are tradeoffs. Mistakenly targeting a live service can lead to cold starts and unwanted latency in spinning up instances. For certain spiky traffic patterns, that can be a problem.

## FinOps and Observability

A growing array of cloud-native and platform enhancements aims to take up the slack. Vendors such as Google, Flexera, Datadog, and IBM offer tools that correlate billing and performance metrics to highlight charged resources with zero use; continuously index assets to detect inactivity and broken parent-child services; and apply policies with auto-remediation to tear down processes that lurk unnoticed in data center environments.

## Microservices Made Zombies Harder to Kill

The endlessly running program, of course, predates the cloud. It dates to mainframes and persisted through client/server and virtual machines. A tried-and-true operating model was to “pull the plug and wait for somebody to scream,” said Eric Newcomer, analyst at Intellyx and a veteran of middleware and distributed computing.

That changed when cloud-native microservices gained traction. “Cloud computing brought with it ‘headless’ services that run in the background to serve up some data to another service or the like. A lot of such microservices-based applications can be composed of even hundreds of services,” Newcomer said. “If the application stops, the zombie services might still keep going – and do nothing.”

Cloud computing pioneered the use of commodity hardware, massive scale, aggressive replication, and container orchestration tools like Kubernetes. But these innovations don’t easily map to today’s LLMs and agentic AI. The GPU-based hardware behind modern AI is far different from the commodity setups built for cloud-style distributed computing – and AI zombies cost more, in both money and electricity.

Newcomer noted that, under the auspices of the Linux Foundation and the Cloud Native Computing Foundation, Kubernetes is evolving to better support AI workloads, where new kinds of zombies emerge. Kubernetes innovations are spreading across new AI-focused cloud providers, but much of this work is still in its early stages.

## It’s a New Age: GPU-Era AI Raises the Stakes

Supporting AI on Kubernetes is complicated by the moving-target nature of generative and agentic AI. There’s still a lot of R&D. Teams routinely encounter long-running jobs, mid-flight pipeline crashes, and orchestration scripts that fail to clean up jobs. Add to that the complexity of managing model weights, data ingestion, and more.

That creates a very different workload profile, according to Graziano Casto, a developer relations engineer at AI optimization platform maker Akamas and a Cloud Native Computing Foundation Ambassador, in an email exchange with Data Center Knowledge.

“GPUs are more expensive than CPU cores, so inefficiencies that were just rounding errors on a cloud bill become very real numbers. A GPU sitting idle isn’t a minor waste,” Casto said. “What changed with the LLM era is that the cost of ignoring inefficiency went up by an order of magnitude almost overnight.”

> "What changed with the LLM era is that the cost of ignoring inefficiency went up by an order of magnitude almost overnight.” – Graziano Casto, developer relations engineer at Akamas and Cloud Native Computing Foundation Ambassador

While expected in R&D, that inefficiency becomes a major problem in production, introducing a costly new strain of zombies: idle or abandoned GPUs.

“Kubernetes is growing into this space, and it's happening fast, and the ecosystem is visibly building new primitives to catch up,” Casto said. “But it's still exposing every place where the platform’s original assumptions were built for a cheaper, more elastic kind of workload than the one it’s increasingly being asked to run.”

Kubernetes is evolving for AI with dynamic resource allocation for specialized hardware and smarter batch job scheduling that accounts for the topology and locality requirements many AI jobs have.

Not surprisingly, efficiency monitoring and zombie workload hunting now rely on vigilant GPU health and utilization checks. Chip-level observability is becoming table stakes for smooth operations.

Casto pointed to NVIDIA’s Data Center GPU Manager (DCGM) as a monitoring mainstay, useful for gauging how busy the GPU appears to be. “The catch is that ‘appears to be’ part," he said. “A GPU can look busy on a DCGM’s utilization number while it is actually just sitting there waiting on [some other action], like data that hasn’t arrived yet, or another GPU running behind on the same job.”

Network activity can be a blind spot in these setups. That’s why ongoing OpenTelemetry standards work matters, Casto added: different tools must interoperate to correlate performance signals. Those efforts are creating a common vocabulary for this AI job monitoring, he said.

## Back to Basics: Policies and Ownership

The hunt for new zombie variants shouldn’t overshadow the basics, Strukhoff emphasized. Organizations need clear policies to eliminate zombie workloads. “Individual users must be reminded to close their instances when finished with them. Or better, organizations should institute monitors that remind administrators of zombie workloads and set policies to close those that are not being used,” he said.

In practice, that means routine environment-wide scans of all usage and a cleanup methodology – especially after internal consolidations or acquisitions. The tools are improving, and the workloads are changing, but the rule remains: what you don’t turn off will cost you.



[Published by Codemotion](https://www.codemotion.com/magazine/devops/cloud/kubernetes-5-questions-for-graziano-casto-codemotion-ai-and-tech-conference-2026/)

_An interview with Graziano Casto ahead of the Codemotion AI and Tech Conference 2026, exploring Kubernetes' In-Place Pod Resize feature and its implications for resource efficiency and developer experience._

Ahead of his talk at [Codemotion AI & Tech Conference 2026](https://conferences.codemotion.com/milan/?_gl=1*1fuwhy3*_gcl_au*OTQ5NDM3MDYwLjE3ODk5Nzc5MDAuLS4tLjE3ODk5NzgzNTMuMTEyMjczMjk1Ni4xNzg5OTc4MzU0LjE3ODk5NzgzNjU.*_ga*MjA1NjE0NDA3Mi4xNzg5OTc3ODg3*_ga_C9WPJMT5LH*czE3ODk5Nzc5MDAkbzEkZzEkdDE3ODk5Nzg0ODUkajEkbDAkaDA.*_ga_52S30H0VCG*czE3ODk5Nzc5MDAkbzEkZzEkdDE3ODk5Nzg2NjYkajYwJGwwJGgw&_ga=2.3524601.836988764.1789977887-2056144072.1789977887), we sat down with [Graziano Casto](https://castograziano.com/) to talk about one of Kubernetes’ quieter but consequential recent additions — In-Place Pod Resize — and what it actually means for cloud spend, developer experience, and where the platform still falls short. Here’s what he had to say.

## Kubernetes

**Most developers have heard “just give your pods more resources” as the default answer to slow startup times. What’s the real cost of that advice at scale — and why has it taken this long for Kubernetes to offer something better?**

Throwing more CPU or memory at a pod is the easiest lever anyone can pull, and that’s exactly the problem. It works in the moment because more resources usually do shave a few seconds off a cold start, but nobody goes back later and tunes it down once the deadline pressure is gone. Multiply that by every service in a large fleet and you get a cluster running at a fraction of its real utilization, because requests were set for the worst-case startup spike rather than steady-state load. Teams end up paying for headroom they use for maybe thirty seconds at boot and then never touch again, and that shows up directly in the cloud bill and in how many nodes you need just to fit everyone’s inflated requests.

The reason Kubernetes couldn’t offer a cleaner answer for so long comes down to how the resource model was built from day one. Pod resource requests and limits were treated as effectively immutable once the pod started running, so if you wanted different resources for startup versus steady state, your only option was to kill the pod and bring up a new one with new values — which obviously defeats the purpose if what you’re trying to fix is slow startup in the first place. Changing that meant touching the scheduler, the kubelet, the API machinery, and every controller that assumes resources don’t move once a pod is admitted, which is a lot of surface area to get right without breaking existing workloads. That’s why in-place resource resizing took years of design work through the KEP process rather than being a quick patch. It’s only recently gotten stable enough that you can actually resize a running pod without a restart, which finally gives people a real alternative to just padding the numbers and hoping for the best.

## Cloud Costs

**In-Place Pod Resize sounds like an infrastructure detail, but you mention it could free up thousands of wasted cores across a cluster. Can you put that in terms a developer or engineering manager would feel in their cloud bill?**

Think about what actually happens today without in-place resize. A service needs four CPUs for the first twenty seconds while it warms up caches and loads config, then it settles down and uses maybe half a CPU for the rest of its life. Since you can’t change that request without a restart, you set it at four CPUs permanently, because nobody wants a pod that gets OOMKilled or throttled during startup in production. Now multiply that by however many replicas you run and however many services look like this across the fleet, and you’ve got a huge chunk of your cluster capacity reserved for something that’s only true for a few seconds out of the pod’s entire lifetime.

That reserved capacity isn’t free just because it’s idle. The scheduler still has to find room for those four CPUs on some node, so you end up provisioning more nodes than the actual workload needs, and those nodes cost money whether the CPUs are busy or sitting there as a safety margin. With in-place resize, you can request something closer to what the pod actually needs on average, let it burst up temporarily during startup, and then scale it back down automatically once it’s stable, all without a restart. For an engineering manager looking at a cloud bill, that’s the difference between provisioning for peak-forever and provisioning for what you actually use, which on a large fleet can mean cutting node count — and therefore spend — by a meaningful percentage without touching a single line of application code.

## Developer Experience

**You work on both the Kubernetes release team and developer experience at CNCF. Do features like this one actually change how developers think about writing applications, or does the “right” behaviour still get pushed down to platform teams to figure out?**

Honestly, in most organizations this still lands on the platform team, and I don’t think that’s a bad thing right now. In-place resize is a primitive, not a policy, and application developers generally shouldn’t need to know that Kubernetes finally lets you change CPU without a restart any more than they need to know how the scheduler binpacks nodes. What changes is that platform teams now have a lever they didn’t have before, so instead of telling application teams “just bump your requests and don’t worry about it,” they can build an internal system that sets a lower baseline request and lets it flex up automatically when a service is actually starting up or under load. The developer experience improvement isn’t that every engineer suddenly understands resource tuning — it’s that the platform can stop asking them to guess at numbers they were never equipped to guess at accurately anyway.

Where it does start to change developer behavior is at the edges, mostly for teams building their own operators or anyone working close to the workload API, because now there’s a real reason to think about resource profiles as something dynamic rather than a static number you set once and forget. And that’s actually the pattern I care about most from the DevEx side. A feature like this only pays off broadly if the tooling and defaults around it are good enough that most developers never have to think about it at all, and the harder cases get handled by people who actually understand the tradeoffs. So the short answer is that the primitive is what unlocks the smarter platform behavior, but the platform team is still the one that has to build that behavior and expose it as something sane — whether that’s a VPA-style controller, a policy engine, or an internal tool that just does the right thing by default.

## Tech Conferences

**You contributed directly to Kubernetes v1.35 and now you’re here explaining it on stage. How important are conferences like Codemotion in closing the gap between “the feature shipped” and “developers actually know it exists and how to use it”?**

Honestly, that gap is bigger than most people realize, and it’s one of the reasons I care so much about doing this kind of talk. A feature can go through years of KEP design, implementation, testing, graduate to stable, get its release notes written, and still sit there almost invisible to the person who could actually benefit from it, because the release notes are one line in a changelog most developers never read, buried among forty other changes that shipped the same cycle. Being Comms Lead for 1.35 made that really obvious to me. We spend enormous effort making sure a feature is technically correct and well tested, and comparatively little effort making sure someone building a service on Tuesday afternoon actually knows the option exists.

Conferences like Codemotion close that gap in a way that documentation alone just can’t, because you get to show the actual cost in terms people recognize from their own bill or their own on-call rotation, rather than a paragraph in an API reference. Someone can read “KEP-1287, in-place pod vertical scaling, promoted to stable” and feel nothing, but if I show them the same idea framed as “your JVM warm-up is costing you idle cores you’re paying for right now,” that lands differently and sticks. It also works in both directions. I don’t just get to explain the feature, I get to hear from the room whether it actually solves their problem or whether there’s a rough edge nobody flagged during the KEP review, and that feedback goes straight back into the project. So I’d say the real value of a conference talk isn’t distribution in the marketing sense — it’s translation, taking something that shipped correctly and making sure it actually gets used correctly too.

## What’s Next

**You describe yourself as “trying to make the cloud faster, greener, and significantly less annoying for everyone.” If In-Place Pod Resize is one piece of that puzzle — what’s the piece that’s still frustratingly missing?**

What’s still missing, honestly, is that these pieces still get treated as separate problems when they’re really the same problem seen from different angles. HPA and VPA are a good example. HPA decides how many replicas you need, VPA decides how big each one should be, and the two don’t actually coordinate, so you can end up with both firing on the same spike, scaling out and up at once when one of those alone would have been enough. There’s early work starting to bring these together into something closer to a genuinely multidimensional scheduler — one component that reasons about both axes at once instead of two controllers reacting independently to the same signal — and I think that’s going to matter a lot more than either optimization on its own once it actually lands.

The second piece is that even with a smarter scheduler, the platform still has no visibility into what the application runtime is actually doing inside the container. It sees CPU and memory at the cgroup boundary, but it has no idea what the application runtime is doing inside the container, so resource decisions get made on numbers that are technically accurate but don’t explain why the application is behaving the way it is.

And that connects to the third piece: PSI, pressure stall information. As of Kubernetes 1.36 the PSI metrics themselves are stable and exposed by the kubelet, so the data problem is basically solved. What’s still missing is that the schedulers and autoscalers most people actually run — the built-in HPA and VPA — don’t natively reason off that signal yet. Today it’s mostly custom controllers pulling PSI in on their own rather than it being the default input the ecosystem scales on, and until that changes, you’re still making resource decisions based on a partially complete signal.



[Published by CNCF](https://www.cncf.io/blog/2026/07/20/why-goodput-matters-more-than-throughput-for-llm-serving/)

When we benchmark an LLM serving setup, the number almost everyone reaches for first is throughput: how many requests per second the system can push through. It is easy to measure, easy to compare, and it lines up nicely with the thing finance actually cares about, which is dollars per request. So we chase it, tuning batching, raising concurrency, squeezing the GPU, and watching requests per second climb.

The trouble is that this number, on its own, can keep rising while your service quietly gets worse. I ran into exactly that while tuning vLLM on a single GPU and the gap was wide enough to be worth walking through because the lesson travels far beyond my particular setup.

## Throughput counts plates, goodput counts happy diners

Picture an LLM endpoint as a restaurant kitchen. Throughput is the number of plates that leave the pass every hour. It tells you the kitchen is busy but it says nothing about whether the food arrived hot, on time or at the right table. A kitchen can double its plates per hour and still send a full room of cold and late dinners.

The metric that captures what we actually want is [goodput](https://haoailab.com/blogs/distserve/). The definition is refreshingly plain: it is the number of completed requests per second that also meet your latency targets. For LLM serving those targets are usually time to first token (TTFT), the wait before anything appears on screen, and time per output token (TPOT), the pace at which the rest of the answer streams out. A request that finishes but blows past either target is a plate that left the kitchen cold. It still counts toward throughput but it does not count toward goodput.

## The setup, and how I drove load

Everything here runs on parts you probably already have. I served Qwen2.5-7B with vLLM on a single NVIDIA A10G with 24 GB of memory inside an EKS cluster, drove load with [GuideLLM](https://github.com/vllm-project/guidellm), and watched the system through Prometheus and Grafana with GPU level metrics coming from NVIDIA’s DCGM exporter. The entire setup is on GitHub, Kubernetes manifests and all, so you can reproduce my runs and, more usefully, aim the same benchmark at your own model and real traffic instead of the synthetic load I used.

One honest note before the numbers. A single A10G is not the GPU you would pick for a production-grade LLM service, so the absolute numbers here will look modest next to an H100 cluster. I picked it on purpose to keep these studies cheap enough to repeat often. The point of this post is not the ceiling, it is the shape of the trade-off between throughput and latency, and that shape holds regardless of how big the GPU is.

I focused on changing only three vLLM settings: `gpu_memory_utilization, the batched token budget (max_num_batched_tokens)`, and the cap on `concurrent sequences (max_num_seqs)`. I picked these three on purpose because together they decide the throughput against latency trade-off, they control how much GPU memory is left for the KV cache, how much work goes into each batch step and how many requests run at the same time. They are also the settings an operator can safely change in production without touching the model, the hardware, or the cluster, so holding everything else fixed keeps the results easy to read and easy to reproduce.

How you send the traffic matters just as much as how you tune it, so it is worth being explicit about it. I used GuideLLM in two ways. A rate sweep ramps the request rate up gradually and lets me read latencies while the server still keeps up, so the numbers reflect real serving and not a backlog. A throughput run does the opposite, firing requests as fast as the server will take them to find the ceiling, which pushes it into saturation. I will use both in this post, and the contrast between them is part of the lesson.

## What the studies show

I ran the same setup against three kinds of traffic because the shape of the traffic decides which latency target actually bites. A chatbot sends a fairly short prompt and gets a fairly short answer. A reasoning workload sends a short prompt and gets a very long answer, on the order of 4,000 tokens. An agentic workload sends many short calls one after another to finish a single task.

| **Workload** | **Prompt / Output (tokens)** | **What the SLO hangs on** |
| --- | --- | --- |
| Chatbot | ~485 / 121 | tight TTFT, medium TPOT |
| Reasoning | ~400 / 4000 | TPOT, with TTFT loose |
| Agentic | ~512 / 128 | per-call TTFT that adds up |

The chatbot study is the clearest case, so start there, and start with what I actually asked it to do. The goal was to maximize combined token throughput, `prefill_token_throughput + decode_token_throughput`, under one constraint: average TTFT had to stay at or below 1.5 seconds. I also added a windowing rule, scoring each config from a stretch of eight consecutive data points where prefill throughput had settled down rather than from any single sample. The reason is simple and worth keeping in mind for your own benchmarks too. A live metric like token throughput jitters as individual requests start and finish so one sample can land on a lucky peak or an unlucky dip. Requiring eight steady points in a row is an easy way to score the system’s real running rate instead of a blip.

You can see that jitter, and the windowing rule doing its job in the raw metrics from two configs I will use as our main example. Both ramp up over the run as GuideLLM raises the load, both wobble step to step and both still let you read off a believable plateau once the ramp settles which is exactly the stretch the windowing rule is built to find.

![Image one of the raw metrics from the two configs used as the main example.](https://www.cncf.io/wp-content/uploads/2026/07/image-13.png)

![Image two of the raw metrics from the two configs used as the main example.](https://www.cncf.io/wp-content/uploads/2026/07/image-15.png)

![Image three of the raw metrics from the two configs used as the main example.](https://www.cncf.io/wp-content/uploads/2026/07/image-16.png)

| **Config** | **gpu_memory_utilization** | **max_num_batched_tokens** | **max_num_seqs** | **TTFT avg** | **Prefill tput** | **Decode tput** |
| --- | --- | --- | --- | --- | --- | --- |
| Exp 1 | 0.89 | 4160 | 212 | 1395 ms | 1925 tok/s | 478 tok/s |
| Exp 20 | 0.87 | 2150 | 271 | 1403 ms | 2934 tok/s | 690 tok/s |

Both configs land at almost the same TTFT, right under the 1.5 second ceiling I set. That is not a coincidence, it is the constraint doing exactly what I asked it to do: hold the first-token wait at the edge of what I said was acceptable then squeeze as much combined throughput out of the rest of the system as it can. Exp 20 wins that squeeze by a wide margin, with about **50 percent more combined throughput** than Exp 1.

That extra throughput is not free, and the bill shows up somewhere I did not constrain. Exp 1’s **p95 TPOT sits at about 50 milliseconds**. Exp 20’s sits at about **494 milliseconds**, nearly ten times higher. I told the search to protect the first token. I never told it to protect the pace of the stream that follows, so it spent that budget without asking. If your target user is reading a chat response as it streams in, a half-second gap between tokens is the difference between a smooth reply and one that visibly stutters even though both configs kept their promise on the metric I actually constrained.

The two latencies move in different directions because they come from two stages of the same request. TTFT comes out of the prefill stage, where the model reads your prompt, and bounding it directly is what keeps both configs near the same 1.4 second mark. Every token after that comes from the decode stage, where vLLM advances all the active requests at once, one token to each per step. Pushing more combined throughput means packing more work into each of those steps, and a fuller step takes longer to run, so every request gets its next token a little later. Picture the kitchen sending one course to every table each round: a fuller room makes each round longer and every table waits more between courses. Constrain how fast the kitchen seats you and brings the first course, and the wait gets pushed onto the gap between the courses that follow instead.  
  
Reasoning and agentic ran a different experiment on purpose. I kept the same goal, maximizing combined prefill and decode throughput, but with no TTFT constraint at all, and I drove load with a throughput run instead of a sweep, sending requests as fast as the server would take them to find the ceiling. The chatbot study answers “how much throughput can we get while keeping the first token fast”. These two answer “how much throughput is actually available”, which is a fair question on its own, as long as you read the latency that comes with the answer rather than assuming it does not matter.

It still does not come for free, and it still varies between configs that post similar throughput. In the agentic study, the two best configs I found were nearly tied on requests per second, 6.20 against 6.11, yet their p95 TPOT differed by about 220 milliseconds, 682 against 902. Same throughput, a noticeably different stream once you are inside one of those many chained calls. In the reasoning study, where each answer runs to about 4,000 tokens, the config with the best raw throughput was not the one you would want once you cared about latency: asking the same search to respect a tighter inter-token target pushed the winner to a different configuration entirely. Three different studies, three different ways of asking the question, and the same answer comes back each time: a throughput number with nothing else attached does not tell you which configuration to run.

## Optimize for the thing you actually sell

Three lessons come out of this for anyone running inference in production.

First, a throughput number with no SLO next to it is marketing, not engineering. The honest target is goodput, meaning requests per second that actually met your latency targets. Count anything else and you are counting cold plates.

Second, the config with the best goodput is not the one you would guess. In my reasoning runs, two settings that looked almost the same ended up far apart and one nearby setting failed outright. The space is full of small cliffs like that, so hand tuning tends to stop at the first number that looks good instead of the best one. The reliable way through it is an automated, data-driven search that tries many configs against your real targets rather than a few hand-picked guesses.

Third, none of this stays put. Traffic, prompts, and models all change over time, and the best config moves with them so the search has to run again, not once.

There is a fourth point worth making even though my study did not test it directly. I changed only three serving knobs and the gap was already this wide. The same trade-off lives in the layers I held fixed too, in the GPU settings, the runtime, and the scheduling, and the largest gains tend to come from tuning the whole stack together rather than one layer at a time since the layers interact.

The practical move is small and worth doing early. Set your TTFT and TPOT targets first, then look for the config that gives you the most throughput while still meeting them instead of chasing throughput on its own. You will often ship a smaller requests-per-second number and a clearly better service, the trade your users would make every time.

If you want to see the gap on your own hardware the full setup is on GitHub at [github.com/graz-dev/vllm-benchmark](https://github.com/graz-dev/vllm-benchmark). Clone it, point GuideLLM at your own model and traffic, set your SLOs, and search for the best goodput rather than the best raw throughput. That gap is usually where your real headroom is hiding.


[Published by Akamas](https://akamas.io/resources/gpu-utilization-llm-inference-optimization/)

AI is reshaping every industry, and agents are at the center of that shift. Many companies are racing to put them to work building assistants that handle customer requests around the clock, pipelines that classify and route data without human input or systems that reason across tools and act on their own. The appeal is obvious. What’s easy to miss in all the excitement is that every one of those agents, no matter how capable it looks from the outside, runs on a model, and that model runs on a GPU that is the engine keeping the whole thing moving.

Enterprises are pouring billions into GPUs to run LLMs and AI agents. For the first time in computing history, we are seeing data-center build-outs at a scale never witnessed before, where the GPU portion alone can account for as much as [70% of total investment](https://www.innovationendeavors.com/insights/future-data-centers), according to recent industry estimates.

Then the first real utilization reports come in. The number is hard to ignore: GPU usage as low as 5%. You might be spending millions on accelerators while [95% of that capacity sits idle](https://akamas.io/resources/gpu-utilization-llm-inference-optimization/#:~:text=AI%20is%20reshaping,the%20same%20hardware.) and yet almost every platform and engineering leader we talk to is trying to do the same thing: buy more GPUs to support the GenAI use cases moving into production this year. **The bottleneck, as it turns out, is usually not capacity but configuration.** On a stack as deep as the GenAI one, the gap between a default setup and a properly tuned one can easily reach 2x throughput or more on the same hardware.

![](https://akamas.io/wp-content/uploads/2026/06/AI-factory-ROI-equals-GPU-utilization.png)

This is the central tension we explored in our recent [webinar](https://akamas.io/events/webinar-genai-optimization-kubernetes/) with Daniele Zonca and Roland Huß from Red Hat, who quite literally wrote the book on running generative AI on Kubernetes.

## We’ve Already Seen This Pattern with CPUs

Here’s a number that should worry anyone treating GPUs the way they treated CPUs. Across one of our customers, with several dozen Kubernetes clusters and around 60,000 CPUs, average utilization over a full day was **5%**. At cloud prices, that adds up to roughly $15 million a year spent on resources that are almost entirely idle.

Those are CPUs. Now apply the same operational habits (over-provisioning, ignoring efficiency, assuming the answer to every capacity question is “add more”) to hardware that costs roughly 40x more per unit and is genuinely scarce. In much of Europe you simply can’t procure H100s on demand, no matter how big your budget is. If 5% utilization on cheap, abundant CPUs costs **$15 million a year**, the equivalent waste on GPUs is an order of magnitude worse. It can quietly block the ROI of an entire GenAI initiative.

## Why GPUs Are So Easy to Waste

GPUs are nothing like CPUs, and that difference is exactly why they’re hard to keep busy.

A **CPU is built for latency**: most of its silicon is dedicated to control logic and caches that predict what comes next, so that a handful of threads can execute general-purpose work as fast as possible otherwise a **GPU is built for throughput**: almost the entire chip is devoted to computation, with thousands of cores optimized for the matrix multiplications that LLM serving depends on. Think of a CPU as a fast motorbike and a GPU as a cargo ship.

The catch is feeding that cargo ship. Every computation requires data to move from high-bandwidth memory (HBM) into the cores, and that transfer frequently becomes the real bottleneck, which is the first reason GPU utilization ends up so low.

This gets sharper once you look at how inference actually runs, across two very different phases:

1. **Prefill** processes the entire input prompt in parallel to produce the first token. It is compute-bound and drives your time to first token (TTFT), which is the waiting time your user feels before anything appears on screen.
2. **Decode** generates tokens one at a time, each step reusing the KV cache built from all previous tokens. It is memory-bound and drives your time per output token (TPOT), which measures how fast the response streams out.

Because the same hardware handles two completely different access patterns, optimizing for one does not automatically improve the other. That mismatch is a core source of complexity in the whole stack.

## The Trade-off You’re Probably Losing On

NVIDIA’s Jensen Huang has popularized a chart worth understanding. When you plot total token throughput (your revenue proxy, since AI services are priced per token) against per-user interactivity (latency), you don’t get a single best point. You get a **Pareto front**: a curve of configurations where improving one dimension always costs you something on the other.

![](https://akamas.io/wp-content/uploads/2026/06/Extracting-2x-throughput-from-the-same-GPUs.png)

Push for maximum throughput and single-user latency suffers. Push for fast responses and you sacrifice a large share of GPU capacity.

Your use case fixes one constraint on that curve. A chatbot needs to emit tokens faster than human reading speed (around 200ms per token) or it feels sluggish and users leave. That means you have a fixed latency limit you cannot cross.

But along that line sit hundreds of possible configurations, and most teams are unknowingly sitting on a suboptimal one. A single misconfiguration can cut effective throughput in half, leading teams to conclude they’re maxed out and order more GPUs, when in reality they’re using only half of what they already paid for.

With proper tuning, doubling the throughput from existing hardware is not unusual. That is the real opportunity.

## The Playbook: Getting 2x from the GPUs You Already Have

The GenAI stack has hundreds of tuning knobs, from the model down to the kernel and Kubernetes itself. That depth is the real challenge. The skills needed to navigate it effectively are genuinely hard to find, which means most teams end up working with defaults that no one has had the time or expertise to question. Here are the levers with the highest payoff.

![](https://akamas.io/wp-content/uploads/2026/06/The-full-stack-LLM-inference-optimization-layers.png)

1. **Right-size the model, then quantize it**

Defaulting to the biggest model for every request is rarely the right call. Smaller and reasoning-tuned models can match quality on the right tasks, and a well-designed agent framework can maintain accuracy while running a smaller model. Quantization can compress a model to as little as 25–30% of its original size while preserving most of its accuracy, which is a significant reduction in resource use for almost no quality cost.

2. **Route intelligently and exploit cache locality**

Pick the right model per request, but also send follow-up messages to the replica that already holds the relevant KV cache. Multi-turn chat and agentic flows send the whole growing conversation back and forth, so cache-aware routing is critical to avoid recomputing work you’ve already done.

3. **Tune the inference engine and the parallelism strategy**

The engine layer around vLLM (or any other inference server) is evolving fast. It pays to match the parallelism strategy to your model:

- *Data parallelism* (replicas behind a load balancer) scales concurrency but not single-request latency.
- *Pipeline parallelism* splits a too-big model across nodes by layers.
- *Tensor parallelism* splits within a node using NVLink/NVSwitch to actually improve latency.

You can combine pipeline and tensor parallelism to get the best of both, but neither is free of trade-offs. Pipeline parallelism is constrained by inter-node network latency, which can be 25–50x slower than staying on a single node. Tensor parallelism introduces its own synchronization overhead across the NVLink fabric inside the node. The right combination always needs to be validated against your specific setup rather than assumed to work out of the box.

4. **Match hardware to the bottleneck**

If your constraint is in the decode phase, the real problem is memory bandwidth. Buying the GPU with the most teraflops will not help. Disaggregated serving (separating prefill and decode pools, as projects like llm-d enable) lets you scale each phase independently and route large inputs to dedicated prefill instances. As inference scales out, expect training-grade networking such as RDMA and InfiniBand to start mattering for inference as well.

5. **Partition GPUs with care**

If a GPU is allocated but underused, you may be able to host a second model on it, but GPUs have both a compute side and a bandwidth side. Partitioning can create a noisy-neighbor effect: spare capacity on paper does not guarantee you can add load without degrading both workloads. Always validate the impact before packing more onto a single card.

6. **Measure what you optimize**

None of these changes are safe to make without proper visibility. Track LLM-specific metrics (TTFT, TPOT), lean on the OpenTelemetry semantic conventions emerging for GenAI, and use tracing tools like MLflow to follow non-deterministic, multi-hop agentic requests. This also gives you the data for internal cost allocation across teams.

GPU-level metrics deserve special attention because they are significantly harder to interpret than application-level ones. Tools like NVIDIA’s DCGM give you utilization numbers, but those numbers are often too coarse to identify whether your actual bottleneck is compute, memory bandwidth, or something else. GPU memory bandwidth, for instance, cannot be directly measured from DCGM at all: vLLM can expose an estimation through the –enable-mfu-metrics flag, but it remains an approximation. To make things more confusing, “GPU utilization” means very different things depending on which layer of the stack you’re looking at (allocation, kernel execution, or model FLOP/s), and conflating them leads to wrong conclusions.

7. **Load test with your own traffic, not a vendor benchmark**

Traditional tools like k6, JMeter, and Locust work at the application level, but purpose-built tools like GuideLLM (part of the vLLM project) let you build realistic, record-and-replay workloads that reflect how your system is actually used. The most important point here is to always build your test dataset from real traffic, because a vendor’s benchmark on a foundation model says nothing about how the system will behave under your specific workload. The optimization space for GenAI is too workload-dependent for generic results to transfer.

## Too Many Knobs for a Human to Tune

With hundreds of knobs spread across three fundamentally different layers (the model, the inference server, and the GPU itself), millions of possible combinations, and a target that shifts with every new use case and every new model release, this is not a one-off exercise you do at launch. It is a continuous practice.

Each layer also requires different expertise. The people who understand model selection and quantization are rarely the same people who can tune vLLM internals or reason about GPU memory bandwidth, and finding someone who spans all three is genuinely uncommon. That makes this well beyond what any team can manage by hand.

The emerging answer is to use AI to optimize AI systems: autonomous tuning that explores the configuration space and finds the settings that maximize throughput while staying within your latency and cost targets. This is exactly the problem [Akamas](https://akamas.io/akamas-ai-infrastructure-optimization-launch/) was built to solve, first for traditional Kubernetes workloads and now for GenAI inference, and we have something significant coming on that front that we will be sharing very soon.

## The Bottom Line

As enterprises move AI agents into production and pour significant capital into the GPU infrastructure to support them, the pressure to show returns on that investment is growing fast. But utilization tells a different story, and the 5% figure is not a hardware problem. It is what happens when a genuinely complex stack gets deployed with default settings and no one has the time to tune it properly.

The good news is that the same depth that makes the GenAI stack hard to configure also creates real headroom. The difference between where most teams are parked today and what their existing hardware can actually deliver is large enough to change the ROI equation entirely.

You do not need more GPUs to get there. You need visibility into your actual bottleneck, a realistic picture of your workload, and a systematic way to explore the configuration space. Everything else in this article follows from those three things.







[Published by PlatformEngineering](https://platformengineering.org/blog/transforming-optimization-into-an-invisible-platform-capability)

In the high-stakes world of cloud-native delivery, organizations are frequently paralyzed by an "invisible wall" that fundamentally separates FinOps professionals, Site Reliability Engineers, and Developers. 

This structural friction strongly resembles a dysfunctional michelin-starred kitchen where the night’s success depends on characters who rarely see eye-to-eye on the operational priorities required to serve a flawless meal. 

![](https://cdn.prod.website-files.com/6489e23dd070ba71d41a33b2/6a0c8288c5dd8bfc13916685_Screenshot%202026-05-19%20at%2017.32.11.png)

Because these highly specialized teams speak entirely different languages regarding risk and reward, this dynamic creates a zero-sum game where the organization must continuously trade off cost for reliability or speed for efficiency. 

When a FinOps spreadsheet indicates that a Kubernetes pod is mostly idle, the SRE immediately perceives a threat to their carefully calculated reliability buffer, while the developers simply lack the contextual data to know how much compute space their specific application actually requires under realistic load. 

This operational paralysis is further exacerbated by the shared ownership model prevalent in many modern enterprises, where a lack of singular accountability means that reactive cost-cutting and conservative over-provisioning remain the default strategies.

## The day 2 operational gap

To break this destructive cycle, we must stop treating operational efficiency as a manual "spring cleaning" task and start recognizing it as an essential requirement that must be embedded deeply within modern platform architecture. 

While the platform engineering community has excelled at creating golden paths to streamline the initial developer experience and deployment pipelines, the lifecycle of applications reaching production has been largely neglected. Currently, our primary method of assisting developers with post-deployment optimization relies on providing access to passive monitoring dashboards rather than delivering automated, actionable intelligence. 

Organizations are striving to run highly performant applications while simultaneously maintaining strict cost controls and ensuring absolute reliability, but static golden paths are simply insufficient for managing this delicate trade-off. 

When building deployment pipelines, everyone shares the single goal of releasing software, which allows for strict and opinionated automation. However, production optimization requires a highly configurable approach because different applications inherently require dynamic tuning strategies tailored to their specific risk profiles and performance needs. For these reasons, static deployment pipelines inevitably fall short during Day 2 operations and lead to systemic resource waste.

## The full-stack optimization puzzle

True optimization represents a complex multi-layered puzzle that extends far beyond simply resizing Kubernetes pods or shutting down idle development environments after business hours. 

Deep efficiency must begin at the application runtime level by tuning elements like the Java Virtual Machine heap size or garbage collection settings to ensure the core workload operates at peak health. Once the runtime is properly configured and understood, engineers can accurately size the container requests and limits to perfectly encase that specific application without causing artificial throttling or fatal memory errors. 

This layered approach must also carefully incorporate how you scale horizontally and the underlying node groups to ensure the entire system remains stable when infrastructure density increases. Without looking at this vertical stack of dependencies, scaling policies might rely on superficial metrics that do not align with the newly established pod boundaries.

![](https://cdn.prod.website-files.com/6489e23dd070ba71d41a33b2/6a0c82b03624c5b2b527b085_Screenshot%202026-05-19%20at%2017.32.51.png)

This inevitably causes the system to fluctuate unpredictably and degrade the performance of your applications. You cannot effectively optimize a system if you treat the container as an opaque black box, yet industry data shows that very few companies actually attempt to tune the container and the runtime in tandem.

### Resolving the kitchen conflict

To solve this puzzle, we must address the conflicting motivations of the specialized roles mentioned earlier. The "invisible wall" persists because each player in the kitchen has a different definition of success:

- The **FinOps** team acts as the restaurant owner who scrutinizes every cloud expenditure and demands higher utilization rates to justify the infrastructure investments.
- The **Site Reliability Engineers** operate as the executive chefs who prioritize absolute uptime and system stability, often demanding massive resource buffers to absorb unexpected traffic spikes.
- The **platform team** serves as the overwhelmed manager attempting to standardize infrastructure while keeping the peace.
- The **developers** act as line cooks who desperately try to innovate without clear guidance on the resource constraints they must operate within.

## Integrating efficiency into the GitOps workflow

By integrating performance gates directly into the continuous integration and continuous deployment pipeline, the platform team establishes a controlled environment where efficiency becomes a core component of the software release lifecycle. 

These purpose-built platform capabilities can continuously analyze historical observability data to generate intelligent recommendations based on predefined tuning profiles that strictly respect the safety limits established by the SRE team.

![](https://cdn.prod.website-files.com/6489e23dd070ba71d41a33b2/6a0c82cefc9e4e1f6cdac159_Screenshot%202026-05-19%20at%2017.33.24.png)

The most effective way to implement these data-driven recommendations involves establishing a human-in-the-loop workflow driven entirely through established GitOps principles. Whenever the platform identifies a more efficient configuration that satisfies both cost constraints and performance requirements, it seamlessly opens a pull request directly in the developer's repository containing the evidence-based sizing adjustments. 

Developers can easily review these insights within their existing tools and approve the new configuration with absolute confidence, knowing that the proposed changes have been mathematically validated against the organizational reliability standards. 

By embedding these automated right-sizing mechanisms directly into the golden path, the platform truly becomes a unified bridge that eliminates the historical friction between engineering departments. This approach moves the organization beyond simple infrastructure provisioning to create a self-improving ecosystem where cost-awareness and performance are continuously balanced without requiring constant human intervention.

By transforming optimization from a manual chore into an invisible platform capability, we solve the "Michelin kitchen" dilemma. This shift allows developers to focus entirely on innovation and feature delivery while the underlying system ensures the organizational menu is always perfectly optimized for both the business and the end user. When efficiency is automated, the invisible wall disappears, replaced by a seamless path toward sustainable, high-performance delivery.


[Published by CNCF co-authored with Hélia Barroso, Alessandro Pomponio, Sonali Srivastava](https://www.cncf.io/blog/2026/04/29/the-state-of-ai-in-cncf-projects-a-first-look-at-the-data/)

At **CNCF TAG Developer Experience**, we recently set out to understand how Artificial Intelligence is shaping open-source development. The response from the community has been impressive in its scale, with nearly half of our initial responses arriving within the first week alone. This immediate engagement highlights the urgency of the topic and the community’s desire for shared guidelines for AI-assisted development.

The data collected from 133 respondents so far represents **nearly 100 unique projects**, giving us confidence that these findings reflect the cloud-native ecosystem at large rather than a narrow subset of projects. This article serves as a sneak peek into our initial findings; a more comprehensive analysis will follow as we process further data.

### Who is answering?

The feedback primarily reflects the perspectives of those on the front lines. The vast majority of participants are code-centric contributors focusing on submission, CI/CD, and infrastructure, while approximately 20% combine engineering with critical roles like release management and documentation.

### How maintainers use AI: Tools and workflows

Modern AI tools have moved beyond simple web chatbots and are now deeply integrated into daily routines. Nearly half of respondents actively use AI assistants directly within their IDEs or command-line interfaces.

![A blue bar graph showing the levels of basic, advanced, other and intermediate. ](https://www.cncf.io/wp-content/uploads/2026/04/image-14.png)

A blue bar graph showing the levels of basic, advanced, other and intermediate.

When it comes to tooling, Claude Code and Github Copilot emerge as clear leaders in the space. Interestingly, only a small fraction (roughly 10%) of contributors still rely on basic chatbots via manual copy-pasting. Meanwhile, a similar percentage of advanced users has already moved toward “high-level integration,” where AI is built directly into project automation for PR reviews and issue triaging.

### Where AI helps the most

Contributors are seeing the most significant boosts in productivity within a few specific areas:

- Writing and refactoring code.
- Improving documentation and debugging.
- Understanding unfamiliar codebases.
- Analyzing Pull Requests.

The high ranking of “understanding the codebase” suggests that AI is acting as a knowledgeable guide, helping developers navigate the inherent complexity of large-scale projects.

### The gap between AI use and official policies

One of our most striking findings is the disconnect between individual AI usage and formal project governance. While local AI use is widespread, the adoption of official policies has lagged behind.

Roughly two-thirds of respondents are either unaware of any specific AI guidelines or confirmed that no official policies exist in their main repositories. Furthermore, the vast majority of projects make no mention of AI usage in their public-facing documentation or contributing guides. While a few pioneering projects are setting the pace with clear policies, the ecosystem is largely operating in an environment that is still figuring out how to govern automated code generation.

![An orange bar graph showing levels of clear policies, informal acceptance and unaware/no policy ](https://www.cncf.io/wp-content/uploads/2026/04/image-15.png)

An orange bar graph showing levels of clear policies, informal acceptance and unaware/no policy

### Community sentiment and code reviews

Despite the lack of formal rules, the general “vibe” toward AI is open and accepting. Roughly one-third of contributors noted that AI usage is generally allowed. Conversely, only a tiny minority (less than 4%) reported that AI usage is explicitly prohibited in their environments.

![A green bar graph showing levels of 'unsure', 'not required', 'specific cases only' and 'always require disclosure.' ](https://www.cncf.io/wp-content/uploads/2026/04/image-16.png)

A green bar graph showing levels of 'unsure', 'not required', 'specific cases only' and 'always require disclosure.'

This pragmatic approach extends to how maintainers handle AI-generated contributions:

- A solid majority follow their standard review process without applying special filters.
- Over a quarter of maintainers prefer a collaborative approach, asking contributors to refine AI-generated code to meet quality standards rather than rejecting it.
- Only a nominal percentage automatically reject suspected AI PRs.

### Top concerns and the call for transparency

While the community is optimistic, several valid concerns remain. Maintainers are particularly worried about:

1. Security vulnerabilities
2. License compliance.
3. The burden on reviewers, caused by **a potential flood of low-effort PRs**.

To mitigate these risks, there is a strong collective desire for visibility. Over half of respondents believe that AI-assisted contributions should always require formal disclosure (such as an “AI-authored” tag). An additional 20% feel disclosure should be required in specific cases. This suggests that while maintainers are willing to accept AI-generated code, they want the transparency necessary to adjust their review efforts accordingly.

### Wrapping up

This first batch of data confirms that AI integration is no longer a trend, it is a core part of the modern workflow. As we move forward, the challenge for the cloud-native community will be balancing this new productivity with the high standards of security and manual oversight that enterprise-grade open source requires.

We aren’t finished yet! To ensure our final report truly reflects the diverse cloud-native landscape, we need your voice. The survey will remain open until Monday, May 18 (End of Day, Anywhere on Earth).If you haven’t shared your experience yet, we invite you to [**contribute to the survey**](https://forms.gle/tBjQp9sFnBF3CXE17) and help us build a more accurate and comprehensive picture of AI’s role in our community.

Graziano Casto, Developer Advocate at Akuity (Argo CD, Kargo). Making cloud native continuous delivery easier. Co-Chair of CNCF TAG Developer Experience.

Graziano Casto | Developer Advocate @ Akuity, Co-Chair @ CNCF TAG DevEx

Graziano Casto

Developer Advocate @ Akuity · Co-Chair @ CNCF TAG DevEx · K8s Contributor · CNCF Ambassador · Platform Engineering Ambassador

Developer Advocate

Akuity

Making continuous delivery easier
one release at a time.

Latest Activity

Featured Articles

Upcoming Events

Making continuous delivery easierone release at a time.

Latest Activity

Featured Articles

Upcoming Events

Making continuous delivery easier
one release at a time.