State of the Model Serving Communities - February 2026

Most recent updates from several AI/ML model inference communities that our teams at Red Hat AI are contributing to.

Hi everyone,

This newsletter will provide you with recent updates on various model serving communities, keep you informed about Red Hat AI’s contributions to upstream communities, and foster collaborations across teams and organizations.

In case you missed it, we launched this newsletter publicly on Substack last year and now we have over 1,200 subscribers already! Feel free to share with others who might be interested in receiving future updates.

Contributors: Nir Rozenbaum, Jooho Lee, Sasa Zelenovic, Pete Cheslock, Wentao Ye, Pierangelo Di Pilato, Yuan Tang

Executive Summary

• Community Outreach:

  • KubeCon Europe: the conference in March in Amsterdam will feature a Red Hat booth, a KServe project kiosk, and KServe maintainer session and project lightning talks.

  • vLLM: The community covered topics like batch invariant, Speculators, LLM Compressor Update, and CPU Offloading. The community has also launched a new website and events calendar. For vLLM Meetups, 15 hands-on workshops and production stories are planned for Q1 across major cities globally, in collaboration with partners like AMD, NVIDIA, and IBM.

  • llm-d: The community celebrated the 0.5 release, which introduced production-scale inference enhancements such as one-command benchmarking, memory-saving KV offloading, LoRA-aware scheduling, and scale-to-zero capabilities. An inaugural llm-d meetup is scheduled for March 10 in NYC with IBM and AMD. The community’s YouTube channel offers demos, including a Red Hat demonstration on scaling LLM inference with KServe, llm-d, and Envoy Gateway, and an IBM Research video on achieving 100% reproducibility for llm-d deployments using a standardized benchmarking suite.

• WG Serving:

  • GIE: New proposals include Flow Control improvements and integrating the P/D protocol. New features added are OpenAI Response API support and distributed tracing. Improvements were made to the Data Layer for pluggable data collection and a fix to conformance tests to allow report submission. A project meeting is also scheduled for KubeCon EU.

  • Serving Catalog: Work is in progress to add support for various models and clouds, including Azure support for LLaMA3-8B, LWS tpu7x components, and AWS support for Llama3-8b, Llama3-70b, and Gemma-2b.

  • Inference Perf: Recent updates include a published paper in the Journal of Open Source Software, adding shuffle to multi-round prompt generation, enabling multiple report analysis via a CLI tool, adding end-to-end testing with llm-d-inference-sim, and adding multi-LoRA support.

• KServe: The community has a few notable community PRs, including centralizing scripts and refactoring installation for a monthly release cadence, and bumping the Gateway API Inference Extension to v1.2.0. For KubeCon Europe in Amsterdam, attendees can visit the KServe project kiosk and check out the maintainer session and project lightning talk.

• llm-d: Updates from various SIGs from the llm-d community:

  • Inference Scheduling & LoRA Support: Introduces LoRA-precise scheduling to optimize multi-tenant environments by routing requests based on adapter locality, maximizing cache hits, and addressing the “thundering herd” problem.

  • Advanced Flow Control (Ongoing Work): Developing gradient-based saturation detection to replace the current binary open/closed throttle gate with a normalized signal of fullness. This enables priority-based holdback and in-flight request eviction.

  • Inference Scheduling and Autoscaling: Adds Scale-to-Zero support for intermittent workloads, ensuring incoming requests immediately trigger provisioning without being dropped.

  • Disaggregated Inference and KV-Cache Offloading: Integration of the UCCL backend into the NIXL layer to manage transport on the host CPU, achieving 2.4x greater resilience to network congestion and protecting tail latency in noisy networks. Implements a three-tier memory hierarchy (GPU, CPU, and Disk) to decouple cache capacity from GPU VRAM, allowing the active working set to exceed available RAM. This resulted in a 13.9x improvement in throughput under high concurrency.

  • Benchmarking: Adopts a strict policy where all performance claims are backed by a version-controlled configuration for easy replication and validation by users.

• vLLM:

  • vLLM 0.14.0 focuses on higher throughput and better usability with async scheduling enabled by default, a new gRPC server entrypoint, automatic --max-model-len fitting to avoid OOM, and richer model inspection plus experimental Model Runner V2 upgrades.

  • vLLM 0.15.0 expands model/LoRA/VLM and embedding support while further strengthening the engine core (KV cache/memory management, Triton attention, prefix-cache and DP optimizations, and more robust async + speculative decoding behavior).

• Llama Stack: Recent LlamaStack releases (up to v0.5.0) introduced several notable changes. New features include migrating the models and conversations APIs to use a FastAPI router. CI/Build improvements involve adding a CodeQL scanning workflow and the ability to build ARM64-based UBI starter images. Bug fixes and chores addressed security vulnerabilities (aiohttp HTTP Parser, fonttools) and added backward compatibility for vector store deserialization in Faiss, Milvus, and Qdrant. Other notable changes included breaking changes in configuration, API removals and renames, and API behavior changes.

Community Outreach

• KubeCon Europe in Amsterdam next month

  • Visit the Red Hat booth in the sponsor showcase floor.

  • Visit the KServe project kiosk (from Tuesday to Thursday mornings) in the project pavilion to chat with the project maintainers. In addition, check out KServe’s maintainer session as well as the project lightning talk.

  • Check out all the sessions and events from Red Hat at the main conference as well as Cloud Native AI Day that we are organizing.

• vLLM community:

  • vLLM Office Hours

    • We ran four vLLM office hours sessions in January instead of the usual two to keep up with content demand:

      • #39: Intro to batch invariant in vLLM

      • #40: Intro to Speculators

      • #41: LLM Compressor Update & Case Study

      • #42: Deep Dive Into the vLLM CPU Offloading Connector

    • We launched a new vLLM website. Take a look, and check out the events calendar to stay up to date with all vLLM events like office hours, meetups, SIGs, and more.

  • vLLM Meetups

    • We are planning 15 vLLM meetups in Q1 together with partners like AMD, NVIDIA, IBM, HPE, Cisco, and more. We’re leaning harder into hands-on workshops and real production stories, like Canva sharing their vLLM deployment experience at the Vienna meetup in March. Q1 cities include San Francisco, Bengaluru, Munich, Pune, Tokyo, Hong Kong, Wellington, Warsaw, Vienna, Beijing, Boston, Stockholm, Bangkok, and Mumbai.

• llm-d community:

  • Congrats to the llm-d community for the latest 0.5 release, which supercharges production-scale inference with one-command benchmarking, memory-saving KV offloading, LoRA-aware scheduling, and cost-efficient scale-to-zero capabilities.

  • In case you missed it last month, you can also follow the llm-d community on Bluesky. We’ll be posting community updates there along with Twitter/X and LinkedIn.

  • We’re planning our first llm-d meetup on March 10 at the IBM office in NYC with IBM and AMD. Stay tuned for a registration link on the events page.

  • Subscribe to the llm-d project YouTube channel as we release more demos and technical deep dives from the community

    • Watch Sean Condon from Red Hat demonstrate how to scale LLM inference by using KServe for orchestration, llm-d for intelligent traffic distribution, and Envoy Gateway for model-based routing across high-performance GPU clusters.

    • See how Dmitri Pikus from IBM Research shows how the SIG-benchmarking team delivers 100% reproducibility for llm-d deployments by providing a standardized benchmarking suite and simplified scripts to verify local environment performance against community baselines.

WG Serving

• Communications

  • #wg-serving channel in Kubernetes Slack and mailing list

• Subprojects updates (note: you’ll need to join the WG Serving mailing list to access some of the documents)

  • Gateway API Inference Extension (GIE):

    • New proposals, including Flow Control improvements and upstreaming P/D protocol from llm-d to IGW

    • Added support for OpenAI Response API and distributed tracing.

    • Improvements

      • Enhanced the new Data Layer, allowing pluggable data collection with custom DataSource and Data Extractors

      • Fix in conformance tests that unblock the ability to submit conformance reports (removed the dependency on old gateway api which caused conflicts).

    • Additionally to the accepted KubeCon sessions, a project meeting will be hosted in KubeCon EU Maintainer Summit.

  • Serving Catalog:

    • Several in-progress work:

      • Add Azure support for LLaMA3-8B

      • Add LWS tpu7x 2x2x2 components

      • Add AWS support to ai-inference llama3-8b/llama3-70b/gemma-2b

  • Inference Perf:

    • Added Journal of Open Source Software paper on inference-perf

    • Added shuffle to multi-round prompt generation

    • Enabling multiple report analysis using CLI tool

    • Added end-to-end testing using llm-d-inference-sim

    • Added multi-lora support

KServe

• Communications: #kserve channel in the CNCF Slack

• Community PRs worth paying attention to:

  • Centralized all the scripts and refactorying installation/helm/kustomize for 1 month cadence release

  • Bump Gateway API Inference Extension (GIE) to v1.2.0

• Notable recent changes from the OpenShift AI Serving team:

  • RHOAI 3.3 ships MLServer ServingRuntime as TP

• KubeCon Europe in Amsterdam

  • Visit the KServe project kiosk (from Tuesday to Thursday mornings) in the project pavilion to chat with the project maintainers.

  • Check out KServe’s maintainer session as well as the project lightning talk.

llm-d

• Communications: Slack, mailing list, and community meeting

• Inference scheduler

  • Smarter Scheduling & LoRA Support: Multi-tenant environments can suffer from the “thundering herd” problem when everyone tries to load adapters at once. Our new LoRA-precise scheduling optimizes routing based on adapter locality. This ensures requests go where the data already resides, maximizing cache hits and minimizing overhead

• Ongoing Work: Advanced Flow Control: We are also overhauling our flow control layer to better handle multi-tenancy and mixed-priority traffic. Currently, saturation detection is a binary gate (open or closed), which often leads to “throttle thrash” where the system flips rapidly between full dispatch and full closure. To solve this, we are developing Gradient-based Saturation Detection, which converts that binary gate into a normalized signal of fullness. This precise signal enables two key capabilities:

  • Priority-based Holdback: You can define a usage limit for specific priorities, ensuring that lower-priority requests are throttled before the pool is full to reserve capacity for critical and high-priority workloads.

  • In-Flight Request Eviction: The system can identifying and evict “negative priority” requests that are already in flight, instantly reclaiming resources for higher-priority traffic.

• Inference scheduler and autoscaling

  • Scale-to-Zero: For intermittent workloads (like dev clusters or internal RAG apps), keeping GPUs running 24/7 is a waste. v0.5 introduces scale-to-zero support. Crucially, incoming requests trigger provisioning immediately without being dropped.

• Disaggregated inference and KV-cache offloading

  • Resilient Networking with UCCL: In distributed systems, tail latency is often at the mercy of the network. We have integrated our UCCL backend into the NIXL layer. By managing transport logic on the host CPU rather than relying solely on hardware offload, we achieved 2.4x greater resilience to network congestion. This protects your tail latency even when the network is noisy

  • Hierarchical KV Offloading: Breaking the VRAM Ceiling The finite capacity of GPU HBM has always been a hard ceiling for concurrency. We introduced a new three-tier memory hierarchy (GPU, CPU, and Disk) that decouples cache capacity from GPU memory. llm-d can now sustain throughput even when the active working set far exceeds available RAM. In our testing, this achieved a 13.9x improvement in throughput under high concurrency. A new storage connector plugs directly into vLLM’s native interface, allowing blocks to be offloaded to a shared file system asynchronously, so your compute path never gets blocked

• Benchmarking

  • We know the frustration of seeing benchmark numbers that you can’t replicate. We are changing that. We have adopted a strict policy where every performance claim in v0.5 is backed by a version-controlled configuration. You can now validate our “well-lit paths” with a single command using our updated benchmarking tools.

vLLM

• Communications: Slack

• Notable additions since the last update:

  • Highlights for vLLM 0.14.0

    • Async scheduling enabled by default: Overlaps engine core scheduling with GPU execution, improving throughput without user configuration. Now also works with speculative decoding and structured outputs.

    • gRPC server entrypoint: Alternative to REST API with binary protocol, HTTP/2 multiplexing.

    • Add --max-model-len auto: Automatically fits context length to available GPU memory, eliminating OOM startup failures.

    • Model inspection view: View the modules, attention backends, and quantization of your model in vLLM by specifying VLLM_LOG_MODEL_INSPECTION=1 or by simply printing the LLM object.

    • Model Runner V2 enhancements: UVA block tables, M-RoPE, logit_bias/allowed_token_ids/min_tokens.

    • Please note that Model Runner V2 is still experimental and disabled by default. Set VLLM_USE_V2_MODEL_RUNNER=1

  • Highlights for vLLM 0.15.0

    • Model Support

      • New architectures: Kimi-K2.5, Molmo2, Step3vl 10B, Step1, GLM-MoE-Lite , Eagle2.5-8B VLM.

      • LoRA expansion: Nemotron-H, InternVL2, MiniMax M2.

      • Speculative decoding: EAGLE3 for Pixtral/Llava, Qwen3 VL MoE, generalized draft model support.

      • Embeddings: BGE-M3 sparse embeddings and ColBERT.

      • Model enhancements: Voxtral streaming architecture, SharedFusedMoE for Qwen3MoE, dynamic resolution for Nemotron Nano VL, Molmo2 vision quantization.

    • Engine Core

      • Async scheduling default with spec decode and structured outputs

      • Hybrid allocator + KV connector with multiple KV cache groups

      • Triton attention: encoder-only/cross attention, cross-layer blocks

      • Mamba2 prefix cache optimization

      • Batch invariant LoRA

      • LoRA name in BlockStored for KV-cache reconstruction

      • Request ID collision prevention

      • Dense model DP without overhead

      • Async + spec decode penalties/bad_words

• Articles and blog posts

  • Building Mixture-of-Models on AMD GPUs with vLLM-SR

  • Inside vLLM’s New KV Offloading Connector: Smarter Memory Transfer for Maximizing Inference Throughput

  • vLLM Semantic Router v0.1 Iris: The First Major Release

  • Introducing vLLM Playground: A Modern Web Interface for Managing and Interacting with vLLM Servers

Llama Stack

• Communications

  • Discord server

  • Community office hours happen weekly Thursdays at 12pm EST on Discord

• Recent releases (full list): v0.5.0, v0.4.4, v0.4.3, v0.4.2, v0.4.1. Notable changes include:

  • New Features

    • API Migration: The models API and the Conversations API were converted to use a FastAPI router.

  • CI/Build Improvements:

    • Added a CodeQL scanning workflow for continuous security analysis.

    • Added a feature to build ARM64-based UBI starter images.

  • Documentation:

    • New documentation for building agents with OCI.

    • A new guide for migrating from the older Agents component to the Responses API.

  • Bug Fixes & Chores

    • Security: Fixed a security vulnerability related to the aiohttp HTTP Parser’s auto_decompress feature and a fonttools security advisory.

    • Vector Store: Added backward compatibility fixes for deserialization of embedded chunks in Faiss, Milvus, and Qdrant.

    • Testing: Fixed issues in the Vector Store and integration tests.

  • Other notable changes:

    • Breaking changes in configuration files, API removals, VectorStore API renames, and API behavior changes. Please check out v0.4.0 release notes for details.