Blog

Kubernetes 1.37 Garhwal: Gang Scheduling and Device Taints Fix the GPU Deadlock

Aug 30, 2026

Christopher Shaun Godwin

Author

Kubernetes v1.37 “Garhwal” landed on August 26, 2026. This is the release where Kubernetes stops scheduling pods one at a time and starts scheduling AI workloads as groups. The release carries 67 enhancements: 16 to Stable, 23 to Beta, 27 into Alpha, and 1 deprecation (Kubernetes Blog, 2026).

If you run GPUs, this release changes your day job. Dynamic Resource Allocation (DRA) moved four features to Stable in one shot. Gang scheduling, the fix for the distributed-training deadlock, hit Beta. And the removal list put kube-dns and IPVS on the clock.

The release by the numbers

67 enhancements: 16 Stable, 23 Beta, 27 Alpha, 1 deprecation (Kubernetes Blog, 2026).
A 15-week cycle with contributions from over 1,700 individuals across 212 companies (The Register, 2026).
118 API reviews, the most in project history, up from 88 in v1.36 and 41 in v1.21 back in April 2021 (Cloud Native Now, 2026).
Release lead Dipesh Rawat, and a theme named Garhwal, the Himalayan region of Uttarakhand in northern India (Network World, 2026).

The 118-review number is the hard one to absorb. The team handled it because the project formalized declarative validation in the previous release. That system generates API validation code from the type definitions instead of hand-writing every check (Cloud Native Now, 2026).

DRA grows up

DRA is how Kubernetes assigns GPUs and other accelerators. The DRA core APIs reached general availability in v1.34 (Cloud Native Now, 2026). v1.37 graduates four more pieces to Stable (Kubermatic, 2026).

ResourceClaim device status (KEP-4817) — drivers report per-device data on .status.devices. Before this field, a Pod could request a DRA-managed network device and no other component could learn the IP assigned to it. DRA is now fully usable for attaching secondary network interfaces (Kubernetes Blog, 2026).
Extended resources through a DRA driver (KEP-5004) — assign an extended resource name such as abc.example/gpu: 3 to a DeviceClass. DRA allocates the device with no device plugin in the path. Existing manifests keep working, which makes this the migration route off device plugins (Kubermatic, 2026).
Device taints and tolerations (KEP-5055) — taint one overheating GPU instead of draining the node it sits in. A driver can taint a single device, or an administrator can write a DeviceTaintRule by selection criteria, such as every device one driver manages. Claims that still need the degraded device declare tolerations (Kubernetes Blog, 2026). The old model was worse. A driver’s only option for a failing device was to pull it out of the ResourceSlice entirely. That stopped everyone from using it, including healthy running workloads (MetalBear, 2026).
A standard numaNode device attribute — resource.kubernetes.io/numaNode becomes the shared attribute name. Devices managed by different DRA drivers now compare on the same NUMA placement (Kubernetes Blog, 2026).

Gang scheduling kills the partial-deadlock

The default scheduler places pods one by one. A 32-replica training job with only 20 free GPUs used to take the 20 and sit. Those 20 pods hold resources that can never finish the job, and other workloads starve behind them (Kubernetes Blog, 2026).

Gang scheduling flips the rule. The scheduler commits the group only when cluster capacity covers the whole group. The beta runs through the Workload API and the PodGroup concept as an all-or-nothing strategy (KEP-4671) (Kubernetes Blog, 2026).

The beta also adds two supporting pieces. Workload-aware preemption stops the scheduler from evicting a job for a preemptor that would not make progress anyway. PodGroup queueing coordinates competing workloads and stops the livelock where two large jobs keep bouncing each other without progress (Kubernetes Blog, 2026).

One caveat before you upgrade. Gang scheduling does not activate on its own. You opt in through the Workload API and PodGroup objects (TechTimes, 2026).

HPA can scale to zero

Scale to zero for the HorizontalPodAutoscaler reached Beta and is enabled by default. Set spec.minReplicas: 0 on a queue consumer, batch job, or GPU workload. The workload drops to zero pods when idle and comes back when demand returns. It works with object or external metrics only, because CPU and memory scaling needs live pods to produce numbers (Kubernetes Blog, 2026).

The control plane gets calmer under load

Resilient watchcache initialization completed its graduation. When the API server restarts, the watchcache warm-up no longer fires a list-and-watch storm at etcd. The API server bounds the burst and rejects the rest with HTTP 429 plus a Retry-After header (Kubernetes Blog, 2026).
Pod-level checkpoint and restore entered Alpha as KEP-5823. The CRI gains CheckpointPod and RestorePod RPCs. Your container runtimes must implement them before this is useful (Kubernetes Blog, 2026, and The Register, 2026).

The theme of the release, according to the trade press, is control-plane hardening for production AI workloads rather than shiny additions (ITOps Times, 2026).

The removal list is your upgrade checklist

Garhwal prunes as hard as it adds. The Register called it “Kubernetes cleans house” (The Register, 2026).

kube-dns is deprecated. CoreDNS has been the default cluster DNS since v1.13, and kube-dns never gained EndpointSlices or dual-stack Services. No new kube-dns packages ship after v1.40 (Kubernetes Blog, 2026).
kube-proxy IPVS mode is deprecated. IPVS arrived in v1.8 to fix iptables bottlenecks, but it still leans on iptables underneath. The project is consolidating on nftables, and clusters running IPVS now log a startup deprecation warning (Kubernetes Blog, 2026, and Network World, 2026).
cgroup v1 is on the way out. Since v1.35 the kubelet failCgroupV1 setting defaults to true. Nodes still on cgroup v1 fail kubelet init unless you apply a temporary override. Memory QoS and in-place memory-volume scaling work only on cgroup v2 (Kubernetes Blog, 2026).
Smaller cuts: kubectl run --filename/-f is deprecated, and static Pods can no longer reference Secrets or ConfigMaps (Kubernetes Blog, 2026).

What to do this week

Verify cgroup v2 on every node before you schedule the 1.37 upgrade.
Inventory kube-dns. Start the CoreDNS migration before v1.40 ends the packages.
If kube-proxy runs IPVS, test nftables mode on a canary node.
Start moving device plugins to DRA. KEP-5004 keeps your old extended-resource manifests working during the move.
Opt in to gang scheduling for training jobs through the Workload API. It does not auto-enable.
Teach your controllers to handle HTTP 429 with exponential backoff. The API server now expects it.

The bottom line

Kubernetes 1.37 treats an AI training run like a unit of work, not a pile of pods. DRA finished growing into a real device scheduler, and gang scheduling ends the deadlock that has haunted distributed AI training for years (TechTimes, 2026). The release logo shows terraced Himalayan fields, each shelf cut one step at a time and each shelf holding the next one up. That image fits the work: no single leap, four DRA graduations, one beta that rewrites the scheduling contract, and a removal list that tells you where the next three upgrades bite. The release is available for download on GitHub (GitHub, 2026).

OpenAI Cuts Off Cursor on November 12: Read the Change-of-Control Clause

Aug 29, 2026

Christopher Shaun Godwin

Author

OpenAI just showed every developer what a model supplier exit looks like. On Friday, the company notified SpaceX that it will wind down the contract providing OpenAI models to Cursor, with a proposed shutoff date of November 12, 2026 (Devdiscourse, 2026). OpenAI published the decision in its own statement and said it is giving developers the maximum notice its contract allows (OpenAI on X, 2026).

This is not a product dispute. It is a contract clause firing exactly as written, two weeks after SpaceX closed its $60 billion all-stock acquisition of Anysphere, the company behind Cursor (ExplainX, 2026).

The statement, in full

OpenAI’s statement is short. Here is the core of it:

“Today, we notified SpaceX that we intend to wind down our contract providing OpenAI models to Cursor, with a proposed shutoff date of November 12, 2026. To maximize the time that developers can retain access to our models through Cursor, we are giving the maximum notice provided by our contract.” (Livemint, 2026)

The reason follows in the same statement. OpenAI wrote that it is “making this choice because we cannot be confident that SpaceX will use our technology within our terms of service, based on our experience with Elon Musk’s companies violating contracts” (Business Insider, 2026). The company also called the decision tough and said it wants its models to stay broadly available to developers (Storyboard18, 2026).

The mechanism: a change-of-control clause

The timeline shows a clause doing its job:

Date	Event
April 2026	SpaceX and Cursor announce a strategic partnership with an option: SpaceX can acquire Cursor for $60B or pay a $10B break-up fee (Dealroom, 2026)
June 16, 2026	SpaceX files to acquire Anysphere in an all-stock deal (ExplainX, 2026)
August 14, 2026	The acquisition closes and Cursor joins the SpaceXAI group (ExplainX, 2026)
August 28, 2026	OpenAI serves notice, invoking the change-of-control provision in its agreement (Devdiscourse, 2026)
November 12, 2026	Proposed shutoff. That works out to 76 days of notice (Livemint, 2026)

OpenAI’s custom agreement with Cursor carried a cancellation window that opened when ownership changed (AI Tools Recap, 2026). The acquisition triggered it.

The precedent list in OpenAI’s reasoning is specific. After Musk acquired Twitter, the company broke the terms of its OpenAI contract. xAI, now part of SpaceX, admitted violating OpenAI’s terms of service (AI Tools Recap, 2026).

What changes for Cursor users

GPT models keep working until November 12. Cursor’s site lists OpenAI’s GPT-5.6 Luna, Sol, and Terra models for paid users today (Business Insider, 2026).
The model menu freezes now. OpenAI will ship no future models to Cursor in the meantime, including Astra, its next model (Business Insider, 2026).
Anthropic has not said publicly whether Claude stays in Cursor (AI Tools Recap, 2026).
SpaceX and OpenAI spokespeople did not immediately return press requests for comment (Business Insider, 2026).

The migration path inside the deal is already visible. Cursor’s official account announced it partnered with SpaceXAI to train Grok 4.5, a model built for large codebases and long-running, multi-repository tasks (VentureBeat, 2026). Four days after the close, Cursor shipped Origin, an in-app code hosting platform, and Grok 4.6 reached Amazon Bedrock (ExplainX, 2026).

Why this matters beyond Cursor

SpaceX now holds compute, model, and distribution in one stack. The company owns the Colossus training infrastructure, the Grok model family, and the developer base of a dominant AI coding tool (VentureBeat, 2026).

The tooling layer between model suppliers and developers is being bought, not built. Stripe bought OpenRouter earlier this month (we covered that deal here), and SpaceX bought Cursor. When the aggregator becomes a subsidiary of a supplier or a competitor, suppliers gain a new reason to walk. Ownership changes become outages with a 76-day countdown.

What your team should do now

Treat model access as a dependency. Pin model IDs in config, log the provider and version behind every call, and alert on changes the way you alert on image digest drift.
Keep evals provider-neutral. A suite that runs per provider turns a forced migration into a measured comparison instead of a leap of faith.
Own the routing layer. BYOK or a gateway you control makes a supplier exit a config change, not a rewrite. The Stripe-OpenRouter deal taught the same lesson.
Read the change-of-control clause. Your own contracts have one. This episode shows the exit fires on ownership, not on failure, and 76 days is what maximum notice looks like.

Model supply chains now behave like every other supply chain. They are single-sourced, exposed to politics, and one acquisition away from a forced migration. Build for that.

Nvidia Is Buying Hugging Face for $12.9 Billion: The GitHub of AI Gets a New Owner

Aug 28, 2026

Christopher Shaun Godwin

Author

The default home of open-weight AI models may soon belong to the company that sells most of the silicon those models run on. Nvidia agreed to buy Hugging Face for $12.9 billion, according to a report in The Information (TechCrunch, 2026). Neither company has confirmed the deal.

That matters because Hugging Face is the open-source hub where developers build, test, and distribute AI models and tools (CNBC, 2026). It is often described as the GitHub for AI models (LinkedIn News, 2026). If the deal closes, the dominant AI chip supplier would also own the main distribution point for the open ecosystem.

The deal, by the numbers

The Information reported the agreement on Wednesday night, citing a person familiar with the deal. Business Insider reported the same night that talks had not yet produced a signed agreement and could still fall apart (TechCrunch, 2026).

Spec	Value
Reported price	$12.9 billion
Hugging Face annualized revenue	About $150 million
Price-to-revenue multiple	About 86x
Last funding round (2023)	$235 million at a $4.5 billion valuation
Rejected Nvidia investment (late 2025)	$500 million at a $7 billion valuation

The revenue figure comes from The Information. It puts Hugging Face at about $150 million in annualized revenue, up from roughly $100 million two months earlier, which makes the reported price about 86 times sales (The New Stack, 2026).

The price is nearly triple the $4.5 billion valuation Hugging Face carried out of its 2023 round. Salesforce Ventures led that round, with money from Alphabet’s GV, IBM Ventures, and Nvidia itself (TechCrunch, 2026).

Hugging Face turned Nvidia down once before. In late 2025 it rejected a $500 million investment that would have valued the company at $7 billion. The stated reason was governance: the company argued that no single investor should hold that much sway over neutral infrastructure (TNW, 2026). Nine months later, the company reportedly agreed to sell outright for close to double that valuation.

Why Nvidia wants the hub

The logic is pull-through economics.

Developers who download open-weight models run them on their own infrastructure, and that hardware overwhelmingly relies on Nvidia GPUs. Hugging Face also uses Nvidia GPUs to power its paid hosting services (Fortune, 2026). Owning one of the most widely used platforms for open models connects developer demand directly to the silicon.

The deal also defends the core business. OpenAI, Google, Amazon, and Anthropic are building their own chips to reduce their reliance on Nvidia hardware. A strong open-source ecosystem keeps more of the market tied to Nvidia’s stack (Fortune, 2026).

Timing tells its own story. The report landed the same day Nvidia posted earnings that pushed its shares up 4% after hours, on top of a reported $20 billion licensing deal with AI chip startup Groq (CNBC, 2026).

The neutrality problem

Hugging Face’s value comes from neutrality. The platform works with AMD, AWS, and Google, alongside its 2023 partnership that connected Hugging Face models to Nvidia’s DGX Cloud for training and fine-tuning (TechSpot, 2026).

Open licenses guarantee access to weights. They do not guarantee equal treatment in discovery, ranking, optimization, or product priorities. An analysis in TechStrong AI states the risk directly: Hugging Face does not have to stop supporting Nvidia competitors for the easiest and best-supported path to gradually become the Nvidia path (TechStrong AI, 2026).

Vinay Thakker, CTO of KloudStax, framed the operations angle. Many teams ran open models through Hugging Face because they could self-host and avoid lock-in to one frontier lab. The fallback option now sits inside the same company that sells the chips (TechStrong AI, 2026).

Antitrust review is the other open question. Nvidia is already under scrutiny over its position in AI accelerators, and buying the ecosystem’s default hosting layer is a vertical move that regulators in more than one jurisdiction will find interesting (TNW, 2026).

The precedent: GitHub, one layer down

Microsoft paid $7.5 billion for GitHub in 2018. GitHub was the place developers already worked, and Azure became the default next step out of it. Nvidia is running the same play one layer down: own the place developers get models, and the path to your hardware becomes the path of least resistance (LinkedIn News, 2026).

What your team should do now

Nothing is signed yet, so treat this as risk planning, not alarm:

Pin your models. Reference exact model versions and checksums in your deployments, the same way you pin container images.
Mirror what you cannot lose. Copy the weights and tokenizers your production stack depends on into your own registry or object storage.
Read the license, then the terms of service. Model licenses survive an acquisition. Hosted tooling, rate limits, and discovery ranking can change under new ownership.
Track the deal status. No signed agreement exists yet, which means the outcome is still open (TechCrunch, 2026).

The open-weight ecosystem grew on neutrality. The question now is whether the hub that made it accessible stays neutral under the ownership of the company that sells the hardware underneath it.

Gitea Has a Critical RCE Being Exploited Right Now — Patch to 1.27.1

Aug 27, 2026

Christopher Shaun Godwin

Author

CISA added a critical Gitea remote code execution (RCE) flaw to its Known Exploited Vulnerabilities catalog on August 25, 2026, after confirming attackers are using it in the wild (CISA, 2026). The flaw is tracked as CVE-2026-60004 and carries a CVSS score of 9.8 (Help Net Security, 2026).

If you self-host Gitea, this is a patch-today situation. Attackers are already dropping miner-like payloads on vulnerable servers (The Hacker News, 2026). A successful exploit runs shell commands as the Gitea service account, which means full code execution on the machine that hosts your repositories.

What the vulnerability is

CVE-2026-60004 is a code injection flaw, classified as CWE-94, inside Gitea’s diffpatch feature (SecurityWeek, 2026). Gitea is an open-source, Go-based Git platform that teams host on their own servers (Help Net Security, 2026). The bug lets an attacker who holds write access to a repository push a malicious patch to the diffpatch endpoint and plant an executable Git hook that runs shell commands as the Gitea service account (CISA, 2026).

The attack chain

The exploit does not crash the platform with a packet flood. It uses Gitea’s own patch machinery.

The attacker submits a tricked patch to the diffpatch endpoint, which lives on the repo path (/api/v1/repos/{owner}/{repo}/diffpatch).
Gitea runs the patch through Git’s apply inside a temporary bare clone (runZero, 2026).
A crafted add/add collision in the patch content lets the attacker write an executable Git hook into the repository directory (runZero, 2026).
Gitea and Git then execute that hook as the Gitea service account (SecurityWeek, 2026).

The result is remote code execution on the host. An attacker who controls the Gitea service account can read repositories, steal secrets from hooks or environment, and pivot to the wider internal network (BleepingComputer, 2026).

Which versions are affected

The advisory stretches versions from 1.17 up to, but not including, 1.27.1 (runZero, 2026). Exploitation also depends on Git version 2.32 or newer and an enabled diffpatch route (SecurityWeek, 2026). Gitea shipped the fix in version 1.27.1 (Gitea Security Advisories, 2026).

A rough blast radius estimate is straightforward. Open-registration instances let anyone sign up and create a repository (FOFA note, 2026). With write access already granted, the attacker skips the hardest part of the chain.

What CISA ordered

CISA placed CVE-2026-60004 into its Known Exploited Vulnerabilities catalog with a tag that it is also used to infect systems with ransomware (CISA KEV Catalog, 2026). Federal agencies must patch within their binding operational directive window, laid out as a three-day deadline (BleepingComputer, 2026).

What to do now

Upgrade every Gitea server to version 1.27.1 or later. This is the single effective fix (Gitea Security Advisories, 2026).
Treat any public Gitea as exposed until upgraded (BleepingComputer, 2026).
Review which IPs can reach the diffpatch endpoint and disable the route if your workflow does not use it.
Disable open registration (unsigned-up instance signup) so strangers cannot self-register and obtain write access (The Hacker News, 2026).

Self-hosted DevOps platforms are a high-value target for attackers precisely because they cluster code and secrets in one trust zone (Help Net Security, 2026). The security reaction pattern of the developer ecosystem is now very consistent: KEV catalog entry, patch, then fresh scrutiny of any self-hosted Git forge. Patch first, plan after.

TRACE Is the Tamper-Proof Receipt for AI Agents

Aug 26, 2026

Christopher Shaun Godwin

Author

TRACE Is the Tamper-Proof Receipt for AI Agents

AI agents now touch sensitive systems, call internal tools, and act on classified data. A team that deploys an agent cannot always prove what it actually did. That trust gap breaks audits and slows production rollouts.

The Linux Foundation adopted TRACE on August 25, 2026. TRACE stands for Trust, Runtime Attestation, and Compliance Evidence. It is an open specification for hardware-attested AI agent governance records. AMD, Intel, Microsoft, OPAQUE, and the Technology Innovation Institute (TII) developed it together.

What a TRACE Trust Record proves

A TRACE Trust Record answers five questions. Any third party can verify the answers without trusting the operator that produced them.

Question	Claim recorded
What model ran?	`model.model_id` + `model.weights_digest`
Where did it run?	`runtime.platform` + `runtime.measurement`
Under which policy?	`policy.bundle_hash` + `policy.enforcement_mode`
What data did it touch?	`data_class`
Which tools were called?	`tool_transcript.hash` + `tool_transcript.call_count`

The record is one signed artifact. It binds the runtime environment, policy, data classification, and tool use into a single cryptographically verifiable blob, per the TRACE specification.

It builds on standards you already know

TRACE does not invent a new framework. It composes existing open standards into one evidence layer.

RFC 9711 (EAT) — the claim envelope
RFC 9334 (RATS) — the attester, verifier, relying-party roles
SCITT draft — transparency-ledger anchoring

The Linux Foundation confirmed the design builds on RATS, EAT, SLSA, SCITT, SPIFFE, and EAR. It targets CoSAI WS4 interoperability, and the reference implementation is Confidential MCP (cMCP), which emits a Trust Record on every session.

Why adoption is picking

TRACE development preview launched at the Confidential Computing Summit in June 2026. The official Python library is agentrust-trace, and the identity records drew nearly 135,000 PyPI downloads within ten weeks.

Hardware-rooted attestation is the engine. A Trusted Platform Module or a Trusted Execution Environment signs a measurement of the code and configuration running inside an isolated space. A verifier compares that evidence against expected reference values before releasing keys, credentials, or access. The same pattern now extends to agent execution.

What this means for your stack

A Trust Record gives every agent run a signed log line you can prove. That matters for compliance, for multi-agent handoffs, and for the “who do we blame when the agent does the wrong thing” question.

The hardware certification layer stays cheap. You do not need a new cloud. The standard works across enterprise, cloud, and sovereign AI infrastructure. The specification, docs, and reference implementation are public today.

Roll with it on your next audit trail. One signed artifact that survives a third-party check is stronger than the most detailed log you wrote yourself.