Kubernetes cache architecture explained: why reads go through local copies
A Kubernetes blog post corrects prior technical inaccurate descriptions of how controller-runtime caches objects and how changes propagate through the system.
Disclaimer
This article was produced by Scroll Insights News Desk using automated systems and published under our standing editorial policy. It is compiled from the primary sources linked above and is provided for general information only — it is not legal, financial, investment, tax or professional advice, and no decision should be taken on it without independent verification against those sources. Errors can be reported to corrections@scrollinsights.com and are corrected on the record.
Controller-runtime operates against a local copy of data populated through list and watch rather than querying the API server directly 1. Reads via r.Get() and r.List() inside a reconciler read from this in-memory cache instead of the API server, while writes go straight to the server and do not pass through the cache.
How the cache fills and stays current
The Reflector opens a watch with sendInitialEvents=true by default, causing the API server to begin the stream with synthetic ADDED events for the whole current state before switching to live changes. A reconciler does not start invoking Reconcile until every source it owns reports synced, including its event handlers having processed the initial snapshot. Between execution of Update and reflection of the new state in the cache there is a window, usually milliseconds with no guaranteed upper bound.
Why reads cache and writes do not
Mgr.GetClient() reads go through the cache, while mgr.GetAPIReader() goes straight to the API server. Writes (Create, Update, Patch, Apply, Delete, DeleteAllOf) go straight to the API server in client.Client. A Get for a type nothing registered a watch for starts a new informer on the spot and blocks until it is warm. The regular client returns ErrCacheNotStarted if you attempt to read objects before mgr.Start().
How contention at scale led to architectural choices
The Indexer is backed by a ThreadSafeStore which uses a map keyed by namespace/name, plus a single sync.RWMutex that guards both the store and every index. A bottleneck in kube-controller-manager at scale was caused by this sync.RWMutex guarding the store, documented in kubernetes#130767. Predicates and handlers in controller subscriptions receive the same objects that live in the informer's shared store, not copies, so operations on them do not require additional locking.
Queue defaults changed in client-go 1.36
RealFIFO is a flat, strictly ordered slice of deltas with no deduplication and no per-object slots. Since client-go 1.36, DeltaFIFO cannot be switched back on and RealFIFO is now the default queue used by shared informers. PopBatch for processing several deltas in one pass has been on by default since client-go 1.35. Deduplication of reconcile requests happens in the controller's workqueue layer, not in the delta queue.