CPU affinity and load balancing
On a machine with several CPUs, a scheduler usually gives each CPU its own run queue. A single shared queue would need a lock that every CPU contends for on every switch, and a thread that keeps returning to the same CPU finds its data still in that CPU's caches. Per-CPU queues bring two decisions of their own. Placement picks a CPU for a thread when it is created or woken, typically the least loaded one. Load balancing moves threads afterwards, when one queue runs long while another CPU sits idle: the idle CPU pulls, or steals, a thread from a busy one, or a busy CPU pushes one away.
Affinity limits where a thread may go. Hard affinity, or pinning, restricts a thread to one CPU or a set of them, for a per-CPU thread such as an idle thread or for code that touches per-CPU state. Soft affinity is only a preference, the cache warmth a scheduler weighs against the load.
The trap is per-CPU state that travels with the thread. A policy that keeps a per-CPU clock, such as the virtual time of a fair-share scheduler, has to rebase a migrating thread on the destination's clock, or the thread arrives far ahead of or far behind every thread there. A migration also touches the queues of two CPUs at once, so it has to be safe against the ticks of both, and a balancer must never steal a pinned thread.
In Cosmos the kernel runs on one CPU today. The hooks that place and balance threads, SelectCpu and Balance, have no caller, and OnThreadMigrate runs only when a policy's own Balance calls it. The shape is per-CPU all the same: every hook that touches a run structure receives the PerCpuState it operates on, a policy keeps its run structure in that state's SchedulerData, and SchedulerThread.CpuId names the CPU a thread is assigned to, so a policy written per CPU now carries over. SchedulerThreadFlags.Pinned marks a thread that must not migrate; only the idle threads carry it, since a kernel can read a thread's flags but not set them. Stride's Balance steals the tail of the longest queue when its own is empty and that queue holds more than one thread, unless the tail is pinned, and its OnThreadMigrate rebases the thread's Pass on the destination's GlobalPass plus the thread's saved Remain. See Limitations and evolution.