Important Things
1) CPU/latency-bound
Kernel launch overheads, especially in low-batch size scenarios, can be significant. Low batch sizes imply less work for the GPU, meaning the GPU is more likely to finish its work before the CPU can launch the next task.
2) TKLQT as a metric
Total kernel launch and queuing time. The measured time between start of kernel launch on the host and start of kernel execution on the GPU.
Conceptually, kernel launch overhead is the first half of TKLQT, and queuing time is the second half.
At low batch sizes, if TKLQT is flat while GPU utilization is low, likely to be dominated by kernel launch overhead.
At high batch sizes, TKLQT can still be high despite high GPU utilization, indicating GPU queuing time.
3) Important of single-thread performance
Fast clock, high IPC, and low-latency memory accesses are all ideal for the host CPU, who is responsible for kernel launch and orchestration.
4) CPU choice matters for launch overheads, interconnect choice matters for kernel performance
CPU choice matters less for the kernel performance (duh). The interconnect needs both high bandwidth for fast data movement on closely/tightly coupled systems.
Glaring Deficiency
The paper, over a year old, is already a little dated. Study of encoder-only models, focus on small batch size / low-latency as “chat/conversation” workloads.
It does not study multi-GPU interactions, larger models, nor more realistic input/output request shapes (agentic workloads of 2026).
Takeaways
Focusing on GPU performance and utilization makes the CPU-side of LLM inference acceleration an Amdahl’s law problem. Push the GPU side of the balloon down, and the CPU side inflates. Kernels get faster, meaning the CPU needs to be faster to keep the GPU busy.
Thus highlighting the heavy importance of operator fusion where possible up until GPU utilization is saturated.