v0.31.1rc0: [Metrics] Expose cached prompt tokens by cache tier (#56318)
<p>Signed-off-by: Cam Quilici <a href="mailto:[email protected]">[email protected]</a><br> Signed-off-by: Cam Quilici <a href="mailto:[email protected]">[email protected]</a><br> Co-authored-by: Cam Qui...
Verified State Diff
Impact & Verification Analysis
Developers, software engineers, and active vLLM users.
Enhances platform capabilities, developer velocity, and product capabilities.
Full Fact Overview
<p>Signed-off-by: Cam Quilici <a href="mailto:[email protected]">[email protected]</a><br> Signed-off-by: Cam Quilici <a href="mailto:[email protected]">[email protected]</a><br> Co-authored-by: Cam Quilici <a href="mailto:[email protected]">[email protected]</a><br> Co-authored-by: Nick Hill <a href="mailto:[email protected]">[email protected]</a><br> Co-authored-by: Yifan Qiao <a href="mailto:[email protected]">[email protected]</a></p>