Cutting LLM Batch Inference Time in Half: Dynamic Prefix Bucketing at Scale

5 points

by ykev

4 hours ago

1 comments