Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

132 points

by NicoConstant

6 hours ago

65 comments