What’s Changed
Amazon SageMaker Inference has introduced “prefix-aware routing,” a new routing strategy to reduce the inference latency of large language models (LLMs). Traditional random routing dispersed the same prompt prefixes across multiple instances, making cache reuse difficult. To solve this problem, a mechanism has been introduced to always send requests with the same prefix to the same instance.
(Reference: aws.amazon.com)
How It Works
Prefix-aware routing determines the routing destination based on the initial part of the request. This allows requests with the same prefix to be aggregated to the same instance, enabling effective cache utilization. Two safeguards have been implemented:
- Overload Protection: If a popular prefix is concentrated on a specific instance, the load of that instance is avoided by distributing requests to other instances.
- Scaling Stability: When instances are added or removed, most requests are sent to the previous instance, minimizing cache invalidation.
(Reference: Ibid [aws.amazon.com])
Performance Characteristics
In a benchmark using Llama 3.1 70B Instruct, the following improvements were confirmed with prefix-aware routing:
- P50 TTFT (time to first token) was reduced by up to 77%
- Total throughput was improved by up to 16%
- KV cache hit rate was improved from 25% to over 80%
(Reference: Ibid [aws.amazon.com])
Summary
- Utilize prefix-aware routing to reduce LLM inference latency by up to 77%. Cache reuse of prefixes improves processing efficiency.
- Ensure cache stability during scaling. Cache invalidation due to instance addition or removal is minimized.
- Improve model throughput by up to 16%, enabling efficient resource utilization.