Aug-28-2024, 07:02 AM
When we were conducting stress testing on vllm with a load of 30QPS, we found an anomaly with a 300ms delay, which occurred more frequently as the QPS increased. Looking from nsight, one thread's utilization rate was at 100%, but the Python stack was empty.
The input token for the experimental data was 40, and the output token was 20.
This situation is very strange.
For more specific experimental data, please see
https://github.com/vllm-project/vllm/issues/7540
The input token for the experimental data was 40, and the output token was 20.
This situation is very strange.
For more specific experimental data, please see
https://github.com/vllm-project/vllm/issues/7540
