mirror of
https://github.com/haproxy/haproxy.git
synced 2026-11-04 14:59:44 +00:00
Willy Tarreau
345aa2b73e
Common HTTP load generation tools are not convenient to measure latency through haproxy when it comes to server-sent events, because the measurements do not apply to the partial contents that SSE are. These are particularly important for AI environments where agents need to minimize latency. This commit introduces a new tool that is both a client and a server, and that is meant to measure an intermediary's added latency. For this, it connects to itself to measure its own inflicted latency (loopback test) then does it again through the configured proxy, and reports both as well as the traversal time. The principle is to note the timestamp at which each SSE is sent, and measure the difference when received by the client, since both sides are the same process and use the same clock reference. The output shows the reference time (loopback), the gateway time, and the gateway overhead as the difference between the two. The tool supports adding more clients, doing more requests, delaying the first token response, and setting the inter-token latency. An example haproxy config is provided (gateway.cfg) that measures both with and without option http-no-delay. Without http-no-delay (+ ~100ms, max 200 due to Nagle): === loopback (direct to listen address): 1 requests, 1 ok, 0 failed, 200 tokens === TTFT (client) n=1 mean=0.04ms min=0.04ms p50=0.04ms p95=0.04ms p99=0.04ms max=0.04ms TTFT emit->recv(1st) n=1 mean=0.00ms min=0.00ms p50=0.00ms p95=0.00ms p99=0.00ms max=0.00ms inter-token (client) n=199 mean=5.00ms min=4.07ms p50=5.13ms p95=5.33ms p99=5.36ms max=5.38ms emit->recv (tokens) n=199 mean=0.04ms min=0.01ms p50=0.03ms p95=0.07ms p99=0.18ms max=0.19ms === gateway: 1 requests, 1 ok, 0 failed, 1000 tokens === TTFT (client) n=1 mean=207.17ms min=207.17ms p50=207.17ms p95=207.17ms p99=207.17ms max=207.17ms TTFT emit->recv(1st) n=1 mean=206.86ms min=206.86ms p50=206.86ms p95=206.86ms p99=206.86ms max=206.86ms inter-token (client) n=999 mean=4.80ms min=0.00ms p50=0.00ms p95=0.00ms p99=207.98ms max=208.97ms emit->recv (tokens) n=999 mean=104.04ms min=1.02ms p50=103.88ms p95=197.67ms p99=205.89ms max=208.07ms === gateway overhead (gateway minus loopback; negative = loopback noise) === emit->recv (1st token) mean +206.851ms p50 +206.851ms p95 +206.851ms p99 +206.851ms emit->recv (tokens) mean +104.002ms p50 +103.851ms p95 +197.599ms p99 +205.709ms TTFT (client) mean +207.122ms p50 +207.122ms p95 +207.122ms p99 +207.122ms inter-token (client) mean -0.205ms p50 -5.128ms p95 -5.331ms p99 +202.624ms With http-no-delay (+ ~40 microseconds max): === loopback (direct to listen address): 1 requests, 1 ok, 0 failed, 200 tokens === TTFT (client) n=1 mean=0.05ms min=0.05ms p50=0.05ms p95=0.05ms p99=0.05ms max=0.05ms TTFT emit->recv(1st) n=1 mean=0.01ms min=0.01ms p50=0.01ms p95=0.01ms p99=0.01ms max=0.01ms inter-token (client) n=199 mean=5.00ms min=4.03ms p50=5.06ms p95=5.27ms p99=5.34ms max=5.59ms emit->recv (tokens) n=199 mean=0.04ms min=0.01ms p50=0.03ms p95=0.07ms p99=0.18ms max=0.49ms === gateway: 1 requests, 1 ok, 0 failed, 1000 tokens === TTFT (client) n=1 mean=0.18ms min=0.18ms p50=0.18ms p95=0.18ms p99=0.18ms max=0.18ms TTFT emit->recv(1st) n=1 mean=0.03ms min=0.03ms p50=0.03ms p95=0.03ms p99=0.03ms max=0.03ms inter-token (client) n=999 mean=5.00ms min=4.04ms p50=5.10ms p95=5.31ms p99=5.37ms max=5.45ms emit->recv (tokens) n=999 mean=0.07ms min=0.03ms p50=0.07ms p95=0.13ms p99=0.20ms max=0.28ms === gateway overhead (gateway minus loopback; negative = loopback noise) === emit->recv (1st token) mean +0.026ms p50 +0.026ms p95 +0.026ms p99 +0.026ms emit->recv (tokens) mean +0.034ms p50 +0.036ms p95 +0.059ms p99 +0.029ms TTFT (client) mean +0.135ms p50 +0.135ms p95 +0.135ms p99 +0.135ms inter-token (client) mean -0.002ms p50 +0.042ms p95 +0.041ms p99 +0.035ms