Draft: Probe simulation until request complete
I modified the prober to implement simulation-based probing and experimented with the impact of checking one iteration vs. multiple iterations. The ProbeType parameter configures the metric the global scheduler will search for, and we have different policies. Some as before only need one iteration (E.g., PREFILL_E2E, SCHEDULE_DELAY). Some on the other hand will continue until the target request is finished.
TPOT metric is improved when we consider the impact of our scheduling decision on the rest of the requests in the replica:
Purple curve considering TPOT90 for requests before the new scheduled request is finished.
TPOT:
Scheduling Delay:
TTFT:
Request E2E Time:
-
I think considering the impact of the scheduler's decision on the other requests has potential to work on. As see, TPOT90 probe outperforms the single iteration metrics for TPOT (which is the goal here). However, at very high load, it messes up the E2E time of requests.
-
This might be because of the phenomenon Erfan mentioned: maybe we should also consider the fact that the last decode iterations are too optimistic since load is not being increased!
-
Prefill_e2e makes more sense as a probe over scheduling delay since it includes the scheduling delay plus execution time.
-
Also, all of these metrics are greedily optimized, we might need to come up with a multi-dimensional metric-policy.
P.S: The running time is not good, but for finding a good policy, we can use smaller scale simulations (4-8 replicas).
After we find the best working solution, we can optimize it further.