Support for priority based queueing in the backend queue

[ziadmoubayed]

Feature request

Add support for priority based queue in the tgi backend. A multi-level priority queue with arrival time tie-breaker for request scheduling in the TGI v3 backend. This queue should prioritize requests based on a client-defined numerical priority value, and within each priority level, use arrival time (First-Come, First-Served) to resolve ties.

Default implementation is the existing FIFO queue. Users can choose a different queue using tgi --batch-queue priority/fifo

Motivation

In a multi-tenant environment, this allows us to guarantee SLAs for high prio inference requests, while maintaining efficient resource utilization. Implementing a priority-based queue allows the system to differentiate between critical and non-critical workloads, ensuring that important requests are processed preferentially. The arrival time tie-breaker within each priority level introduces fairness and prevents newer requests from consistently taking precedence over older ones with the same priority.
With this approach we try to balance between the need for performance guarantees and the desire to maximize resource utilization, ie: minimize costs compared to full isolation strategies.

Your contribution

Happy to pick this up.