vvivekiyer opened a new pull request, #13847: URL: https://github.com/apache/pinot/pull/13847
This PR introduces the BinaryWorkloadScheduler, which categorizes queries into two workloads: 1. Primary: Default workload. 2. Secondary: Assigned when the isSecondaryWorkload query option is specified. Primary Workload: • Queries are executed using the existing unbounded FCFS (First-Come, First-Served) scheduler. Secondary Workload: • Queries are processed with several constraints: • Limited number of in-progress secondary workload queries (excess queries are queued). • Limited number of worker threads per secondary workload query. • Limited number of worker threads across all queries in the secondary workload. • Queries stuck in the queue for an extended period can be pruned based on time or queue length. This ensures that Primary Workload queries are always prioritized, capping the impact of secondary queries on primary queries. Usage at LinkedIn: We've been running this at Linkedin for some time now. • Primary: All production traffic. • Secondary: Traffic from ad-hoc queries via dashboards / notebooks, debugging tools, development environments, one-off tests, etc to protect the availability of cluster. [Benchmarking Results ](https://docs.google.com/document/d/1ZzpZAJ4xAqOuFcEX8j-Mr2AcIaoSDowBov1oh3dVTTo/edit) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
