gortiz commented on issue #16001: URL: https://github.com/apache/pinot/issues/16001#issuecomment-2943796722
I've created #16007, where I'm modifying log in case of error to include the stats. I've also included another log that creates the same summary in case of successful execution, but that log can be very spammy, so it is logged at DEBUG level to be disabled by default. A new marker can be used to enable just that log instead of all debug logs from the class. See the PR to see an example. If needed, Log4j2's [BusterFilter](https://logging.apache.org/log4j/2.x/manual/filters.html#BurstFilter) can be used to apply a rate to these logs. The stats already contain most of the information requested here: 1. Query success or failure: Different messages are logged on each case 2. Query latency: Included in the summary 3. Concise representation of the stage graph (1->[2,3],2->4,etc) including which stages are leafs: This is included in the stats. 4. Which stages of the query succeeded or failed: We cannot include that right now, more info below. 5. For stages that failed, which servers failed the complete the stage: In order to have that info we need to merge https://github.com/apache/pinot/pull/15919 first. 6. How long (wall clock) time did each stage take: Stats for each operator include `clockTimeMs`, which is calculated as `executionTimeMs / parallelism`. Right now, `executionTimeMs` is 0 for stages that fail, but that is something we will try to improve. It is not easy to include how many servers have failed because Pinot aborts the execution as soon as the first error is received. We may decide to change that in the future, but it would require significant changes to the code and likely increase latency in error handling, so it is unlikely that we will invest resources in this area in the short term. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
