dachaostripe opened a new issue, #9545: URL: https://github.com/apache/pinot/issues/9545
Currently there's no effective way of searching and gathering all related logs for a particular query or life cycle/control event. As of now, there's only a numeric requestId like `124927` being generated on broker and get sent to online and offline servers to tag a few log messages, but that doesn't include potential exceptions in case of errors. This can cause huge difficulties when troubleshooting errors in production 1. The simple numeric requestId is highly overlapped among hosts in the fleet, searching for such ids are likely to hit many unrelated logs from other hosts; using UUID or similar is a better choice. 2. Not tagging all related logs(especially exceptions) with the appropriate id means that we have no reliable way of finding the exact error for a given request, especially in high traffic environments where it's impossible to search for "nearby logs" to hopefully find related errors 3. There are similar needs for requests received from controller, and life cycle events detected by ZooKeeper, or new online/offline servers notifying it joining the cluster, etc. Ideally 1. both broker and controller APIs should also accept a special header, like `x-pinot-request-id`, to allow the client to provide an ID that Pinot will use to tag its logs, to allow better correlation with client side logs 2. broker and controller will only generate a unique ID in case `x-pinot-request-id` isn't provided by client; or when this is an internally initiated events, like new hosts joining, unhealthy host detected etc. 3. Broker and controller should also include `x-pinot-request-id` header in the response to client, so client can potentially log it, which is also useful for look up related Pinot logs, in case for some reasons the client doesn't provide this id when making the requests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
