jasonyanwenl edited a comment on pull request #5769: URL: https://github.com/apache/incubator-pinot/pull/5769#issuecomment-670657377
Hi @suvodeep-pyne, thanks again for your feedback! And sorry for the late response. For each of your questions: > I assume that the online worker must be on when such an API request is issued? What is the behavior if that is not the case? Yes. The online worker must be on. If that is not the case, the endpoint will timeout during the polling task status stage and then return a failed response back to users. > I wanted to understand the necessity of having 2 TaskDrivers. How is an online TaskDriver different in behavior than an offline task driver? Same question for online detection task vs offline detection task. There are some unique states needed to be maintained by the online driver. Here are some main differences: * When task driver is acquiring a waiting task, an online task driver should not pick a task created by web app (Similar for the web app driver). Different task drivers will pick different tasks based on the task type (DETECTION or DETECTION_ONLINE). * Online task driver will not retrieve detection config from DB. Instead, it retrieves config from task info. Please check this [comment](https://github.com/apache/incubator-pinot/pull/5769#discussion_r463312439) for the reason. * Online task driver will have two extra ThirdEye metrics: `duration counter` and `tasks counter` > Is this about prioritizing online tasks? If yes, have we considered having a priority attached to a task? Currently, no priority is attached. But we do want to have that in the future. I guess providing dedicated priority algorithms is another topic and a little bit out of the scope of my current intern project given the limited time so I didn't support that now. But I think we should do that in the future. I will keep in mind about your suggestions on readability. They are really practical. Thanks! Thanks again for your feedback! ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
