SamWheating commented on issue #63371: URL: https://github.com/apache/airflow/issues/63371#issuecomment-4176645607
@kacpermuda thanks for the update! That PR is a big improvement but I agree that this is still something we should allow configuring on a case-by-case basis. There might also be cases where users want to set a low default, but override with higher values for a particular hook. Maybe adding something into `BaseHook` which allows us to override `max_assets_per_collector` would be useful here? This would definitely need to be wired into every single lineage-enabled hook (similar to how hook-level lineage was implemented in the first place), but maybe the total effort here isn't too high. I'm a little behind on my understanding of the internals here, but I can take a stab at this later this week / next. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
