vitorsantos-commits commented on issue #6745: URL: https://github.com/apache/hop/issues/6745#issuecomment-4023870933
I understand that from a pure architectural standpoint, this might seem like a design issue. However, let me clarify the practical scenario I’m facing: Imagine a pipeline processing 2,000 rows that requires an authentication token from 'API A' to be used in subsequent calls to 'API B' for each of those records. Currently, if the REST Client is placed in this stream, it would hit the authentication endpoint 2,000 times—one for each row. This is highly inefficient, unnecessary, and often triggers rate-limiting (429 errors) or security flags from the API provider. To solve this today, I have to use a 'Sample Rows' transform followed by a 'Database Join' or 'Stream Lookup' to broadcast that single token back to all 2,000 rows. This adds significant complexity and 'boilerplate' transforms to what should be a straightforward process. By having a 'Send only first row' option, the REST Client could act as a singleton trigger within a dense data stream. It would fetch the token once using the first row's context and then allow the pipeline to propagate that value efficiently without redundant API calls. My goal is to simplify the pipeline design for these common 'setup' or 'authentication' patterns that depend on stream data. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
