davsclaus opened a new pull request, #30440:
URL: https://github.com/apache/camel/pull/30440

   ## Description
   
   [CAMEL-27181](https://issues.apache.org/jira/browse/CAMEL-27181): the Data 
Lake consumer reads every file of the file system (or of `path`) at every poll. 
It had no option to delete a file once its exchange completed: the 
on-completion only logged at TRACE. camel-azure-storage-blob has 
`deleteAfterRead`; `DataLakeConfiguration` had nothing.
   
   - New consumer option `deleteAfterRead` (default `false`, so nothing changes 
for existing routes).
   - The file name is taken when the exchange is created, so a route that 
changes the `CamelAzureStorageDataLakeFileName` header still deletes the file 
it consumed.
   - In `onComplete`, the file is deleted with 
`DataLakeFileClientWrapper.delete` (`deleteWithResponse`).
     - A 404 (already deleted, for example by another consumer) is logged at 
DEBUG.
     - Other failures go to the exception handler of the consumer.
   - A failed exchange keeps the file, so it is read again at the next poll.
   - New paragraph and example in the component doc. The catalog, its doc copy 
and the DSLs are regenerated.
   
   `moveAfterRead`, which the JIRA mentions as a possible follow-up, is not 
part of this PR.
   
   Needed for a delete-after-read option on the `azure-storage-datalake-source` 
Kamelet (apache/camel-kamelets#3280).
   
   ## Tests
   
   `DataLakeConsumerDeleteAfterReadTest` uses a mocked service client, as 
`DataLakeConsumerFileThatCannotBeGotTest` does:
   - files are deleted after they are consumed, and each is consumed once
   - the original file is deleted when the route changes the header
   - a failed exchange keeps the file, which is read again
   - without the option, nothing is deleted
   - a failed delete reaches the exception handler
   
   All 37 unit tests of the module pass. The new test passed 3 runs in a row.
   
   _Claude Code on behalf of Claus Ibsen_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to