NiFi experts, Let's say that I want to send data from NiFi to some destination that works much better when the documents are batched. I do not think this is an unreasonable ask.
I imagine that I would want to first combine all of the records in one processor, and then pass on to a dedicated processor for sending the data? I'm not sure yet if I would be able to use existing processors for this, or if I could create my own, but this part feels fairly straightforward. Next, let's imagine that some document in the batch causes it to fail. I would like to un-batch, and create smaller batches, and try to send those, assuming that some piece of the data was malformed and not a transient error like network unavailable. Is this pattern workable? I can imagine several layers of fail/split/retry to winnow from 1000 documents to 100 to 10 to 1, so that I can still get most of my data sent and know exactly which documents fail. I'm largely thinking out loud here, somebody stop me if I'm off the deep end, or if this has been done before and we have examples (I didn't see any readily apparent). Mike
