Mike, Typing on phone now. Will respond more thoroughly when I'm at computer in a couple of hours. Short answer: look at PutKafka. It does something similar if sending delineated records.
Thanks -Mark Sent from my iPhone > On Feb 24, 2015, at 9:30 AM, Mike Drob <[email protected]> wrote: > > NiFi experts, > > Let's say that I want to send data from NiFi to some destination that works > much better when the documents are batched. I do not think this is an > unreasonable ask. > > I imagine that I would want to first combine all of the records in one > processor, and then pass on to a dedicated processor for sending the data? > I'm not sure yet if I would be able to use existing processors for this, or > if I could create my own, but this part feels fairly straightforward. > > Next, let's imagine that some document in the batch causes it to fail. I > would like to un-batch, and create smaller batches, and try to send those, > assuming that some piece of the data was malformed and not a transient > error like network unavailable. Is this pattern workable? I can imagine > several layers of fail/split/retry to winnow from 1000 documents to 100 to > 10 to 1, so that I can still get most of my data sent and know exactly > which documents fail. > > I'm largely thinking out loud here, somebody stop me if I'm off the deep > end, or if this has been done before and we have examples (I didn't see any > readily apparent). > > Mike
