Toivo, If the results of the calculation are put into FlowFile attributes leaving the content untouched, you could also use a ReplaceText processor after all but one of your calc processors to replace the content with nothing. Then the MergeContent's binary concatenation would still leave you with the original content.
Brandon On Thu, Jun 18, 2015, 6:41 AM Joe Witt <[email protected]> wrote: > Toivo > > You can replicate the relationship post listener to each calculation > process. Send all outputs of calc processes to mergecontent and have it > correlate based on some shared id attribute and keep all unique > attributes. You will end up with the content concatenated but the > calculations would end up on the attributes. > > The latency of this approach will approximately be whatever the length of > the slowest calculation is. > > Thanks > Joe > On Jun 18, 2015 5:52 AM, "Toivo Adams" <[email protected]> wrote: > > > Please see > > http://www.enterpriseintegrationpatterns.com/BroadcastAggregate.html > > > > My use case is little bit different but overall pattern is same. > > Instead of sending requests to different vendors some data must be > > calculated. > > Processors Calc_A, Calc_B, Calc_C and Calc_D calculate values A, B, C and > > D. > > All processors are independent from each other. Each processor use only > > data > > from request as input. > > Every request contains unique requestId which will be attached to every > > FlowFile during processing. > > > > How to implement this in NiFi? > > > > 1. Sequential processing > > Listener -> Calc_A -> Calc_B -> Calc_C -> Calc_D -> > > BuildResponse -> SendResponse > > > > Each Calc_* processor will add calculated value to FlowFile. > > So After Calc_B FlowFile contains A and B values. > > And After Calc_C FlowFile contains A, B and C values. > > BuildResponse incoming FlowFile contains A, B, C and D values and > building > > response is easy. > > > > Sequential processing works but when Calc_* processors are slow overall > > service latency is too big. > > > > 2. Parallel processing > > After Listener we can send request FlowFile to all Calc_* processors > > simultaneously. > > Problem is how to aggregate values A, B, C and D from different > processors? > > > > This question may belong to user list, but I thought solutions are > probably > > not trivial and maybe some overall development is reasonable? > > > > Thanks > > Toivo > > > > > > > > > > -- > > View this message in context: > > > http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/How-to-implement-Scatter-Gather-tp1944.html > > Sent from the Apache NiFi (incubating) Developer List mailing list > archive > > at Nabble.com. > > >
