Toivo You can replicate the relationship post listener to each calculation process. Send all outputs of calc processes to mergecontent and have it correlate based on some shared id attribute and keep all unique attributes. You will end up with the content concatenated but the calculations would end up on the attributes.
The latency of this approach will approximately be whatever the length of the slowest calculation is. Thanks Joe On Jun 18, 2015 5:52 AM, "Toivo Adams" <[email protected]> wrote: > Please see > http://www.enterpriseintegrationpatterns.com/BroadcastAggregate.html > > My use case is little bit different but overall pattern is same. > Instead of sending requests to different vendors some data must be > calculated. > Processors Calc_A, Calc_B, Calc_C and Calc_D calculate values A, B, C and > D. > All processors are independent from each other. Each processor use only > data > from request as input. > Every request contains unique requestId which will be attached to every > FlowFile during processing. > > How to implement this in NiFi? > > 1. Sequential processing > Listener -> Calc_A -> Calc_B -> Calc_C -> Calc_D -> > BuildResponse -> SendResponse > > Each Calc_* processor will add calculated value to FlowFile. > So After Calc_B FlowFile contains A and B values. > And After Calc_C FlowFile contains A, B and C values. > BuildResponse incoming FlowFile contains A, B, C and D values and building > response is easy. > > Sequential processing works but when Calc_* processors are slow overall > service latency is too big. > > 2. Parallel processing > After Listener we can send request FlowFile to all Calc_* processors > simultaneously. > Problem is how to aggregate values A, B, C and D from different processors? > > This question may belong to user list, but I thought solutions are probably > not trivial and maybe some overall development is reasonable? > > Thanks > Toivo > > > > > -- > View this message in context: > http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/How-to-implement-Scatter-Gather-tp1944.html > Sent from the Apache NiFi (incubating) Developer List mailing list archive > at Nabble.com. >
