Bryan,
This is largely what I was trying to touch on in my last response to Toivo. It’s also what I was touching on in the last blog that I wrote: https://blogs.apache.org/nifi/entry/say_good_bye_to_canned This is largely how I test all of my software at my “day job.” I spawn a duplicate feed to push to a test instance. That duplicate feed could then be modified/obfuscated as needed for the test cluster. Or you could filter out only part of it to send to the test cluster. Does this make sense? Or am I just confusing people? 😊 From: Bryan Bende Sent: Sunday, February 22, 2015 7:44 PM To: [email protected] What if you are making significant changes to a custom processor? Using Toivo's example, if there is a custom processor for inserting to a relational database, and the graph has one instance inserting to a production database, and another inserting to a test database, is there a good way to test your latest Nar and not affect the production part of the flow? -Bryan On Sun, Feb 22, 2015 at 5:59 PM, Corey Flowers <[email protected]> wrote: > Good afternoon Toivo, > > I work nifi operations/integration daily, on very vital datasets, > and can tell you that we too had to change the views and procedures of our > customers/leadership to accept this type of thinking. NIFI is a step > forward in the evolution of software development not a software to be > placed in previous software development methodologies. Our team has had > this fight numerous times, from canned data, to test environments to > upgrading policies. We even had one customer tell us, we needed a 17 day > notice just to restart NIFI because all previous versions of their software > had to be started in a sequence, previously took 2 hours to start back up > and this was their current operating posture. In our daily activities our > operations personnel work hand in hand with the developers to integrate new > processors into production environments, usually by cloning production data > into grouped development processors on the production graph. This allows us > to expedite the integration process, save money on building test > environments and also allows us to see real load on the production > suite/cluster. There is no doubt, this is not only a change in development > processes but also the mindsets around software development in general. I > can only assure you the fight is well worth it in the end. > > Corey > > On Sun, Feb 22, 2015 at 4:46 PM, Toivo Adams <[email protected]> > wrote: > > > Joe, > > > > Thank you for explanation. > > > > I hope I understand NiFi main idea. > > And it's really well thought and implemented. > > I like it very much. > > > > But. > > “Ultimately the idea of production versus test often means fairly large > > protracted cycles from idea to production outcome.” > > Welcome to my world. > > I work for financial institution. > > > > Maybe I am naive but I do see NiFi as general way how to do software > > development in many business domains. > > I hate to hear NiFi is not usable in financial institution. > > > > I've seen a lot of different business applications. > > One of worst of them are EJB style monolithic web applications. > > I have learned that software should be implemented as side-effect free > > components which will do only one thing but do it very well. Software > > development is very costly and reusing components is the key to > > keep development cost at reasonable level. > > Also debugging and scalability is much simple using such components. > > So NiFi is perfect fit. > > > > IDEAL world > > Development and operations are separated. > > Almost none are allowed to see live data because it contains highly > > sensitive customer data. > > Breaking rules may lead immediate firing. (this is not theory, it has > > happen > > few times) > > So development must use scrambled test data. > > Before even single simple change can be applied to Live system, strict > > change management procedure must be pass through. Yes, this takes usually > > 1-2 days when everything is OK. > > And deploy will be done by operations (not by developers). > > Positive is what all changes to live systems are recorded and anytime > roll > > back can be done quickly (s*it happens). > > When whatever new application or change is in live, it might be running > > many > > years without any efforts from development team. > > > > REAL world > > Sometimes something still goes wrong whatever the reason is. > > Usually monitoring will get alert and will forward problem to > predetermined > > person(persons) > > When operations (administrations) are unable to resolve problem this will > > end up to some developer leader desk. In this case developer is > authorized > > to be use live data to solve problem quickly. > > Now NiFi can again be very, very valuable. > > > > Summary > > I want to use NiFi but at the same time I must follow our strict > test/live > > environment rules. > > Or NiFi would not be accepted at all. > > > > > > Thanks > > toivo > > > > > > > > > > -- > > View this message in context: > > > http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/Great-question-on-nifi-IRC-room-today-NiFi-BPM-sharing-configuration-tp787p801.html > > Sent from the Apache NiFi (incubating) Developer List mailing list > archive > > at Nabble.com. > > > > > > -- > Corey Flowers > Vice President, Onyx Point, Inc > (410) 541-6699 > [email protected] > > -- This account not approved for unencrypted proprietary information -- >
