Toivo,
I think Corey’s e-mail here is a response directly to your comment of “Before even single simple change can be applied to Live system, strict change management procedure must be pass through. Yes, this takes usually 1-2 days when everything is OK.” I think the only thing that I can add in this regard is that changing your flow is NOT a change to software. It is simply a change to software configuration. If a configuration change requires days then you may have an uphill battle, but just as Corey is recommending, I still think it’s worth a fight. You made the point that “development must use scrambled test data.” I don’t think this is necessarily a problem. We suggested teeing off a copy of the data and pushing to both a production database and a test database. That doesn’t mean that the exact same data must go to both. You could easily have a processor ahead of the “Put to Test Database” processor that performs the obfuscation for you. For example, if the data is XML or JSON, perhaps the standard processors will be all you need to obfuscate it. Otherwise, it’s possible that you'd need to create a new processor that can handle the obfuscation. Then your flow would send the actual data to the production database and then send a modified version to the test database. This way, while you will not see the actual live data as-is, you will see as close as possible to the real thing. This allows you still to operate on the live stream of data as it comes in so that you can determine your ability to handle the load as well as handle the structure of the data (though account numbers, etc. would be randomly generated numbers, for instance, the JSON would be the same structure, at least). This still is far better than “test data” that was generated a while ago. However, I am arguing that this obfuscated test data is great for testing new processors and new code. It should not be needed to test the flow. The “test" version of the flow should be built alongside the production flow and after it has been verified, you just make the switch and delete the old version. I come from an extremely stringent environment as well. This ability of NiFi to take a live stream of data, process it to meet production needs, and then modify the data to send to a test environment (or perhaps filter data and only send data that is deemed OK as-is to a test environment) is in my opinion one of the most powerful features. Does this make sense? From: Corey Flowers Sent: Sunday, February 22, 2015 6:01 PM To: [email protected] Good afternoon Toivo, I work nifi operations/integration daily, on very vital datasets, and can tell you that we too had to change the views and procedures of our customers/leadership to accept this type of thinking. NIFI is a step forward in the evolution of software development not a software to be placed in previous software development methodologies. Our team has had this fight numerous times, from canned data, to test environments to upgrading policies. We even had one customer tell us, we needed a 17 day notice just to restart NIFI because all previous versions of their software had to be started in a sequence, previously took 2 hours to start back up and this was their current operating posture. In our daily activities our operations personnel work hand in hand with the developers to integrate new processors into production environments, usually by cloning production data into grouped development processors on the production graph. This allows us to expedite the integration process, save money on building test environments and also allows us to see real load on the production suite/cluster. There is no doubt, this is not only a change in development processes but also the mindsets around software development in general. I can only assure you the fight is well worth it in the end. Corey On Sun, Feb 22, 2015 at 4:46 PM, Toivo Adams <[email protected]> wrote: > Joe, > > Thank you for explanation. > > I hope I understand NiFi main idea. > And it's really well thought and implemented. > I like it very much. > > But. > “Ultimately the idea of production versus test often means fairly large > protracted cycles from idea to production outcome.” > Welcome to my world. > I work for financial institution. > > Maybe I am naive but I do see NiFi as general way how to do software > development in many business domains. > I hate to hear NiFi is not usable in financial institution. > > I've seen a lot of different business applications. > One of worst of them are EJB style monolithic web applications. > I have learned that software should be implemented as side-effect free > components which will do only one thing but do it very well. Software > development is very costly and reusing components is the key to > keep development cost at reasonable level. > Also debugging and scalability is much simple using such components. > So NiFi is perfect fit. > > IDEAL world > Development and operations are separated. > Almost none are allowed to see live data because it contains highly > sensitive customer data. > Breaking rules may lead immediate firing. (this is not theory, it has > happen > few times) > So development must use scrambled test data. > Before even single simple change can be applied to Live system, strict > change management procedure must be pass through. Yes, this takes usually > 1-2 days when everything is OK. > And deploy will be done by operations (not by developers). > Positive is what all changes to live systems are recorded and anytime roll > back can be done quickly (s*it happens). > When whatever new application or change is in live, it might be running > many > years without any efforts from development team. > > REAL world > Sometimes something still goes wrong whatever the reason is. > Usually monitoring will get alert and will forward problem to predetermined > person(persons) > When operations (administrations) are unable to resolve problem this will > end up to some developer leader desk. In this case developer is authorized > to be use live data to solve problem quickly. > Now NiFi can again be very, very valuable. > > Summary > I want to use NiFi but at the same time I must follow our strict test/live > environment rules. > Or NiFi would not be accepted at all. > > > Thanks > toivo > > > > > -- > View this message in context: > http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/Great-question-on-nifi-IRC-room-today-NiFi-BPM-sharing-configuration-tp787p801.html > Sent from the Apache NiFi (incubating) Developer List mailing list archive > at Nabble.com. > -- Corey Flowers Vice President, Onyx Point, Inc (410) 541-6699 [email protected] -- This account not approved for unencrypted proprietary information --
