Thanks Adam, I guess I was using templates because I didn't know about the existence of flow.xml.gz :). Now, if I have 2 different instances of NiFi, and I want to override the workflow in one of them with the workflow of the other, do I need to do anything else than just overwrite flow.xml.gz?
Regards, XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Esteban Aliverti - Blog @ http://ilesteban.wordpress.com On Wed, Apr 29, 2015 at 2:04 PM, Adam Taft <[email protected]> wrote: > You can get a lot of mileage using the following suggestions: > > - There's nothing preventing you from storing the flow.xml.gz file into > version control. While this file is binary and you won't see any diffs on > it, your VCS (git, svn, etc.) won't have a problem storing it. If storing > the flow.xml.gz as binary is a problem, then you could simply uncompress it > to flow.xml first and version control that. > > - You might likewise want to version control your entire conf directory. > > - For any processor properties which can be configured using NIFI's > expression language, you can reference System properties and/or environment > variables. This allows you to have variant configurations based on the > deployment context. Things like hostnames or database connection properties > could in theory be configured externally. > > - Your custom controller services can likewise be configured to utilize > system properties or environment vars. Most standard controller services > don't have this capability yet, but work is being done that might help in > this area as well. > > I personally don't like using templates to solve your problem. Having flow > snippets that need to be stored and applied between dev/int/prod doesn't > make a lot of sense to me. Better in my mind is to have a single > flow.xml.gz that you can carry between those tiers. NIFI should allow you > to get most of the way there today, but if you run into problems, I'm sure > you could affect additional changes to help with this issue. > > Adam > > > > On Tue, Apr 28, 2015 at 9:58 AM, Esteban Aliverti < > [email protected]> wrote: > > > Joe, > > thanks for your prompt response. > > > > Let me give you more background about what I'm currently doing: > > We are currently a group of 3 developers working on a NiFi workflow. My > two > > colleagues are in charge of building the individual processors required > for > > the workflow we are trying to create. My role, even if I'm also a > Processor > > developer myself, is kind of the "integrator" of these processors. I > have a > > local NiFi running and I'm in charge of the "workflow" design. I use this > > local instance as a playground for develop and initial testing. > > Once I've updated my local workflow, I usually "export" it into another > > instance that is currently being used by QA. This QA instance is only > > modified after I validate that my local instance is stable enough. Given > > that there are a number of other developers using this QA instance as > > clients, the update process is not so easy. Before I introduce any change > > in QA, I need to coordinate with the QA department first. This is one of > > the biggest reasons I keep a local NiFi instance for development. I can > do > > whatever I want whenever I want to in my local instance but I need to be > > very careful when I modify the QA instance. > > > > I have asked similar questions in this forum in the past and one of your > > recommendations was to use the same workflow for both QA and DEV. You > > suggested me to use an attribute in the FlowFiles as some kind of "flag" > so > > it can be routed to the DEV branch of the workflow. I see 2 problems with > > this approach (maybe because of my own development workflow): > > 1.- Sometimes I don't have connectivity to the server where the QA NiFi > > instance is. I would like to be able to continue my development tasks > even > > if I'm offline. > > 2.- My workflow is not stateless. I have a bunch of Controller Services > > that are used through the workflow that intentionally keep state. If I > use > > the same NiFi instance for both QA and DEV, I will not only have to add a > > new "flag" attribute in my workflows, but I will also has to implement > some > > kind of mechanism in my Controller Services as well to separate DEV and > QA > > data. I don't think this is a really good idea. > > > > Regards, > > > > > > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > > > > Esteban Aliverti > > - Blog @ http://ilesteban.wordpress.com > > > > On Tue, Apr 28, 2015 at 3:34 PM, Joe Witt <[email protected]> wrote: > > > > > Esteban, > > > > > > Thank you for providing so much detail to what you're trying to do. I > > > want to walk through this further with you to better understand the > > > perspective. But first I will say that the pain you refer to with > > > removing a flow is something we will sort out. You should be able to > > > simply right-click and delete so we will improve that until it is > > > true. > > > > > > Can you explain why you want to build on one instance, then make a > > > template, then go to another instance? I am wondering if this is a > > > sort of dev/integration/production workflow. > > > > > > I think our general philosophy has been to enable developers to easily > > > build new extensions which are effectively unit tested and then put > > > those out on the instance where the data is flowing. We then optimize > > > extensively around the operations/administrator experience. We've > > > seen a consistent theme of need though to clarify how the developer > > > experience can/should work. So happy to take more feedback on what > > > you're trying to do and also 'why' you're trying to do it so we can > > > help guide and so we can learn how to adjust as needed. > > > > > > Thanks > > > Joe > > > > > > On Tue, Apr 28, 2015 at 8:49 AM, Esteban Aliverti > > > <[email protected]> wrote: > > > > Hi guys, > > > > I was wondering what is the best way to replace a complete workflow > in > > > > NiFi. > > > > My scenario is that I have multiple instances of NiFi running the > same > > > > workflow. At some point, when changes are introduced into one of the > > > > instances (let's say a new processor is introduced) I would like to > > > > replicate the changes to the other instances. For small changes, > things > > > can > > > > be easily handled, but for big changes - introduction of a whole new > > > > branch, for example - things are not so easy. > > > > Given that the workflows I'm currently using are in dev stage, I > don't > > > mind > > > > to lose any FlowFile that already exists in the NiFi instance I'm > > > updating. > > > > > > > > The way I'm currently handling this is: > > > > 1.- I create a new template from the NiFi instance where the changes > > were > > > > introduced. > > > > 2.- I export the template as an xml file. > > > > 3.- I import the template in the NiFi instance where I want the > changes > > > to > > > > be applied. > > > > 4.- I manually remove any processor from the NiFi instance where I > want > > > the > > > > changes to be applied. (for complex workflows, this step is a real > > > pita!). > > > > 5.- I use the template I've previously imported in the NiFi instance > > > where > > > > I want the changes to be applied. > > > > > > > > My question is: what is the best way to achieve this? > > > > Is there any way to completely delete the workflow that NiFi is > > currently > > > > executing? > > > > > > > > Regards, > > > > > > > > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > > > > > > > > Esteban Aliverti > > > > - Blog @ http://ilesteban.wordpress.com > > > > > >
