Thank you for your responses Jonathan and Rob. Appreciate the clarifications.
I am not really trying to do ETL but mainly collating and loading. I will play around with Heka and will get back in touch to see how I can contribute - in travels so trying things out in between. I would like to use Heka for what I am working on. Thank you again for the great product you have designed and are developing. Mono On Wed, Jan 21, 2015 at 1:14 PM, Rob Miller <[email protected]> wrote: > I mostly agree with Jonathan's post. I agree completely with his > descriptions of Heka and Kafka, and how they differ. I wouldn't write Heka > off as a choice for ETL, though. It can be nice when your ETL tool provides > the option for doing crunching, graphing, and alerting on the gathered data > while it's in transit. > > Anyway, my responses to your questions: > > 1. Yes, Heka can be used. It's primary use case is stream processing, but > collecting and shipping data is an important part of stream processing, so > you can pu that functionality to other use. > > 2. Right, you can't find good examples for that, because there aren't any. > AFAIK nobody has written either a postgres or mongo output plugin, those > would have to exist before you could push in. You could probably use the > HttpOutput to push data into at least mongo, though. > > 3. Kafka is heavier, and different. Jonathan covered this nicely. > > 4. If you're still interested in trying Heka for your needs, and are > serious about wanting to contribute, you might start by trying to write a > postgres output. > > Hope this info is useful, sorry we don't have a better solution for you. > > -r > > > On 01/20/2015 09:22 PM, Monosij Dutta-Roy wrote: > >> Hello Rob and Heka users - >> >> I was exploring Heka (on Ubuntu VM) and had a few questions. i will try >> to keep them brief. >> >> ... >> 1. Can Heka be used, being used as an ETL tool to load data from various >> files into DBMS? or is it mainly for log processing and/or stream >> processing? >> >> 2. To me it seems that it can be, but I cannot seem to find good >> examples / tutorials to go through in figuring out config files to load >> into Postgres and MongoDB simultaneously - my use case. >> >> ... >> 3. I was comparing it as an alternative to the Mesos / Kafka / Aurora >> architecture - which seems much heavier than what Heka accomplishes much >> simpler. But again in going through the tutorials on your site I am not >> able to configure for DBMS from the flat files I am loading. >> >> The tutorial seems to want to write a log file in my /var/heka directory >> - whuch I had to remove my root permissions from (in my VM). Perhaps >> some of this baseline files, should be configurable of course, but >> perhaps start off in ~/heka? >> >> ... >> 4. Again, I am interested in using this towards ETL - if there are some >> examples you may be able to point me to would be great. I hope to learn >> to use it and then contribute towards its development as well. >> >> Thank you much for your help. >> >> And thank you for thinking through the details, in identifying, >> collating the shortcomings of other tools, in coming up with this great >> toolset. >> >> Mono >> > >
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
