[
https://issues.apache.org/jira/browse/GEARPUMP-167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16004425#comment-16004425
]
Karol Brejna commented on GEARPUMP-167:
---------------------------------------
[~yanghua] I started developing PoC on the following branch:
https://github.com/karol-brejna-i/incubator-gearpump/tree/GEARPUMP-167-elasticsearch-sink
My idea was to have a simple sink that would just accept a JSON document to be
stored in elasticsearch (not an object that would need to be serialized to
JSON).
I anticipated two cases for the incoming message:
* Tuple2[String, String] - id, body - message gets stored under given id, so
it's actually an upsert
* String - body - message is stored under new id
* A sequence of any of above
I even did some short "demo" that had:
* small app subscribing to Twitter and putting messages to Kafka
* a DAG that consumed messages from Kafka, did wordcount, stored data to
elasticsearch
* Kibana visualization
The code was written in Java. I planned to convert it to Scala but got
distracted by the last release...
[~yanghua] I'd be glad to hear your idea on this. Maybe we could cooperate on
this feature.
> Create Elasticsearch sink and example
> -------------------------------------
>
> Key: GEARPUMP-167
> URL: https://issues.apache.org/jira/browse/GEARPUMP-167
> Project: Apache Gearpump
> Issue Type: Task
> Components: connectors
> Reporter: Karol Brejna
> Assignee: Karol Brejna
> Labels: feature
> Original Estimate: 18h
> Remaining Estimate: 18h
>
> Create a simple Elasticsearch sink that enables indexing messages as
> Elasticsearch docs.
> Write an example demonstrating usage of the sink
> Write a doc on setting up local development env (installing and confifuring
> elastic, etc.) for the users of the sink
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
