[
https://issues.apache.org/jira/browse/HADOOP-14623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079033#comment-16079033
]
Hongyuan Li edited comment on HADOOP-14623 at 7/8/17 9:15 AM:
--------------------------------------------------------------
I highly recommend Four steps:
1、should use {{acks}} = {{1}}.
2、add {{https://repository.apache.org/content/repositories/releases}} repo,
the {{apache snapshot rep}} doesnot have a higher version kafka module, the
version of which is less than {{0.8.2}}
3、update kafka client version to at least {{0.10.1.0}}, which has a
IntegerSerializer class If kafka sink want to generate a kafka producer with
the the type of key being Integer.
4、 Use ProducerConfig.XXX instead of using string value directly. For example,
use
{{ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG}} instead of {{key.serializer}}
Thanks for any advice.
was (Author: hongyuan li):
I highly recommend Five steps:
1、should use {{acks}} = {{1}}.
2、add {{https://repository.apache.org/content/repositories/releases}} repo,
the {{apache snapshot rep}} doesnot have a higher version kafka module, the
version of which is less than {{0.8.2}}
3、update kafka client version to at least {{0.10.1.0}}, which has a
IntegerSerializer class If kafka sink want to generate a kafka producer with
the the type of key being Integer.
4、add {{callback}} when using new {{KafkaProducer}}#{{send}}
5、 Use ProducerConfig.XXX instead of using string value directly. For example,
use
{{ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG}} instead of {{key.serializer}}
Thanks for any advice.
> fixed some bugs in KafkaSink
> -----------------------------
>
> Key: HADOOP-14623
> URL: https://issues.apache.org/jira/browse/HADOOP-14623
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, tools
> Affects Versions: 3.0.0-alpha3
> Reporter: Hongyuan Li
> Assignee: Hongyuan Li
> Attachments: HADOOP-14623-001.patch
>
>
> {{KafkaSink}}#{{init}} should set ack to *1* to make sure the message has
> been written to the broker at least.
> current code list below:
> {code}
>
> props.put("request.required.acks", "0");
> {code}
> *Update*
> find another bug about this class, {{key.serializer}} used
> {{org.apache.kafka.common.serialization.ByteArraySerializer}}, however, the
> key properties of Producer is Integer, codes list below:
> {code}
> props.put("key.serializer",
> "org.apache.kafka.common.serialization.ByteArraySerializer");
> ……………
> producer = new KafkaProducer<Integer, byte[]>(props);
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
