Mikołaj Stefaniak created KAFKA-9066:
----------------------------------------
Summary: Kafka Connect JMX : source & sink task metrics missing
for tasks in failed state
Key: KAFKA-9066
URL: https://issues.apache.org/jira/browse/KAFKA-9066
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Affects Versions: 2.1.1
Reporter: Mikołaj Stefaniak
h2. Overview
Kafka Connect exposes various metrics via JMX. Those metrics can be exported
i.e. by _Prometheus JMX Exporter_ for further processing.
One of crucial attributes is connector's *task status.*
According to official Kafka docs, status is available as +status+ attribute of
following MBean:
{quote}kafka.connect:type=connector-task-metrics,connector="\{connector}",task="\{task}"status
- The status of the connector task. One of 'unassigned', 'running', 'paused',
'failed', or 'destroyed'.
{quote}
h2. Issue
Generally +connector-task-metrics+ are exposed propery for tasks in +running+
status but not exposed at all if task is +failed+.
Failed Task *appears* properly with failed status when queried via *REST API*:
{code:java}
$ curl -X GET -u 'user:pass'
http://kafka-connect.mydomain.com/connectors/customerconnector/status
{"name":"customerconnector","connector":{"state":"RUNNING","worker_id":"kafka-connect.mydomain.com:8080"},"tasks":[{"id":0,"state":"FAILED","worker_id":"kafka-connect.mydomain.com:8080","trace":"org.apache.kafka.connect.errors.ConnectException:
Received DML 'DELETE FROM mysql.rds_sysinfo ......"}],"type":"source"}
$ {code}
Failed Task *doesn't appear* as bean with +connector-task-metrics+ type when
queried via *JMX*:
{code:java}
$ echo "beans -d kafka.connect" | java -jar
target/jmxterm-1.1.0-SNAPSHOT-uber.jar -l localhost:8081 -n -v silent | grep
connector=customerconnector
kafka.connect:connector=customerconnector,task=0,type=task-error-metricskafka.connect:connector=customerconnector,type=connector-metrics
$
{code}
h2. Expected result
It is expected, that bean with +connector-task-metrics+ type will appear also
for tasks that failed.
Below is example of how beans are properly registered for tasks in Running
state:
{code:java}
$ echo "get -b
kafka.connect:connector=sinkConsentSubscription-1000,task=0,type=connector-task-metrics
status" | java -jar target/jmxterm-1.1.0-SNAPSHOT-ube r.jar -l localhost:8081
-n -v silent
status = running;
$
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
