[
https://issues.apache.org/jira/browse/KAFKA-16961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855205#comment-17855205
]
Luke Chen edited comment on KAFKA-16961 at 6/15/24 9:52 AM:
------------------------------------------------------------
caused by KAFKA-16969
was (Author: showuon):
caused by KAFKA-16961
> TestKRaftUpgrade system tests fail in v3.7.1 RC1
> ------------------------------------------------
>
> Key: KAFKA-16961
> URL: https://issues.apache.org/jira/browse/KAFKA-16961
> Project: Kafka
> Issue Type: Test
> Reporter: Luke Chen
> Priority: Major
>
>
>
> {code:java}
> ================================================================================
> SESSION REPORT (ALL TESTS)
> ducktape version: 0.11.4
> session_id: 2024-06-14--003
> run time: 86 minutes 13.705 seconds
> tests run: 24
> passed: 18
> flaky: 0
> failed: 6
> ignored: 0
> ================================================================================
> test_id:
> kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.1.2.use_new_coordinator=False.metadata_quorum=ISOLATED_KRAFT
> status: PASS
> run time: 3 minutes 44.680 seconds
> --------------------------------------------------------------------------------
> test_id:
> kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.1.2.use_new_coordinator=True.metadata_quorum=ISOLATED_KRAFT
> status: PASS
> run time: 3 minutes 42.627 seconds
> --------------------------------------------------------------------------------
> test_id:
> kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.2.3.use_new_coordinator=False.metadata_quorum=ISOLATED_KRAFT
> status: PASS
> run time: 3 minutes 28.205 seconds
> --------------------------------------------------------------------------------
> test_id:
> kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.2.3.use_new_coordinator=True.metadata_quorum=ISOLATED_KRAFT
> status: PASS
> run time: 3 minutes 42.388 seconds
> --------------------------------------------------------------------------------
> test_id:
> kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.3.2.use_new_coordinator=False.metadata_quorum=ISOLATED_KRAFT
> status: PASS
> run time: 2 minutes 57.679 seconds
> --------------------------------------------------------------------------------
> test_id:
> kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.3.2.use_new_coordinator=True.metadata_quorum=ISOLATED_KRAFT
> status: PASS
> run time: 2 minutes 57.238 seconds
> --------------------------------------------------------------------------------
> test_id:
> kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.4.1.use_new_coordinator=False.metadata_quorum=ISOLATED_KRAFT
> status: PASS
> run time: 2 minutes 52.545 seconds
> --------------------------------------------------------------------------------
> test_id:
> kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.4.1.use_new_coordinator=True.metadata_quorum=ISOLATED_KRAFT
> status: PASS
> run time: 2 minutes 56.289 seconds
> --------------------------------------------------------------------------------
> test_id:
> kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.5.2.use_new_coordinator=False.metadata_quorum=ISOLATED_KRAFT
> status: PASS
> run time: 2 minutes 54.953 seconds
> --------------------------------------------------------------------------------
> test_id:
> kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.5.2.use_new_coordinator=True.metadata_quorum=ISOLATED_KRAFT
> status: PASS
> run time: 2 minutes 59.579 seconds
> --------------------------------------------------------------------------------
> test_id:
> kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=dev.use_new_coordinator=False.metadata_quorum=ISOLATED_KRAFT
> status: PASS
> run time: 3 minutes 21.016 seconds
> --------------------------------------------------------------------------------
> test_id:
> kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=dev.use_new_coordinator=True.metadata_quorum=ISOLATED_KRAFT
> status: PASS
> run time: 2 minutes 56.175 seconds
> --------------------------------------------------------------------------------
> test_id:
> kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=3.1.2.use_new_coordinator=False.metadata_quorum=COMBINED_KRAFT
> status: PASS
> run time: 3 minutes 6.505 seconds
> --------------------------------------------------------------------------------
> test_id:
> kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=3.1.2.use_new_coordinator=True.metadata_quorum=COMBINED_KRAFT
> status: PASS
> run time: 3 minutes 8.059 seconds
> --------------------------------------------------------------------------------
> test_id:
> kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=3.2.3.use_new_coordinator=False.metadata_quorum=COMBINED_KRAFT
> status: PASS
> run time: 3 minutes 19.250 seconds
> --------------------------------------------------------------------------------
> test_id:
> kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=3.2.3.use_new_coordinator=True.metadata_quorum=COMBINED_KRAFT
> status: PASS
> run time: 3 minutes 33.163 seconds
> --------------------------------------------------------------------------------
> test_id:
> kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=3.3.2.use_new_coordinator=False.metadata_quorum=COMBINED_KRAFT
> status: FAIL
> run time: 4 minutes 19.037 seconds
> TimeoutError("Kafka server didn't finish startup in 120 seconds")
> Traceback (most recent call last):
> File
> "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py",
> line 186, in _do_run
> data = self.run_test()
> File
> "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py",
> line 246, in run_test
> return self.test_context.function(self.test)
> File "/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line
> 433, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
> File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py",
> line 116, in test_combined_mode_upgrade
> self.run_upgrade(from_kafka_version)
> File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py",
> line 105, in run_upgrade
> self.run_produce_consume_validate(core_test_action=lambda:
> self.perform_version_change(from_kafka_version))
> File "/opt/kafka-dev/tests/kafkatest/tests/produce_consume_validate.py",
> line 105, in run_produce_consume_validate
> core_test_action(*args)
> File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py",
> line 105, in <lambda>
> self.run_produce_consume_validate(core_test_action=lambda:
> self.perform_version_change(from_kafka_version))
> File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py",
> line 63, in perform_version_change
> self.kafka.controller_quorum.start_node(node)
> File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 894, in
> start_node
> self.wait_for_start(node, monitor, timeout_sec)
> File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 898, in
> wait_for_start
> monitor.wait_until("Kafka\s*Server.*started", timeout_sec=timeout_sec,
> backoff_sec=.25,
> File
> "/usr/local/lib/python3.9/dist-packages/ducktape/cluster/remoteaccount.py",
> line 753, in wait_until
> return wait_until(lambda: self.acct.ssh("tail -c +%d %s | grep '%s'" %
> (self.offset + 1, self.log, pattern),
> File "/usr/local/lib/python3.9/dist-packages/ducktape/utils/util.py", line
> 58, in wait_until
> raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from
> last_exception
> ducktape.errors.TimeoutError: Kafka server didn't finish startup in 120
> seconds--------------------------------------------------------------------------------
> test_id:
> kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=3.3.2.use_new_coordinator=True.metadata_quorum=COMBINED_KRAFT
> status: FAIL
> run time: 4 minutes 51.680 seconds
> TimeoutError("Kafka server didn't finish startup in 120 seconds")
> Traceback (most recent call last):
> File
> "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py",
> line 186, in _do_run
> data = self.run_test()
> File
> "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py",
> line 246, in run_test
> return self.test_context.function(self.test)
> File "/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line
> 433, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
> File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py",
> line 116, in test_combined_mode_upgrade
> self.run_upgrade(from_kafka_version)
> File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py",
> line 105, in run_upgrade
> self.run_produce_consume_validate(core_test_action=lambda:
> self.perform_version_change(from_kafka_version))
> File "/opt/kafka-dev/tests/kafkatest/tests/produce_consume_validate.py",
> line 105, in run_produce_consume_validate
> core_test_action(*args)
> File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py",
> line 105, in <lambda>
> self.run_produce_consume_validate(core_test_action=lambda:
> self.perform_version_change(from_kafka_version))
> File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py",
> line 63, in perform_version_change
> self.kafka.controller_quorum.start_node(node)
> File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 894, in
> start_node
> self.wait_for_start(node, monitor, timeout_sec)
> File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 898, in
> wait_for_start
> monitor.wait_until("Kafka\s*Server.*started", timeout_sec=timeout_sec,
> backoff_sec=.25,
> File
> "/usr/local/lib/python3.9/dist-packages/ducktape/cluster/remoteaccount.py",
> line 753, in wait_until
> return wait_until(lambda: self.acct.ssh("tail -c +%d %s | grep '%s'" %
> (self.offset + 1, self.log, pattern),
> File "/usr/local/lib/python3.9/dist-packages/ducktape/utils/util.py", line
> 58, in wait_until
> raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from
> last_exception
> ducktape.errors.TimeoutError: Kafka server didn't finish startup in 120
> seconds--------------------------------------------------------------------------------
> test_id:
> kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=3.4.1.use_new_coordinator=False.metadata_quorum=COMBINED_KRAFT
> status: FAIL
> run time: 5 minutes 1.860 seconds
> TimeoutError("Kafka server didn't finish startup in 120 seconds")
> Traceback (most recent call last):
> File
> "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py",
> line 186, in _do_run
> data = self.run_test()
> File
> "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py",
> line 246, in run_test
> return self.test_context.function(self.test)
> File "/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line
> 433, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
> File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py",
> line 116, in test_combined_mode_upgrade
> self.run_upgrade(from_kafka_version)
> File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py",
> line 105, in run_upgrade
> self.run_produce_consume_validate(core_test_action=lambda:
> self.perform_version_change(from_kafka_version))
> File "/opt/kafka-dev/tests/kafkatest/tests/produce_consume_validate.py",
> line 105, in run_produce_consume_validate
> core_test_action(*args)
> File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py",
> line 105, in <lambda>
> self.run_produce_consume_validate(core_test_action=lambda:
> self.perform_version_change(from_kafka_version))
> File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py",
> line 63, in perform_version_change
> self.kafka.controller_quorum.start_node(node)
> File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 894, in
> start_node
> self.wait_for_start(node, monitor, timeout_sec)
> File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 898, in
> wait_for_start
> monitor.wait_until("Kafka\s*Server.*started", timeout_sec=timeout_sec,
> backoff_sec=.25,
> File
> "/usr/local/lib/python3.9/dist-packages/ducktape/cluster/remoteaccount.py",
> line 753, in wait_until
> return wait_until(lambda: self.acct.ssh("tail -c +%d %s | grep '%s'" %
> (self.offset + 1, self.log, pattern),
> File "/usr/local/lib/python3.9/dist-packages/ducktape/utils/util.py", line
> 58, in wait_until
> raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from
> last_exception
> ducktape.errors.TimeoutError: Kafka server didn't finish startup in 120
> seconds--------------------------------------------------------------------------------
> test_id:
> kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=3.4.1.use_new_coordinator=True.metadata_quorum=COMBINED_KRAFT
> status: FAIL
> run time: 4 minutes 50.405 seconds
> TimeoutError("Kafka server didn't finish startup in 120 seconds")
> Traceback (most recent call last):
> File
> "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py",
> line 186, in _do_run
> data = self.run_test()
> File
> "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py",
> line 246, in run_test
> return self.test_context.function(self.test)
> File "/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line
> 433, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
> File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py",
> line 116, in test_combined_mode_upgrade
> self.run_upgrade(from_kafka_version)
> File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py",
> line 105, in run_upgrade
> self.run_produce_consume_validate(core_test_action=lambda:
> self.perform_version_change(from_kafka_version))
> File "/opt/kafka-dev/tests/kafkatest/tests/produce_consume_validate.py",
> line 105, in run_produce_consume_validate
> core_test_action(*args)
> File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py",
> line 105, in <lambda>
> self.run_produce_consume_validate(core_test_action=lambda:
> self.perform_version_change(from_kafka_version))
> File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py",
> line 63, in perform_version_change
> self.kafka.controller_quorum.start_node(node)
> File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 894, in
> start_node
> self.wait_for_start(node, monitor, timeout_sec)
> File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 898, in
> wait_for_start
> monitor.wait_until("Kafka\s*Server.*started", timeout_sec=timeout_sec,
> backoff_sec=.25,
> File
> "/usr/local/lib/python3.9/dist-packages/ducktape/cluster/remoteaccount.py",
> line 753, in wait_until
> return wait_until(lambda: self.acct.ssh("tail -c +%d %s | grep '%s'" %
> (self.offset + 1, self.log, pattern),
> File "/usr/local/lib/python3.9/dist-packages/ducktape/utils/util.py", line
> 58, in wait_until
> raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from
> last_exception
> ducktape.errors.TimeoutError: Kafka server didn't finish startup in 120
> seconds--------------------------------------------------------------------------------
> test_id:
> kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=3.5.2.use_new_coordinator=False.metadata_quorum=COMBINED_KRAFT
> status: FAIL
> run time: 4 minutes 48.846 seconds
> TimeoutError("Kafka server didn't finish startup in 120 seconds")
> Traceback (most recent call last):
> File
> "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py",
> line 186, in _do_run
> data = self.run_test()
> File
> "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py",
> line 246, in run_test
> return self.test_context.function(self.test)
> File "/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line
> 433, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
> File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py",
> line 116, in test_combined_mode_upgrade
> self.run_upgrade(from_kafka_version)
> File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py",
> line 105, in run_upgrade
> self.run_produce_consume_validate(core_test_action=lambda:
> self.perform_version_change(from_kafka_version))
> File "/opt/kafka-dev/tests/kafkatest/tests/produce_consume_validate.py",
> line 105, in run_produce_consume_validate
> core_test_action(*args)
> File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py",
> line 105, in <lambda>
> self.run_produce_consume_validate(core_test_action=lambda:
> self.perform_version_change(from_kafka_version))
> File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py",
> line 63, in perform_version_change
> self.kafka.controller_quorum.start_node(node)
> File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 894, in
> start_node
> self.wait_for_start(node, monitor, timeout_sec)
> File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 898, in
> wait_for_start
> monitor.wait_until("Kafka\s*Server.*started", timeout_sec=timeout_sec,
> backoff_sec=.25,
> File
> "/usr/local/lib/python3.9/dist-packages/ducktape/cluster/remoteaccount.py",
> line 753, in wait_until
> return wait_until(lambda: self.acct.ssh("tail -c +%d %s | grep '%s'" %
> (self.offset + 1, self.log, pattern),
> File "/usr/local/lib/python3.9/dist-packages/ducktape/utils/util.py", line
> 58, in wait_until
> raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from
> last_exception
> ducktape.errors.TimeoutError: Kafka server didn't finish startup in 120
> seconds--------------------------------------------------------------------------------
> test_id:
> kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=3.5.2.use_new_coordinator=True.metadata_quorum=COMBINED_KRAFT
> status: FAIL
> run time: 4 minutes 19.672 seconds
> TimeoutError("Kafka server didn't finish startup in 120 seconds")
> Traceback (most recent call last):
> File
> "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py",
> line 186, in _do_run
> data = self.run_test()
> File
> "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py",
> line 246, in run_test
> return self.test_context.function(self.test)
> File "/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line
> 433, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
> File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py",
> line 116, in test_combined_mode_upgrade
> self.run_upgrade(from_kafka_version)
> File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py",
> line 105, in run_upgrade
> self.run_produce_consume_validate(core_test_action=lambda:
> self.perform_version_change(from_kafka_version))
> File "/opt/kafka-dev/tests/kafkatest/tests/produce_consume_validate.py",
> line 105, in run_produce_consume_validate
> core_test_action(*args)
> File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py",
> line 105, in <lambda>
> self.run_produce_consume_validate(core_test_action=lambda:
> self.perform_version_change(from_kafka_version))
> File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py",
> line 63, in perform_version_change
> self.kafka.controller_quorum.start_node(node)
> File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 894, in
> start_node
> self.wait_for_start(node, monitor, timeout_sec)
> File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 898, in
> wait_for_start
> monitor.wait_until("Kafka\s*Server.*started", timeout_sec=timeout_sec,
> backoff_sec=.25,
> File
> "/usr/local/lib/python3.9/dist-packages/ducktape/cluster/remoteaccount.py",
> line 753, in wait_until
> return wait_until(lambda: self.acct.ssh("tail -c +%d %s | grep '%s'" %
> (self.offset + 1, self.log, pattern),
> File "/usr/local/lib/python3.9/dist-packages/ducktape/utils/util.py", line
> 58, in wait_until
> raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from
> last_exception
> ducktape.errors.TimeoutError: Kafka server didn't finish startup in 120
> seconds--------------------------------------------------------------------------------
> test_id:
> kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=dev.use_new_coordinator=False.metadata_quorum=COMBINED_KRAFT
> status: PASS
> run time: 3 minutes 11.294 seconds
> --------------------------------------------------------------------------------
> test_id:
> kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=dev.use_new_coordinator=True.metadata_quorum=COMBINED_KRAFT
> status: PASS
> run time: 3 minutes 9.933 seconds
> --------------------------------------------------------------------------------
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
