This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel.git
The following commit(s) were added to refs/heads/main by this push:
new 112995b37383 CAMEL-26774: camel-chaos - experiments in the journal of
an exploration: a hypothesis, its rules and expectations, and a verdict (#29480)
112995b37383 is described below
commit 112995b37383081db2a295a5a6b0c0ca26bfee35
Author: Claus Ibsen <[email protected]>
AuthorDate: Mon Nov 9 16:14:49 2026 +0100
CAMEL-26774: camel-chaos - experiments in the journal of an exploration: a
hypothesis, its rules and expectations, and a verdict (#29480)
Co-Authored-By: Claude Opus 5.5 <[email protected]>
---
.../org/apache/camel/catalog/docs/chaos.adoc | 44 ++++
components/camel-chaos/src/main/docs/chaos.adoc | 44 ++++
.../camel/component/chaos/ChaosExperiment.java | 242 +++++++++++++++++++++
.../camel/component/chaos/ChaosExploration.java | 51 ++++-
.../apache/camel/component/chaos/ChaosManager.java | 117 +++++++++-
.../camel/component/chaos/ChaosExperimentTest.java | 186 ++++++++++++++++
.../apache/camel/cli/connector/ChaosAction.java | 19 +-
.../LocalCliConnectorChaosActionTest.java | 33 +++
.../dsl/jbang/core/commands/process/ListChaos.java | 61 +++++-
.../jbang/core/commands/process/ListChaosTest.java | 25 ++-
10 files changed, 810 insertions(+), 12 deletions(-)
diff --git
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/chaos.adoc
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/chaos.adoc
index 746bab5ac884..00bdae584b34 100644
---
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/chaos.adoc
+++
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/chaos.adoc
@@ -376,6 +376,50 @@ $ camel get chaos --report
baseline (before the first rule): orders 30 completed, 0 failed
----
+==== Experiments
+
+The agent tries one thing at a time as an experiment: a hypothesis, such as
"an order never fails when payment answers
+503", and the rules that test it. At least one of the rules has an expectation
(`expect=`), which decides the
+hypothesis. The rules of an experiment are added together, all or none, and
the counts per route are read when it
+begins, so its end says what changed while it ran. One experiment is open at a
time, and only while an exploration
+runs.
+
+When it ends, camel-chaos reads what came of it and then removes its rules:
per route the difference of the counts, the
+faults of each rule, the state of each expectation, and a verdict:
+
+* `FAIL`: an expectation is unmet.
+* `PASS`: all expectations are met.
+* `PENDING`: it is still open and an expectation is not decided yet.
+* `INCONCLUSIVE`: it ended before an expectation was decided, so the messages
may still have come.
+
+The experiment is ended with the exploration when that finishes, or when its
time is up, with what came of it so far.
+
+The `chaos` action of the CLI connector begins one (`{"action": "chaos",
"command": "experiment-begin", "hypothesis":
+"an order never fails", "rule": "endpoint=http:payment* error status=503
until=1m expect=orders:failed<=0
+within=70s"}`), with several rules separated by `;`, and ends it
(`experiment-end`, with an optional `note` of what the
+agent makes of it). In Java, these are
`ChaosManager.beginExperiment(hypothesis, rules)` and `endExperiment(note)`.
+
+The journal has them as `experiments`, each with:
+
+* `number`, `hypothesis`, the `rules` (their ids) and `ruleTexts`
+* `open`, `startedAt` and `elapsed`
+* the `verdict` and the `expectations`: per rule its `expect`, `state` (`met`,
`unmet` or `pending`), the `actual` value,
+and the `value` in words
+* the `faults` per rule
+* the `routes` that did something while it ran, with the same counts as the
baseline
+* once ended, `endedAt`, `endedBy` (`ended`, `finished` or `deadline`) and the
`note`
+
+[source,text]
+----
+$ camel get chaos --report
+...
+ experiments:
+ 1 FAIL an order never fails (r1, r2, ended after 1m4s)
+ expect orders:failed<=0: unmet, 9 failed
+ routes: orders 41 completed, 9 failed
+ note: orders has no error handler
+----
+
== Scenarios
A scenario is the rules of an experiment in a file of the project, so it can
be run again, shared and kept with the
diff --git a/components/camel-chaos/src/main/docs/chaos.adoc
b/components/camel-chaos/src/main/docs/chaos.adoc
index 746bab5ac884..00bdae584b34 100644
--- a/components/camel-chaos/src/main/docs/chaos.adoc
+++ b/components/camel-chaos/src/main/docs/chaos.adoc
@@ -376,6 +376,50 @@ $ camel get chaos --report
baseline (before the first rule): orders 30 completed, 0 failed
----
+==== Experiments
+
+The agent tries one thing at a time as an experiment: a hypothesis, such as
"an order never fails when payment answers
+503", and the rules that test it. At least one of the rules has an expectation
(`expect=`), which decides the
+hypothesis. The rules of an experiment are added together, all or none, and
the counts per route are read when it
+begins, so its end says what changed while it ran. One experiment is open at a
time, and only while an exploration
+runs.
+
+When it ends, camel-chaos reads what came of it and then removes its rules:
per route the difference of the counts, the
+faults of each rule, the state of each expectation, and a verdict:
+
+* `FAIL`: an expectation is unmet.
+* `PASS`: all expectations are met.
+* `PENDING`: it is still open and an expectation is not decided yet.
+* `INCONCLUSIVE`: it ended before an expectation was decided, so the messages
may still have come.
+
+The experiment is ended with the exploration when that finishes, or when its
time is up, with what came of it so far.
+
+The `chaos` action of the CLI connector begins one (`{"action": "chaos",
"command": "experiment-begin", "hypothesis":
+"an order never fails", "rule": "endpoint=http:payment* error status=503
until=1m expect=orders:failed<=0
+within=70s"}`), with several rules separated by `;`, and ends it
(`experiment-end`, with an optional `note` of what the
+agent makes of it). In Java, these are
`ChaosManager.beginExperiment(hypothesis, rules)` and `endExperiment(note)`.
+
+The journal has them as `experiments`, each with:
+
+* `number`, `hypothesis`, the `rules` (their ids) and `ruleTexts`
+* `open`, `startedAt` and `elapsed`
+* the `verdict` and the `expectations`: per rule its `expect`, `state` (`met`,
`unmet` or `pending`), the `actual` value,
+and the `value` in words
+* the `faults` per rule
+* the `routes` that did something while it ran, with the same counts as the
baseline
+* once ended, `endedAt`, `endedBy` (`ended`, `finished` or `deadline`) and the
`note`
+
+[source,text]
+----
+$ camel get chaos --report
+...
+ experiments:
+ 1 FAIL an order never fails (r1, r2, ended after 1m4s)
+ expect orders:failed<=0: unmet, 9 failed
+ routes: orders 41 completed, 9 failed
+ note: orders has no error handler
+----
+
== Scenarios
A scenario is the rules of an experiment in a file of the project, so it can
be run again, shared and kept with the
diff --git
a/components/camel-chaos/src/main/java/org/apache/camel/component/chaos/ChaosExperiment.java
b/components/camel-chaos/src/main/java/org/apache/camel/component/chaos/ChaosExperiment.java
new file mode 100644
index 000000000000..fe2a5bd5c87b
--- /dev/null
+++
b/components/camel-chaos/src/main/java/org/apache/camel/component/chaos/ChaosExperiment.java
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.chaos;
+
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import java.util.function.Function;
+
+import org.apache.camel.component.chaos.ChaosRule.Expectation;
+import org.apache.camel.component.chaos.ChaosRule.Expectation.Outcome;
+import org.apache.camel.util.json.JsonArray;
+import org.apache.camel.util.json.JsonObject;
+
+/**
+ * An experiment of an exploration (CAMEL-26774): a hypothesis, the rules that
test it with their expectations, and what
+ * came of it. It begins with its rules added together and the counts per
route read, and ends with its rules removed:
+ * the difference of the counts, the faults of each rule and the states of the
expectations are kept in the journal,
+ * with a verdict.
+ * <p/>
+ * The verdict is {@code FAIL} when an expectation is unmet and {@code PASS}
when all are met. While it runs and an
+ * expectation is not decided it is {@code PENDING}, and an experiment that
ended before an expectation was decided is
+ * {@code INCONCLUSIVE}: the messages may still have come. The manager guards
it with its lock.
+ */
+final class ChaosExperiment {
+
+ static final String PASS = "PASS";
+ static final String FAIL = "FAIL";
+ static final String PENDING = "PENDING";
+ static final String INCONCLUSIVE = "INCONCLUSIVE";
+
+ /** How an experiment ended. */
+ enum End {
+ // endExperiment
+ ENDED("ended"),
+ // the exploration was finished with the experiment open
+ FINISHED("finished"),
+ // the time of the exploration was up
+ DEADLINE("deadline");
+
+ final String key;
+
+ End(String key) {
+ this.key = key;
+ }
+ }
+
+ final int number;
+ final String hypothesis;
+ final List<String> rules;
+ final long startedAt;
+ // the counts per route when it began: completed, failed, kept by the
consumer, handled, redeliveries
+ private final Map<String, long[]> before = new LinkedHashMap<>();
+ // the report was reset while it ran: the counts of the routes started
again from zero
+ private boolean countsReset;
+ // what came of it, once it ended
+ private JsonObject result;
+ private String resultText;
+
+ ChaosExperiment(int number, String hypothesis, List<String> rules,
Map<String, ChaosManager.RouteCounts> counts,
+ long startedAt) {
+ this.number = number;
+ this.hypothesis = hypothesis != null && !hypothesis.isBlank() ?
hypothesis.strip() : null;
+ this.rules = List.copyOf(rules);
+ this.startedAt = startedAt;
+ counts.forEach((route, c) -> before.put(route, read(c)));
+ }
+
+ boolean isOpen() {
+ return result == null;
+ }
+
+ /** The counts of the routes were reset (a scenario was loaded): they
count from zero now. */
+ void reportReset() {
+ before.clear();
+ countsReset = true;
+ }
+
+ /**
+ * Ends it: what came of it is read now, before its rules are removed (a
removed rule cancels its expectation if not
+ * decided).
+ */
+ void end(
+ Function<String, ChaosRule> rule, Map<String,
ChaosManager.RouteCounts> counts, long now, End by,
+ String note) {
+ JsonObject jo = read(rule, counts, now, true);
+ jo.put("endedAt", now);
+ jo.put("endedBy", by.key);
+ if (note != null && !note.isBlank()) {
+ jo.put("note", note.strip());
+ }
+ resultText = text(jo);
+ result = jo;
+ }
+
+ /** What came of it as it ended, or null while it runs. */
+ JsonObject result() {
+ return result;
+ }
+
+ /** What came of it: as it ended, or as it is now while it runs. */
+ JsonObject toJson(Function<String, ChaosRule> rule, Map<String,
ChaosManager.RouteCounts> counts, long now) {
+ return result != null ? result : read(rule, counts, now, false);
+ }
+
+ String text(Function<String, ChaosRule> rule, Map<String,
ChaosManager.RouteCounts> counts, long now) {
+ return result != null ? resultText : text(read(rule, counts, now,
false));
+ }
+
+ private JsonObject read(
+ Function<String, ChaosRule> rule, Map<String,
ChaosManager.RouteCounts> counts, long now, boolean ended) {
+ JsonObject jo = new JsonObject();
+ jo.put("number", number);
+ if (hypothesis != null) {
+ jo.put("hypothesis", hypothesis);
+ }
+ jo.put("open", !ended);
+ jo.put("startedAt", startedAt);
+ jo.put("elapsed", now - startedAt);
+ jo.put("rules", new JsonArray(rules));
+ boolean unmet = false;
+ boolean undecided = false;
+ JsonArray texts = new JsonArray();
+ JsonObject faults = new JsonObject();
+ JsonArray expectations = new JsonArray();
+ for (String id : rules) {
+ ChaosRule r = rule.apply(id);
+ if (r == null) {
+ continue;
+ }
+ texts.add(r.text);
+ long fired = 0;
+ for (ChaosRule.Hook hook : ChaosRule.Hook.values()) {
+ fired += r.firedAt(hook);
+ }
+ faults.put(id, fired);
+ Expectation e = r.expectation;
+ if (e != null) {
+ Expectation.Reading reading = e.read(now);
+ // an expectation the experiment did not wait for is
undecided, not cancelled
+ Outcome outcome = reading.outcome() == Outcome.CANCELLED ?
Outcome.PENDING : reading.outcome();
+ unmet |= outcome == Outcome.UNMET;
+ undecided |= outcome == Outcome.PENDING;
+ JsonObject x = new JsonObject();
+ x.put("rule", id);
+ x.put("expect", e.spec());
+ x.put("state", outcome.key);
+ x.put("actual", reading.value());
+ x.put("value", e.valueText(reading));
+ expectations.add(x);
+ }
+ }
+ jo.put("ruleTexts", texts);
+ jo.put("faults", faults);
+ jo.put("expectations", expectations);
+ jo.put("verdict", unmet ? FAIL : !undecided ? PASS : ended ?
INCONCLUSIVE : PENDING);
+ JsonArray routes = new JsonArray();
+ counts.forEach((route, c) -> {
+ long[] now0 = read(c);
+ long[] then = before.getOrDefault(route, new long[5]);
+ long[] delta = new long[5];
+ boolean any = false;
+ for (int i = 0; i < delta.length; i++) {
+ delta[i] = Math.max(0, now0[i] - then[i]);
+ any |= delta[i] > 0;
+ }
+ if (any) {
+ JsonObject r = new JsonObject();
+ r.put("route", route);
+ r.put("completed", delta[0]);
+ r.put("failed", delta[1]);
+ r.put("failedKeptByConsumer", delta[2]);
+ r.put("handled", delta[3]);
+ r.put("redeliveries", delta[4]);
+ routes.add(r);
+ }
+ });
+ jo.put("routes", routes);
+ if (countsReset) {
+ jo.put("countsReset", true);
+ }
+ return jo;
+ }
+
+ private static long[] read(ChaosManager.RouteCounts c) {
+ return new long[] {
+ c.completed.sum(), c.failed.sum(), c.failedKept.sum(),
c.handled.sum(), c.redeliveries.sum() };
+ }
+
+ /** One line for the experiment and one per expectation, as the report
prints them. */
+ private static String text(JsonObject jo) {
+ StringBuilder sb = new StringBuilder();
+ sb.append("\n ").append(jo.getInteger("number")).append('
').append(jo.getString("verdict"));
+ String hypothesis = jo.getString("hypothesis");
+ if (hypothesis != null) {
+ sb.append(" ").append(hypothesis);
+ }
+ sb.append(" (").append(String.join(", ", jo.<List<String>>
getCollection("rules"))).append(", ");
+ if (jo.getBooleanOrDefault("open", false)) {
+ sb.append("running for
").append(ChaosRuleParser.printTime(jo.getLong("elapsed")));
+ } else {
+ String by = jo.getString("endedBy");
+ sb.append(End.DEADLINE.key.equals(by)
+ ? "ended at the deadline" : End.FINISHED.key.equals(by) ?
"ended with the exploration" : "ended")
+ .append(" after
").append(ChaosRuleParser.printTime(jo.getLong("elapsed")));
+ }
+ sb.append(')');
+ for (Object o : jo.getCollection("expectations")) {
+ JsonObject x = (JsonObject) o;
+ sb.append("\n expect
").append(x.getString("expect")).append(": ").append(x.getString("state"))
+ .append(", ").append(x.getString("value"));
+ }
+ StringJoiner sj = new StringJoiner(", ");
+ for (Object o : jo.getCollection("routes")) {
+ JsonObject r = (JsonObject) o;
+ sj.add(r.getString("route") + " " + r.getLong("completed") + "
completed, " + r.getLong("failed") + " failed");
+ }
+ if (sj.length() > 0) {
+ sb.append("\n routes: ").append(sj);
+ }
+ String note = jo.getString("note");
+ if (note != null) {
+ sb.append("\n note: ").append(note);
+ }
+ return sb.toString();
+ }
+}
diff --git
a/components/camel-chaos/src/main/java/org/apache/camel/component/chaos/ChaosExploration.java
b/components/camel-chaos/src/main/java/org/apache/camel/component/chaos/ChaosExploration.java
index 89a9e3560e9f..511bc4c8d5f7 100644
---
a/components/camel-chaos/src/main/java/org/apache/camel/component/chaos/ChaosExploration.java
+++
b/components/camel-chaos/src/main/java/org/apache/camel/component/chaos/ChaosExploration.java
@@ -21,6 +21,7 @@ import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.StringJoiner;
+import java.util.function.Function;
import org.apache.camel.util.json.JsonArray;
import org.apache.camel.util.json.JsonObject;
@@ -32,7 +33,8 @@ import org.apache.camel.util.json.JsonObject;
* the deadline the rules are removed and the exploration is finished,
whatever the agent does.
* <p/>
* It has the baseline, the counts per route when it started, so the failures
that were there before are not blamed on
- * chaos, and the rules added while it ran. The manager guards it with its
lock.
+ * chaos, the rules added while it ran, and its experiments (CAMEL-26774): a
hypothesis and the rules that test it, one
+ * at a time. The manager guards it with its lock.
*/
final class ChaosExploration {
@@ -66,6 +68,8 @@ final class ChaosExploration {
private boolean baselineReset;
// the ids of the rules added while it ran (a changed rule keeps its id)
private final List<String> rules = new ArrayList<>();
+ // the experiments, the open one last
+ private final List<ChaosExperiment> experiments = new ArrayList<>();
private long finishedAt;
private Finish finishedBy;
private int removed;
@@ -97,6 +101,28 @@ final class ChaosExploration {
void reportReset() {
baseline.clear();
baselineReset = true;
+ ChaosExperiment open = openExperiment();
+ if (open != null) {
+ open.reportReset();
+ }
+ }
+
+ /** The experiment that is open, or null. */
+ ChaosExperiment openExperiment() {
+ ChaosExperiment last = experiments.isEmpty() ? null :
experiments.get(experiments.size() - 1);
+ return last != null && last.isOpen() ? last : null;
+ }
+
+ /** Begins an experiment with the rules just added for it (and taken in
the rules tried). */
+ ChaosExperiment beginExperiment(
+ String hypothesis, List<String> ruleIds, Map<String,
ChaosManager.RouteCounts> counts, long now) {
+ ChaosExperiment experiment = new ChaosExperiment(experiments.size() +
1, hypothesis, ruleIds, counts, now);
+ experiments.add(experiment);
+ return experiment;
+ }
+
+ int experimentCount() {
+ return experiments.size();
}
void finished(long now, Finish by, int removedRules) {
@@ -106,6 +132,14 @@ final class ChaosExploration {
}
JsonObject toJson(long now) {
+ return toJson(now, id -> null, Map.of());
+ }
+
+ /**
+ * As JSON, with the experiments: the rules of an open one are looked up
by their id, and the counts of the routes
+ * now give what it did so far.
+ */
+ JsonObject toJson(long now, Function<String, ChaosRule> rule, Map<String,
ChaosManager.RouteCounts> counts) {
JsonObject jo = new JsonObject();
jo.put("running", isRunning());
jo.put("startedAt", startedAt);
@@ -141,6 +175,11 @@ final class ChaosExploration {
jo.put("finishedBy", finishedBy.key);
jo.put("removed", removed);
}
+ JsonArray tried = new JsonArray();
+ for (ChaosExperiment experiment : experiments) {
+ tried.add(experiment.toJson(rule, counts, now));
+ }
+ jo.put("experiments", tried);
return jo;
}
@@ -149,8 +188,18 @@ final class ChaosExploration {
}
String text(long now) {
+ return text(now, id -> null, Map.of());
+ }
+
+ String text(long now, Function<String, ChaosRule> rule, Map<String,
ChaosManager.RouteCounts> counts) {
StringBuilder sb = new StringBuilder();
appendText(sb, now);
+ if (!experiments.isEmpty()) {
+ sb.append("\n experiments:");
+ for (ChaosExperiment experiment : experiments) {
+ sb.append(experiment.text(rule, counts, now));
+ }
+ }
return sb.toString();
}
diff --git
a/components/camel-chaos/src/main/java/org/apache/camel/component/chaos/ChaosManager.java
b/components/camel-chaos/src/main/java/org/apache/camel/component/chaos/ChaosManager.java
index 5c88b3888480..404e77c66d24 100644
---
a/components/camel-chaos/src/main/java/org/apache/camel/component/chaos/ChaosManager.java
+++
b/components/camel-chaos/src/main/java/org/apache/camel/component/chaos/ChaosManager.java
@@ -23,6 +23,7 @@ import java.util.Collection;
import java.util.Deque;
import java.util.HashMap;
import java.util.HashSet;
+import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.Set;
@@ -655,7 +656,8 @@ public final class ChaosManager {
lostAtDlq, recentFaults.latestFirst(),
ChaosTargets.reach(camelContext), tried);
if (explored != null) {
synchronized (lock) {
- report.exploration(explored.toJson(now), explored.text(now));
+ Map<String, RouteCounts> live = new TreeMap<>(routes);
+ report.exploration(explored.toJson(now, this::findRule, live),
explored.text(now, this::findRule, live));
}
}
return report;
@@ -715,8 +717,109 @@ public final class ChaosManager {
throw new IllegalStateException("No exploration is running");
}
finish(now, ChaosExploration.Finish.FINISHED);
- return exploration.toJson(now);
+ return exploration.toJson(now, this::findRule, new
TreeMap<>(routes));
+ }
+ }
+
+ /**
+ * Begins an experiment of the running exploration (CAMEL-26774): a
hypothesis and the rules that test it, added
+ * together (all or none), with at least one expectation ({@code expect=})
to decide it. The counts per route are
+ * read now, so its end says what changed while it ran. One experiment is
open at a time.
+ *
+ * @param hypothesis what the experiment tests, in a few
words, such as "a failed refund is read
+ * again"
+ * @param rules the rules in their one-line text form,
separated by {@code ;}
+ * @return the experiment, as in the report
+ * @throws IllegalArgumentException when a rule is not valid, or no rule
has an expectation
+ * @throws IllegalStateException when no exploration is running, or an
experiment is open
+ */
+ public Map<String, Object> beginExperiment(String hypothesis, String
rules) {
+ List<String> texts = ChaosRuleParser.splitRules(rules);
+ if (texts.isEmpty()) {
+ throw new IllegalArgumentException("The chaos rule is empty");
+ }
+ List<ChaosRule.Builder> definitions = new ArrayList<>();
+ for (String text : texts) {
+ definitions.add(parse(text));
+ }
+ if (definitions.stream().allMatch(d -> d.expectRoute == null)) {
+ throw new IllegalArgumentException(
+ "An experiment needs an expectation (expect=) on one of
its rules to decide its hypothesis, such as"
+ + " expect=orders:failed<=0
within=1m");
+ }
+ synchronized (lock) {
+ long now = TimeSource.currentTimeMillis();
+ checkDeadline(now);
+ if (exploration == null || !exploration.isRunning()) {
+ throw new IllegalStateException("No exploration is running: an
experiment is part of one");
+ }
+ ChaosExperiment open = exploration.openExperiment();
+ if (open != null) {
+ throw new IllegalStateException("Experiment " + open.number +
" is open: end it first");
+ }
+ Map<String, RouteCounts> counts = new TreeMap<>(routes);
+ List<String> ids = add(definitions);
+ ChaosExperiment experiment =
exploration.beginExperiment(hypothesis, ids, counts, now);
+ LOG.info("Chaos experiment {} began with rules {}{}",
experiment.number, ids,
+ experiment.hypothesis != null ? ": " +
experiment.hypothesis : "");
+ return experiment.toJson(this::findRule, counts, now);
+ }
+ }
+
+ /**
+ * Ends the open experiment: what came of it is read (the difference of
the counts per route, the faults of each
+ * rule, the states of the expectations and the verdict), its rules are
removed, and it stays in the journal.
+ *
+ * @param note what the agent makes of it (optional)
+ * @return the experiment, as in the report
+ * @throws IllegalStateException when no experiment is open
+ */
+ public Map<String, Object> endExperiment(String note) {
+ synchronized (lock) {
+ long now = TimeSource.currentTimeMillis();
+ // an experiment open at the deadline was ended with the
exploration
+ checkDeadline(now);
+ ChaosExperiment open = exploration != null ?
exploration.openExperiment() : null;
+ if (open == null) {
+ throw new IllegalStateException("No experiment is open");
+ }
+ endExperiment(open, now, ChaosExperiment.End.ENDED, note);
+ return open.result();
+ }
+ }
+
+ // reads what came of the experiment, and then removes its rules; with the
lock
+ private int endExperiment(ChaosExperiment experiment, long now,
ChaosExperiment.End by, String note) {
+ experiment.end(this::findRule, new TreeMap<>(routes), now, by, note);
+ int count = 0;
+ for (String id : experiment.rules) {
+ int index = indexOf(id);
+ if (index >= 0) {
+ ChaosRule removed = rules.remove(index);
+ removed.removed(State.REMOVED, now);
+ remember(removed);
+ count++;
+ }
+ }
+ update();
+ LOG.info("Chaos experiment {} ended: {}", experiment.number,
+ experiment.result().getString("verdict"));
+ return count;
+ }
+
+ // the rule with the id: the one in place, else the last one removed (a
changed rule keeps its id); with the lock
+ private ChaosRule findRule(String id) {
+ int index = indexOf(id);
+ if (index >= 0) {
+ return rules.get(index);
+ }
+ for (Iterator<ChaosRule> it = history.descendingIterator();
it.hasNext();) {
+ ChaosRule rule = it.next();
+ if (rule.id.equals(id)) {
+ return rule;
+ }
}
+ return null;
}
// finishes the running exploration whose time is up; with the lock
@@ -727,7 +830,15 @@ public final class ChaosManager {
}
private void finish(long now, ChaosExploration.Finish by) {
- int removedRules = clearRules();
+ ChaosExperiment open = exploration.openExperiment();
+ int removedRules = 0;
+ if (open != null) {
+ // read before the rules are removed
+ removedRules = endExperiment(open, now,
+ by == ChaosExploration.Finish.DEADLINE ?
ChaosExperiment.End.DEADLINE : ChaosExperiment.End.FINISHED,
+ null);
+ }
+ removedRules += clearRules();
exploration.finished(now, by, removedRules);
// no rule is left and the exploration is done: the counting stops
update();
diff --git
a/components/camel-chaos/src/test/java/org/apache/camel/component/chaos/ChaosExperimentTest.java
b/components/camel-chaos/src/test/java/org/apache/camel/component/chaos/ChaosExperimentTest.java
new file mode 100644
index 000000000000..7b58a415864f
--- /dev/null
+++
b/components/camel-chaos/src/test/java/org/apache/camel/component/chaos/ChaosExperimentTest.java
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.chaos;
+
+import java.util.List;
+import java.util.Map;
+
+import org.apache.camel.builder.RouteBuilder;
+import org.apache.camel.util.json.JsonObject;
+import org.junit.jupiter.api.Test;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+/**
+ * The experiments of an exploration (CAMEL-26774): a hypothesis and the rules
that test it, and what came of it in the
+ * journal: the counts per route while it ran, the faults of each rule, the
expectations and the verdict.
+ */
+class ChaosExperimentTest extends ChaosTestSupport {
+
+ private FakeClock clock;
+
+ @Override
+ protected void setupResources() throws Exception {
+ super.setupResources();
+ clock = FakeClock.install();
+ }
+
+ @Override
+ protected void cleanupResources() throws Exception {
+ clock.close();
+ super.cleanupResources();
+ }
+
+ @Override
+ protected RouteBuilder createRouteBuilder() {
+ return new RouteBuilder() {
+ @Override
+ public void configure() {
+ from("direct:orders").routeId("orders").to("mock:shipped");
+ }
+ };
+ }
+
+ private void send(int count) {
+ for (int i = 0; i < count; i++) {
+ // a failed exchange is counted by chaos, and not thrown here
+ template.send("direct:orders", e ->
e.getMessage().setBody("order"));
+ }
+ }
+
+ private JsonObject experiment(int number) {
+ JsonObject exploration = (JsonObject)
chaos().report().toJson().get("exploration");
+ return (JsonObject) list(exploration, "experiments").get(number - 1);
+ }
+
+ private static List<Object> list(JsonObject jo, String key) {
+ return jo.getCollection(key);
+ }
+
+ private static JsonObject first(JsonObject jo, String key) {
+ return (JsonObject) list(jo, key).get(0);
+ }
+
+ @Test
+ void aHypothesisThatFails() {
+ chaos().startExploration("10m", null);
+ // before the experiment: not counted in it
+ send(1);
+
+ Map<String, Object> begun = chaos().beginExperiment("an order never
fails",
+ "endpoint=mock:shipped error rate=every:2
expect=orders:failed<=0 within=1m");
+ assertThat(begun).containsEntry("number", 1).containsEntry("open",
true).containsEntry("verdict", "PENDING");
+ send(4);
+ clock.advance(5_000);
+
+ // the expectation is unmet as soon as an order fails, before the
experiment ends
+ assertThat(experiment(1).getString("verdict")).isEqualTo("FAIL");
+ Map<String, Object> ended = chaos().endExperiment("orders has no error
handler");
+
+ assertThat(ended).containsEntry("open",
false).containsEntry("verdict", "FAIL").containsEntry("endedBy", "ended")
+ .containsEntry("elapsed", 5_000L).containsEntry("note",
"orders has no error handler");
+ JsonObject result = experiment(1);
+ assertThat(list(result, "ruleTexts")).containsExactly(
+ "endpoint=mock:shipped error rate=every:2
expect=orders:failed<=0 within=1m");
+ assertThat(((JsonObject)
result.get("faults")).getLong("r1")).isEqualTo(2);
+ JsonObject expectation = first(result, "expectations");
+
assertThat(expectation.getString("expect")).isEqualTo("orders:failed<=0");
+ assertThat(expectation.getString("state")).isEqualTo("unmet");
+ assertThat(expectation.getLong("actual")).isEqualTo(2);
+ JsonObject orders = first(result, "routes");
+ assertThat(orders.getString("route")).isEqualTo("orders");
+ assertThat(orders.getLong("completed")).isEqualTo(2);
+ assertThat(orders.getLong("failed")).isEqualTo(2);
+ // its rules are removed, and it stays in the journal
+ assertThat(chaos().rules()).isEmpty();
+ assertThat(chaos().report().toText()).contains(
+ "experiments:", "1 FAIL an order never fails (r1, ended after
5s)",
+ "expect orders:failed<=0: unmet, 2 failed", "routes: orders 2
completed, 2 failed",
+ "note: orders has no error handler");
+ }
+
+ @Test
+ void aHypothesisThatHoldsAndOneNotDecided() {
+ chaos().startExploration("10m", null);
+ chaos().beginExperiment("orders keep up with a slow shipping",
+ "endpoint=mock:shipped delay duration=10ms expect=orders:2
within=1m");
+ send(1);
+ assertThat(experiment(1).getString("verdict")).isEqualTo("PENDING");
+ send(1);
+ assertThat(experiment(1).getString("verdict")).isEqualTo("PASS");
+ assertThat(chaos().endExperiment(null)).containsEntry("verdict",
"PASS").doesNotContainKey("note");
+
+ // ended before its expectation was decided: the messages may still
have come
+ chaos().beginExperiment(null, "endpoint=mock:shipped delay
duration=10ms expect=orders:5 within=1m");
+ send(1);
+ Map<String, Object> ended = chaos().endExperiment(null);
+ assertThat(ended).containsEntry("number", 2).containsEntry("verdict",
"INCONCLUSIVE");
+ assertThat(first(experiment(2),
"expectations").getString("state")).isEqualTo("pending");
+ assertThat(list((JsonObject)
chaos().report().toJson().get("exploration"), "rules"))
+ .containsExactly("r1", "r2");
+ }
+
+ @Test
+ void oneAtATimeWithAnExpectationAndInAnExploration() {
+ String rule = "endpoint=mock:shipped error expect=orders:failed<=0
within=1m";
+ assertThatThrownBy(() -> chaos().beginExperiment("h",
rule)).isInstanceOf(IllegalStateException.class)
+ .hasMessageContaining("No exploration is running");
+
+ chaos().startExploration("10m", null);
+ assertThatThrownBy(() -> chaos().beginExperiment("h",
"endpoint=mock:shipped error"))
+
.isInstanceOf(IllegalArgumentException.class).hasMessageContaining("needs an
expectation (expect=)");
+ assertThatThrownBy(() -> chaos().beginExperiment("h", rule + ";
endpoint=mock:shipped nonsense"))
+ .isInstanceOf(IllegalArgumentException.class);
+ assertThat(chaos().rules()).isEmpty();
+ assertThatThrownBy(() ->
chaos().endExperiment(null)).isInstanceOf(IllegalStateException.class)
+ .hasMessage("No experiment is open");
+
+ // a pair of faults: two rules, one of them with the expectation
+ Map<String, Object> begun = chaos().beginExperiment("h", rule + ";
endpoint=mock:shipped delay duration=10ms");
+ assertThat(list(new JsonObject(begun), "rules")).containsExactly("r1",
"r2");
+ assertThatThrownBy(() -> chaos().beginExperiment("h",
rule)).isInstanceOf(IllegalStateException.class)
+ .hasMessage("Experiment 1 is open: end it first");
+ // a rule added by hand is not part of the experiment, and stays when
it ends
+ String other = chaos().addRule("endpoint=mock:shipped delay
duration=1ms");
+ chaos().endExperiment(null);
+ assertThat(chaos().rules()).extracting(r ->
r.id).containsExactly(other);
+ }
+
+ @Test
+ void anExperimentOpenAtTheDeadlineEndsWithTheExploration() {
+ chaos().startExploration("1m", null);
+ chaos().beginExperiment("h", "endpoint=mock:shipped error
expect=orders:failed<=0 within=5m");
+ send(1);
+ clock.advance(60_000);
+
+ JsonObject experiment = experiment(1);
+ assertThat(experiment.getString("endedBy")).isEqualTo("deadline");
+ assertThat(experiment.getString("verdict")).isEqualTo("FAIL");
+ JsonObject exploration = (JsonObject)
chaos().report().toJson().get("exploration");
+ assertThat(exploration.getLong("removed")).isEqualTo(1);
+ assertThat(chaos().report().toText()).contains("1 FAIL h (r1, ended
at the deadline after 1m0s)");
+
+ // finished by hand, with an experiment open
+ chaos().startExploration("1m", null);
+ chaos().beginExperiment("h", "endpoint=mock:shipped delay duration=1ms
expect=orders:1 within=5m");
+ chaos().finishExploration();
+ assertThat(experiment(1).getString("endedBy")).isEqualTo("finished");
+
assertThat(experiment(1).getString("verdict")).isEqualTo("INCONCLUSIVE");
+ assertThat(chaos().rules()).isEmpty();
+ }
+}
diff --git
a/dsl/camel-cli-connector/src/main/java/org/apache/camel/cli/connector/ChaosAction.java
b/dsl/camel-cli-connector/src/main/java/org/apache/camel/cli/connector/ChaosAction.java
index 554ac4415f43..e313d421dca3 100644
---
a/dsl/camel-cli-connector/src/main/java/org/apache/camel/cli/connector/ChaosAction.java
+++
b/dsl/camel-cli-connector/src/main/java/org/apache/camel/cli/connector/ChaosAction.java
@@ -151,9 +151,26 @@ final class ChaosAction {
@SuppressWarnings("unchecked")
Map<String, Object> finished = (Map<String, Object>)
call(type, chaos, "finishExploration");
answer.putAll(finished);
+ } else if ("experiment-begin".equals(command)) {
+ if (isBlank(rule)) {
+ output.error("The rules of the experiment are required");
+ return;
+ }
+ // an experiment of the exploration (CAMEL-26774): its rules
are added together, with an expectation
+ @SuppressWarnings("unchecked")
+ Map<String, Object> begun = (Map<String, Object>) call(type,
chaos, "beginExperiment",
+ root.getString("hypothesis"), rule);
+ answer.putAll(begun);
+ } else if ("experiment-end".equals(command)) {
+ // what came of it, and its rules are removed
+ @SuppressWarnings("unchecked")
+ Map<String, Object> ended = (Map<String, Object>) call(type,
chaos, "endExperiment",
+ root.getString("note"));
+ answer.putAll(ended);
} else {
output.error("Unknown chaos command: " + command
- + " (add, check, change, remove, clear, load,
suggest, explore-start or explore-finish)");
+ + " (add, check, change, remove, clear, load,
suggest, explore-start, explore-finish,"
+ + " experiment-begin or experiment-end)");
return;
}
} catch (IllegalArgumentException | IllegalStateException e) {
diff --git
a/dsl/camel-cli-connector/src/test/java/org/apache/camel/cli/connector/LocalCliConnectorChaosActionTest.java
b/dsl/camel-cli-connector/src/test/java/org/apache/camel/cli/connector/LocalCliConnectorChaosActionTest.java
index 73b090e315a0..eae9ee24a76d 100644
---
a/dsl/camel-cli-connector/src/test/java/org/apache/camel/cli/connector/LocalCliConnectorChaosActionTest.java
+++
b/dsl/camel-cli-connector/src/test/java/org/apache/camel/cli/connector/LocalCliConnectorChaosActionTest.java
@@ -26,6 +26,7 @@ import java.util.Map;
import java.util.concurrent.TimeUnit;
import org.apache.camel.CamelContext;
+import org.apache.camel.Exchange;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.component.chaos.ChaosManager;
import org.apache.camel.impl.DefaultCamelContext;
@@ -248,6 +249,38 @@ class LocalCliConnectorChaosActionTest {
assertThat(chaos("explore-finish", null, null).error()).isEqualTo("No
exploration is running");
}
+ @Test
+ void anExperimentBeginsAndEnds() throws Exception {
+ start(null, Map.of("camel.chaos.enabled", "true"));
+ assertThat(chaos("explore-start", null, null).error()).isNull();
+ JsonObject action = new JsonObject();
+ action.put("action", "chaos");
+ action.put("command", "experiment-begin");
+ action.put("hypothesis", "start never fails");
+ action.put("rule", "endpoint=mock:target error expect=start:failed<=0
within=1m");
+
+ CliActionOutcome begun = CliActionOutcome.dispatch(connector,
action).get(10, TimeUnit.SECONDS);
+ assertThat(begun.error()).isNull();
+ assertThat(begun.result().getInteger("number")).isEqualTo(1);
+ assertThat(begun.result().getString("verdict")).isEqualTo("PENDING");
+ // one at a time: the reason goes to the tool
+ assertThat(CliActionOutcome.dispatch(connector, action).get(10,
TimeUnit.SECONDS).error())
+ .isEqualTo("Experiment 1 is open: end it first");
+
+ Exchange failed =
context.createProducerTemplate().send("direct:start", e ->
e.getMessage().setBody("x"));
+ assertThat(failed.getException()).isNotNull();
+ JsonObject end = new JsonObject();
+ end.put("action", "chaos");
+ end.put("command", "experiment-end");
+ end.put("note", "no error handler");
+ CliActionOutcome ended = CliActionOutcome.dispatch(connector,
end).get(10, TimeUnit.SECONDS);
+ assertThat(ended.error()).isNull();
+ assertThat(ended.result().getString("verdict")).isEqualTo("FAIL");
+ assertThat(ended.result().getString("note")).isEqualTo("no error
handler");
+ assertThat(chaos("experiment-end", null, null).error()).isEqualTo("No
experiment is open");
+ assertThat(chaos("experiment-begin", null,
null).error()).isEqualTo("The rules of the experiment are required");
+ }
+
@Test
void rulesAreAddedTogetherWithWarningsForTheOnesThatMatchNothing() throws
Exception {
start(null, Map.of("camel.chaos.enabled", "true"));
diff --git
a/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/process/ListChaos.java
b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/process/ListChaos.java
index 34c710feb859..f81650fb1806 100644
---
a/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/process/ListChaos.java
+++
b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/process/ListChaos.java
@@ -308,8 +308,8 @@ public class ListChaos extends ProcessWatchCommand {
}
/**
- * The exploration running now or the last one (CAMEL-26773): its time,
the rules tried, and the baseline before its
- * first rule.
+ * The exploration running now or the last one (CAMEL-26773): its time,
the rules tried, the baseline before its
+ * first rule, and its experiments (CAMEL-26774).
*/
private void printExploration(JsonObject report) {
JsonObject exploration = report.getMap("exploration");
@@ -338,16 +338,65 @@ public class ListChaos extends ProcessWatchCommand {
printer().println(" baseline: taken when the first rule is
added");
return;
}
+ printer().println(" baseline (before the first rule): " +
routeCounts(exploration.getCollection("baseline")));
+ List<Object> experiments = exploration.getCollection("experiments");
+ if (experiments != null && !experiments.isEmpty()) {
+ printer().println(" experiments:");
+ for (Object o : experiments) {
+ printExperiment((JsonObject) o);
+ }
+ }
+ }
+
+ /** An experiment: its verdict and hypothesis, its rules, its
expectations, and what the routes did meanwhile. */
+ private void printExperiment(JsonObject experiment) {
+ StringBuilder sb = new StringBuilder("
").append(experiment.getIntegerOrDefault("number", 0)).append(' ')
+ .append(experiment.getString("verdict"));
+ String hypothesis = experiment.getString("hypothesis");
+ if (hypothesis != null) {
+ sb.append(" ").append(hypothesis);
+ }
+ List<Object> rules = experiment.getCollection("rules");
+ sb.append(" (").append(rules == null ? "" :
rules.stream().map(String::valueOf).collect(Collectors.joining(", ")))
+ .append(", ");
+ String elapsed =
TimeUtils.printDuration(experiment.getLongOrDefault("elapsed", 0));
+ if (experiment.getBooleanOrDefault("open", false)) {
+ sb.append("running for ").append(elapsed);
+ } else {
+ String by = experiment.getString("endedBy");
+ sb.append("deadline".equals(by)
+ ? "ended at the deadline" : "finished".equals(by) ? "ended
with the exploration" : "ended")
+ .append(" after ").append(elapsed);
+ }
+ printer().println(sb.append(')').toString());
+ List<Object> expectations = experiment.getCollection("expectations");
+ if (expectations != null) {
+ for (Object o : expectations) {
+ JsonObject x = (JsonObject) o;
+ printer().println(" expect " + x.getString("expect") +
": " + x.getString("state") + ", "
+ + x.getString("value"));
+ }
+ }
+ List<Object> routes = experiment.getCollection("routes");
+ if (routes != null && !routes.isEmpty()) {
+ printer().println(" routes: " + routeCounts(routes));
+ }
+ String note = experiment.getString("note");
+ if (note != null) {
+ printer().println(" note: " + note);
+ }
+ }
+
+ private static String routeCounts(List<Object> routes) {
StringJoiner sj = new StringJoiner(", ");
- List<Object> baseline = exploration.getCollection("baseline");
- if (baseline != null) {
- for (Object o : baseline) {
+ if (routes != null) {
+ for (Object o : routes) {
JsonObject route = (JsonObject) o;
sj.add(route.getString("route") + " " +
route.getLongOrDefault("completed", 0) + " completed, "
+ route.getLongOrDefault("failed", 0) + " failed");
}
}
- printer().println(" baseline (before the first rule): " +
(sj.length() == 0 ? "no exchanges" : sj));
+ return sj.length() == 0 ? "no exchanges" : sj.toString();
}
/**
diff --git
a/dsl/camel-jbang/camel-jbang-core/src/test/java/org/apache/camel/dsl/jbang/core/commands/process/ListChaosTest.java
b/dsl/camel-jbang/camel-jbang-core/src/test/java/org/apache/camel/dsl/jbang/core/commands/process/ListChaosTest.java
index 56107e222941..6763a6dfd9f9 100644
---
a/dsl/camel-jbang/camel-jbang-core/src/test/java/org/apache/camel/dsl/jbang/core/commands/process/ListChaosTest.java
+++
b/dsl/camel-jbang/camel-jbang-core/src/test/java/org/apache/camel/dsl/jbang/core/commands/process/ListChaosTest.java
@@ -201,6 +201,26 @@ class ListChaosTest extends ProcessCommandTestSupport {
orders.put("completed", 30L);
orders.put("failed", 0L);
exploration.put("baseline", new JsonArray(List.of(orders)));
+ JsonObject experiment = new JsonObject();
+ experiment.put("number", 1);
+ experiment.put("hypothesis", "an order never fails");
+ experiment.put("open", false);
+ experiment.put("endedBy", "ended");
+ experiment.put("elapsed", 64_000L);
+ experiment.put("rules", new JsonArray(List.of("r1", "r2")));
+ experiment.put("verdict", "FAIL");
+ JsonObject expectation = new JsonObject();
+ expectation.put("expect", "orders:failed<=0");
+ expectation.put("state", "unmet");
+ expectation.put("value", "9 failed");
+ experiment.put("expectations", new JsonArray(List.of(expectation)));
+ JsonObject during = new JsonObject();
+ during.put("route", "orders");
+ during.put("completed", 41L);
+ during.put("failed", 9L);
+ experiment.put("routes", new JsonArray(List.of(during)));
+ experiment.put("note", "orders has no error handler");
+ exploration.put("experiments", new JsonArray(List.of(experiment)));
((JsonObject) chaos.get("report")).put("exploration", exploration);
writeStatus(chaos);
@@ -208,7 +228,10 @@ class ListChaosTest extends ProcessCommandTestSupport {
String out = printer.getStdout();
assertThat(out).contains("exploration: running, 6m12s left of 10m0s,
focus: orders",
- "rules tried: r1, r2", "baseline (before the first rule):
orders 30 completed, 0 failed");
+ "rules tried: r1, r2", "baseline (before the first rule):
orders 30 completed, 0 failed",
+ "experiments:", "1 FAIL an order never fails (r1, r2, ended
after 1m4s)",
+ "expect orders:failed<=0: unmet, 9 failed", "routes: orders 41
completed, 9 failed",
+ "note: orders has no error handler");
}
@Test
