This is an automated email from the ASF dual-hosted git repository. davsclaus pushed a commit to branch fix/CAMEL-25481 in repository https://gitbox.apache.org/repos/asf/camel.git
commit c6bb588e69352ee6a8032b8a58658001c1025ce7 Author: Claus Ibsen <[email protected]> AuthorDate: Fri Oct 9 10:29:34 2026 +0200 CAMEL-25481: camel-dataweave - convert XML namespaces, attributes and the DataSonnet XML model The DataWeave to DataSonnet conversion now follows the DataWeave XML semantics. XML input: a selector matches an element by its local name whatever its namespace prefix (such as payload.Envelope.Body of a SOAP message), .ns#name only in the namespace of an ns header directive, .@ gives all the attributes and ..* all the descendants, an empty element is null, and the object functions see the child elements in document order. An output other than XML writes the elements without the attributes, namespace declarations and positions of the DataSonnet XML model (dw.output). XML output: attributes (key @(name: value)), namespace-qualified keys (ns0#key), null as an empty element, arrays, object spreads and keys that repeat as elements that repeat (in the order of the script), elements of the input with their attributes and namespaces, and the writeDeclaration and skipNullOn writer properties. DataSonnet does not keep the value of a function argument (each use evaluates it again), so nested calls of the camel-dataweave.libsonnet functions took exponential time in the depth of the nesting. The functions now evaluate their arguments once. The corpus has 29 new XML entries, verified with the DataWeave CLI, and compares XML outputs without formatting. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Signed-off-by: Claus Ibsen <[email protected]> --- .../camel/catalog/docs/datasonnet-language.adoc | 31 +- .../src/main/docs/datasonnet-language.adoc | 31 +- .../src/main/resources/camel-dataweave.libsonnet | 525 ++++++++++++++------- .../language/datasonnet/DataWeaveCorpusTest.java | 58 ++- .../src/test/resources/dataweave-corpus/corpus.txt | 205 +++++++- .../camel/component/dataweave/DataWeaveAst.java | 37 +- .../component/dataweave/DataWeaveConverter.java | 241 ++++++++-- .../camel/component/dataweave/DataWeaveLexer.java | 38 +- .../camel/component/dataweave/DataWeaveParser.java | 94 ++-- .../dataweave/DataWeaveConverterTest.java | 109 ++++- .../component/dataweave/DataWeaveParserTest.java | 34 +- .../ROOT/pages/camel-4x-upgrade-guide-4_23.adoc | 6 +- 12 files changed, 1123 insertions(+), 286 deletions(-) diff --git a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/datasonnet-language.adoc b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/datasonnet-language.adoc index af3222598ad6..9a4894c4de70 100644 --- a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/datasonnet-language.adoc +++ b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/datasonnet-language.adoc @@ -542,10 +542,13 @@ The conversion either produces a script that gives the same result as the DataWe === Supported Constructs - *Header*: `%dw`, `output` (JSON, XML, CSV, Java and text; for JSON the writer properties `skipNullOn`, `indent` and - `encoding`), `input`, `import` of the `dw::core` modules, `var` and `fun` declarations (also with default parameters, - recursive and calling each other), and `type` declarations (ignored). + `encoding`, for XML `writeDeclaration`, `skipNullOn` (`elements`, `attributes` or `everywhere`), `indent`, + `encoding` and `inlineCloseOn`), `input`, `ns` namespace declarations, `import` of the `dw::core` modules, `var` and + `fun` declarations (also with default parameters, recursive and calling each other), and `type` declarations + (ignored). - *Selectors*: `.field`, `."quoted field"`, `[index]` (negative from the end), `[from to to]`, `["key"]`, `.*field`, - `..field`, `.@attribute`, `field?` and `[?(condition)]`. + `..field`, `..*field`, `.@attribute`, `.@` (all the attributes), `.ns#field` (in the namespace declared as `ns`), + `field?` and `[?(condition)]`. - *Mule variables and attributes*: `payload` is the message body, `vars.name` (or `vars["name"]`) an exchange variable, and `attributes.headers`, `attributes.queryParams` and `attributes.uriParams` message headers. `attributes.method`, `requestPath`, `requestUri`, `queryString`, `statusCode` and `reasonPhrase` are the corresponding Camel HTTP headers. @@ -564,6 +567,17 @@ The conversion either produces a script that gives the same result as the DataWe `partition`, `firstWith` and `mergeWith`. - *Objects*: dynamic keys `(key): value`, conditional fields `(key: value) if condition`, and object spread `{ (expression) }` of an object or an array of objects. +- *XML input*: as in DataWeave, a selector matches an element by its local name whatever its namespace prefix (such as + `payload.Envelope.Body` of a SOAP message), and `.ns#field` only in the given namespace. The value of an element + without child elements is its text (`null` when empty), `.field` of an element that repeats is the first one and + `.*field` all of them, and the attributes are read with `.@name` and `.@` (also of an element in a variable or + given to a lambda). The object functions (`keysOf`, `sizeOf`, `mapObject`, `pluck`, `entriesOf`, ...) see the child + elements in document order, without the attributes. In an output other than XML, an element is written as its text, + or as an object of its child elements, without its attributes and namespace prefixes. +- *XML output*: attributes `key @(name: value)`, namespace-qualified keys `ns0#key` (declared on the root element), + `null` as an empty element, and arrays, object spreads and keys that repeat as elements that repeat, in the order of + the script. An element of the input is written with its child elements, attributes and namespaces; under another key + it gets the attributes of that key, as in DataWeave. - *Strings*: interpolation (`"Hello $(expression)"`, `"Hello $name"` and `$` of a lambda), and regular expressions `/.../` in `replace` (also with a function of the matched groups: `replace /(\d+)/ with ($[1] ++ "!")`), `contains`, `splitBy`, `matches`, `scan` and `find`. @@ -577,10 +591,13 @@ The conversion either produces a script that gives the same result as the DataWe The expected output of the DataWeave scripts in the tests of the conversion is verified with the DataWeave CLI. These constructs are not converted (the conversion fails): the `update` operator, `try` (`dw::Runtime`), overloaded -functions, custom modules, `as` with a custom type, XML namespaces, XML attributes in the output, and writer properties -other than the ones listed above. The XML support is limited: an element with text is read as its text (as in -DataWeave), but the attributes of such an element are only kept when they are selected directly (`payload.order.@id`) -or in a lambda (`payload.orders.*order map $.@id`). +functions, custom modules, `as` with a custom type, the `..@` selector, a namespace prefix that is not declared, and +writer properties other than the ones listed above. + +A DataSonnet object can not repeat a key, so where DataWeave repeats a key in a JSON or Java output, the converted +script gives an array of the values: for elements that repeat under the same name (in an XML input, such as +`payload.order` with two `item` elements), and for the text of mixed content (`__text`). An XML output does repeat the +elements. == Dependencies diff --git a/components/camel-datasonnet/src/main/docs/datasonnet-language.adoc b/components/camel-datasonnet/src/main/docs/datasonnet-language.adoc index af3222598ad6..9a4894c4de70 100644 --- a/components/camel-datasonnet/src/main/docs/datasonnet-language.adoc +++ b/components/camel-datasonnet/src/main/docs/datasonnet-language.adoc @@ -542,10 +542,13 @@ The conversion either produces a script that gives the same result as the DataWe === Supported Constructs - *Header*: `%dw`, `output` (JSON, XML, CSV, Java and text; for JSON the writer properties `skipNullOn`, `indent` and - `encoding`), `input`, `import` of the `dw::core` modules, `var` and `fun` declarations (also with default parameters, - recursive and calling each other), and `type` declarations (ignored). + `encoding`, for XML `writeDeclaration`, `skipNullOn` (`elements`, `attributes` or `everywhere`), `indent`, + `encoding` and `inlineCloseOn`), `input`, `ns` namespace declarations, `import` of the `dw::core` modules, `var` and + `fun` declarations (also with default parameters, recursive and calling each other), and `type` declarations + (ignored). - *Selectors*: `.field`, `."quoted field"`, `[index]` (negative from the end), `[from to to]`, `["key"]`, `.*field`, - `..field`, `.@attribute`, `field?` and `[?(condition)]`. + `..field`, `..*field`, `.@attribute`, `.@` (all the attributes), `.ns#field` (in the namespace declared as `ns`), + `field?` and `[?(condition)]`. - *Mule variables and attributes*: `payload` is the message body, `vars.name` (or `vars["name"]`) an exchange variable, and `attributes.headers`, `attributes.queryParams` and `attributes.uriParams` message headers. `attributes.method`, `requestPath`, `requestUri`, `queryString`, `statusCode` and `reasonPhrase` are the corresponding Camel HTTP headers. @@ -564,6 +567,17 @@ The conversion either produces a script that gives the same result as the DataWe `partition`, `firstWith` and `mergeWith`. - *Objects*: dynamic keys `(key): value`, conditional fields `(key: value) if condition`, and object spread `{ (expression) }` of an object or an array of objects. +- *XML input*: as in DataWeave, a selector matches an element by its local name whatever its namespace prefix (such as + `payload.Envelope.Body` of a SOAP message), and `.ns#field` only in the given namespace. The value of an element + without child elements is its text (`null` when empty), `.field` of an element that repeats is the first one and + `.*field` all of them, and the attributes are read with `.@name` and `.@` (also of an element in a variable or + given to a lambda). The object functions (`keysOf`, `sizeOf`, `mapObject`, `pluck`, `entriesOf`, ...) see the child + elements in document order, without the attributes. In an output other than XML, an element is written as its text, + or as an object of its child elements, without its attributes and namespace prefixes. +- *XML output*: attributes `key @(name: value)`, namespace-qualified keys `ns0#key` (declared on the root element), + `null` as an empty element, and arrays, object spreads and keys that repeat as elements that repeat, in the order of + the script. An element of the input is written with its child elements, attributes and namespaces; under another key + it gets the attributes of that key, as in DataWeave. - *Strings*: interpolation (`"Hello $(expression)"`, `"Hello $name"` and `$` of a lambda), and regular expressions `/.../` in `replace` (also with a function of the matched groups: `replace /(\d+)/ with ($[1] ++ "!")`), `contains`, `splitBy`, `matches`, `scan` and `find`. @@ -577,10 +591,13 @@ The conversion either produces a script that gives the same result as the DataWe The expected output of the DataWeave scripts in the tests of the conversion is verified with the DataWeave CLI. These constructs are not converted (the conversion fails): the `update` operator, `try` (`dw::Runtime`), overloaded -functions, custom modules, `as` with a custom type, XML namespaces, XML attributes in the output, and writer properties -other than the ones listed above. The XML support is limited: an element with text is read as its text (as in -DataWeave), but the attributes of such an element are only kept when they are selected directly (`payload.order.@id`) -or in a lambda (`payload.orders.*order map $.@id`). +functions, custom modules, `as` with a custom type, the `..@` selector, a namespace prefix that is not declared, and +writer properties other than the ones listed above. + +A DataSonnet object can not repeat a key, so where DataWeave repeats a key in a JSON or Java output, the converted +script gives an array of the values: for elements that repeat under the same name (in an XML input, such as +`payload.order` with two `item` elements), and for the text of mixed content (`__text`). An XML output does repeat the +elements. == Dependencies diff --git a/components/camel-datasonnet/src/main/resources/camel-dataweave.libsonnet b/components/camel-datasonnet/src/main/resources/camel-dataweave.libsonnet index 88c068bb7953..95394c7c1ec3 100644 --- a/components/camel-datasonnet/src/main/resources/camel-dataweave.libsonnet +++ b/components/camel-datasonnet/src/main/resources/camel-dataweave.libsonnet @@ -20,55 +20,159 @@ // In DataWeave a missing field, an index out of range and a selector on null give null, a field selector on an // array selects the field of every element, and most functions accept null. In Jsonnet these are errors. // Functions that are given a function call it with all the arguments DataWeave passes (such as item and index). +// +// DataSonnet evaluates the argument of a function each time the function uses it, and an element of an array each +// time it is used: only the value of an object field is kept. A function that uses an argument more than once starts +// with strict, else nested calls are evaluated over and over (exponentially in the depth of the nesting). { local dw = self, local indexes(a) = std.range(0, std.length(a) - 1), + // f(x), with x evaluated once (the initial value of std.foldl is evaluated once, and kept) + local strict(x, f) = std.foldl(function(v, _) f(v), [0], x), - // -- Selectors - - // An XML element with only text (and attributes) is read as an object with the text in '$' - local isText(x) = std.isObject(x) && std.objectHas(x, '$') - && std.all([k == '$' || k == '~' || std.startsWith(k, '@') for k in std.objectFields(x)]), - local text(x) = if isText(x) then x['$'] else x, + // -- XML + // + // The DataSonnet XML reader gives an element as an object with its position in '~', its attributes in '@name' keys, + // its namespace declarations in '@xmlns', its text in '$' (mixed content in '$1', '$2', ..., CDATA also in '#1', + // ...), and its child elements by name (with the prefix of their namespace, such as 'soapenv:Body'); an element + // repeated under the same name is an array. In DataWeave the value of an element without child elements is its text + // (null when empty), a selector matches the local name of an element whatever its namespace, and the attributes are + // metadata that only the .@ selector and an XML output see. - // An XML element is read as an object with its position in '~' local isElement(x) = std.isObject(x) && std.objectHas(x, '~'), + // the reader gives an array only for a repeated element, so the first element tells + local isRepeated(v) = std.isArray(v) && std.length(v) > 0 && isElement(v[0]), + // an element, or a document (the object with the root element) + local isXml(x) = + isElement(x) || std.isObject(x) && (local fs = std.objectFields(x); + std.length(fs) == 1 && (isElement(x[fs[0]]) || isRepeated(x[fs[0]]))), + local isMeta(k) = k == '~' || std.startsWith(k, '@'), + local isTextKey(k) = std.startsWith(k, '$') || std.startsWith(k, '#'), + local isText(x) = isElement(x) && std.all([isMeta(k) || isTextKey(k) for k in std.objectFields(x)]), + local textOf(x) = strict(std.join('', [x[k] for k in std.objectFields(x) if std.startsWith(k, '$')]), + function(t) if t == '' then null else t), + // (an element with child elements is marked as selected: written as XML under another key, its attributes are left + // out, as they are those of its key in the input) + local text(x0) = strict(x0, function(x) + if isText(x) then textOf(x) else if isElement(x) then x + { '~selected':: true } else x), + local localName(k) = local i = std.findSubstr(':', k); if std.length(i) == 0 then k else k[i[std.length(i) - 1] + 1:], + // the key of the child k of an XML element: k, or else a child with the local name k + local childKey(x, k) = + if std.objectHas(x, k) then k + else if !isXml(x) then null + else strict([f for f in std.objectFields(x) if !isMeta(f) && !isTextKey(f) && localName(f) == k], + function(m) if std.length(m) == 0 then null else m[0]), + local first(v) = if isRepeated(v) then v[0] else v, + // a child element with the namespace declarations in scope of its parent, so its namespace can be resolved (and it + // can be written as XML) without its ancestors + local inScope(x, v0) = strict(v0, function(v) + if !std.isObject(x) || !std.objectHas(x, '@xmlns') then v + else if isElement(v) then v { '@xmlns': x['@xmlns'] + (if std.objectHas(v, '@xmlns') then v['@xmlns'] else {}) } + else if isRepeated(v) then [inScope(x, e) for e in v] + else v), + // the position of a child in its element: the order of child elements, or the number of a text key ($2) + local position(k, v) = + if isElement(v) then v['~'] + else if std.length(k) > 1 && isTextKey(k) then std.parseInt(k[1:]) + else 0, + local attrsOf(x0) = strict(x0, function(x) + if !isElement(x) then {} + else { [k[1:]]: x[k] for k in std.objectFields(x) if std.startsWith(k, '@') && k != '@xmlns' }), + + // The value of an XML element as DataWeave sees it: its text, or its child elements + text(x):: text(x), + // The entries of an object as DataWeave sees them: for an XML element without its attributes and order, with the + // local names, an entry for each element repeated under a name, and the text of the elements without children + local entry(k, e) = { k: localName(k), v: text(e), a: attrsOf(e), p: position(k, e) }, + local entries(o0) = strict(o0, function(o) + if !isXml(o) then [{ k: k, v: o[k], a: {} } for k in std.objectFields(o)] + else + local keys = [k for k in std.objectFields(o) if !isMeta(k) && !std.startsWith(k, '#')]; + local es = std.flattenArrays([ + if isTextKey(k) then [{ k: '__text', v: o[k], a: {}, p: position(k, o[k]) }] + else strict(inScope(o, o[k]), function(v) if isRepeated(v) then [entry(k, e) for e in v] else [entry(k, v)]) + for k in keys + ]); + // in document order, which the keys are unless an element repeats with others in between, or text is mixed in + if std.length(keys) > 1 && std.any([std.isArray(o[k]) || isTextKey(k) for k in keys]) + then std.sort(es, function(e) e.p) + else es), + // An object of entries; the values of a key that repeats become an array (an object can not repeat a key) + local fromEntries(es0) = strict(es0, function(es) + local keys = std.foldl(function(acc, e) if std.member(acc, e.k) then acc else acc + [e.k], es, []); + { [k]: strict([e.v for e in es if e.k == k], function(vs) if std.length(vs) == 1 then vs[0] else vs) for k in keys }), + entries(o):: entries(o), + // The DataWeave value of something read from XML, for writing it in another format than XML: elements as their text + // or as objects of their child elements, without attributes, order and namespace prefixes + fromXml(x0):: strict(x0, function(x) + if isText(x) then textOf(x) + else if std.isArray(x) then std.map(dw.fromXml, x) + else if isXml(x) then fromEntries([{ k: e.k, v: dw.fromXml(e.v) } for e in entries(x)]) + else if std.isObject(x) then { [k]: dw.fromXml(x[k]) for k in std.objectFields(x) } + else x), + // The result of a script written in another format than XML: when the input is XML, as DataWeave would write it + output(result, input):: if isXml(input) then dw.fromXml(result) else result, + + // -- Selectors // x.k: null when missing; on an array the field of every element that has it; of an XML element repeated under - // the same name (read as an array) the first - selRaw(x, k):: - if std.isObject(x) then ( - if !std.objectHas(x, k) then null - else - local v = x[k]; - if std.isArray(v) && std.length(v) > 0 && std.all(std.map(isElement, v)) then v[0] else v - ) - else if std.isArray(x) then [e[k] for e in x if std.isObject(e) && std.objectHas(e, k)] - else null, - sel(x, k):: - local v = dw.selRaw(x, k); - if std.isArray(x) then std.map(text, v) else text(v), + // the same name the first; selRaw keeps an XML element as an object (for its attributes) + selRaw(x0, k):: strict(x0, function(x) + if std.isObject(x) then strict(childKey(x, k), function(key) if key == null then null else inScope(x, first(x[key]))) + else if std.isArray(x) then std.flattenArrays([ + strict(e, function(e) + if !std.isObject(e) then [] + else strict(childKey(e, k), function(key) + if key == null then [] else if isRepeated(e[key]) then inScope(e, e[key]) else [inScope(e, e[key])])) + for e in x + ]) + else null), + sel(x0, k):: strict(x0, function(x) + strict(dw.selRaw(x, k), function(v) if std.isArray(x) then std.map(text, v) else text(v))), + // x.ns#k: the child k in the namespace with the given URI; a child whose prefix is not declared in scope (as when + // it was not selected from its parent) is taken to be in that namespace + selNsRaw(x0, uri, k):: strict(x0, function(x) + local ns = if std.isObject(x) && std.objectHas(x, '@xmlns') then x['@xmlns'] else {}; + local inNamespace(f) = + local i = std.findSubstr(':', f); + local prefix = if std.length(i) == 0 then '$' else f[:i[0]]; + strict(first(x[f]), function(v) + strict(ns + (if isElement(v) && std.objectHas(v, '@xmlns') then v['@xmlns'] else {}), function(declarations) + !std.objectHas(declarations, prefix) || declarations[prefix] == uri)); + if !isXml(x) then dw.selRaw(x, k) + else strict([f for f in std.objectFields(x) if !isMeta(f) && !isTextKey(f) && localName(f) == k && inNamespace(f)], + function(m) if std.length(m) == 0 then null else inScope(x, first(x[m[0]])))), + selNs(x, uri, k):: text(dw.selNsRaw(x, uri, k)), // x.a.b.c pathRaw(x, ks):: std.foldl(function(v, k) dw.selRaw(v, k), ks, x), path(x, ks):: std.foldl(function(v, k) dw.sel(v, k), ks, x), // x.@a - attr(x, a):: dw.selRaw(x, '@' + a), + attr(x0, a):: strict(x0, function(x) + if std.isArray(x) then [dw.attr(e, a) for e in x] + else if std.isObject(x) && std.objectHas(x, '@' + a) then x['@' + a] + else null), // x.*k: all the values of k (an XML element repeated under the same name is read as an array) - multiRaw(x, k):: - if std.isObject(x) then (if !std.objectHas(x, k) then null else if std.isArray(x[k]) then x[k] else [x[k]]) - else if std.isArray(x) then std.flattenArrays([dw.multiRaw(e, k) for e in x if dw.multiRaw(e, k) != null]) - else null, - multi(x, k):: local v = dw.multiRaw(x, k); if v == null then null else std.map(text, v), - // x..k: the values of k at any depth - desc(x, k):: + multiRaw(x0, k):: strict(x0, function(x) + if std.isObject(x) then strict(childKey(x, k), function(key) + if key == null then null else if std.isArray(x[key]) then inScope(x, x[key]) else [inScope(x, x[key])]) + else if std.isArray(x) then + std.flattenArrays([strict(dw.multiRaw(e, k), function(v) if v == null then [] else v) for e in x]) + else null), + multi(x, k):: strict(dw.multiRaw(x, k), function(v) if v == null then null else std.map(text, v)), + // x..k: the values of k at any depth (of an XML element repeated under the same name the first), x..*k all of them + local descendants(x0, k, all) = strict(x0, function(x) local walk(v) = if std.isObject(v) then - (if std.objectHas(v, k) then [text(v[k])] else []) + std.flattenArrays([walk(v[f]) for f in std.objectFields(v)]) + strict(childKey(v, k), function(key) + if key == null then [] else if all && isRepeated(v[key]) then std.map(text, v[key]) else [text(first(v[key]))]) + + std.flattenArrays([walk(v[f]) for f in std.objectFields(v) if !isMeta(f)]) else if std.isArray(v) then std.flattenArrays([walk(e) for e in v]) else []; - if x == null then null else walk(x), + if x == null then null else walk(x)), + desc(x, k):: descendants(x, k, false), + descAll(x, k):: descendants(x, k, true), // x[i]: from the end when negative, null when out of range; x["k"] is x.k - idx(x, i):: + idx(x0, i):: strict(x0, function(x) if x == null || i == null then null else if std.isString(i) then dw.sel(x, i) else if std.isArray(x) || std.isString(x) then ( @@ -81,9 +185,9 @@ local j = if i < 0 then std.length(ks) + i else i; if j < 0 || j >= std.length(ks) then null else x[ks[j]] ) - else null, + else null), // x[a to b]: inclusive, from the end when negative, reversed when a > b - slice(x, a, b):: + slice(x0, a, b):: strict(x0, function(x) if x == null then null else local n = std.length(x); @@ -94,210 +198,301 @@ local part = if std.isString(x) then std.substr(x, lo, hi - lo + 1) else x[lo:hi + 1]; if from <= to then part else if std.isString(x) then std.join('', std.reverse(std.stringChars(part))) - else std.reverse(part), + else std.reverse(part)), // a to b range(a, b):: if a <= b then std.range(a, b) else std.reverse(std.range(b, a)), // x.k? - has(x, k):: std.isObject(x) && std.objectHas(x, k), + has(x0, k):: strict(x0, function(x) std.isObject(x) && childKey(x, k) != null), // -- Values - default(v, fallback):: if v == null then fallback else v, + default(v0, fallback):: strict(v0, function(v) if v == null then fallback else v), // a value in a string (interpolation, dynamic key) - str(v):: if std.isString(v) then v else std.toString(v), - toString(v):: if v == null || std.isString(v) then v else std.toString(v), - nullSafe(f, x):: if x == null then null else f(x), + str(v0):: strict(v0, function(v) if std.isString(v) then v else std.toString(v)), + toString(v0):: strict(v0, function(v) if v == null || std.isString(v) then v else std.toString(v)), + nullSafe(f, x0):: strict(x0, function(x) if x == null then null else f(x)), // a function given as a value, called with up to three arguments - fn(f):: + fn(f0):: strict(f0, function(f) local n = std.length(f); if n == 0 then function(a=null, b=null, c=null) f() else if n == 1 then function(a, b=null, c=null) f(a) else if n == 2 then function(a, b=null, c=null) f(a, b) - else f, - typeOf(x):: + else f), + typeOf(x0):: strict(x0, function(x) if x == null then 'Null' else if std.isString(x) then 'String' else if std.isNumber(x) then 'Number' else if std.isBoolean(x) then 'Boolean' else if std.isArray(x) then 'Array' else if std.isObject(x) then 'Object' - else 'Function', + else 'Function'), + // key @(attributes): value, written as XML (a null attribute as "null", or left out with skipNull); the attributes + // of an element of the input are those of its key in the input, so they are replaced + withAttributes(v0, attrs0, skipNull=false):: strict(v0, function(v) strict(attrs0, function(attrs) + local a = { ['@' + k]: dw.str(attrs[k]) for k in std.objectFields(attrs) if !(skipNull && attrs[k] == null) }; + if std.isArray(v) then [dw.withAttributes(e, attrs, skipNull) for e in v] + else if isElement(v) then withoutAttributes(v) + a + { '~keyAttributes':: true } + else if std.isObject(v) then v + a + else if v == null then a + else a { '$': dw.str(v) })), + // The DataSonnet XML writer writes the child elements in the order of their position in '~' (else of their field): + // the elements of an object made by the script are given their position in it, as an element of the input has its + // position in the input (the output of the script is not read anymore, so a value is written as an element) + // an element of the input without its attributes (but with its namespace declarations), as the value of a key + local withoutAttributes(v) = + std.foldl(function(acc, k) acc + { [k]:: null }, [k for k in std.objectFields(v) if std.startsWith(k, '@') && k != '@xmlns'], v), + local positioned(v, i) = + if std.isObject(v) then v { '~': i } + else if v == null then { '~': i } + else { '$': dw.str(v), '~': i }, + local ordered(x0) = strict(x0, function(x) + if std.isArray(x) then std.map(ordered, x) + else if !std.isObject(x) then x + // an element of the input selected under a key of the script: written with the attributes of that key (none or + // those of withAttributes) + else if isElement(x) then + (if std.objectHasAll(x, '~selected') && !std.objectHasAll(x, '~keyAttributes') then withoutAttributes(x) else x) + else + local fields = [k for k in std.objectFields(x) if !isMeta(k) && !isTextKey(k)]; + // an empty array is written as an empty element + local count(k) = if std.isArray(x[k]) then std.max(1, std.length(x[k])) else 1; + local fieldOrder = std.flattenArrays([[k for _ in std.range(1, count(k))] for k in fields]); + // the keys of the elements in the order of the object, or as kept by xmlObject + strict(if std.objectHasAll(x, '~order') && std.length(x['~order']) == std.length(fieldOrder) + && std.all([std.length([o for o in x['~order'] if o == k]) == count(k) for k in fields]) + then x['~order'] else fieldOrder, function(order) + { [k]: x[k] for k in std.objectFields(x) if isMeta(k) || isTextKey(k) } + + { [k]: strict([i for i in indexes(order) if order[i] == k], function(ps) + if std.isArray(x[k]) && std.length(x[k]) > 0 then [positioned(ordered(x[k][j]), ps[j]) for j in indexes(x[k])] + else positioned(if std.isArray(x[k]) then null else ordered(x[k]), ps[0])) + for k in fields })), + // The result of a script written as XML: an element of the input (such as payload.Envelope.Body) is written as its + // content; the namespaces of the header, and those of the input in scope, are declared on the root element + xmlOutput(doc0, ns):: strict(doc0, function(doc) + local inherited = if isElement(doc) && std.objectHas(doc, '@xmlns') then doc['@xmlns'] else {}; + strict(if isElement(doc) then { [k]: doc[k] for k in std.objectFields(doc) if !isMeta(k) && !isTextKey(k) } else doc, + function(content) + if !std.isObject(content) || std.length(std.objectFields(content)) != 1 then content + else + local root = std.objectFields(content)[0]; + strict(ordered(content[root]), function(v) + if inherited + ns == {} then { [root]: v } + else + local own = if std.isObject(v) && std.objectHas(v, '@xmlns') then v['@xmlns'] else {}; + { [root]: (if std.isObject(v) then v else if v == null then {} else { '$': dw.str(v) }) + + { '@xmlns': inherited + ns + own } }))), + // x.@: the attributes of an XML element + attrs(x0):: strict(x0, function(x) if std.isArray(x) then [attrsOf(e) for e in x] else attrsOf(first(x))), // a ~= b - similar(a, b):: a == b || (a != null && b != null && dw.str(a) == dw.str(b)), + similar(a0, b0):: strict(a0, function(a) strict(b0, function(b) + a == b || (a != null && b != null && dw.str(a) == dw.str(b)))), // an object spread { (x) }: an object, or an array of objects - toObject(x):: + toObject(x0):: strict(x0, function(x) if x == null then {} else if std.isArray(x) then std.foldl(function(acc, o) acc + dw.toObject(o), x, {}) - else x, + else if isXml(x) then fromEntries(entries(x)) + else x), + // An object written as XML, of its parts (objects of its fields, and object spreads): a key that repeats is an + // element that repeats (DataWeave keeps every field of a key that repeats, where Jsonnet keeps the last) + local fieldsOf(x0) = strict(x0, function(x) + if x == null then [] + else if std.isArray(x) then std.flattenArrays([fieldsOf(e) for e in x]) + else if isElement(x) then [{ k: k, v: x[k] } for k in std.objectFields(x) if !isMeta(k) && !isTextKey(k)] + else [{ k: k, v: x[k] } for k in std.objectFields(x)]), + // (the items of an array are elements that repeat); the order of the elements is kept in the hidden field '~order' + xmlObject(parts):: strict(std.flattenArrays([fieldsOf(p) for p in parts]), function(fs) + strict(std.flattenArrays([strict(f.v, function(v) if std.isArray(v) then [{ k: f.k, v: e } for e in v] + else [{ k: f.k, v: v }]) + for f in fs]), function(es) + local keys = std.foldl(function(acc, e) if std.member(acc, e.k) then acc else acc + [e.k], es, []); + { [k]: strict([e.v for e in es if e.k == k], function(vs) if std.length(vs) == 1 then vs[0] else vs) for k in keys } + + { '~order':: [e.k for e in es] })), // obj - "key", array - element - minus(a, b):: - if std.isObject(a) then std.foldl(function(acc, k) if k == b then acc else acc { [k]: a[k] }, std.objectFields(a), {}) + minus(a0, b):: strict(a0, function(a) + if std.isObject(a) then fromEntries([e for e in entries(a) if e.k != b]) else if std.isArray(a) then [e for e in a if e != b] - else a - b, + else a - b), // obj -- ["k1", "k2"] (or an object with the keys), array -- elements - removeAll(a, b):: - local keys = if std.isObject(b) then std.objectFields(b) else b; - if a == null then null - else if std.isObject(a) then std.foldl(function(acc, k) if std.member(keys, k) then acc else acc { [k]: a[k] }, - std.objectFields(a), {}) - else [e for e in a if !std.member(keys, e)], - // the skipNullOn writer property: everywhere, objects or arrays - skipNulls(v, where):: - local objects = where == 'everywhere' || where == 'objects'; + removeAll(a0, b0):: strict(a0, function(a) strict(b0, function(b) + strict(if std.isObject(b) then [e.k for e in entries(b)] else b, function(keys) + if a == null then null + else if std.isObject(a) then fromEntries([e for e in entries(a) if !std.member(keys, e.k)]) + else [e for e in a if !std.member(keys, e)]))), + // the skipNullOn writer property: everywhere, objects or arrays (JSON), elements (XML; attributes are left out + // by withAttributes) + skipNulls(v0, where):: strict(v0, function(v) + local objects = where == 'everywhere' || where == 'objects' || where == 'elements'; local arrays = where == 'everywhere' || where == 'arrays'; if std.isObject(v) then - std.foldl(function(acc, k) if objects && v[k] == null then acc else acc { [k]: dw.skipNulls(v[k], where) }, - std.objectFields(v), {}) - else if std.isArray(v) then [dw.skipNulls(e, where) for e in v if !(arrays && e == null)] - else v, + { [k]: dw.skipNulls(v[k], where) for k in std.objectFields(v) if !(objects && v[k] == null) } + + (if std.objectHasAll(v, '~order') then { '~order':: [k for k in v['~order'] if !(objects && v[k] == null)] } + else {}) + else if std.isArray(v) then + std.flattenArrays([strict(e, function(e) if arrays && e == null then [] else [dw.skipNulls(e, where)]) for e in v]) + else v), // -- Strings - upper(s):: if s == null then null else std.asciiUpper(s), - lower(s):: if s == null then null else std.asciiLower(s), - trim(s):: if s == null then null else std.stripChars(s, ' \t\n\r'), - isBlank(s):: s == null || std.length(std.stripChars(s, ' \t\n\r')) == 0, - contains(a, b):: + upper(s0):: strict(s0, function(s) if s == null then null else std.asciiUpper(s)), + lower(s0):: strict(s0, function(s) if s == null then null else std.asciiLower(s)), + trim(s0):: strict(s0, function(s) if s == null then null else std.stripChars(s, ' \t\n\r')), + isBlank(s0):: strict(s0, function(s) s == null || std.length(std.stripChars(s, ' \t\n\r')) == 0), + contains(a0, b):: strict(a0, function(a) if a == null then false else if std.isArray(a) then std.member(a, b) - else std.length(std.findSubstr(b, a)) > 0, + else std.length(std.findSubstr(b, a)) > 0), // contains /regex/, with scan the ds.scan function - containsMatch(s, scan, regex):: s != null && std.length(scan(s, regex)) > 0, + containsMatch(s0, scan, regex):: strict(s0, function(s) s != null && std.length(scan(s, regex)) > 0), // splitBy /regex/, with split the ds.splitBy function - splitByMatch(s, split, regex):: if s == null then null else split(s, regex), - startsWith(s, prefix):: s != null && std.startsWith(s, prefix), - endsWith(s, suffix):: s != null && std.endsWith(s, suffix), - splitBy(s, separator):: + splitByMatch(s0, split, regex):: strict(s0, function(s) if s == null then null else split(s, regex)), + startsWith(s0, prefix):: strict(s0, function(s) s != null && std.startsWith(s, prefix)), + endsWith(s0, suffix):: strict(s0, function(s) s != null && std.endsWith(s, suffix)), + splitBy(s0, separator):: strict(s0, function(s) if s == null then null else if separator == '' then std.stringChars(s) - else std.split(s, separator), - joinBy(a, separator):: if a == null then null else std.join(separator, [dw.str(e) for e in a]), - replace(s, target, replacement):: if s == null then null else std.strReplace(s, target, replacement), + else std.split(s, separator)), + joinBy(a0, separator):: strict(a0, function(a) + if a == null then null else std.join(separator, [dw.str(e) for e in a])), + replace(s0, target, replacement):: strict(s0, function(s) + if s == null then null else std.strReplace(s, target, replacement)), // replace /regex/ with a function of the match (the matched text and its groups), with starts and matches as given // by ds.find and ds.scan - replaceMatches(s, starts, matches, f):: - if s == null then null - else - local r = std.foldl(function(acc, i) { - out: acc.out + std.substr(s, acc.pos, starts[i] - acc.pos) + dw.str(f(matches[i])), - pos: starts[i] + std.length(matches[i][0]), - }, indexes(matches), { out: '', pos: 0 }); - r.out + std.substr(s, r.pos, std.length(s) - r.pos), + replaceMatches(s0, starts0, matches0, f):: strict(s0, function(s) strict(starts0, function(starts) + strict(matches0, function(matches) + if s == null then null + else + strict(std.foldl(function(acc, i) { + out: acc.out + std.substr(s, acc.pos, starts[i] - acc.pos) + dw.str(f(matches[i])), + pos: starts[i] + std.length(matches[i][0]), + }, indexes(matches), { out: '', pos: 0 }), function(r) r.out + std.substr(s, r.pos, std.length(s) - r.pos))))), // -- Arrays and objects + // + // A function given to them is called with the item evaluated once (an item of an array computed by another + // function is evaluated each time it is used). - sizeOf(x):: if x == null then null else std.length(x), - isEmpty(x):: x == null || std.length(x) == 0, - map(a, f):: if a == null then null else std.mapWithIndex(function(i, x) f(x, i), a), - filter(a, f):: + sizeOf(x0):: strict(x0, function(x) + if x == null then null else if isXml(x) then std.length(entries(x)) else std.length(text(x))), + isEmpty(x0):: strict(x0, function(x) + x == null || strict(text(x), function(t) t == null || std.length(if isXml(x) then entries(x) else t) == 0)), + map(a0, f):: strict(a0, function(a) + if a == null then null else std.mapWithIndex(function(i, x) strict(x, function(x) f(x, i)), a)), + filter(a0, f):: strict(a0, function(a) if a == null then null else if std.isString(a) then ( local cs = std.stringChars(a); std.join('', [cs[i] for i in indexes(cs) if f(cs[i], i)]) ) else if std.isObject(a) then dw.filterObject(a, function(v, k, i) f(v, k)) - else [a[i] for i in indexes(a) if f(a[i], i)], - flatMap(a, f):: if a == null then null else std.flattenArrays(dw.map(a, f)), + else std.flattenArrays([strict(a[i], function(x) if f(x, i) then [x] else []) for i in indexes(a)])), + flatMap(a, f):: strict(dw.map(a, f), function(m) if m == null then null else std.flattenArrays(m)), // reduce with an initial accumulator, and without (starting with the first item) - reduce(a, f, init):: if a == null then null else std.foldl(function(acc, x) f(x, acc), a, init), - reduce1(a, f):: + reduce(a0, f, init):: strict(a0, function(a) + if a == null then null else std.foldl(function(acc, x) strict(x, function(x) f(x, acc)), a, init)), + reduce1(a0, f):: strict(a0, function(a) if a == null || std.length(a) == 0 then null - else std.foldl(function(acc, x) f(x, acc), a[1:], a[0]), - distinctBy(a, f):: + else std.foldl(function(acc, x) strict(x, function(x) f(x, acc)), a[1:], a[0])), + distinctBy(a0, f):: strict(a0, function(a) if a == null then null else if std.isObject(a) then - std.foldl(function(acc, k) - local g = f(a[k], k); - if std.member(acc.keys, g) then acc else { keys: acc.keys + [g], obj: acc.obj { [k]: a[k] } }, - std.objectFields(a), { keys: [], obj: {} }).obj + fromEntries(std.foldl(function(acc, e) + strict(f(e.v, e.k), function(g) + if std.member(acc.keys, g) then acc else { keys: acc.keys + [g], es: acc.es + [e] }), + entries(a), { keys: [], es: [] }).es) else std.foldl(function(acc, i) - local g = f(a[i], i); - if std.member(acc.keys, g) then acc else { keys: acc.keys + [g], items: acc.items + [a[i]] }, - indexes(a), { keys: [], items: [] }).items, - groupBy(a, f):: + strict(a[i], function(x) strict(f(x, i), function(g) + if std.member(acc.keys, g) then acc else { keys: acc.keys + [g], items: acc.items + [x] })), + indexes(a), { keys: [], items: [] }).items), + groupBy(a0, f):: strict(a0, function(a) if a == null then null else if std.isObject(a) then - std.foldl(function(acc, k) - local g = dw.str(f(a[k], k)); - acc { [g]: (if std.objectHas(acc, g) then acc[g] else {}) + { [k]: a[k] } }, - std.objectFields(a), {}) + strict([e { g: dw.str(f(e.v, e.k)) } for e in entries(a)], function(es) + { [g]: fromEntries([e for e in es if e.g == g]) + for g in std.foldl(function(acc, e) if std.member(acc, e.g) then acc else acc + [e.g], es, []) }) else std.foldl(function(acc, i) - local g = dw.str(f(a[i], i)); - acc { [g]: (if std.objectHas(acc, g) then acc[g] else []) + [a[i]] }, - indexes(a), {}), - // a stable sort by key (std.sort of DataSonnet ignores keyF for an array of numbers or strings) - local sortBy(xs, key) = - local keyed = [{ k: key(x), x: x } for x in xs]; - std.flattenArrays([[e.x for e in keyed if e.k == k] for k in std.set([e.k for e in keyed])]), - orderBy(a, f):: + strict(a[i], function(x) strict(dw.str(f(x, i)), function(g) + acc { [g]: (if std.objectHas(acc, g) then acc[g] else []) + [x] })), + indexes(a), {})), + // a stable sort by key, with the key of each item computed once (std.sort of DataSonnet ignores keyF for an array + // of numbers or strings, so it sorts objects with the indexes) + local sortBy(xs0, key) = strict(xs0, function(xs) { + ks: { [std.toString(i)]: key(xs[i], i) for i in indexes(xs) }, + r: [xs[e.i] for e in std.sort([{ i: i } for i in indexes(xs)], function(e) self.ks[std.toString(e.i)])], + }.r), + orderBy(a0, f):: strict(a0, function(a) if a == null then null - else if std.isObject(a) then - dw.toObject(sortBy([{ [k]: a[k] } for k in std.objectFields(a)], - function(e) local k = std.objectFields(e)[0]; f(e[k], k))) - else std.map(function(e) e.x, sortBy([{ i: i, x: a[i] } for i in indexes(a)], function(e) f(e.x, e.i))), - mapObject(o, f):: + else if std.isObject(a) then fromEntries(sortBy(entries(a), function(e, i) f(e.v, e.k))) + else sortBy(a, function(x, i) strict(x, function(x) f(x, i)))), + mapObject(o0, f):: strict(o0, function(o) if o == null then null - else - local ks = std.objectFields(o); - std.foldl(function(acc, i) acc + f(o[ks[i]], ks[i], i), indexes(ks), {}), - filterObject(o, f):: + else strict(entries(o), function(es) + fromEntries(std.flattenArrays([strict(es[i], function(e) entries(f(e.v, e.k, i))) for i in indexes(es)])))), + filterObject(o0, f):: strict(o0, function(o) if o == null then null - else - local ks = std.objectFields(o); - std.foldl(function(acc, i) if f(o[ks[i]], ks[i], i) then acc { [ks[i]]: o[ks[i]] } else acc, indexes(ks), {}), - pluck(o, f):: + else strict(entries(o), function(es) + fromEntries(std.flattenArrays([strict(es[i], function(e) if f(e.v, e.k, i) then [e] else []) + for i in indexes(es)])))), + pluck(o0, f):: strict(o0, function(o) if o == null then null - else - local ks = std.objectFields(o); - [f(o[ks[i]], ks[i], i) for i in indexes(ks)], - keysOf(o):: if o == null then null else std.objectFields(o), + else strict(entries(o), function(es) [strict(es[i], function(e) f(e.v, e.k, i)) for i in indexes(es)])), + keysOf(o0):: strict(o0, function(o) if o == null then null else [e.k for e in entries(o)]), namesOf(o):: dw.keysOf(o), - valuesOf(o):: if o == null then null else [o[k] for k in std.objectFields(o)], - entriesOf(o):: if o == null then null else [{ key: k, value: o[k], attributes: {} } for k in std.objectFields(o)], - flatten(a):: if a == null then null else std.flattenArrays([if std.isArray(e) then e else [e] for e in a]), - zip(a, b):: + valuesOf(o0):: strict(o0, function(o) if o == null then null else [e.v for e in entries(o)]), + entriesOf(o0):: strict(o0, function(o) + if o == null then null else [{ key: e.k, value: e.v, attributes: e.a } for e in entries(o)]), + flatten(a0):: strict(a0, function(a) + if a == null then null else std.flattenArrays([strict(e, function(e) if std.isArray(e) then e else [e]) for e in a])), + zip(a0, b0):: strict(a0, function(a) strict(b0, function(b) if a == null || b == null then null - else [[a[i], b[i]] for i in std.range(0, std.min(std.length(a), std.length(b)) - 1)], + else [[a[i], b[i]] for i in std.range(0, std.min(std.length(a), std.length(b)) - 1)])), // x then f - pipe(v, f):: f(v), - sum(a):: if a == null then null else std.foldl(function(acc, x) acc + x, a, 0), - avg(a):: if a == null then null else dw.sum(a) / std.length(a), - min(a):: if a == null || std.length(a) == 0 then null else std.foldl(function(m, x) if x < m then x else m, a[1:], a[0]), - max(a):: if a == null || std.length(a) == 0 then null else std.foldl(function(m, x) if x > m then x else m, a[1:], a[0]), - maxBy(a, f):: + pipe(v, f):: strict(v, f), + sum(a0):: strict(a0, function(a) if a == null then null else std.foldl(function(acc, x) acc + x, a, 0)), + avg(a0):: strict(a0, function(a) if a == null then null else dw.sum(a) / std.length(a)), + min(a0):: strict(a0, function(a) + if a == null || std.length(a) == 0 then null + else std.foldl(function(m, x) strict(x, function(x) if x < m then x else m), a[1:], a[0])), + max(a0):: strict(a0, function(a) if a == null || std.length(a) == 0 then null - else std.foldl(function(m, x) if f(x) > f(m) then x else m, a[1:], a[0]), - minBy(a, f):: + else std.foldl(function(m, x) strict(x, function(x) if x > m then x else m), a[1:], a[0])), + maxBy(a0, f):: strict(a0, function(a) if a == null || std.length(a) == 0 then null - else std.foldl(function(m, x) if f(x) < f(m) then x else m, a[1:], a[0]), - sumBy(a, f):: if a == null then null else std.foldl(function(acc, x) acc + f(x), a, 0), - countBy(a, f):: if a == null then null else std.length([x for x in a if f(x)]), - every(a, f):: a == null || std.all([f(x) for x in a]), - some(a, f):: a != null && std.any([f(x) for x in a]), - firstWith(a, f):: if a == null then null else (local m = [x for x in a if f(x)]; if std.length(m) == 0 then null else m[0]), - partition(a, f):: if a == null then null else { success: [x for x in a if f(x)], failure: [x for x in a if !f(x)] }, - indexOf(a, x):: + else std.foldl(function(m, x) strict(x, function(x) if f(x) > f(m) then x else m), a[1:], a[0])), + minBy(a0, f):: strict(a0, function(a) + if a == null || std.length(a) == 0 then null + else std.foldl(function(m, x) strict(x, function(x) if f(x) < f(m) then x else m), a[1:], a[0])), + sumBy(a0, f):: strict(a0, function(a) if a == null then null else std.foldl(function(acc, x) acc + f(x), a, 0)), + countBy(a0, f):: strict(a0, function(a) if a == null then null else std.length([x for x in a if f(x)])), + every(a0, f):: strict(a0, function(a) a == null || std.all([f(x) for x in a])), + some(a0, f):: strict(a0, function(a) a != null && std.any([f(x) for x in a])), + firstWith(a0, f):: strict(a0, function(a) + if a == null then null + else strict([x for x in a if f(x)], function(m) if std.length(m) == 0 then null else m[0])), + partition(a0, f):: strict(a0, function(a) + if a == null then null else { success: [x for x in a if f(x)], failure: [x for x in a if !f(x)] }), + indexOf(a0, x):: strict(a0, function(a) if a == null then -1 - else if std.isString(a) then (local found = std.findSubstr(x, a); if std.length(found) == 0 then -1 else found[0]) - else (local found = [i for i in indexes(a) if a[i] == x]; if std.length(found) == 0 then -1 else found[0]), - lastIndexOf(a, x):: + else strict(if std.isString(a) then std.findSubstr(x, a) else [i for i in indexes(a) if a[i] == x], + function(found) if std.length(found) == 0 then -1 else found[0])), + lastIndexOf(a0, x):: strict(a0, function(a) if a == null then -1 - else if std.isString(a) then (local found = std.findSubstr(x, a); if std.length(found) == 0 then -1 else found[std.length(found) - 1]) - else (local found = [i for i in indexes(a) if a[i] == x]; if std.length(found) == 0 then -1 else found[std.length(found) - 1]), - take(a, n):: if a == null then null else a[0:std.max(0, std.min(n, std.length(a)))], - drop(a, n):: if a == null then null else a[std.max(0, std.min(n, std.length(a))):], - splitAt(a, n):: if a == null then null else { l: dw.take(a, n), r: dw.drop(a, n) }, - divideBy(a, n):: if a == null then null else [a[i:std.min(i + n, std.length(a))] for i in std.range(0, std.length(a) - 1) if i % n == 0], + else strict(if std.isString(a) then std.findSubstr(x, a) else [i for i in indexes(a) if a[i] == x], + function(found) if std.length(found) == 0 then -1 else found[std.length(found) - 1])), + take(a0, n):: strict(a0, function(a) if a == null then null else a[0:std.max(0, std.min(n, std.length(a)))]), + drop(a0, n):: strict(a0, function(a) if a == null then null else a[std.max(0, std.min(n, std.length(a))):]), + splitAt(a0, n):: strict(a0, function(a) if a == null then null else { l: dw.take(a, n), r: dw.drop(a, n) }), + divideBy(a0, n):: strict(a0, function(a) + if a == null then null else [a[i:std.min(i + n, std.length(a))] for i in std.range(0, std.length(a) - 1) if i % n == 0]), // -- Numbers - abs(x):: if x == null then null else if x < 0 then -x else x, + abs(x0):: strict(x0, function(x) if x == null then null else if x < 0 then -x else x), // half up, as DataWeave - round(x):: if x == null then null else if x < 0 then -std.floor(-x + 0.5) else std.floor(x + 0.5), + round(x0):: strict(x0, function(x) if x == null then null else if x < 0 then -std.floor(-x + 0.5) else std.floor(x + 0.5)), isEven(x):: x % 2 == 0, isOdd(x):: x % 2 != 0, - isInteger(x):: std.isNumber(x) && std.floor(x) == x, - isDecimal(x):: std.isNumber(x) && std.floor(x) != x, + isInteger(x0):: strict(x0, function(x) std.isNumber(x) && std.floor(x) == x), + isDecimal(x0):: strict(x0, function(x) std.isNumber(x) && std.floor(x) != x), } diff --git a/components/camel-datasonnet/src/test/java/org/apache/camel/language/datasonnet/DataWeaveCorpusTest.java b/components/camel-datasonnet/src/test/java/org/apache/camel/language/datasonnet/DataWeaveCorpusTest.java index e15ccbb0dfa6..8885cc025994 100644 --- a/components/camel-datasonnet/src/test/java/org/apache/camel/language/datasonnet/DataWeaveCorpusTest.java +++ b/components/camel-datasonnet/src/test/java/org/apache/camel/language/datasonnet/DataWeaveCorpusTest.java @@ -18,6 +18,7 @@ package org.apache.camel.language.datasonnet; import java.io.IOException; import java.io.InputStream; +import java.io.StringReader; import java.nio.charset.StandardCharsets; import java.util.ArrayList; import java.util.LinkedHashMap; @@ -25,8 +26,19 @@ import java.util.List; import java.util.Locale; import java.util.Map; import java.util.Properties; +import java.util.TreeMap; import java.util.stream.Stream; +import javax.xml.XMLConstants; +import javax.xml.parsers.DocumentBuilderFactory; + +import org.w3c.dom.Element; +import org.w3c.dom.NamedNodeMap; +import org.w3c.dom.Node; +import org.w3c.dom.Text; + +import org.xml.sax.InputSource; + import com.fasterxml.jackson.databind.ObjectMapper; import org.apache.camel.Exchange; import org.apache.camel.Expression; @@ -40,6 +52,7 @@ import org.junit.jupiter.params.provider.Arguments; import org.junit.jupiter.params.provider.MethodSource; import org.skyscreamer.jsonassert.JSONAssert; +import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertThrows; import static org.junit.jupiter.api.Assertions.fail; @@ -163,11 +176,54 @@ class DataWeaveCorpusTest extends CamelTestSupport { fail("Converted without error but fails when evaluated: " + exchange.getException().getMessage(), exchange.getException()); } - if (!"?".equals(expected)) { + if ("?".equals(expected)) { + return; + } + if (entry.directive("outType", JSON).endsWith("xml")) { + assertEquals(canonicalXml(expected), canonicalXml(actual), actual); + } else { JSONAssert.assertEquals("{\"v\":" + expected + "}", "{\"v\":" + actual + "}", true); } } + // An XML document without its formatting: whether it has an XML declaration, and its elements with their namespace, + // attributes (in order of name, without the namespace declarations, which can be on another element) and text + private static String canonicalXml(String xml) throws Exception { + DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance(); + factory.setNamespaceAware(true); + factory.setFeature("http://apache.org/xml/features/disallow-doctype-decl";, true); + String document = xml.strip(); + StringBuilder sb = new StringBuilder(document.startsWith("<?xml") ? "<?xml?>" : ""); + canonicalXml(factory.newDocumentBuilder().parse(new InputSource(new StringReader(document))).getDocumentElement(), + sb); + return sb.toString(); + } + + private static void canonicalXml(Element element, StringBuilder sb) { + sb.append('<').append(element.getTagName()); + if (element.getNamespaceURI() != null) { + sb.append(" {").append(element.getNamespaceURI()).append('}'); + } + Map<String, String> attributes = new TreeMap<>(); + NamedNodeMap map = element.getAttributes(); + for (int i = 0; i < map.getLength(); i++) { + Node attribute = map.item(i); + if (!XMLConstants.XMLNS_ATTRIBUTE_NS_URI.equals(attribute.getNamespaceURI())) { + attributes.put(attribute.getNodeName(), attribute.getNodeValue()); + } + } + attributes.forEach((name, value) -> sb.append(' ').append(name).append("=\"").append(value).append('"')); + sb.append('>'); + for (Node child = element.getFirstChild(); child != null; child = child.getNextSibling()) { + if (child instanceof Element e) { + canonicalXml(e, sb); + } else if (child instanceof Text text && !text.getData().isBlank()) { + sb.append(text.getData().strip()); + } + } + sb.append("</").append(element.getTagName()).append('>'); + } + private Expression createExpression(Entry entry, String dataWeave) { return context.resolveLanguage("datasonnet").createExpression(dataWeave, new Object[] { diff --git a/components/camel-datasonnet/src/test/resources/dataweave-corpus/corpus.txt b/components/camel-datasonnet/src/test/resources/dataweave-corpus/corpus.txt index 33592c0f4102..26ae8d818f15 100644 --- a/components/camel-datasonnet/src/test/resources/dataweave-corpus/corpus.txt +++ b/components/camel-datasonnet/src/test/resources/dataweave-corpus/corpus.txt @@ -17,7 +17,8 @@ # snippet is the output of the DataWeave CLI (dw 2.12, English locale); vars and attributes are given to dw as inputs. # Each entry starts with "=== <id> <title>", followed by directives and then the DataWeave script: # -# @exp <json> the output DataWeave produces (as JSON); "?" = any output, PARSE-ERROR = invalid DataWeave +# @exp <json> the output DataWeave produces (as JSON, or the XML of an XML output, compared without formatting +# and namespace declarations); "?" = any output, PARSE-ERROR = invalid DataWeave # @in <text> the body (default: default.json) # @inType / @outType the body and output media types (default: application/json) # @vars / @headers exchange variables / message headers as a JSON object @@ -122,7 +123,7 @@ fun fullName(c) = c.first ++ " " ++ c.last } === H02 output xml @outType application/xml -@exp ? +@exp <?xml version='1.0' encoding='UTF-8'?><order><id>7</id><name>Ann</name></order> %dw 2.0 output application/xml --- @@ -163,9 +164,8 @@ output application/json --- countBy(payload.nums, (n) -> n > 2) === H07 ns declaration and ns-qualified key -@unsupported XML namespaces @outType application/xml -@exp ? +@exp <?xml version='1.0' encoding='UTF-8'?><ns0:order xmlns:ns0="http://example.com/order";><id>7</id></ns0:order> %dw 2.0 output application/xml ns ns0 http://example.com/order @@ -681,10 +681,9 @@ payload.customer ++ { vip: true } === O11 quoted keys @exp {"first-name":"Ann"} { "first-name": payload.customer.first } -=== O12 object key with ns/attrs -@unsupported XML attributes in output +=== O12 object key with attributes @outType application/xml -@exp ? +@exp <?xml version='1.0' encoding='UTF-8'?><order id="7"><name>Ann</name></order> %dw 2.0 output application/xml --- @@ -719,11 +718,201 @@ payload.order.item payload.orders.*order map $.@id === XM04 xml output with list @outType application/xml -@exp ? +@exp <?xml version='1.0' encoding='UTF-8'?><items><item>A1</item><item>B2</item><item>A1</item></items> %dw 2.0 output application/xml --- { items: { (payload.items map ((i) -> { item: i.sku })) } } +=== XM05 SOAP selectors ignore namespace prefixes +@inType application/xml +@in <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"; xmlns:ns0="http://example.com/orders";><soapenv:Body><ns0:getOrderResponse><ns0:order id="7"><ns0:item>apple</ns0:item></ns0:order></ns0:getOrderResponse></soapenv:Body></soapenv:Envelope> +@exp ["7","apple"] +[payload.Envelope.Body.getOrderResponse.order.@id, payload.Envelope.Body.getOrderResponse.order.item] +=== XM06 namespace-qualified selectors +@inType application/xml +@in <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"; xmlns:ns0="http://example.com/orders";><soapenv:Body><ns0:getOrderResponse><ns0:order id="7"><ns0:item>apple</ns0:item></ns0:order></ns0:getOrderResponse></soapenv:Body></soapenv:Envelope> +@exp ["apple","7",null] +%dw 2.0 +output application/json +ns o http://example.com/orders +ns s http://schemas.xmlsoap.org/soap/envelope/ +--- +[payload.s#Envelope.s#Body.o#getOrderResponse.o#order.o#item, payload.Envelope.Body.o#getOrderResponse.order.@id, payload.s#Envelope.o#Body] +=== XM07 element with child elements written as JSON +@inType application/xml +@in <order id="1"><item>apple</item><note>n</note><empty/></order> +@exp {"order":{"item":"apple","note":"n","empty":null}} +payload +=== XM08 empty element, keys, size and attributes of an element +@inType application/xml +@in <order id="1" status="new"><item>apple</item><item>pear</item><note>n</note><empty/></order> +@exp [null,["item","item","note","empty"],4,["apple","pear"],{"id":"1","status":"new"}] +[payload.order.empty, keysOf(payload.order), sizeOf(payload.order), payload.order.*item, payload.order.@] +=== XM09 elements repeated with others in between keep the document order +@inType application/xml +@in <a><x>1</x><y>2</y><x>3</x></a> +@exp [["x","y","x"],["1","2","3"]] +[keysOf(payload.a), valuesOf(payload.a)] +=== XM10 attributes and text of the items of an implicit lambda +@inType application/xml +@in <orders><order id="1">a</order><order id="2">b</order></orders> +@exp [{"id":"1","text":"a","all":{"id":"1"}},{"id":"2","text":"b","all":{"id":"2"}}] +payload.orders.*order map { id: $.@id, text: $, all: $.@ } +=== XM11 attributes of a lambda parameter +@inType application/xml +@in <orders><order id="1">a</order><order id="2">b</order></orders> +@exp [{"id":"1","v":"a","i":0},{"id":"2","v":"b","i":1}] +payload.orders.*order map ((o, i) -> { id: o.@id, v: o, i: i }) +=== XM12 existence of elements and attributes +@inType application/xml +@in <order id="1"><item>apple</item><empty/></order> +@exp [true,false,true,true,false] +[payload.order.item?, payload.order.missing?, payload.order.empty?, payload.order.@id?, payload.order.@missing?] +=== XM13 object functions on an element +@inType application/xml +@in <order id="1"><item>apple</item><note>n</note><empty/></order> +@exp [{"ITEM":"apple","NOTE":"n","EMPTY":null},["item","note","empty"],{"item":"apple","note":"n"},{"item":"apple","note":"n","empty":null},{"item":"apple","empty":null},{"item":"apple"},["apple","n",null]] +[payload.order mapObject ((v, k) -> { (upper(k)): v }), payload.order pluck $$, payload.order filterObject ((v) -> v != null), { (payload.order) }, payload.order - "note", payload.order -- ["note", "empty"], valuesOf(payload.order)] +=== XM14 entriesOf with the attributes of the elements +@inType application/xml +@in <order><item sku="A1">apple</item><note>n</note></order> +@exp [{"key":"item","value":"apple","attributes":{"sku":"A1"}},{"key":"note","value":"n","attributes":{}}] +entriesOf(payload.order) +=== XM15 CDATA +@inType application/xml +@in <a><![CDATA[x < y]]></a> +@exp "x < y" +payload.a +=== XM16 descendants +@inType application/xml +@in <orders><order><item>a</item></order><order><item>b</item><item>c</item></order></orders> +@exp [["a","b"],["a","b","c"]] +[payload..item, payload..*item] +=== XM17 attributes of a variable +@inType application/xml +@in <order id="7"><item sku="A1">apple</item><note>n</note></order> +@exp ["7","A1"] +%dw 2.0 +output application/json +var o = payload.order +--- +[o.@id, o.item.@sku] +=== XM18 groupBy on an element +@inType application/xml +@in <order><item>apple</item><item>pear</item><note>n</note></order> +@exp {"apple":{"item":"apple"},"pear":{"item":"pear"},"n":{"note":"n"}} +payload.order groupBy $ +=== XM19 XML to XML +@inType application/xml +@outType application/xml +@in <order id="7"><item>a</item><empty/></order> +@exp <?xml version='1.0' encoding='UTF-8'?><order id="7"><item>a</item><empty/></order> +%dw 2.0 +output application/xml +--- +payload +=== XM20 SOAP body content as XML +@inType application/xml +@outType application/xml +@in <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"; xmlns:ns0="http://example.com/orders";><soapenv:Body><ns0:getOrderResponse><ns0:order id="7"><ns0:item>apple</ns0:item></ns0:order></ns0:getOrderResponse></soapenv:Body></soapenv:Envelope> +@exp <?xml version='1.0' encoding='UTF-8'?><ns0:getOrderResponse xmlns:ns0="http://example.com/orders";><ns0:order id="7"><ns0:item>apple</ns0:item></ns0:order></ns0:getOrderResponse> +%dw 2.0 +output application/xml +--- +payload.Envelope.Body +=== XM21 element of the input under another key +@inType application/xml +@outType application/xml +@in <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"; xmlns:ns0="http://example.com/orders";><soapenv:Body><ns0:getOrderResponse><ns0:order id="7"><ns0:item>apple</ns0:item></ns0:order></ns0:getOrderResponse></soapenv:Body></soapenv:Envelope> +@exp <?xml version='1.0' encoding='UTF-8'?><resp><status>ok</status><order><ns0:item xmlns:ns0="http://example.com/orders";>apple</ns0:item></order></resp> +%dw 2.0 +output application/xml +--- +{ resp: { status: "ok", order: payload.Envelope.Body.getOrderResponse.order } } +=== XM22 attributes, null attribute and null element +@outType application/xml +@exp <?xml version='1.0' encoding='UTF-8'?><root><price currency="USD" none="null">10</price><flag a="true">true</flag><empty/></root> +%dw 2.0 +output application/xml +--- +{ root: { price @(currency: "USD", none: null): 10, flag @(a: true): true, empty: null } } +=== XM23 skipNullOn everywhere in XML +@outType application/xml +@exp <?xml version='1.0' encoding='UTF-8'?><root><price currency="USD">10</price></root> +%dw 2.0 +output application/xml skipNullOn="everywhere" +--- +{ root: { price @(currency: "USD", none: null): 10, empty: null } } +=== XM24 skipNullOn attributes +@outType application/xml +@exp <?xml version='1.0' encoding='UTF-8'?><root><price currency="USD">10</price><empty/></root> +%dw 2.0 +output application/xml skipNullOn="attributes" +--- +{ root: { price @(currency: "USD", none: null): 10, empty: null } } +=== XM25 namespaces of the header in XML +@outType application/xml +@exp <?xml version='1.0' encoding='UTF-8'?><o:a xmlns:o="http://x"; id="1"><o:b>1</o:b><c><o:d>2</o:d></c></o:a> +%dw 2.0 +output application/xml +ns o http://x +--- +{ o#a @(id: 1): { o#b: "1", c: { o#d: 2 } } } +=== XM26 writeDeclaration false +@outType application/xml +@exp <root><a>1</a></root> +%dw 2.0 +output application/xml writeDeclaration=false +--- +{ root: { a: 1 } } +=== XM27 arrays and object spreads are elements that repeat +@outType application/xml +@exp <?xml version='1.0' encoding='UTF-8'?><root><head>1</head><item>A1</item><item>B2</item><item>A1</item><n>1</n><n>2</n><tail>2</tail></root> +%dw 2.0 +output application/xml +--- +{ root: { head: 1, (payload.items map { item: $.sku }), n: [1, 2], tail: 2 } } +=== XM28 keys that repeat keep their order +@outType application/xml +@exp <?xml version='1.0' encoding='UTF-8'?><root><a>1</a><b>2</b><a>3</a></root> +%dw 2.0 +output application/xml +--- +{ root: { a: 1, b: 2, a: 3 } } +=== XM29 repeated elements with attributes from XML +@inType application/xml +@outType application/xml +@in <orders><order id="1"><item>a</item></order><order id="2"><item>b</item></order></orders> +@exp <?xml version='1.0' encoding='UTF-8'?><ids><id n="0">1</id><id n="1">2</id></ids> +%dw 2.0 +output application/xml +--- +{ ids: { (payload.orders.*order map { id @(n: $$): $.@id }) } } +=== XM30 copy of an element with an extra element +@inType application/xml +@outType application/xml +@in <order id="7"><item>a</item><item>b</item><note>n</note></order> +@exp <?xml version='1.0' encoding='UTF-8'?><copy><item>a</item><item>b</item><note>n</note><extra>x</extra></copy> +%dw 2.0 +output application/xml +--- +{ copy: { (payload.order), extra: "x" } } +=== XM31 element of the input under a key with attributes +@inType application/xml +@outType application/xml +@in <orders><order id="1"><item>a</item></order></orders> +@exp <?xml version='1.0' encoding='UTF-8'?><x n="1"><item>a</item></x> +%dw 2.0 +output application/xml +--- +{ x @(n: "1"): payload.orders.order } +=== XM32 empty object and array, numbers and escaping in XML +@outType application/xml +@exp <?xml version='1.0' encoding='UTF-8'?><root><e/><ea/><n>1.5</n><t>true</t><s>x & y < z</s></root> +%dw 2.0 +output application/xml +--- +{ root: { e: {}, ea: [], n: 1.50, t: true, s: "x & y < z" } } === CS01 csv input @inType application/csv @in name,age\nann,30\nbob,40 diff --git a/components/camel-dataweave/src/main/java/org/apache/camel/component/dataweave/DataWeaveAst.java b/components/camel-dataweave/src/main/java/org/apache/camel/component/dataweave/DataWeaveAst.java index cc89bb27adc2..8ab1e633409b 100644 --- a/components/camel-dataweave/src/main/java/org/apache/camel/component/dataweave/DataWeaveAst.java +++ b/components/camel-dataweave/src/main/java/org/apache/camel/component/dataweave/DataWeaveAst.java @@ -34,10 +34,11 @@ public sealed interface DataWeaveAst { /** * The header directives. {@code outputProperties} are the writer properties of the output directive (such as - * {@code skipNullOn}), and {@code imports} the imported module paths (such as {@code dw::core::Strings}). + * {@code skipNullOn}), {@code imports} the imported module paths (such as {@code dw::core::Strings}), and + * {@code namespaces} the URIs of the declared XML namespace prefixes ({@code ns prefix uri}). */ record Header(String version, String outputType, Map<String, String> outputProperties, List<InputDecl> inputs, - List<String> imports) + List<String> imports, Map<String, String> namespaces) implements DataWeaveAst { } @@ -91,11 +92,21 @@ public sealed interface DataWeaveAst { /** * An object entry {@code key: value}, where a {@code dynamic} key is an expression ({@code (expr): value}). A * {@code null} key is an object spread {@code (expr)}, which adds the entries of an object (or array of objects). - * The {@code condition} of a conditional entry {@code (key: value) if cond} is null for an unconditional entry. + * The {@code condition} of a conditional entry {@code (key: value) if cond} is null for an unconditional entry, and + * the {@code attributes} are the XML attributes of the key ({@code key @(name: value): ...}). */ - record ObjectEntry(DataWeaveAst key, DataWeaveAst value, boolean dynamic, DataWeaveAst condition) + record ObjectEntry(DataWeaveAst key, DataWeaveAst value, boolean dynamic, DataWeaveAst condition, + List<ObjectEntry> attributes) implements DataWeaveAst { + + public ObjectEntry(DataWeaveAst key, DataWeaveAst value, boolean dynamic, DataWeaveAst condition) { + this(key, value, dynamic, condition, List.of()); + } + } + + /** A name in an XML namespace: {@code prefix#name}, with the prefix declared in the header. */ + record QName(String prefix, String name) implements DataWeaveAst { } record ArrayLit(List<DataWeaveAst> elements) implements DataWeaveAst { @@ -120,12 +131,26 @@ public sealed interface DataWeaveAst { record AttributeAccess(DataWeaveAst object, String attribute) implements DataWeaveAst { } + /** Attributes selector {@code .@}: all the attributes of an XML element. */ + record AllAttributes(DataWeaveAst object) implements DataWeaveAst { + } + + /** Single-value selector of an element in an XML namespace {@code .prefix#field}. */ + record QualifiedFieldAccess(DataWeaveAst object, String prefix, String field) implements DataWeaveAst { + } + /** Multi-value selector {@code .*field}. */ record MultiValueSelector(DataWeaveAst object, String field) implements DataWeaveAst { } - /** Descendants selector {@code ..field}. */ - record DescendantSelector(DataWeaveAst object, String field) implements DataWeaveAst { + /** + * Descendants selector {@code ..field}, or {@code ..*field} (multi) for all the values of an XML element that + * repeats. + */ + record DescendantSelector(DataWeaveAst object, String field, boolean multi) implements DataWeaveAst { + DescendantSelector(DataWeaveAst object, String field) { + this(object, field, false); + } } /** Index or dynamic key selector {@code [expr]}; the index is a {@link Range} for {@code [a to b]}. */ diff --git a/components/camel-dataweave/src/main/java/org/apache/camel/component/dataweave/DataWeaveConverter.java b/components/camel-dataweave/src/main/java/org/apache/camel/component/dataweave/DataWeaveConverter.java index 0064db61b0d7..fd086377392c 100644 --- a/components/camel-dataweave/src/main/java/org/apache/camel/component/dataweave/DataWeaveConverter.java +++ b/components/camel-dataweave/src/main/java/org/apache/camel/component/dataweave/DataWeaveConverter.java @@ -28,6 +28,7 @@ import java.util.Map; import java.util.Set; import java.util.function.Predicate; +import org.apache.camel.component.dataweave.DataWeaveAst.AllAttributes; import org.apache.camel.component.dataweave.DataWeaveAst.ArrayLit; import org.apache.camel.component.dataweave.DataWeaveAst.AttributeAccess; import org.apache.camel.component.dataweave.DataWeaveAst.BinaryOp; @@ -57,6 +58,8 @@ import org.apache.camel.component.dataweave.DataWeaveAst.NumberLit; import org.apache.camel.component.dataweave.DataWeaveAst.ObjectEntry; import org.apache.camel.component.dataweave.DataWeaveAst.ObjectLit; import org.apache.camel.component.dataweave.DataWeaveAst.Parens; +import org.apache.camel.component.dataweave.DataWeaveAst.QName; +import org.apache.camel.component.dataweave.DataWeaveAst.QualifiedFieldAccess; import org.apache.camel.component.dataweave.DataWeaveAst.Range; import org.apache.camel.component.dataweave.DataWeaveAst.RegexLit; import org.apache.camel.component.dataweave.DataWeaveAst.Script; @@ -183,6 +186,12 @@ public class DataWeaveConverter { private final Deque<Map<String, Binding>> scopes = new ArrayDeque<>(); // the variables $, $$ and $$$ refer to in the body of an implicit lambda, or null private List<String> dollars; + // the lambda parameters that are XML elements kept as objects for their attributes (by their Jsonnet name) + private final Set<String> rawNames = new HashSet<>(); + // whether the script writes XML, the XML namespaces of its header, and whether null attributes are left out + private boolean xmlOutput; + private Map<String, String> namespaces = Map.of(); + private boolean skipNullAttributes; public DataWeaveConverter() { } @@ -231,32 +240,59 @@ public class DataWeaveConverter { needsDataWeaveLib = false; scopes.clear(); dollars = null; + rawNames.clear(); + xmlOutput = false; + namespaces = Map.of(); + skipNullAttributes = false; } // -- Script private String emitScript(Script script) { Header header = script.header(); - StringBuilder sb = new StringBuilder(); String output = header.outputType() != null ? mediaType(header.outputType()) : null; + xmlOutput = output != null && output.contains("xml"); + namespaces = header.namespaces(); + String skipNullOn = xmlOutput ? header.outputProperties().get("skipNullOn") : null; + skipNullAttributes = "attributes".equals(skipNullOn) || "everywhere".equals(skipNullOn); + StringBuilder todos = new StringBuilder(); + for (String module : header.imports()) { + if (!KNOWN_MODULES.contains(module)) { + todos.append(todoComment("import of module", module)); + } + } + // DataSonnet writes a null as an empty XML element only when asked to, DataWeave always does + List<String> outputParameters = new ArrayList<>(); + if (xmlOutput) { + outputParameters.add("nullasemptyelement=true"); + } + String body = applyWriterProperties(emit(script.body()), output, header.outputProperties(), outputParameters); + if (xmlOutput) { + // an element of the input is written as its content, the elements in the order of the script, and the + // namespaces declared on the root element + List<String> declarations = new ArrayList<>(); + namespaces.forEach((prefix, uri) -> declarations + .add((isJsonnetIdentifier(prefix) ? prefix : string(prefix)) + ": " + string(uri))); + body = lib("xmlOutput") + "(" + body + ", " + + (declarations.isEmpty() ? "{}" : "{ " + String.join(", ", declarations) + " }") + ")"; + } else if (output != null) { + // the elements of an XML input are written as DataWeave writes them, without the XML details + body = lib("output") + "(" + body + ", body)"; + } + + StringBuilder sb = new StringBuilder(); if (output != null || !header.inputs().isEmpty()) { sb.append("/** DataSonnet\nversion=2.0\n"); if (output != null) { - sb.append("output ").append(output).append('\n'); + sb.append("output ").append(output); + outputParameters.forEach(p -> sb.append("; ").append(p)); + sb.append('\n'); } for (InputDecl input : header.inputs()) { sb.append("input ").append(input.name()).append(' ').append(mediaType(input.mediaType())).append('\n'); } sb.append("*/\n"); } - StringBuilder todos = new StringBuilder(); - for (String module : header.imports()) { - if (!KNOWN_MODULES.contains(module)) { - todos.append(todoComment("import of module", module)); - } - } - String body = emit(script.body()); - body = applyWriterProperties(body, output, header.outputProperties()); if (needsDataWeaveLib) { sb.append(DW_IMPORT); } @@ -274,17 +310,26 @@ public class DataWeaveConverter { }; } - private String applyWriterProperties(String body, String output, Map<String, String> properties) { + private String applyWriterProperties( + String body, String output, Map<String, String> properties, List<String> outputParameters) { String result = body; for (Map.Entry<String, String> property : properties.entrySet()) { String name = property.getKey(); - if ("indent".equals(name) || "encoding".equals(name)) { + String value = property.getValue(); + if ("indent".equals(name) || "encoding".equals(name) || xmlOutput && "inlineCloseOn".equals(name)) { continue; // formatting only } - if ("skipNullOn".equals(name) && (output == null || output.endsWith("json"))) { - result = lib("skipNulls") + "(" + result + ", " + string(property.getValue()) + ")"; + if (xmlOutput && "skipNullOn".equals(name) && "attributes".equals(value)) { + continue; // left out by withAttributes + } + if ("skipNullOn".equals(name) && (output == null || output.endsWith("json") || xmlOutput)) { + result = lib("skipNulls") + "(" + result + ", " + string(value) + ")"; + } else if (xmlOutput && "writeDeclaration".equals(name)) { + if ("false".equalsIgnoreCase(value)) { + outputParameters.add("omitxmldeclaration=true"); + } } else { - result = todoComment("writer property", name + "=" + property.getValue()) + result; + result = todoComment("writer property", name + "=" + value) + result; } } return result; @@ -312,12 +357,15 @@ public class DataWeaveConverter { return emitIdentifier(id); } else if (node instanceof Dollar d) { return emitDollar(d); - } else if (node instanceof FieldAccess || node instanceof IndexAccess || node instanceof MultiValueSelector) { + } else if (node instanceof FieldAccess || node instanceof IndexAccess || node instanceof MultiValueSelector + || node instanceof QualifiedFieldAccess) { return emitSelector(node, false); + } else if (node instanceof AllAttributes aa) { + return lib("attrs") + "(" + emitSelector(aa.object(), true) + ")"; } else if (node instanceof AttributeAccess aa) { return lib("attr") + "(" + emitSelector(aa.object(), true) + ", " + string(aa.attribute()) + ")"; } else if (node instanceof DescendantSelector ds) { - return lib("desc") + "(" + emit(ds.object()) + ", " + string(ds.field()) + ")"; + return lib(ds.multi() ? "descAll" : "desc") + "(" + emit(ds.object()) + ", " + string(ds.field()) + ")"; } else if (node instanceof FilterSelector fs) { return lib("filter") + "(" + emit(fs.object()) + ", " + emitImplicitLambda(fs.condition(), ITEM_INDEX) + ")"; @@ -387,6 +435,9 @@ public class DataWeaveConverter { // a date coerced with a format is written in that format return formatDate(binding.name(), binding.date().properties()); } + if (binding != null && rawNames.contains(binding.name())) { + return lib("text") + "(" + binding.name() + ")"; + } if (binding != null) { return binding.name(); } @@ -399,7 +450,8 @@ public class DataWeaveConverter { private String emitDollar(Dollar d) { if (dollars != null && d.level() <= dollars.size()) { - return dollars.get(d.level() - 1); + String name = dollars.get(d.level() - 1); + return rawNames.contains(name) ? lib("text") + "(" + name + ")" : name; } return todo("$ outside of a lambda", "$".repeat(d.level())); } @@ -408,9 +460,21 @@ public class DataWeaveConverter { // A selector; raw keeps an XML element with only text as an object (for its attributes) private String emitSelector(DataWeaveAst node, boolean raw) { + String rawName = rawReference(node); + if (rawName != null) { + return rawName; + } if (isExchangeAccess(node)) { return emitExchangeAccess(node); } + if (node instanceof QualifiedFieldAccess q) { + String uri = namespaces.get(q.prefix()); + if (uri == null) { + return todo("undeclared namespace prefix", q.prefix() + "#" + q.field()); + } + return lib(raw ? "selNsRaw" : "selNs") + "(" + emitSelector(q.object(), true) + ", " + string(uri) + ", " + + string(q.field()) + ")"; + } if (node instanceof FieldAccess fa && DATE_PARTS.contains(fa.field()) && isDateValue(fa.object())) { return "cml.datePart(" + emitDate(fa.object()) + ", " + string(fa.field()) + ")"; } @@ -426,9 +490,7 @@ public class DataWeaveConverter { fields.add(0, string(f.field())); base = f.object(); } - String object = base instanceof FieldAccess || base instanceof IndexAccess - || base instanceof MultiValueSelector - ? emitSelector(base, true) : emit(base); + String object = emitSelector(base, true); if (fields.size() == 1) { return lib(raw ? "selRaw" : "sel") + "(" + object + ", " + fields.get(0) + ")"; } @@ -444,6 +506,17 @@ public class DataWeaveConverter { return emit(node); } + // The Jsonnet name of a lambda parameter that is an XML element kept as an object, or null + private String rawReference(DataWeaveAst node) { + String name = null; + if (node instanceof Identifier id && lookup(id.name()) != null) { + name = lookup(id.name()).name(); + } else if (node instanceof Dollar d && dollars != null && d.level() <= dollars.size()) { + name = dollars.get(d.level() - 1); + } + return name != null && rawNames.contains(name) ? name : null; + } + // A date or time: a date literal, now(), a coercion to a date or time type, or a variable with one private boolean isDateValue(DataWeaveAst node) { DataWeaveAst n = unwrap(node); @@ -554,6 +627,9 @@ public class DataWeaveConverter { // -- Objects private String emitObject(ObjectLit obj) { + if (xmlOutput && (obj.entries().stream().anyMatch(e -> e.key() == null) || hasRepeatedKey(obj))) { + return emitXmlObject(obj); + } // fields are emitted as object literals, and an object spread (expr) is merged in between List<String> parts = new ArrayList<>(); List<String> fields = new ArrayList<>(); @@ -578,6 +654,52 @@ public class DataWeaveConverter { return parts.size() == 1 ? parts.get(0) : "(" + String.join(" + ", parts) + ")"; } + // An object written as XML: a key that repeats (also by object spreads) is an element that repeats, so its parts + // (fields, and object spreads) are merged by dw.xmlObject + private String emitXmlObject(ObjectLit obj) { + List<String> parts = new ArrayList<>(); + List<String> fields = new ArrayList<>(); + Set<String> names = new HashSet<>(); + for (ObjectEntry entry : obj.entries()) { + String name = entry.key() != null && !entry.dynamic() ? keyName(entry.key()) : null; + if (entry.key() == null || name != null && names.contains(name)) { + if (!fields.isEmpty()) { + parts.add(objectLiteral(fields)); + fields = new ArrayList<>(); + names.clear(); + } + } + if (entry.key() == null) { + String spread = emit(entry.value()); + parts.add(entry.condition() != null + ? "(if " + emit(entry.condition()) + " then " + spread + " else null)" : spread); + } else { + if (name != null) { + names.add(name); + } + fields.add(emitField(entry)); + } + } + if (!fields.isEmpty()) { + parts.add(objectLiteral(fields)); + } + return lib("xmlObject") + "([" + String.join(", ", parts) + "])"; + } + + private static boolean hasRepeatedKey(ObjectLit obj) { + Set<String> names = new HashSet<>(); + for (ObjectEntry entry : obj.entries()) { + if (entry.key() != null && !entry.dynamic() && !names.add(keyName(entry.key()))) { + return true; + } + } + return false; + } + + private static String keyName(DataWeaveAst key) { + return key instanceof QName q ? q.prefix() + "#" + q.name() : ((StringLit) key).value(); + } + private static String objectLiteral(List<String> fields) { if (fields.isEmpty()) { return "{}"; @@ -587,22 +709,42 @@ public class DataWeaveConverter { private String emitField(ObjectEntry entry) { String key; + String name = null; if (entry.dynamic()) { DataWeaveAst k = entry.key(); key = k instanceof StringLit || k instanceof Interpolation ? emit(k) : lib("str") + "(" + emit(k) + ")"; } else { - key = string(((StringLit) entry.key()).value()); + name = staticName(entry.key()); + key = string(name); } String value = emit(entry.value()); + if (!entry.attributes().isEmpty() && xmlOutput) { + // attributes are only written in XML (DataWeave leaves them out in the other formats) + List<String> attributes = new ArrayList<>(); + for (ObjectEntry attribute : entry.attributes()) { + attributes.add(emitField(attribute)); + } + value = lib("withAttributes") + "(" + value + ", { " + String.join(", ", attributes) + " }" + + (skipNullAttributes ? ", true" : "") + ")"; + } + String todo = entry.key() instanceof QName q && xmlOutput && !namespaces.containsKey(q.prefix()) + ? todoComment("undeclared namespace prefix", q.prefix() + "#" + q.name()) : ""; if (entry.condition() != null) { // a field with a null name is left out - return "[if " + emit(entry.condition()) + " then " + key + " else null]: " + value; + return todo + "[if " + emit(entry.condition()) + " then " + key + " else null]: " + value; } if (entry.dynamic()) { return "[" + key + "]: " + value; } - String name = ((StringLit) entry.key()).value(); - return (isJsonnetIdentifier(name) ? name : key) + ": " + value; + return todo + (isJsonnetIdentifier(name) ? name : key) + ": " + value; + } + + // The name of a static key: a name, or prefix#name (written with the prefix in XML only, as DataWeave does) + private String staticName(DataWeaveAst key) { + if (key instanceof QName q) { + return xmlOutput ? q.prefix() + ":" + q.name() : q.name(); + } + return ((StringLit) key).value(); } private static boolean isJsonnetIdentifier(String name) { @@ -854,10 +996,11 @@ public class DataWeaveConverter { return wrongArguments(name, args); } List<String> emitted = new ArrayList<>(); + boolean rawItems = functionArg != null && readsItemAttributes(args.get(functionArg.position())); for (int i = 0; i < args.size(); i++) { if (functionArg != null && i == functionArg.position()) { - emitted.add(emitFunctionArg(args.get(i), functionArg.params())); - } else if (functionArg != null && i == 0 && readsItemAttributes(args.get(functionArg.position()))) { + emitted.add(emitFunctionArg(args.get(i), functionArg.params(), rawItems)); + } else if (rawItems && i == 0) { // keep XML elements with only text as objects, as the function reads their attributes emitted.add(emitSelector(args.get(i), true)); } else { @@ -896,14 +1039,19 @@ public class DataWeaveConverter { // A function argument of a higher-order function, which DataWeave calls with params.size() arguments private String emitFunctionArg(DataWeaveAst arg, List<String> params) { + return emitFunctionArg(arg, params, false); + } + + // rawItem: the first parameter is an XML element kept as an object, because the function reads its attributes + private String emitFunctionArg(DataWeaveAst arg, List<String> params, boolean rawItem) { if (arg instanceof Lambda lam) { if (lam.params().size() > params.size()) { return todo("lambda with " + lam.params().size() + " parameters", "expected at most " + params.size()); } - return emitLambda(lam, params.size()); + return emitLambda(lam, params.size(), rawItem); } if (containsDollar(arg)) { - return emitImplicitLambda(arg, params); + return emitImplicitLambda(arg, params, rawItem); } if (arg instanceof Identifier id && lookup(id.name()) != null && lookup(id.name()).arity() >= 0) { // a function by name: pass the arguments it accepts @@ -924,6 +1072,10 @@ public class DataWeaveConverter { // An expression with $, $$, $$$ (such as payload map $.name): a function of the parameters private String emitImplicitLambda(DataWeaveAst body, List<String> params) { + return emitImplicitLambda(body, params, false); + } + + private String emitImplicitLambda(DataWeaveAst body, List<String> params, boolean rawItem) { List<String> names = freshNames(params); Map<String, Binding> scope = new HashMap<>(); for (String name : names) { @@ -932,9 +1084,15 @@ public class DataWeaveConverter { List<String> saved = dollars; dollars = names; scopes.push(scope); + if (rawItem) { + rawNames.add(names.get(0)); + } try { return "function(" + String.join(", ", names) + ") " + emit(body); } finally { + if (rawItem) { + rawNames.remove(names.get(0)); + } scopes.pop(); dollars = saved; } @@ -942,6 +1100,10 @@ public class DataWeaveConverter { // A lambda with at least the given number of parameters (DataWeave passes arguments a lambda does not declare) private String emitLambda(Lambda lam, int arity) { + return emitLambda(lam, arity, false); + } + + private String emitLambda(Lambda lam, int arity, boolean rawFirst) { Map<String, Binding> scope = new HashMap<>(); List<String> params = new ArrayList<>(); Set<String> used = new HashSet<>(); @@ -965,9 +1127,16 @@ public class DataWeaveConverter { List<String> saved = dollars; dollars = null; scopes.push(scope); + String raw = rawFirst && !lam.params().isEmpty() ? jsonnetName(lam.params().get(0).name()) : null; + if (raw != null) { + rawNames.add(raw); + } try { return "function(" + String.join(", ", params) + ") " + emit(lam.body()); } finally { + if (raw != null) { + rawNames.remove(raw); + } scopes.pop(); dollars = saved; } @@ -987,17 +1156,23 @@ public class DataWeaveConverter { return names; } - // Whether a function argument reads an attribute of its (first) parameter: (e) -> e.@id, or $.@id + // Whether a function argument reads the attributes of its (first) parameter: (e) -> e.@id, $.@id or $.@ private static boolean readsItemAttributes(DataWeaveAst fn) { + Predicate<DataWeaveAst> item; + DataWeaveAst body; if (fn instanceof Lambda lam) { if (lam.params().isEmpty()) { return false; } String name = lam.params().get(0).name(); - return anyNode(lam.body(), n -> n instanceof AttributeAccess aa && aa.object() instanceof Identifier id - && id.name().equals(name)); + item = n -> n instanceof Identifier id && id.name().equals(name); + body = lam.body(); + } else { + item = n -> n instanceof Dollar d && d.level() == 1; + body = fn; } - return anyNode(fn, n -> n instanceof AttributeAccess aa && aa.object() instanceof Dollar d && d.level() == 1); + return anyNode(body, n -> n instanceof AttributeAccess aa && item.test(aa.object()) + || n instanceof AllAttributes all && item.test(all.object())); } private static boolean anyNode(DataWeaveAst node, Predicate<DataWeaveAst> predicate) { diff --git a/components/camel-dataweave/src/main/java/org/apache/camel/component/dataweave/DataWeaveLexer.java b/components/camel-dataweave/src/main/java/org/apache/camel/component/dataweave/DataWeaveLexer.java index 72b40e9d76d9..015ee3335295 100644 --- a/components/camel-dataweave/src/main/java/org/apache/camel/component/dataweave/DataWeaveLexer.java +++ b/components/camel-dataweave/src/main/java/org/apache/camel/component/dataweave/DataWeaveLexer.java @@ -19,6 +19,8 @@ package org.apache.camel.component.dataweave; import java.util.ArrayList; import java.util.List; import java.util.Set; +import java.util.regex.Matcher; +import java.util.regex.Pattern; /** * Tokenizer for DataWeave 2.0 scripts. @@ -83,6 +85,10 @@ public class DataWeaveLexer { } } + private static final Pattern HEADER_SEPARATOR_LINE = Pattern.compile("(?m)^[ \\t]*---[ \\t]*\\r?$"); + private static final Pattern NAMESPACE_DIRECTIVE + = Pattern.compile("ns[ \\t]+([A-Za-z_][A-Za-z0-9_]*)[ \\t]+(\\S+)[ \\t]*(//.*)?\\r?"); + // A '/' after one of these identifiers starts a regular expression literal rather than a division private static final Set<String> REGEX_PREFIX_IDENTIFIERS = Set.of( "replace", "contains", "matches", "splitBy", "scan", "find", "match", "case", "startsWith", "endsWith"); @@ -92,6 +98,8 @@ public class DataWeaveLexer { private int pos; private int line; private int col; + // whether the --- separator of the header was read + private boolean body; public DataWeaveLexer(String input) { this.input = input; @@ -106,12 +114,40 @@ public class DataWeaveLexer { if (pos >= input.length()) { break; } - tokens.add(readToken()); + if (!readNamespaceDirective()) { + Token token = readToken(); + body |= token.type() == TokenType.HEADER_SEPARATOR; + tokens.add(token); + } } tokens.add(new Token(TokenType.EOF, "", line, col)); return tokens; } + // A namespace declaration of the header, "ns prefix uri" on a line of its own before the --- separator: the URI is + // read as a string (it has characters such as // that start a comment elsewhere) + private boolean readNamespaceDirective() { + if (body || !input.startsWith("ns", pos) || !tokens.isEmpty() && tokens.get(tokens.size() - 1).line() == line + || !HEADER_SEPARATOR_LINE.matcher(input).region(pos, input.length()).find()) { + return false; + } + int end = input.indexOf('\n', pos); + String rest = input.substring(pos, end < 0 ? input.length() : end); + Matcher m = NAMESPACE_DIRECTIVE.matcher(rest); + if (!m.matches()) { + return false; + } + int startLine = line; + int startCol = col; + tokens.add(token(TokenType.IDENTIFIER, "ns", startLine, startCol)); + tokens.add(token(TokenType.IDENTIFIER, m.group(1), startLine, startCol + m.start(1))); + tokens.add(token(TokenType.STRING, m.group(2), startLine, startCol + m.start(2))); + for (int i = 0; i < rest.length(); i++) { + advance(); + } + return true; + } + private void skipWhitespaceAndComments() { while (pos < input.length()) { char c = input.charAt(pos); diff --git a/components/camel-dataweave/src/main/java/org/apache/camel/component/dataweave/DataWeaveParser.java b/components/camel-dataweave/src/main/java/org/apache/camel/component/dataweave/DataWeaveParser.java index 74c148495047..cee119a1e277 100644 --- a/components/camel-dataweave/src/main/java/org/apache/camel/component/dataweave/DataWeaveParser.java +++ b/components/camel-dataweave/src/main/java/org/apache/camel/component/dataweave/DataWeaveParser.java @@ -22,6 +22,7 @@ import java.util.List; import java.util.Map; import java.util.Set; +import org.apache.camel.component.dataweave.DataWeaveAst.AllAttributes; import org.apache.camel.component.dataweave.DataWeaveAst.ArrayLit; import org.apache.camel.component.dataweave.DataWeaveAst.AttributeAccess; import org.apache.camel.component.dataweave.DataWeaveAst.BinaryOp; @@ -51,6 +52,8 @@ import org.apache.camel.component.dataweave.DataWeaveAst.NumberLit; import org.apache.camel.component.dataweave.DataWeaveAst.ObjectEntry; import org.apache.camel.component.dataweave.DataWeaveAst.ObjectLit; import org.apache.camel.component.dataweave.DataWeaveAst.Parens; +import org.apache.camel.component.dataweave.DataWeaveAst.QName; +import org.apache.camel.component.dataweave.DataWeaveAst.QualifiedFieldAccess; import org.apache.camel.component.dataweave.DataWeaveAst.Range; import org.apache.camel.component.dataweave.DataWeaveAst.RegexLit; import org.apache.camel.component.dataweave.DataWeaveAst.Script; @@ -97,7 +100,7 @@ public class DataWeaveParser { * Parses a complete script: an optional header (directives and declarations ending with {@code ---}) and the body. */ public DataWeaveAst parse() { - Header header = new Header("2.0", null, Map.of(), List.of(), List.of()); + Header header = new Header("2.0", null, Map.of(), List.of(), List.of(), Map.of()); List<DataWeaveAst> declarations = new ArrayList<>(); if (hasHeaderSeparator()) { header = parseHeader(declarations); @@ -146,6 +149,7 @@ public class DataWeaveParser { Map<String, String> outputProperties = new LinkedHashMap<>(); List<InputDecl> inputs = new ArrayList<>(); List<String> imports = new ArrayList<>(); + Map<String, String> namespaces = new LinkedHashMap<>(); while (!check(TokenType.HEADER_SEPARATOR)) { Token start = current(); @@ -169,12 +173,10 @@ public class DataWeaveParser { } else if (checkIdentifier("import")) { imports.add(parseImport()); } else if (checkIdentifier("ns")) { - StringBuilder text = new StringBuilder(); - while (current().line() == start.line() && !check(TokenType.EOF)) { - text.append(current().value()).append(' '); - advance(); - } - declarations.add(new Unsupported(text.toString().trim(), "XML namespace declaration")); + // ns prefix uri (the lexer reads the URI as a string) + advance(); + String prefix = expectName(); + namespaces.put(prefix, expect(TokenType.STRING).value()); } else if (checkIdentifier("type")) { skipTypeDeclaration(); } else if (checkIdentifier("var") || checkIdentifier("fun")) { @@ -184,7 +186,7 @@ public class DataWeaveParser { } } advance(); // --- - return new Header(version, outputType, outputProperties, inputs, imports); + return new Header(version, outputType, outputProperties, inputs, imports, namespaces); } private String parseMediaType() { @@ -553,9 +555,17 @@ public class DataWeaveParser { if (check(TokenType.DOT) && isTokenAhead(1, TokenType.DOT)) { advance(); // . advance(); // . - if (check(TokenType.STAR) || check(TokenType.AT)) { - expr = new Unsupported(".." + current().value(), "descendants selector ..* / ..@"); - advance(); + if (check(TokenType.STAR)) { + advance(); // * + expr = new DescendantSelector(expr, expectSelectorName(), true); + } else if (check(TokenType.AT)) { + advance(); // @ + String name = ""; + if (check(TokenType.IDENTIFIER)) { + name = current().value(); + advance(); + } + expr = new Unsupported("..@" + name, "descendants selector ..@"); } else { expr = new DescendantSelector(expr, expectSelectorName()); } @@ -597,7 +607,7 @@ public class DataWeaveParser { if (check(TokenType.IDENTIFIER) || check(TokenType.STRING)) { return new AttributeAccess(expr, expectSelectorName()); } - return new Unsupported(".@", "all attributes selector .@"); + return new AllAttributes(expr); } if (check(TokenType.STAR)) { advance(); // * @@ -619,7 +629,7 @@ public class DataWeaveParser { if (check(TokenType.IDENTIFIER) && isTokenAhead(1, TokenType.HASH)) { String prefix = expectName(); advance(); // # - return new Unsupported("." + prefix + "#" + expectSelectorName(), "XML namespace selector"); + return new QualifiedFieldAccess(expr, prefix, expectSelectorName()); } return new FieldAccess(expr, expectSelectorName()); } @@ -791,7 +801,7 @@ public class DataWeaveParser { advance(); // ) DataWeaveAst condition = parseEntryCondition(); for (ObjectEntry entry : group) { - entries.add(new ObjectEntry(entry.key(), entry.value(), entry.dynamic(), condition)); + entries.add(new ObjectEntry(entry.key(), entry.value(), entry.dynamic(), condition, entry.attributes())); } } else if (check(TokenType.LPAREN)) { // (expr): value is a dynamic key, and (expr) alone an object spread @@ -799,9 +809,9 @@ public class DataWeaveParser { DataWeaveAst expr = parseExpression(); expect(TokenType.RPAREN); if (check(TokenType.AT)) { - entries.add(unsupportedEntry(skipAttributes(), "XML attributes")); + List<ObjectEntry> attributes = parseAttributes(); expect(TokenType.COLON); - parseExpression(); + entries.add(new ObjectEntry(expr, parseExpression(), true, null, attributes)); } else if (check(TokenType.COLON)) { advance(); entries.add(new ObjectEntry(expr, parseExpression(), true, null)); @@ -856,45 +866,33 @@ public class DataWeaveParser { } else { key = new StringLit(expectName()); } - if (check(TokenType.HASH)) { + if (check(TokenType.HASH) && !dynamic && key instanceof StringLit prefix) { + // prefix#name: a name in an XML namespace advance(); // # - String name = expectName(); - expect(TokenType.COLON); - parseExpression(); - return unsupportedEntry(token.value() + "#" + name, "XML namespace prefix"); - } - if (check(TokenType.AT)) { - String attributes = skipAttributes(); - expect(TokenType.COLON); - parseExpression(); - return unsupportedEntry(token.value() + " " + attributes, "XML attributes"); + key = new QName(prefix.value(), expectName()); } + List<ObjectEntry> attributes = check(TokenType.AT) ? parseAttributes() : List.of(); expect(TokenType.COLON); - return new ObjectEntry(key, parseExpression(), dynamic, null); - } - - private static ObjectEntry unsupportedEntry(String text, String reason) { - return new ObjectEntry(null, new Unsupported(text, reason), false, null); + return new ObjectEntry(key, parseExpression(), dynamic, null, attributes); } - private String skipAttributes() { + // The XML attributes of a key: @(name: value, ...) + private List<ObjectEntry> parseAttributes() { advance(); // @ - StringBuilder text = new StringBuilder("@"); - if (check(TokenType.LPAREN)) { - int depth = 0; - do { - if (check(TokenType.LPAREN)) { - depth++; - } else if (check(TokenType.RPAREN)) { - depth--; - } else if (check(TokenType.EOF)) { - throw error("unterminated attributes"); - } - text.append(current().value()); - advance(); - } while (depth > 0); + expect(TokenType.LPAREN); + List<ObjectEntry> attributes = new ArrayList<>(); + while (!check(TokenType.RPAREN)) { + ObjectEntry attribute = parseObjectEntry(); + if (!attribute.attributes().isEmpty()) { + throw error("an attribute has no attributes"); + } + attributes.add(attribute); + if (!check(TokenType.RPAREN)) { + expect(TokenType.COMMA); + } } - return text.toString(); + advance(); // ) + return attributes; } private DataWeaveAst parseArray() { diff --git a/components/camel-dataweave/src/test/java/org/apache/camel/component/dataweave/DataWeaveConverterTest.java b/components/camel-dataweave/src/test/java/org/apache/camel/component/dataweave/DataWeaveConverterTest.java index 086b3004e9c9..3b321c6dbbaf 100644 --- a/components/camel-dataweave/src/test/java/org/apache/camel/component/dataweave/DataWeaveConverterTest.java +++ b/components/camel-dataweave/src/test/java/org/apache/camel/component/dataweave/DataWeaveConverterTest.java @@ -66,9 +66,11 @@ class DataWeaveConverterTest { --- { a: 1 } """); - assertEquals("/** DataSonnet\nversion=2.0\noutput application/json\ninput payload application/xml\n*/\n{\n a: 1\n}", + // an XML input is written as DataWeave writes it (dw.output) + assertEquals("/** DataSonnet\nversion=2.0\noutput application/json\ninput payload application/xml\n*/\n" + DW_IMPORT + + "dw.output({\n a: 1\n}, body)", result); - assertFalse(converter.needsDataWeaveLib()); + assertTrue(converter.needsDataWeaveLib()); assertEquals(0, converter.getTodoCount()); } @@ -81,7 +83,7 @@ class DataWeaveConverterTest { @Test void testSkipNullOn() { String result = converter.convert("%dw 2.0\noutput application/json skipNullOn=\"everywhere\", indent=false\n---\n1"); - assertTrue(result.endsWith("dw.skipNulls(1, \"everywhere\")"), result); + assertTrue(result.endsWith("dw.output(dw.skipNulls(1, \"everywhere\"), body)"), result); assertEquals(0, converter.getTodoCount()); } @@ -283,6 +285,105 @@ class DataWeaveConverterTest { assertEquals("{}", expr("{}")); } + // -- XML + + @Test + void testXmlOutput() { + String result = converter.convert(""" + %dw 2.0 + output application/xml writeDeclaration=false, indent=false + ns ns0 http://example.com/order + --- + { ns0#order @(id: payload.id): { item: payload.sku } } + """); + assertEquals(""" + /** DataSonnet + version=2.0 + output application/xml; nullasemptyelement=true; omitxmldeclaration=true + */ + """ + DW_IMPORT + """ + dw.xmlOutput({ + "ns0:order": dw.withAttributes({ + item: dw.sel(body, "sku") + }, { id: dw.sel(body, "id") }) + }, { ns0: "http://example.com/order"; })""", result); + assertEquals(0, converter.getTodoCount()); + } + + @Test + void testXmlOutputSkipsNullAttributes() { + String result + = converter.convert("%dw 2.0\noutput application/xml skipNullOn=\"attributes\"\n---\n{ a @(x: null): 1 }"); + assertTrue(result.endsWith("dw.xmlOutput({\n a: dw.withAttributes(1, { x: null }, true)\n}, {})"), result); + result = converter.convert("%dw 2.0\noutput application/xml skipNullOn=\"everywhere\"\n---\n{ a @(x: null): 1 }"); + assertTrue(result.endsWith( + "dw.xmlOutput(dw.skipNulls({\n a: dw.withAttributes(1, { x: null }, true)\n}, \"everywhere\"), {})"), + result); + assertEquals(0, converter.getTodoCount()); + } + + @Test + void testXmlObjectWithRepeatedKeys() { + // in XML a key that repeats is an element that repeats: the fields and object spreads are merged by dw.xmlObject + String result = converter.convert("%dw 2.0\noutput application/xml\n---\n{ root: { a: 1, (payload.items), a: 2 } }"); + assertTrue(result.endsWith("dw.xmlOutput({\n root: dw.xmlObject([{\n a: 1\n }, dw.sel(body, \"items\"), {\n" + + " a: 2\n }])\n}, {})"), + result); + assertEquals(0, converter.getTodoCount()); + } + + @Test + void testXmlKeysInOtherOutputs() { + // attributes and namespace prefixes are only written in XML + assertTrue(converter.convert("%dw 2.0\noutput application/json\n---\n{ a @(x: 1): 2, o#b: 3 }") + .endsWith("dw.output({\n a: 2,\n b: 3\n}, body)")); + assertEquals(0, converter.getTodoCount()); + } + + @Test + void testNamespaceDirectiveOnlyInHeader() { + // a line "ns a b" in the body is an expression (ns is a variable) + String result = converter.convert("%dw 2.0\noutput application/json\nvar ns = [1]\n---\nns map $"); + assertTrue(result.endsWith("dw.output(local ns = [1];\ndw.map(ns, function(item, index) item), body)"), result); + assertEquals(0, converter.getTodoCount()); + } + + @Test + void testUndeclaredNamespacePrefixIsTodo() { + String result = converter.convert("%dw 2.0\noutput application/xml\n---\n{ x#a: 1 }"); + assertTrue(result.contains("/* TODO: manual conversion needed -- undeclared namespace prefix: x#a */"), result); + assertEquals(1, converter.getTodoCount()); + converter.convert("%dw 2.0\noutput application/json\n---\npayload.x#a"); + assertEquals(1, converter.getTodoCount()); + } + + @Test + void testXmlSelectors() { + String result = converter.convert(""" + %dw 2.0 + output application/json + ns o http://example.com/orders + --- + [payload.o#order.@, payload..*item, payload.order.o#item] + """); + assertTrue(result.endsWith("dw.output([dw.attrs(dw.selNsRaw(body, \"http://example.com/orders\";, \"order\")), " + + "dw.descAll(body, \"item\"), dw.selNs(dw.selRaw(body, \"order\"), " + + "\"http://example.com/orders\";, \"item\")], body)"), + result); + assertEquals(0, converter.getTodoCount()); + } + + @Test + void testAttributesOfLambdaItems() { + // the items are kept as XML elements for their attributes, and used as their text otherwise + assertEquals("dw.map(dw.multiRaw(body, \"order\"), function(item, index) {\n id: dw.attr(item, \"id\"),\n" + + " text: dw.text(item)\n})", + expr("payload.*order map { id: $.@id, text: $ }")); + assertEquals("dw.map(dw.multiRaw(body, \"order\"), function(o, _1) {\n all: dw.attrs(o),\n v: dw.text(o)\n})", + expr("payload.*order map ((o) -> { all: o.@, v: o })")); + assertEquals(0, converter.getTodoCount()); + } + // -- Match @Test @@ -363,7 +464,7 @@ class DataWeaveConverterTest { void testUnsupportedConstructs() { assertTrue(converter.convertExpression("payload update { case .a -> 1 }").contains("update operator")); assertEquals(1, converter.getTodoCount()); - converter.convertExpression("{ a @(id: 1): 2 }"); + converter.convertExpression("payload..@id"); assertEquals(1, converter.getTodoCount()); } diff --git a/components/camel-dataweave/src/test/java/org/apache/camel/component/dataweave/DataWeaveParserTest.java b/components/camel-dataweave/src/test/java/org/apache/camel/component/dataweave/DataWeaveParserTest.java index 981bdee5faae..aa522dfeb61a 100644 --- a/components/camel-dataweave/src/test/java/org/apache/camel/component/dataweave/DataWeaveParserTest.java +++ b/components/camel-dataweave/src/test/java/org/apache/camel/component/dataweave/DataWeaveParserTest.java @@ -19,6 +19,7 @@ package org.apache.camel.component.dataweave; import java.util.List; import java.util.Map; +import org.apache.camel.component.dataweave.DataWeaveAst.AllAttributes; import org.apache.camel.component.dataweave.DataWeaveAst.ArrayLit; import org.apache.camel.component.dataweave.DataWeaveAst.AttributeAccess; import org.apache.camel.component.dataweave.DataWeaveAst.BinaryOp; @@ -42,6 +43,8 @@ import org.apache.camel.component.dataweave.DataWeaveAst.NumberLit; import org.apache.camel.component.dataweave.DataWeaveAst.ObjectEntry; import org.apache.camel.component.dataweave.DataWeaveAst.ObjectLit; import org.apache.camel.component.dataweave.DataWeaveAst.Parens; +import org.apache.camel.component.dataweave.DataWeaveAst.QName; +import org.apache.camel.component.dataweave.DataWeaveAst.QualifiedFieldAccess; import org.apache.camel.component.dataweave.DataWeaveAst.Range; import org.apache.camel.component.dataweave.DataWeaveAst.RegexLit; import org.apache.camel.component.dataweave.DataWeaveAst.Script; @@ -120,10 +123,11 @@ class DataWeaveParserTest { } @Test - void shouldRepresentNamespaceDeclarationAsUnsupported() { - Script script = parseScript("%dw 2.0\nns ns0 http://example.com\n---\npayload";); - Block block = assertInstanceOf(Block.class, script.body()); - assertInstanceOf(Unsupported.class, block.declarations().get(0)); + void shouldParseNamespaceDeclaration() { + Script script + = parseScript("%dw 2.0\nns ns0 http://example.com/a?b=1 // the orders\noutput application/xml\n---\npayload"); + assertEquals(Map.of("ns0", "http://example.com/a?b=1";), script.header().namespaces()); + assertEquals("payload", name(script.body())); } @Test @@ -287,6 +291,13 @@ class DataWeaveParserTest { assertEquals("id", assertInstanceOf(AttributeAccess.class, parseExpr("payload.order.@id")).attribute()); assertEquals("item", assertInstanceOf(MultiValueSelector.class, parseExpr("payload.*item")).field()); assertEquals("sku", assertInstanceOf(DescendantSelector.class, parseExpr("payload..sku")).field()); + assertTrue(assertInstanceOf(DescendantSelector.class, parseExpr("payload..*sku")).multi()); + assertInstanceOf(Unsupported.class, parseExpr("payload..@id")); + assertEquals("order", assertInstanceOf(FieldAccess.class, + assertInstanceOf(AllAttributes.class, parseExpr("payload.order.@")).object()).field()); + QualifiedFieldAccess qualified = assertInstanceOf(QualifiedFieldAccess.class, parseExpr("payload.o#order")); + assertEquals("o", qualified.prefix()); + assertEquals("order", qualified.field()); assertInstanceOf(ExistenceCheck.class, parseExpr("payload.a?")); assertInstanceOf(FilterSelector.class, parseExpr("payload.items[?($.qty > 1)]")); Range range = assertInstanceOf(Range.class, assertInstanceOf(IndexAccess.class, parseExpr("xs[-1 to 0]")).index()); @@ -309,6 +320,21 @@ class DataWeaveParserTest { assertEquals("default", assertInstanceOf(StringLit.class, entries.get(4).key()).value()); } + @Test + void shouldParseXmlKeys() { + ObjectLit obj = assertInstanceOf(ObjectLit.class, parseExpr("{ o#a @(id: 1, o#b: x): 2, (k) @(c: 3): 4, d: 5 }")); + List<ObjectEntry> entries = obj.entries(); + QName qName = assertInstanceOf(QName.class, entries.get(0).key()); + assertEquals("o", qName.prefix()); + assertEquals("a", qName.name()); + assertEquals(2, entries.get(0).attributes().size()); + assertEquals("id", assertInstanceOf(StringLit.class, entries.get(0).attributes().get(0).key()).value()); + assertInstanceOf(QName.class, entries.get(0).attributes().get(1).key()); + assertTrue(entries.get(1).dynamic()); + assertEquals(1, entries.get(1).attributes().size()); + assertTrue(entries.get(2).attributes().isEmpty()); + } + @Test void shouldParseArrayLiteral() { ArrayLit arr = assertInstanceOf(ArrayLit.class, parseExpr("[1, 2, 3]")); diff --git a/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_23.adoc b/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_23.adoc index c57fe1b0eac5..3e8135647cc7 100644 --- a/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_23.adoc +++ b/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_23.adoc @@ -4993,8 +4993,10 @@ DataSonnet directly, or moved to a `.ds` file. The conversion also follows the DataWeave semantics more closely, which changes the result of some scripts that were converted before. For example, a selector on a missing field now gives `null` (as in DataWeave) instead of failing, `default` applies to missing fields, and a field selector on an array -selects the field of every element. Unknown functions and variables are no longer passed through -unchanged, but fail the conversion. The converted scripts use the new `camel-dataweave.libsonnet` library of +selects the field of every element. With an XML input, a selector matches an element by its local name +(whatever its namespace prefix), an empty element is `null`, and an element written to another format than XML +no longer includes the attributes, namespace declarations and positions of the DataSonnet XML model. Unknown +functions and variables are no longer passed through unchanged, but fail the conversion. The converted scripts use the new `camel-dataweave.libsonnet` library of `camel-datasonnet`, so DataSonnet scripts generated with `camel transform dataweave` need `camel-datasonnet` 4.23 or newer.
