yurivish opened a new issue, #610:
URL: https://github.com/apache/arrow-julia/issues/610
Example:
```
using Arrow
# Two independent single-batch streams, same schema, different dict pools.
# Each is written fresh, so each DictionaryBatch is "initial"
(isDelta=false) —
# exactly what a producer like pyarrow emits when per-batch pools differ.
open(io -> Arrow.write(io, (a=Arrow.DictEncode(["alpha", "beta"]),)),
"s1.arrows", "w")
open(io -> Arrow.write(io, (a=Arrow.DictEncode(["gamma", "delta"]),)),
"s2.arrows", "w")
# Splice: [schema, dict1, rb1] from s1 + [dict2, rb2] from s2 + EOS.
# This reproduces a foreign stream whose 2nd DictionaryBatch replaces dict
id 0.
function messages(bytes)
spans, pos = UnitRange{Int}[], 1
while pos + 3 <= length(bytes) && Arrow.readbuffer(bytes, pos, UInt32)
== Arrow.CONTINUATION_INDICATOR_BYTES
pos += 4
msglen = Arrow.readbuffer(bytes, pos, Int32)
msglen == 0 && break
pos += 4
msg = Arrow.FlatBuffers.getrootas(Arrow.Meta.Message, bytes, pos - 1)
start = pos - 8
pos += msglen + msg.bodyLength
push!(spans, start:(pos - 1))
end
spans
end
b1, b2 = read("s1.arrows"), read("s2.arrows")
s1, s2 = messages(b1), messages(b2)
eos = reinterpret(UInt8, [Arrow.CONTINUATION_INDICATOR_BYTES, UInt32(0)])
write("combined.arrows", vcat(b1[s1[1]], b1[s1[2]], b1[s1[3]], b2[s2[2]],
b2[s2[3]], eos))
println(collect(Arrow.Table("combined.arrows").a)) #
["gamma","delta","gamma","delta"] -- WRONG
println([collect(b.a) for b in Arrow.Stream("combined.arrows")]) #
[["alpha","beta"],["gamma","delta"]] -- correct
```
The `pyarrow` version behaves correctly in this case:
```python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
