alamb commented on issue #8777: URL: https://github.com/apache/datafusion/issues/8777#issuecomment-5209646498
You can probably work around this limitation by implementing some sort of
"buffer" node that saves the intermediate result (the output to be shared) and
then implement "datasource" nodes that read from that intermediate result
So the ExecutionPlan would something like this:
```text
plan as DataFusion sees it (a tree)
┌──────────────────────────┐
│ UnionExec │
│ │
└──────────────────────────┘
▲ ▲
┌──────────────┘ └───────────────┐
│ │
│ │
┌──────────────────────────┐ ┌──────────────────────────┐
│ Aggregate │ │ AggregateExec │
│ │ │ │
└──────────────────────────┘ └──────────────────────────┘
▲ ▲
│ │
│ │
┌──────────────────────────┐ ┌──────────────────────────┐
│ BufferReader (new node) │ │ BufferReader (new node) │
│ │ │ │
└──────────────────────────┘ └──────────────────────────┘
▲
│
│
┌──────────────────────────┐
│ Buffer (new node) │
│ │
└──────────────────────────┘
▲
│
│
┌──────────────────────────┐
│ JoinExec │
│ │
└──────────────────────────┘
▲
│
│
│
┌──────────────────────────┐
│ Input... │
│ │
└──────────────────────────┘
```
However internally, there would be some connection between Buffer and
BufferReader, similar to how dynamic predicates are used to send information
between nodes.
```text
┌──────────────────────────┐
│ UnionExec │
│ │
└──────────────────────────┘
▲ ▲
┌──────────────┘ └───────────────┐
│ │
│ │
┌──────────────────────────┐ ┌──────────────────────────┐
│ Aggregate │ │ AggregateExec │
│ │ │ │
└──────────────────────────┘ └──────────────────────────┘
▲ ▲
│ │
│ │
┌──────────────────────────┐ ┌──────────────────────────┐
│ BufferReader (new node) │ │ BufferReader (new node) │
│ │ │ │
└──────────────────────────┘ └──────────────────────────┘
▲ ▲
│
│ │
┌──────────────────────────┐
│ Buffer (new node) │ │
│ │─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
└──────────────────────────┘ This is not modeled in the input
▲ plan (it would be some sort of
│ connection DataFusion is not
│ aware of, similar to how
┌──────────────────────────┐ DynamicPredicates are modeled
│ JoinExec │
│ │
└──────────────────────────┘
▲
│
│
│
┌──────────────────────────┐
│ Input... │
│ │
└──────────────────────────┘
```
It is sort of a hack for sure, but I think it would work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
