anishgirianish commented on issue #44146:
URL: https://github.com/apache/airflow/issues/44146#issuecomment-4225699528
@Lee-W this sketch is really helpful — the consistency across TaskFlow,
`@asset`, `@asset.multi`, and traditional operators is nice, and reusing the
existing `outlet_events` accessor feels much cleaner than a separate
`PartitionAtRuntime(name=...)` object as the AIP suggests. +1 from me on that
direction.
One thing I wanted to run by you: all four forms in your sketch set a single
partition key per run, which handles the override case beautifully (and
resolves #58474 on the way). The AIP's motivating example, though, is the
runtime fan-out — looping over a discovered set and emitting one event per
partition in a single task run:
```python
@task(outlets=[my_asset])
def process(*, outlet_events):
for customer in get_customers():
outlet_events[my_asset].add_partition(key=customer, extra={"rows":
...})
```
One direction that composes cleanly with your sketch: add an
`add_partition(key=, extra=)` method on the accessor alongside `.partition_key
= ...`. Under the hood, `OutletEventAccessor` would hold a list of pending
`(partition_key, extra)` events — assigning `.partition_key` / `.extra` mutates
a default single event, `add_partition` appends. Today's single-event behavior
is preserved as the default, and the same extension lands naturally on all four
of your forms (`self.add_partition(...)` inside a `@asset` body, etc.).
One small AIP constraint worth surfacing somewhere in the final design:
*"Runtime partitioning cannot be combined with statically defined partitioning
with `PartitionByProduct`."*
Curious whether `add_partition` on the accessor matches where you were
headed, or whether you had a different shape in mind for the fan-out case? If
this direction resonates, I'd love to take a crack at the PR and bring it back
to you for review 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
