cgpoh commented on PR #12543: URL: https://github.com/apache/iceberg/pull/12543#issuecomment-2803679796
> @wypoon , the motivation for this PR is when I’m trying to capture data lineage using DataHub in spark streaming mode. In the DataHub [code](https://github.com/datahub-project/datahub/blob/master/metadata-integration/java/acryl-spark-lineage/src/main/java/datahub/spark/converter/SparkStreamingEventToDatahub.java#L145), the regular expression matcher is looking for e.g. `Kafka[…]` to extract the data platform for DataHub to show lineages. Without the prefix `iceberg[`, DataHub will return empty lineage. I’m wondering whether to change the code at DataHub side instead but I saw Deltalake has this `DeltaSink[` prefix to identify their sink and thought it’s a good idea to make the changes in iceberg. Wdyt? Hi @manuzhang , As explained above, I'm using DataHub acryl-spark-lineage library to capture the lineage when starting a spark job. Without the prefix `iceberg[`, DataHub will return empty lineage. AFAIK, Iceberg catalog doesn't have any information on runtime lineage. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
