moomindani commented on PR #16574: URL: https://github.com/apache/iceberg/pull/16574#issuecomment-5234643885
One more data point for why filtering before the reporter matters, which I hadn't been able to quantify earlier. The OpenTelemetry metrics SDK applies a [cardinality limit per metric stream](https://opentelemetry.io/docs/specs/otel/metrics/sdk/), defaulting to 2000 attribute combinations. Beyond it, the SDK doesn't drop measurements — it folds them into one data point marked `otel.metric.overflow=true` and strips the original attributes. Totals remain correct while the per-table breakdown silently degrades, with no error and no missing metric to notice. That puts a number on the "per-table cardinality" concern from the #16250 dev@ thread: with a table-name attribute in play, more than ~2000 distinct tables is enough to lose the breakdown. Filtering at the framework layer addresses it at the source — the reports never become time series in the first place — whereas the SDK-side options are to raise the limit or drop the attribute wholesale, and the Collector-side option only exists for OTel users. To be clear about scope: this is an argument for filtering existing somewhere, not for any particular shape of it. Your two concerns are still the open questions, and I'm happy to follow your read on Q1. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
