zn-danielbariudin commented on PR #4384: URL: https://github.com/apache/iceberg-python/pull/4384#issuecomment-6945897805
Hey @Fokko! Fair concerns, so I profiled both on a table with 53 columns and 3 partition fields (day, identity, bucket), 150 manifests: | | Result | |---|---| | Avro header reads / distinct schema strings | 151 / 2 | | Schema string length | 4,256 chars | | Hash of a freshly decoded string | 0.71 µs | | Cache hit (hash + equality check) | 0.90 µs | | Conversion without the cache (`json.loads` + `avro_to_iceberg`) | 244 µs | | Memory retained per cache entry (`Schema` + key string) | 47 KiB | | `plan_files()` peak memory, without / with cache | 2.92 / 2.95 MiB | **Hashing**: hashing the schema string does cost something, but ~0.9 µs per lookup vs ~244 µs for the conversion it replaces. The numbers use a fresh `str` per lookup, since each header read decodes a new string with no precomputed hash. **Memory**: the manifest schema depends on the partition spec, not on the table's columns, so a scan only sees a couple of distinct strings. Peak memory of `plan_files()` is unchanged. Some examples of what this means: - **This run (150 manifests)**: without the cache, 151 × 244 µs ≈ **36.8 ms** of conversion. With it, 2 conversions + 149 lookups ≈ **0.6 ms**, for **~94 KiB** retained. - **1,000 manifests**: ≈ **244 ms → 1.4 ms**, same ~94 KiB, since it's still 2 distinct strings. - **A service reading 10 tables with different partition specs**: ~11 distinct strings ≈ **0.5 MiB** retained, regardless of how many scans it runs. - **Worst case**: bounded by the `lru_cache` default of 128 entries ≈ **5.9 MiB**, which needs 128+ distinct manifest schemas in one process. If you'd prefer a tighter bound, I'm happy to set an explicit `maxsize`, e.g. `maxsize=32` caps it at ~1.5 MiB while still covering typical workloads. Let me know what you prefer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
