zn-danielbariudin commented on PR #4384:
URL: https://github.com/apache/iceberg-python/pull/4384#issuecomment-6945897805

   Hey @Fokko!
   Fair concerns, so I profiled both on a table with 53 columns and 3 partition 
fields (day, identity, bucket), 150 manifests:
   
   | | Result |
   |---|---|
   | Avro header reads / distinct schema strings | 151 / 2 |
   | Schema string length | 4,256 chars |
   | Hash of a freshly decoded string | 0.71 µs |
   | Cache hit (hash + equality check) | 0.90 µs |
   | Conversion without the cache (`json.loads` + `avro_to_iceberg`) | 244 µs |
   | Memory retained per cache entry (`Schema` + key string) | 47 KiB |
   | `plan_files()` peak memory, without / with cache | 2.92 / 2.95 MiB |
   
   **Hashing**: hashing the schema string does cost something, but ~0.9 µs per 
lookup vs ~244 µs for the conversion it replaces. The numbers use a fresh `str` 
per lookup, since each header read decodes a new string with no precomputed 
hash.
   
   **Memory**: the manifest schema depends on the partition spec, not on the 
table's columns, so a scan only sees a couple of distinct strings. Peak memory 
of `plan_files()` is unchanged.
   
   Some examples of what this means:
   
   - **This run (150 manifests)**: without the cache, 151 × 244 µs ≈ **36.8 
ms** of conversion. With it, 2 conversions + 149 lookups ≈ **0.6 ms**, for 
**~94 KiB** retained.
   - **1,000 manifests**: ≈ **244 ms → 1.4 ms**, same ~94 KiB, since it's still 
2 distinct strings.
   - **A service reading 10 tables with different partition specs**: ~11 
distinct strings ≈ **0.5 MiB** retained, regardless of how many scans it runs.
   - **Worst case**: bounded by the `lru_cache` default of 128 entries ≈ **5.9 
MiB**, which needs 128+ distinct manifest schemas in one process.
   
   If you'd prefer a tighter bound, I'm happy to set an explicit `maxsize`, 
e.g. `maxsize=32` caps it at ~1.5 MiB while still covering typical workloads. 
Let me know what you prefer.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to