[
https://issues.apache.org/jira/browse/THRIFT-5943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jens Geyer updated THRIFT-5943:
-------------------------------
Description:
In this area of code, there already exists an optimization where we skip
calling klass(**kwargs) on the top-level struct and instead init an empty
object then PyObject_SetAttr the properties onto it.
This was done because
Python's keyword argument matching is O(n) string comparisons per argument,
making this expensive for structs with many fields.
However, nested structs which are decoded within the C extension did not get
this same optimization, causing them to go through the slow class
initialization path.
In this change, for mutable nested structs, we create the instance up front
with a no-arg constructor (klass()) and set decoded fields directly via
PyObject_SetAttr (the same way we fast-path the top-level struct).
Immutable structs (TFrozenBase subclasses) continue to use the kwargs path
since their generated __setattr__ blocks attribute mutation.
Immutability is detected via (1) PyObject_IsSubclass against TFrozenBase with
the class reference cached in a function-local static or (2) and overridden
__setattr__.
Benchmarks show up to 3.6x speedup for deeply nested struct hierarchies, with
no impact on flat structs. I had Claude generate me some benchmarks to showcase
performance in different nested scenarios:
https://gist.github.com/markjm/e8610298d30d32869f46cf521272c85c
was:described in https://github.com/apache/thrift/pull/3377
> Improve performance of constructing deeply nested thrift structs
> ----------------------------------------------------------------
>
> Key: THRIFT-5943
> URL: https://issues.apache.org/jira/browse/THRIFT-5943
> Project: Thrift
> Issue Type: Improvement
> Components: Python - Library
> Reporter: Mark Molinaro
> Assignee: Mark Molinaro
> Priority: Minor
> Fix For: 0.24.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> In this area of code, there already exists an optimization where we skip
> calling klass(**kwargs) on the top-level struct and instead init an empty
> object then PyObject_SetAttr the properties onto it.
This was done because
> Python's keyword argument matching is O(n) string comparisons per argument,
> making this expensive for structs with many fields.
> However, nested structs which are decoded within the C extension did not get
> this same optimization, causing them to go through the slow class
> initialization path.
> In this change, for mutable nested structs, we create the instance up front
> with a no-arg constructor (klass()) and set decoded fields directly via
> PyObject_SetAttr (the same way we fast-path the top-level struct).
> Immutable structs (TFrozenBase subclasses) continue to use the kwargs path
> since their generated __setattr__ blocks attribute mutation.
> Immutability is detected via (1) PyObject_IsSubclass against TFrozenBase with
> the class reference cached in a function-local static or (2) and overridden
> __setattr__.
> Benchmarks show up to 3.6x speedup for deeply nested struct hierarchies, with
> no impact on flat structs. I had Claude generate me some benchmarks to
> showcase performance in different nested scenarios:
> https://gist.github.com/markjm/e8610298d30d32869f46cf521272c85c
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
