mneha05 opened a new pull request, #2192: URL: https://github.com/apache/libcloud/pull/2192
## Fix excessive memory usage in local storage range downloads ### Description Fixes #2191. The local driver's range-stream method reads the entire source file to determine its size, then yields the whole requested range as one chunk. Even a tiny range therefore allocates memory proportional to the source size, and `chunk_size` is ignored. The file-download method then buffers the complete range before writing. Use `os.fstat` on the open descriptor for the size, seek directly to the requested offset, and read at most the requested chunk size and remaining range length. Write each chunk to the destination as it arrives. Preserve non-inclusive end offsets and existing range validation. Use the standard default chunk size when omitted or zero, reject negative chunk sizes, and stop on early EOF. Add five regression test methods covering bounded/default chunks, exact range reads, incremental writes, negative chunk sizes, and early EOF. All five fail against the original implementation. Add a storage changelog entry. A local `tracemalloc` experiment requested 1 KiB from a 64 MiB sparse file with `chunk_size=128`: | Implementation | Peak traced Python allocation | Output | | --- | ---: | --- | | Original | 67,113,884 bytes | One 1,024-byte chunk | | Fixed | 6,242 bytes | Eight 128-byte chunks | These are allocations for this single experiment, not total process memory or a general benchmark. ### Status Done, ready for review. Local checks pass; upstream CI has not yet run. Validation on Python 3.12.14 / Linux: - `pytest -q libcloud/test/storage libcloud/test/test_utils.py --ignore=libcloud/test/storage/test_list_objects_filtering_performance.py`: **982 passed, 12 skipped**, with six additional passing subtests. Existing deprecation warnings remain. - All applicable prek hooks for the three changed files pass, including Black, isort, flake8, pyupgrade, and codespell. - `git diff --check` passes. ### Checklist - [x] [Code linting](https://libcloud.readthedocs.io/en/latest/development.html#code-style-guide) - [x] Documentation — storage changelog entry - [x] [Tests](https://libcloud.readthedocs.io/en/latest/testing.html) - [ ] [ICLA](https://libcloud.readthedocs.io/en/latest/development.html#contributing-bigger-changes) (required for bigger changes) ### AI assistance Generated-by: OpenAI Codex (GPT-6) Codex assisted with investigation, implementation, regression tests, local validation, and this description. This assistance is disclosed following the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
