[issue21321] itertools.islice() doesn't release reference to the source iterator when the slice is exhausted
New submission from Anton Afanasyev: This issue results in redundant memory consumption for e.g. in this case: from itertools import * def test_islice(): items, lookahead = tee(repeat(1, int(1e9))) lookahead = islice(lookahead, 10) for item in lookahead: pass for item in items: pass if __name__ == "__main__": test_islice() This demo is taken from real case where one needs to look ahead input stream before processing it. For my PC this demo stops with 'Segmentation fault' message after exhausting all PC memory, while running it with patched python consumes only 0.1% of memory till the end. When one uses pure pythonic implementation of itertools.islice() (taken from docs), the issue goes away as well, since this implementation doesn't store redundant reference to source iterator. def islice(iterable, *args): s = slice(*args) it = iter(xrange(s.start or 0, s.stop or sys.maxint, s.step or 1)) nexti = next(it) for i, element in enumerate(iterable): if i == nexti: yield element nexti = next(it) Attaching patch for this issue. Have to change '__reduce__()' method since now unpickling of exhausted 'islice()' object cannot return old source iterator. -- components: Extension Modules files: patch_3.4_8c8315bac6a8.diff keywords: patch messages: 216939 nosy: Anton.Afanasyev, rhettinger priority: normal severity: normal status: open title: itertools.islice() doesn't release reference to the source iterator when the slice is exhausted type: resource usage versions: Python 2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4 Added file: http://bugs.python.org/file34990/patch_3.4_8c8315bac6a8.diff ___ Python tracker <http://bugs.python.org/issue21321> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21321] itertools.islice() doesn't release reference to the source iterator when the slice is exhausted
Anton Afanasyev added the comment: Added patch for 2.7 version (no need to change '__reduce__()' method since it's not implemented). -- Added file: http://bugs.python.org/file34991/issue21321_2.7_e3217efa6edd.diff ___ Python tracker <http://bugs.python.org/issue21321> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21321] itertools.islice() doesn't release reference to the source iterator when the slice is exhausted
Anton Afanasyev added the comment: Hi Raymond, do you mean allocation exceptions handling should be more accurate? Attaching fixed version for 3.4 branch. -- Added file: http://bugs.python.org/file35003/issue21321_3.4_8c8315bac6a8_2.diff ___ Python tracker <http://bugs.python.org/issue21321> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21321] itertools.islice() doesn't release reference to the source iterator when the slice is exhausted
Anton Afanasyev added the comment: Hi Antoine, I have no found a way to check resource usage in test infrastructure and I don't think it could be done carefully. The only method I found to test issue is straightforward: just to check source iterator is not referenced from itertools.islice() after the latter has been exhausted: a = [random.random() for i in range(10)] before = sys.getrefcount(a) b = islice(a, 5) for i in b: pass after = sys.getrefcount(a) self.assertEqual(before, after) Attaching "issue21321_2.7_e3217efa6edd_3.diff" and "issue21321_3.4_8c8315bac6a8_3.diff" patches with this test included in "Lib/test/test_itertools.py". -- Added file: http://bugs.python.org/file35076/issue21321_2.7_e3217efa6edd_3.diff ___ Python tracker <http://bugs.python.org/issue21321> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21321] itertools.islice() doesn't release reference to the source iterator when the slice is exhausted
Changes by Anton Afanasyev : Added file: http://bugs.python.org/file35077/issue21321_3.4_8c8315bac6a8_3.diff ___ Python tracker <http://bugs.python.org/issue21321> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21321] itertools.islice() doesn't release reference to the source iterator when the slice is exhausted
Anton Afanasyev added the comment: Hi Antoine, my test works for me. It can be either >>> a = [1, 2, 3] or >>> a = iter([1, 2, 3]) , no matter: both objects will be +1 referenced after taking >>> b = islice(a, 1) . My test failed without patch and passed with one. But your test is more straightforward, thanks. Attaching patches with your test. -- ___ Python tracker <http://bugs.python.org/issue21321> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21321] itertools.islice() doesn't release reference to the source iterator when the slice is exhausted
Changes by Anton Afanasyev : Added file: http://bugs.python.org/file35087/issue21321_3.4_8c8315bac6a8_4.diff ___ Python tracker <http://bugs.python.org/issue21321> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21321] itertools.islice() doesn't release reference to the source iterator when the slice is exhausted
Changes by Anton Afanasyev : Added file: http://bugs.python.org/file35086/issue21321_2.7_e3217efa6edd_4.diff ___ Python tracker <http://bugs.python.org/issue21321> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21321] itertools.islice() doesn't release reference to the source iterator when the slice is exhausted
Anton Afanasyev added the comment: Hi Antoine, oops you are right about leaks: fixed them in new attached patch. As for testing changes in "reduce()": they are already covered by "self.pickletest(islice(range(100), *args))". Function "pickletest()" covers case for pickle dumping/loading of exhausted iterator. -- Added file: http://bugs.python.org/file35091/issue21321_3.4_8c8315bac6a8_5.diff ___ Python tracker <http://bugs.python.org/issue21321> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21321] itertools.islice() doesn't release reference to the source iterator when the slice is exhausted
Anton Afanasyev added the comment: Antoine, not sure about 2.7. The issue first arose for me at Python 2.7, so I would prefer "issue21321_2.7_e3217efa6edd_4.diff" patch be applied. -- ___ Python tracker <http://bugs.python.org/issue21321> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
