[issue12190] intern filenames in bytecode
New submission from Mike Solomon : I work on a large app and we noticed that a surprising portion of our heap was filenames embedded the the bytecode. This one-line patch to intern filenames reduces our on-disk size about ~15% and brings down our heap and in-memory object count by a similar percentage. -- components: Interpreter Core files: codegen.patch keywords: patch messages: 136997 nosy: Mike.Solomon priority: normal severity: normal status: open title: intern filenames in bytecode type: performance versions: Python 2.6, Python 2.7 Added file: http://bugs.python.org/file22131/codegen.patch ___ Python tracker <http://bugs.python.org/issue12190> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12190] intern filenames in bytecode
Mike Solomon added the comment: If you have a file with say a hundred functions, and each function contains the full path of that file on disk, your pyc file will contain about (100*(path_size+overhead)) bytes. In some cases, this is pretty significant. On Thu, May 26, 2011 at 12:47 PM, Benjamin Peterson wrote: > > Benjamin Peterson added the comment: > > How exactly does it bring down your disk space? > > -- > nosy: +benjamin.peterson > > ___ > Python tracker > <http://bugs.python.org/issue12190> > ___ > -- Added file: http://bugs.python.org/file22136/unnamed ___ Python tracker <http://bugs.python.org/issue12190> ___If you have a file with say a hundred functions, and each function contains the full path of that file on disk, your pyc file will contain about (100*(path_size+overhead)) bytes. In some cases, this is pretty significant. On Thu, May 26, 2011 at 12:47 PM, Benjamin Peterson <mailto:rep...@bugs.python.org";>rep...@bugs.python.org> wrote: Benjamin Peterson <mailto:benja...@python.org";>benja...@python.org> added the comment: How exactly does it bring down your disk space? -- nosy: +benjamin.peterson ___ Python tracker <mailto:rep...@bugs.python.org";>rep...@bugs.python.org> <http://bugs.python.org/issue12190"; target="_blank">http://bugs.python.org/issue12190> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12190] intern filenames in bytecode
Mike Solomon added the comment: The in-memory fix is really the most important - the disk space was a bonus and an easy metric to gather. Unfortunately, our app won't be upgrading to python 3.x. On Fri, May 27, 2011 at 7:10 AM, Benjamin Peterson wrote: > > Benjamin Peterson added the comment: > > As you can see, I've implemented a similar solution in 3.3. It should have > the same memory savings but not disk space saving. (This would require > reintroducing the marshal feature for interned strings.) > > -- > > ___ > Python tracker > <http://bugs.python.org/issue12190> > ___ > -- Added file: http://bugs.python.org/file22159/unnamed ___ Python tracker <http://bugs.python.org/issue12190> ___The in-memory fix is really the most important - the disk space was a bonus and an easy metric to gather. Unfortunately, our app won't be upgrading to python 3.x.On Fri, May 27, 2011 at 7:10 AM, Benjamin Peterson <mailto:rep...@bugs.python.org";>rep...@bugs.python.org> wrote: Benjamin Peterson <mailto:benja...@python.org";>benja...@python.org> added the comment: As you can see, I've implemented a similar solution in 3.3. It should have the same memory savings but not disk space saving. (This would require reintroducing the marshal feature for interned strings.) -- ___ Python tracker <mailto:rep...@bugs.python.org";>rep...@bugs.python.org> <http://bugs.python.org/issue12190"; target="_blank">http://bugs.python.org/issue12190> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
