Cool :)
About reading in a file, a little bit at a time, it’s not as difficult as
it seems.
f = open("100gb.json")
At this point, you’ve opened a file of an imaginary 100 gigabytes. What is
the effect on memory? 0. Nada.
Now, if you were to this..
data = f.read()
You’d be in trouble.
Now you’ve told Python (and the OS) to go out there and bring back the
entire contents of this file and put it in your variable data. That’s no
good.
But there are other ways of reading from a file.
first_line = next(f)
Bam, you’ve opened a huge file, read the first line and stopped. No more
data is read, memory is barely affected.
You can iterate this too.
for line in f:
print(line)
It will read one line at a time, print it and throw the data away. Memory
is barely affected.
The thing about various file formats, is that some of them can’t be read
like this. Some of them won’t make sense until you’ve read the entire file.
For example, consider JSON.
{
"key": "value"}
That first line is meaningless. The second line too. For this file to make
sense, you will need to read the entire file.
Some formats, including a variation of JSON, support “streaming”, which is
what we did up there. So if you’re looking to conserve memory, you’ll have
to add this criteria to your desired file format.
ps. Don’t forget to close the file, or use a context manager.
f.close()
On 6 October 2016 at 10:13, fruity <[email protected]> wrote:
> actually, the file would be split into 2 parts : the first one would be
> the infos that i want the user to be able to modify (a couple of lines),
> the second one would be a text of n lines (n being a number of vertices for
> a mesh, for instance), that the user will definetly not modify. So you're
> right, maybe i should split it into 2 files, the first in json or similar,
> the second encoded..
> For the memory leak, yes, i did experience something similar, during my
> last project (yes, in rigging too, it was nothing but fun on this project
> =p ), but with in-house tools. So i've seen it, but didn't do it myself.
> But that's why i'm concerned about that. And by memory leak, i'm also
> talking about memory management. For instance, you work on super heavy sets
> (like really super heavy sets.... ;-), and you want to load some datas
> attached to this set. By default, you'll have to load the entire data file,
> which will result in a huge consommation of RAM, and ultimately, probably a
> crash because of a lack of memory. So it wouldn't be a memory leak so to
> speak, but something that you'd have to handle by reading your file chunk
> after chunk, and flush the memory after each iteration. I'm not sure any
> module would do that automatically, though.
> I'll think at my problem differently, and try to split it into 2 parts,
> that seems so obvious now you mentionned it ! Thanks !
>
>
> Le jeudi 6 octobre 2016 10:29:18 UTC+2, Marcus Ottosson a écrit :
>>
>> cPickle looks great
>>
>> If you want it to be human readable, you can’t encrypt, compress or
>> otherwise obfuscate it, so both pickling and zlib is out. But think about
>> that for a second. If you are talking about multi-megabyte/gigabytes worth
>> of data, the mere quantity would make it uneditable regardless of the
>> format.
>>
>> The part that i’m not sure of is the memory management when working with
>> huge chunks of data, and i’m even not comfortable at how to mesure and
>> control it.
>>
>> I’m not sure why you would worry about that, unless you’ve experienced
>> some issues already? So long as you read the file into a variable, and that
>> variable isn’t kept around globally, then garbage collection would handle
>> this for you no matter which format you choose.
>>
>> To get a memory leak, you’d really have to try.
>>
>> import json
>>
>> leak = list()
>> def read_file(fname):
>> with open(fname) as f:
>> data = json.load(f)
>> leak.append(data) # builds up over time
>>
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Python Programming for Autodesk Maya" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/python_inside_maya/211afd52-15e9-4b4f-81bb-
> 05d11fb702b4%40googlegroups.com
> <https://groups.google.com/d/msgid/python_inside_maya/211afd52-15e9-4b4f-81bb-05d11fb702b4%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>
--
*Marcus Ottosson*
[email protected]
--
You received this message because you are subscribed to the Google Groups
"Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOAC7MZiDpKSWcUqGOqeStVY8v%2B2x4kOGhSFZMsPCy-oEw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
