[Tutor] Memory error - how to manage large data sets?
Hi, I am new to Python programming, I was trying to work out a few problems in order to grasp the knowledge gained after going through the basic chapters on Python programming. I got stuck with a memory error. Following is what I did, 1. I need to find the sum of all numbers at even positions in the Fibonacci series upto 2 million. 2. I have used lists to achieve this. 3. My program works good with smaller ranges. Say till 10,000 or even 100,000. However when I compute the sum for bigger ranges it gives me the memory error. 4. Also could someone tell me how to get the result in the form of an exponent. For instance, I would prefer 10^5 rather 10. Thanks in advance, Karthik ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Memory error - how to manage large data sets?
Forgot to include the following information, Platform - win32 Version - 2.5.1 Error message: Traceback (most recent call last): File "C:\Python25\programs\fibo.py", line 10, in if i % 2 == 0: MemoryError Code: fib = [] even = [] def fibonacci(x,y): return x+y for i in range (0,100): if i < 2: fib.append(i) else: i = fib[i-1] + fib[i-2] if i % 2 == 0: fib.append(i) even.append(i) else: fib.append(i) total = reduce(fibonacci,even) print total Any pointers would be of great help to me. Regards, Karthik From: Karthik [mailto:[EMAIL PROTECTED] Sent: Monday, July 28, 2008 9:27 PM To: 'tutor@python.org' Subject: Memory error - how to manage large data sets? Hi, I am new to Python programming, I was trying to work out a few problems in order to grasp the knowledge gained after going through the basic chapters on Python programming. I got stuck with a memory error. Following is what I did, 1. I need to find the sum of all numbers at even positions in the Fibonacci series upto 2 million. 2. I have used lists to achieve this. 3. My program works good with smaller ranges. Say till 10,000 or even 100,000. However when I compute the sum for bigger ranges it gives me the memory error. 4. Also could someone tell me how to get the result in the form of an exponent. For instance, I would prefer 10^5 rather 10. Thanks in advance, Karthik ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
[Tutor] Memory Error problem solved
I was amazed by the response of the Python community. Hats off to all. I
stopped using the lists and got the issue resolved using just the
variables. Nevertheless, i learned a lot by starting this thread.
Thanks a million to Alan, Chris, John for spending your quality time in
helping this newbie.
Hope i havent spoiled Alan's sleep much.
On 7/29/08, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
>
> Send Tutor mailing list submissions to
>tutor@python.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>http://mail.python.org/mailman/listinfo/tutor
> or, via email, send a message with subject or body 'help' to
>[EMAIL PROTECTED]
>
> You can reach the person managing the list at
>[EMAIL PROTECTED]
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Tutor digest..."
>
>
> Today's Topics:
>
> 1. Re: Memory error - how to manage large data sets? (John Fouhy)
> 2. Re: Memory error - how to manage large data sets? (Chris Fuller)
> 3. Re: Memory error - how to manage large data sets? (Alan Gauld)
> 4. Re: Turtle problem: how to exit the .exe? (Alan Gauld)
> 5. Re: Turtle problem: how to exit the .exe? (Dick Moores)
> 6. Re: Memory error - how to manage large data sets?
> (Daniel Sarmiento)
> 7. Re: Tutor Digest, Vol 53, Issue 99 (kinuthiA muchanE)
>
>
> --
>
> Message: 1
> Date: Tue, 29 Jul 2008 11:48:11 +1200
> From: "John Fouhy" <[EMAIL PROTECTED]>
> Subject: Re: [Tutor] Memory error - how to manage large data sets?
> To: "Daniel Sarmiento" <[EMAIL PROTECTED]>
> Cc: tutor@python.org
> Message-ID:
><[EMAIL PROTECTED]>
> Content-Type: text/plain; charset=ISO-8859-1
>
> On 29/07/2008, Daniel Sarmiento <[EMAIL PROTECTED]> wrote:
> > I tried to run your code and checked (with top) the memory ussage and
> > it uses more than 2 Gb of memory.
> >
> > I tried to modify the code a little bit to use less memory and came up
> > with this:
> >
> > fib = {0:0,1:1}
> >
> > even = []
> >
> > def fibonacci(x,y):
> >return x+y
> >
> > for j in xrange (2,100):
> > i = fib[j-1] + fib[j-2]
> > if i % 2 == 0:
> > even.append(i)
> > fib = {j-1:fib[j-1], j:i}
> >
> >
> > total = reduce(fibonacci,even)
> > print total
> >
> > It looks like the progam still hangs and I did not notice any memory
> > imrovements when running it with 1 000 000
>
> Well, let's see. You're still storing all the even fibonacci numbers
> that you compute. By the looks of things, one third of fibonacci
> numbers are even, so that's about 333,333 numbers that we're storing.
>
> How big do these numbers get? There's a closed-form expression for
> Fibonacci numbers; it's: fib(n) = (phi**n - (1-phi)**n)/sqrt(5), where
> phi is the golden ratio (about 1.6). 1-phi is -0.6, so when n is
> large, (1-phi)**n is practically zero. So fib(n) is roughly
> phi**n/sqrt(5). These numbers will quickly get beyond the size of
> normal integers, and into long integers. I don't know how many bytes
> a long integer takes up, but I guess we can estimate it by looking at
> the log to base 2.
>
> So let's consider the millionth Fibonacci number. fib(100) ~=
> phi**100/sqrt(5).
> So log(fib(100)) ~= log(phi**100/sqrt(5)) = 100*log(phi) -
> log(sqrt(5)).
>
> Taking logs to base 2, we get:
>
> >>> 100*log(phi, 2) - log(sqrt(5), 2)
> 694240.75266657001
>
> In other words, the best possible representation of the millionth
> Fibonacci number will take almost 700,000 bits, or around 85
> kilobytes. I don't know how Python long integers actually work; it
> may take up more space than that.
>
> Of course, that's just the biggest one we're working out. Let's try
> to work out how much space the first million will take up. This is:
>
> sum_{i=1}^{100} i*log(phi, 2) - log(sqrt(5), 2)
>
> == -100*log(sqrt(5), 2) + log(phi, 2)*sum_{i=1}^{100} i
>
> Remembering the formula for summing integers (n(n+1)/2), this is:
>
> >>> -100*log(sqrt(5), 2) + log(phi, 2)*(100*101/2)
> 347120142972.21808
>
> This is a number of bits; divide by 8 to get bytes; divide by 8*1024
> to get kilobytes, etc:
>
> >>> _/(8*1024*1024*1024)
> 40.410103156722393
>
> So ... that's about 40 gigabytes worth of numbers in this list you're
> trying to build. Well, actually you're only storing a third of the
> Fibonacci numbers (the even ones), so we can cut that down to thirteen
> gigabytes. Assuming my maths is correct and there's not too much
> overhead in Python long integers :-)
>
> (the solution, of course, is to avoid storing all those numbers in the
> first place)
>
> --
> John.
>
>
> --
>
> Message: 2
> Date: Mon, 28 Jul 2008 19:18:48 -0500
> From: Chris Fuller <[EMAIL PROTECTED]>
> Subject: Re: [Tutor] Memory error - how to manage large data sets?
> To: tutor@python.org
> Message-ID: <[EMAIL PROTECTED]>
> Content-Type
[Tutor] Facebook apps with python
Well, my question is simple.. How do I create facebook apps with python. I have couple of interesting/ funky programs and want to make them as apps. So, 1. What all things I should know for writing facebook apps. 2. I read that we should first upload our app to 'google app engine' and need do link it to facebook.. Is that right? 3. Actually, I am not aware of Network/ Web programming.. can I be able to do that? 4. Please do mention a couple of books (ebooks) from which I can learn.. That will help me. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
[Tutor] query from sqlalchemy returns AttributeError: 'NoneType' object
ent.ofp # The actual ofp_packet_in message.
#self.act_like_hub(packet, packet_in)
self.act_like_switch(packet, packet_in)
def launch ():
"""
Starts the component
"""
def start_switch (event):
log.debug("Controlling %s" % (event.connection,))
Tutorial(event.connection)
core.openflow.addListenerByName("ConnectionUp", start_switch)
When I run the above code I get the following error:
The problem that I am facing is for some reason if I use
if session.query(exists().where(SourcetoPort.src_address ==
str(packet.dst))).scalar() is not None:
in place of count query.
#if
session.query(SourcetoPort).filter_by(src_address=str(packet.dst)).count():
The querying from the database
q_res =
session.query(SourcetoPort).filter_by(src_address=str(packet.dst)).first()
self.send_packet(packet_in.buffer_id, packet_in.data,q_res.port_no,
packet_in.in_port)
is giving the following error:
DEBUG:core:POX 0.1.0 (betta) going up...
DEBUG:core:Running on CPython (2.7.3/Aug 1 2012 05:14:39)
DEBUG:core:Platform is
Linux-3.5.0-23-generic-x86_64-with-Ubuntu-12.04-precise
INFO:core:POX 0.1.0 (betta) is up.
DEBUG:openflow.of_01:Listening on 0.0.0.0:6633
INFO:openflow.of_01:[00-00-00-00-00-02 1] connected
DEBUG:tutorial:Controlling [00-00-00-00-00-02 1]
got info from the database
ERROR:core:Exception while handling Connection!PacketIn...
Traceback (most recent call last):
File "/home/karthik/pox/pox/lib/revent/revent.py", line 234, in
raiseEventNoErrors
return self.raiseEvent(event, *args, **kw)
File "/home/karthik/pox/pox/lib/revent/revent.py", line 281, in
raiseEvent
rv = event._invoke(handler, *args, **kw)
File "/home/karthik/pox/pox/lib/revent/revent.py", line 159, in
_invoke
return handler(self, *args, **kw)
File "/home/karthik/pox/tutorial.py", line 118, in _handle_PacketIn
self.act_like_switch(packet, packet_in)
File "/home/karthik/pox/tutorial.py", line 86, in act_like_switch
self.send_packet(packet_in.buffer_id, packet_in.data,q_res.port_no,
packet_in.in_port)
AttributeError: 'NoneType' object has no attribute 'port_no'
got info from the database
ERROR:core:Exception while handling Connection!PacketIn...
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
[Tutor] Defining variable arguments in a function in python
Hello,
I have the following piece of code. In this, I wanted to make use
of the optional parameter given to 'a', i.e- '5', and not '1'
def fun_varargs(a=5, *numbers, **dict):
print("Value of a is",a)
for i in numbers:
print("Value of i is",i)
for i, j in dict.items():
print("The value of i and j are:",i,j)
fun_varargs(1,2,3,4,5,6,7,8,9,10,Jack=111,John=222,Jimmy=333)
How do I make the tuple 'number' contain the first element to be 1 and not
2?
--
Regards,
Karthik A Bhat
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Defining variable arguments in a function in python
Thank you all for the quick response!
On Sun, Dec 30, 2018 at 10:39 AM Avi Gross wrote:
> I have my usual off the wall answer.
>
> OK, seriously. Not exactly an answer but perhaps an experiment.
>
> The question was how to have a non-named first argument to a function with
> some form of default.
>
> As was pointed out, this does not fit well with being able to have python
> gather all positional arguments after it as well as all keyword arguments.
>
> But bear with me. Say I want to have a way to signal that I want a default
> for the first argument?
>
> An empty comma fails but try this:
>
> def hello(a, *n, **m) :
> if a == None: a=5
> print(a)
> print(*n)
> print(**m)
>
> The above says "a" is required. It can be followed by any number of
> positional args gathered into "n" and any number of keyword args gathered
> into "m"
>
> But what if you define a sentinel to watch for such as None, in the above?
>
> If the first and only arg is None, it switches to the default of 5.
>
> >>> hello(None)
> 5
>
> Add a few more args and it properly takes it.
>
> >>> hello(1,2,3)
> 1
> 2 3
>
> Switch the first to None:
>
> >>> hello(None,2,3)
> 5
> 2 3
>
> The keywords don't work for print but no biggie.
>
> But is this only for None? What I say any negative arg is replaced by 5?
>
> def hello(a, *n, **m) :
> if a < 0: a=5
> print(a)
> print(*n)
>
> Seems to work fine:
>
> >>> hello(-666, 2, 3, 4)
> 5
> 2 3 4
>
> And I wonder if we can use the darn ellipsis for something useful?
>
> def hello(a, *n, **m) :
> if a == ... : a=5
> print(a)
> print(*n)
>
> >>> hello(1,2,3)
> 1
> 2 3
> >>> hello(...,2,3)
> 5
> 2 3
> >>> hello(...,2,...)
> 5
> 2 Ellipsis
>
> OK, all kidding aside, is this helpful? I mean if you want a function where
> you MUST give at least one arg and specify the first arg can be some odd
> choice (as above) and then be replaced by a default perhaps it would be
> tolerable to use None or an Ellipsis.
>
> Or on a more practical level, say a function wants an input from 1 to 10.
> The if statement above can be something like:
>
> >>> def hello(a, *n, **m) :
> if not (1 <= a <= 10) : a=5
> print(a)
> print(*n)
>
>
> >>> hello(1,2,3)
> 1
> 2 3
> >>> hello(21,2,3)
> 5
> 2 3
> >>> hello(-5,2,3)
> 5
> 2 3
> >>> hello("infinity and beyond",2,3)
> Traceback (most recent call last):
> File "", line 1, in
> hello("infinity and beyond",2,3)
> File "", line 2, in hello
> if not (1 <= a <= 10) : a=5
> TypeError: '<=' not supported between instances of 'int' and 'str'
>
> As expected, it may take a bit more code such as checking if you got an int
> but the idea may be solid enough. It is NOT the same as having a default
> from the command line but it may satisfy some need.
>
> Other than that, I fully agree that the current python spec cannot support
> anything like this in the function definition.
>
> Side note: To spare others, I sent Steven alone a deeper reply about ways
> to
> select random rows from a pandas DataFrame. I am still learning how pandas
> works and doubt many others here have any immediate needs.
>
>
>
>
>
>
>
>
>
> -Original Message-
> From: Tutor On Behalf Of
> Steven D'Aprano
> Sent: Saturday, December 29, 2018 6:02 AM
> To: tutor@python.org
> Subject: Re: [Tutor] Defining variable arguments in a function in python
>
> On Sat, Dec 29, 2018 at 11:42:16AM +0530, Karthik Bhat wrote:
> > Hello,
> >
> > I have the following piece of code. In this, I wanted to make
> > use of the optional parameter given to 'a', i.e- '5', and not '1'
> >
> > def fun_varargs(a=5, *numbers, **dict):
> [...]
> >
> > fun_varargs(1,2,3,4,5,6,7,8,9,10,Jack=111,John=222,Jimmy=333)
> >
> > How do I make the tuple 'number' contain the first element to be 1 and
> not
> 2?
>
>
> You can't. Python allocates positional arguments like "a" first, and only
> then collects whatever is left over in *numbers. How else would you expect
> it to work? Suppose you called:
>
> fun_varargs(1, 2, 3)
>
> wanting a to get the value 1, and numbers to get the values (2, 3). And
> then
> im
[Tutor] Off-Topic: Tutor group specific to Java
Hello Guys, This is kind of off-topic, but I would really appreciate it if anyone could provide me with a tutor mailing list/group specific to Java. I am a beginner, and it would be really helpful for me. -- Thanks & Regards, Karthik A Bhat ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
