date:20131212

On Thu, Dec 12, 2013 at 6:25 PM, Robert Voigtländer
 wrote:
> I need to find a -performant- way to transform this into a list with tuples 
> (a[0],[a[0][1]min],[a[0][1]max]).
>
> Hard to explaint what I mean .. [0] of the first three tuples is 52. [1] is 
> 193,193 and 192.
> What I need as result for these three tuples is: (52,192,193).
>
> For the next five tuples it is (51,188,193).
>
>
> Extra challenges:
> - This list is sorted. For performance reasons I would like to keep it 
> unsorted.
> - There may be tuples where min=max.
> - There my be tupples where [0] only exists once. So mix is automatically max

Yep, I see what you mean! Apart from the first of the challenges,
which is ambiguous: do you mean you'd rather be able to work with it
unsorted, or is that a typo, "keep it sorted"?

This is a common task of aggregation. Your list is of (key, value)
tuples, and you want to do some per-key statistics. Here are three
variants on the code:

# Fastest version, depends on the keys being already grouped
# and the values sorted within each group. It actually returns
# the last and first, not the smallest and largest.
def min_max_1(lst):
prev_key = None
for key, value in lst:
if key != prev_key:
if prev_key is not None: yield prev_key, value, key_max
key_max = value
if prev_key is not None: yield prev_key, value, key_max

# This version depends on the keys being grouped, but
# not on them being sorted within the groups.
def min_max_2(lst):
prev_key = None
for key, value in lst:
if key != prev_key:
if prev_key is not None: yield prev_key, key_min, key_max
key_min = key_max = value
else:
key_min = min(key_min, value)
key_max = min(key_max, value)
if prev_key is not None: yield prev_key, key_min, key_max

# Slowest version, does not depend on either the keys
# or the values being sorted. Will iterate over the entire
# list before producing any results. Returns tuples in
# arbitrary order, unlike the others (which will retain).
def min_max_3(lst):
data = {}
for key, value in lst:
if key not in data:
data[key]=(value, value)
else:
data[key][0] = min(data[key][0], value)
data[key][1] = min(data[key][1], value)
for key, minmax in data.items():
yield key, minmax[0], minmax[1]

Each of these is a generator that yields (key, min, max) tuples. The
third one needs the most memory and execution time; the others simply
take the input as it comes. None of them actually requires that the
input be a list - any iterable will do.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: [newbie] trying socket as a replacement for nc

On Thu, Dec 12, 2013 at 7:08 PM, Jean Dubois  wrote:
> Thanks for the reply, I changed the line you mentioned to
> s.send('*IDN?\n')

See if there's a newline issue - you might need \r\n here.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Threading In Python

2013-12-12 Thread marcinmltd

Adding subject to the message.

Sent from Samsung Mobile on O2

 Original message 
From: marcinmltd  
Date:  
To: [email protected] 
Subject:  

Hello,

I'm big fan of multiprocessing module, but recently I started looking at 
threading in Python more closely and got couple of questions I hope You can 
help me with:

1. When I run two or more threads in my python process are they really run 
concurrently on mulicore machine?

2. Browsing through documentation it looks like python interpreter protects its 
sensitive states by using GIL. Can you guys list situations when this happens?

2. What would be general advice from python experts on when to use threadng and 
when switch to multliprocessing in python? Is the decision still influenced by 
how often we need to comunicate between the tasks as it's in C\C++?

thanks in advance for help!
Marcin

-- 
https://mail.python.org/mailman/listinfo/python-list

[no subject]

2013-12-12 Thread marcinmltd

Hello,

I'm big fan of multiprocessing module, but recently I started looking at 
threading in Python more closely and got couple of questions I hope You can 
help me with:

1. When I run two or more threads in my python process are they really run 
concurrently on mulicore machine?

2. Browsing through documentation it looks like python interpreter protects its 
sensitive states by using GIL. Can you guys list situations when this happens?

2. What would be general advice from python experts on when to use threadng and 
when switch to multliprocessing in python? Is the decision still influenced by 
how often we need to comunicate between the tasks as it's in C\C++?

thanks in advance for help!
Marcin


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: [newbie] trying socket as a replacement for nc

2013-12-12 Thread Jean Dubois

On Thursday, December 12, 2013 12:20:36 AM UTC+1, Dan Stromberg wrote:
> On Wed, Dec 11, 2013 at 3:08 PM, Jean Dubois  wrote:
> 
> I have an ethernet-rs232 adapter which allows me to connect to a measurement 
> instrument by means of netcat on a linux system.
> 
> 
> e.g. entering nc 10.128.59.63 7000
> 
> allows me to enter e.g.
> 
> *IDN?
> 
> after which I get an identification string of the measurement instrument back.
> 
> I thought I could accomplish the same using the python module "socket"
> 
> and tried out the sample program below which doesn't work however:
> 
> 
> 
> Sockets reserve the right to split one socket.send() into multiple 
> socket.recv()'s on the other end of the communication, or to aggregate 
> multiple socket.send()'s into a single socket.recv() - pretty much any way 
> the relevant IP stacks and communications equipment feel like for the sake of 
> performance or reliability.
> 
> 
> The confusing thing about this is, it won't be done on every transmission - 
> in fact, it'll probably happen rather seldom unless you're on a heavy loaded 
> network or have some MTU issues (see Path MTU Discovery, and bear in mind 
> that paths can change during a TCP session).  But writing your code assuming 
> it will never happen is a bad idea.
> 
> 
> 
> For this reason, I wrote http://stromberg.dnsalias.org/~strombrg/bufsock.html 
> , which abstracts away these complications, and actually makes things pretty 
> simple.  There are examples on the web page.
> 
> 
> 
> HTH

Dear Dan, 
Could you copy paste here the code for your function I have to add to my 
"program"?

thanks in advance
jean
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: min max from tuples in list

Robert Voigtländer wrote:

> Hi,
> 
> I have a list like this:
> 
> a = [(52, 193), (52, 193), (52, 192), (51, 193), (51, 191), (51, 190),
> (51, 189), (51, 188), (50, 194), (50, 187), (50, 186), (50, 185), (50,
> 184), (49, 194), (49, 183), (49, 182), (49, 181), (48, 194), (48, 180),
> (48, 179), (48, 178), (48, 177), (47, 194), (47, 176), (47, 175), (47,
> 174), (47, 173), (46, 195), (46, 172), (46, 171), (46, 170), (46, 169),
> (45, 195), (45, 168), (45, 167), (45, 166), (44, 195), (44, 165), (44,
> 164), (44, 163), (44, 162), (43, 195), (43, 161), (43, 160), (43, 159),
> (43, 158), (42, 196), (42, 157), (42, 156), (42, 155), (41, 196), (41,
> 154), (41, 153), (41, 152), (41, 151), (40, 196), (40, 150), (40, 149),
> (40, 148), (40, 147), (39, 196), (39, 146), (39, 145), (39, 144), (39,
> 143), (38, 196), (38, 142), (38, 141), (38, 140), (37, 197), (37, 139),
> (37, 138), (37
>  , 137), (37, 136), (36, 197), (36, 135), (36, 134), (36, 133)]
> 
> 
> I need to find a -performant- way to transform this into a list with
> tuples (a[0],[a[0][1]min],[a[0][1]max]).
> 
> Hard to explaint what I mean .. [0] of the first three tuples is 52. [1]
> is 193,193 and 192. What I need as result for these three tuples is:
> (52,192,193).
> 
> For the next five tuples it is (51,188,193).
> 
> 
> Extra challenges:
> - This list is sorted. For performance reasons I would like to keep it
> unsorted. - There may be tuples where min=max.
> - There my be tupples where [0] only exists once. So mix is automatically
> max
> 
> 
> I hope I was able to explain what I mean.

I have a hunch that sorting the list might be quite efficient. You should at 
least try

import operator
import itertools

a = ...
a.sort()
result = []
for key, group in itertools.groupby(a, key=operator.itemgetter(0)):
minpair = maxpair = next(group)
for maxpair in group:
pass
result.append((key, minpair[1], maxpair[1]))

for item in result:
print(item)

to see whether it is good enough.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Threading In Python

On Thu, Dec 12, 2013 at 7:08 PM, marcinmltd  wrote:
> I'm big fan of multiprocessing module, but recently I started looking at
> threading in Python more closely and got couple of questions I hope You can
> help me with:

The first thing to note is that there are two completely separate concepts here:

1) Python is a language. It is defined by a set of specifications
covering syntax, semantics, etc, etc, etc.

2) CPython, Jython, IronPython, and PyPy are four implementations of
the Python language. (There are others, too, but those are the four
biggest.) They're all a bit different in how they behave.

Most likely, you're talking about CPython, which is the most
commonly-used implementation of Python. I'm going to discuss CPython
here, but be aware that the other Pythons may well (and in some cases
definitely DO) behave differently.

> 1. When I run two or more threads in my python process are they really run
> concurrently on mulicore machine?
>
> 2. Browsing through documentation it looks like python interpreter protects
> its sensitive states by using GIL. Can you guys list situations when this
> happens?

I'll answer these two together. Most likely the two threads will *not*
run concurrently, because CPython uses the Global Interpreter Lock
around all Python code execution. The GIL is released any time there's
going to be a long-running C-level call, especially something that
waits; so if your threads are spending most of their time, say, trying
to read from sockets, then there's no problem and the GIL won't get in
your way. There are a very VERY few CPU-bound operations that release
the GIL (some numpy calculations, I think, but I'm not familiar with
that library), but the general rule of thumb is that whenever the
CPU's busy, you own the GIL.

> 2. What would be general advice from python experts on when to use threadng
> and when switch to multliprocessing in python? Is the decision still
> influenced by how often we need to comunicate between the tasks as it's in
> C\C++?

Use threading when you're waiting for something else, and
multiprocessing when you need to exercise multiple CPU cores. Again,
that's just a rule of thumb, but it's close enough for a base
understanding of the difference.

Since I don't have real-world experience with any Python other than
CPython, I won't speak to those; but do expect that there'll be
differences. (I think, for instance, Jython's threading module is
backed by Java's threads, with all the advantages and disadvantages
that entails.) But the beauty of this mailing list is that someone
else will fill that in :)

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: min max from tuples in list

2013-12-12 Thread Robert Voigtländer

Wow, thanks for the educating answer. I'll work through all the varaints.
And yes, I meant keep it unsorted.

As I read it, sorting may be required then if I don't want to use the slowest 
variant. I'll test them all.

Thanks
Robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Threading In Python

2013-12-12 Thread Ian Kelly

On Thu, Dec 12, 2013 at 1:08 AM, marcinmltd  wrote:
> Adding subject to the message.
> Hello,
>
> I'm big fan of multiprocessing module, but recently I started looking at
> threading in Python more closely and got couple of questions I hope You can
> help me with:
>
> 1. When I run two or more threads in my python process are they really run
> concurrently on mulicore machine?

No.  The GIL allows the Python interpreter to run in only thread at a time.

> 2. Browsing through documentation it looks like python interpreter protects
> its sensitive states by using GIL. Can you guys list situations when this
> happens?

Any time Python code is being run, the GIL is held.  C extensions have
the option to release the GIL for long-running operations (e.g.
waiting on a network socket), but they are not permitted to work with
Python objects while the GIL is released.

> 2. What would be general advice from python experts on when to use threadng
> and when switch to multliprocessing in python? Is the decision still
> influenced by how often we need to comunicate between the tasks as it's in
> C\C++?

Generally speaking, use threading for programs that are IO-bound
(where there would be little concurrency anyway) and multiprocessing
for programs that are CPU-bound.  Communication between processes is
more expensive than communication between threads, so that could be an
important criterion.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: min max from tuples in list

On Thu, Dec 12, 2013 at 7:34 PM, Robert Voigtländer
 wrote:
> Wow, thanks for the educating answer. I'll work through all the varaints.
> And yes, I meant keep it unsorted.
>
> As I read it, sorting may be required then if I don't want to use the slowest 
> variant. I'll test them all.

Sorting would be slower than using the dictionary method. Much slower.
If you don't need to sort for any other reason, don't sort!

By the way, it's usually courteous to quote at least some of the text
you're responding to, to give context. Otherwise nobody knows what
you're talking about. This isn't a PHPBB forum; each post actually is
quite separate. You may find that Google Groups makes your life
unnecessarily difficult in this, so it's probably easier to use the
mailing list:

https://mail.python.org/mailman/listinfo/python-list

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: min max from tuples in list

2013-12-12 Thread Jussi Piitulainen

Robert Voigtländer writes:

> Hi,
> 
> I have a list like this:

# shortened:
a = [(52, 193), (52, 193), (52, 192),
 (51, 193), (51, 191), (51, 190), (51, 189), (51, 188),
 (50, 194)]

> I need to find a -performant- way to transform this into a list with
> tuples (a[0],[a[0][1]min],[a[0][1]max]).
> 
> Hard to explaint what I mean .. [0] of the first three tuples is
> 52. [1] is 193,193 and 192.  What I need as result for these three
> tuples is: (52,192,193).
> 
> For the next five tuples it is (51,188,193).

I think you want [(50, 194, 194), (51, 188, 193), (52, 192, 193)] for
the shortened list, in some order. Then you might be happy with a dict
instead. If I misunderstand, the following will be irrelevant.

> Extra challenges:
> - This list is sorted. For performance reasons I would like to keep
>   it unsorted.
> - There may be tuples where min=max.
> - There my be tupples where [0] only exists once. So mix is
>   automatically max

Scan the list and keep the running min and max for each key in a dict.

mix = dict()
for k,v in a:
   mix[k] = ( min(mix.get(k, (v,v))[0], v),
  max(mix.get(k, (v,v))[1], v) )

# mix == {50: (194, 194), 51: (188, 193), 52: (192, 193)}
# As list: [ (k,v[0],v[1]) for k,v in mix.items() ]
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: min max from tuples in list

Peter Otten wrote:

> Robert Voigtländer wrote:
> 
>> Hi,
>> 
>> I have a list like this:
>> 
>> a = [(52, 193), (52, 193), (52, 192), (51, 193), (51, 191), (51, 190),
>> (51, 189), (51, 188), (50, 194), (50, 187), (50, 186), (50, 185), (50,
>> 184), (49, 194), (49, 183), (49, 182), (49, 181), (48, 194), (48, 180),
>> (48, 179), (48, 178), (48, 177), (47, 194), (47, 176), (47, 175), (47,
>> 174), (47, 173), (46, 195), (46, 172), (46, 171), (46, 170), (46, 169),
>> (45, 195), (45, 168), (45, 167), (45, 166), (44, 195), (44, 165), (44,
>> 164), (44, 163), (44, 162), (43, 195), (43, 161), (43, 160), (43, 159),
>> (43, 158), (42, 196), (42, 157), (42, 156), (42, 155), (41, 196), (41,
>> 154), (41, 153), (41, 152), (41, 151), (40, 196), (40, 150), (40, 149),
>> (40, 148), (40, 147), (39, 196), (39, 146), (39, 145), (39, 144), (39,
>> 143), (38, 196), (38, 142), (38, 141), (38, 140), (37, 197), (37, 139),
>> (37, 138), (37
>>  , 137), (37, 136), (36, 197), (36, 135), (36, 134), (36, 133)]
>> 
>> 
>> I need to find a -performant- way to transform this into a list with
>> tuples (a[0],[a[0][1]min],[a[0][1]max]).
>> 
>> Hard to explaint what I mean .. [0] of the first three tuples is 52. [1]
>> is 193,193 and 192. What I need as result for these three tuples is:
>> (52,192,193).
>> 
>> For the next five tuples it is (51,188,193).
>> 
>> 
>> Extra challenges:
>> - This list is sorted. For performance reasons I would like to keep it
>> unsorted. - There may be tuples where min=max.
>> - There my be tupples where [0] only exists once. So mix is automatically
>> max
>> 
>> 
>> I hope I was able to explain what I mean.
> 
> I have a hunch that sorting the list might be quite efficient. You should
> at least try
> 
> import operator
> import itertools
> 
> a = ...
> a.sort()
> result = []
> for key, group in itertools.groupby(a, key=operator.itemgetter(0)):
> minpair = maxpair = next(group)
> for maxpair in group:
> pass
> result.append((key, minpair[1], maxpair[1]))
> 
> for item in result:
> print(item)
> 
> to see whether it is good enough.

On second thought -- Chris Angelico may be right ;)
So here's my dict-based variant: 

def keyminmax(items):
d = collections.defaultdict(list)
for key, value in items:
d[key].append(value)
for key, values in d.items(): # d.iteritems() in python 2
yield key, min(values), max(values)

for item in keyminmax(a):
print(item)

It uses a bit more memory than Chris' code, but has fewer min()/max() calls.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Experiences/guidance on teaching Python as a first programming language

2013-12-12 Thread wxjmfauth

Le mercredi 11 décembre 2013 11:45:43 UTC+1, Chris Angelico a écrit :
> On Wed, Dec 11, 2013 at 7:43 PM, Chris Angelico  wrote:
> 
> > When you tell a story, it's important to engage the reader from the
> 
> > start.
> 
> 
> 
> On Wed, Dec 11, 2013 at 8:39 PM,   wrote:
> 
> > A few practical considerations, far away from theoretical
> 
> > aspects. Mainly for non ascii, understand non native English
> 
> > speakers.
> 
> 
> 
> And then, shortly after the beginning of the story, you need to
> 
> introduce the villain. Thanks, jmf, for taking that position in our
> 
> role-play storytelling scenario! A round of applause for jmf, folks,
> 
> for doing a brilliant impression of the uninformed-yet-fanatical
> 
> Knight Templar villain!
> 

I know Python since ver. 1.5.6 and used it intensively
since ver. 2.0 or 2.1 (?). I acquired some user experience.

Windows, Py2.(7), ascii. It is not a secret Python uses
ascii for the representation. As an example, this guy
who some time ago exposed his own solution to solve that
problem (btw, elegant and correct). ---  you wrote blah, blah
about his "mysterious code point", you did not recognize
he is (was) using Turkish Windows with the code
page cp1254 ---. It is a little bit fascinating, 20 years
after the creation a language, people are still fighting
to write text in a human way.

Unicode. For a first language, it may be not a bad idea
to use a language which uses "unicode à la unicode".

Windows, Py3, unicode. It is is infortunate, but it is
a fact Python has some problems with that platform (file
sytem encoding), -> potential problems which should not
exist for a beginner.
I am the first to recognize the win console is all but
friendly. If one wishes to use a unicode code page, Python
fails [*].

Python has plenty of good qualities, you (and others)
are discussing plenty of theoretical aspects.
I'm pointing the fact, one may be stuck simply because
one cannot display a piece of of text!
I'm not so sure, such a behaviour is expected from a
beginner learning a computer language.

[*] I toyed with go(lang) and ruby 2 (only in a unicode
perspective), I should say I had no problems. Why? No
idea, it is too far beyond my knowlege.

jmf

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: [newbie] trying socket as a replacement for nc

2013-12-12 Thread Jean Dubois

On Thursday, December 12, 2013 9:21:32 AM UTC+1, Chris Angelico wrote:
> On Thu, Dec 12, 2013 at 7:08 PM, Jean Dubois  wrote:
> 
> > Thanks for the reply, I changed the line you mentioned to
> 
> > s.send('*IDN?\n')
> 
> 
> 
> See if there's a newline issue - you might need \r\n here.
> 
> 
> 
> ChrisA

I changed it as you suggested to:

 s.send('*IDN?\r\n')

unfortunately this doesn't change the result, first run give as response:
Received: 
second run makes the program hang and the adapter needs rebooting

kind regards,
jean
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Comparing values of counter in python 3.3

2013-12-12 Thread Wolfgang Maier

> I want to print only key,values in Counter2 which have values > then 
> corresponding value in Counter1.
> E.g
> Counter1={97:1,99:2,196:2,198:1}
> Counter2={97:1 ,99:3, 196:1,198:1}
> 
> # Output
> [99,3]
>

Try:

[[key, Counter2[key]] for key in Counter1 if Counter2[key] > Counter1[key]]

for a start.
If you can't guarantee that every key from Counter1 is also in Counter2 you 
could use something like:

[[key, Counter2[key]] for key in Counter1 if key in Counter2 and Counter2[key] 
> Counter1[key]]

Best,
Wolfgang
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Comparing values of counter in python 3.3


On 12/12/2013 09:55, Wolfgang Maier wrote:

I want to print only key,values in Counter2 which have values > then 
corresponding value in Counter1.
E.g
Counter1={97:1,99:2,196:2,198:1}
Counter2={97:1 ,99:3, 196:1,198:1}

# Output
[99,3]



Try:

[[key, Counter2[key]] for key in Counter1 if Counter2[key] > Counter1[key]]

for a start.
If you can't guarantee that every key from Counter1 is also in Counter2 you 
could use something like:

[[key, Counter2[key]] for key in Counter1 if key in Counter2 and Counter2[key] 
> Counter1[key]]

Best,
Wolfgang



Personal preference I suppose, but give me a for loop any day of the 
week, guess I just find them more readable :)


--
My fellow Pythonistas, ask not what our language can do for you, ask 
what you can do for our language.


Mark Lawrence

--
https://mail.python.org/mailman/listinfo/python-list

Re: Experiences/guidance on teaching Python as a first programming language

On Thu, Dec 12, 2013 at 8:17 PM,   wrote:
> Windows, Py2.(7), ascii. It is not a secret Python uses
> ascii for the representation.

Actually no, it doesn't.

Python 2.7.4 (default, Apr  6 2013, 19:54:46) [MSC v.1500 32 bit
(Intel)] on win32
>>> s = "abcd\xa9"
>>> print(s)
abcd©

The copyright symbol is not in ASCII. Are you suggesting that Python
uses a 7-bit internal representation of this data? Because a quick
squiz at the source code will prove that wrong. This is not ASCII.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Trouble with Multi-threading

On Wed, 11 Dec 2013 21:00:38 -0500, Roy Smith wrote:

> I believe I started off the chain of responses you're referring to.  I 
> meant it semi-humorously, but also with a point.  It is clear it turned 
> out to be harmful, and for that I apologize.
> 
> I share your dismay at where this group is going, and feel bad that I
> inadvertently moved it further in that direction.

Apology accepted. We're all only human. Except for 8 Dihedral.



-- 
Steven
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: The increasing disempowerment of the computer user

On Thu, 12 Dec 2013 13:35:37 +1100, Ben Finney wrote:

> Hmm, interesting Freudian slip there. I meant “cloud computing”, of
> course. That's where the computer owner pretends their service is always
> available and easy to access, while having terms of service that give
> them unilateral power to kick you off with no warning, no explanation,
> no accountability, and no recourse.

Now Ben, you know that's not true. Everybody has the only recourse that 
matters: buy the company and make them do what you want them to do. How 
hard could that possibly be?



-- 
Steven
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Optimizing list processing

2013-12-12 Thread Stefan Behnel

Terry Reedy, 12.12.2013 03:26:
> from itertools import count
> table = sorted(t for t in zip(iterable, count))

This is a useless use of a generator expression. sorted(zip(...)) is enough
(and likely to be substantially faster also).

Stefan


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Disable HTML in forum messages (was: Movie (MPAA) ratings and Python?)

On Wed, 11 Dec 2013 19:23:39 -0800, rusi wrote:

> The problem is that then your other mails (may) become plain text and
> your friends/recipients will wonder whether you've entered a
> time-machine and gone back to 1990!!

Not everything that's changed since 1990 has been an improvement.

> Many people find it simpler to just use Google groups.  It also has its
> problems (as do all methods!) but in sum its the easiest option to use.

How ironic. After mocking those of us who prefer to send and receive 
plain text, you then recommend that people use a delivery mechanism which 
sends plain text.

-- 
Steven
-- 
https://mail.python.org/mailman/listinfo/python-list

Pep8 in SetupTools

2013-12-12 Thread Chandru Rajendran

Hi all,

Please help me with running Pep8  using setuptools. Also help me how to Pep8 
for files in a folder.

Thanks & regards,
Chandru



 CAUTION - Disclaimer *
This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely
for the use of the addressee(s). If you are not the intended recipient, please
notify the sender by e-mail and delete the original message. Further, you are 
not
to copy, disclose, or distribute this e-mail or its contents to any other 
person and
any such actions are unlawful. This e-mail may contain viruses. Infosys has 
taken
every reasonable precaution to minimize this risk, but is not liable for any 
damage
you may sustain as a result of any virus in this e-mail. You should carry out 
your
own virus checks before opening the e-mail or attachment. Infosys reserves the
right to monitor and review the content of all messages sent to or from this 
e-mail
address. Messages sent to or from this e-mail address may be stored on the
Infosys e-mail system.
***INFOSYS End of Disclaimer INFOSYS***
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: min max from tuples in list

On Wed, 11 Dec 2013 23:25:53 -0800, Robert Voigtländer wrote:

> Hi,
> 
> I have a list like this:
> 
> a = [(52, 193), (52, 193), (52, 192), ...
> 
> 
> I need to find a -performant- way to transform this into a list with
> tuples (a[0],[a[0][1]min],[a[0][1]max]).

I'm afraid I don't know what you mean by "performant". It doesn't appear 
to be an English word, so far as I can tell. Do you mean efficient?

> Hard to explaint what I mean .. [0] of the first three tuples is 52. [1]
> is 193,193 and 192. What I need as result for these three tuples is:
> (52,192,193).
> 
> For the next five tuples it is (51,188,193).
> 
> 
> Extra challenges:
> - This list is sorted. For performance reasons I would like to keep it
> unsorted. 

Normally when people talk about "performance", they mean to *maximise* 
it, not minimise it. If you can guarantee that your data is already pre-
sorted, then the resulting algorithm is likely to be much more efficient 
than one which doesn't require sorted data.

In any case, sorting in Python is amazingly fast. You may be pleasantly 
surprised that a version that sorts your data, while nominally 
O(N log N), may be much faster than an O(N) solution that doesn't require 
sorted data. If I were a betting man, I'd be willing to wager a shiny new 
dollar[1] that sorting works out faster for reasonable sized sets of data.

One general piece of advice for speeding up Python code: the more work 
you can delegate to Python built-ins, like sort, min, max and so forth, 
the faster your code is likely to be.

> - There may be tuples where min=max.
> - There my be tupples where [0] only exists once. So mix is
> automatically max

Neither of these should be a problem.

The first approach I'd try would be:

- sort the list, which will group the tuples by their first item;

- conveniently, it will also ensure that the first tuple in each group 
will have the minimum second item, and the last tuple of that group the 
maximum value;

- use itertools.groupby to extract each group;

- grab the first tuple from the group, and take both its items;

- grab the last tuple from the group (if any), and take its second item;

- yield those three items.

Something like this should work, I expect:

from collections import deque
from itertools import groupby
from operator import itemgetter

def collect(data):
data = sorted(data)
groups = groupby(data, itemgetter(0))
d = deque([], maxlen=1)
for key, subiter in groups:
smallest = largest = next(subiter)[1]
d.extend(subiter)
try:
largest = d.pop()[1]
except IndexError:
pass
yield (key, smallest, largest)

And in use:

py> data = [(1, 4), (5, 3), (2, 7), (2, 3), (1, 3), (2, 5), 
... (5, 3), (4, 9)]
py> list(collect(data))
[(1, 3, 4), (2, 3, 7), (4, 9, 9), (5, 3, 3)]

[1] Betting man or not, nobody ever accused me of being a spend-thrift.

-- 
Steven
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: min max from tuples in list

2013-12-12 Thread Tim Chase

On 2013-12-12 11:44, Steven D'Aprano wrote:
> In any case, sorting in Python is amazingly fast. You may be
> pleasantly surprised that a version that sorts your data, while
> nominally O(N log N), may be much faster than an O(N) solution that
> doesn't require sorted data. If I were a betting man, I'd be
> willing to wager a shiny new dollar[1] that sorting works out
> faster for reasonable sized sets of data.

An interesting observation given the "Optimizing list processing"
thread you recently opened about algorithms for processing large
volumes of data and finding that elbow where two algorithms cross on
the performance graph. :-)

-tkc

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Optimizing list processing

On Wed, 11 Dec 2013 21:26:51 -0500, Terry Reedy wrote:

> On 12/11/2013 6:54 PM, Steven D'Aprano wrote:
>> I have some code which produces a list from an iterable using at least
>> one temporary list, using a Decorate-Sort-Undecorate idiom.
> 
> It is a non-standard version thereof, as DSU usually means to decorate
> with a key that gets discarded.

I do decorate with a key that gets discarded. The actual values that are 
being sorted are the enumerated values 0, 1, 2, ... and the temporary key 
values come from the iterable.

Confused yet? :-)

> A couple of examples of input and expected output would have been good
> ;-).

Only if I wanted you to understand the algorithm, which isn't really 
important. My question is about swapping between in-place modifications 
and external copying. If I wanted to be really mysterious (annoying), I 
could have just give pseudo-code :-)

Please don't focus on the algorithm I gave. Focus on the fact that I 
could write it like this:

if some condition to do with the computer's available memory:
 make modifications in place
else:
 make a copy of the data containing the modifications

if only I knew what that condition was and how to test for it.

>> The algorithm looks something like this (simplified):
>>
>> table = sorted([(x, i) for i,x in enumerate(iterable)])
> 
> This makes two temporaries when only one is needed, and shows why we
> added generator expressions.

You may be right in this particular case, I don't know I haven't tried 
it, but there are cases where list comps are significantly faster than 
generator expressions. For example, when passed to str.join, list comps 
are (on my computer) consistently 15% faster:

[steve@ando ~]$ python3.3 -m timeit "''.join([chr(n) for n in range(1, 
201)])"
1 loops, best of 3: 100 usec per loop
[steve@ando ~]$ python3.3 -m timeit "''.join([chr(n) for n in range(1, 
201)])"
1 loops, best of 3: 94.1 usec per loop
[steve@ando ~]$ python3.3 -m timeit "''.join(chr(n) for n in range(1, 
201))"
1 loops, best of 3: 116 usec per loop
[steve@ando ~]$ python3.3 -m timeit "''.join(chr(n) for n in range(1, 
201))"
1 loops, best of 3: 109 usec per loop

> table = sorted((x, i) for i,x in enumerate(iterable)) is equivalent to
> the table; table.sort lines below.
> 
> The following (untested) should be faster as it avoids tuple unpacking
> and repacking.

I'm not terribly fussed about micro-optimizations here. I'm concerned 
about *macro-optimizing* the case where creating two (or more) lists 
forces the computer to page memory in and out of virtual memory. Saving a 
few milliseconds? I don't care. Saving 5 or 6 seconds? That I care about.

[...]
> I expect the list comp be faster than in-place as long as the result
> list can be allocated and held in memory without paging. (This of course
> depends on system memory and other memory uses.)  

Exactly!

> List iterators have a
> __length_hint__ method giving the length of the underlying list, so the
> list comp result list can potentially be allocated once and then filled
> in by enumeration and replacement, but in C rather than Python code.

I don't want to pre-allocate the list comp. I want to avoid having to 
have two LARGE lists in memory at the same time, one containing the 
decorated values, and one not. I know that this is a good optimization 
for large enough lists. On my computer, ten million items is sufficient 
to demonstrate the optimization, and with sufficient testing I could 
determine a rough cut-off value, below which list comps are more 
efficient and above which in-place modifications are better.

But I don't know how to generalize that cut-off value. If I buy extra 
RAM, the cut-off value will shift. If I run it on another computer, it 
will shift.

P.S. The algorithm I'm working on is a way of generating index and rank 
tables. Not that it really matters -- what matters is determining whether 
or not to shift from "make a copy of the list" to "modify the list in 
place".

-- 
Steven
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: min max from tuples in list


On 12/12/2013 11:44, Steven D'Aprano wrote:

On Wed, 11 Dec 2013 23:25:53 -0800, Robert Voigtländer wrote:


Hi,

I have a list like this:

a = [(52, 193), (52, 193), (52, 192), ...


I need to find a -performant- way to transform this into a list with
tuples (a[0],[a[0][1]min],[a[0][1]max]).


I'm afraid I don't know what you mean by "performant". It doesn't appear
to be an English word, so far as I can tell. Do you mean efficient?


[snip]

There's some debate over whether it's English or not:

http://english.stackexchange.com/questions/38945/what-is-wrong-with-the-word-performant

--
https://mail.python.org/mailman/listinfo/python-list

Re: Optimizing list processing

On Thu, Dec 12, 2013 at 11:08 PM, Steven D'Aprano
 wrote:
> P.S. The algorithm I'm working on is a way of generating index and rank
> tables. Not that it really matters -- what matters is determining whether
> or not to shift from "make a copy of the list" to "modify the list in
> place".

So you're currently looking at...

if len(table) < ?:
table = [i for x,i in table]
else:
for x, i in table:
table[i] = x

Can I throw a spanner [1] in the works with other suggestions to try timing?

table[:] = [i for x,i in table]  # Does slice assignment get optimized?

SPLIT = 1048576  # Pick some useful cutoff
for part in range(0,len(table),SPLIT):
table[part:part+SPLIT] = [i for x,i in table[part:part+SPLIT]]

If slice assignment is reasonably fast (which I suspect it is), the
one-liner should be practically identical to your small-table
one-liner. Then if the million-record splits can be done inside
memory, it ought to be possible to do this in comparable time, even if
the total table length is huge. The choice of SPLIT would then matter
a lot less than the cutoff that you're trying to find; you might have
been able to do it in half the number of sections, but that won't make
as big a difference as suddenly paging out. Ideally, what I'd like to
see is that a higher SPLIT improves performance slightly until it gets
too high, at which point you go bust and the dealer wins... but the
critical word there is "slightly", meaning that it wouldn't cost too
much for SPLIT to be lower than it needs to be. That's the criteria
for the experiment; do you have the data on which to try it?

[1] Monkey wrench, for the Yanks in the room

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: min max from tuples in list

Steven D'Aprano wrote:

> In any case, sorting in Python is amazingly fast. You may be pleasantly
> surprised that a version that sorts your data, while nominally
> O(N log N), may be much faster than an O(N) solution that doesn't require
> sorted data. If I were a betting man, I'd be willing to wager a shiny new
> dollar[1] that sorting works out faster for reasonable sized sets of data.

Well, that was my first reaction, too. But then

 
$ cat keyminmax.py
import operator
import itertools
import collections

def minmax_groupby(items):
for key, group in itertools.groupby(sorted(items), 
key=operator.itemgetter(0)):
minpair = maxpair = next(group)
for maxpair in group:
pass
yield key, minpair[1], maxpair[1]

def minmax_dict(items):
d = collections.defaultdict(list)
for key, value in items:
d[key].append(value)
for key, values in d.items():
yield key, min(values), max(values)

a = [(52, 193), (52, 193), (52, 192), (51, 193), (51, 191), (51, 190),
 (51, 189), (51, 188), (50, 194), (50, 187),(50, 186), (50, 185),
 (50, 184), (49, 194), (49, 183), (49, 182), (49, 181), (48, 194),
 (48, 180), (48, 179), (48, 178), (48, 177), (47, 194), (47, 176),
 (47, 175), (47, 174), (47, 173), (46, 195), (46, 172), (46, 171),
 (46, 170), (46, 169), (45, 195), (45, 168), (45, 167), (45, 166),
 (44, 195), (44, 165), (44, 164), (44, 163), (44, 162), (43, 195),
 (43, 161), (43, 160), (43, 159), (43, 158), (42, 196), (42, 157),
 (42, 156), (42, 155), (41, 196), (41, 154), (41, 153), (41, 152),
 (41, 151), (40, 196), (40, 150), (40, 149), (40, 148), (40, 147),
 (39, 196), (39, 146), (39, 145), (39, 144), (39, 143), (38, 196),
 (38, 142), (38, 141), (38, 140), (37, 197), (37, 139), (37, 138),
 (37, 137), (37, 136), (36, 197), (36, 135), (36, 134), (36, 133)]

from collections import deque
from itertools import groupby
from operator import itemgetter

def collect(data):
data = sorted(data)
groups = groupby(data, itemgetter(0))
d = deque([], maxlen=1)
for key, subiter in groups:
smallest = largest = next(subiter)[1]
d.extend(subiter)
try:
largest = d.pop()[1]
except IndexError:
pass
yield (key, smallest, largest)


def time_dict():
for item in minmax_dict(a):
pass

def time_groupby():
for item in minmax_groupby(a):
pass

def time_daprano():
for item in collect(a):
pass

$ python -m timeit -s 'from keyminmax import time_groupby as t' 't()'
1 loops, best of 3: 68.6 usec per loop
$ python -m timeit -s 'from keyminmax import time_dict as t' 't()'
1 loops, best of 3: 53.3 usec per loop
$ python -m timeit -s 'from keyminmax import time_daprano as t' 't()'
1 loops, best of 3: 75.7 usec per loop

So yes, sorting seems to be slower even for small datasets.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Pep8 in SetupTools

2013-12-12 Thread Glenn Hutchings

On Thursday, 12 December 2013 11:13:51 UTC, Chandru Rajendran  wrote:

> Please help me with running Pep8  using setuptools. Also help me how to Pep8 
> for files in a folder.

The tool you're looking for is flake8.  It integrates with setuptools, so that 
you can run 'python setup.py flake8'.  More information here:

http://flake8.readthedocs.org/en/2.0/setuptools.html
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Optimizing list processing


On 12/12/2013 12:25, Chris Angelico wrote:

On Thu, Dec 12, 2013 at 11:08 PM, Steven D'Aprano
 wrote:

P.S. The algorithm I'm working on is a way of generating index and rank
tables. Not that it really matters -- what matters is determining whether
or not to shift from "make a copy of the list" to "modify the list in
place".


So you're currently looking at...

if len(table) < ?:
 table = [i for x,i in table]
else:
 for x, i in table:
 table[i] = x


Can I throw a spanner [1] in the works with other suggestions to try timing?

table[:] = [i for x,i in table]  # Does slice assignment get optimized?


[snip]

If you're trying that, you could also try:

table[:] = (i for x,i in table)

--
https://mail.python.org/mailman/listinfo/python-list

Re: Disable HTML in forum messages (was: Movie (MPAA) ratings and Python?)

2013-12-12 Thread Steve Hayes

On 12 Dec 2013 11:05:35 GMT, Steven D'Aprano
 wrote:

>On Wed, 11 Dec 2013 19:23:39 -0800, rusi wrote:
>
>> The problem is that then your other mails (may) become plain text and
>> your friends/recipients will wonder whether you've entered a
>> time-machine and gone back to 1990!!
>
>Not everything that's changed since 1990 has been an improvement.

And vice versa. 


-- 
Steve Hayes from Tshwane, South Africa
Web:  http://www.khanya.org.za/stevesig.htm
Blog: http://khanya.wordpress.com
E-mail - see web page, or parse: shayes at dunelm full stop org full stop uk
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Experiences/guidance on teaching Python as a first programming language

2013-12-12 Thread Ned Batchelder


On 12/12/13 4:17 AM, [email protected] wrote:

Le mercredi 11 décembre 2013 11:45:43 UTC+1, Chris Angelico a écrit :

On Wed, Dec 11, 2013 at 7:43 PM, Chris Angelico  wrote:


When you tell a story, it's important to engage the reader from the



start.




On Wed, Dec 11, 2013 at 8:39 PM,   wrote:


A few practical considerations, far away from theoretical



aspects. Mainly for non ascii, understand non native English



speakers.




And then, shortly after the beginning of the story, you need to

introduce the villain. Thanks, jmf, for taking that position in our

role-play storytelling scenario! A round of applause for jmf, folks,

for doing a brilliant impression of the uninformed-yet-fanatical

Knight Templar villain!



I know Python since ver. 1.5.6 and used it intensively
since ver. 2.0 or 2.1 (?). I acquired some user experience.

Windows, Py2.(7), ascii. It is not a secret Python uses
ascii for the representation.


It is incorrect that Py2.x uses ASCII strings.  It uses byte strings. 
Source files are assumed to be encoded in ASCII, so byte strings often 
are ASCII.  But as Chris has pointed out, bytestrings can hold any byte 
data, including UTF-8 if you wish.


JMF, I think you are clever enough and care enough about these issues to 
get this straight.


Many people seem to like the Pragmatic Unicode presentation I did, it 
may clear up some issues:  http://nedbatchelder.com/text/unipain.html


I'd be glad to have an extended conversation with you offline if you 
don't want to get into details here.



As an example, this guy
who some time ago exposed his own solution to solve that
problem (btw, elegant and correct). ---  you wrote blah, blah
about his "mysterious code point", you did not recognize
he is (was) using Turkish Windows with the code
page cp1254 ---. It is a little bit fascinating, 20 years
after the creation a language, people are still fighting
to write text in a human way.


"This guy": I have no idea who you are talking about.



Unicode. For a first language, it may be not a bad idea
to use a language which uses "unicode à la unicode".

Windows, Py3, unicode. It is is infortunate, but it is
a fact Python has some problems with that platform (file
sytem encoding), -> potential problems which should not
exist for a beginner.


File system encodings are very difficult.  Linux uses byte strings for 
file names, with no attempt to record the encoding, so there's a strong 
possibility that the declared encoding for the filesystem is wrong, or 
that your guess at the encoding will be wrong.



I am the first to recognize the win console is all but
friendly. If one wishes to use a unicode code page, Python
fails [*].


Yes, the Windows console and Python don't get along well with Unicode. 
This is a long-standing ticket: http://bugs.python.org/issue1602  I'm 
sure they would welcome your contribution towards a solution.  When I 
use Windows, I often wish this were solved.




Python has plenty of good qualities, you (and others)
are discussing plenty of theoretical aspects.
I'm pointing the fact, one may be stuck simply because
one cannot display a piece of of text!
I'm not so sure, such a behaviour is expected from a
beginner learning a computer language.


[*] I toyed with go(lang) and ruby 2 (only in a unicode
perspective), I should say I had no problems. Why? No
idea, it is too far beyond my knowlege.

jmf





--
Ned Batchelder, http://nedbatchelder.com

--
https://mail.python.org/mailman/listinfo/python-list

Re: Pep8 in SetupTools


On 12/12/2013 11:13, Chandru Rajendran wrote:

Hi all,

Please help me with running Pep8  using setuptools. Also help me how to
Pep8 for files in a folder.


I don't understand what you're asking for.  Can you please rephrase your 
question.




Thanks & regards,
Chandru



--
My fellow Pythonistas, ask not what our language can do for you, ask 
what you can do for our language.


Mark Lawrence

--
https://mail.python.org/mailman/listinfo/python-list

Re: Movie (MPAA) ratings and Python?

2013-12-12 Thread Dave Angel

On Wed, 11 Dec 2013 23:22:14 -0700, Michael Torrie 
 wrote:
From what I can see gmail is producing a multipart message that has 

a
plaint text part and an html part.  This is what gmail normally 

does and
as far as I know it's RFC-compliant and that's what gmail always 

does.

"Always does" doesn't mean it's a good idea on a text newsgroup. 

Very often the pretty text in the html part is mangled in the text 
part. Most often this is just indentation,  but for Python that's a 
biggie. It also means that we don't all see the same thing. 

Including both makes the download slower and more expensive. 

Some text newsreaders refuse to show anything if there's an html 
part.  Mine (groundhog on android) apparently shows the text part if 
it follows the html part.


--
DaveA

--
https://mail.python.org/mailman/listinfo/python-list

Formatting text in a table with reportlab

2013-12-12 Thread Joseph L. Casale

I sent off a msg to the reportlab list but didn't find an answer, hoping someone
here might have come across this...

I am generating a table to hold text oriented by the specification of the label
it gets printed on. I need to compress the vertical size of the table a little
more but the larger text in (1, 0) and (1, 1) as compared to the rest causes it
to migrate downwards away from the upper boundary of the cell that it's in.

Is there a better way to accomplish this or some way I can prevent the
behavior? I have tried adjusting with the rowHeight in Table() but for the
larger text in (1, 0) and (1, 1) to not migrate out of the cell, the overall
cell height has to be too large. The ('VALIGN', (1, 0), (1, 1), 'TOP') seems to
only have a minimal affect if the rowHeight is still larger than what I want
versus [None] for example.

# Define table data.
data = [
['foo', xxx, yyy, zzz, www],
['bar', ttt, uuu],
['\n'.join((iii, jjj, kkk))],
]

# Define table dimensions.
table = Table(
data,
[.625 * inch] + [None] * 4,
[None] * 3
)

# Define table styles.
table_style = [
('VALIGN', (0, 0), (-1, -1), 'BOTTOM'),
('VALIGN', (1, 0), (1, 1), 'TOP'),
('ALIGN', (0, 0), (0, 2), 'LEFT'),
('ALIGN', (1, 0), (1, 1), 'LEFT'),
('ALIGN', (2, 0), (2, 0), 'CENTER'),
('SPAN', (2, 1), (4, 1)),
('SPAN', (0, 2), (4, 2)),
('FONTSIZE', (0, 0), (0, 2), 10),
('TEXTFONT', (0, 0), (0, 2), 'Times'),
('FONTSIZE', (1, 0), (1, 1), 18),
('TEXTFONT', (1, 0), (1, 1), 'Times-Bold')
]

Thanks!
jlc

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Optimizing list processing

On Fri, Dec 13, 2013 at 12:32 AM, MRAB  wrote:
>> table[:] = [i for x,i in table]  # Does slice assignment get optimized?
>>
> [snip]
>
> If you're trying that, you could also try:
>
> table[:] = (i for x,i in table)

Right, a tweak which could be applied also to the split version. Thanks MRAB.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: [newbie] trying socket as a replacement for nc

2013-12-12 Thread Alister

On Thu, 12 Dec 2013 01:21:27 -0800, Jean Dubois wrote:

> On Thursday, December 12, 2013 9:21:32 AM UTC+1, Chris Angelico wrote:
>> On Thu, Dec 12, 2013 at 7:08 PM, Jean Dubois 
>> wrote:
>> 
>> > Thanks for the reply, I changed the line you mentioned to
>> 
>> > s.send('*IDN?\n')
>> 
>> 
>> 
>> See if there's a newline issue - you might need \r\n here.
>> 
>> 
>> 
>> ChrisA
> 
> I changed it as you suggested to:
> 
>  s.send('*IDN?\r\n')
> 
> unfortunately this doesn't change the result, first run give as
> response:
> Received:
> second run makes the program hang and the adapter needs rebooting
> 
> kind regards,
> jean

you probably need to use something like wireshark to see what is actually 
happening and compare it to a good connection in the normal way.




-- 
Save energy:  Drive a smaller shell.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: [newbie] trying socket as a replacement for nc

2013-12-12 Thread Alister

On Thu, 12 Dec 2013 01:21:27 -0800, Jean Dubois wrote:

> On Thursday, December 12, 2013 9:21:32 AM UTC+1, Chris Angelico wrote:
>> On Thu, Dec 12, 2013 at 7:08 PM, Jean Dubois 
>> wrote:
>> 
>> > Thanks for the reply, I changed the line you mentioned to
>> 
>> > s.send('*IDN?\n')
>> 
>> 
>> 
>> See if there's a newline issue - you might need \r\n here.
>> 
>> 
>> 
>> ChrisA
> 
> I changed it as you suggested to:
> 
>  s.send('*IDN?\r\n')
> 
> unfortunately this doesn't change the result, first run give as
> response:
> Received:
> second run makes the program hang and the adapter needs rebooting
> 
> kind regards,
> jean

you probably need to use something like wireshark to see what is actually 
happening and compare it to a good connection in the normal way.




-- 
Save energy:  Drive a smaller shell.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: [newbie] trying socket as a replacement for nc

2013-12-12 Thread Grant Edwards

On 2013-12-11, Dan Stromberg  wrote:
> On Wed, Dec 11, 2013 at 3:08 PM, Jean Dubois wrote:
>
>> I have an ethernet-rs232 adapter which allows me to connect to a
>> measurement instrument by means of netcat on a linux system.
>> e.g. entering nc 10.128.59.63 7000
>> allows me to enter e.g.
>> *IDN?
>> after which I get an identification string of the measurement instrument
>> back.
>> I thought I could accomplish the same using the python module "socket"
>> and tried out the sample program below which doesn't work however:
>>
>
> Sockets reserve the right to split one socket.send() into multiple
> socket.recv()'s on the other end of the communication, or to aggregate
> multiple socket.send()'s into a single socket.recv() - pretty much any way
> the relevant IP stacks and communications equipment feel like for the sake
> of performance or reliability.

Just to be pedantic: _TCP_ sockets reserver that right.  UDP sockets
do not, and do in fact guarantee that each message is discrete.  [It
appears that the OP is undoubtedly using TCP sockets.]

> The confusing thing about this is, it won't be done on every
> transmission - in fact, it'll probably happen rather seldom unless
> you're on a heavy loaded network or have some MTU issues (see Path
> MTU Discovery, and bear in mind that paths can change during a TCP
> session).  But writing your code assuming it will never happen is a
> bad idea.

And it _will_ fail someday in some odd circumstance when, for example,
some customer is be using it via a dial-up PPP connection, or there is
a satellite link in the path, or there's a flakey router somewhere,
or...

-- 
Grant Edwards   grant.b.edwardsYow! Those people look
  at   exactly like Donnie and
  gmail.comMarie Osmond!!
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: [newbie] trying socket as a replacement for nc

On Fri, Dec 13, 2013 at 1:16 AM, Grant Edwards  wrote:
> And it _will_ fail someday in some odd circumstance when, for example,
> some customer is be using it via a dial-up PPP connection, or there is
> a satellite link in the path, or there's a flakey router somewhere,
> or...

Or you write, write, write in quick succession. Nagle's Algorithm
means that if the first one hasn't yet been acknowledged, the second
one will be delayed, which means it'll probably be combined with the
third and sent when the first one's ACK comes through. Stream sockets
guarantee a stream of bytes; datagram sockets guarantee datagrams.
Weird how that goes, isn't it?

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Experiences/guidance on teaching Python as a first programming language


On 12/12/2013 13:52, Ned Batchelder wrote:


I'd be glad to have an extended conversation with you offline if you
don't want to get into details here.


Would you please to kind enough to keep it online, I enjoy a good 
comedian and our erstwhile unicode expert is one of the best, if not the 
best, in this field.


If you do take it offline would you please ask him to stop sending via 
google groups, or to take appropriate action to stop the extremely 
annoying double spacing.



Yes, the Windows console and Python don't get along well with Unicode.
This is a long-standing ticket: http://bugs.python.org/issue1602  I'm
sure they would welcome your contribution towards a solution.  When I
use Windows, I often wish this were solved.



There's also http://bugs.python.org/issue14170 and 
http://bugs.python.org/issue15809 amongst others.  Contributing towards 
a solution is a simple thing to do.  Finding a solution isn't as simple 
see e.g. http://www.gossamer-threads.com/lists/python/dev/731701


--
My fellow Pythonistas, ask not what our language can do for you, ask 
what you can do for our language.


Mark Lawrence

--
https://mail.python.org/mailman/listinfo/python-list

Re: Experiences/guidance on teaching Python as a first programming language

2013-12-12 Thread wxjmfauth

Le jeudi 12 décembre 2013 11:28:35 UTC+1, Chris Angelico a écrit :
> On Thu, Dec 12, 2013 at 8:17 PM,   wrote:
> 
> > Windows, Py2.(7), ascii. It is not a secret Python uses
> 
> > ascii for the representation.
> 
> 
> 
> Actually no, it doesn't.
> 
> 
> 
> Python 2.7.4 (default, Apr  6 2013, 19:54:46) [MSC v.1500 32 bit
> 
> (Intel)] on win32
> 
> >>> s = "abcd\xa9"
> 
> >>> print(s)
> 
> abcd©
> 
> 
> 
> The copyright symbol is not in ASCII. Are you suggesting that Python
> 
> uses a 7-bit internal representation of this data? Because a quick
> 
> squiz at the source code will prove that wrong. This is not ASCII.
> 
> 

>>> sys.version
'2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)]'
>>> sys.stdout.encoding
'cp1252'
>>> s = 'abc\xa9'
>>> t = 'abc©'
>>> s
'abc\xa9'
>>> t
'abc\xa9'
>>> print s, t, (s, t)
abc© abc© ('abc\xa9', 'abc\xa9')
>>> def HumanStr(o):
t = repr(o)
newt = t.replace('\\xa9', '©')
return newt

>>> print s, t, (s, t), HumanStr((s, t))
abc© abc© ('abc\xa9', 'abc\xa9') ('abc©', 'abc©')
>>> 

jmf

PS I do not insist on "sys.displayhook"

PS2 I can only congratulate this Turkish guy for
his understanding of Python
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: [newbie] trying socket as a replacement for nc


On 12/12/2013 14:05, Alister wrote:


you probably need to use something like wireshark to see what is actually
happening and compare it to a good connection in the normal way.



You've sent this twice old chap, you've a configuration issue somewhere 
I'd guess :)


--
My fellow Pythonistas, ask not what our language can do for you, ask 
what you can do for our language.


Mark Lawrence

--
https://mail.python.org/mailman/listinfo/python-list

Re: Experiences/guidance on teaching Python as a first programming language

On Fri, Dec 13, 2013 at 1:34 AM,   wrote:
> Le jeudi 12 décembre 2013 11:28:35 UTC+1, Chris Angelico a écrit :
>> On Thu, Dec 12, 2013 at 8:17 PM,   wrote:
>>
>> > Windows, Py2.(7), ascii. It is not a secret Python uses
>> > ascii for the representation.
>>
>> Actually no, it doesn't.
>
 sys.version
> '2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)]'
 sys.stdout.encoding
> 'cp1252'

What has this to do with ASCII or with Python's internal
representation? All you've proven is that you can convert the repr of
a string back into a byte-string, by replacing "\\xa9" with "\xa9",
and then shown that you can successfully render that as CP-1252 and it
displays as a copyright symbol. Meanwhile when I try the same thing on
my Windows box, the default encoding is cp437, so it throws. Proves
nothing about ASCII, as neither of those encodings is ASCII, and A9
does not decode as ASCII.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

input() on python 2.7.5 vs 3.3.2

2013-12-12 Thread stephen . boulet

Can someone explain? Thanks.

Python 3.3.2 (v3.3.2:d047928ae3f6, May 16 2013, 00:06:53) [MSC v.1600 64 bit 
(AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> x = input()
Hello there
>>> print(x)
Hello there

Python 2.7.5 (default, May 15 2013, 22:43:36) [MSC v.1500 32 bit (Intel)] on 
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> x = input()
Hello there
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 1
Hello there
  ^
SyntaxError: unexpected EOF while parsing
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: input() on python 2.7.5 vs 3.3.2

On Fri, Dec 13, 2013 at 1:45 AM,   wrote:
> Can someone explain? Thanks.
>
> Python 3.3.2 (v3.3.2:d047928ae3f6, May 16 2013, 00:06:53) [MSC v.1600 64 bit 
> (AMD64)] on win32
> Type "help", "copyright", "credits" or "license" for more information.
 x = input()
> Hello there
 print(x)
> Hello there
>
> Python 2.7.5 (default, May 15 2013, 22:43:36) [MSC v.1500 32 bit (Intel)] on 
> win32
> Type "help", "copyright", "credits" or "license" for more information.
 x = input()
> Hello there
> Traceback (most recent call last):
>   File "", line 1, in 
>   File "", line 1
> Hello there
>   ^
> SyntaxError: unexpected EOF while parsing

It's very simple: The input() function in Python 2.x is a very
dangerous one - it's equivalent to eval(input()) in Python 3. The
equivalent function in Python 2 is called raw_input. For safety and
compatibility, just do this at the beginning of your interactive
session or the top of your script:

input = raw_input

or, in a way that'll work in Python 3 as well, with no changes:

try:
input = raw_input
except NameError:
pass

After that, you can safely call input() and get back a string.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: input() on python 2.7.5 vs 3.3.2

2013-12-12 Thread Amit Saha

On Fri, Dec 13, 2013 at 12:45 AM,   wrote:
> Can someone explain? Thanks.
>
> Python 3.3.2 (v3.3.2:d047928ae3f6, May 16 2013, 00:06:53) [MSC v.1600 64 bit 
> (AMD64)] on win32
> Type "help", "copyright", "credits" or "license" for more information.
 x = input()
> Hello there
 print(x)
> Hello there

In Python 3, input() considers an input as a string and returns the
input as a string. This is the behavior of raw_input() in Python 2.

>
> Python 2.7.5 (default, May 15 2013, 22:43:36) [MSC v.1500 32 bit (Intel)] on 
> win32
> Type "help", "copyright", "credits" or "license" for more information.
 x = input()
> Hello there
> Traceback (most recent call last):
>   File "", line 1, in 
>   File "", line 1
> Hello there
>   ^
> SyntaxError: unexpected EOF while parsing

In Python 2, input() expects valid Python as it's input. If you
provide your input as 'Hello there' (a  Python string), it won't
complain.

HTH,
Amit.


-- 
http://echorand.me
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Experiences/guidance on teaching Python as a first programming language


On 12/12/2013 14:34, [email protected] wrote:

Le jeudi 12 décembre 2013 11:28:35 UTC+1, Chris Angelico a écrit :

On Thu, Dec 12, 2013 at 8:17 PM,   wrote:


Windows, Py2.(7), ascii. It is not a secret Python uses



ascii for the representation.




Actually no, it doesn't.



Python 2.7.4 (default, Apr  6 2013, 19:54:46) [MSC v.1500 32 bit

(Intel)] on win32


s = "abcd\xa9"



print(s)


abcd©



The copyright symbol is not in ASCII. Are you suggesting that Python

uses a 7-bit internal representation of this data? Because a quick

squiz at the source code will prove that wrong. This is not ASCII.





sys.version

'2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)]'

sys.stdout.encoding

'cp1252'

s = 'abc\xa9'
t = 'abc©'
s

'abc\xa9'

t

'abc\xa9'

print s, t, (s, t)

abc© abc© ('abc\xa9', 'abc\xa9')

def HumanStr(o):

t = repr(o)
newt = t.replace('\\xa9', '©')
return newt


print s, t, (s, t), HumanStr((s, t))

abc© abc© ('abc\xa9', 'abc\xa9') ('abc©', 'abc©')




jmf

PS I do not insist on "sys.displayhook"

PS2 I can only congratulate this Turkish guy for
his understanding of Python



I understand that this Turkish guy will be added to the list here 
http://en.wikipedia.org/wiki/Turing_Award next year for his stunning 
contribution to the field of computer science.  The year after he will 
win the award again for his outstanding contribution which prevents 
people from sending double spaced crap to this list.


--
My fellow Pythonistas, ask not what our language can do for you, ask 
what you can do for our language.


Mark Lawrence

--
https://mail.python.org/mailman/listinfo/python-list

Re: min max from tuples in list

2013-12-12 Thread Roy Smith

In article <[email protected]>,
 Steven D'Aprano  wrote:

> I'm afraid I don't know what you mean by "performant".

I've heard the term used often.  It means something like, "performs 
well" or "runs fast".  It may or may not be an English word, but that 
doesn't stop people from using it :-)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Optimizing list processing

Steven D'Aprano wrote:

> I have some code which produces a list from an iterable using at least
> one temporary list, using a Decorate-Sort-Undecorate idiom. The algorithm
> looks something like this (simplified):
> 
> table = sorted([(x, i) for i,x in enumerate(iterable)])
> table = [i for x,i in table]
> 
> The problem here is that for large iterables, say 10 million items or so,
> this is *painfully* slow, as my system has to page memory like mad to fit
> two large lists into memory at once. So I came up with an in-place
> version that saves (approximately) two-thirds of the memory needed.
> 
> table = [(x, i) for i,x in enumerate(iterable)]
> table.sort()
> for x, i in table:
> table[i] = x
> 
> For giant iterables (ten million items), this version is a big
> improvement, about three times faster than the list comp version. Since
> we're talking about the difference between 4 seconds and 12 seconds (plus
> an additional 40-80 seconds of general slow-down as the computer pages
> memory into and out of virtual memory), this is a good, solid
> optimization.
> 
> Except that for more reasonably sized iterables, it's a pessimization.
> With one million items, the ratio is the other way around: the list comp
> version is 2-3 times faster than the in-place version. For smaller lists,
> the ratio varies, but the list comp version is typically around twice as
> fast. A good example of trading memory for time.
> 
> So, ideally I'd like to write my code like this:
> 
> 
> table = [(x, i) for i,x in enumerate(iterable)]
> table.sort()
> if len(table) < ?:
> table = [i for x,i in table]
> else:
> for x, i in table:
> table[i] = x
> 
> where ? no doubt will depend on how much memory is available in one
> contiguous chunk.
> 
> Is there any way to determine which branch I should run, apart from hard-
> coding some arbitrary and constant cut-off value?

How about using two lists?

keys = list(iterable)
values = range(len(keys))
values.sort(key=keys.__getitem__)
del keys

The intention is to save memory used for the 2-tuples; I don't know if they 
pop up elsewhere.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: input() on python 2.7.5 vs 3.3.2


On 12/12/2013 14:56, Amit Saha wrote:

On Fri, Dec 13, 2013 at 12:45 AM,   wrote:

Can someone explain? Thanks.

Python 3.3.2 (v3.3.2:d047928ae3f6, May 16 2013, 00:06:53) [MSC v.1600 64 bit 
(AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

x = input()

Hello there

print(x)

Hello there


In Python 3, input() considers an input as a string and returns the
input as a string. This is the behavior of raw_input() in Python 2.



Python 2.7.5 (default, May 15 2013, 22:43:36) [MSC v.1500 32 bit (Intel)] on 
win32
Type "help", "copyright", "credits" or "license" for more information.

x = input()

Hello there
Traceback (most recent call last):
   File "", line 1, in 
   File "", line 1
 Hello there
   ^
SyntaxError: unexpected EOF while parsing


In Python 2, input() expects valid Python as it's input. If you
provide your input as 'Hello there' (a  Python string), it won't
complain.

HTH,
Amit.



I much prefer Chris Angelico's response "The input() function in Python 
2.x is a very dangerous one - it's equivalent to eval(input()) in Python 3."


--
My fellow Pythonistas, ask not what our language can do for you, ask 
what you can do for our language.


Mark Lawrence

--
https://mail.python.org/mailman/listinfo/python-list

Re: min max from tuples in list


On 12/12/2013 15:02, Roy Smith wrote:

In article <[email protected]>,
  Steven D'Aprano  wrote:


I'm afraid I don't know what you mean by "performant".


I've heard the term used often.  It means something like, "performs
well" or "runs fast".  It may or may not be an English word, but that
doesn't stop people from using it :-)



If "google" can be used to mean "make huge amouts of money with a 
product that is inherently flawed" then I'll happily accept "performant" 
as an English word, regardless of whether the English variant is UK, US, 
Australian, New Zealand, Soth African, Geordie, Glaswegian or any other :)


--
My fellow Pythonistas, ask not what our language can do for you, ask 
what you can do for our language.


Mark Lawrence

--
https://mail.python.org/mailman/listinfo/python-list

Re: input() on python 2.7.5 vs 3.3.2

On Fri, Dec 13, 2013 at 2:04 AM, Mark Lawrence  wrote:
> I much prefer Chris Angelico's response "The input() function in Python 2.x
> is a very dangerous one - it's equivalent to eval(input()) in Python 3."

Just to clarify: If you *want* eval, then you know you want it, and
you are (or should be) aware of its dangers. The problem, imo, is
hiding something as powerful and dangerous as code evaluation behind
the innocuous name "input". If I were coding a Python 2.x REPL, I
would probably write eval(raw_input()) rather than input(), just for
clarity.

But I'm more likely to just code for Python 3 anyway. :)

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Experiences/guidance on teaching Python as a first programming language

2013-12-12 Thread wxjmfauth

Le jeudi 12 décembre 2013 15:47:40 UTC+1, Chris Angelico a écrit :
> On Fri, Dec 13, 2013 at 1:34 AM,   wrote:
> 
> > Le jeudi 12 décembre 2013 11:28:35 UTC+1, Chris Angelico a écrit :
> 
> >> On Thu, Dec 12, 2013 at 8:17 PM,   wrote:
> 
> >>
> 
> >> > Windows, Py2.(7), ascii. It is not a secret Python uses
> 
> >> > ascii for the representation.
> 
> >>
> 
> >> Actually no, it doesn't.
> 
> >
> 
>  sys.version
> 
> > '2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)]'
> 
>  sys.stdout.encoding
> 
> > 'cp1252'
> 
> 
> 
> What has this to do with ASCII or with Python's internal
> 
> representation? All you've proven is that you can convert the repr of
> 
> a string back into a byte-string, by replacing "\\xa9" with "\xa9",
> 
> and then shown that you can successfully render that as CP-1252 and it
> 
> displays as a copyright symbol. Meanwhile when I try the same thing on
> 
> my Windows box, the default encoding is cp437, so it throws. Proves
> 
> nothing about ASCII, as neither of those encodings is ASCII, and A9
> 
> does not decode as ASCII.
> 
> 


Are you understanding Python by chance? print, __repr__, __str__,
sys.std*.encoding, ...
Are you understanding Windows? CHCP
Are you understanding the coding of the characters? cp1252, cp850, cp437, ...

Python (2) is managing  all this very well. Unfortunately, not in
a friendly way.

jmf
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Experiences/guidance on teaching Python as a first programming language

2013-12-12 Thread bob gailer


On 12/11/2013 9:07 PM, Larry Martell wrote:

Nope. Long before that I was working on computers that didn't boot 
when you powered them up, You had to manually key in a bootstrap 
program from the front panel switches.

PDP8? RIM loader, BIN loader?

--
https://mail.python.org/mailman/listinfo/python-list

Re: Experiences/guidance on teaching Python as a first programming language

2013-12-12 Thread Ned Batchelder


On 12/12/13 11:20 AM, [email protected] wrote:

Le jeudi 12 décembre 2013 15:47:40 UTC+1, Chris Angelico a écrit :

On Fri, Dec 13, 2013 at 1:34 AM,   wrote:


Le jeudi 12 décembre 2013 11:28:35 UTC+1, Chris Angelico a écrit :



On Thu, Dec 12, 2013 at 8:17 PM,   wrote:







Windows, Py2.(7), ascii. It is not a secret Python uses



ascii for the representation.







Actually no, it doesn't.







sys.version



'2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)]'



sys.stdout.encoding



'cp1252'




What has this to do with ASCII or with Python's internal

representation? All you've proven is that you can convert the repr of

a string back into a byte-string, by replacing "\\xa9" with "\xa9",

and then shown that you can successfully render that as CP-1252 and it

displays as a copyright symbol. Meanwhile when I try the same thing on

my Windows box, the default encoding is cp437, so it throws. Proves

nothing about ASCII, as neither of those encodings is ASCII, and A9

does not decode as ASCII.





Are you understanding Python by chance? print, __repr__, __str__,
sys.std*.encoding, ...
Are you understanding Windows? CHCP
Are you understanding the coding of the characters? cp1252, cp850, cp437, ...



Before we talk about Unicode, we should talk about the process of 
convincing people of things.


Asking questions won't convince anyone of anything.  If you have new 
information, then present it to us.  Presenting it means: show some 
code, show some bad outcome, and then explain what you you have 
demonstrated.  Be specific about what problem you are showing.


You said "Python uses ASCII."  Then you showed us Python code with 
non-ASCII characters.  We are confused what you are trying to tell us.


Python 2 uses byte strings.  Those byte strings can contain any bytes, 
conforming to any encoding the developer desires.  You asserted that it 
uses ASCII.  That is incorrect.


We have discussed Unicode with you enough to believe that we are not 
going to agree with you.  You hold a (very) minority view about what 
Python does with text, and you are not able to convince people of your 
view.  Isn't that frustrating?  Perhaps you need a new approach.



Python (2) is managing  all this very well. Unfortunately, not in
a friendly way.

jmf




--
Ned Batchelder, http://nedbatchelder.com

--
https://mail.python.org/mailman/listinfo/python-list

Re: Experiences/guidance on teaching Python as a first programming language


On 12/12/2013 4:17 AM, [email protected] wrote:


Windows, Py3, unicode. It is is infortunate, but it is
a fact Python has some problems with that platform (file
sytem encoding), -> potential problems which should not
exist for a beginner.


Some disappear if, for instance, one uses Idle.


I am the first to recognize the win console is all but
friendly.


Something we agree on. It is a piece crap. I believe that MS keeps it 
crippled to discourage use. (They apparently would like to remove it.)



If one wishes to use a unicode code page, Python fails [*].


If you mean cp65xxx (I forget exact numbers), MS Command Prompt fails, 
not Python. One should not use any other code page, but only other code 
pages work.


Tk, and hence tkinter and idle work fine with the entire BMP. If one 
tells tk to use a font that can handle the entire BMP, it displays the 
entire BMP.



Python has plenty of good qualities, you (and others)
are discussing plenty of theoretical aspects.
I'm pointing the fact, one may be stuck simply because
one cannot display a piece of of text!


Most text display problems disappear if one avoids Command Prompt and 
uses a GUI that can handle at least the BMP. Idle works on Windows after 
Python is installed unless one tells the installer to not install tcl/tk.


--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

Re: Optimizing list processing


On 12/12/2013 6:09 AM, Stefan Behnel wrote:

Terry Reedy, 12.12.2013 03:26:

from itertools import count
table = sorted(t for t in zip(iterable, count))


This is a useless use of a generator expression. sorted(zip(...)) is enough
(and likely to be substantially faster also).


Yes, definitely, and thank you. I did not squeeze enough.

--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

Python Script

2013-12-12 Thread Amimo Benja

I have an issue with a Python script that I will show as follows:
http://codepad.org/G8Z2ConI

Assume that you have three (well defined) classes: AirBase and VmNet, . VmNet 
has got a method that is called recursively each time an HTTP response is 
received. The variable recordTuple needs to be built independently for each 
instance of VmNet that is created. However, the mentioned variable is being 
overwritten across every instance, so if you try to get it from 
vmnet_instance_y, you would get exactly the same than retrieving it from 
vmnet_instance_x. 

• What is the code issue? I need to use this script in a project and I don't 
know how to proceed.

Actually, the script aims to follow the principle don't repeat yourself (DRY). 
As you may notice, VmNet and AirBase does not have def __init__(self), so 
self.recordTupleBase does not probably exist. Additionally, many other 
subclasses, similar to VmNet, can implement the recursive method using that 
recordTupleBase.

* I will gladly appreciate any help thanks
-- 
https://mail.python.org/mailman/listinfo/python-list

Tree library - multiple children


I need to use a tree structure. Is there a good and known library?
Doesn't have to be binary tree, I need to have multiple children per node.

Thanks

--
https://mail.python.org/mailman/listinfo/python-list

Re: Python Script

2013-12-12 Thread Gary Herron


On 12/12/2013 10:05 AM, Amimo Benja wrote:

I have an issue with a Python script that I will show as follows:
http://codepad.org/G8Z2ConI

Assume that you have three (well defined) classes: AirBase and VmNet, . VmNet 
has got a method that is called recursively each time an HTTP response is 
received. The variable recordTuple needs to be built independently for each 
instance of VmNet that is created. However, the mentioned variable is being 
overwritten across every instance, so if you try to get it from 
vmnet_instance_y, you would get exactly the same than retrieving it from 
vmnet_instance_x.

• What is the code issue? I need to use this script in a project and I don't 
know how to proceed.

Actually, the script aims to follow the principle don't repeat yourself (DRY). 
As you may notice, VmNet and AirBase does not have def __init__(self), so 
self.recordTupleBase does not probably exist. Additionally, many other 
subclasses, similar to VmNet, can implement the recursive method using that 
recordTupleBase.

* I will gladly appreciate any help thanks


You haven't actually asked a question here.  You say you don't know how 
to proceed with "a project", but we don't know what that project is.  In 
fact, I can't even figure out if your trouble is with the script, or 
with using the script in this unknown project.


Also, if you repost, please include the script in the email, not as a 
pointer to somewhere else.



Gary Herron

--
https://mail.python.org/mailman/listinfo/python-list

PyDev 3.1.0 released

2013-12-12 Thread Fabio Zadrozny

Hi All,

PyDev 3.1.0 has been released

Details on PyDev: http://pydev.org

Details on its development: http://pydev.blogspot.com

LiClipse (PyDev standalone with goodies such as support for Django
Templates, Mako Templates, Html, Javascript, etc):
http://brainwy.github.io/liclipse/

Release Highlights:
---

* **Important**: PyDev requires Eclipse 3.8 or 4.3 onwards and Java 7! For
older versions, keep using PyDev 2.x.

* **Refactoring**:

* It's now possible to rename a module (using F2 or drag and drop in
the pydev package explorer).

* Multiple improvements on the rename refactoring.

* **Debugger**:

* **Automatic code reloading on the debugger** (based on xreload).

* When a file is changed and a debug session is on, PyDev will
automatically reload it (based on xreload).

* View
https://github.com/fabioz/Pydev/blob/development/plugins/org.python.pydev/pysrc/pydevd_reload.pyfor
caveats/limitations.

* **Get referrers on debug**

* Right-click expression or variable in debugger and select 'Get
Referrers'

* Note: may not work on some Python variants as it needs access to
the gc module.

* **Stackless python** is now supported in the debugger, showing all
the suspended tasklets in the stack view.

* Automatically force focus to Eclipse on breakpoint hit (Enable in
prefereces > pydev > debug).

* The remote debugger can be left 'always on' (Enable in prefereces >
pydev > debug).

* If there's an exception while evaluating a conditional breakpoint the
thread is suspended and the issue reported.

* Option to skip caught exceptions thrown and handled in the same
context.

* A comment with @IgnoreException can be added to lines where an
exception is thrown to have that exception ignored by the debugger when
caught exceptions support is turned on.

* Improved visualization of frame objects.

* Bug-fixes on Jython debugging.

* **Unittest**:

* Django: The default PyDev unittest runner can now run Django tests
properly

* Selecting a unit-test method in the editor and **right-click > run as
unit-test** will run only the selected unit-test.

* **Ctrl+F9** with test selected will pre-select only that test to run
in unit-test.

* **General**:

* Improvements on search for references (Ctrl+Shift+G).

* Fixed some racing conditions related to the plugin startup.

* Organize imports has option to add from imports before other imports.

* Improved connection to shell that does code-completion.

* Properly supporting creation of shell inside a Jython VM in Eclipse.

What is PyDev?
---

PyDev is a plugin that enables users to use Eclipse for Python, Jython and
IronPython development -- making Eclipse a first class Python IDE -- It
comes with many goodies such as code completion, syntax highlighting,
syntax analysis, refactor, debug and many others.

Cheers,

--
Fabio Zadrozny
--
Software Developer

LiClipse
http://brainwy.github.io/liclipse

PyDev - Python Development Environment for Eclipse
http://pydev.org
http://pydev.blogspot.com
--
https://mail.python.org/mailman/listinfo/python-list

Re: Optimizing list processing


On 12/12/2013 7:08 AM, Steven D'Aprano wrote:


Please don't focus on the algorithm I gave. Focus on the fact that I
could write it like this:

if some condition to do with the computer's available memory:
  make modifications in place
else:
  make a copy of the data containing the modifications

if only I knew what that condition was and how to test for it.


In other words, you want a magic formula that depends on just about 
everything: the (static) memory in the machine, the other programs 
running, the memory otherwise used by the current program, the number 
items, the nature of the items, and possibly the memory fragmentation state.



You may be right in this particular case, I don't know I haven't tried
it, but there are cases where list comps are significantly faster than
generator expressions.


Stefan noted that neither is needed if zip produces the items wanted.


I'm not terribly fussed about micro-optimizations here. I'm concerned
about *macro-optimizing* the case where creating two (or more) lists
forces the computer to page memory in and out of virtual memory. Saving a
few milliseconds? I don't care. Saving 5 or 6 seconds? That I care about.


Why would you spend an hour or more of your time to save 5 seconds? 
Theoretical interest or a practical problem with enough runs to turns 
seconds into hours or days? A non-trivial practical answer will require 
more info about the practical context, including the distribution of 
machine memory sizes and of problem sizes.



I don't want to pre-allocate the list comp. I want to avoid having to
have two LARGE lists in memory at the same time, one containing the
decorated values, and one not. I know that this is a good optimization
for large enough lists. On my computer, ten million items is sufficient
to demonstrate the optimization, and with sufficient testing I could
determine a rough cut-off value, below which list comps are more
efficient and above which in-place modifications are better.

But I don't know how to generalize that cut-off value. If I buy extra
RAM, the cut-off value will shift. If I run it on another computer, it
will shift.


The simplest answer is to always avoid catastrophe by always modifying 
the one list. For 'short' lists, the time difference will be relatively 
small.


If the fraction of short lists is large enough to make the cumulative 
time difference large enough, and the list object sizes are more or less 
constant or at least bounded, a possible answer is to pick a minimum 
memory size, get a machine with that size, and work out a list size 
small enough to avoid thrashing.


--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

Re: Python Script


On 12/12/2013 1:35 PM, Gary Herron wrote:

On 12/12/2013 10:05 AM, Amimo Benja wrote:

I have an issue with a Python script that I will show as follows:
http://codepad.org/G8Z2ConI

Assume that you have three (well defined) classes: AirBase and VmNet,
. VmNet has got a method that is called recursively each time an HTTP
response is received. The variable recordTuple needs to be built
independently for each instance of VmNet that is created. However, the
mentioned variable is being overwritten across every instance, so if
you try to get it from vmnet_instance_y, you would get exactly the
same than retrieving it from vmnet_instance_x.

• What is the code issue? I need to use this script in a project and I
don't know how to proceed.

Actually, the script aims to follow the principle don't repeat
yourself (DRY). As you may notice, VmNet and AirBase does not have def
__init__(self), so self.recordTupleBase does not probably exist.
Additionally, many other subclasses, similar to VmNet, can implement
the recursive method using that recordTupleBase.

* I will gladly appreciate any help thanks


You haven't actually asked a question here.  You say you don't know how
to proceed with "a project", but we don't know what that project is.  In
fact, I can't even figure out if your trouble is with the script, or
with using the script in this unknown project.

Also, if you repost, please include the script in the email, not as a
pointer to somewhere else.


And reduce the script to the minimum needed to reproduce the problem. 
The effort to do that may reveal the solution.


--
Terry Jan Reedy


--
https://mail.python.org/mailman/listinfo/python-list

Re: Tree library - multiple children


On 12/12/2013 1:14 PM, Ricardo Aráoz wrote:

I need to use a tree structure. Is there a good and known library?


Search tools, both for the web and on pypi.python.org, are your friend.

--
Terry Jan Reedy


--
https://mail.python.org/mailman/listinfo/python-list

Re: Python Script


On 12/12/2013 18:05, Amimo Benja wrote:

I have an issue with a Python script that I will show as follows:
http://codepad.org/G8Z2ConI

Assume that you have three (well defined) classes: AirBase and VmNet,
. VmNet has got a method that is called recursively each time an HTTP
response is received. The variable recordTuple needs to be built
independently for each instance of VmNet that is created. However,
the mentioned variable is being overwritten across every instance, so
if you try to get it from vmnet_instance_y, you would get exactly the
same than retrieving it from vmnet_instance_x.

• What is the code issue? I need to use this script in a project and
I don't know how to proceed.

Actually, the script aims to follow the principle don't repeat
yourself (DRY). As you may notice, VmNet and AirBase does not have
def __init__(self), so self.recordTupleBase does not probably exist.
Additionally, many other subclasses, similar to VmNet, can implement
the recursive method using that recordTupleBase.

* I will gladly appreciate any help thanks


This line:

recordTuple = AirBase.recordTupleBase

makes recordTuple refer to the same object as AirBase.recordTupleBase 
and AirBase.recordTupleBase is an attribute of the class AirBase itself.


You're re-using the same object. That's why it's being overwritten.
--
https://mail.python.org/mailman/listinfo/python-list

Re: Tree library - multiple children

2013-12-12 Thread Neil Cerutti

On 2013-12-12, Ricardo Aráoz  wrote:
> I need to use a tree structure. Is there a good and known library?
> Doesn't have to be binary tree, I need to have multiple children per node.

Have you tried nested lists?

[[1, 2], [3, 4]

Can represent

 root
 /  \
  1-23-4

Python makes it very easy to manipulate such a structure. It
isn't clear that you need more than that yet.

-- 
Neil Cerutti

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Tree library - multiple children

2013-12-12 Thread Joel Goldstick

On Thu, Dec 12, 2013 at 2:26 PM, Neil Cerutti  wrote:

> On 2013-12-12, Ricardo Aráoz  wrote:
> > I need to use a tree structure. Is there a good and known library?
> > Doesn't have to be binary tree, I need to have multiple children per
> node.
>
> Have you tried nested lists?
>
> [[1, 2], [3, 4]
>

I think that was a type for this:

[[1, 2], [3, 4]]


>
> Can represent
>
>  root
>  /  \
>   1-23-4
>
> Python makes it very easy to manipulate such a structure. It
> isn't clear that you need more than that yet.
>
> --
> Neil Cerutti
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>



-- 
Joel Goldstick
http://joelgoldstick.com
-- 
https://mail.python.org/mailman/listinfo/python-list

ABC hackery

2013-12-12 Thread Demian Brecht

Hey all,

Thought I'd share some fun I had some time ago when digging into abcs
and metaclasses: https://gist.github.com/demianbrecht/6944269. The
implementation piggy backs off of abcs to achieve "class A should look
like class B" type functionality without using inheritance.

Not recommended for anything outside of POC :)

-- 
Demian Brecht
http://demianbrecht.github.com
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: ABC hackery

On Fri, Dec 13, 2013 at 6:26 AM, Demian Brecht  wrote:
> Not recommended for anything outside of POC :)

I love that kind of code :)

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Tree library - multiple children

On 12/12/2013 19:30, Joel Goldstick wrote:

On Thu, Dec 12, 2013 at 2:26 PM, Neil Cerutti mailto:[email protected]>> wrote:

On 2013-12-12, Ricardo Aráoz mailto:[email protected]>> wrote:
 > I need to use a tree structure. Is there a good and known library?
 > Doesn't have to be binary tree, I need to have multiple children
per node.

Have you tried nested lists?

[[1, 2], [3, 4]

I think that was a type for this:

I think that was a typo for "typo". :-)

[[1, 2], [3, 4]]

Can represent

  root
  /  \
   1-23-4

Python makes it very easy to manipulate such a structure. It
isn't clear that you need more than that yet.

--
https://mail.python.org/mailman/listinfo/python-list

Re: Tree library - multiple children

2013-12-12 Thread Joel Goldstick

On Thu, Dec 12, 2013 at 2:38 PM, MRAB  wrote:

> On 12/12/2013 19:30, Joel Goldstick wrote:
>
>> On Thu, Dec 12, 2013 at 2:26 PM, Neil Cerutti > > wrote:
>>
>> On 2013-12-12, Ricardo Aráoz > > wrote:
>>  > I need to use a tree structure. Is there a good and known library?
>>  > Doesn't have to be binary tree, I need to have multiple children
>> per node.
>>
>> Have you tried nested lists?
>>
>> [[1, 2], [3, 4]
>>
>> I think that was a type for this:
>>
>>  I think that was a typo for "typo". :-)


OMG!   I saw that too!

>
>
>  [[1, 2], [3, 4]]
>>
>>
>> Can represent
>>
>>   root
>>   /  \
>>1-23-4
>>
>> Python makes it very easy to manipulate such a structure. It
>> isn't clear that you need more than that yet.
>>
>>
> --
> https://mail.python.org/mailman/listinfo/python-list
>



-- 
Joel Goldstick
http://joelgoldstick.com
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Experiences/guidance on teaching Python as a first programming language

2013-12-12 Thread Wolfgang Keller

> I'm particularly interested to know if anyone can share experience of
> switching to teaching Python as a first programming language in a
> similar context. A written up case study that I could circulate among
> the relevant staff would be especially useful.

Just one experience from the "other" (student's) side.

When I started to study engineering science in 1991, Pascal was the
"first language" at the university in question (there were no
programming classes at highschool over here at that time yet). The class
was quite motivating and taught me some essential basics, I think.
Although issues such as object-orientation or event-based (GUI)
programming were not even mentioned, which is something that I'm
desperately missing today.

When I went to a different university (in 1993), still in engineering
science, they used C as the "first language" in the class there. The
result was that I tried (and succeeded) to pass that class with the
strict minimum of effort possible and deliberately forgot everything
that I had to learn about C as quickly as possible afterwards. I was a
"very good" student back then otherwise, so this was not due to
general laziness. What that class has taught me, essentially, was to
*hate* C. And it was not an issue of bad teachers. And they didn't mean
to make me hate C, after all, it was them who had chosen that language.
I never ever used C for anything (outside of that class). And ever
after that experience, I avoided all languages that were even remotely
similar to C, such as C++, Java, C#, Javascript, PHP etc.

In numerics classes and for research projects, I had to learn and use
Fortran, which was easy after the introduction with Pascal. The
teachers who taught me Fortran easily were the same as those who made me
hate C. 

Then, I accidentally got in touch with Python (in 1994 iirc) and
thought it was interesting and useful. In fact Python is the only
programming language that I ever learned without being obliged to do
so. And the only one that I keep using whenever I have the choice.

Since then, the university that once used C as a "first language"
has switched to Python. Which is a good thing, imho. If I had had to
learn the basics of programming with C instead of Pascal, I most
certainly would have avoided anything even remotely connected to
programming ever since, even "office automation" through scripting
(which is what I use Python for today).

Sincerely,

Wolfgang
-- 
https://mail.python.org/mailman/listinfo/python-list

Downloading file

2013-12-12 Thread Matt Graves

I have direct links to a number of csv files to download. Copying and pasting 
it to my browser would take too long, how would i go to this site for example 
and get the file? Right when you go to the site the download should start

www.example.com/files/document.csv
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Tree library - multiple children


On 12/12/2013 18:56, Terry Reedy wrote:

On 12/12/2013 1:14 PM, Ricardo Aráoz wrote:

I need to use a tree structure. Is there a good and known library?


Search tools, both for the web and on pypi.python.org, are your friend.



stackoverflow is another useful site to search and see also 
http://kmike.ru/python-data-structures/.


--
My fellow Pythonistas, ask not what our language can do for you, ask 
what you can do for our language.


Mark Lawrence

--
https://mail.python.org/mailman/listinfo/python-list

Code suggestion - List comprehension

2013-12-12 Thread Shyam Parimal Katti

Hello,

I have a list of sql queries, some which are split across multiple list
elements e.x.
['drop table sample_table;', 'create table sample_test', '(col1 int);',
'select col1 from', ' sample_test;']

A semi-colon in the string value  indicates the termination of a sql query.
So the expected out come is a conversion to a list of valid sql queries:
['drop table sample_table;', 'create table sample_test (col1 int);',
'select col1 from sample_test;']

Here is the code that does that:

sample = ['drop table sample_table;', 'create table sample_test', '(col1
int);', 'select col1 from', ' sample_test;']
pure_sqls = []
query_holder= ''
for each_line in sample:
query_holder += each_line
if query_holder.endswith(';'):
pure_sqls.append(query_holder)
query_holder = ''


Is there a way to do this by eliminating explicit creation of new
list(pure_sqls) and a temporary variable(query_holder)? Using list
comprehension? Though I don't want to put the shorter version in
production(if it is difficult to understand), I am looking if this can be
done with list comprehension since I am trying to learn list comprehension
by using it in such scenarios.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Experiences/guidance on teaching Python as a first programming language

On Fri, Dec 13, 2013 at 7:36 AM, Wolfgang Keller  wrote:
> And ever
> after that experience, I avoided all languages that were even remotely
> similar to C, such as C++, Java, C#, Javascript, PHP etc.

I think that's disappointing, for two reasons. Firstly, C syntax isn't
that terrible. You might prefer Python syntax to it, but it's
undeniably better than several of its predecessors (I do not want to
write in COBOL, tyvm!), and there are recent languages that manage to
get some things so crazily backward (like abolishing operator
precedence so 2 + 3 * 4 = 24 not 14) that I wouldn't want to use them.
And secondly, C is very much the language of Unix. Sure, its best job
is implementing high level languages so day-to-day code doesn't need
to use it, but it's still important when you need to get to some
lower-level facilities. For those two reasons, I think a basic working
knowledge of C is useful for working with computers, networking,
pretty much everything these days. It won't break your brain to
understand multiple styles, and it might help you to remember why it
is you love Python syntax so much :)

Last time I had to do some C work, I was reminded just how amazingly
convenient a high-level string class is... all I wanted to do was
concatenate a bunch of strings with spaces between them, and I had to
go through so many hoops!

" ".join(list_of_strings)

Et voila.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Tree library - multiple children

2013-12-12 Thread Michael Torrie

On 12/12/2013 11:14 AM, Ricardo Aráoz wrote:
> I need to use a tree structure. Is there a good and known library?
> Doesn't have to be binary tree, I need to have multiple children per node.

There are lots of types of tree structures that may or may not be
applicable to your problem.  And it depends on what kind of data you're
storing.  For example, I wrote a parser years ago (in C) that processed
BER-encoded structured data (sort of like binary xml).  Turned out that
the nested structure of BER-encoded data lends itself well to
"left-child, right-sibling" trees (and it happens to be binary, which
makes for easy traversal).

In any even Python's data primitives are powerful enough that you don't
need a library at all.  Just use Python's built-in primitives.  You can
do most tree structures with just list manipulation, without any class
overhead at all.

In fact in my case, my "left-child right sibling" trees are by
definition lists (think LISP car and cdr) or tuples.  The LISP-esque
nature of Python's data types always did make Python appeal to me.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re:

2013-12-12 Thread Dan Stromberg

On Thu, Dec 12, 2013 at 12:06 AM, marcinmltd  wrote:
> Hello,
>
> I'm big fan of multiprocessing module, but recently I started looking at
> threading in Python more closely and got couple of questions I hope You can
> help me with:
>
> 1. When I run two or more threads in my python process are they really run
> concurrently on mulicore machine?
>
> 2. Browsing through documentation it looks like python interpreter protects
> its sensitive states by using GIL. Can you guys list situations when this
> happens?
>
> 2. What would be general advice from python experts on when to use threadng
> and when switch to multliprocessing in python? Is the decision still
> influenced by how often we need to comunicate between the tasks as it's in
> C\C++?

Jython and IronPython reportedly thread well generally.

CPython threads I/O-bound tasks well, but as soon as you introduce one
CPU-bound thread, the other threads start to have problems.  Pypy is
probably the same as CPython, at least until they get their STM
working well.

Multiprocessing can do CPU-bound and I/O-bound workloads, but starting
a new process takes longer than starting a new thread (especially on
Windows).  Inter-thread/process communication is likely slower on
multiprocessing than multithreading as well, but this is merely my
inference.

Note: CPython is what a lot of people call "Python"; CPython is a new
term for the original implementation to distinguish it from the other
implementations that exist today.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Experiences/guidance on teaching Python as a first programming language

2013-12-12 Thread Larry Martell

On Thu, Dec 12, 2013 at 11:51 AM, bob gailer  wrote:
> On 12/11/2013 9:07 PM, Larry Martell wrote:
>
>> Nope. Long before that I was working on computers that didn't boot when
>> you powered them up, You had to manually key in a bootstrap program from the
>> front panel switches.
>
> PDP8? RIM loader, BIN loader?

Data General Nova 3
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: [newbie] trying socket as a replacement for nc

2013-12-12 Thread Dan Stromberg

On Thu, Dec 12, 2013 at 12:28 AM, Jean Dubois  wrote:
> On Thursday, December 12, 2013 12:20:36 AM UTC+1, Dan Stromberg wrote:
>> On Wed, Dec 11, 2013 at 3:08 PM, Jean Dubois  wrote:
>>
>> I have an ethernet-rs232 adapter which allows me to connect to a measurement 
>> instrument by means of netcat on a linux system.
>>
>>
>> e.g. entering nc 10.128.59.63 7000
>>
>> allows me to enter e.g.
>>
>> *IDN?
>>
>> after which I get an identification string of the measurement instrument 
>> back.
>>
>> I thought I could accomplish the same using the python module "socket"
>>
>> and tried out the sample program below which doesn't work however:
>>
>>
>>
>> Sockets reserve the right to split one socket.send() into multiple 
>> socket.recv()'s on the other end of the communication, or to aggregate 
>> multiple socket.send()'s into a single socket.recv() - pretty much any way 
>> the relevant IP stacks and communications equipment feel like for the sake 
>> of performance or reliability.
>>
>>
>> The confusing thing about this is, it won't be done on every transmission - 
>> in fact, it'll probably happen rather seldom unless you're on a heavy loaded 
>> network or have some MTU issues (see Path MTU Discovery, and bear in mind 
>> that paths can change during a TCP session).  But writing your code assuming 
>> it will never happen is a bad idea.
>>
>>
>>
>> For this reason, I wrote 
>> http://stromberg.dnsalias.org/~strombrg/bufsock.html , which abstracts away 
>> these complications, and actually makes things pretty simple.  There are 
>> examples on the web page.
>>
>>
>>
>> HTH
>
> Dear Dan,
> Could you copy paste here the code for your function I have to add to my 
> "program"?

This is untested, but it should be something like the following:

#!/usr/bin/env python

"""
A simple echo client
"""
import socket as socket_mod
import bufsock as bufsock_mod
host = '10.128.59.63'
port = 7000
size = 10
socket = socket_mod.socket(socket.AF_INET, socket.SOCK_STREAM)
socket.connect((host,port))
bufsock = bufsock_mod.bufsock(socket)
bufsock.send('*IDN?')
data = bufsock.recv(size)
bufsock.close()
print 'Received:', data

You might look over
http://stackoverflow.com/questions/19918307/retrieve-file-information-located-on-a-different-application-server-using-python/19918706#19918706
for a more complete example.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: [newbie] trying socket as a replacement for nc

2013-12-12 Thread Dan Stromberg

On Thu, Dec 12, 2013 at 6:16 AM, Grant Edwards  wrote:

>> Sockets reserve the right to split one socket.send() into multiple
>> socket.recv()'s on the other end of the communication, or to aggregate
>> multiple socket.send()'s into a single socket.recv() - pretty much any way
>> the relevant IP stacks and communications equipment feel like for the sake
>> of performance or reliability.
>
> Just to be pedantic: _TCP_ sockets reserver that right.  UDP sockets
> do not, and do in fact guarantee that each message is discrete.  [It
> appears that the OP is undoubtedly using TCP sockets.]

I haven't done a lot of UDP, but are you pretty sure UDP can't at
least fragment large packets?  What's a router or switch to do if the
Path MTU isn't large enough for an original packet?

http://www.gamedev.net/topic/343577-fragmented-udp-packets/
-- 
https://mail.python.org/mailman/listinfo/python-list

Downloading multiple files based on info extracted from CSV

2013-12-12 Thread Matt Graves

I have a CSV file containing a bunch of URLs I have to download a file from for 
clients (Column 7) and the clients names (Column 0) I tried making a script to 
go down the .csv file and just download each file from column 7, and save the 
file as [clientname].csv

I am relatively new to python, so this may be way off but…






import urllib 
import csv
urls = []
clientname = []

###This will set column 7 to be a list of urls
with open('clients.csv', 'r') as f:
reader = csv.reader(f)
for column in reader:
urls.append(column[7])

###And this will set column 0 as a list of client names
with open('clients.csv', 'r') as g:
reader = csv.reader(g)
for column in reader:
clientname.append(column[0])

###This SHOULD plug in the URL for F, and the client name for G.
def downloadFile(urls, clientname):
urllib.urlretrieve(f, "%g.csv") % clientname


downloadFile(f,g)



When I run it, I get : AttributeError: 'file' object has no attribute 'strip'
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Code suggestion - List comprehension

On Fri, Dec 13, 2013 at 7:40 AM, Shyam Parimal Katti  wrote:
> A semi-colon in the string value  indicates the termination of a sql query.
> So the expected out come is a conversion to a list of valid sql queries:
> ['drop table sample_table;', 'create table sample_test (col1 int);', 'select
> col1 from sample_test;']

Hmm. Just to clarify, does a semicolon _anywhere_ in the string
terminate the query? If so, you have a problem of parsing, and the
best thing to do is to enable multi-query processing on your database
connection and send them all across. (You'd have to skip any
semicolons inside quoted strings, for instance.) Your current code
depends on the semi being at the end of one of the strings, which is a
much safer proposition.

If you really mean to split it anywhere, the easiest is to simply join
the whole lot and then split on the semi:

sample = ['drop table sample_table;', 'create table sample_test',
'(col1 int);', 'select col1 from', ' sample_test;']
pure_sqls = ''.join(sample).split(';')[:-1]

Note that the last element from the split is NOT a valid query. If
all's well, it should be an empty string (as it will be in this
sample), but if it's not empty, it's a partial query. I don't know
what you want to do with those; your code above will simply ignore
them, so I've done the same here, applying the "trim off the last
element" smiley operator [:-1] before assigning to pure_sqls.

Parenthesis: I just asked the members of Threshold RPG what they
thought [:-1] meant. 15 seconds after I asked, three responses came in
almost simultaneously.

Zeala: pothead smoking a roach
Claudas: fallen jaw?
Tharl: constipated, but happy.

I don't know what that means for the OP's code. Somehow it doesn't
seem a good omen. End parenthesis.

For what you're doing, a list comp isn't really appropriate. Broadly,
a list comp should be creating zero or one elements from each element
of the source list; what you're trying to do here is stitching things
together, which requires state. You can't do that with a list comp.
The best way is probably what you already have, but if you'd like it
to be shorter, you need simply invent a split marker that can't
possibly exist in your queries, and use that. Let's suppose "\0" can't
ever occur (that's likely, given that you're working with SQL).

sample = ['drop table sample_table;', 'create table sample_test',
'(col1 int);', 'select col1 from', ' sample_test;']
pure_sqls = [s.replace('\0','') for s in
'\0'.join(sample+['']).split(';\0') if s!='']

Assuming the exact sequence ";\0" never comes up in your text, this
will work perfectly. You could change out the replace call to put a
delimiter in, if that made sense:
pure_sqls = [s.replace('\0',' ') for s in
'\0'.join(sample+['']).split(';\0') if s!='']

This is also a reasonable example of a filtered list comp, as it'll
suppress any blank entries in the result. Whether this is better or
worse than trimming off the last unit depends on how you want to treat
text after the last semicolon.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Code suggestion - List comprehension


On 12/12/2013 20:40, Shyam Parimal Katti wrote:

Hello,

I have a list of sql queries, some which are split across multiple list
elements e.x.
['drop table sample_table;', 'create table sample_test', '(col1 int);',
'select col1 from', ' sample_test;']

A semi-colon in the string value  indicates the termination of a sql
query. So the expected out come is a conversion to a list of valid sql
queries:
['drop table sample_table;', 'create table sample_test (col1 int);',
'select col1 from sample_test;']

Here is the code that does that:

sample = ['drop table sample_table;', 'create table sample_test', '(col1
int);', 'select col1 from', ' sample_test;']
pure_sqls = []
query_holder= ''
for each_line in sample:
 query_holder += each_line
 if query_holder.endswith(';'):
 pure_sqls.append(query_holder)
 query_holder = ''


Is there a way to do this by eliminating explicit creation of new
list(pure_sqls) and a temporary variable(query_holder)? Using list
comprehension? Though I don't want to put the shorter version in
production(if it is difficult to understand), I am looking if this can
be done with list comprehension since I am trying to learn list
comprehension by using it in such scenarios.



I don't think this can be done with a list comprehension.  As you appear 
to have a perfectly good piece of code, if it ain't broke, don't fix it 
:)  Maybe change one line.


if query_holder[-1] == ';':

--
My fellow Pythonistas, ask not what our language can do for you, ask 
what you can do for our language.


Mark Lawrence

--
https://mail.python.org/mailman/listinfo/python-list

Re: [newbie] trying socket as a replacement for nc

On Fri, Dec 13, 2013 at 8:27 AM, Dan Stromberg  wrote:
> On Thu, Dec 12, 2013 at 6:16 AM, Grant Edwards  
> wrote:
>
>>> Sockets reserve the right to split one socket.send() into multiple
>>> socket.recv()'s on the other end of the communication, or to aggregate
>>> multiple socket.send()'s into a single socket.recv() - pretty much any way
>>> the relevant IP stacks and communications equipment feel like for the sake
>>> of performance or reliability.
>>
>> Just to be pedantic: _TCP_ sockets reserver that right.  UDP sockets
>> do not, and do in fact guarantee that each message is discrete.  [It
>> appears that the OP is undoubtedly using TCP sockets.]
>
> I haven't done a lot of UDP, but are you pretty sure UDP can't at
> least fragment large packets?  What's a router or switch to do if the
> Path MTU isn't large enough for an original packet?
>
> http://www.gamedev.net/topic/343577-fragmented-udp-packets/

I'm no expert on this (mostly I do TCP, or UDP with fairly small
packets), but the packet should be reassembled at the far end. When
your application comes to receive it, it'll receive the entire UDP
packet as a whole.

UDP fragmentation has several problems. First, if any fragment is
lost, it won't be retransmitted (as TCP will), so the whole datagram
is lost. And secondly, if you stream data across the network in a
series of packets just a little too large to fit, each one will get
split in two and you'll end up with twice as many packets going out,
ergo abysmal performance. With TCP, there's the chance that the sender
and receiver can between them figure out what packet size to use (cf
path MTU discovery), but that won't happen with UDP unless the
application consciously does it. So it's something to be cautious of
in terms of performance, but if you want to send large UDP packets
because they make sense, just go ahead and do it.

Now, if you want reliability AND datagrams, it's a lot easier to add
boundaries to a TCP stream (sentinel or length prefixes) than to add
reliability to UDP...

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Tracking the status of python script execution

2013-12-12 Thread Ned Batchelder


On 12/11/13 2:26 PM, Shyam Parimal Katti wrote:

Hello All,

I am looking for a library that can help me trace the status of a live
python script execution. i.e if I have a python script `x.py` with 200
lines, when I execute the script with `python x.py`, is there a way to
trace the status of this execution in terms of number of lines executed
so far?

Background: We have a Web page with "Run" button that executes the
program `x.py` when a user clicks it. We were looking of a way to keep
the user informed about the status of run by using: (no. of lines
executed/total lines) *100. Since the script `x.py` is running multiple
sql queries, it usually won't be the case that the script would complete
within few seconds of its execution.


Using sys.settrace, you could write a tool to track the lines being 
executed in any Python program.  But your problem is different than that 
in two ways:


1) You don't need to track any Python program, you need to track your 
particular Python program.


2) You want the output to take into account the "total number of lines", 
which means you have to somehow configure it ahead of time.


Both of these factors point to using a more specialized approach, by way 
of modifying your program.  I like Chris' idea of simply tracking the 
progress of the SQL queries since they are taking the time.



--
Ned Batchelder, http://nedbatchelder.com

--
https://mail.python.org/mailman/listinfo/python-list

Re: Tree library - multiple children


El 12/12/13 16:26, Neil Cerutti escribió:

On 2013-12-12, Ricardo Aráoz  wrote:

I need to use a tree structure. Is there a good and known library?
Doesn't have to be binary tree, I need to have multiple children per node.

Have you tried nested lists?

[[1, 2], [3, 4]

Can represent

  root
  /  \
   1-23-4

Python makes it very easy to manipulate such a structure. It
isn't clear that you need more than that yet.



And what if "2" has a couple of children? And one of those children has 
children of it's own?
You see, I will be needing multiple levels and will need to know if a 
node is already there at some level, and be able to add a child to that 
node on the fly, and to be able to traverse the tree in different ways, 
so I would eventually develop a tree library which is what I'm looking for.


--
https://mail.python.org/mailman/listinfo/python-list

Re: Tree library - multiple children


El 12/12/13 15:56, Terry Reedy escribió:

On 12/12/2013 1:14 PM, Ricardo Aráoz wrote:

I need to use a tree structure. Is there a good and known library?


Search tools, both for the web and on pypi.python.org, are your friend.



I thought it was obvious that I've already looked around.
I'm using treelib right now but as my post says I was wondering if there 
was "a good and known library". I wanted to gather the previous 
experience of members of the list.


--
https://mail.python.org/mailman/listinfo/python-list

Re: Tree library - multiple children


El 12/12/13 18:01, Mark Lawrence escribió:

On 12/12/2013 18:56, Terry Reedy wrote:

On 12/12/2013 1:14 PM, Ricardo Aráoz wrote:

I need to use a tree structure. Is there a good and known library?


Search tools, both for the web and on pypi.python.org, are your friend.



stackoverflow is another useful site to search and see also
http://kmike.ru/python-data-structures/.



Nice site, thanks. Unfortunately there are tries and all kind of search 
trees, but not simple unbalanced, multiple children trees.

Thanks.

--
https://mail.python.org/mailman/listinfo/python-list

Re: Code suggestion - List comprehension

On Fri, Dec 13, 2013 at 7:40 AM, Shyam Parimal Katti  wrote:
> sample = ['drop table sample_table;', 'create table sample_test', '(col1
> int);', 'select col1 from', ' sample_test;']
> pure_sqls = []
> query_holder= ''
> for each_line in sample:
> query_holder += each_line
> if query_holder.endswith(';'):
> pure_sqls.append(query_holder)
> query_holder = ''

By the way, side point. It's generally considered good programming
practice to use shorter names for short-lived variables and longer
names for things that hang around. I'd spell this slightly
differently:

sample = ['drop table sample_table;', 'create table sample_test',
'(col1 int);', 'select col1 from', ' sample_test;']
pure_sqls = []
cur = ''
for line in sample:
cur += line
if cur.endswith(';'):  # or line.endswith, or line[-1]==';'
pure_sqls.append(cur)
cur = ''

The short one-token names go with the short usage; the longer name
pure_sqls outlives them. Makes it easier to figure out what's
important and what's intermediate. The name 'cur' there is debatable;
I use it all over the place as a generic accumulator for the "current"
whatever I'm working with, you might prefer to use "query" or even
"q", take your pick.

Makes no difference to the code, but might make it easier for someone
to glance over your code and figure out what it's doing.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Downloading multiple files based on info extracted from CSV


On 12/12/2013 21:43, Matt Graves wrote:

I have a CSV file containing a bunch of URLs I have to download a file from for 
clients (Column 7) and the clients names (Column 0) I tried making a script to 
go down the .csv file and just download each file from column 7, and save the 
file as [clientname].csv

I am relatively new to python, so this may be way off but…

import urllib
import csv
urls = []
clientname = []


I assume clientnames.



###This will set column 7 to be a list of urls
with open('clients.csv', 'r') as f:
 reader = csv.reader(f)
 for column in reader:
 urls.append(column[7])

###And this will set column 0 as a list of client names
with open('clients.csv', 'r') as g:
 reader = csv.reader(g)
 for column in reader:
 clientname.append(column[0])


You could do the above in one hit.

with open('clients.csv', 'r') as f:
 reader = csv.reader(f)
 for row in reader:
 urls.append(row[7])
 clientnames.append(row[0])

Note that you're reading rows, not columns.



###This SHOULD plug in the URL for F, and the client name for G.


What makes you think this, f and g are file handles?


def downloadFile(urls, clientname):
 urllib.urlretrieve(f, "%g.csv") % clientname



If you want one file at a time you'd want url, clientname.



downloadFile(f,g)


I think you want something like.

for url, clientname in zip(urls, clientnames):
downloadFile(url, clientname)



When I run it, I get : AttributeError: 'file' object has no attribute 'strip'



When you get a traceback like this please cut and paste all it of, not 
just the last line.  Here it seems likely that your call to downloadFile 
doesn't like you passing in the file handle as I've explained above (I 
hope :)


--
My fellow Pythonistas, ask not what our language can do for you, ask 
what you can do for our language.


Mark Lawrence

--
https://mail.python.org/mailman/listinfo/python-list

Re: Downloading multiple files based on info extracted from CSV

On Fri, Dec 13, 2013 at 8:43 AM, Matt Graves  wrote:
> ###This SHOULD plug in the URL for F, and the client name for G.
> def downloadFile(urls, clientname):
> urllib.urlretrieve(f, "%g.csv") % clientname
>
> downloadFile(f,g)
>
> When I run it, I get : AttributeError: 'file' object has no attribute 'strip'

When showing errors like this, you really need to copy and paste.
Fortunately, I can see where the problem is, here. You're referencing
the file object still in f, which is now a closed file object, instead
of the parameter urls.

But you're also passing f and g as parameters, instead of urls and
clientname. In fact, the downloadFile function isn't really achieving
much; you'd do better to simply inline its code into the main routine
and save yourself the hassle.

While you're at it, there are two more problems in that line of code.
Firstly, you're going to save everything into a file called "%g.csv",
and then try to modulo the return value of urlretrieve with the
clientname; I think you want the close parens at the very end of that
line. And secondly, %g is a floating-point encoder - you want %s here,
or simply use string concatenation:

urllib.urlretrieve(urls, clientname + ".csv")

Except that those are your lists, so that won't work without another
change. We'll fix that later...

> ###This will set column 7 to be a list of urls
> with open('clients.csv', 'r') as f:
> reader = csv.reader(f)
> for column in reader:
> urls.append(column[7])
>
> ###And this will set column 0 as a list of client names
> with open('clients.csv', 'r') as g:
> reader = csv.reader(g)
> for column in reader:
> clientname.append(column[0])

You're reading the file twice. There's no reason to do that; you can
read both columns at once. (By the way, what you're iterating over is
actually rows; for each row that comes out of the reader, do something
with one element from it. So calling it "column" is a bit confusing.)
So now we come to a choice. Question: Is it okay to hold the CSV file
open while you do the downloading? If it is, you can simplify the code
way way down:

import urllib
import csv

# You actually could get away with not using a with
# block here, but may as well keep it for best practice
with open('clients.csv') as f:
for client in csv.reader(f):
urllib.urlretrieve(client[7], client[0] + ".csv")

Yep, that's it! That's all you need. But retrieving all that might
take a long time, so it might be better to do all your CSV reading
first and only *then* start downloading. In that case, I'd make a
single list of tuples:

import urllib
import csv

clients = []
with open('clients.csv') as f:
for client in csv.reader(f):
clients.append((client[7], client[0] + ".csv"))

for client in clients:
urllib.urlretrieve(client[0], client[1])

And since the "iterate and append to a new list" idiom is so common,
it can be simplified down to a list comprehension; and since "call
this function with this tuple of arguments" is so common, it has its
own syntax. So the code looks like this:

import urllib
import csv

with open('clients.csv') as f:
clients = [client[7], client[0]+".csv" for client in csv.reader(f)]

for client in clients:
urllib.urlretrieve(*client)

Again, it's really that simple! :)

Enjoy!

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Downloading multiple files based on info extracted from CSV

2013-12-12 Thread John Gordon

In <[email protected]> Matt Graves 
 writes:

> import urllib
> import csv
> urls = []
> clientname = []

> ###This will set column 7 to be a list of urls
> with open('clients.csv', 'r') as f:
> reader = csv.reader(f)
> for column in reader:
> urls.append(column[7])

> ###And this will set column 0 as a list of client names
> with open('clients.csv', 'r') as g:
> reader = csv.reader(g)
> for column in reader:
> clientname.append(column[0])

> ###This SHOULD plug in the URL for F, and the client name for G.
> def downloadFile(urls, clientname):
> urllib.urlretrieve(f, "%g.csv") % clientname

> downloadFile(f,g)

> When I run it, I get : AttributeError: 'file' object has no attribute
> 'strip'

I think you're passing the wrong arguments to downloadFile().  You're
calling downloadFile(f, g), but f and g are file objects.  Don't you want
to pass urls and clientname instead?

Even if the correct arguments are passed to downloadFile, I think you're
using them incorrectly.  You don't even use the urls argument, and
clientname is supposed to be a list, so why aren't you looping through
it?

You aren't using string interpolation correctly on the call to urlretrieve.
Assuming your intent was to build a string and pass it as the second
argument, you have the close-parenthesis in the wrong place.  The call
should look like this:

urllib.urlretrieve(f, "%g.csv" % clientname)

"%g" returns a floating-point value.  Did you mean "%s" instead?)

-- 
John Gordon Imagine what it must be like for a real medical doctor to
[email protected] 'House', or a real serial killer to watch 'Dexter'.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Code suggestion - List comprehension

2013-12-12 Thread Ben Finney

Shyam Parimal Katti  writes:

> A semi-colon in the string value  indicates the termination of a sql query.
> So the expected out come is a conversion to a list of valid sql queries:
> ['drop table sample_table;', 'create table sample_test (col1 int);',
> 'select col1 from sample_test;']

I presume these strings are split from lines of input. If so, you would
be better advised to use an SQL parsing library in the first place
https://pypi.python.org/pypi/sqlparse/>, to get distinct SQL
statements from a text stream.

-- 
 \   “We must find our way to a time when faith, without evidence, |
  `\disgraces anyone who would claim it.” —Sam Harris, _The End of |
_o__) Faith_, 2004 |
Ben Finney

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Downloading file


On 12/12/2013 20:45, Matt Graves wrote:

I have direct links to a number of csv files to download. Copying and pasting 
it to my browser would take too long, how would i go to this site for example 
and get the file? Right when you go to the site the download should start

www.example.com/files/document.csv


Have a look in the Python docs for "urlretrieve".
--
https://mail.python.org/mailman/listinfo/python-list

Re: Tree library - multiple children

2013-12-12 Thread Ethan Furman


On 12/12/2013 02:01 PM, Ricardo Aráoz wrote:

El 12/12/13 15:56, Terry Reedy escribió:

On 12/12/2013 1:14 PM, Ricardo Aráoz wrote:

I need to use a tree structure. Is there a good and known library?


Search tools, both for the web and on pypi.python.org, are your friend.


I thought it was obvious that I've already looked around.


Why would that be obvious from the one line you wrote?  Python values explicitness over implicitness, and so do many of 
its users.


In other words, tell us what you've done, don't assume we're mind readers. ;)  (Unfortunately, many before you have been 
too lazy to do any work themselves and just post one line questions -- so we're a bit leery of such postings.)




I'm using treelib right now but as my post says I was wondering if there was "a good 
and known library". I wanted to
gather the previous experience of members of the list.


Now that you've given some background you may get better answers.

--
~Ethan~
--
https://mail.python.org/mailman/listinfo/python-list

Re: min max from tuples in list