Re: Writing Donald E. Knuth based code in Python, cont'd

[Juhani Ylikoski]

There were there two (2) bugs in the code that I posted.  Thanks anyway.

A. J. Y.

"Vincent Vande Vyvre"  kirjoitti 
viestissä:mailman.3596.1352758176.27098.python-l...@python.org...


Le 12/11/12 22:02, Juhani Ylikoski a écrit :

Following comes a working, debugged Python program which computes the
permutations of the integers 1, 2, 3 - n after Donald E. Knuth.  I
present it as an example of writing straightforward, easy Knuth-based
code in a language with no GOTO statement.

The Python program has been written after the DFA construct I
previously discussed in this newsgroup, and after Knuth's discussion
of the solution of the problem; and according the (very good)
discussions in this newsgroup.  To my opinion, it no more is a "crow's
nest" as they say in Finnish.

This program was needed for a real problem, namely computing optimal
tournament tables for a Bughouse (Tandem) chess tournament.  See

http://en.wikipedia.org/wiki/Bughouse_chess

Knuth became criticized in the newsgroup; but to my opinion his books
are still useful and nontrivially needed.


---


yours sincerely, Antti J Ylikoski
Helsinki, Finland
PhD student in the Aalto University


Thanks,

One comment in:

  def E1(self): # Phase 1 in Knuth's text
  self.app = self.listofPerm.append(self.a[1:self.n+1])
  return self.E2 # next state: E2

append() return None and self.app is no longer used in the code.

Missing something ?

--
Vincent V.V.
Oqapy  . Qarte
 . PaQager  


--
http://mail.python.org/mailman/listinfo/python-list

Re: Bugs: Content-Length not updated by reused urllib.request.Request / has_header() case-sensitive

[Johannes Kleese]

Terry Reedy wrote:
> On 11/12/2012 10:52 AM, Johannes Kleese wrote:

>> Tested with Python 3.1.3 and Python 3.1.4.
> 
> 3.1 only gets security fixes. Consider upgrading. 

Stuck with Debian on a server, thus stuck with 3.1 on development machine.

> exhibits the same behavior in 3.3.0 of printing ('Content-length', '1')
> in the last output. I agree that that looks wrong, but I do not know if
> such re-use is supposed to be supported.

The Request object should then either get it right on re-use (which I'd
prefer), or block re-use.

>> While at it, I noticed that urllib.request.Request.has_header() and
>> .get_header() are case sensitive,
> 
> Python is case sensitive.

True, of course, but

>> HTTP headers are not (RFC 2616, 4.2).

and the functions work on HTTP data, not Python data. After all, we are
lucky to have functions here and not just a dictionary.


Anyway, thanks for reporting!
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Getting module path string from a class instance

[Steven D'Aprano]

On Tue, 13 Nov 2012 07:54:32 +, Some Developer wrote:

> On 13/11/2012 07:19, Steven D'Aprano wrote:
>> On Tue, 13 Nov 2012 06:38:31 +, Some Developer wrote:
>>
>>> I'm trying to find a way to get a string of the module path of a
>>> class.
>>>
>>> So for instance say I have class Foo and it is in a module called
>>> my.module. I want to be able to get a string that is equal to this:
>>> "my.module.Foo". I'm aware of the __repr__ method but it does not do
>>> what I want it to do in this case.
>>>
>>> Can anyone offer any advice at all?
>> py> from multiprocessing.pool import Pool py> repr(Pool)
>> ""
>>
>> Seems pretty close to what you ask for. You can either pull that string
>> apart:
>>
>> py> s = repr(Pool)
>> py> start = s.find("'")
>> py> end = s.rfind("'")
>> py> s[start+1:end]
>> 'multiprocessing.pool.Pool'
>>
>> or you can construct it yourself:
>>
>> py> Pool.__module__ + '.' + Pool.__name__ 'multiprocessing.pool.Pool'
>>
>>
> Yeah I considered doing it this way but was wary of that method because
> of possible changes to the implementation of the __repr__ method in the
> upstream code. If the Django developers don't consider the __repr__
> method a public API then it could change in the future breaking my code.

I didn't call SomeClass.__repr__. That is an implementation detail of 
SomeClass, and could change.

I called repr(SomeClass), which calls the *metaclass* __repr__. That is 
less likely to change, although not impossible.


If you're worried, just use the second way:

SomeClass.__module__ + '.' + SomeClass.__name__
 


> Of course this might not happen but I was hoping that there was a more 
> generic way of doing it that did not rely on a certain implementation 
> being in existence.

SomeClass.__name__ is the official way to get the name of a class; 
SomeClass.__module__ is the official way to get the name of the module or 
package it comes from.


-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Understanding Code

[subhabangalore]

Dear Group,
To improve my code writing I am trying to read good codes. Now, I have received 
a code,as given below,(apology for slight indentation errors) the code is 
running well. 
Now to comprehend the code, I am looking to understand it completely.

class Calculate:
  def __init__(self):
self.prior = {}  
self.total = {}  
self.count = 0   
  def add(self, cls, obs):
self.prior[cls] = self.prior.get(cls, 0) + 1
for idx, val in enumerate(obs):
key = cls, idx, val
self.total[key] = self.total.get(key, 0) + 1
self.count += 1
  def discr(self, cls, obs):
result = self.prior[cls]/self.count
for idx, val in enumerate(obs):
freq = self.total.get((cls, idx, val), 0)
result *= freq/self.prior[cls]
return result
  def classify(self, obs):
candidates = [(self.discr(c, obs), c) for c in self.prior]
return max(candidates)[1]

I am not understanding many parts of it, I am understanding many parts of it 
also.

So I am looking for an exercise what are the things I should know to understand 
it, (please do not give answers I would get back with the answers in a week and 
would discuss even how to write better than this).

If any one of the expert members of the room kindly help me to do this.

Thanking You in Advance,
Regards,
Subhabrata. 
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Simple Question regarding running .py program

[Dave Angel]

On 11/12/2012 09:45 PM, Caroline Hou wrote:

> 
> Hi Dave!
> 
> thank you very much for your quick reply! I did manage to get the program run 
> from cmd.exe. 
> So does it mean that if I want to use python interactively,I should use the 
> interpreter,while if I just want to run a python program, I should use DOS 
> shell instead?

The DOS shell is one answer that settled both of your original
questions.  It's also how I run about 90% of the time.  But different
people have different preferences.

The interpreter is great for experimenting.  But if you have non-trivial
code (presumably written in a file), then you have to import it.  Which
means references to the stuff there are done with mymodule.myfunction.
That can get tiresome after a while.  And if you have to change the
source file, it's not always safe to reload it (so I never do).  If I've
imported something, and that something has changed, I quit the
interpreter and start it over.

When you're running the script as a whole from the DOS box, it's always
a clean start.

> Also, how could I edit my script? I have sth called "IDLE" installed along 
> with python. Is it the right place to write/edit my script?

Any text editor will do, but especially one with some knowledge of the
Python syntax.  Don't use Notepad.  I've never used IDLE (I use a
commercial one called Komodo IDE), so I can't say how good it is.  Many
people love IDLE, though.

The thing is, any IDE will require some setup (setting directory paths,
project settings, etc.), and some getting used to.  Some don't work very
well for GUI programs, others truncate traceback listings (maybe giving
you a GUI view of the same information).  Some apparently won't even let
you copy/paste a traceback into a mail message.  So it's very useful to
also get thoroughly acquainted with the cmd prompt.

> Sorry about these semi-idiot questions but it is really hard to find an 
> article or book that covers such basic stuffs! 

Nothing wrong with those questions.  Welcome to Python-list.

-- 

DaveA
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Getting module path string from a class instance

[Dave Angel]

On 11/13/2012 01:38 AM, Some Developer wrote:
> I'm trying to find a way to get a string of the module path of a class.
>
> So for instance say I have class Foo and it is in a module called
> my.module. I want to be able to get a string that is equal to this:
> "my.module.Foo". I'm aware of the __repr__ method but it does not do
> what I want it to do in this case.
>
> Can anyone offer any advice at all?

So you have:

import my.module
theclass = my.module.Foo
print tellme(theClass)

and you want to know how to write tellme?  Why not just change it to
take a string, and pass it  "my.module.Foo" ?

If you have stored the class away somewhere, and want to figure it out
from there, you could look at the __module__ attribute.  that'll tell
you the module name, but not the class name.


If you're really asking how to get that string from an INSTANCE of the
class, then try the __class__ attribute of that instance.




-- 

DaveA

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Understanding Code

[Peter Otten]

subhabangal...@gmail.com wrote:

> Dear Group,
> To improve my code writing I am trying to read good codes. Now, I have
> received a code,as given below,(apology for slight indentation errors) the
> code is running well. Now to comprehend the code, I am looking to
> understand it completely.
> 
> class Calculate:
>   def __init__(self):
> self.prior = {}
> self.total = {}
> self.count = 0
>   def add(self, cls, obs):
> self.prior[cls] = self.prior.get(cls, 0) + 1
> for idx, val in enumerate(obs):
> key = cls, idx, val
> self.total[key] = self.total.get(key, 0) + 1
> self.count += 1
>   def discr(self, cls, obs):
> result = self.prior[cls]/self.count
> for idx, val in enumerate(obs):
> freq = self.total.get((cls, idx, val), 0)
> result *= freq/self.prior[cls]
> return result
>   def classify(self, obs):
> candidates = [(self.discr(c, obs), c) for c in self.prior]
> return max(candidates)[1]
> 
> I am not understanding many parts of it, I am understanding many parts of
> it also.
> 
> So I am looking for an exercise what are the things I should know to
> understand it, (please do not give answers I would get back with the
> answers in a week and would discuss even how to write better than this).

Start with running the code for the simplest piece of the class:
>>> c = Calculate()
>>> c.add("x", [1,2,3])

Then inspect the attributes:

>>> c.prior
{'x': 1}
>>> c.total
{('x', 2, 3): 1, ('x', 1, 2): 1, ('x', 0, 1): 1}
>>> c.count

Now read the code for Calculate.add(). Do you understand what

> self.prior[cls] = self.prior.get(cls, 0) + 1

does? Experiment with a dict and its get() method in the interactive 
interpreter. Next to the loop.

> for idx, val in enumerate(obs):
> key = cls, idx, val
> self.total[key] = self.total.get(key, 0) + 1
> self.count += 1

Do you understand what enumerate() does? If not read its documentation with

>>> help(enumerate)

Do you understand what key looks like? If you don't add a print statement

> for idx, val in enumerate(obs):
> key = cls, idx, val
  print key
> self.total[key] = self.total.get(key, 0) + 1
> self.count += 1

What does

> self.total[key] = self.total.get(key, 0) + 1

do? Note that this line is very similar to

> self.prior[cls] = self.prior.get(cls, 0) + 1

which you have studied before.

> self.count += 1

This like the rest of your class is left as an exercise. The routine is 
always the same: 

- break parts that you don't understand into smaller parts
- consult the documentation on unknown classes, functions, methods, 
preferrably with help(some_obj) or dir(some_obj)
- run portions of the code or similar code in the interactive interpreter or 
with a little throw-away script.
- add print statements to inspect variables at interesting points in your 
script.

-- 
http://mail.python.org/mailman/listinfo/python-list

Passing functions as parameter (multiprocessing)

[Jean-Michel Pichavant]

Fellows,

I'm having problems understanding an issue with passing function as parameters.

I'm sending some functions to the multiprocessing module (python 2.5 with the 
proper backport).
I'm iterating on a list of functions, however it seems that only the last 
function implementation is used for 
all the subprocesses.

Here's a code that triggers the issue:

 
import multiprocessing

def f1():
print 'I am f1'
def f2(foo):
print 'I am f2 %s' % foo

workers = [
(f1,tuple()),
(f2,(5,)),
]

procs=[]
for func, parameters in workers:
# here it should be decorated, but for this example to be kept simple, the 
function is only wrapped, doing nothing special
def subproc(*args, **kwargs):
return func(*args, **kwargs)
procs.append(multiprocessing.Process(target=subproc, args=parameters))

for proc in procs:
proc.start()
for proc in procs:
proc.join()


Here's the result:
> run test.py
Process Process-1:
Traceback (most recent call last):
  File 
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.2.1-py2.5-linux-i686.egg/multiprocessing/process.py",
 line 237, in _bootstrap
self.run()
  File 
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.2.1-py2.5-linux-i686.egg/multiprocessing/process.py",
 line 93, in run
self._target(*self._args, **self._kwargs)
  File "test.py", line 17, in subproc
return func(*args, **kwargs)
TypeError: f2() takes exactly 1 argument (0 given)
I am f2 5

It looks like the first subprocess is called with f2 instead of f1.

Any idea ?

JM


-- IMPORTANT NOTICE: 

The contents of this email and any attachments are confidential and may also be 
privileged. If you are not the intended recipient, please notify the sender 
immediately and do not disclose the contents to any other person, use it for 
any purpose, or store or copy the information in any medium. Thank you.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Division matrix

[Cleuson Alves]

Thanks, I'm starting to plan now, so I'm still confused with the production
code, but what I need is to divide array 2x2 or 3x3.
I still can not!



2012/11/12 Joshua Landau 

> On 13 November 2012 01:00, Cleuson Alves  wrote:
>
>> Hello, I need to solve an exercise follows, first calculate the inverse
>> matrix and then multiply the first matrix.
>>
>
> This list isn't to give answers for homeworks, and this sounds like one.
> We *do* give help to those who have a specific problem and who preferably
> show that they are trying to help up help them.
>
>
>> I await help.
>>
>
> For what?! You haven't asked a question.
> 1) What do you need. Answer precisely, preferably giving an example output
> for the input you want
> 2) What techniques have you tried? You have given code below in an
> extremely un-obvious manner. What does it do, crash? Is it the wrong
> output? What is wrong about the output?
> 3) What do you *think* has gone wrong? What about the output seems to be
> wrong?
>
>
>> Thank you.
>> follows the code below incomplete.
>>
>
> On the basis that we can get some foreign people who maybe don't have
> English as a commonly used language, muddled grammar isn't that big deal.
> However, when asking for help it is worth checking that you've asked in a
> clear way.
>
>
>> m = [[1,2,3],[4,5,6],[7,8,9]]
>> x = []
>> for i in [0,1,2]:
>> y = []
>> for linha in m:
>> y.append(linha[i])
>> x.append(y)
>>
>> print x
>> [[1, 4, 7], [2, 5, 8], [3, 6, 9]]
>>
>
> Is this the right output? Is this what you *meant* by inverse? As Ian
> Kelly (who is undoubtedly more learned in this area) said, this is a
> transpose, so you have just swapped the one axis with another.
>
>
>> def ProdMatrix(x,b):
>> tamL = len(x)
>> tamC = len(x[0])
>> c = nullMatrix(tamL,tamC)
>> for i in range(tamL):
>> for j in range(tamC):
>> val = 0
>> for k in range(len(b)):
>> val = val + x[i][l]*b[k][j]
>> c[i][j]
>> return c
>>
>
> You haven't given the full code. This crashes because we don't have
> "nullMatrix" defined. It would be nice if we had something we could try to
> run.
>
> Then, the error is: "NameError: global name 'l' is not defined". On the
> line "val = val + x[i][l]*b[k][j]" you use a variable "l" which doesn't
> exist. What should you be using, or should you have created it?
>
> *Then* you get an output of pure 0s. This is because you forgot to put
> your result into c, your new matrix.
> You probably just forgot to finish the line "c[i][j]", which does nothing
> now.
>
> You're actually really close on the second part. I have no idea if you're
> doing what you want for the first part, but it's not what's traditionally
> called an inverse.
>



-- 
Cleuson de Oliveira Alves
Rio de Janeiro - RJ
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Simple Question regarding running .py program

[Ramchandra Apte]

On Tuesday, 13 November 2012 08:15:45 UTC+5:30, Caroline Hou  wrote:
> On Monday, 12 November 2012 21:25:08 UTC-5, Dave Angel  wrote:
> 
> > On 11/12/2012 09:02 PM, Caroline Hou wrote:
> 
> > 
> 
> > > Hi all!
> 
> > 
> 
> > >
> 
> > 
> 
> > > I just started learning Python by myself and I have an extremely simple 
> > > question now!
> 
> > 
> 
> > > I am in my Python interpreter now and I want to open/edit a program 
> > > called nobel.py. But when I typed >>> python nobel.py, it gave me a 
> > > "SyntaxError:invalid syntax”( I've changed to the correct directory)what 
> > > should I do?
> 
> > 
> 
> > > I also want to run the program, but as I double-clicked the program, a 
> > > command window pops up and closes immediately. How can I see the result 
> > > of the program run?
> 
> > 
> 
> > > Could anyone help me please? I am pretty confused here...Thank you!
> 
> > 
> 
> > 
> 
> > 
> 
> > It'd be nice to specify that you're running Windows, and also what
> 
> > 
> 
> > version of the interpreter, although in this case the latter doesn't matter.
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > Go to a shell (cmd.exe), change to the directory containing that script,
> 
> > 
> 
> > and type the command as you did.
> 
> > 
> 
> > 
> 
> > 
> 
> > On linux:  davea@think:~$ python nobel.py
> 
> > 
> 
> > On Windows:   c:\mydir\myscript > python nobel.py
> 
> > 
> 
> > 
> 
> > 
> 
> > If you're already in the python interpreter, then running python is
> 
> > 
> 
> > useless -- it's already running.  In that case, you might want to use
> 
> > 
> 
> > import.  However, I recommend against it at first, as it opens up some
> 
> > 
> 
> > other problems you haven't experience with yet.
> 
> > 
> 
> > 
> 
> > 
> 
> > When you say you "double clicked the program', we have to guess you
> 
> > 
> 
> > might have meant in MS Explorer.  If you do that, it launches a cmd, it
> 
> > 
> 
> > runs the python system, and it closes the cmd.  Blame Windows for not
> 
> > 
> 
> > reading your mind.  If you want the cmd window to stick around, you
> 
> > 
> 
> > COULD end your program with an raw_input function call, but frequently
> 
> > 
> 
> > that won't work.  The right answer is the first one above...   Open a
> 
> > 
> 
> > shell (perhaps with a menu like  DOS BOX), change...
> 
> > 
> 
> > 
> 
> > 
> 
> > That way, when the program finishes, you can see what happened, or
> 
> > 
> 
> > didn't happen, and you can run it again using the uparrow key.
> 
> > 
> 
> > 
> 
> > 
> 
> > BTW, you don't need to send email to both the python-list and to the
> 
> > 
> 
> > newsgroup.  The newsgroup is automatically fed from the list.  But since
> 
> > 
> 
> > you're posting from google groups, that's just one of the bugs.  Many
> 
> > 
> 
> > folks here simply filter out everything from google groups, so your post
> 
> > 
> 
> > is invisible to them.
> 
> > 
> 
> >
> 
> > 
> 
> > 
> 
> > 
> 
> > -- 
> 
> > 
> 
> > 
> 
> > 
> 
> > DaveA
> 
> 
> 
> Hi Dave!
> 
> 
> 
> thank you very much for your quick reply! I did manage to get the program run 
> from cmd.exe. 
> 
> So does it mean that if I want to use python interactively,I should use the 
> interpreter,while if I just want to run a python program, I should use DOS 
> shell instead?
> 
> Also, how could I edit my script? I have sth called "IDLE" installed along 
> with python. Is it the right place to write/edit my script?
> 
> Sorry about these semi-idiot questions but it is really hard to find an 
> article or book that covers such basic stuffs! 
> 
> Thank you!
> 
> 
> 
> Caroline Hou

IDLE is recommended for newbies like you because an IDE requires too much 
configuration.
When you start writing a big project, you can use an IDE.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Passing functions as parameter (multiprocessing)

[MRAB]

On 2012-11-13 12:19, Jean-Michel Pichavant wrote:

Fellows,

I'm having problems understanding an issue with passing function as parameters.

I'm sending some functions to the multiprocessing module (python 2.5 with the 
proper backport).
I'm iterating on a list of functions, however it seems that only the last 
function implementation is used for
all the subprocesses.

Here's a code that triggers the issue:


import multiprocessing

def f1():
 print 'I am f1'
def f2(foo):
 print 'I am f2 %s' % foo

workers = [
 (f1,tuple()),
 (f2,(5,)),
 ]

procs=[]
for func, parameters in workers:
 # here it should be decorated, but for this example to be kept simple, the 
function is only wrapped, doing nothing special
 def subproc(*args, **kwargs):
 return func(*args, **kwargs)
 procs.append(multiprocessing.Process(target=subproc, args=parameters))

for proc in procs:
 proc.start()
for proc in procs:
 proc.join()


Here's the result:

run test.py

Process Process-1:
Traceback (most recent call last):
   File 
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.2.1-py2.5-linux-i686.egg/multiprocessing/process.py",
 line 237, in _bootstrap
 self.run()
   File 
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.2.1-py2.5-linux-i686.egg/multiprocessing/process.py",
 line 93, in run
 self._target(*self._args, **self._kwargs)
   File "test.py", line 17, in subproc
 return func(*args, **kwargs)
TypeError: f2() takes exactly 1 argument (0 given)
I am f2 5

It looks like the first subprocess is called with f2 instead of f1.

I believe the problem is that 'subproc' calls 'func', which is rebound 
on the each iteration.


--
http://mail.python.org/mailman/listinfo/python-list

Re: Passing functions as parameter (multiprocessing)

[Peter Otten]

Jean-Michel Pichavant wrote:

> I'm having problems understanding an issue with passing function as
> parameters.

> Here's a code that triggers the issue:
> 
>  
> import multiprocessing
> 
> def f1():
> print 'I am f1'
> def f2(foo):
> print 'I am f2 %s' % foo
> 
> workers = [
> (f1,tuple()),
> (f2,(5,)),
> ]
> 
> procs=[]
> for func, parameters in workers:

> def subproc(*args, **kwargs):
> return func(*args, **kwargs)
> procs.append(multiprocessing.Process(target=subproc, args=parameters))

Python has late binding, and when the loop has finished the name func is 
bound to f2. You have created multiple subproc functions, but that doesn't 
matter as they all invoke func aka f2.

A possible fix:

def make_subproc(func):
def subproc(*args, **kwargs):
return func(*args, **kwargs)
return subproc

procs=[]
for func, parameters in workers:
procs.append(multiprocessing.Process(target=make_subproc(func), 
args=parameters))


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: stackoverflow quote on Python

[Alister]

On Tue, 13 Nov 2012 03:08:54 +, Mark Lawrence wrote:

> http://stackoverflow.com/questions/tagged/python
> 
> "Python has two major versions (2 and 3) in use which have significant
> differences."
> 
> I believe that this is incorrect.  The warts have been removed, but
> significant differences, not in my book.  If there is agreement about
> there not being significant differences, should stackoverflow be asked
> to change their wording?
I think you are confusing significant with fundamental

they are significant differences because failing to take them into 
account may cause incorrect operation.
they are not fundamental differences because someone familiar with on 
version should have minimal difficulty in understanding the other.




-- 
Lee's Law:
Mother said there would be days like this,
but she never said that there'd be so many!
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: stackoverflow quote on Python

[wxjmfauth]

Le mardi 13 novembre 2012 06:42:19 UTC+1, Steven D'Aprano a écrit :
> On Tue, 13 Nov 2012 03:08:54 +, Mark Lawrence wrote:
> 
> 
> 
> 
> * strings are now proper text strings (Unicode), not byte strings;
> 
 


Let me laugh.

jmf
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Help building a dictionary of lists

[Thomas Bach]

On Mon, Nov 12, 2012 at 11:41:59PM +, Joshua Landau wrote:
> 
> Dict comprehension:
> {i:[] for i in ["Test 1", "Test 2", "Test 3"]}

In Python 2.6 this syntax is not supported. You can achieve the same
there via

dict((i, []) for i in ['Test 1', 'Test 2', 'Test 3'])

Also have a look at ``collections.defaultdict``.

Regards,
Thomas.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: xml data or other?

[shivers . paul]

On Friday, November 9, 2012 12:54:56 PM UTC, Artie Ziff wrote:
> Hello,
> 
> 
> 
> I want to process XML-like data like this:
> 
> 
> 
> 
> 
>   
> 
>   ACPI (Advanced Control Power & Integration) testscript for 2.5 
> kernels.
> 
> 
> 
>   <\description>
> 
>   
> 
>   ltp/testcases/kernel/device-drivers/acpi/ltpacpi.sh
> 
>   <\test_location>
> 
> <\testname>
> 
> 
> 
> 
> 
> After manually editing the data above, the python module 
> 
> xml.etree.ElementTree parses it without failing due to error in the data 
> 
> structure.
> 
> 
> 
> Edits were substituting '/' for '\' on the end tags, and adding the 
> 
> following structure:
> 
> 
> 
> 
> 
> 
> 
>
> 
>  ...
> 
><\testname>
> 
> 
> 
> 
> 
> 
> 
> Is there a name for the format above (perhaps xhtml)?
> 
> I'd like to find a python module that can translate it to proper xml. 
> 
> Does one exist? etree?
> 
> 
> 
> Many thanks!
> 
> az

maybe an xml tool would be better, a good list of xml tools here; 
http://www.xml-data.info
-- 
http://mail.python.org/mailman/listinfo/python-list

Templating and XML modelling

[Martin Sand Christensen]

Hi!

At our IT department we've developed a basic templating system for web 
apps in the spirit of Meld3 (which appears to have been abandoned some 
time ago), based on lxml. Here's what we like about it:


* It's just a library, not a template language
* It uses templates that are valid XHTML
* It's trivial to generate tables and forms from database metadata
* It's trivial to fill named elements á la format strings

While we like it, it's more code to maintain. Between the time when we 
started coding this and now, more new templating systems have appeared 
than I can reasonably evaluate. So now the question is whether we can 
find a good replacement or whether we should publish our code and hope 
that more people will adopt and help maintain it.


So...

1) Can you suggest a good alternative that sounds like a good fit?
2) Does our templating system sound like just what you've been looking for?

--
Martin Sand Christensen
IT Services, Dept. of Electronic Systems
--
http://mail.python.org/mailman/listinfo/python-list

Error messages from format()

[Colin J. Williams]

Is there some way to get more informative error messages from the 
builtin format?


Most messages are such as:
Traceback (most recent call last):
  File "", line 1, in 
ValueError: Invalid conversion specification

This example doesn't point to the first invalid case.

[Dbg]>>> format((25, 31),'{0^9o} a(1:9x}')
Traceback (most recent call last):
  File "", line 1, in 
ValueError: Invalid conversion specification

Basically, I'm trying to make use of the format function with Python 
3.2, but find little in the way of examples in the docs.


Colin W.
--
http://mail.python.org/mailman/listinfo/python-list

Re: Passing functions as parameter (multiprocessing)

[Oscar Benjamin]

On 13 November 2012 12:51, Peter Otten <__pete...@web.de> wrote:
> Jean-Michel Pichavant wrote:
>
>> I'm having problems understanding an issue with passing function as
>> parameters.
>
>> Here's a code that triggers the issue:
>>
>>
>> import multiprocessing
>>
>> def f1():
>> print 'I am f1'
>> def f2(foo):
>> print 'I am f2 %s' % foo
>>
>> workers = [
>> (f1,tuple()),
>> (f2,(5,)),
>> ]
>>
>> procs=[]
>> for func, parameters in workers:
>
>> def subproc(*args, **kwargs):
>> return func(*args, **kwargs)
>> procs.append(multiprocessing.Process(target=subproc, args=parameters))

I don't know if this is to do with the way that the code was
simplified before posting but this subproc function wrapper does
nothing (even after Peter fixed it below). This code is needlessly
complicated for what it does.

> Python has late binding, and when the loop has finished the name func is
> bound to f2. You have created multiple subproc functions, but that doesn't
> matter as they all invoke func aka f2.
>
> A possible fix:
>
> def make_subproc(func):
> def subproc(*args, **kwargs):
> return func(*args, **kwargs)
> return subproc
>
> procs=[]
> for func, parameters in workers:
> procs.append(multiprocessing.Process(target=make_subproc(func),
> args=parameters))

The above is one way to wrap a function but it wraps the function into
a function that is exactly the same as the original function. I assume
that the reason for having the subproc function was to be able to bind
parameters to the function like so:

def make_subproc(func, args):
def subproc():
return func(*args)
return subproc

procs=[]
for func, parameters in workers:
procs.append(multiprocessing.Process(target=make_subproc(func, parameters)))

However, as you have already noticed, multiprocess.Process already has
the machinery to do this. If you pass in a value for args you can
leave out the subproc/make_subproc function and write:

procs=[]
for func, parameters in workers:
procs.append(multiprocessing.Process(target=func, args=parameters))

A loop like this can just be a list comprehension:

procs = [multiprocessing.Process(target=func, args=args) for func,
args in workers]


Oscar
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: stackoverflow quote on Python

[Mark Lawrence]

On 13/11/2012 13:21, wxjmfa...@gmail.com wrote:

Le mardi 13 novembre 2012 06:42:19 UTC+1, Steven D'Aprano a écrit :

On Tue, 13 Nov 2012 03:08:54 +, Mark Lawrence wrote:

* strings are now proper text strings (Unicode), not byte strings;





Let me laugh.

jmf



Presumably because you're looking at yourself in a mirror, and have 
finally realised that you've completely misunderstood the work done with 
unicode in Python 3, specifically Python 3.3?


--
Cheers.

Mark Lawrence.

--
http://mail.python.org/mailman/listinfo/python-list

Re: creating size-limited tar files

[Ian Kelly]

On Tue, Nov 13, 2012 at 3:31 AM, andrea crotti
 wrote:
> but it's a bit ugly.  I wonder if I can use the subprocess PIPEs to do
> the same thing, is it going to be as fast and work in the same way??

It'll look something like this:

>>> p1 = subprocess.Popen(cmd1, shell=True, stdout=subprocess.PIPE, 
>>> stderr=subprocess.PIPE)
>>> p2 = subprocess.Popen(cmd2, shell=True, stdin=p1.stdout, 
>>> stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> p1.communicate()
('', '')
>>> p2.communicate()
('', '')
>>> p1.wait()
0
>>> p2.wait()
0

Note that there's a subtle potential for deadlock here.  During the
p1.communicate() call, if the p2 output buffer fills up, then it will
stop accepting input from p1 until p2.communicate() can be called, and
then if that buffer also fills up, p1 will hang.  Additionally, if p2
needs to wait on the parent process for some reason, then you end up
effectively serializing the two processes.

Solution would be to poll all the open-ended pipes in a select() loop
instead of using communicate(), or perhaps make the two communicate
calls simultaneously in separate threads.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Getting module path string from a class instance

[Some Developer]

On 13/11/2012 08:49, Steven D'Aprano wrote:

On Tue, 13 Nov 2012 07:54:32 +, Some Developer wrote:


On 13/11/2012 07:19, Steven D'Aprano wrote:

On Tue, 13 Nov 2012 06:38:31 +, Some Developer wrote:


I'm trying to find a way to get a string of the module path of a
class.

So for instance say I have class Foo and it is in a module called
my.module. I want to be able to get a string that is equal to this:
"my.module.Foo". I'm aware of the __repr__ method but it does not do
what I want it to do in this case.

Can anyone offer any advice at all?

py> from multiprocessing.pool import Pool py> repr(Pool)
""

Seems pretty close to what you ask for. You can either pull that string
apart:

py> s = repr(Pool)
py> start = s.find("'")
py> end = s.rfind("'")
py> s[start+1:end]
'multiprocessing.pool.Pool'

or you can construct it yourself:

py> Pool.__module__ + '.' + Pool.__name__ 'multiprocessing.pool.Pool'



Yeah I considered doing it this way but was wary of that method because
of possible changes to the implementation of the __repr__ method in the
upstream code. If the Django developers don't consider the __repr__
method a public API then it could change in the future breaking my code.

I didn't call SomeClass.__repr__. That is an implementation detail of
SomeClass, and could change.

I called repr(SomeClass), which calls the *metaclass* __repr__. That is
less likely to change, although not impossible.


If you're worried, just use the second way:

SomeClass.__module__ + '.' + SomeClass.__name__
  


Ah, my mistake. Thanks. That sounds exactly like what I want.


Of course this might not happen but I was hoping that there was a more
generic way of doing it that did not rely on a certain implementation
being in existence.

SomeClass.__name__ is the official way to get the name of a class;
SomeClass.__module__ is the official way to get the name of the module or
package it comes from.

--
http://mail.python.org/mailman/listinfo/python-list

Re: Passing functions as parameter (multiprocessing)

[Peter Otten]

Oscar Benjamin wrote:

> I don't know if this is to do with the way that the code was
> simplified before posting but this subproc function wrapper does
> nothing (even after Peter fixed it below). This code is needlessly
> complicated for what it does.

Jean-Michel's Post had the following comment preceding his subproc() 
definition

>>># here it should be decorated, but for this example to be kept
>>> simple, the function is only wrapped, doing nothing special

which I unhelpfully removed...

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Getting module path string from a class instance

[Some Developer]

On 13/11/2012 09:36, Dave Angel wrote:

On 11/13/2012 01:38 AM, Some Developer wrote:

I'm trying to find a way to get a string of the module path of a class.

So for instance say I have class Foo and it is in a module called
my.module. I want to be able to get a string that is equal to this:
"my.module.Foo". I'm aware of the __repr__ method but it does not do
what I want it to do in this case.

Can anyone offer any advice at all?

So you have:

import my.module
theclass = my.module.Foo
print tellme(theClass)

and you want to know how to write tellme?  Why not just change it to
take a string, and pass it  "my.module.Foo" ?

If you have stored the class away somewhere, and want to figure it out
from there, you could look at the __module__ attribute.  that'll tell
you the module name, but not the class name.


If you're really asking how to get that string from an INSTANCE of the
class, then try the __class__ attribute of that instance.

I'm actually writing a dynamic importer.

Basically when the program starts it queries all the loaded modules for 
a certain package. It then stores the module and class information in a 
database and then when a certain view is called the specific data 
required is loaded by calling importlib.import_module('module.class'). 
This then populates the view with the required data.


Basically the idea is to be able to display any data required without 
having to tightly couple the view code to a specific model type.


Thanks for the help.
--
http://mail.python.org/mailman/listinfo/python-list

Re: creating size-limited tar files

[Ian Kelly]

On Tue, Nov 13, 2012 at 9:07 AM, Ian Kelly  wrote:
> It'll look something like this:
>
 p1 = subprocess.Popen(cmd1, shell=True, stdout=subprocess.PIPE, 
 stderr=subprocess.PIPE)
 p2 = subprocess.Popen(cmd2, shell=True, stdin=p1.stdout, 
 stdout=subprocess.PIPE, stderr=subprocess.PIPE)
 p1.communicate()
> ('', '')
 p2.communicate()
> ('', '')
 p1.wait()
> 0
 p2.wait()
> 0
>
> Note that there's a subtle potential for deadlock here.  During the
> p1.communicate() call, if the p2 output buffer fills up, then it will
> stop accepting input from p1 until p2.communicate() can be called, and
> then if that buffer also fills up, p1 will hang.  Additionally, if p2
> needs to wait on the parent process for some reason, then you end up
> effectively serializing the two processes.
>
> Solution would be to poll all the open-ended pipes in a select() loop
> instead of using communicate(), or perhaps make the two communicate
> calls simultaneously in separate threads.

Sorry, the example I gave above is wrong.  If you're calling
p1.communicate(), then you need to first remove the p1.stdout pipe
from the Popen object.  Otherwise, the communicate() call will try to
read data from it and may "steal" input from p2.  It should look more
like this:

>>> p1 = subprocess.Popen(cmd1, shell=True, stdout=subprocess.PIPE, 
>>> stderr=subprocess.PIPE)
>>> p2 = subprocess.Popen(cmd2, shell=True, stdin=p1.stdout, 
>>> stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> p1.stdout = None
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: creating size-limited tar files

[Ian Kelly]

On Tue, Nov 13, 2012 at 9:25 AM, Ian Kelly  wrote:
> Sorry, the example I gave above is wrong.  If you're calling
> p1.communicate(), then you need to first remove the p1.stdout pipe
> from the Popen object.  Otherwise, the communicate() call will try to
> read data from it and may "steal" input from p2.  It should look more
> like this:
>
 p1 = subprocess.Popen(cmd1, shell=True, stdout=subprocess.PIPE, 
 stderr=subprocess.PIPE)
 p2 = subprocess.Popen(cmd2, shell=True, stdin=p1.stdout, 
 stdout=subprocess.PIPE, stderr=subprocess.PIPE)
 p1.stdout = None

Per the docs, that third line should be "p1.stdout.close()".  :-P
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: stackoverflow quote on Python

[Ethan Furman]

wxjmfa...@gmail.com wrote:

Le mardi 13 novembre 2012 06:42:19 UTC+1, Steven D'Aprano a écrit :

On Tue, 13 Nov 2012 03:08:54 +, Mark Lawrence wrote:

* strings are now proper text strings (Unicode), not byte strings;

Let me laugh.


*plonk*
--
http://mail.python.org/mailman/listinfo/python-list

Re: xml data or other?

[Dave Angel]

On 11/09/2012 07:54 AM, Artie Ziff wrote:
> Hello,
>
> I want to process XML-like data like this:
>
> 
> 
> ACPI (Advanced Control Power & Integration) testscript for 2.5
> kernels.
>
> <\description>
> 
> ltp/testcases/kernel/device-drivers/acpi/ltpacpi.sh
> <\test_location>
> <\testname>
> 
>
>
> Is there a name for the format above (perhaps xhtml)?

The only word I can think of is "broken."  xml and html and xhtml all
use forward slashes.

> I'd like to find a python module that can translate it to proper xml.
> Does one exist? etree?
>

I think you've already figured it out.Just take your description and
turn it into Python.  in other words, replace all "<\" with "" with " /", although your example doesn't happen to have
any of these.  Tack a  xml header on, and try to parse it with etree. 
If you can't, then let someone manually fix it.

Or better, fix the program upstream that's creating this mess.  There
isn't a reliable way to "fix" all the possible broken xml it might be
creating, without reverse engineering it.



-- 

DaveA

-- 
http://mail.python.org/mailman/listinfo/python-list

Upcoming Webinar "SQLAlchemy Creator Mike Bayer on How and Why to Integrate Akiban and SQLAlchemy"

[Robert Riegel]

 [image: Akiban]
*Upcoming Webinar:*
"SQLAlchemy Creator Mike Bayer on How and Why to Integrate Akiban and
SQLAlchemy"

Hi guys,

I wanted to take a minute and invite you and the Python user group to our
next free webinar on SQLAchemy:
Presented by: Mike Bayer, Creator of SQLAchemy,Ori Herrnstadt, Creator of
Akiban Date:  Thursday, December 13, 2012[image: Register
Here]Time:
 2:30 pm ET / 11:30 am PT

*SQLAlchemy + Akiban: A very interesting combo*

What good is a database if you can’t use it with the language or framework
you depend on? From a developer’s perspective, not very. Even though Akiban
understands SQL, developers often work at higher level with
Object-Relational Mappers (ORM) like Hibernate, SQLAlchemy, or
ActiveRecord. Very few developers operate on pure SQL these days.

Clearly, we’ve been thinking about this question at Akiban. As we’ve been
developing our new database we’ve taken time to work closely with open
source developers like Mike Bayer, creator of the SQLAlchemy project, to
make sure that developers have access to libraries that understand the
Akiban database.

Akiban's unique storage approach means that not only does SQL run a lot
faster because most joins are free, but also it is possible to access a
full object directly within the database. Through the use of Akiban,
businesses can quickly implement new services such as real-time dynamic
search, social-local-mobile applications, relationship discovery and
real-time insight. Akiban enables immediate access to your operational data
without negatively impacting operational performance. This also eliminates
the Extract-Transform-Load effort and time delay to transfer operational
data into a data mart or data warehouse.

SQLAlchemy is the first ORM we’ve approached about table grouping, our
novel approach to storing data and returning nested result sets in JSON. As
Bayer explains, while other databases have provided a JSON view of a
relational database, Akiban’s unique approach is at a much deeper level.
We’re not just returning data as JSON, we’ve decided to store our data in a
hierarchical format on disk. It didn’t take Bayer long to adapt SQLAlchemy
to these nested results, nor did it take long to write an extension to
psyopg2 that could understand the output of a query that involved table
groups.

What you'll take away from this webinar:

   - Brief intro to Akiban.
   - Snapshot of SQLAlchemy 0.8, new features and additions.
   - Example of how easy it is to execute a query that returns a nested
   resultset in Python using SQLAlchemy.
   - Overview of two recently developed projects:


   1. *Akiban for Python* - an extension for the psycopg2 DBAPI.
   2. *SQLAlchemy Akiban* - an Akiban dialect and ORM extension for
   SQLAlchemy.

Duration: 1 hour

[image: 
Register]

Hope to see you there.

Regards,
Akiban Team

*P.S.* If you are unable to make it to the live webinar, register
anyway,
and we'll share the recording after the event.

*Akiban* 560 Harrison Avenue, Suite 411 Boston, MA 02445 United States of
America
Contact Us 
-- 
Robert Riegel @akiban 
skype: robert.c.riegel.III | 617-826-9990
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How to only get a list of the names of the non-directory files in current directory ('.')?

[emile]

On 11/06/2012 03:12 PM, iMath wrote:


how to get a list of names of everything in the current directory ?


Start by working through the tutorial to get familiar with python at
   http://docs.python.org/2/tutorial/

then for your specific question, review the content at
http://www.diveintopython.net/file_handling/os_module.html

Emile


BTW, googling for "python how to get a list of names of everything in 
the current directory" yields some good information as well.  Google is 
your friend for this level of question.  Not sure anymore beyond that...






--
http://mail.python.org/mailman/listinfo/python-list

Re: stackoverflow quote on Python

[Thomas Rachel]

Am 13.11.2012 14:21 schrieb wxjmfa...@gmail.com:


* strings are now proper text strings (Unicode), not byte strings;


Let me laugh.


Do so.


Thomas
--
http://mail.python.org/mailman/listinfo/python-list

Re: Error messages from format()

[Steven D'Aprano]

On Tue, 13 Nov 2012 10:08:59 -0500, Colin J. Williams wrote:

> Is there some way to get more informative error messages from the
> builtin format?

Yes -- post a feature request on the Python bug tracker, then wait until 
Python 3.4 comes out in about 16 months.

:(


> Most messages are such as:
> Traceback (most recent call last):
>File "", line 1, in 
> ValueError: Invalid conversion specification
> 
> This example doesn't point to the first invalid case.

Better error messages would be valuable.


> [Dbg]>>> format((25, 31),'{0^9o} a(1:9x}')
> Traceback (most recent call last):
>File "", line 1, in 
> ValueError: Invalid conversion specification

I see at least three problems.

(1) The first brace substitution is missing the colon between the 
argument selector "0" and the format spec "^9o": should be "{0:^9o}".

(2) The second format string has an opening round bracket instead of 
brace: (1:9x}

(3) But you are confusing the str.format method with the format function. 
The format function doesn't take brace substitutions!

The string format method takes a template including brace substitutions, 
plus multiple "objects to be substituted", like this:

py> '{0:^9o} a{1:9x}'.format(25, 31)
'   31 a   1f'

In this case, the template '{0:^9o} a{1:9x}' requires two arguments since 
it has two substitutions, {0} and {1}. Each substitution has a format 
spec following the colon: {0:^9o} and {1:9x}

But the format function only takes a single "object to be substituted", 
and so doesn't take a brace substitution. Instead, it just takes the 
format spec part:


py> format(25, '^9o')
'   31'
py> format(31, '^9o')
'   37'

format will not split a tuple into multiple arguments for you, since the 
tuple is considered a single object.



-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Passing functions as parameter (multiprocessing)

[Jean-Michel Pichavant]

- Original Message -
> Oscar Benjamin wrote:
> 
> > I don't know if this is to do with the way that the code was
> > simplified before posting but this subproc function wrapper does
> > nothing (even after Peter fixed it below). This code is needlessly
> > complicated for what it does.
> 
> Jean-Michel's Post had the following comment preceding his subproc()
> definition
> 
> >>># here it should be decorated, but for this example to be kept
> >>> simple, the function is only wrapped, doing nothing special
> 
> which I unhelpfully removed...
> 
> --
> http://mail.python.org/mailman/listinfo/python-list
> 

Anyway thank you all, I solved the issue with Peter's code.
func was rebound after each iteration, though I though it wasn't.

I would not have figured it out by myself, my mind was lost in the mysteries of 
subprocesses, decorated and wrapped methods.

Cheers,

JM








-- IMPORTANT NOTICE: 

The contents of this email and any attachments are confidential and may also be 
privileged. If you are not the intended recipient, please notify the sender 
immediately and do not disclose the contents to any other person, use it for 
any purpose, or store or copy the information in any medium. Thank you.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: how to simulate tar filename substitution across piped subprocess.Popen() calls?

[Thomas Rachel]

Am 09.11.2012 02:12 schrieb Hans Mulder:


That's what 'xargs' will do for you.  All you need to do, is invoke
xargs with arguments containing '{}'.  I.e., something like:

cmd1 = ['tar', '-czvf', 'myfile.tgz', '-c', mydir, 'mysubdir']
first_process = subprocess.Popen(cmd1, stdout=subprocess.PIPE)

cmd2 = ['xargs', '-I', '{}', 'sh', '-c', "test -f %s/'{}'" % mydir]
second_process = subprocess.Popen(cmd2, stdin=first_process.stdout)


After launching second_process, it might be useful to 
firstprocess.stdout.close(). If you fail to do so, your process is a 
second reader which might break things apart.


At least, I once hat issues with it; I currently cannot recapitulate 
what these were nor how they could arise; maybe there was just the open 
file descriptor which annoyed me.



Thomas
--
http://mail.python.org/mailman/listinfo/python-list

Welcome to Horror !!!!!!

[MoneyMaker]

This site is intended for all horror fans.

The site have been collected and collected links to horror fans

and joining a membership you can add the best moments of

terror.  http://horrorhorrorhorror.webs.com/
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Error messages from format()

[Colin J. Williams]

On 13/11/2012 1:38 PM, Steven D'Aprano wrote:

On Tue, 13 Nov 2012 10:08:59 -0500, Colin J. Williams wrote:


Is there some way to get more informative error messages from the
builtin format?


Yes -- post a feature request on the Python bug tracker, then wait until
Python 3.4 comes out in about 16 months.

:(


Many thanks :)

I am working on the assumption that the first argument of the format 
builtin function and be a sequence of values, which can be selected

with {1:}, {2:}, {0:} etc.

The docs don't make this clear.  I would appreciate advice.

Colin W.

[snip]

--
http://mail.python.org/mailman/listinfo/python-list

Re: How to only get a list of the names of the non-directory files in current directory ('.')?

[Chris Angelico]

On Wed, Nov 14, 2012 at 5:16 AM, emile  wrote:
> BTW, googling for "python how to get a list of names of everything in the
> current directory" yields some good information as well.  Google is your
> friend for this level of question.  Not sure anymore beyond that...

It yields it? You mean Google is an iterator?


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How to only get a list of the names of the non-directory files in current directory ('.')?

[John Gordon]

In  Chris Angelico 
 writes:

> It yields it? You mean Google is an iterator?

ITYM generator.

-- 
John Gordon   A is for Amy, who fell down the stairs
gor...@panix.com  B is for Basil, assaulted by bears
-- Edward Gorey, "The Gashlycrumb Tinies"

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Error messages from format()

[Dave Angel]

On 11/13/2012 03:24 PM, Colin J. Williams wrote:
> 
>
> I am working on the assumption that the first argument of the format
> builtin function and be a sequence of values, which can be selected
> with {1:}, {2:}, {0:} etc.
>
> The docs don't make this clear.  I would appreciate advice.
>

The built-in function format():

http://docs.python.org/3.3/library/functions.html?highlight=format%20builtin#format

The first parameter is a single object, NOT a sequence.  One object, one
format.  If you want more generality, use the str.format() method:

http://docs.python.org/3.3/library/stdtypes.html?highlight=format#str.format

where you can supply a list or a dictionary of multiple items to be
formatted into a single string.  That's the one where you supply the
curly braces.


-- 

DaveA

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How to only get a list of the names of the non-directory files in current directory ('.')?

[Chris Angelico]

On Wed, Nov 14, 2012 at 8:05 AM, John Gordon  wrote:
> In  Chris Angelico 
>  writes:
>
>> It yields it? You mean Google is an iterator?
>
> ITYM generator.

Yeah, that thing.

Troll fail.

*whoops*

ChrisA
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How to only get a list of the names of the non-directory files in current directory ('.')?

[emile]

On 11/13/2012 01:19 PM, Chris Angelico wrote:

Troll fail.

*whoops*


*sigh* mine too.

Emile


--
http://mail.python.org/mailman/listinfo/python-list

Re: how to simulate tar filename substitution across piped subprocess.Popen() calls?

[Thomas Rachel]

Am 12.11.2012 19:30 schrieb Hans Mulder:


This will break if there are spaces in the file name, or other
characters meaningful to the shell.  If you change if to

 xargsproc.append("test -f '%s/{}'&&  md5sum '%s/{}'"
  % (mydir, mydir))

, then it will only break if there are single quotes in the file name.


And if you do mydir_q = mydir.replace("'", "'\\''") and use mydir_q, you 
should be safe...



Thomas
--
http://mail.python.org/mailman/listinfo/python-list

Re: Error messages from format()

[Colin J. Williams]

  
  

On 13/11/2012 4:18 PM, Dave Angel
  wrote:


  On 11/13/2012 03:24 PM, Colin J. Williams wrote:

  


I am working on the assumption that the first argument of the format
builtin function and be a sequence of values, which can be selected
with {1:}, {2:}, {0:} etc.

The docs don't make this clear.  I would appreciate advice.


  
  
The built-in function format():

http://docs.python.org/3.3/library/functions.html?highlight=format%20builtin#format

The first parameter is a single object, NOT a sequence.  One object, one
format.  If you want more generality, use the str.format() method:

http://docs.python.org/3.3/library/stdtypes.html?highlight=format#str.format

where you can supply a list or a dictionary of multiple items to be
formatted into a single string.  That's the one where you supply the
curly braces.




The docs for the first case leave open the possibility of using a
sequence when they say:
"Convert a value to a “formatted” representation, as
  controlled by
  format_spec. The
  interpretation of format_spec
  will depend on the type
  of the value argument, however there is a standard formatting syntax that
  is used by most built-in types: Format
  Specification Mini-Language."

I hope that, as time goes by, consideration will be given to
permitting a sequence.  It would appear to be a relatively simple
change.  This would extend the generality of the format function.

Thanks for clarifying this.  It confirmed my trial and error
results.

Colin W.





  

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Division matrix

[R. Michael Weylandt]

On Tue, Nov 13, 2012 at 1:00 AM, Cleuson Alves  wrote:
> Hello, I need to solve an exercise follows, first calculate the inverse 
> matrix and then multiply the first matrix.

I would just point out that in most numerical applications, you rarely
need to calculate the intermediate of the matrix inverse directly.
See, e.g., http://www.johndcook.com/blog/2010/01/19/dont-invert-that-matrix/

Of course, if this hasn't been said yet: NumPy.

Michael
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Error messages from format()

[Steven D'Aprano]

On Tue, 13 Nov 2012 15:24:53 -0500, Colin J. Williams wrote:

> On 13/11/2012 1:38 PM, Steven D'Aprano wrote:
>> On Tue, 13 Nov 2012 10:08:59 -0500, Colin J. Williams wrote:
>>
>>> Is there some way to get more informative error messages from the
>>> builtin format?
>>
>> Yes -- post a feature request on the Python bug tracker, then wait
>> until Python 3.4 comes out in about 16 months.
>>
>> :(
>>
> Many thanks :)
> 
> I am working on the assumption that the first argument of the format
> builtin function and be a sequence of values, which can be selected with
> {1:}, {2:}, {0:} etc.

Um, did you read the rest of my post? I already told you that this is 
incorrect.



-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Generate unique ID for URL

[Richard]

Hello,

I want to create a URL-safe unique ID for URL's.
Currently I use:
url_id = base64.urlsafe_b64encode(url)

>>> base64.urlsafe_b64encode('docs.python.org/library/uuid.html')
'ZG9jcy5weXRob24ub3JnL2xpYnJhcnkvdXVpZC5odG1s'

I would prefer more concise ID's. 
What do you recommend? - Compression?

Richard
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Generate unique ID for URL

[John Gordon]

In <0692e6a2-343c-4eb0-be57-fe5c815ef...@googlegroups.com> Richard 
 writes:

> I want to create a URL-safe unique ID for URL's.
> Currently I use:
> url_id = base64.urlsafe_b64encode(url)

> >>> base64.urlsafe_b64encode('docs.python.org/library/uuid.html')
> 'ZG9jcy5weXRob24ub3JnL2xpYnJhcnkvdXVpZC5odG1s'

> I would prefer more concise ID's. 
> What do you recommend? - Compression?

Does the ID need to contain all the information necessary to recreate the
original URL?

-- 
John Gordon   A is for Amy, who fell down the stairs
gor...@panix.com  B is for Basil, assaulted by bears
-- Edward Gorey, "The Gashlycrumb Tinies"

-- 
http://mail.python.org/mailman/listinfo/python-list

Zipping files

[Smaran Harihar]

Hi Guys,

I am trying to create a script which will zip a bunch of files that I have
obtained using the 'glob'

So I have a bunch of files,

glob.glob("shp/file1*)

And I want to zip all the files which are returned by the above command.
How can I do that? Is there a zip library in python?

-- 
Thanks & Regards
Smaran Harihar
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Generate unique ID for URL

[Richard]

Good point - one way encoding would be fine.

Also this is performed millions of times so ideally efficient.


On Wednesday, November 14, 2012 10:34:03 AM UTC+11, John Gordon wrote:
> In <0692e6a2-343c-4eb0-be57-fe5c815ef...@googlegroups.com> Richard 
>  writes:
> 
> 
> 
> > I want to create a URL-safe unique ID for URL's.
> 
> > Currently I use:
> 
> > url_id = base64.urlsafe_b64encode(url)
> 
> 
> 
> > >>> base64.urlsafe_b64encode('docs.python.org/library/uuid.html')
> 
> > 'ZG9jcy5weXRob24ub3JnL2xpYnJhcnkvdXVpZC5odG1s'
> 
> 
> 
> > I would prefer more concise ID's. 
> 
> > What do you recommend? - Compression?
> 
> 
> 
> Does the ID need to contain all the information necessary to recreate the
> 
> original URL?
> 
> 
> 
> -- 
> 
> John Gordon   A is for Amy, who fell down the stairs
> 
> gor...@panix.com  B is for Basil, assaulted by bears
> 
> -- Edward Gorey, "The Gashlycrumb Tinies"

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Generate unique ID for URL

[Chris Kaynor]

One option would be using a hash. Python's built-in hash, a 32-bit
CRC, 128-bit MD5, 256-bit SHA or one of the many others that exist,
depending on the needs. Higher bit counts will reduce the odds of
accidental collisions; cryptographically secure ones if outside
attacks matter. In such a case, you'd have to roll your own means of
converting the hash back into the string if you ever need it for
debugging, and there is always the possibility of collisions. A
similar solution would be using a pseudo-random GUID using the url as
the seed.

You could use a counter if all IDs are generated by a single process
(and even in other cases with some work).

If you want to be able to go both ways, using base64 encoding is
probably your best bet, though you might get benefits by using
compression.
Chris


On Tue, Nov 13, 2012 at 3:56 PM, Richard  wrote:
> Good point - one way encoding would be fine.
>
> Also this is performed millions of times so ideally efficient.
>
>
> On Wednesday, November 14, 2012 10:34:03 AM UTC+11, John Gordon wrote:
>> In <0692e6a2-343c-4eb0-be57-fe5c815ef...@googlegroups.com> Richard 
>>  writes:
>>
>>
>>
>> > I want to create a URL-safe unique ID for URL's.
>>
>> > Currently I use:
>>
>> > url_id = base64.urlsafe_b64encode(url)
>>
>>
>>
>> > >>> base64.urlsafe_b64encode('docs.python.org/library/uuid.html')
>>
>> > 'ZG9jcy5weXRob24ub3JnL2xpYnJhcnkvdXVpZC5odG1s'
>>
>>
>>
>> > I would prefer more concise ID's.
>>
>> > What do you recommend? - Compression?
>>
>>
>>
>> Does the ID need to contain all the information necessary to recreate the
>>
>> original URL?
>>
>>
>>
>> --
>>
>> John Gordon   A is for Amy, who fell down the stairs
>>
>> gor...@panix.com  B is for Basil, assaulted by bears
>>
>> -- Edward Gorey, "The Gashlycrumb Tinies"
>
> --
> http://mail.python.org/mailman/listinfo/python-list
-- 
http://mail.python.org/mailman/listinfo/python-list

Detect file is locked - windows

[Ali Akhavan]

I am trying to open a file in 'w' mode open('file', 'wb'). open() will throw 
with IOError with errno 13 if the file is locked by another application or if 
user does not have permission to open/write to the file. 

How can I distinguish these two cases ? Namely, if some application has the 
file open or not. 

Thanks,
nomadali
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Zipping files

[Mark Lawrence]

On 13/11/2012 23:52, Smaran Harihar wrote:

Hi Guys,

I am trying to create a script which will zip a bunch of files that I have
obtained using the 'glob'

So I have a bunch of files,

glob.glob("shp/file1*)

And I want to zip all the files which are returned by the above command.
How can I do that? Is there a zip library in python?



http://docs.python.org/2/library/zipfile.html

Any reason that you couldn't have found this out for yourself, given 
that there are several search engines available?


--
Cheers.

Mark Lawrence.

--
http://mail.python.org/mailman/listinfo/python-list

Re: Generate unique ID for URL

[Richard Baron Penman]

I found the MD5 and SHA hashes slow to calculate.
The builtin hash is fast but I was concerned about collisions. What
rate of collisions could I expect?

Outside attacks not an issue and multiple processes would be used.


On Wed, Nov 14, 2012 at 11:26 AM, Chris Kaynor  wrote:
> One option would be using a hash. Python's built-in hash, a 32-bit
> CRC, 128-bit MD5, 256-bit SHA or one of the many others that exist,
> depending on the needs. Higher bit counts will reduce the odds of
> accidental collisions; cryptographically secure ones if outside
> attacks matter. In such a case, you'd have to roll your own means of
> converting the hash back into the string if you ever need it for
> debugging, and there is always the possibility of collisions. A
> similar solution would be using a pseudo-random GUID using the url as
> the seed.
>
> You could use a counter if all IDs are generated by a single process
> (and even in other cases with some work).
>
> If you want to be able to go both ways, using base64 encoding is
> probably your best bet, though you might get benefits by using
> compression.
> Chris
>
>
> On Tue, Nov 13, 2012 at 3:56 PM, Richard  wrote:
>> Good point - one way encoding would be fine.
>>
>> Also this is performed millions of times so ideally efficient.
>>
>>
>> On Wednesday, November 14, 2012 10:34:03 AM UTC+11, John Gordon wrote:
>>> In <0692e6a2-343c-4eb0-be57-fe5c815ef...@googlegroups.com> Richard 
>>>  writes:
>>>
>>>
>>>
>>> > I want to create a URL-safe unique ID for URL's.
>>>
>>> > Currently I use:
>>>
>>> > url_id = base64.urlsafe_b64encode(url)
>>>
>>>
>>>
>>> > >>> base64.urlsafe_b64encode('docs.python.org/library/uuid.html')
>>>
>>> > 'ZG9jcy5weXRob24ub3JnL2xpYnJhcnkvdXVpZC5odG1s'
>>>
>>>
>>>
>>> > I would prefer more concise ID's.
>>>
>>> > What do you recommend? - Compression?
>>>
>>>
>>>
>>> Does the ID need to contain all the information necessary to recreate the
>>>
>>> original URL?
>>>
>>>
>>>
>>> --
>>>
>>> John Gordon   A is for Amy, who fell down the stairs
>>>
>>> gor...@panix.com  B is for Basil, assaulted by bears
>>>
>>> -- Edward Gorey, "The Gashlycrumb Tinies"
>>
>> --
>> http://mail.python.org/mailman/listinfo/python-list
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Generate unique ID for URL

[Christian Heimes]

Am 14.11.2012 01:26, schrieb Chris Kaynor:
> One option would be using a hash. Python's built-in hash, a 32-bit
> CRC, 128-bit MD5, 256-bit SHA or one of the many others that exist,
> depending on the needs. Higher bit counts will reduce the odds of
> accidental collisions; cryptographically secure ones if outside
> attacks matter. In such a case, you'd have to roll your own means of
> converting the hash back into the string if you ever need it for
> debugging, and there is always the possibility of collisions. A
> similar solution would be using a pseudo-random GUID using the url as
> the seed.

A hash is the wrong answer to the issue as a hash is open to all sorts
of attack vectors like length extension attack. If Robert needs to
ensure any kind of collision resistance than he needs a MAC, for example
a HMAC with a secret key.

If he needs some kind of persistent identifier than some like a URN or
DOI may be a better answer.

Christian
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Generate unique ID for URL

[Richard]

These URL ID's would just be used internally for quick lookups, not exposed 
publicly in a web application.

Ideally I would want to avoid collisions altogether. But if that means 
significant extra CPU time then 1 collision in 10 million hashes would be 
tolerable.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Generate unique ID for URL

[Christian Heimes]

Am 14.11.2012 01:41, schrieb Richard Baron Penman:
> I found the MD5 and SHA hashes slow to calculate.
> The builtin hash is fast but I was concerned about collisions. What
> rate of collisions could I expect?

Seriously? It takes about 1-5msec to sha1() one MB of data on a modern
CPU, 1.5 on my box. The openssl variants of Python's hash code release
the GIL so you use the power of all cores.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Generate unique ID for URL

[Christian Heimes]

Am 14.11.2012 01:50, schrieb Richard:
> These URL ID's would just be used internally for quick lookups, not exposed 
> publicly in a web application.
> 
> Ideally I would want to avoid collisions altogether. But if that means 
> significant extra CPU time then 1 collision in 10 million hashes would be 
> tolerable.

Are you storing the URLs in any kind of database like a SQL database? A
proper index on the data column will avoid full table scans. It will
give you almost O(1) complexity on lookups and O(n) worst case
complexity for collisions.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Detect file is locked - windows

[Mark Lawrence]

On 14/11/2012 00:33, Ali Akhavan wrote:

I am trying to open a file in 'w' mode open('file', 'wb'). open() will throw 
with IOError with errno 13 if the file is locked by another application or if 
user does not have permission to open/write to the file.

How can I distinguish these two cases ? Namely, if some application has the 
file open or not.

Thanks,
nomadali



Anything here help http://www.python.org/dev/peps/pep-3151/ ?

--
Cheers.

Mark Lawrence.

--
http://mail.python.org/mailman/listinfo/python-list

Re: Generate unique ID for URL

[Richard]

I found md5 / sha 4-5 times slower than hash. And base64 a lot slower.

No database or else I would just use their ID.


On Wednesday, November 14, 2012 11:59:55 AM UTC+11, Christian Heimes wrote:
> Am 14.11.2012 01:41, schrieb Richard Baron Penman:
> 
> > I found the MD5 and SHA hashes slow to calculate.
> 
> > The builtin hash is fast but I was concerned about collisions. What
> 
> > rate of collisions could I expect?
> 
> 
> 
> Seriously? It takes about 1-5msec to sha1() one MB of data on a modern
> 
> CPU, 1.5 on my box. The openssl variants of Python's hash code release
> 
> the GIL so you use the power of all cores.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Generate unique ID for URL

[Roy Smith]

In article <0692e6a2-343c-4eb0-be57-fe5c815ef...@googlegroups.com>,
 Richard  wrote:

> Hello,
> 
> I want to create a URL-safe unique ID for URL's.
> Currently I use:
> url_id = base64.urlsafe_b64encode(url)
> 
> >>> base64.urlsafe_b64encode('docs.python.org/library/uuid.html')
> 'ZG9jcy5weXRob24ub3JnL2xpYnJhcnkvdXVpZC5odG1s'
> 
> I would prefer more concise ID's. 
> What do you recommend? - Compression?

If you're generating random id strings, there's only two ways to make 
them shorter.  Either encode fewer bits of information, or encode them 
more compactly.

Let's start with the second one.  You're already using base64, so you're 
getting 6 bits per character.  You can do a little better than that, but 
not much.  The set of URL-safe characters is the 96-ish printable ascii 
set, minus a few pieces of punctuation.  Maybe you could get it up to 
6.3 or 6.4 bits per character, but that's about it.  For the complexity 
this would add it's probably not worth it.

The next step is to reduce the number of bits you are encoding.  You 
said in another post that "1 collision in 10 million hashes would be 
tolerable".  So you need:

>>> math.log(10*1000*1000, 2)
23.25349666421154

24 bits worth of key.  Base64 encoded, that's only 4 characters.  
Actually, I probably just proved that I don't really understand how 
probabilities work, so maybe what you really need is 32 or 48 or 64 
bits.  Certainly not the 264 bits you're encoding with your example 
above.

So, something like:

hash = md5.md5('docs.python.org/library/uuid.html').digest()
hash64 = base64.urlsafe_b64encode(hash)
id = hash64[:8]  # or 12, or whatever

But, I still don't really understand your use case.  You've already 
mentioned the following requirements:

"just be used internally for quick lookups, not exposed publicly"
"URL-safe"
"unique"
"1 collision in 10 million hashes would be tolerable"
"one way encoding would be fine"
"performed millions of times so ideally efficient"

but haven't really explained what it is that you're trying to do.

If they're not going to be exposed publicly, why do you care if they're 
URL-safe?

What's wrong with just using the URLs directly as dictionary keys and 
not worrying about it until you've got some hard data showing that this 
is not sufficient?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Generate unique ID for URL

[Richard]

I am dealing with URL's rather than integers
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Generate unique ID for URL

[Richard]

So the use case - I'm storing webpages on disk and want a quick retrieval 
system based on URL. 
I can't store the files in a single directory because of OS limitations so have 
been using a sub folder structure.
For example to store data at URL "abc": a/b/c/index.html
This data is also viewed locally through a web app.

If you can suggest a better approach I would welcome it. 
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Generate unique ID for URL

[Richard]

> The next step is to reduce the number of bits you are encoding.  You 
> 
> said in another post that "1 collision in 10 million hashes would be 
> 
> tolerable".  So you need:
> 
> 
> 
> >>> math.log(10*1000*1000, 2)
> 
> 23.25349666421154


I think a difficulty would be finding a hash algorithm that maps evenly across 
those bits.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Simple Question regarding running .py program

[Caroline Hou]

On Tuesday, November 13, 2012 7:35:32 AM UTC-5, Ramchandra Apte wrote:
> On Tuesday, 13 November 2012 08:15:45 UTC+5:30, Caroline Hou  wrote:
> 
> > On Monday, 12 November 2012 21:25:08 UTC-5, Dave Angel  wrote:
> 
> > 
> 
> > > On 11/12/2012 09:02 PM, Caroline Hou wrote:
> 
> > 
> 
> > > 
> 
> > 
> 
> > > > Hi all!
> 
> > 
> 
> > > 
> 
> > 
> 
> > > >
> 
> > 
> 
> > > 
> 
> > 
> 
> > > > I just started learning Python by myself and I have an extremely simple 
> > > > question now!
> 
> > 
> 
> > > 
> 
> > 
> 
> > > > I am in my Python interpreter now and I want to open/edit a program 
> > > > called nobel.py. But when I typed >>> python nobel.py, it gave me a 
> > > > "SyntaxError:invalid syntax”( I've changed to the correct 
> > > > directory)what should I do?
> 
> > 
> 
> > > 
> 
> > 
> 
> > > > I also want to run the program, but as I double-clicked the program, a 
> > > > command window pops up and closes immediately. How can I see the result 
> > > > of the program run?
> 
> > 
> 
> > > 
> 
> > 
> 
> > > > Could anyone help me please? I am pretty confused here...Thank you!
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > It'd be nice to specify that you're running Windows, and also what
> 
> > 
> 
> > > 
> 
> > 
> 
> > > version of the interpreter, although in this case the latter doesn't 
> > > matter.
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > Go to a shell (cmd.exe), change to the directory containing that script,
> 
> > 
> 
> > > 
> 
> > 
> 
> > > and type the command as you did.
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > On linux:  davea@think:~$ python nobel.py
> 
> > 
> 
> > > 
> 
> > 
> 
> > > On Windows:   c:\mydir\myscript > python nobel.py
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > If you're already in the python interpreter, then running python is
> 
> > 
> 
> > > 
> 
> > 
> 
> > > useless -- it's already running.  In that case, you might want to use
> 
> > 
> 
> > > 
> 
> > 
> 
> > > import.  However, I recommend against it at first, as it opens up some
> 
> > 
> 
> > > 
> 
> > 
> 
> > > other problems you haven't experience with yet.
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > When you say you "double clicked the program', we have to guess you
> 
> > 
> 
> > > 
> 
> > 
> 
> > > might have meant in MS Explorer.  If you do that, it launches a cmd, it
> 
> > 
> 
> > > 
> 
> > 
> 
> > > runs the python system, and it closes the cmd.  Blame Windows for not
> 
> > 
> 
> > > 
> 
> > 
> 
> > > reading your mind.  If you want the cmd window to stick around, you
> 
> > 
> 
> > > 
> 
> > 
> 
> > > COULD end your program with an raw_input function call, but frequently
> 
> > 
> 
> > > 
> 
> > 
> 
> > > that won't work.  The right answer is the first one above...   Open a
> 
> > 
> 
> > > 
> 
> > 
> 
> > > shell (perhaps with a menu like  DOS BOX), change...
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > That way, when the program finishes, you can see what happened, or
> 
> > 
> 
> > > 
> 
> > 
> 
> > > didn't happen, and you can run it again using the uparrow key.
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > BTW, you don't need to send email to both the python-list and to the
> 
> > 
> 
> > > 
> 
> > 
> 
> > > newsgroup.  The newsgroup is automatically fed from the list.  But since
> 
> > 
> 
> > > 
> 
> > 
> 
> > > you're posting from google groups, that's just one of the bugs.  Many
> 
> > 
> 
> > > 
> 
> > 
> 
> > > folks here simply filter out everything from google groups, so your post
> 
> > 
> 
> > > 
> 
> > 
> 
> > > is invisible to them.
> 
> > 
> 
> > > 
> 
> > 
> 
> > >
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > -- 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > DaveA
> 
> > 
> 
> > 
> 
> > 
> 
> > Hi Dave!
> 
> > 
> 
> > 
> 
> > 
> 
> > thank you very much for your quick reply! I did manage to get the program 
> > run from cmd.exe. 
> 
> > 
> 
> > So does it mean that if I want to use python interactively,I should use the 
> > interpreter,while if I just want to run a python program, I should use DOS 
> > shell instead?
> 
> > 
> 
> > Also, how could I edit my script? I have sth called "IDLE" installed along 
> > with python. Is it the right place to write/edit my script?
> 
> > 
> 
> > Sorry about these semi-idiot questions but it is really hard to find an 
> > article or book that covers such basic stuffs! 
> 
> > 
> 
> > Thank you!
> 
> > 
> 
> > 
> 
> > 
> 
> > Caroline Hou
> 
> 
> 
> IDLE is recommended for newbies like you because an IDE requires too much 
> configuration.
> 
> When you start writing a big project, you can use an IDE.

Thank you Dave and everybody here for your helpful comments!This place is 
awesome! I found this group when I googled python-list. Seems like this is not 
the usual way you guys

Re: Generate unique ID for URL

[Roy Smith]

In article <1ce88f36-bfc7-4a55-89f8-70d1645d2...@googlegroups.com>,
 Richard  wrote:

> So the use case - I'm storing webpages on disk and want a quick retrieval 
> system based on URL. 
> I can't store the files in a single directory because of OS limitations so 
> have been using a sub folder structure.
> For example to store data at URL "abc": a/b/c/index.html
> This data is also viewed locally through a web app.
> 
> If you can suggest a better approach I would welcome it. 

Ah, so basically, you're reinventing Varnish?

Maybe do what Varnish (and MongoDB, and a few other things) do?  Bypass 
the file system entirely.  Juar mmap() a chunk of memory large enough to 
hold everything and let the OS figure out how to page things to disk.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Generate unique ID for URL

[Richard]

thanks for pointer to Varnish. 

I found MongoDB had a lot of size overhead so that it ended up using 4x the 
data stored. 
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Generate unique ID for URL

[Chris Angelico]

On Wed, Nov 14, 2012 at 2:25 PM, Richard  wrote:
> So the use case - I'm storing webpages on disk and want a quick retrieval 
> system based on URL.
> I can't store the files in a single directory because of OS limitations so 
> have been using a sub folder structure.
> For example to store data at URL "abc": a/b/c/index.html
> This data is also viewed locally through a web app.
>
> If you can suggest a better approach I would welcome it.

The cost of a crypto hash on the URL will be completely dwarfed by the
cost of storing/retrieving on disk. You could probably do some
arithmetic and figure out exactly how many URLs (at an average length
of, say, 100 bytes) you can hash in the time of one disk seek.

ChrisA
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Simple Question regarding running .py program

[Chris Angelico]

On Wed, Nov 14, 2012 at 2:31 PM, Caroline Hou  wrote:
> Thank you Dave and everybody here for your helpful comments!This place is 
> awesome! I found this group when I googled python-list. Seems like this is 
> not the usual way you guys access the list?

There are several ways to communicate with this list.

* The comp.lang.python newsgroup - get a newsreader (there are plenty
around), and either connect to your ISP's news server (if they have
one that carries c.l.p) or to a public server, some of which cost
money.
* Use a news-to-web gateway such as Google Groups. That specific one
is deprecated on this list, as there's more noise than signal from
Google Groups.
* The mailing list python-list, delivered directly to your inbox many
times a day. This is what I personally use.

Try here:
http://mail.python.org/mailman/listinfo/python-list

ChrisA
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Generate unique ID for URL

[Richard]

yeah good point - I have gone with md5 for now.


On Wednesday, November 14, 2012 3:06:18 PM UTC+11, Chris Angelico wrote:
> On Wed, Nov 14, 2012 at 2:25 PM, Richard  wrote:
> 
> > So the use case - I'm storing webpages on disk and want a quick retrieval 
> > system based on URL.
> 
> > I can't store the files in a single directory because of OS limitations so 
> > have been using a sub folder structure.
> 
> > For example to store data at URL "abc": a/b/c/index.html
> 
> > This data is also viewed locally through a web app.
> 
> >
> 
> > If you can suggest a better approach I would welcome it.
> 
> 
> 
> The cost of a crypto hash on the URL will be completely dwarfed by the
> 
> cost of storing/retrieving on disk. You could probably do some
> 
> arithmetic and figure out exactly how many URLs (at an average length
> 
> of, say, 100 bytes) you can hash in the time of one disk seek.
> 
> 
> 
> ChrisA

-- 
http://mail.python.org/mailman/listinfo/python-list

Subprocess puzzle and two questions

[wrw]

I need to time the operation of a command-line utility (specifically nslookup) 
from within a python program I'm writing.  I don't want to use python's timeit 
function because I'd like to avoid python's subprocess creation overhead.  That 
leads me to the standard UNIX time function.  So for example, in my bash shell, 
if I enter:

$ time nslookup www.es.net 8.8.4.4

I get:

Server: 8.8.4.4
Address:8.8.4.4#53

Non-authoritative answer:
www.es.net  canonical name = www3.es.net.
Name:   www3.es.net
Address: 128.55.22.201

 real   0m0.069s
 user   0m0.006s
 sys0m0.004s 

The first lines are the result of an nslookup of the IP address of "www.es.net" 
using the server at 8.8.4.4 (Google's public DNS server b).
The last three lines are what I'm after: the real elapsed wall-clock time, the 
time spent in user space and the time spent in kernel space.

However, if I try the same operation in the python interpreter using 
subprocess.Popen like so:

>>> import subprocess
>>> result = subprocess.Popen(['time', 'nslookup', 'www.es.net', '8.8.4.4'], 
>>> shell = False, stdout = subprocess.PIPE, stderr = 
>>> subprocess.PIPE).communicate()
>>> print result
('Server:\t\t8.8.4.4\nAddress:\t8.8.4.4#53\n\nNon-authoritative 
answer:\nwww.es.net\tcanonical name = 
www3.es.net.\nName:\twww3.es.net\nAddress: 128.55.22.201\n\n', '0.06 
real 0.00 user 0.00 sys\n')

And the timing information I'm after has been truncated to two digits after the 
decimal.  It appears that Popen is applying a default format. If I do explicit 
formatting:

>>> time = result[1].lstrip().split(' ')[0]
>>> formatted_time = '{: >7.3f}'.format(float(time))
>>> print formatted_time
  0.060

I get three digits, BUT that third digit isn't real, the format operation has 
simply appended a zero.  So:

1) how can I recover that third digit from the subprocess?
2) is there a more pythonic way to do what I'm trying to do?

python 2.7, OS-X 10.8.2

Thanks in advance -
Bill Wing

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Subprocess puzzle and two questions

[Roy Smith]

In article ,
 w...@mac.com wrote:

> I need to time the operation of a command-line utility (specifically 
> nslookup) from within a python program I'm writing.

Ugh.  Why are you doing this?  Shelling out to nslookup is an incredibly 
slow and clumsy way of doing name translation.  What you really want to 
be doing is calling getaddrinfo() directly.

See http://docs.python.org/2/library/socket.html#socket.getaddrinfo for 
details.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Simple Question regarding running .py program

[rurpy]

On 11/13/2012 09:10 PM, Chris Angelico wrote:
> On Wed, Nov 14, 2012 at 2:31 PM, Caroline Hou wrote:
>> Thank you Dave and everybody here for your helpful comments!This
>> place is awesome! I found this group when I googled python-list.
>> Seems like this is not the usual way you guys access the list?
> 
> There are several ways to communicate with this list.
> [...]
> * Use a news-to-web gateway such as Google Groups. That
> specific one is deprecated on this list, as there's more
> noise than signal from Google Groups.

Caroline, Chris is mistaken about this, if for no other
reason than there is no authority here empowered to decide
to deprecate anything.  What Chris should have said is 
that there are some people on this list who don't like 
Google Groups for whatever reason and encourage others
to ignore posts from Google Groups.

How successful this boycott effort is is not clear.

I use Google Groups as it suits my needs better than
any of the alternatives, and so do many others.  Both 
of the other alternatives Chris mentioned involve too 
much setup or overhead for those who read/post here 
only occasionally.  GG fills this niche adequately 
if used with care.

If you do use Google Groups to post, there are a 
couple of things you should be careful of:

* You'll sometimes see a checkbox above the GG send
window that is a CC to the python mailing list 
() which is checked by default.
Uncheck that before sending, or the list will get 
two copies of your message.  

* GG doesn't do a very good job in quoting the post 
you are replying to.  If you look at your recent 
post here:
  http://mail.python.org/pipermail/python-list/2012-November/635070.html
(or on GG) you will see lots and lots of lines empty 
save for the ">" quote markers.  This makes a post 
hard to read.

A way to avoid this is to remove the blank extra blank 
lines in the GG send window by hand before posting.

Alternatively, many email programs have a "paste as 
quotation" option when writing mail.  What I do is
to open a blank new email message, copy the original
post I'm replying to from GG, paste-as-quotation into
the new mail window, then copy and paste back into the
GG send window.  Pretty easy to do once you get used
to it.  

Hope this helps and provides a little more accurate 
info about posting from GG than has been provided so
far.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Simple Question regarding running .py program

[Chris Angelico]

On Wed, Nov 14, 2012 at 4:08 PM,   wrote:
> On 11/13/2012 09:10 PM, Chris Angelico wrote:
>> * Use a news-to-web gateway such as Google Groups. That
>> specific one is deprecated on this list, as there's more
>> noise than signal from Google Groups.
>
> Caroline, Chris is mistaken about this, if for no other
> reason than there is no authority here empowered to decide
> to deprecate anything.  What Chris should have said is
> that there are some people on this list who don't like
> Google Groups for whatever reason and encourage others
> to ignore posts from Google Groups.
>
> How successful this boycott effort is is not clear.

To be more accurate: This is deprecated *by members of* this list. As
there is no commanding/controlling entity here, it's up to each
individual to make a decision - for instance, abusive users get
killfiled rather than banned. The use of Google Groups to post is
deprecated in the original sense of the word: strongly disapproved of.

My own opinion on the matter is that if it takes as much effort as you
describe to use GG properly, it's wasting your time on a massive
scale. Surely it's easier to read and post email?

ChrisA
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Subprocess puzzle and two questions

[William Ray Wing]

On Nov 13, 2012, at 11:41 PM, Roy Smith  wrote:

> In article ,
> w...@mac.com wrote:
> 
>> I need to time the operation of a command-line utility (specifically 
>> nslookup) from within a python program I'm writing.
> 
> Ugh.  Why are you doing this?  Shelling out to nslookup is an incredibly 
> slow and clumsy way of doing name translation.  What you really want to 
> be doing is calling getaddrinfo() directly.
> 
> See http://docs.python.org/2/library/socket.html#socket.getaddrinfo for 
> details.
> -- 
Because, unless I'm badly mistaken (very possible), getaddrinfo doesn't let me 
specify the server from which the name is returned. I'm really not after the 
name, what I'm REALLY after is the fact that a path exists to the name server I 
specify (and how long it takes to respond). In the "good old days" I would just 
have ping'd it, but these days more and more DNS boxes (and servers of all 
sorts) are shutting off their ping response.

Thanks, Bill
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: creating size-limited tar files

[Kushal Kumaran]

Ian Kelly  writes:

> On Tue, Nov 13, 2012 at 3:31 AM, andrea crotti
>  wrote:
>> but it's a bit ugly.  I wonder if I can use the subprocess PIPEs to do
>> the same thing, is it going to be as fast and work in the same way??
>
> It'll look something like this:
>
 p1 = subprocess.Popen(cmd1, shell=True, stdout=subprocess.PIPE, 
 stderr=subprocess.PIPE)
 p2 = subprocess.Popen(cmd2, shell=True, stdin=p1.stdout, 
 stdout=subprocess.PIPE, stderr=subprocess.PIPE)
 p1.communicate()
> ('', '')
 p2.communicate()
> ('', '')
 p1.wait()
> 0
 p2.wait()
> 0
>
> Note that there's a subtle potential for deadlock here.  During the
> p1.communicate() call, if the p2 output buffer fills up, then it will
> stop accepting input from p1 until p2.communicate() can be called, and
> then if that buffer also fills up, p1 will hang.  Additionally, if p2
> needs to wait on the parent process for some reason, then you end up
> effectively serializing the two processes.
>
> Solution would be to poll all the open-ended pipes in a select() loop
> instead of using communicate(), or perhaps make the two communicate
> calls simultaneously in separate threads.

Or, you could just change the p1's stderr to an io.BytesIO instance.
Then call p2.communicate *first*.

-- 
regards,
kushal
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Subprocess puzzle and two questions

[Kushal Kumaran]

w...@mac.com writes:

> I need to time the operation of a command-line utility (specifically 
> nslookup) from within a python program I'm writing.  I don't want to use 
> python's timeit function because I'd like to avoid python's subprocess 
> creation overhead.  That leads me to the standard UNIX time function.  So for 
> example, in my bash shell, if I enter:
>

It is unclear to me what overhead you are avoiding.

>   $ time nslookup www.es.net 8.8.4.4
>
> I get:
>
>   Server: 8.8.4.4
>   Address:8.8.4.4#53
>
>   Non-authoritative answer:
>   www.es.net  canonical name = www3.es.net.
>   Name:   www3.es.net
>   Address: 128.55.22.201
>
>real   0m0.069s
>user   0m0.006s
>sys0m0.004s 
>
> The first lines are the result of an nslookup of the IP address of 
> "www.es.net" using the server at 8.8.4.4 (Google's public DNS server b).
> The last three lines are what I'm after: the real elapsed wall-clock time, 
> the time spent in user space and the time spent in kernel space.
>
> However, if I try the same operation in the python interpreter using 
> subprocess.Popen like so:
>
 import subprocess
 result = subprocess.Popen(['time', 'nslookup', 'www.es.net', '8.8.4.4'], 
 shell = False, stdout = subprocess.PIPE, stderr = 
 subprocess.PIPE).communicate()
 print result
> ('Server:\t\t8.8.4.4\nAddress:\t8.8.4.4#53\n\nNon-authoritative 
> answer:\nwww.es.net\tcanonical name = 
> www3.es.net.\nName:\twww3.es.net\nAddress: 128.55.22.201\n\n', '0.06 
> real 0.00 user 0.00 sys\n')
>
> And the timing information I'm after has been truncated to two digits after 
> the decimal.  It appears that Popen is applying a default format. If I do 
> explicit formatting:
>
 time = result[1].lstrip().split(' ')[0]
 formatted_time = '{: >7.3f}'.format(float(time))
 print formatted_time
>   0.060
>
> I get three digits, BUT that third digit isn't real, the format operation has 
> simply appended a zero.  So:
>
> 1) how can I recover that third digit from the subprocess?
> 2) is there a more pythonic way to do what I'm trying to do?
>
> python 2.7, OS-X 10.8.2
>

It is possible that the "time" invocation from the shell is invoking
your shell's builtin time implementation, and your python code is
running /usr/bin/time or /bin/time.  You should see the same behaviour
from the shell if you run /bin/time or /usr/bin/time (whatever you have)
instead of just "time".  subprocess.Popen should never modify the output
of programs it runs.

-- 
regards,
kushal
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: access spreadsheet data

[Mark Lawrence]

On 14/11/2012 06:35, Amit Agrawal wrote:

my problem is, i want to access data in spreadsheet to python code manualy
My data is

1/1982 8:00:00 0



Please tell us your spreadsheet (CSV file, M$ Excel, OS X Numbers, 
what?), your OS and Python version, and what exactly you're trying to 
achieve, then you're far more likely to get some useful answers.


--
Cheers.

Mark Lawrence.

--
http://mail.python.org/mailman/listinfo/python-list

Re: Simple Question regarding running .py program

[rurpy]

On 11/13/2012 11:02 PM, Chris Angelico wrote:
> On Wed, Nov 14, 2012 at 4:08 PM, rurpy wrote:
>> On 11/13/2012 09:10 PM, Chris Angelico wrote:
>>> * Use a news-to-web gateway such as Google Groups. That
>>> specific one is deprecated on this list, as there's more
>>> noise than signal from Google Groups.
>>
>> Caroline, Chris is mistaken about this, if for no other
>> reason than there is no authority here empowered to decide
>> to deprecate anything.  What Chris should have said is
>> that there are some people on this list who don't like
>> Google Groups for whatever reason and encourage others
>> to ignore posts from Google Groups.
>>
>> How successful this boycott effort is is not clear.
> 
> To be more accurate: This is deprecated *by members of* this list. As
> there is no commanding/controlling entity here, it's up to each
> individual to make a decision - for instance, abusive users get
> killfiled rather than banned. The use of Google Groups to post is
> deprecated in the original sense of the word: strongly disapproved of.

s/deprecated *by members of*/deprecated *by some members of*/

(and accuracy could probably be increased further by replacing
"some" with "a few".)

And again,

s/strongly disapproved of/strongly disapproved of by some/

I was objecting to your attempts to make it sound like a 
fact that GG posts were nearly universally condemned here, 
an error you repeat again above.

> My own opinion on the matter is that if it takes as much effort as you
> describe to use GG properly, it's wasting your time on a massive
> scale. Surely it's easier to read and post email?

"that much effort"?  Do you consider a couple of copy-
pastes to be "that much effort"?  "wasting your time 
on a massive scale"?  Sheesh, get a grip man, let's not
get silly.

On the other hand finding and configuring a newsreader 
for someone whose never done it before, as you recommend, 
is a major time consumer.  And signing up for python-list 
email, posting, dealing with several days' volume, and 
then signing off because one does not want to read it 
full time[*], is no picnic either compared to using GG.

Sorry, but there are *good* reasons for some people
to use GG whether you want to admit it or not.

==
[*] Actually, now that I think about it, IIRC one can sign
up for python-list email, and go into the mailman settings
and disable mail delivery, allowing one to post to the list 
via email yet read the list via GG, Gmane or whatever.
However, this is not going to be obvious to many occasional
posters, and is still a PITA compared to just posting from 
GG as our hypothetical user does for all the other groups 
he/she participates in.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Subprocess puzzle and two questions

[Tim Roberts]

w...@mac.com wrote:
>...
>However, if I try the same operation in the python interpreter using 
>subprocess.Popen like so:
>
 import subprocess
 result = subprocess.Popen(['time', 'nslookup', 'www.es.net', '8.8.4.4'], 
 shell = False, stdout = subprocess.PIPE, stderr = 
 subprocess.PIPE).communicate()
 print result
>('Server:\t\t8.8.4.4\nAddress:\t8.8.4.4#53\n\nNon-authoritative 
>answer:\nwww.es.net\tcanonical name = 
>www3.es.net.\nName:\twww3.es.net\nAddress: 128.55.22.201\n\n', '0.06 
>real 0.00 user 0.00 sys\n')
>
>And the timing information I'm after has been truncated to two digits after 
>the decimal.  It appears that Popen is applying a default format. 

No, that's silly.  A few minutes thought should have told you that.  In
your standalone test, you are getting the "time" command that is built in
to bash.  In the subprocess example, you've specified "shell = False", so
you are using the external "time" command (/usr/bin/time in my system), and
that command has a different output format.  The csh "time" command is
different yet again.

>1) how can I recover that third digit from the subprocess?

Do you actually believe that the third decimal place has any meaning at
all?  It doesn't.
-- 
Tim Roberts, t...@probo.com
Providenza & Boekelheide, Inc.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: creating size-limited tar files

[Ian Kelly]

On Tue, Nov 13, 2012 at 11:05 PM, Kushal Kumaran
 wrote:
> Or, you could just change the p1's stderr to an io.BytesIO instance.
> Then call p2.communicate *first*.

This doesn't seem to work.

>>> b = io.BytesIO()
>>> p = subprocess.Popen(["ls", "-l"], stdout=b)
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib64/python3.2/subprocess.py", line 711, in __init__
errread, errwrite) = self._get_handles(stdin, stdout, stderr)
  File "/usr/lib64/python3.2/subprocess.py", line 1112, in _get_handles
c2pwrite = stdout.fileno()
io.UnsupportedOperation: fileno

I think stdout and stderr need to be actual file objects, not just
file-like objects.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Simple Question regarding running .py program

[rusi]

On Nov 14, 12:02 pm, ru...@yahoo.com wrote:
>
> ==
> [*] Actually, now that I think about it, IIRC one can sign
> up for python-list email, and go into the mailman settings
> and disable mail delivery, allowing one to post to the list
> via email yet read the list via GG, Gmane or whatever.
> However, this is not going to be obvious to many occasional
> posters, and is still a PITA compared to just posting from
> GG as our hypothetical user does for all the other groups
> he/she participates in.

Yes this would be (for me) my most preferred option if I could figure
out a way of threading into a preexisting thread.
-- 
http://mail.python.org/mailman/listinfo/python-list