Timestamps for TCP packets?

[Thomas Johnson]

Is there any way to get kernel-level timestamps for TCP packets while
still using the standard python sockets library for communication? I
need to communicate over a TCP connection as easily as possible, but
also record the timestamps of the incoming and outgoing timestamps at
microsecond or nanosecond resolution.

The sockets library is of course great for the communication, and I've
seen some python libraries that do packet sniffing and record
timestamps, but it's not clear that I can do both at the same time.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Open file on remote linux server

[Martien Verbruggen]

On Wed, 23 Sep 2009 23:41:35 +0200,
Diez B. Roggisch  wrote:
> The Bear schrieb:
>> Hi I'm looking to do something like this
>> 
>> f = f.openfileobj(remotefileloc, localfilelikeobj)
>> 
>> my remote files are on a solaris box that i can access using ssh (could
>> prehap request othe protocols if necessary)
>> 
>> anyone got any ideas?
>
> try paramiko. Or just use subprocess to scp and open the file locally.


import paramiko

ssh = paramiko.SSHClient()
ssh.load_system_host_keys(os.environ['HOME'] + '/.ssh/known_hosts')
ssh.connect('localhost')
try:
ftp = ssh.open_sftp()
# To write
fh = ftp.file('/tmp/foo.test', 'w')
fh.write('This is a test string\nAnd this is another one')
fh.close()
# To read
fh = ftp.file('/tmp/foo.test', 'r')
for l in fh:
print l,
fh.close()
finally:
ssh.close()

You may need to add some password handling in there (for me my
environment takes care of that).

Martien
-- 
 | 
Martien Verbruggen   | You can't have everything, where would
first.l...@heliotrope.com.au | you put it?
 | 
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Regular expression to structure HTML

[Bruno Desthuilliers]

504cr...@gmail.com a écrit :

I'm kind of new to regular expressions, and I've spent hours trying to
finesse a regular expression to build a substitution.

What I'd like to do is extract data elements from HTML and structure
them so that they can more readily be imported into a database.

No -- sorry -- I don't want to use BeautifulSoup (though I have for
other projects). Humor me, please -- I'd really like to see if this
can be done with just regular expressions.



I'm kind of new to hammers, and I've spent hours trying to find out how 
to drive a screw with a hammer. No -- sorry -- I don't want to use a 
screwdriver.



--
http://mail.python.org/mailman/listinfo/python-list

Re: Timestamps for TCP packets?

[Miles Kaufmann]

On Oct 2, 2009, at 12:03 AM, Thomas Johnson wrote:


Is there any way to get kernel-level timestamps for TCP packets while
still using the standard python sockets library for communication? I
need to communicate over a TCP connection as easily as possible, but
also record the timestamps of the incoming and outgoing timestamps at
microsecond or nanosecond resolution.

The sockets library is of course great for the communication, and I've
seen some python libraries that do packet sniffing and record
timestamps, but it's not clear that I can do both at the same time.


Have you tried it?  I don't know of any reason that using sockets and  
doing a packet capture would interfere with each other.  What are you  
trying to accomplish with the packet sniffing, though?


-Miles
--
http://mail.python.org/mailman/listinfo/python-list

Re: PyOpenGL and graphics card support

[jefm]

that works. Thx

below is the output for my system:

gluGetString - GLU_VERSION: 1.2.2.0 Microsoft
Corporation
gluGetString - GLU_EXTENSIONS:  GL_EXT_bgra
glGetString  - GL_VENDOR:   NVIDIA Corporation
glGetString  - GL_RENDERER: GeForce 9500 GT/PCI/SSE2
glGetString  - GL_SHADING_LANGUAGE_VERSION: 1.40 NVIDIA via Cg
compiler
glGetString  - GL_EXTENSIONS:
 GL_ARB_color_buffer_float
 GL_ARB_compatibility
 GL_ARB_copy_buffer
 GL_ARB_depth_buffer_float
 GL_ARB_depth_texture
 GL_ARB_draw_buffers
 GL_ARB_draw_instanced
 GL_ARB_fragment_program
 GL_ARB_fragment_program_shadow
 GL_ARB_fragment_shader
 GL_ARB_half_float_pixel
 GL_ARB_half_float_vertex
 GL_ARB_framebuffer_object
 GL_ARB_geometry_shader4
 GL_ARB_imaging
 GL_ARB_map_buffer_range
 GL_ARB_multisample
 GL_ARB_multitexture
 GL_ARB_occlusion_query
 GL_ARB_pixel_buffer_object
 GL_ARB_point_parameters
 GL_ARB_point_sprite
 GL_ARB_shadow
 GL_ARB_shader_objects
 GL_ARB_shading_language_100
 GL_ARB_texture_border_clamp
 GL_ARB_texture_buffer_object
 GL_ARB_texture_compression
 GL_ARB_texture_cube_map
 GL_ARB_texture_env_add
 GL_ARB_texture_env_combine
 GL_ARB_texture_env_dot3
 GL_ARB_texture_float
 GL_ARB_texture_mirrored_repeat
 GL_ARB_texture_non_power_of_two
 GL_ARB_texture_rectangle
 GL_ARB_texture_rg
 GL_ARB_transpose_matrix
 GL_ARB_vertex_array_object
 GL_ARB_vertex_buffer_object
 GL_ARB_vertex_program
 GL_ARB_vertex_shader
 GL_ARB_window_pos
 GL_ATI_draw_buffers
 GL_ATI_texture_float
 GL_ATI_texture_mirror_once
 GL_S3_s3tc
 GL_EXT_texture_env_add
 GL_EXT_abgr
 GL_EXT_bgra
 GL_EXT_blend_color
 GL_EXT_blend_equation_separate
 GL_EXT_blend_func_separate
 GL_EXT_blend_minmax
 GL_EXT_blend_subtract
 GL_EXT_compiled_vertex_array
 GL_EXT_Cg_shader
 GL_EXT_bindable_uniform
 GL_EXT_depth_bounds_test
 GL_EXT_direct_state_access
 GL_EXT_draw_buffers2
 GL_EXT_draw_instanced
 GL_EXT_draw_range_elements
 GL_EXT_fog_coord
 GL_EXT_framebuffer_blit
 GL_EXT_framebuffer_multisample
 GL_EXT_framebuffer_object
 GL_EXTX_framebuffer_mixed_formats
 GL_EXT_framebuffer_sRGB
 GL_EXT_geometry_shader4
 GL_EXT_gpu_program_parameters
 GL_EXT_gpu_shader4
 GL_EXT_multi_draw_arrays
 GL_EXT_packed_depth_stencil
 GL_EXT_packed_float
 GL_EXT_packed_pixels
 GL_EXT_pixel_buffer_object
 GL_EXT_point_parameters
 GL_EXT_provoking_vertex
 GL_EXT_rescale_normal
 GL_EXT_secondary_color
 GL_EXT_separate_shader_objects
 GL_EXT_separate_specular_color
 GL_EXT_shadow_funcs
 GL_EXT_stencil_two_side
 GL_EXT_stencil_wrap
 GL_EXT_texture3D
 GL_EXT_texture_array
 GL_EXT_texture_buffer_object
 GL_EXT_texture_compression_latc
 GL_EXT_texture_compression_rgtc
 GL_EXT_texture_compression_s3tc
 GL_EXT_texture_cube_map
 GL_EXT_texture_edge_clamp
 GL_EXT_texture_env_combine
 GL_EXT_texture_env_dot3
 GL_EXT_texture_filter_anisotropic
 GL_EXT_texture_integer
 GL_EXT_texture_lod
 GL_EXT_texture_lod_bias
 GL_EXT_texture_mirror_clamp
 GL_EXT_texture_object
 GL_EXT_texture_sRGB
 GL_EXT_texture_swizzle
 GL_EXT_texture_shared_exponent
 GL_EXT_timer_query
 GL_EXT_vertex_array
 GL_EXT_vertex_array_bgra
 GL_IBM_rasterpos_clip
 GL_IBM_texture_mirrored_repeat
 GL_KTX_buffer_region
 GL_NV_blend_square
 GL_NV_copy_depth_to_color
 GL_NV_copy_image
 GL_NV_depth_buffer_float
 GL_NV_conditional_render
 GL_NV_depth_clamp
 GL_NV_explicit_multisample
 GL_NV_fence
 GL_NV_float_buffer
 GL_NV_fog_distance
 GL_NV_fragment_program
 GL_NV_fragment_program_option
 GL_NV_fragment_program2
 GL_NV_framebuffer_multisample_coverage
 GL_NV_geometry_shader4
 GL_NV_gpu_program4
 GL_NV_half_float
 GL_NV_light_max_exponent
 GL_NV_multisample_coverage
 GL_NV_multisample_filter_hint
 GL_NV_occlusion_query
 GL_NV_packed_depth_stencil
 GL_NV_parameter_buffer_object
 GL_NV_parameter_buffer_object2
 GL_NV_pixel_data_range
 GL_NV_point_sprite
 GL_NV_primitive_restart
 GL_NV_register_combiners
 GL_NV_register_combiners2
 GL_NV_texgen_reflection
 GL_NV_texture_compression_vtc
 GL_NV_texture_env_combine4
 GL_NV_texture_expand_normal
 GL_NV_texture_rectangle
 GL_NV_texture_shader
 GL_NV_texture_shader2
 GL_NV_texture_shader3
 GL_NV_transform_feedback
 GL_NV_vertex_array_range
 GL_NV_vertex_array_range2
 GL_NV_vertex_program
 GL_NV_vertex_program1_1
 GL_NV_vertex_program2
 GL_NV_vertex_program2_option
 GL_NV_vertex_program3
 GL_NVX_conditional_render
 GL_NV_vertex_buffer_unified_memory

Re: Regular expression to structure HTML

[Paul McGuire]

On Oct 2, 12:10 am, "504cr...@gmail.com" <504cr...@gmail.com> wrote:
> I'm kind of new to regular expressions, and I've spent hours trying to
> finesse a regular expression to build a substitution.
>
> What I'd like to do is extract data elements from HTML and structure
> them so that they can more readily be imported into a database.

Oy! If I had a nickel for every misguided coder who tried to scrape
HTML with regexes...

Some reasons why RE's are no good at parsing HTML:
- tags can be mixed case
- tags can have whitespace in many unexpected places
- tags with no body can combine opening and closing tag with a '/'
before the closing '>', as in ""
- tags can have attributes that you did not expect (like "")
- attributes can occur in any order within the tag
- attribute names can also be in unexpected upper/lower case
- attribute values can be enclosed in double quotes, single quotes, or
even (surprise!) NO quotes

For HTML that is machine-generated, you *may* be able to make some
page-specific assumptions.  But if edited by human hands, or if you
are trying to make a generic page scraper, RE's will never cut it.

-- Paul

-- 
http://mail.python.org/mailman/listinfo/python-list

weak reference to bound method

[Ole Streicher]

Hi group,

I am trying to use a weak reference to a bound method:

class MyClass(object):
def myfunc(self):
pass

o = MyClass()
print o.myfunc
   >

import weakref
r = weakref.ref(o.myfunc)
print r()
   None

This is what I do not understand. The object "o" is still alive, and
therefore the bound method "o.myfunc" shall exists.

Why does the weak reference claim that it is removed? And how can I hold
the reference to the method until the object is removed?

Is this a bug or a feature? (Python 2.6)

Best regards

Ole
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: weak reference to bound method

[Peter Otten]

Ole Streicher wrote:

> I am trying to use a weak reference to a bound method:
> 
> class MyClass(object):
> def myfunc(self):
> pass
> 
> o = MyClass()
> print o.myfunc
>      0xc675d0>>
> 
> import weakref
> r = weakref.ref(o.myfunc)
> print r()
>   None
> 
> This is what I do not understand. The object "o" is still alive, and
> therefore the bound method "o.myfunc" shall exists.

No. o.myfunc is a different object, a bound method, and every time you 
access o's myfunc attribute a new bound method is created:

>>> class MyClass(object):
... def myfunc(self):
... pass
...
>>> o = MyClass()
>>> a = o.myfunc
>>> b = o.myfunc
>>> a == b
True
>>> id(a) == id(b)
False

The bound method holds references to the instance and the function, not the 
other way around:

>>> a.im_class, a.im_self, a.im_func
(, <__main__.MyClass object at 0x7f1437146710>, 
)

> Why does the weak reference claim that it is removed? 

Because there are indeed no more strong references to the bound method.

> And how can I hold the reference to the method until the object is 
removed?

If you kept a strong reference to the bound method it would in turn keep the 
MyClass instance alive. Maybe you could store the bound method in the 
instance and rely on cyclic garbage collection.

Is there an actual use case?

Peter

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: weak reference to bound method

[Ole Streicher]

Hi Thomas,

Thomas Lehmann  writes:
>> r = weakref.ref(o.myfunc)
>> print r()
>>    None
> k = o.myfunc
> r = weakref.ref(k)
> print r()
>>  

> Don't ask me why! I have just been interested for what you are trying...

This is clear: in your case, o.myfunc is explicitely referenced by k,
this avoids the garbage collection.

My problem is that I have a class that delegates a function call, like:

8<--
import weakref

class WeakDelegator(object):
def __init__(self, func):
self._func = weakref.ref(func)

def __call__(self):
func = self._func()
return func() if func else None
8<--

This does not work for bound methods because the weak reference to a
bound method will always point to None, even if the object still exists.

Why is that the case and how can I implement such a class properly?

Best regards

Ole
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: custom data warehouse in python vs. out-of-the-box ETL tool

[M.-A. Lemburg]

Tony Schmidt wrote:
> Hi, Marc-Andre - well, so far you seem to be the only one suggesting
> that cross-database joins is the way to go - everyone else has been
> telling me to build a warehouse.  I initially was trying to avoid the
> warehouse idea to "avoid going through the external temporary
> resource", as you say.  But then the situation came up where the data
> warehouse would give me another benefit by enabling power users with
> tools like OOBase to do their own reporting.  So I started to re-
> consider it, and since then, no one has advised the direct approach
> with connections/joins on multiple databases.
> 
> Have you had much luck circumventing the need for a data warehouse in
> this way?

Depending on what your data sources are, you will end up
using a data warehouse sooner or later.

We've worked with clients from the middle office banking world
and there you typically have to aggregate data, polish it up,
transform it and then send it off for others to process.

Cross-database joins don't work in such a scenario because
the data sources are too diverse. Logging and auditing
requirements can also lead to having to use a warehouse
approach.

However, if you just to put together data from a few local
sources without too much data to work on, I'd suggest to
try the direct Python approach first. If you have enough
RAM to work with, Python can easily handle millions
of records and if you'd rather use a SQL based approach,
it's also possible to temporarily store the data in a SQLite
database as on-demand data warehouse.

Regards,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Oct 02 2009)
>>> Python/Zope Consulting and Support ...http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/


> Thanks.
> 
> On Sep 24, 4:10 am, "M.-A. Lemburg"  wrote:
>> snfctech wrote:
>>> @Lemburg: Thanks for the suggestion.  I'm sure you make a fine
>>> product, but my development platform is Linux, and I don't want any
>>> additional Windows servers to deal with (than the ones I'm already
>>> stuck with.)
>>
>> Strange, EasySoft used to support their product on Linux as well...
>>
>> http://download.freshmeat.net/projects/easysoftsqlengine
>>
>> Looks like they stopped with supporting it on Linux and now only
>> have Windows downloads available.
>>
>> mxODBC and mxODBC Connect work just fine on Linux, so you could use
>> them to build a custom joining solution in Python.
>>
>> It really depends on whether you need to run those joins on
>> an on-demand basis or not:
>>
>> AFAIK, Talend can only run jobs which
>> then apply the JOINs and put the resulting data somewhere, e.g.
>> into a file or another database table.
>>
>> With Python, you could do the join straight away and use the
>> data directly without the Python script - without having
>> to go through an external temporary resource.
>>
>> Anyway, just a suggestion.
>>
>>
>>
>>> On Sep 23, 2:02 am, "M.-A. Lemburg"  wrote:
 snfctech wrote:
> Does anyone have experience building a datawarehousein python?  Any
> thoughts on custom vs using an out-of-the-box product like Talend or
> Informatica?
>>
> I have an integrated system Dashboard project that I was going to
> build using cross-vendor joins on existing DBs, but I keep hearing
> that a datawarehouseis the way to go.  e.g. I want to create orders
> and order_items with relations to members (MS Access DB), products
> (flat file) and employees (MySQL).
>>
> Thanks in advance for any tips.
>>
 You might want to look at this solution for doing cross-database JOINs:
>>
 EasySoft ODBC Join-Engine:
>>
http://www.easysoft.com/products/data_access/odbc_odbc_join_engine/in...
>>
 and then use our mxODBC to access EasySoft's Engine:
>>
http://www.egenix.com/products/python/mxODBC/
>>
 or mxODBC Connect, if you have a client-server setup:
>>
http://www.egenix.com/products/python/mxODBCConnect/
>>
 Some database engines also allow integrating external ODBC
 data sources - the external tables then look like normal
 database tables and can be used in JOINs as well. I know that DB2
 and Oracle support this kind of setup. You can access those using
 mxODBC as well.
>>
 --
 Marc-Andre Lemburg
 eGenix.com
>>
 Professional Python Services directly from the Source  (#1, Sep 23 
 2009)>>> Python/Zope Consulting and Support ...
 http://www.egeni

Re: weak reference to bound method

[Miles Kaufmann]

On Oct 2, 2009, at 1:54 AM, Ole Streicher wrote:

I am trying to use a weak reference to a bound method:

class MyClass(object):
   def myfunc(self):
   pass

o = MyClass()
print o.myfunc
 0xc675d0>>


import weakref
r = weakref.ref(o.myfunc)
print r()

 None


This is what I do not understand. The object "o" is still alive, and
therefore the bound method "o.myfunc" shall exists.


Like Peter said, bound methods are created on demand when they are  
obtained from the instance, not when the instance is created.


Why does the weak reference claim that it is removed? And how can I  
hold

the reference to the method until the object is removed?


You could combine unbound methods with a weakref to the object:

r = weakref.ref(o)
MyClass.myfunc(r())

You could also create a wrapper object that holds a weak reference to  
the instance and creates a bound method on demand:


class WeakMethod(object):
def __init__(self, bound_method):
self.im_func = bound_method.im_func
self.im_self = weakref.ref(bound_method.im_self)
self.im_class = bound_method.im_class

def __call__(self):
obj = self.im_self()
if obj is None: return None
return types.MethodType(self.im_func, obj, self.im_class)
# could alternately act like a callableproxy

-Miles

--
http://mail.python.org/mailman/listinfo/python-list

Re: weak reference to bound method

[Ole Streicher]

Hello Peter,

Peter Otten <__pete...@web.de> writes:
> Is there an actual use case?

I discussed this in the german newsgroup. Here is the use in my class:
-8<---
import threading
import weakref

class DoAsync(threading.Thread):
def __init__(self, func):
threading.Thread.__init__(self)
self.setDaemon(True)
self._cond = threading.Condition()
self.scheduled = False
self._func = weakref.ref(func, self._cleanup)
self.start()

def run(self):
while self._func():
with self._cond:
while not self.scheduled and self._func():
self._cond.wait()
self.scheduled = False
func = self._func()
if func:
func()

def __call__(self):
with self._cond:
self.scheduled = True
self._cond.notify()

def _cleanup(self, ref):
self()
-8<---

The use for this callable class is to take a function call, and whenever
the DoAsync object is called, trigger a call to the stored function.

Other classes use it like:

-8<---
class MyClass:
def __init__(self):
 ...
 self.update = DoAsync(self._do_update)

def _do_update(self):
 do_something_that_takes_long_and_shall_be_done_after_an_update()
-8<---

Since DoAsync starts its own thread, I get a classical deadlock
situation: DoAsync needs a reference to the method to be called, and as
long as the thread is running, the MyClass object (which contains the
method) cannot be cleaned up. This would be a classic case for a weak
reference, if Python would not create it at calling time.

> No. o.myfunc is a different object, a bound method, and every time you 
> access o's myfunc attribute a new bound method is created:

What is the reason for that behaviour? It looks quite silly to me.

And how can I get a reference to a bound method that lives as long as
the method itself?

Regards

Ole
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: weak reference to bound method

[Peter Otten]

Ole Streicher wrote:

> Hi Thomas,
> 
> Thomas Lehmann  writes:
>>> r = weakref.ref(o.myfunc)
>>> print r()
>>>    None
>> k = o.myfunc
>> r = weakref.ref(k)
>> print r()
>>>  
> 
>> Don't ask me why! I have just been interested for what you are trying...
> 
> This is clear: in your case, o.myfunc is explicitely referenced by k,
> this avoids the garbage collection.
> 
> My problem is that I have a class that delegates a function call, like:
> 
> 8<--
> import weakref
> 
> class WeakDelegator(object):
> def __init__(self, func):
> self._func = weakref.ref(func)
> 
> def __call__(self):
> func = self._func()
> return func() if func else None
> 8<--
> 
> This does not work for bound methods because the weak reference to a
> bound method will always point to None, even if the object still exists.
> 
> Why is that the case and how can I implement such a class properly?

Maybe:

from weakref import ref

class A(object):
def f(self): return "f"

class Method(object):
def __init__(self, obj, func=None):
if func is None:
func = obj.im_func
obj = obj.im_self
self._im_self = ref(obj)
self._im_func = ref(func)
def __call__(self):
obj = self._im_self()
func = self._im_func()
if obj is not None and func is not None:
return func.__get__(obj)()

a = A()
m = Method(a.f)
print m()
del a
print m()

It's still not clear to me why you would want to do that...

Peter

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: cx_freeze problem on Ubuntu

[Paul Boddie]

On 1 Okt, 16:08, John  wrote:
>
> I downloaded the cx_freeze source code 
> fromhttp://cx-freeze.sourceforge.net/into a directory.

[...]

>  From 
> here:http://linux.softpedia.com/get/Programming/Assembler-Tools/cx-Freeze-...
> the directions state:

What about the documentation available from the SourceForge project? I
don't know how reliable or up-to-date the Softpedia stuff is, given
that they're apparently aggregating content from elsewhere on the
Internet.

[...]

> As far as I can tell, no one else has ever posted this problem so I
> don't know what corrective actions to make.  I'd greatly appreciate any
> assistance.

The author appears to use the mailing list for cx_Freeze:

https://lists.sourceforge.net/lists/listinfo/cx-freeze-users

Perhaps you can get some help in that forum.

Paul
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: weak reference to bound method

[Ole Streicher]

Hi Miles,

Miles Kaufmann  writes:
> You could also create a wrapper object that holds a weak reference to  the
> instance and creates a bound method on demand:
> class WeakMethod(object):
> def __init__(self, bound_method):
> self.im_func = bound_method.im_func
> self.im_self = weakref.ref(bound_method.im_self)
> self.im_class = bound_method.im_class

In this case, I can use it only for bound methods, so I would need to
handle the case of unbound methods separately.

Is there a way to find out whether a function is bound? Or do I have to
use hasattr(im_func) and hasattr(im_self) and hasattr(im_class)?

Best regards

Ole
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: weak reference to bound method

[Thomas Lehmann]

> I am trying to use a weak reference to a bound method:
>
> class MyClass(object):
> def myfunc(self):
> pass
>
> o = MyClass()
> print o.myfunc
>    >
>
> import weakref
> r = weakref.ref(o.myfunc)
> print r()
>    None
>
> This is what I do not understand. The object "o" is still alive, and
> therefore the bound method "o.myfunc" shall exists.
>
> Why does the weak reference claim that it is removed? And how can I hold
> the reference to the method until the object is removed?
>

k = o.myfunc
r = weakref.ref(k)
print r()
>  

Don't ask me why! I have just been interested for what you are
trying...
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python and lost files

[Dave Angel]

Carl Banks wrote:

On Sep 30, 11:35 pm, "Timothy W. Grove"  wrote:
  

Recently I purchased some software to recover some files which I had
lost. (A python project, incidentally! Yes, I should have kept better
backups!) They were nowhere to found in the file system, nor in the
recycle bin, but this software was able to locate them and restore them.



I could have used that yesterday, if it were able to work for a
network Samba drive.  (Yeah, not likely.)


  

I was just wondering if there was a way using python to view and recover
files from the hard drive which would otherwise remain lost forever?



Obviously, if that program was able to do it, it's possible.

On Unix-like OSes, and probably others, it's possible to read the raw
data on a disk the same way as you would read any file.  So Python can
do it without any system-level programming.  Recent versions (I think
2.6+) can use mmap, too, now that it supports an offset parameter.

I don't think you can do that in Windows, though.  I think you'd have
to use special system calls (via ctypes, for example).


Carl Banks

  
To write such a program, you have two challenges.  First is to get 
read-access to the raw sectors of the partition, and second, to analyze 
them to discover which ones are interesting, and how they need to be 
combined to reconstruct the lost data.


In Windows, the first challenge is pretty easy for drives other than the 
system drive (usually drive C:, but not necessarily.)  You use one of 
the following:

   \\.\X:where X: is the logical drive letter
  or   \\.\PhysicalDriveN   where N is the hard drive # (0, 1, 2...)   
Normally you'd use this only if the data is on a "deleted" or "foreign" 
partition that Windows doesn't recognize.


Naturally, make sure the scratch files and result files you create are 
going to a different partition/drive.


The second challenge is the file system format.  If you go with physical 
drive, you'll have to parse the partitioning information to find 
individual partitions within the drive, and once you get to a partition, 
you have to parse the particular file system.  Most likely NTFS (which 
has had several versions).  But could be FAT32, FAT16, or a couple of 
other less likely candidates.


While you can also do this for a system partition, there are some 
challenges that I've no relevant experience with.


DaveA

--
http://mail.python.org/mailman/listinfo/python-list

setuptools, accessing ressource files

[Patrick Sabin]

I use setuptools to create a package. In this package I included some 
images and I checked that they are in the egg-file. The problem is how 
can I access the images in the package?


I tried pkgutil.get_data, but only got an IOError, because the EGG-INFO 
directory doesn't exist.


I tried
pkg_resources.get_distribution('dist').get_metadata('images/image.png')
with a similar error (IOError)

What is the best way to access images distributed in an egg file?

-Patrick
--
http://mail.python.org/mailman/listinfo/python-list

Re: easy question, how to double a variable

[Albert van der Horst]

In article ,
daggerdvm   wrote:
>you brain needs error checking!

Whose brain? At least I know this:

Your brain is beyond repair. Go for a brain transplant.

Groetjes Albert

--
-- 
Albert van der Horst, UTRECHT,THE NETHERLANDS
Economic growth -- being exponential -- ultimately falters.
alb...@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst

-- 
http://mail.python.org/mailman/listinfo/python-list

New Python Novice

[baboucarr sanneh]

Hello Everyone,

My name is Baboucarr ..am from the gambia (west africa)..I just read about 
python and i want to know how to program with it..
I would like you guys to help me in my road to becoming a python guru..Am a 
novice so i would welcome any suggestions etc..
Will be posting again soon.Thanks 

Regards, 


$LIM $...@dy

  
_
Show them the way! Add maps and directions to your party invites. 
http://www.microsoft.com/windows/windowslive/products/events.aspx-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Threaded GUI slowing method execution?

[Dave Angel]

Aaron Hoover wrote:
I have a 
wx GUI application that connects to a serial port in a separate 
thread, reads from the port, and then is supposed to put the data it 
finds into a queue to be used by the main GUI thread. Generally 
speaking, it's working as expected.


However, one method (that's part of a library I've written to parse 
the packet structure of the data that's coming over the serial port) 
executes approximately 1000 times slower (50ms vs. 50us) when called 
from the serial management thread in the GUI as compared to calling 
the same function from within a command line Python script. I checked 
it by wrapping the call as follows in both cases  (GUI and command 
line script):


tic = time.time()

print time.time() - tic

All the thread is doing most of the time is sitting around checking 
the serial port for waiting data, reading it, and appending it to a 
list when it finds it. Then, in the same thread, the method that seems 
to be remarkably slow works its way through that list looking for 
packets of data and appending the packet payloads it finds to a queue 
to be handled in some way by the GUI.


My question is, what am I missing about either threading or the fact 
that this is running in a GUI that might explain such a huge slowdown. 
I'm sending data over the serial at a true rate of about 24k bytes per 
second, or approximately 2 packets per ms. Is it too much to ask to be 
able to process this data in realtime from within a GUI (I'm not 
talking about plotting or anything - just read it and find packets)? 
Like I said, the process pretty much runs in realtime from a command 
line script.


This packet parsing needs to happen continuously, so it seems calling 
join() to ensure it's not interrupted by the GUI thread, won't work.


Thanks in advance for your help.

Aaron



Threading in any language is tricky, so it may be just that.  But in 
Python in particular, you have the GIL (Global Interpreter Lock, I 
believe), which usually makes writing threaded code easier, but can 
really hurt efficiency.   If you do a search of this list for GIL, 
you'll see lots of discussion.  Or you can google for "GIL python".  
Look also at http://www.dabeaz.com/python/GIL.pdf


I'm no expert here, but let me try to summarize what I recall.  The GIL 
makes writing background threads easier when all but one of the threads 
are frequently blocked for system calls.  But as soon as you have two 
threads doing "busy work," instead of them getting 50% each, the 
threading overhead goes way up, and you spend much of your time just 
deciding what not to do next.  And it gets enormously worse than that 
when there are more than one CPU available (eg. multicore).


Many people have concluded that (in Python) much of what threads are 
used for should be done with processes.


DaveA

--
http://mail.python.org/mailman/listinfo/python-list

Re: New Python Novice

[Simon Brunning]

2009/10/2 baboucarr sanneh :
> Hello Everyone,
>
> My name is Baboucarr ..am from the gambia (west africa)..

I visited some years back. Friendly people.

> I just read about
> python and i want to know how to program with it..
> I would like you guys to help me in my road to becoming a python guru..Am a
> novice so i would welcome any suggestions etc..

You might want to start with .

-- 
Cheers,
Simon B.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Q: sort's key and cmp parameters

[Duncan Booth]

Paul Rubin  wrote:

> Duncan Booth  writes:
>> > Is there a real-life sorting task that requires (or is far more
>> > efficient with) cmp and can't be easily achieved with key and reverse?
>> > 
>> There is no sorting task that *requires* cmp. If all else fails you can 
>> define a key class to wrap the original wrapper such that comparing the 
>> keys simply calls the comparison function that you *would* have used.
> 
> I would count that as key being far less efficient, though still
> giving the same result.

You'll notice I carefully didn't comment on the efficiency. However, 
without testing it I'm not convinced that it would be 'far less' efficient.

Using cmp2key you have an overhead of one class per sort plus one instance 
for each element being sorted, and an additional one function call for each 
comparison. The only example given so far that would justify needing this 
(sorting trees which require recursive comparison) by its very nature would 
make both of these additional overheads insignificant as each element 
already contains multiple instances and each comparison contains multiple 
function calls.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: weak reference to bound method

[Peter Otten]

Ole Streicher wrote:

> Peter Otten <__pete...@web.de> writes:
>> class Method(object):
>> def __init__(self, obj, func=None):
>> if func is None:
>> func = obj.im_func
>> obj = obj.im_self
> 
> This requires that func is a bound method. What I want is to have a
> universal class that "always" works: with unbound functions, with
> bound function, with lambda expressions, with locally defined functions,

That's left as an exercise to the reader ;)

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: weak reference to bound method

[Ole Streicher]

Hello Peter,

Peter Otten <__pete...@web.de> writes:
>> What I want is to have a universal class that "always" works: with
>> unbound functions, with bound function, with lambda expressions, with
>> locally defined functions,

> That's left as an exercise to the reader ;)

Do you have the feeling that there exists any reader that is able to
solve this exercise? :-)

I am a bit surprised that already such a simple problem is virtually
unsolvable in python. Do you think that my concept of having a DoAsync
class is wrong?

Best regards

Ole
-- 
http://mail.python.org/mailman/listinfo/python-list

RE: New Python Novice

[baboucarr sanneh]

Thanks simon, Can't wait to get my hands dirty on it..


$LIM $...@dy



> From: si...@brunningonline.net
> Date: Fri, 2 Oct 2009 12:32:53 +0100
> Subject: Re: New Python Novice
> To: sanne...@hotmail.com
> CC: python-list@python.org
> 
> 2009/10/2 baboucarr sanneh :
> > Hello Everyone,
> >
> > My name is Baboucarr ..am from the gambia (west africa)..
> 
> I visited some years back. Friendly people.
> 
> > I just read about
> > python and i want to know how to program with it..
> > I would like you guys to help me in my road to becoming a python guru..Am a
> > novice so i would welcome any suggestions etc..
> 
> You might want to start with .
> 
> -- 
> Cheers,
> Simon B.
  
_
Invite your mail contacts to join your friends list with Windows Live Spaces. 
It's easy!
http://spaces.live.com/spacesapi.aspx?wx_action=create&wx_url=/friends.aspx&mkt=en-us-- 
http://mail.python.org/mailman/listinfo/python-list

Re: New Python Novice

[Deven T]

On Fri, Oct 2, 2009 at 4:34 PM, baboucarr sanneh wrote:

>  Hello Everyone,
>
> My name is Baboucarr ..am from the gambia (west africa)..I just read about
> python and i want to know how to program with it..
> I would like you guys to help me in my road to becoming a python guru..Am a
> novice so i would welcome any suggestions etc..
> Will be posting again soon.Thanks
>
>
This is one is very good for beginners:

http://www.swaroopch.com/notes/Python

-Deven
-- 
http://mail.python.org/mailman/listinfo/python-list

RE: New Python Novice

[baboucarr sanneh]

Thanks Deven...
Appreciate it


$LIM $...@dy



Date: Fri, 2 Oct 2009 17:37:28 +0530
Subject: Re: New Python Novice
From: drt...@gmail.com
To: sanne...@hotmail.com
CC: python-list@python.org



On Fri, Oct 2, 2009 at 4:34 PM, baboucarr sanneh  wrote:






Hello Everyone,

My name is Baboucarr ..am from the gambia (west africa)..I just read about 
python and i want to know how to program with it..
I would like you guys to help me in my road to becoming a python guru..Am a 
novice so i would welcome any suggestions etc..

Will be posting again soon.Thanks 



This is one is very good for beginners:



http://www.swaroopch.com/notes/Python



-Deven
  
_
More than messages–check out the rest of the Windows Live™.
http://www.microsoft.com/windows/windowslive/-- 
http://mail.python.org/mailman/listinfo/python-list

Re: weak reference to bound method

[Peter Otten]

Ole Streicher wrote:

> Hello Peter,
> 
> Peter Otten <__pete...@web.de> writes:
>>> What I want is to have a universal class that "always" works: with
>>> unbound functions, with bound function, with lambda expressions, with
>>> locally defined functions,
> 
>> That's left as an exercise to the reader ;)
> 
> Do you have the feeling that there exists any reader that is able to
> solve this exercise? :-)

I was thinking of you.

> I am a bit surprised that already such a simple problem is virtually
> unsolvable in python. Do you think that my concept of having a DoAsync
> class is wrong?

I don't understand the example you give in the other post. 

If you are trying to use reference counting as a means of inter-thread 
communication, then yes, I think that's a bad idea.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: easy question, how to double a variable

[Chris Colbert]

I come from a scientific background, so my approach to the solution of
this problem is a little different.

It makes use of some numerical approximations, but that's not
necessarily a bad thing, because it helps avoid singularities. So it
could be a little more robust than other solutions presented here.

It also has a little more functionality: Say you wanted to return
three times a variable, you wouldnt want to write another function to
do that for you, so now you just pass in how many times you want the
variable repeated as the first parameter.

Hope this helps! Cheers!

import math

def repeat(how_many_times, x):
def f(n):
return 1./(2**n)

def summation(func, howmany):
if howmany == 1:
return func(1)
else:
return func(howmany) + summation(func, howmany-1)

def eulerify(num):
return abs(math.cos(math.pi) + 1j*(math.sin(math.pi))) * num

def get_coefficient(multiplier):
return eulerify(multiplier * summation(f, 100))

return int(eulerify(get_coefficient(how_many_times) * x))



On Fri, Oct 2, 2009 at 2:08 PM, Albert van der Horst
 wrote:
> In article 
> ,
> daggerdvm   wrote:
>>you brain needs error checking!
>
> Whose brain? At least I know this:
>
> Your brain is beyond repair. Go for a brain transplant.
>
> Groetjes Albert
>
> --
> --
> Albert van der Horst, UTRECHT,THE NETHERLANDS
> Economic growth -- being exponential -- ultimately falters.
> alb...@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Regular expression to structure HTML

[Stefan Behnel]

Paul McGuire wrote:
> On Oct 2, 12:10 am, "504cr...@gmail.com" <504cr...@gmail.com> wrote:
>> I'm kind of new to regular expressions, and I've spent hours trying to
>> finesse a regular expression to build a substitution.
>>
>> What I'd like to do is extract data elements from HTML and structure
>> them so that they can more readily be imported into a database.
> 
> Oy! If I had a nickel for every misguided coder who tried to scrape
> HTML with regexes...
> 
> Some reasons why RE's are no good at parsing HTML:
> - tags can be mixed case
> - tags can have whitespace in many unexpected places
> - tags with no body can combine opening and closing tag with a '/'
> before the closing '>', as in ""
> - tags can have attributes that you did not expect (like " CLEAR=ALL>")
> - attributes can occur in any order within the tag
> - attribute names can also be in unexpected upper/lower case
> - attribute values can be enclosed in double quotes, single quotes, or
> even (surprise!) NO quotes

BTW, BeautifulSoup's parser also uses regexes, so if the OP used it, he/she
could claim to have solved the problem "with regular expressions" without
even lying.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

ANN: Humerus 2.1

[Greg Ewing]

Humerus 2.1 is now available:

   http://www.cosc.canterbury.ac.nz/greg.ewing/python/Albow/Humerus-2.1.0.zip

Online documentation:

   http://www.cosc.canterbury.ac.nz/greg.ewing/python/Albow/Humerus-2.1.0/doc/

In this version, the code for handling levels has been separated out
into a new pair of classes. This makes it easier to use Humerus for
games that don't have levels.

What is Humerus?


Humerus is a companion to the Albow widget library for PyGame. It provides a
framework for games made up of a sequence of levels, including user interface
and back-end logic for saving and restoring game state, loading levels, and
sundry other details. There is also optional support for a built-in level
editor, including code for loading and saving levels to be edited, and asking
whether to save modified levels.

Albow can be found here:

   http://www.cosc.canterbury.ac.nz/greg.ewing/python/Albow/
--
http://mail.python.org/mailman/listinfo/python-list

Re: weak reference to bound method

[Ole Streicher]

Hi Peter,

Peter Otten <__pete...@web.de> writes:
> Ole Streicher wrote:
>> Peter Otten <__pete...@web.de> writes:
 What I want is to have a universal class that "always" works: with
 unbound functions, with bound function, with lambda expressions, with
 locally defined functions,
>>> That's left as an exercise to the reader ;)
>> Do you have the feeling that there exists any reader that is able to
>> solve this exercise? :-)
> I was thinking of you.

I could imagine that. However, I am just a beginner in Python and I dont
know which types "callables" there exist in python and which "smart"
ideas (like re-creating them at every call) additionally occur when I
implement such a beast. For example, for locally defined functions, I
have still no idea at all on how to keep them away from the garbage
collector.

>> I am a bit surprised that already such a simple problem is virtually
>> unsolvable in python. Do you think that my concept of having a DoAsync
>> class is wrong?
> I don't understand the example you give in the other post. 

Hmm. I am programming a GUI client application. The client will receive
some input data (via network, and via user input) and shall be updated
after these data.

Unfortunately, the input data (and ofcourse the user input) do not come
regularly; there may times when the data come too fast to process all
of them.

Imagine, for example, that I want to provide a 2d-gaussian fit to some
region of an image and display the result in a separate window, and
updated this whenever the mouse is moved.

The fit takes (let's say) some seconds, so I cannot just call the fit
routine within the mouse move event (this would block other gui
operations, and f.e. the display of the mouse coordinates). So I need
just to trigger the fit routine on mouse movement, and to check
afterwards whether the mouse position is still current.

This is the reason for the DoAsync class: when it is called, it shall
trigger the function that was given in the constructor, t.m.

class MyClass:
def __init__(self):
self.update_fit = DoAsync(update_the_fit)

def mouse_move(self, event):
self.set_coordinates(event.x, event.y)
self.update_fit() # trigger the update_the_fit() call
...

Thus, the mouse_move() method is fast, even if the update_the_fit()
method takes some time to process.

I want to implement it now that DoAsync will be automatically garbage
collected whenever the MyClass object is deleted. Since DoAsync starts
its own thread (which only shall finish when the MyClass object is
deleted), a reference to MyClass (or one of its functions) will keep the
MyClass object from garbage collection.

> If you are trying to use reference counting as a means of inter-thread 
> communication, then yes, I think that's a bad idea.

No; my problem is:

- a thread started in DoAsync will keep the DoAsync object from
  garbage collection
- a reference to a MyClass realted object (the bound method) in DoAsync 
  will thus also prevent the MyClass object from garbage collection
- Even if I dont use the MyClass object anymore, and nobody else uses 
  the DoAsync object, both stand in memory forever, and the thread also 
  never finishes.

Did you get the problem?

Best regards

Ole
-- 
http://mail.python.org/mailman/listinfo/python-list

PyCon 2010

[baboucarr sanneh]

Hi Guys...want to register to the PyCon 2010 conference but i cannot have 
access to the page because it says the page was not found
on the server..I have seen also that they can give out financial aid for those 
who want to go to the conference.am interested in that too as it will be very 
expensive for me to pay for my self and also i would like to be using python in 
my projects..Thanks


$LIM $...@dy

  
_
More than messages–check out the rest of the Windows Live™.
http://www.microsoft.com/windows/windowslive/-- 
http://mail.python.org/mailman/listinfo/python-list

Re: weak reference to bound method

[Ole Streicher]

Hi Peter,

Peter Otten <__pete...@web.de> writes:
> class Method(object):
> def __init__(self, obj, func=None):
> if func is None:
> func = obj.im_func
> obj = obj.im_self

This requires that func is a bound method. What I want is to have a
universal class that "always" works: with unbound functions, with
bound function, with lambda expressions, with locally defined functions,
...

For a user of my class, there is no visible reason, why some of them
shall work, while others dont.

Viele Grüße

Ole
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: weak reference to bound method

[Peter Otten]

Ole Streicher wrote:

> Hi Peter,
> 
> Peter Otten <__pete...@web.de> writes:
>> Ole Streicher wrote:
>>> Peter Otten <__pete...@web.de> writes:
> What I want is to have a universal class that "always" works: with
> unbound functions, with bound function, with lambda expressions, with
> locally defined functions,
 That's left as an exercise to the reader ;)
>>> Do you have the feeling that there exists any reader that is able to
>>> solve this exercise? :-)
>> I was thinking of you.
> 
> I could imagine that. However, I am just a beginner in Python and I dont
> know which types "callables" there exist in python and which "smart"
> ideas (like re-creating them at every call) additionally occur when I
> implement such a beast. For example, for locally defined functions, I
> have still no idea at all on how to keep them away from the garbage
> collector.
> 
>>> I am a bit surprised that already such a simple problem is virtually
>>> unsolvable in python. 

Btw, have you implemented such a design in another language?

>>> Do you think that my concept of having a DoAsync
>>> class is wrong?
>> I don't understand the example you give in the other post.
> 
> Hmm. I am programming a GUI client application. The client will receive
> some input data (via network, and via user input) and shall be updated
> after these data.
> 
> Unfortunately, the input data (and ofcourse the user input) do not come
> regularly; there may times when the data come too fast to process all
> of them.
> 
> Imagine, for example, that I want to provide a 2d-gaussian fit to some
> region of an image and display the result in a separate window, and
> updated this whenever the mouse is moved.
> 
> The fit takes (let's say) some seconds, so I cannot just call the fit
> routine within the mouse move event (this would block other gui
> operations, and f.e. the display of the mouse coordinates). So I need
> just to trigger the fit routine on mouse movement, and to check
> afterwards whether the mouse position is still current.
> 
> This is the reason for the DoAsync class: when it is called, it shall
> trigger the function that was given in the constructor, t.m.
> 
> class MyClass:
> def __init__(self):
> self.update_fit = DoAsync(update_the_fit)
> 
> def mouse_move(self, event):
> self.set_coordinates(event.x, event.y)
> self.update_fit() # trigger the update_the_fit() call
> ...
> 
> Thus, the mouse_move() method is fast, even if the update_the_fit()
> method takes some time to process.
> 
> I want to implement it now that DoAsync will be automatically garbage
> collected whenever the MyClass object is deleted. Since DoAsync starts
> its own thread (which only shall finish when the MyClass object is
> deleted), a reference to MyClass (or one of its functions) will keep the
> MyClass object from garbage collection.
> 
>> If you are trying to use reference counting as a means of inter-thread
>> communication, then yes, I think that's a bad idea.
> 
> No; my problem is:
> 
> - a thread started in DoAsync will keep the DoAsync object from
>   garbage collection
> - a reference to a MyClass realted object (the bound method) in DoAsync
>   will thus also prevent the MyClass object from garbage collection
> - Even if I dont use the MyClass object anymore, and nobody else uses
>   the DoAsync object, both stand in memory forever, and the thread also
>   never finishes.

I think I'd go for a simpler approach, manage the lifetime of MyClass 
instances manually and add a MyClass.close() method that sets a flag which 
in turn is periodically read by DoAsync() and will eventually make it stop.

Peter

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Regular expression to structure HTML

[John]

On Oct 2, 1:10 am, "504cr...@gmail.com" <504cr...@gmail.com> wrote:
> I'm kind of new to regular expressions, and I've spent hours trying to
> finesse a regular expression to build a substitution.
>
> What I'd like to do is extract data elements from HTML and structure
> them so that they can more readily be imported into a database.
>
> No -- sorry -- I don't want to use BeautifulSoup (though I have for
> other projects). Humor me, please -- I'd really like to see if this
> can be done with just regular expressions.
>
> Note that the output is referenced using named groups.
>
> My challenge is successfully matching the HTML tags in between the
> first table row, and the second table row.
>
> I'd appreciate any suggestions to improve the approach.
>
> rText = "8583 href=lic_details.asp?lic_number=8583>New Horizon Technical Academy,
> Inc #4Jefferson70114 tr>9371 lic_number=9371>Career Learning Center valign=top>Jefferson70113"
>
> rText = re.compile(r'()(?P\d+)()( valign=top>)()(?P[A-
> Za-z0-9#\s\S\W]+)().+$').sub(r'LICENSE:\g|NAME:
> \g\n', rText)
>
> print rText
>
> LICENSE:8583|NAME:New Horizon Technical Academy, Inc #4 valign=top>Jefferson70114 valign=top>9371 lic_number=9371>Career Learning Center|PARISH:Jefferson|ZIP:70113

Some suggestions to start off with:

  * triple-quote your multiline strings
  * consider using the re.X, re.M, and re.S options for re.compile()
  * save your re object after you compile it
  * note that re.sub() returns a new string

Also, it sounds like you want to replace the first 2  elements for
each  element with their content separated by a pipe (throwing
away the  tags themselves), correct?

---John
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Timestamps for TCP packets?

[Grant Edwards]

On 2009-10-02, Thomas Johnson  wrote:

> Is there any way to get kernel-level timestamps for TCP packets while
> still using the standard python sockets library for communication?

libpcap

http://sourceforge.net/projects/pylibpcap/

> I need to communicate over a TCP connection as easily as
> possible, but also record the timestamps of the incoming and
> outgoing timestamps at microsecond or nanosecond resolution.

That shouldn't be a problem, except for the communication API
and the capture-packets-with-timestamp API are separate.
Remember that there's no guarantee that send() recv() calls
correspond in any way to the actual packets sent/recevied.

> The sockets library is of course great for the communication,
> and I've seen some python libraries that do packet sniffing
> and record timestamps, but it's not clear that I can do both
> at the same time.

Yes, you can do both at the same time.  But you have to use
separate APIs.

-- 
Grant Edwards   grante Yow! TONY RANDALL!  Is YOUR
  at   life a PATIO of FUN??
   visi.com
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Q: sort's key and cmp parameters

[Scott David Daniels]

Paul Rubin wrote:

I still have never understood why cmp was removed.  Sure, key is more
convenient a lot (or maybe most) of the time, but it's not always.


Not just more convenient.  cmp will always be N log N, in that _every_
comparison runs your function, while key is linear, in that it is run
once per element.  Most cases are moreeasily done with key, and it is
a good idea to make the most accessible way to a sort be the most
efficient one.  In the rare case that you really want each comparison,
the cmp-injection function will do nicely (and can be written as a
recipe.

In short, make the easy path the fast path, and more will use it;
provide two ways, and the first that springs to mind is the one
used.

--Scott David Daniels
scott.dani...@acm.org
--
http://mail.python.org/mailman/listinfo/python-list

Re: Regular expression to structure HTML

[Brian D]

Yes, John, that's correct. I'm trying to trap and discard the  row
 elements, re-formatting with pipes so that I can more readily
import the data into a database. The tags are, of course, initially
useful for pattern discovery. But there are other approaches -- I
could just replace the tags and capture the data as an array.

I'm well aware of the problems using regular expressions for html
parsing. This isn't merely a question of knowing when to use the right
tool. It's a question about how to become a better developer using
regular expressions.

I'm trying to figure out where the regular expression fails. The
structure of the page I'm scraping is uniform in the production of
tags -- it's an old ASP page that pulls data from a database.

What's different in the first  row is the appearance of a comma, a
# pound symbol, and a number (", Inc #4"). I'm making the assumption
that's what's throwing off the remainder of the regular expression --
because (despite the snark by others above) the expression is working
for every other data row. But I could be wrong. Of course, if I could
identify the problem, I wouldn't be asking. That's why I posted the
question for other eyes to review.

I discovered that I may actually be properly parsing the data from the
tags when I tried this test in a Python interpreter:

>>> s = "New Horizon Technical Academy, Inc #4"
>>> p = re.compile(r'([\s\S\WA-Za-z0-9]*)()')
>>> m = p.match(s)
>>> m = p.match(s)
>>> m.group(0)
"New Horizon Technical Academy, Inc #4"
>>> m.group(1)
"New Horizon Technical Academy, Inc #4"
>>> m.group(2)
''

I found it curious that I was capturing the groups as sequences, but I
didn't understand how to use this knowledge in named groups -- or
maybe I am merely mis-identifying the source of the regular expression
problem.

It's a puzzle. I'm hoping someone will want to share the wisdom of
their experience, not criticize for the attempt to learn. Maybe one
shouldn't learn how to use a hammer on a screw, but I wouldn't say
that I have never hammered a screw into a piece of wood just because I
only had a hammer.

Thanks,
Brian


On Oct 2, 8:38 am, John  wrote:
> On Oct 2, 1:10 am, "504cr...@gmail.com" <504cr...@gmail.com> wrote:
>
>
>
> > I'm kind of new to regular expressions, and I've spent hours trying to
> > finesse a regular expression to build a substitution.
>
> > What I'd like to do is extract data elements from HTML and structure
> > them so that they can more readily be imported into a database.
>
> > No -- sorry -- I don't want to use BeautifulSoup (though I have for
> > other projects). Humor me, please -- I'd really like to see if this
> > can be done with just regular expressions.
>
> > Note that the output is referenced using named groups.
>
> > My challenge is successfully matching the HTML tags in between the
> > first table row, and the second table row.
>
> > I'd appreciate any suggestions to improve the approach.
>
> > rText = "8583 > href=lic_details.asp?lic_number=8583>New Horizon Technical Academy,
> > Inc #4Jefferson70114 > tr>9371 > lic_number=9371>Career Learning Center > valign=top>Jefferson70113"
>
> > rText = re.compile(r'()(?P\d+)()( > valign=top>)()(?P[A-
> > Za-z0-9#\s\S\W]+)().+$').sub(r'LICENSE:\g|NAME:
> > \g\n', rText)
>
> > print rText
>
> > LICENSE:8583|NAME:New Horizon Technical Academy, Inc #4 > valign=top>Jefferson70114 > valign=top>9371 > lic_number=9371>Career Learning Center|PARISH:Jefferson|ZIP:70113
>
> Some suggestions to start off with:
>
>   * triple-quote your multiline strings
>   * consider using the re.X, re.M, and re.S options for re.compile()
>   * save your re object after you compile it
>   * note that re.sub() returns a new string
>
> Also, it sounds like you want to replace the first 2  elements for
> each  element with their content separated by a pipe (throwing
> away the  tags themselves), correct?
>
> ---John

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Regular expression to structure HTML

[Brian D]

The other thought I had was that I may not be properly trapping the
end of the first  row, and the beginning of the next  row.


On Oct 2, 8:38 am, John  wrote:
> On Oct 2, 1:10 am, "504cr...@gmail.com" <504cr...@gmail.com> wrote:
>
>
>
> > I'm kind of new to regular expressions, and I've spent hours trying to
> > finesse a regular expression to build a substitution.
>
> > What I'd like to do is extract data elements from HTML and structure
> > them so that they can more readily be imported into a database.
>
> > No -- sorry -- I don't want to use BeautifulSoup (though I have for
> > other projects). Humor me, please -- I'd really like to see if this
> > can be done with just regular expressions.
>
> > Note that the output is referenced using named groups.
>
> > My challenge is successfully matching the HTML tags in between the
> > first table row, and the second table row.
>
> > I'd appreciate any suggestions to improve the approach.
>
> > rText = "8583 > href=lic_details.asp?lic_number=8583>New Horizon Technical Academy,
> > Inc #4Jefferson70114 > tr>9371 > lic_number=9371>Career Learning Center > valign=top>Jefferson70113"
>
> > rText = re.compile(r'()(?P\d+)()( > valign=top>)()(?P[A-
> > Za-z0-9#\s\S\W]+)().+$').sub(r'LICENSE:\g|NAME:
> > \g\n', rText)
>
> > print rText
>
> > LICENSE:8583|NAME:New Horizon Technical Academy, Inc #4 > valign=top>Jefferson70114 > valign=top>9371 > lic_number=9371>Career Learning Center|PARISH:Jefferson|ZIP:70113
>
> Some suggestions to start off with:
>
>   * triple-quote your multiline strings
>   * consider using the re.X, re.M, and re.S options for re.compile()
>   * save your re object after you compile it
>   * note that re.sub() returns a new string
>
> Also, it sounds like you want to replace the first 2  elements for
> each  element with their content separated by a pipe (throwing
> away the  tags themselves), correct?
>
> ---John

-- 
http://mail.python.org/mailman/listinfo/python-list

int rich comparisons

[George Sakkis]

I stumbled upon the following strangeness (python 2.6.2):

>>> getattr(int, '__gt__')


>>> getattr(5, '__gt__')
Traceback (most recent call last):n
  File "", line 1, in 
AttributeError: 'int' object has no attribute '__gt__'

Is this a bug ?

George
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: AJAX Widget Framework

[Daniel Fetchinson]

> I'm looking for an open source, AJAX based widget/windowing framework.
> Here is what I need:
>
> - end user opens up a browser, points it to a URL, logs in
> - on the server site, sits my application, creating a new session for
> each user that is logged in
> - on the server site, I create windows(frames), put widgets on them,
> write event handlers etc. Just like with wx or pygtk.
> - However, windows are created in the user's browser, and events are
> triggered by Javascript, and sent back to server through AJAX.
> - the server side would be - of course - written in Python.
>
> I was looking these projects:
>
> http://www.uize.com/
> http://pyjs.org/
>
> There are many frameworks listed here which I did not check:
> http://internetmindmap.com/javascript_frameworks. I have no idea which
> has Python support, and probably there are only a few that worth looking
> at. I wonder if you could tell me which are the pros and contras for
> these frameworks. If there is a comparison available on the NET,
> pointing me to the right URL would help a lot.
>
> The bottom line...
>
> My main goal is to develop enterprise class OLTP database applications.
> I'll need grid widget to display data in tabular format, and I'll use
> events heavily for data manipulation, live search, tooltips etc. I'm
> familiar with wxWidgets, pygtk and other toolkits, but not with AJAX. I
> have never used a system like that.

I do things like these via turbogears on the server side and extjs on
the client:

turbogears.org
extjs.com

HTH,
Daniel



-- 
Psss, psss, put it down! - http://www.cafepress.com/putitdown
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Concurrent threads to pull web pages?

[exarkun]

On 05:48 am, wlfr...@ix.netcom.com wrote:

On Fri, 02 Oct 2009 01:33:18 -, exar...@twistedmatrix.com declaimed
the following in gmane.comp.python.general:

There's no need to use threads for this.  Have a look at Twisted:

  http://twistedmatrix.com/trac/


Strange... While I can easily visualize how to convert the 
problem

to a task pool -- especially given that code to do a single occurrence
is already in place...

... conversion to an event-dispatch based system is something 
/I/

can not imagine...


The cool thing is that there's not much conversion to do from the single 
request version to the multiple request version, if you're using 
Twisted.  The single request version looks like this:


   getPage(url).addCallback(pageReceived)

And the multiple request version looks like this:

   getPage(firstURL).addCallback(pageReceived)
   getPage(secondURL).addCallback(pageReceived)

Since the APIs don't block, doing things concurrently ends up being the 
easy thing.


Not to say it isn't a bit of a challenge to get into this mindset, but I 
think anyone who wants to put a bit of effort into it can manage. :) 
Getting used to using Deferreds in the first place (necessary to 
write/use even the single request version) is probably where more people 
have trouble.


Jean-Paul
--
http://mail.python.org/mailman/listinfo/python-list

question about the GC implementation

[Francis Moreau]

Hello,

I'm looking at gcmodule.c and in move_unreachable() function, the code
assumes that if an object has its gc.gc_refs stuff to 0 then it *may*
be unreachable.

How can an object tagged as unreachable could suddenly become
reachable later ?

Thanks
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: weak reference to bound method

[Ole Streicher]

Hi Peter,

Peter Otten <__pete...@web.de> writes:
 I am a bit surprised that already such a simple problem is virtually
 unsolvable in python. 

> Btw, have you implemented such a design in another language?

No. 

> I think I'd go for a simpler approach, manage the lifetime of MyClass 
> instances manually and add a MyClass.close() method that sets a flag which 
> in turn is periodically read by DoAsync() and will eventually make it stop.

Has the disadvantage that I rely on the user. We already have a garbage
collector, so why not use it? It is exactly made for what I want: delete
unused objects.

The solution I have now is still untested, but maybe the separation of
the thread will work:
--<8
import threading
import weakref

class AsyncThread(threading.Thread):
def __init__(self, func):
threading.Thread.__init__(self)
self.setDaemon(True)
self._cond = threading.Condition()
self._func = weakref.ref(func, self.run)
self.start()

def run(self):
while self._func():
with self._cond:
while not self.scheduled:
self._cond.wait()
self.scheduled = False
self._func()
func = self._func()
if func:
func()

def __call__(self):
with self._cond:
self.scheduled = True
self._cond.notify()

class DoAsync(object):
def __init__(self, func):
self._func = func
self._thread = AsyncThread(self._func)

def __call__(self):
self._thread()
--<8

The AsyncThread has now no reference that could prevent the referenced
from the GC. And the function is always stored in the DoAsync object and
will be only collected if this is removed (what is the case if the
parent object is being deleted)

Regards

Ole
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: int rich comparisons

[Mark Dickinson]

On Oct 2, 3:52 pm, George Sakkis  wrote:
> I stumbled upon the following strangeness (python 2.6.2):
>
> >>> getattr(int, '__gt__')
>
> 
>
> >>> getattr(5, '__gt__')
>
> Traceback (most recent call last):n
>   File "", line 1, in 
> AttributeError: 'int' object has no attribute '__gt__'
>
> Is this a bug ?

I don't think so.  Notice that in the first example you're picking
up the __gt__ method of the *type* object, not the int object;  i.e.,
the method that's executed when you do:

>>> int > float
True

You get similar results with __call__:

>>> int.__call__

>>> int.__call__(3)
3
>>> (5).__call__
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: 'int' object has no attribute '__call__'

Mark
-- 
http://mail.python.org/mailman/listinfo/python-list

Simple Python struct issue

[Carlo DiCelico]

I saw an article on O'Reilly about using NumPy and Dislin to analyze
and visualize WAV files. It's a really fantastic article but was a
little out of date. I updated the script to work with the newer
modules &etc but am still having trouble getting it working.

The line

temp[i,:] = array(struct.unpack("%dB"%(fft_length), tempb),Float) -
128.0

always returns the same error: "Traceback (most recent call last):
  File "pysono.py", line 31, in 
   temp[i,:] = array(struct.unpack("%dB"%(fft_length),tempb),float) -
128.0
struct.error: unpack requires a string argument of length 256" when I
do python pysono.py test.wav 256

I'm sure it's probably something simple but I just can't see what it
is!

Here's the original code: http://onlamp.com/python/2001/01/31/graphics/pysono.py

Thanks!
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Simple Python struct issue

[Simon Forman]

On Fri, Oct 2, 2009 at 12:07 PM, Carlo DiCelico
 wrote:
> I saw an article on O'Reilly about using NumPy and Dislin to analyze
> and visualize WAV files. It's a really fantastic article but was a
> little out of date. I updated the script to work with the newer
> modules &etc but am still having trouble getting it working.
>
> The line
>
> temp[i,:] = array(struct.unpack("%dB"%(fft_length), tempb),Float) -
> 128.0
>
> always returns the same error: "Traceback (most recent call last):
>  File "pysono.py", line 31, in 
>   temp[i,:] = array(struct.unpack("%dB"%(fft_length),tempb),float) -
> 128.0
> struct.error: unpack requires a string argument of length 256" when I
> do python pysono.py test.wav 256
>
> I'm sure it's probably something simple but I just can't see what it
> is!
>
> Here's the original code: 
> http://onlamp.com/python/2001/01/31/graphics/pysono.py
>
> Thanks!

In:

struct.unpack("%dB" % (fft_length), tempb)

tempb is not length 256.

Also, note that (foo) is the same as just foo.  To create a tuple of
length 1 you must say (foo,)

HTH,
~Simon
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: re.sub do not replace portion of match

[J Wolfe]

Thanks Duncan,

I did look at that, but it was kinda greek to me.  Thanks for pulling
out the part I was looking for that should do the trick.

Jonathan

> http://www.python.org/doc/current/library/re.html#re.sub
>
> > Backreferences, such as \6, are replaced with the substring matched by
> > group 6 in the pattern.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Simple Python struct issue

[Carlo DiCelico]

On Oct 2, 12:15 pm, Simon Forman  wrote:
> On Fri, Oct 2, 2009 at 12:07 PM, Carlo DiCelico
>
>
>
>
>
>  wrote:
> > I saw an article on O'Reilly about using NumPy and Dislin to analyze
> > and visualize WAV files. It's a really fantastic article but was a
> > little out of date. I updated the script to work with the newer
> > modules &etc but am still having trouble getting it working.
>
> > The line
>
> > temp[i,:] = array(struct.unpack("%dB"%(fft_length), tempb),Float) -
> > 128.0
>
> > always returns the same error: "Traceback (most recent call last):
> >  File "pysono.py", line 31, in 
> >   temp[i,:] = array(struct.unpack("%dB"%(fft_length),tempb),float) -
> > 128.0
> > struct.error: unpack requires a string argument of length 256" when I
> > do python pysono.py test.wav 256
>
> > I'm sure it's probably something simple but I just can't see what it
> > is!
>
> > Here's the original 
> > code:http://onlamp.com/python/2001/01/31/graphics/pysono.py
>
> > Thanks!
>
> In:
>
> struct.unpack("%dB" % (fft_length), tempb)
>
> tempb is not length 256.
>
> Also, note that (foo) is the same as just foo.  To create a tuple of
> length 1 you must say (foo,)
>
> HTH,
> ~Simon

I'm sorry, I'm not sure what you're referring to when you say "Also,
note that (foo) is the same as just foo.  To create a tuple of length
1 you must say (foo,)". The 256 is passed into the script as an
argument and then assigned to the variable fft_length, which you can
see in that line. So, whatever value I pass in comes out in the error.
What you're saying is that the error happens because tempb isn't the
length of any of those values (i.e., 128, 256, 512, etc)? (1) How
could I determine the length of tempb? and (2) It's doing this ->
tempb = fp.readframes(fft_length); right before doing the struct.unpack
(); could the splitting of the frames into frames of length fft_length
be causing this error?

Thanks for the help!
Carlo
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Simple Python struct issue

[MRAB]

Carlo DiCelico wrote:

I saw an article on O'Reilly about using NumPy and Dislin to analyze
and visualize WAV files. It's a really fantastic article but was a
little out of date. I updated the script to work with the newer
modules &etc but am still having trouble getting it working.

The line

temp[i,:] = array(struct.unpack("%dB"%(fft_length), tempb),Float) -
128.0

always returns the same error: "Traceback (most recent call last):
  File "pysono.py", line 31, in 
   temp[i,:] = array(struct.unpack("%dB"%(fft_length),tempb),float) -
128.0
struct.error: unpack requires a string argument of length 256" when I
do python pysono.py test.wav 256

I'm sure it's probably something simple but I just can't see what it
is!

Here's the original code: http://onlamp.com/python/2001/01/31/graphics/pysono.py


.readframes(n) returns (at most) n _frames_ as a bytestring (str), not n
bytes. I tried reading 256 frames from a .wav file containing stereo at
16 bits per channel and got 1024 bytes (4 bytes per frame, not
surprisingly!).
--
http://mail.python.org/mailman/listinfo/python-list

Re: M2Crypto 0.20.1 won't build on Red Hat Linux

[John Nagle]

Heikki Toivonen wrote:

John Nagle wrote:

The right question is "uname --hardware-platform".  That returns "i386" if
running on something emulating a 386, even it it's 64-bit capable.


Thanks, I'll make a note that I'll need to clarify that part.


With that change, the build runs to completion and and the regression tests
mostly run.  Test failure occurred at:

==
FAIL: test_cipher_ok (tests.test_ssl.MiscSSLClientTestCase)
--
Traceback (most recent call last):
  File
"/var/www/vhosts/sitetruth.com/private/downloads/M2Crypto/M2Crypto-0.20.1/tests/test_ssl.py",
line 485, in test_cipher_ok
assert s.get_cipher().name() == 'AES128-SHA', s.get_cipher().name()
AssertionError: AES256-SHA

--
Ran 226 tests in 91.672s

Is 'AES128-SHA' a cypher that Fedora Core doesn't support, or what?


That seems strange. OpenSSL does allow fine tuning what ciphers to build
and operate with, but this is the first I hear AES128-SHA might not be
turned on anymore. And even stranger still is that the test is
explicitly asking for 128 yet it is getting 256 back. Could you rerun
the test again just to make sure you can repeat it, and if so, let me
know the fedora core, openssl, swig and python versions?


  Ran it again.  Same error.

python setup.py test
...
test_blocking0 (tests.test_ssl.MiscSSLClientTestCase) ... ok
test_blocking1 (tests.test_ssl.MiscSSLClientTestCase) ... ok
test_cipher_mismatch (tests.test_ssl.MiscSSLClientTestCase) ... ok
test_cipher_ok (tests.test_ssl.MiscSSLClientTestCase) ... FAIL
test_info_callback (tests.test_ssl.MiscSSLClientTestCase) ... LOOP: SSL connect: 
before/connect initialization

LOOP: SSL connect: SSLv2/v3 write client hello A
...
test_26_compat (tests.test_ssl.FtpslibTestCase) ... ok

==
FAIL: test_cipher_ok (tests.test_ssl.MiscSSLClientTestCase)
--
Traceback (most recent call last):
  File 
"/var/www/vhosts/sitetruth.com/private/downloads/M2Crypto/M2Crypto-0.20.1/tests/test_ssl.py", 
line 485, in test_cipher_ok

assert s.get_cipher().name() == 'AES128-SHA', s.get_cipher().name()
AssertionError: AES256-SHA

--
Ran 226 tests in 166.875s

FAILED (failures=1)

System info:

-bash-3.1$ uname -r -v
2.6.18-1.2239.fc5smp #1 SMP Fri Nov 10 13:22:44 EST 2006

-bash-3.1$ python --version
Python 2.5

-bash-3.1$ swig -version

SWIG Version 1.3.31

Compiled with i386-redhat-linux-g++ [i386-redhat-linux-gnu]
Please see http://www.swig.org for reporting bugs and further information

-bash-3.1$ openssl
OpenSSL> version
OpenSSL 0.9.8a 11 Oct 2005
OpenSSL>

So, per "http://chandlerproject.org/bin/view/Projects/MeTooCrypto":

Required Python version: 2.4 or newer.  Using Python 2.5.  REQUIREMENT MET.
Required SWIG version: 1.3.30 or newer.  Using SWIG 1.3.31  REQUIREMENT MET.
Required OpenSSL version: 0.9.8 or newer: Using OpenSSL 0.9.8a. REQUIREMENT MET.

John Nagle





--
http://mail.python.org/mailman/listinfo/python-list

Re: Threaded GUI slowing method execution?

[sturlamolden]

On 2 Okt, 13:29, Dave Angel  wrote:

> Many people have concluded that (in Python) much of what threads are
> used for should be done with processes.

Remember that threads were invented long before multi-core CPUs were
common. Java had threads before the VM could support more than one
CPU. The early Linux kernel had threads despite of the BKL, etc.
Threads have more usage than concurrent CPU bound busy work.

If you need to program multi-cores for speed, consider that you incur
a 200x speed penalty from Python alone.  If you are worried about
speed, chances are you are not using Python anyway.

If you still have "need for speed" on a multicore, you can use Cython
and release the GIL when appropriate. Then launch multiple Python
threads and be happy.

In C, C++ and Fortran, one would not use threads for multicore
programming at all, but OpenMP.

Using more than one process is always an option, i.e. os.fork if you
have it or multiprocessing if you don't. Processes don't share GIL.

S.M.











-- 
http://mail.python.org/mailman/listinfo/python-list

Fast decimal arithmetic module released

[Stefan Krah]

Hi,

today I have released the following packages for fast arbitrary precision
decimal arithmetic:


1. libmpdec


Libmpdec is a C (C++ ready) library for arbitrary precision decimal
arithmetic.  It is a complete implementation of Mike Cowlishaw's
General Decimal Arithmetic specification.


2. fastdec.so
==

Fastdec.so is a Python C module with the same functionality as decimal.py.
With some restrictions, code written for for decimal.py should work
identically.


3. deccheck.py
===

deccheck.py performs redundant calculations using both decimal.py and
fastdec.so. For each calculation the results of both modules are compared
and an exception is raised if they differ. This module was mainly developed
for testing, but could in principle be used for redundant calculations.



Correctness


Libmpdec passes IBM's official test suite and a multitude of additional tests.
Fastdec.so passes (with minor modifications) all Python unit tests. When run
directly, deccheck.py performs very exhaustive tests that compare fastdec.so
with decimal.py.

All tests complete successfully under Valgrind.


Speed
==

In a couple of initial benchmarks, libmpdec compares very well against
decNumber and the Intel decimal library. For very large numbers, the speed
is roughly the same as the speed of the apfloat library.

Fastdec.so compares quite well against gmpy and even native Python floats.
In the benchmarks, it is significantly faster than Java's BigDecimal class.


Portability


All tests have been completed on Linux 64/32-bit, Windows 64/32-bit, OpenSolaris
32-bit, OpenBSD 32-bit and Debian Mips 32-bit. For 32-bit platforms there is
a pure ANSI C version, 64 bit platforms require a couple of asm lines.



Further information and downloads at:

http://www.bytereef.org/libmpdec.html



Stefan Krah



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Threaded GUI slowing method execution?

[Ole Streicher]

sturlamolden  writes:
> On 2 Okt, 13:29, Dave Angel  wrote:
> If you are worried about speed, chances are you are not using Python
> anyway.

I *do* worry about speed. And I use Python. Why not? There are powerful
libraries available.

> If you still have "need for speed" on a multicore, you can use Cython
> and release the GIL when appropriate. Then launch multiple Python
> threads and be happy.

Usually this is not an option: numpy is AFAIK not available for Cython,
neither is scipy (ofcourse).

Especially for numeric calculations, speed *matters*.

> Using more than one process is always an option, i.e. os.fork if you
> have it or multiprocessing if you don't. Processes don't share GIL.

Not if the threads/processes need to share lots of data. Interprocess
communication can be very expensive -- even more if one needs to share
Python objects. Also, the support of sharing python objects between
processes seems to me not well supported at least by the standard python
libs.

Ole
-- 
http://mail.python.org/mailman/listinfo/python-list

Python SSL Client Authentication

[AJ Mayorga]

I have been able to use the ssl module in Python 2.6 to setup a server that
requires

The client to authenticate via an x509 certificates, but I want to be able
to override the client

Authentication process or catch the exception so that I can do redirects
etc. if the client authentication

Fails. Does anyone have a code snippet for this.

 

AJM

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Threaded GUI slowing method execution?

[sturlamolden]

On 2 Okt, 02:51, Aaron Hoover  wrote:

> All the thread is doing most of the time is sitting around checking  
> the serial port for waiting data, reading it, and appending it to a  
> list when it finds it.

Do your threads ever block waiting for I/O? If they do, is the GIL
released while they are blocked?

If they don't, you could add a time.sleep(0) to the loop that polls
the port. It will return the rest of the current time slice back to
the system.














Then, in the same thread, the method that seems  
> to be remarkably slow works its way through that list looking for  
> packets of data and appending the packet payloads it finds to a queue  
> to be handled in some way by the GUI.
>
> My question is, what am I missing about either threading or the fact  
> that this is running in a GUI that might explain such a huge slowdown.  
> I'm sending data over the serial at a true rate of about 24k bytes per  
> second, or approximately 2 packets per ms. Is it too much to ask to be  
> able to process this data in realtime from within a GUI (I'm not  
> talking about plotting or anything - just read it and find packets)?  
> Like I said, the process pretty much runs in realtime from a command  
> line script.
>
> This packet parsing needs to happen continuously, so it seems calling  
> join() to ensure it's not interrupted by the GUI thread, won't work.
>
> Thanks in advance for your help.
>
> Aaron

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: The Python: Rag October issue available

[steven.oldner]

On Oct 2, 11:14 am, Bernie  wrote:
> The Python: Rag October issue available
>
> The October issue of The Python: Rag is available at:
>
> http://www.pythonrag.org
>
> A monthly, free, community run, Python magazine - issues are in pdf
> format, intended for anyone interested in Python, without being
> particularly serious.  If you have anything you would like to say about
> Python, please contribute.

Thanks!  Any way to subscribe to it?
-- 
http://mail.python.org/mailman/listinfo/python-list

epydoc - can I provide multiple dirs to parse

[Medi]

Can I present multiple directories to epydoc to process. For example

epydoc -option -option -option dir_1 dir_2 dir_3

where dir_i is a directory containing some python src codes ?

I am currently running it on two directories with about 10M + 20 Meg
which takes a very long time (more than 40 minutes) and strace(1) is
showing mmap(2) ops back to backmight not be related but the
process gets wedged somewhere

Thanks
Medi
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Threaded GUI slowing method execution?

[sturlamolden]

On 2 Okt, 20:19, Ole Streicher  wrote:

> I *do* worry about speed. And I use Python. Why not? There are powerful
> libraries available.

I do as well. But "powerful libraries" should release the GIL. Let me
rephrase that: I am not worried about speed in the part of my code
that uses Python.


> Usually this is not an option: numpy is AFAIK not available for Cython,
> neither is scipy (ofcourse).

Anything available to Python is available to Cython.

Cython even has a special syntax for working with NumPy arrays.


> > Using more than one process is always an option, i.e. os.fork if you
> > have it or multiprocessing if you don't. Processes don't share GIL.
>
> Not if the threads/processes need to share lots of data. Interprocess
> communication can be very expensive -- even more if one needs to share
> Python objects.

I have written NumPy arrays that uses named shared memory as buffer.
They are pickled by name (i.e. the buffer is not copied) and therefore
very efficient when used with multiprocessing.
http://folk.uio.no/sturlamo/python/sharedmem-feb13-2009.zip

IPC without shared memory is generally cheap. The overhead of a pipe
or a unix domain socket is little more than that of a memcpy. The
expensive part is serializing and deserializing the Python object.

Also consider that the majority of the world's supercomputers are
programmed with MPI, which uses processes and IPC instead of threads.
On clusters, threads are not even an option. On shared memory
machines, MPI tends to be more efficient than threads/OpenMP (there
are often issues with cache use and false sharing when using threads).

S.M.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Regular expression to structure HTML

[504cr...@gmail.com]

Screw:

>>> html = """  

14313


Python
Hammer Institute #2


Jefferson


70114


  

  

8583


New
Screwdriver Technical Academy, Inc #4


Jefferson


70114


  

  

9371


Career
RegEx Center


Jefferson


70113


  """

Hammer:

First remove line returns.
Then remove extra spaces.
Then insert a line return to restore logical rows on each 
combination. For more information, see: http://www.qc4blog.com/?p=55

>>> s = re.sub(r'\n','', html)
>>> s = re.sub(r'\s{2,}', '', s)
>>> s = re.sub('()()', r'\1\n\2', s)
>>> print s
14313Python Hammer Institute #2Jefferson70114
8583New Screwdriver Technical Academy, Inc #4Jefferson70114
9371Career RegEx CenterJefferson70113
>>> p = re.compile(r"()(?P\d+)()(>> valign=top>)(>> href=lic_details\.asp)(\?lic_number=\d+)(>)(?P[\s\S\WA-Za-z0-9]*?)()()(?:>>  valign=top>)(?P[\s\WA-Za-z]+)()(>> valign=top>)(?P\d+)()()$", re.M)
>>> n = 
>>> p.sub(r'LICENSE:\g|NAME:\g|PARISH:\g|ZIP:\g',
>>>  s)
>>> print n
LICENSE:14313|NAME:Python Hammer Institute #2|PARISH:Jefferson|ZIP:
70114
LICENSE:8583|NAME:New Screwdriver Technical Academy, Inc #4|
PARISH:Jefferson|ZIP:70114
LICENSE:9371|NAME:Career RegEx Center|PARISH:Jefferson|ZIP:70113
>>>

The solution was to escape the period in the ".asp" string, e.g.,
"\.asp". I also had to limit the pattern in the  grouping by
using a "?" qualifier to limit the "greediness" of the "*" pattern
metacharacter.

Now, who would like to turn that re.compile pattern into a MULTILINE
expression, combining the re.M and re.X flags?

Documentation says that one should be able to use the bitwise OR
operator (e.g., re.M | re.X), but I sure couldn't get it to work.

Sometimes a hammer actually is the right tool if you hit the screw
long and hard enough.

I think I'll try to hit some more screws with my new hammer.

Good day.

On Oct 2, 12:10 am, "504cr...@gmail.com" <504cr...@gmail.com> wrote:
> I'm kind of new to regular expressions, and I've spent hours trying to
> finesse a regular expression to build a substitution.
>
> What I'd like to do is extract data elements from HTML and structure
> them so that they can more readily be imported into a database.
>
> No -- sorry -- I don't want to use BeautifulSoup (though I have for
> other projects). Humor me, please -- I'd really like to see if this
> can be done with just regular expressions.
>
> Note that the output is referenced using named groups.
>
> My challenge is successfully matching the HTML tags in between the
> first table row, and the second table row.
>
> I'd appreciate any suggestions to improve the approach.
>
> rText = "8583 href=lic_details.asp?lic_number=8583>New Horizon Technical Academy,
> Inc #4Jefferson70114 tr>9371 lic_number=9371>Career Learning Center valign=top>Jefferson70113"
>
> rText = re.compile(r'()(?P\d+)()( valign=top>)()(?P[A-
> Za-z0-9#\s\S\W]+)().+$').sub(r'LICENSE:\g|NAME:
> \g\n', rText)
>
> print rText
>
> LICENSE:8583|NAME:New Horizon Technical Academy, Inc #4 valign=top>Jefferson70114 valign=top>9371 lic_number=9371>Career Learning Center|PARISH:Jefferson|ZIP:70113

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Fast decimal arithmetic module released

[Mark Dickinson]

On Oct 2, 8:53 pm, Stefan Krah  wrote:
> Hi,
>
> today I have released the following packages for fast arbitrary precision
> decimal arithmetic:
>
[...]

Nice!  I'd been wondering how you'd been finding all those decimal.py
bugs.  Now I know.  :)

--
Mark
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Simple Python struct issue

[Simon Forman]

On Fri, Oct 2, 2009 at 12:35 PM, Carlo DiCelico
 wrote:
> On Oct 2, 12:15 pm, Simon Forman  wrote:
>> On Fri, Oct 2, 2009 at 12:07 PM, Carlo DiCelico
>>  wrote:
>> > I saw an article on O'Reilly about using NumPy and Dislin to analyze
>> > and visualize WAV files. It's a really fantastic article but was a
>> > little out of date. I updated the script to work with the newer
>> > modules &etc but am still having trouble getting it working.
>>
>> > The line
>>
>> > temp[i,:] = array(struct.unpack("%dB"%(fft_length), tempb),Float) -
>> > 128.0
>>
>> > always returns the same error: "Traceback (most recent call last):
>> >  File "pysono.py", line 31, in 
>> >   temp[i,:] = array(struct.unpack("%dB"%(fft_length),tempb),float) -
>> > 128.0
>> > struct.error: unpack requires a string argument of length 256" when I
>> > do python pysono.py test.wav 256
>>
>> > I'm sure it's probably something simple but I just can't see what it
>> > is!
>>
>> > Here's the original 
>> > code:http://onlamp.com/python/2001/01/31/graphics/pysono.py
>>
>> > Thanks!
>>
>> In:
>>
>> struct.unpack("%dB" % (fft_length), tempb)
>>
>> tempb is not length 256.
>>
>> Also, note that (foo) is the same as just foo.  To create a tuple of
>> length 1 you must say (foo,)
>>
>> HTH,
>> ~Simon
>
> I'm sorry, I'm not sure what you're referring to when you say "Also,
> note that (foo) is the same as just foo.  To create a tuple of length
> 1 you must say (foo,)".

Sorry about that.  All I mean is that in python putting parentheses
around an expression "(some_object)" does not create a tuple unless
you also use a comma ","

I hope the following examples show what I mean:

In [1]: (2)
Out[1]: 2

In [2]: (2,)
Out[2]: (2,)

In [3]: 2,
Out[3]: (2,)

In [4]: n = 2,

In [5]: "%s" % (n,)
Out[5]: '(2,)'

In [6]: "%s" % (n)
Out[6]: '2'

In [7]: "%s" % n
Out[7]: '2'


Your code:

"%dB" % (fft_length)

is the same as

"%dB" % fft_length

But what you probably meant was

"%dB" % (fft_length,)

Note the comma.  People do this to prevent string formatting errors if
fft_length should ever happen to be a tuple rather than an int (or
whatever.)


> The 256 is passed into the script as an
> argument and then assigned to the variable fft_length, which you can
> see in that line. So, whatever value I pass in comes out in the error.
> What you're saying is that the error happens because tempb isn't the
> length of any of those values (i.e., 128, 256, 512, etc)?

Yes, whatever tempb is, it isn't "a string argument of length 256" as
the traceback says.

> (1) How
> could I determine the length of tempb?

If it's a string, len(tempb) will return it's length.

> and (2) It's doing this ->
> tempb = fp.readframes(fft_length); right before doing the struct.unpack
> (); could the splitting of the frames into frames of length fft_length
> be causing this error?

See MRAB's reply. :]

> Thanks for the help!

You're very welcome.

One other thing, in the code you posted it has "Float", but in the
traceback it has "float".  Generally speaking when you post code to a
newsgroup for help you should paste in the exact code, rather than
retyping it manually.

Warm regards, and happy hacking!
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: AJAX Widget Framework

[Aaron Watters]

On Oct 1, 2:01 pm, Laszlo Nagy  wrote:
> I'm looking for an open source, AJAX based widget/windowing framework.
> Here is what I need:
>
> - end user opens up a browser, points it to a URL, logs in
> - on the server site, sits my application, creating a new session for
> each user that is logged in
> - on the server site, I create windows(frames), put widgets on them,
> write event handlers etc. Just like with wx or pygtk.
> - However, windows are created in the user's browser, and events are
> triggered by Javascript, and sent back to server through AJAX.
> - the server side would be - of course - written in Python

WHIFF provides support for these type of AJAX
capabilities.  Please have a look at the AJAX calculator
tutorial at

http://aaron.oirt.rutgers.edu/myapp/docs/W1100_1400.calc

for an example with discussion of
some of the built in support for AJAX.
You could probably use this as a starting point for building
your "grid" widget.

I've used WHIFF with jquery and mootools javascript libraries
with much success.

Please let me know what you think.

   -- Aaron Watters

===

Ban DHMO!
   http://www.dhmo.org/
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Threaded GUI slowing method execution?

[Dave Angel]

(you responded off-list, which isn't the way these mailing lists work.  
So I'm pasting your message back to the list, with my response at the end)


Aaron Hoover wrote:




But as soon as you have two threads doing "busy work," instead of 
them getting 50% each, the threading overhead goes way up, and you 
spend much of your time just deciding what not to do next.


Yeah, I had briefly read some grumblings regarding the GIL, but 
naively assumed it wouldn't bite me too badly.



And it gets enormously worse than that when there are more than one 
CPU available (eg. multicore).


Well, I'm running on a MacBook Pro, so count me in for that multicore 
performance degradation!





Many people have concluded that (in Python) much of what threads are 
used for should be done with processes.


Presumably you're referring to the processing module (pre 2.6 - 
multiprocessing in 2.6)? Do you have any canonical references or 
example code for using processing in the producer/consumer type 
paradigm I'm trying to use in my application. Specially, do you know 
if it's possible for processes to post wx events that can be handled 
by the main GUI thread?



Thanks for the help.

Cheers,
Aaron
Actually, I was thinking of the subprocess module (introduced in 2.4).  
But the multiprocessing module would be useful if you were porting 
threading code to a process model.


The real point is the distinction between threads and separate 
processes.  If the processes are separate, then the only time 
synchronization is needed is when they communicate with each other, or 
with other shared services.  So on a multicore system they each run at 
full tilt till they communicate.


There are tons of ways to communicate between processes, though you 
can't do the simple variable sharing that threads can (sometimes) get 
away with.  I would normally point you to queues, but there a number of 
possibilities.  And since the one process is running a GUI event loop, 
you might want to piggyback on the OS capability to post events between 
processes.  The code might end up OS-dependent, but I'd bet the overhead 
will be minimal.  What is your target operating system?


Your numbers in the original message make me nervous;  sending an event 
between processes (or even threads) every 0.5 millisecond is 
impossible.  But I think you might have been confusing bytes and packets.


There could very well be multiprocess support in wxPython.  I'd check 
there first, before re-inventing the wheel.  Presumably you know of the 
wxPython news group, hosted on Google groups?


DaveA
--
http://mail.python.org/mailman/listinfo/python-list

Opinions, please, on PEP 8 and local, 3rd party imports

[Philip Semanchuk]

Hi all,
Our project uses some libraries that were written by 3rd parties (i.e.  
not us). These libraries fit into a single Python file and live in our  
source tree alongside other modules we've written. When our app is  
distributed, they'll be included in the installation. In other words,  
they're not prerequisites that we'll make the user install, and they  
won't live in site-packages.


For example, we use configobj from , and I think Beautiful Soup can be installed as a single file too,  
although we're not using that particular module.


PEP 8  says the following:

   Imports should be grouped in the following order:
   1. standard library imports
   2. related third party imports
   3. local application/library specific imports


I'm not sure in which category to place local, 3rd-party modules like  
configobj.


Clearly, they belong in category 3 since they're local.

Clearly, they also belong in category 2 since they're 3rd party  
modules, and explicitly labeling an imported module as "this is code  
we didn't write" is useful. But I've always reserved category 2  for  
external libraries (e.g. numpy, wxPython, Twisted, PIL, etc.).


Clearly, the best choice is category 2.5?

In your Pythonic opinion, should 3rd-party modules that live alongside  
homegrown code be listed in import category 2 or 3?


Thanks
Philip
--
http://mail.python.org/mailman/listinfo/python-list

store encrypted data in sqlite ?

[Stef Mientki]

hello,

I want to store some fields in an sqlite database.

I use ezPyCrypto to encrypt and decrypt:

User = ['z684684', 'Mientki, Stef', 1,1,0,1,1 ]

encryption_key_1 = ezPyCrypto.key ( 512 )

SQL_Base = 'insert or replace into __USERS__ values ('
for field in User :
   SQL += ",'" + encryption_key_1.encString ( str ( item ))+ "'"
SQL += ')'


Now this fails, probably, because the second character of the encrypted 
string is a binary zero.


By trial and error, I found a work around,
but I'm not sure this will garantee that it will work always:
by converting the encrypted buffer with base64.encode:

   SQL += ",'" + base64.encodestring(EnCrypt_1 ( str ( item )))+ "'"

Will this method work always ?
Are there better methods ?

thanks,
Stef Mientki


--
http://mail.python.org/mailman/listinfo/python-list

Re: Opinions, please, on PEP 8 and local, 3rd party imports

[Duncan Booth]

Philip Semanchuk  wrote:

> In your Pythonic opinion, should 3rd-party modules that live alongside  
> homegrown code be listed in import category 2 or 3?
> 
PEP 8 also starts by saying "This document gives coding conventions for the 
Python code comprising the standard library in the main Python 
distribution."

So while you are free to try to apply it to other code there's no 
obligation on you to take all or even any of it as gospel.

You suggest option 2.5 sounds best. Sounds good to me. If it's good for you 
then go for it.

-- 
http://mail.python.org/mailman/listinfo/python-list

RELEASED Python 2.6.3

[Barry Warsaw]

On behalf of the Python community, I'm happy to announce the  
availability of Python 2.6.3.  This is the latest production-ready  
version in the Python 2.6 series.  Somewhere near 100 bugs have been  
fixed since Python 2.6.2 was released in April 2009.  Please see the  
NEWS file for all the gory details.


http://www.python.org/download/releases/2.6.3/NEWS.txt

Source tarballs, Windows installers and the Mac OS X disk image can be  
downloaded from the Python 2.6.3 page.


http://www.python.org/download/releases/2.6.3/

For more information on Python 2.6 in general, please see

http://docs.python.org/whatsnew/2.6.html

Please report bugs for any Python version in the Python tracker.

http://bugs.python.org

Enjoy,
-Barry

Barry Warsaw
ba...@python.org
Python 2.6 Release Manager
(on behalf of the entire python-dev team)



PGP.sig
Description: This is a digitally signed message part
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: store encrypted data in sqlite ?

[Jonathan Gardner]

On Oct 2, 11:53 am, Stef Mientki  wrote:
>
> Will this method work always ?
> Are there better methods ?
>

I SQLite doesn't like raw data (with all its \0 glory), you're out of
luck, unfortunately. Base64 encoding is a really good solution for
places like this.

You are aware, of course, of the dangers of storing sensitive data in
memory? That is, if you are storing the sensitive data anywhere in a
Python variable, it is possible for someone with access to the memory
of the machine to discover it.

If it is only the storage of the sensitive data you are concerned
about, or the sensitivity of the data if it is transferred over the
network, there are other, easier ways to protect the data than in an
encrypted field.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: store encrypted data in sqlite ?

[Carsten Haese]

Stef Mientki wrote:
> hello,
> 
> I want to store some fields in an sqlite database.
> 
> I use ezPyCrypto to encrypt and decrypt:
> 
> User = ['z684684', 'Mientki, Stef', 1,1,0,1,1 ]
> 
> encryption_key_1 = ezPyCrypto.key ( 512 )
> 
> SQL_Base = 'insert or replace into __USERS__ values ('
> for field in User :
>SQL += ",'" + encryption_key_1.encString ( str ( item ))+ "'"
> SQL += ')'
> 
> 
> Now this fails, probably, because the second character of the encrypted
> string is a binary zero.
> 
> By trial and error, I found a work around,
> but I'm not sure this will garantee that it will work always:
> by converting the encrypted buffer with base64.encode:
> 
>SQL += ",'" + base64.encodestring(EnCrypt_1 ( str ( item )))+ "'"
> 
> Will this method work always ?
> Are there better methods ?

There is definitely a better method! You should use parameter binding
instead of rolling the query by hand:

SQL = "insert or replace into __USERS__ values (?,?,?,?,?,?,?)"
params = [ encryption_key_1.encString(str(x)) for x in User ]
cur.execute(SQL, params)

That way, the parameters are passed separately and safely, and the query
syntax is protected from all the dangerous characters that are floating
around in the parameters.

HTH,

--
Carsten Haese
http://informixdb.sourceforge.net

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: epydoc - can I provide multiple dirs to parse

[Jonathan Gardner]

On Oct 2, 11:38 am, Medi  wrote:
> Can I present multiple directories to epydoc to process. For example
>
> epydoc -option -option -option dir_1 dir_2 dir_3
>

I know nothing of epydoc.

However, it looks like it should be something like:

  epydoc -option dir_1 -option dir_2 -option dir_3

...if anything would work at all.

You may want to contact the epydoc mailing list for better guidance,
though.
-- 
http://mail.python.org/mailman/listinfo/python-list

numpy f2py question

[George Trojan]

I have a problem with numpy's vectorize class and f2py wrapped old 
FORTRAN code. I found that the function _get_nargs() in 
site-packages/numpy/lib/function_base.py tries to find the number of 
arguments for a function from an error message generated when the 
function is invoked with no arguments. However, with numpy 1.3.0 the 
error message is different that the code expects. Is there any known 
solution? I don';t know where the message is coming from, a cursory 
check of numpy files did not yield any hits. A ctypes issue?
I did find an  unanswered seven weeks old related posting 
http://article.gmane.org/gmane.comp.python.numeric.general/32168/match=f2py+number+arguments 
though I don't think this is gfortran related. Mine is 4.1.2.


George
--
http://mail.python.org/mailman/listinfo/python-list

Re: int rich comparisons

[Robert Kern]

George Sakkis wrote:

I stumbled upon the following strangeness (python 2.6.2):


getattr(int, '__gt__')




getattr(5, '__gt__')

Traceback (most recent call last):n
  File "", line 1, in 
AttributeError: 'int' object has no attribute '__gt__'

Is this a bug ?


I believe that ints in the 2.x series still use __cmp__. In Python 3.x, __gt__ 
and the rest are defined.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

--
http://mail.python.org/mailman/listinfo/python-list

Re: Q: sort's key and cmp parameters

[Paul Rubin]

Scott David Daniels  writes:
> Most cases are moreeasily done with key, and it is
> a good idea to make the most accessible way to a sort be the most
> efficient one.  In the rare case that you really want each comparison,
> the cmp-injection function will do nicely (and can be written as a
> recipe.

I don't think wrapping the sorted objects in an otherwise useless
special purpose class is "nicely", either from a performance or from a
code verbosity point of view.  I avoid Java and its useless extra
classes for a reason ;-).

> In short, make the easy path the fast path, and more will use it;
> provide two ways, and the first that springs to mind is the one
> used.

I think we are saying the same thing.  Python 2.x provides two ways
and you can use whichever one fits the application better.  I have
never understood why Python 3.x finds it necessary to break one of
them.  Maybe I can migrate to Haskell by the time Python 2.x becomes
deprecated.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Simple Python struct issue

[Carlo DiCelico]

On Oct 2, 3:17 pm, Simon Forman  wrote:
> On Fri, Oct 2, 2009 at 12:35 PM, Carlo DiCelico
>
>
>
>
>
>  wrote:
> > On Oct 2, 12:15 pm, Simon Forman  wrote:
> >> On Fri, Oct 2, 2009 at 12:07 PM, Carlo DiCelico
> >>  wrote:
> >> > I saw an article on O'Reilly about using NumPy and Dislin to analyze
> >> > and visualize WAV files. It's a really fantastic article but was a
> >> > little out of date. I updated the script to work with the newer
> >> > modules &etc but am still having trouble getting it working.
>
> >> > The line
>
> >> > temp[i,:] = array(struct.unpack("%dB"%(fft_length), tempb),Float) -
> >> > 128.0
>
> >> > always returns the same error: "Traceback (most recent call last):
> >> >  File "pysono.py", line 31, in 
> >> >   temp[i,:] = array(struct.unpack("%dB"%(fft_length),tempb),float) -
> >> > 128.0
> >> > struct.error: unpack requires a string argument of length 256" when I
> >> > do python pysono.py test.wav 256
>
> >> > I'm sure it's probably something simple but I just can't see what it
> >> > is!
>
> >> > Here's the original 
> >> > code:http://onlamp.com/python/2001/01/31/graphics/pysono.py
>
> >> > Thanks!
>
> >> In:
>
> >> struct.unpack("%dB" % (fft_length), tempb)
>
> >> tempb is not length 256.
>
> >> Also, note that (foo) is the same as just foo.  To create a tuple of
> >> length 1 you must say (foo,)
>
> >> HTH,
> >> ~Simon
>
> > I'm sorry, I'm not sure what you're referring to when you say "Also,
> > note that (foo) is the same as just foo.  To create a tuple of length
> > 1 you must say (foo,)".
>
> Sorry about that.  All I mean is that in python putting parentheses
> around an expression "(some_object)" does not create a tuple unless
> you also use a comma ","
>
> I hope the following examples show what I mean:
>
> In [1]: (2)
> Out[1]: 2
>
> In [2]: (2,)
> Out[2]: (2,)
>
> In [3]: 2,
> Out[3]: (2,)
>
> In [4]: n = 2,
>
> In [5]: "%s" % (n,)
> Out[5]: '(2,)'
>
> In [6]: "%s" % (n)
> Out[6]: '2'
>
> In [7]: "%s" % n
> Out[7]: '2'
>
> Your code:
>
> "%dB" % (fft_length)
>
> is the same as
>
> "%dB" % fft_length
>
> But what you probably meant was
>
> "%dB" % (fft_length,)
>
> Note the comma.  People do this to prevent string formatting errors if
> fft_length should ever happen to be a tuple rather than an int (or
> whatever.)
>
> > The 256 is passed into the script as an
> > argument and then assigned to the variable fft_length, which you can
> > see in that line. So, whatever value I pass in comes out in the error.
> > What you're saying is that the error happens because tempb isn't the
> > length of any of those values (i.e., 128, 256, 512, etc)?
>
> Yes, whatever tempb is, it isn't "a string argument of length 256" as
> the traceback says.
>
> > (1) How
> > could I determine the length of tempb?
>
> If it's a string, len(tempb) will return it's length.
>
> > and (2) It's doing this ->
> > tempb = fp.readframes(fft_length); right before doing the struct.unpack
> > (); could the splitting of the frames into frames of length fft_length
> > be causing this error?
>
> See MRAB's reply. :]
>
> > Thanks for the help!
>
> You're very welcome.
>
> One other thing, in the code you posted it has "Float", but in the
> traceback it has "float".  Generally speaking when you post code to a
> newsgroup for help you should paste in the exact code, rather than
> retyping it manually.
>
> Warm regards, and happy hacking!

Ah, yes, makes so much more sense now. Thanks very much for the help!
The discrepancy between Float and float has to do with the fact that
in the code I linked to, he's using an old version of numpy. In my
code, I'm using the new version of numpy, which uses float instead of
Float. Sorry about that, thanks again.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Simple Python struct issue

[Carlo DiCelico]

On Oct 2, 12:49 pm, MRAB  wrote:
> Carlo DiCelico wrote:
> > I saw an article on O'Reilly about using NumPy and Dislin to analyze
> > and visualize WAV files. It's a really fantastic article but was a
> > little out of date. I updated the script to work with the newer
> > modules &etc but am still having trouble getting it working.
>
> > The line
>
> > temp[i,:] = array(struct.unpack("%dB"%(fft_length), tempb),Float) -
> > 128.0
>
> > always returns the same error: "Traceback (most recent call last):
> >   File "pysono.py", line 31, in 
> >    temp[i,:] = array(struct.unpack("%dB"%(fft_length),tempb),float) -
> > 128.0
> > struct.error: unpack requires a string argument of length 256" when I
> > do python pysono.py test.wav 256
>
> > I'm sure it's probably something simple but I just can't see what it
> > is!
>
> > Here's the original 
> > code:http://onlamp.com/python/2001/01/31/graphics/pysono.py
>
> .readframes(n) returns (at most) n _frames_ as a bytestring (str), not n
> bytes. I tried reading 256 frames from a .wav file containing stereo at
> 16 bits per channel and got 1024 bytes (4 bytes per frame, not
> surprisingly!).

Yes! Thank you very much for pointing this out, don't know how I
missed it!
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Threaded GUI slowing method execution?

[Aaron Hoover]

On Oct 2, 2009, at 12:30 PM, Dave Angel wrote:

(you responded off-list, which isn't the way these mailing lists  
work.  So I'm pasting your message back to the list, with my  
response at the end)




Sorry about that - a slip of the "reply" button.



Actually, I was thinking of the subprocess module (introduced in  
2.4).  But the multiprocessing module would be useful if you were  
porting threading code to a process model.


Previously, I had been using threads, so I just tried moving to  
multiprocessing because it required few changes.


There are tons of ways to communicate between processes, though you  
can't do the simple variable sharing that threads can (sometimes)  
get away with.  I would normally point you to queues, but there a  
number of possibilities.  And since the one process is running a GUI  
event loop, you might want to piggyback on the OS capability to post  
events between processes.  The code might end up OS-dependent, but  
I'd bet the overhead will be minimal.  What is your target operating  
system?


My target OS is Mac OS X Leopard. I've decided just to try and trigger  
WX events in the GUI based on data put in one of the queues by the  
processing thread.




Your numbers in the original message make me nervous;  sending an  
event between processes (or even threads) every 0.5 millisecond is  
impossible.  But I think you might have been confusing bytes and  
packets.


My external hardware is actually sending 2000 packets per second right  
now (but that can also be changed). Each packet currently contains 6  
bytes of data and 6 bytes of overhead. So, 12 bytes per packet * 2000  
packets per second is 24k bytes per second. However, the serial  
processing process should be finding the packet payload data and  
adding to a queue for the GUI to deal with and plot (obviously at a  
much lower framerate) (at an ideal rate of 2KHz).




There could very well be multiprocess support in wxPython.  I'd  
check there first, before re-inventing the wheel.  Presumably you  
know of the wxPython news group, hosted on Google groups?


I actually tried using multiprocess to create a separate process for  
reading the serial data. The process gets passed a TX queue and an RX  
queue in its constructor, then it's supposed to enter an infinite loop  
where it waits for commands on the TX queue and reads data and puts in  
in the RX queue. However, I've discovered for some reason that the  
infinite loop terminates after the first call I make to either of the  
queues. Perhaps this behavior should be expected, but this is my first  
go-round with processes, so I was a little surprised by this result.  
Maybe someone with more multiprocessing experience can give me some  
pointers.


Maybe I am barking up the wrong tree with Python from a speed  
perspective, but it's just some much faster and more fun to write than  
anything else...


Aaron

--
http://mail.python.org/mailman/listinfo/python-list

Multiple Select

[Victor Subervi]

Hi;
I'm trying to create an HTML  with several options in a form that is
posted to a script. I want to enable the user to select multiple options.
The problem is I don't know how to call those options in my cgi calls in the
script to which it posts. Only one of them gets associated with the variable
in the . What do?
TIA
Victor
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Multiple Select

[Rami Chowdhury]

On Fri, 02 Oct 2009 14:25:55 -0700, Victor Subervi  
 wrote:



Hi;
I'm trying to create an HTML  with several options in a form  
that is

posted to a script. I want to enable the user to select multiple options.
The problem is I don't know how to call those options in my cgi calls in  
the
script to which it posts. Only one of them gets associated with the  
variable

in the . What do?
TIA
Victor


How are you retrieving the values in the CGI script?

--
Rami Chowdhury
"Never attribute to malice that which can be attributed to stupidity" --  
Hanlon's Razor

408-597-7068 (US) / 07875-841-046 (UK) / 0189-245544 (BD)
--
http://mail.python.org/mailman/listinfo/python-list

Re: Multiple Select

[Victor Subervi]

I have the select called variable "these". Whichever option is selected is
passed through the cgi:

import cgitb; cgitb.enable()
import cgi

form = cgi.FieldStorage()
these = form.getfirst('these', '')

As I mentioned before, I get the variable all right. That is not the issue.
What is the issue is that I would like to enable the user to select
*several* options; in essence, to turn "these" into a tuple. That way, I
could capture *several* options and enter them all into my database at once.
TIA,
V


On Fri, Oct 2, 2009 at 4:46 PM, Rami Chowdhury wrote:

> On Fri, 02 Oct 2009 14:25:55 -0700, Victor Subervi <
> victorsube...@gmail.com> wrote:
>
>  Hi;
>> I'm trying to create an HTML  with several options in a form that
>> is
>> posted to a script. I want to enable the user to select multiple options.
>> The problem is I don't know how to call those options in my cgi calls in
>> the
>> script to which it posts. Only one of them gets associated with the
>> variable
>> in the . What do?
>> TIA
>> Victor
>>
>
> How are you retrieving the values in the CGI script?
>
> --
> Rami Chowdhury
> "Never attribute to malice that which can be attributed to stupidity" --
> Hanlon's Razor
> 408-597-7068 (US) / 07875-841-046 (UK) / 0189-245544 (BD)
>
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Multiple Select

[Ethan Furman]

Victor Subervi wrote:
I have the select called variable "these". Whichever option is selected 
is passed through the cgi:


import cgitb; cgitb.enable()
import cgi

form = cgi.FieldStorage()
these = form.getfirst('these', '')

As I mentioned before, I get the variable all right. That is not the 
issue. What is the issue is that I would like to enable the user to 
select *several* options; in essence, to turn "these" into a tuple. That 
way, I could capture *several* options and enter them all into my 
database at once.

TIA,
V



Please don't top post.

From the docs:

getfirst( name[, default])

This method always *returns* *only* *one* value associated with form 
field name. The method returns only the first value in case that more 
values were posted under such name. Please note that the order in which 
the values are received may vary from browser to browser and should not 
be counted on.18.1 If no such form field or value exists then the method 
returns the value specified by the optional parameter default. This 
parameter defaults to None if not specified.


getlist( name)

This method always *returns* *a* *list* of values associated with form 
field name. The method returns an empty list if no such form field or 
value exists for name. It returns a list consisting of one item if only 
one such value exists.



Hope this helps.

~Ethan~
--
http://mail.python.org/mailman/listinfo/python-list

Re: Threaded GUI slowing method execution?

[sturlamolden]

On 2 Okt, 22:29, Aaron Hoover  wrote:

> My external hardware is actually sending 2000 packets per second right  
> now (but that can also be changed). Each packet currently contains 6  
> bytes of data and 6 bytes of overhead. So, 12 bytes per packet * 2000  
> packets per second is 24k bytes per second.


> Maybe I am barking up the wrong tree with Python from a speed  
> perspective, but it's just some much faster and more fun to write than  
> anything else...

I have processed data in "real time" with Python from a system that
delivered data from 64 channels at 48 kHz, 2 bytes per sample, video
tracker, I/O ports, etc. Each package was 432 bytes and arrived at 16
kHz, that is 6.59 MiB per second. Python is much faster than you
think. You just have to do some buffering. :-)

If your hardware does not provide sufficient buffering, I'd write a
small C function that takes care of this. This would be a function
that collects data as it comes in, writes to a queue, and waits for
more. Run this C function from a background Python thread using
ctypes.CDLL.














-- 
http://mail.python.org/mailman/listinfo/python-list

Python resident memory retention & Evan Jones' improvements

[Matt Ernst]

My apologies in advance if this has been addressed before. Google does
not presently seem to return search results for this group from more
than a couple of months ago.

I have some long-running Python processes that slowly increase in
resident memory size, and whose resident size goes down only when they
are restarted. I spent hours with gc and heapy but was unable to
identify obvious culprits. I eventually tracked the problem down to
buffering data to a queue for later processing. Putting to the queue
increases resident size, but getting from it never decreases resident
size. In fact I see the same behavior when I use plain lists instead
of queue objects.

I thought Evan Jones altered Python to deal with this very problem,
and the change went into the release of 2.5.

Here is Tim Peters announcing the change:
http://mail.python.org/pipermail/python-dev/2006-March/061991.html

He included this simple test program to show the improvement:
"""
x = []
for i in xrange(100):
   x.append([])
raw_input("full ")
del x[:]
raw_input("empty ")
"""

If you look at resident size in the "full" stage, the interpreter has
grown to tens of megabytes. If you look at it in the "empty" stage, it
goes back down to less than 10 megabytes. But if you run this trivial
variation on the same program, memory use goes up and stays up:

"""
x = []
for i in xrange(100):
   x.append([])
raw_input("full ")
del x[:]
for i in xrange(100):
   x.append([])
del x[:]
raw_input("empty ")
"""

At the "empty" prompt resident memory size has not decreased. I see
this pattern of behavior in CPython 3.1.1, 2.6.3, 2.5.2, and Jython
2.5.1. I have tested under 32 and 64 bit Intel Linux.

At this point I suspect that I am not going to be able to force my
long-running processes to shrink their resident size, since I can't
force it in much simpler tests. I am curious about why it happens
though. That the second program should retain a larger resident memory
footprint than the first is (to me) quite surprising.
-- 
http://mail.python.org/mailman/listinfo/python-list

multiprocessing newbie - semaphore question

[xera121]

Hi,
I'm interested in this type of programming but need some fundamental
questions answered if someone could oblige:

In the docs for the mutiprocessing module it mentions right at the
beginning about an underlying semaphore system - would MPI2 fit the
bill running on a Beowulf cluster?

Is the mutiprocessing module usable for writing apps to run on a
Beowulf cluster - i.e. can I use it for an app that will share the
work amongst the nodes?

I am looking for a way to share the processing on a task which
requires intensive and repetitive work done over a large dataset in a
sqlite table and I wondered if this would be a way to share the load
over a number of pcs I have available for this task.

My other idea is a simple(ish) client/server model which I am pretty
confident I can execute whereas the setup of the cluster and the
writing of a multiprocessing app would require a lot of prep and study/
testing.

Therefore the bottom line is is it worth it? I know I haven't been too
specific of what I'm doing but I just wanted a quick feel for the task
from those who have more knowledge of the subject - Oh and btw I am
NOT an expert programmer by any means and would expect almost all my
code to be deemed as amateurish - but it keeps me happy and only
breaks every other iteration ;-)

xera121
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: numpy f2py question

[sturlamolden]

On 2 Okt, 22:41, George Trojan  wrote:
> I have a problem with numpy's vectorize class and f2py wrapped old
> FORTRAN code. I found that the function _get_nargs() in
> site-packages/numpy/lib/function_base.py tries to find the number of
> arguments for a function from an error message generated when the
> function is invoked with no arguments. However, with numpy 1.3.0 the
> error message is different that the code expects. Is there any known
> solution? I don';t know where the message is coming from, a cursory
> check of numpy files did not yield any hits. A ctypes issue?
> I did find an  unanswered seven weeks old related 
> postinghttp://article.gmane.org/gmane.comp.python.numeric.general/32168/matc...
> though I don't think this is gfortran related. Mine is 4.1.2.

The wrappers generated by f2py just calls PyArg_ParseTupleAndKeywords
and return NULL if it fails. The error message from
PyArg_ParseTupleAndKeywords does not fit the regex used by _get_nargs:

re.compile(r'.*? takes exactly (?P\d+) argument(s|) \((?
P\d+) given\)')

It should have said

re.compile(r'.*? takes (exactly|at least) (?P\d+) argument(s|)
\((?P\d+) given\)')

It is not just an f2py issue. It affects any function that uses
keyword arguments. I'll file this as a bug on NumPy trac.

S.M.






-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Fast decimal arithmetic module released

[Raymond Hettinger]

[Stefan Krah]
> today I have released the following packages for fast arbitrary precision
> decimal arithmetic:

Nice.


Raymond
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: numpy f2py question

[sturlamolden]

http://projects.scipy.org/numpy/ticket/1247
-- 
http://mail.python.org/mailman/listinfo/python-list

Haskell's new logo, and the idiocy of tech geekers

[Xah Lee]

Haskell has a new logo. A fantastic one. Beautiful. For creator,
context, detail, see bottom of:

• A Lambda Logo Tour
  http://xahlee.org/UnixResource_dir/lambda_logo.html

this is posted here because it relates to various computer software/
language's logo, a subject discussed by me several times in the past
years in comp.lang.lisp, comp.lang.scheme, comp.lang.python, as well
discussed now and then in these communities often without my
involvement.

for my criticism or comment on logos, typical response by these people
are showcases of complete ignorance of social function of logos, and a
complete iliteracy in visual design (that any art student learns in
first year), usually coupled with a teen-age militant attitude.
(usually, these tech geeker's mentality is like: “you are a fucking
idiot... we don't need a logo. Logos are for kids and blood-sucking
marketers.”)

Some of these tech geekers, are motherfucking aggresive, so that they
would ban me, or call me as a troll, or call upon whatever rules to
oust me (such as “off topicality”) , for any criticism on them. (if
requested, i'll take some time to cite the exact url for numerous of
these threads that happened in the past 5 years. My frustration in
discussion logos happened in python, lojban, lisp communities. Note:
not all people in these communities are idiotic about logos.
Typically, it's just few of priest fuckheads (e.g. a newsgroup
regular, or somewhat significant contributor of the particular
community). This particularly happened in several threads in the
lojban community where i criticized its logo during mid 2000s. (have a
look at lojban's motherfucking idiotic logo here: 
http://en.wikipedia.org/wiki/Lojban
))

• The Purpose of Logos and Principles of Logo Design
  http://xahlee.org/UnixResource_dir/logo_design.html

• Lisp Needs A Logo
  http://xahlee.org/UnixResource_dir/writ/logo_lisp.html


computer language and software logo gallery:

• The Unix Pestilence
  http://xahlee.org/UnixResource_dir/freebooks.html

• A Lambda Logo Tour
  http://xahlee.org/UnixResource_dir/lambda_logo.html

• LISP Logo
  http://xahlee.org/emacs/lisp_logo.html

• Qi Language Logo
  http://xahlee.org/emacs/qi_logo.html

• Haskell Logo
  http://xahlee.org/haskell/haskell-logo.html

• The Emacs Logo
  http://xahlee.org/emacs/emacs_logo.html

• Xah's Java Logo
  http://xahlee.org/java-a-day/java_logo.html

  Xah
∑ http://xahlee.org/

☄
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Haskell's new logo, and the idiocy of tech geekers

[Kenneth Tilton]

Xah Lee wrote:

Haskell has a new logo. A fantastic one. Beautiful. For creator,
context, detail, see bottom of:

• A Lambda Logo Tour
  http://xahlee.org/UnixResource_dir/lambda_logo.html


Cool survey, and yes, that is a nice new one for Haskell.

I saw beauty the other day changing an application to talk to RDF for 
what it used to get from CLOS and seeing it Just Work after a couple of 
hours of poking around.


Connecting those two paragraphs left as an exercise.

kt
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python resident memory retention & Evan Jones' improvements

[Andrew MacIntyre]

Matt Ernst wrote:

{...}


I thought Evan Jones altered Python to deal with this very problem,
and the change went into the release of 2.5.

Here is Tim Peters announcing the change:
http://mail.python.org/pipermail/python-dev/2006-March/061991.html

He included this simple test program to show the improvement:
"""
x = []
for i in xrange(100):
   x.append([])
raw_input("full ")
del x[:]
raw_input("empty ")
"""

If you look at resident size in the "full" stage, the interpreter has
grown to tens of megabytes. If you look at it in the "empty" stage, it
goes back down to less than 10 megabytes. But if you run this trivial
variation on the same program, memory use goes up and stays up:

"""
x = []
for i in xrange(100):
   x.append([])
raw_input("full ")
del x[:]
for i in xrange(100):
   x.append([])
del x[:]
raw_input("empty ")
"""

At the "empty" prompt resident memory size has not decreased. I see
this pattern of behavior in CPython 3.1.1, 2.6.3, 2.5.2, and Jython
2.5.1. I have tested under 32 and 64 bit Intel Linux.

At this point I suspect that I am not going to be able to force my
long-running processes to shrink their resident size, since I can't
force it in much simpler tests. I am curious about why it happens
though. That the second program should retain a larger resident memory
footprint than the first is (to me) quite surprising.


There are two things you need to be aware of in this situation:

- not all Python's memory is allocated through Python's specialised
  malloc() - int and float objects in particular (in 2.x at least) are
  allocated directly via a privately managed free list, and any
  allocation requiring more than 256 bytes is directed to the platform
  malloc().  Any memory not allocated via Python's malloc() is not
  subject to the memory release facility referred to above.

  Python 2.6 does improve the management of memory consumed by int and
  float objects via the garbage collector.

- while Python attempts to maximally utilise memory arenas to improve
  the chances of being able to free them, Python's malloc() does not do
  any compaction of memory (ie moving allocations between arenas) within
  existing arenas.  Nor does garbage collection do this.  So fragmented
  allocations can cause the retention of nearly empty arenas.

I suspect that what you see with the second test script above is caused
by some sort of fragmentation.

My recollection of Evan's objective with his work was to deal with the
case where long running processes created lots of objects on startup,
but having initialised no longer need most of the created objects.
Without his modification, the memory would have been retained unused for
the remaining life of the process.  It also helps with cyclic bursts of
object creation/deletion.

But there are circumstances where it doesn't kick in.

To get a deeper understanding of your issue will require deeper
debugging.  I have done this at times by building Python with a wrapper
around malloc() (& friends) to log memory allocation activity.

--
-
Andrew I MacIntyre "These thoughts are mine alone..."
E-mail: andy...@bullseye.apana.org.au  (pref) | Snail: PO Box 370
   andy...@pcug.org.au (alt) |Belconnen ACT 2616
Web:http://www.andymac.org/   |Australia
--
http://mail.python.org/mailman/listinfo/python-list

Re: Haskell's new logo, and the idiocy of tech geekers

[Pascal J. Bourguignon]

Kenneth Tilton  writes:

> Xah Lee wrote:
>> Haskell has a new logo. A fantastic one. Beautiful. For creator,
>> context, detail, see bottom of:
>> • A Lambda Logo Tour
>>   http://xahlee.org/UnixResource_dir/lambda_logo.html

Don't do that!  
If you want to watch the logo, just google for haskell logo.
There's no need to go thru xahlee.org.


-- 
__Pascal Bourguignon__
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Concurrent threads to pull web pages?

[Kyle Terry]

On Thu, Oct 1, 2009 at 6:33 PM,  wrote:

> On 1 Oct, 09:28 am, nos...@nospam.com wrote:
>
>> Hello
>>
>>I recently asked how to pull companies' ID from an SQLite database,
>> have multiple instances of a Python script download each company's web
>> page from a remote server, eg. www.acme.com/company.php?id=1, and use
>> regexes to extract some information from each page.
>>
>> I need to run multiple instances to save time, since each page takes
>> about 10 seconds to be returned to the script/browser.
>>
>> Since I've never written a multi-threaded Python script before, to
>> save time investigating, I was wondering if someone already had a
>> script that downloads web pages and save some information into a
>> database.
>>
>
> There's no need to use threads for this.  Have a look at Twisted:
>
>  http://twistedmatrix.com/trac/
>
> Here's an example of how to use the Twisted HTTP client:
>
>  http://twistedmatrix.com/projects/web/documentation/examples/getpage.py
>

I don't think he was looking for a framework... Specifically a framework
that you work on.


>
> Jean-Paul
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Opinions, please, on PEP 8 and local, 3rd party imports

[Simon Forman]

On Fri, Oct 2, 2009 at 3:50 PM, Philip Semanchuk  wrote:
> Hi all,
> Our project uses some libraries that were written by 3rd parties (i.e. not
> us). These libraries fit into a single Python file and live in our source
> tree alongside other modules we've written. When our app is distributed,
> they'll be included in the installation. In other words, they're not
> prerequisites that we'll make the user install, and they won't live in
> site-packages.
>
> For example, we use configobj from
> , and I think Beautiful
> Soup can be installed as a single file too, although we're not using that
> particular module.
>
> PEP 8  says the following:
>
>   Imports should be grouped in the following order:
>   1. standard library imports
>   2. related third party imports
>   3. local application/library specific imports
>
>
> I'm not sure in which category to place local, 3rd-party modules like
> configobj.
>
> Clearly, they belong in category 3 since they're local.
>
> Clearly, they also belong in category 2 since they're 3rd party modules, and
> explicitly labeling an imported module as "this is code we didn't write" is
> useful. But I've always reserved category 2  for external libraries (e.g.
> numpy, wxPython, Twisted, PIL, etc.).
>
> Clearly, the best choice is category 2.5?
>
> In your Pythonic opinion, should 3rd-party modules that live alongside
> homegrown code be listed in import category 2 or 3?
>
> Thanks
> Philip
> --
> http://mail.python.org/mailman/listinfo/python-list
>

2.5 +1

;]
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Q: sort's key and cmp parameters

[Raymond Hettinger]

[Paul Rubin]
> Yes, think of sorting tree structures where you have to recursively
> compare them til you find an unequal pair of nodes.  

I'm not sure what you mean by this.  What are the semantics of
sorting a tree?  Can you outline an example of tree that
could be sorted easily with a cmp function but not a key function?

t = [[9,6],[7,1]],[[2,5],[4,3]]   # inputs
t.sort(mycmp) # what would the cmp function be?
print t   # what would the output be


Raymond
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Haskell's new logo, and the idiocy of tech geekers

[Kenneth Tilton]

Pascal J. Bourguignon wrote:

Kenneth Tilton  writes:


Xah Lee wrote:

Haskell has a new logo. A fantastic one. Beautiful. For creator,
context, detail, see bottom of:
• A Lambda Logo Tour
  http://xahlee.org/UnixResource_dir/lambda_logo.html


Don't do that!  
If you want to watch the logo, just google for haskell logo.

There's no need to go thru xahlee.org.


Hunh? He has done a nice job of collecting different logos and putting 
them all in one place where one can see them all just by scrolling. ie, 
it's a cool web page with added value available nowhere else.


kt

--

http://thelaughingstockatpngs.com/
http://www.facebook.com/pages/The-Laughingstock/115923141782?ref=nf
--
http://mail.python.org/mailman/listinfo/python-list

organizing your scripts, with plenty of re-use

[bukzor]

I would assume that putting scripts into a folder with the aim of re-
using pieces of them would be called a package, but since this is an
"anti-pattern" according to Guido, apparently I'm wrong-headed here.
(Reference: http://mail.python.org/pipermail/python-3000/2007-April/006793.html
)

Say you have ~50 scripts or so with lots of re-use (importing from
each other a lot) and you want to organize them into folders. How do
you do this simply?

The intent is to have people be able to check out the directly from
CVS and have the scripts "just work", even if they're not directly on
the PYTHONPATH.

This seems to be the best discussion on the topic, but the conclusion
seemed to be that there's no good way. That seems unthinkable
considering python's dedication to simplicity and elegance.
http://groups.google.com/group/comp.lang.python/browse_thread/thread/c44c769a72ca69fa/


It looks like I'm basically restating this post, which sadly got
dropped without further comment:
http://mail.python.org/pipermail/python-3000/2007-April/006814.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: int rich comparisons

[Terry Reedy]

Robert Kern wrote:

George Sakkis wrote:

I stumbled upon the following strangeness (python 2.6.2):


getattr(int, '__gt__')




getattr(5, '__gt__')

Traceback (most recent call last):n
  File "", line 1, in 
AttributeError: 'int' object has no attribute '__gt__'

Is this a bug ?


I believe that ints in the 2.x series still use __cmp__. In Python 3.x, 
__gt__ and the rest are defined.


Correct:
Python 3.1 (r31:73574, Jun 26 2009, 20:21:35) [MSC v.1500 32 bit 
(Intel)] on win32

Type "copyright", "credits" or "license()" for more information.
>>> 5 .__gt__ # notice space after 5


--
http://mail.python.org/mailman/listinfo/python-list

Re: Q: sort's key and cmp parameters

[Paul Rubin]

Raymond Hettinger  writes:
> I'm not sure what you mean by this.  What are the semantics of
> sorting a tree?  Can you outline an example of tree that
> could be sorted easily with a cmp function but not a key function?

The idea was that you have a list of trees that you want to sort, and
an ordering relation between trees:

   def gt(tree1, tree2): ...

where you recursively descend both trees until you find an unequal
pair of nodes.  You're not trying to sort the nodes within a single
tree.
-- 
http://mail.python.org/mailman/listinfo/python-list