parallel (concurrent) eventlet

[David Gabriel]

Dears,

I have an issue when I use eventlet Api to create parallel threads.
In fact, when I run the below code, only the program dealing with the
synchronozation with ldap data base is working and is continuously blocking
the others to run.
But, when I use the 'thread' Api the program is working fine without any
blocking issue. However, I can not use thread Api and I must to use
eventlet.
So I am wondering how to get the thread Api behavior using the eventlet Api
?

Could you please inform me how to fix this issue ?

Kindly find below my code.
But you need some configurations regarding ldap server/client.

#!/usr/bin/python
# -*- coding: utf-8 -*-
"""
This script implements a syncrepl consumer which syncs data from an OpenLDAP
server to a local (shelve) database.
Notes:
The bound user needs read access to the attributes entryDN and entryCSN.
This needs the following software:
Python
pyasn1 0.1.4+
pyasn1-modules
python-ldap 2.4.10+
"""

# Import the python-ldap modules
import ldap,ldapurl
# Import specific classes from python-ldap
from ldap.ldapobject import ReconnectLDAPObject
from ldap.syncrepl import SyncreplConsumer

# Import modules from Python standard lib
import shelve,signal,time,sys,logging
import eventlet
#import thread
eventlet.monkey_patch()

# Global state
watcher_running = True
ldap_connection = False



class SyncReplConsumer(ReconnectLDAPObject,SyncreplConsumer):
"""
Syncrepl Consumer interface
"""
def __init__(self,db_path,*args,**kwargs):
# Initialise the LDAP Connection first
ldap.ldapobject.ReconnectLDAPObject.__init__(self, *args, **kwargs)
# Now prepare the data store
self.__data = shelve.open(db_path, 'c')
# We need this for later internal use
self.__presentUUIDs = dict()

def __del__(self):
# Close the data store properly to avoid corruption
self.__data.close()

def syncrepl_get_cookie(self):
if 'cookie' in self.__data:
return self.__data['cookie']

def syncrepl_set_cookie(self,cookie):
self.__data['cookie'] = cookie

def syncrepl_entry(self,dn,attributes,uuid):

# First we determine the type of change we have here (and store
away the previous data for later if needed)
previous_attributes = dict()
if uuid in self.__data:
change_type = 'modify'
previous_attributes = self.__data[uuid]
else:
change_type = 'add'
# Now we store our knowledge of the existence of this entry
(including the DN as an attribute for convenience)
attributes['dn'] = dn
self.__data[uuid] = attributes
# Debugging
print 'Detected', change_type, 'of entry:', dn
# If we have a cookie then this is not our first time being run, so
it must be a change
if 'ldap_cookie' in self.__data:
self.perform_application_sync(dn, attributes,
previous_attributes)

def syncrepl_delete(self,uuids):
# Make sure we know about the UUID being deleted, just in case...
uuids = [uuid for uuid in uuids if uuid in self.__data]
# Delete all the UUID values we know of
for uuid in uuids:
print 'Detected deletion of entry:', self.__data[uuid]['dn']
del self.__data[uuid]

def syncrepl_present(self,uuids,refreshDeletes=False):
# If we have not been given any UUID values, then we have recieved
all the present controls...
if uuids is None:
# We only do things if refreshDeletes is false as the syncrepl
extension will call syncrepl_delete instead when it detects a delete notice
if refreshDeletes is False:
deletedEntries = [uuid for uuid in self.__data.keys() if
uuid not in self.__presentUUIDs and uuid != 'ldap_cookie']
self.syncrepl_delete( deletedEntries )
# Phase is now completed, reset the list
self.__presentUUIDs = {}
else:
# Note down all the UUIDs we have been sent
for uuid in uuids:
self.__presentUUIDs[uuid] = True

def perform_application_sync(self,dn,attributes,previous_attributes):
print 'Performing application sync for:', dn
return True


# Shutdown handler
#def commenceShutdown(signum, stack):
def commenceShutdown():
# Declare the needed global variables
global watcher_running, ldap_connection
print 'Shutting down!'

# We are no longer running
watcher_running = False

# Tear down the server connection
if( ldap_connection ):
del ldap_connection

# Shutdown
sys.exit(0)

def mainOfSyncrepl(threadName):
 # Time to actually begin execution
 # Install our signal handlers
# signal.signal(signal.SIGTERM,commenceShutdown)
# signal.signal(signal.SIGINT,commenceShutdown)
 try:
   ldap_url =
ldapurl.LDAPUrl('ldap://localhost/dc=example,dc=org?*?sub?(objectClass=*)?bindname=cn=admin%2cdc=test

Re: Powerful perl paradigm I don't find in python

[Charles T. Smith]

On Fri, 15 Jan 2016 14:20:17 +0100, Wolfgang Maier wrote:

> pattern = pattern_str.compile()
> try:
>  matches = pattern.findall(some_str, endpos=some_str.index(tail))
> except ValueError:
>  # do something if tail is not found
>  pass

Oh!  I think that's it!


matches = findall (pattern, string)
for file in matches:
use (file)

Totally cool!  Thank you.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Powerful perl paradigm I don't find in python

[Wolfgang Maier]

On 1/18/2016 14:05, Charles T. Smith wrote:

On Fri, 15 Jan 2016 14:20:17 +0100, Wolfgang Maier wrote:


pattern = pattern_str.compile()
try:
  matches = pattern.findall(some_str, endpos=some_str.index(tail))
except ValueError:
  # do something if tail is not found
  pass


Oh!  I think that's it!


matches = findall (pattern, string)
for file in matches:
 use (file)

Totally cool!  Thank you.



Great if it helps you. Just beware that this simplified version is not 
exactly equivalent to your initial perl snippet:


Generally, findall will find ALL occurrences of pattern, not just 
adjacent ones.


Since your perl example would never terminate if something non-matching 
is interleaved with pattern matches I figured you never expect that case.


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus


--
https://mail.python.org/mailman/listinfo/python-list

Re: parallel (concurrent) eventlet

[David Gabriel]

Dears,

Let me add one more detail: When I add these two lines to check whether my
modules are monkey_patched or not I get *False* as a result.
I think it is strange to get this result since I patched my modules at the
beginning using: eventlet.monkey_patch() as detailed here
.

print "*is_monkey_patched(ldap.syncrepl) : %s*" %
eventlet.patcher.is_monkey_patched('ldap.syncrepl')
print "*is_monkey_patched(ldap.ldapobject) : %s*" %
eventlet.patcher.is_monkey_patched('ldap.ldapobject')


Please advise me how to fix this issue.
Kind regards.


2016-01-18 12:03 GMT+01:00 David Gabriel :

> Dears,
>
> I have an issue when I use eventlet Api to create parallel threads.
> In fact, when I run the below code, only the program dealing with the
> synchronozation with ldap data base is working and is continuously blocking
> the others to run.
> But, when I use the 'thread' Api the program is working fine without any
> blocking issue. However, I can not use thread Api and I must to use
> eventlet.
> So I am wondering how to get the thread Api behavior using the eventlet
> Api ?
>
> Could you please inform me how to fix this issue ?
>
> Kindly find below my code.
> But you need some configurations regarding ldap server/client.
>
>
> #!/usr/bin/python
> # -*- coding: utf-8 -*-
> """
> This script implements a syncrepl consumer which syncs data from an
> OpenLDAP
> server to a local (shelve) database.
> Notes:
> The bound user needs read access to the attributes entryDN and entryCSN.
> This needs the following software:
> Python
> pyasn1 0.1.4+
> pyasn1-modules
> python-ldap 2.4.10+
> """
>
> # Import the python-ldap modules
> import ldap,ldapurl
> # Import specific classes from python-ldap
> from ldap.ldapobject import ReconnectLDAPObject
> from ldap.syncrepl import SyncreplConsumer
>
> # Import modules from Python standard lib
> import shelve,signal,time,sys,logging
> import eventlet
> #import thread
> eventlet.monkey_patch()
>
> # Global state
> watcher_running = True
> ldap_connection = False
>
>
>
> class SyncReplConsumer(ReconnectLDAPObject,SyncreplConsumer):
> """
> Syncrepl Consumer interface
> """
> def __init__(self,db_path,*args,**kwargs):
> # Initialise the LDAP Connection first
> ldap.ldapobject.ReconnectLDAPObject.__init__(self, *args, **kwargs)
> # Now prepare the data store
> self.__data = shelve.open(db_path, 'c')
> # We need this for later internal use
> self.__presentUUIDs = dict()
>
> def __del__(self):
> # Close the data store properly to avoid corruption
> self.__data.close()
>
> def syncrepl_get_cookie(self):
> if 'cookie' in self.__data:
> return self.__data['cookie']
>
> def syncrepl_set_cookie(self,cookie):
> self.__data['cookie'] = cookie
>
> def syncrepl_entry(self,dn,attributes,uuid):
>
> # First we determine the type of change we have here (and store
> away the previous data for later if needed)
> previous_attributes = dict()
> if uuid in self.__data:
> change_type = 'modify'
> previous_attributes = self.__data[uuid]
> else:
> change_type = 'add'
> # Now we store our knowledge of the existence of this entry
> (including the DN as an attribute for convenience)
> attributes['dn'] = dn
> self.__data[uuid] = attributes
> # Debugging
> print 'Detected', change_type, 'of entry:', dn
> # If we have a cookie then this is not our first time being run,
> so it must be a change
> if 'ldap_cookie' in self.__data:
> self.perform_application_sync(dn, attributes,
> previous_attributes)
>
> def syncrepl_delete(self,uuids):
> # Make sure we know about the UUID being deleted, just in case...
> uuids = [uuid for uuid in uuids if uuid in self.__data]
> # Delete all the UUID values we know of
> for uuid in uuids:
> print 'Detected deletion of entry:', self.__data[uuid]['dn']
> del self.__data[uuid]
>
> def syncrepl_present(self,uuids,refreshDeletes=False):
> # If we have not been given any UUID values, then we have recieved
> all the present controls...
> if uuids is None:
> # We only do things if refreshDeletes is false as the syncrepl
> extension will call syncrepl_delete instead when it detects a delete notice
> if refreshDeletes is False:
> deletedEntries = [uuid for uuid in self.__data.keys() if
> uuid not in self.__presentUUIDs and uuid != 'ldap_cookie']
> self.syncrepl_delete( deletedEntries )
> # Phase is now completed, reset the list
> self.__presentUUIDs = {}
> else:
> # Note down all the UUIDs we have been sent
> for uuid in uuids:
> self.__presentUUIDs[uuid] = 

When is an int not an int? Who can explain this?

[Charles T. Smith]

$ python
Python 2.6.5 (r265:79063, Apr 16 2010, 13:09:56) 
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> type(0) is int
True
...
(PDB)type(0) is int
False
(PDB)type(1) is int
False

(PDB)p 5 + 0
5

(PDB)class c (object): pass
(PDB)type (c()) is c
True



-- 
https://mail.python.org/mailman/listinfo/python-list

Re: When is an int not an int? Who can explain this?

[Chris Angelico]

On Tue, Jan 19, 2016 at 3:11 AM, Charles T. Smith
 wrote:
> $ python
> Python 2.6.5 (r265:79063, Apr 16 2010, 13:09:56)
> [GCC 4.4.3] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
 type(0) is int
> True
> ...
> (PDB)type(0) is int
> False
> (PDB)type(1) is int
> False

Possibility #1: 'int' has been rebound.

Possibility #2: 'type' has been rebound.

I'd check them in that order.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: When is an int not an int? Who can explain this?

[Charles T. Smith]

On Tue, 19 Jan 2016 03:19:59 +1100, Chris Angelico wrote:

> On Tue, Jan 19, 2016 at 3:11 AM, Charles T. Smith
>  wrote:
>> $ python
>> Python 2.6.5 (r265:79063, Apr 16 2010, 13:09:56) [GCC 4.4.3] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
> type(0) is int
>> True
>> ...
>> (PDB)type(0) is int
>> False
>> (PDB)type(1) is int
>> False
> 
> Possibility #1: 'int' has been rebound.
> 
> Possibility #2: 'type' has been rebound.
> 
> I'd check them in that order.
> 
> ChrisA


But:

(PDB)p 5 + 0
5

(PDB)class c (object): pass
(PDB)type (c()) is c
True

Okay, I think I understand it now:

(PDB)type (int)


(PDB)type (float)


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: parallel (concurrent) eventlet

[Ian Kelly]

On Mon, Jan 18, 2016 at 8:03 AM, David Gabriel  wrote:
> Dears,
>
> Let me add one more detail: When I add these two lines to check whether my
> modules are monkey_patched or not I get *False* as a result.
> I think it is strange to get this result since I patched my modules at the
> beginning using: eventlet.monkey_patch() as detailed here
> .
>
> print "*is_monkey_patched(ldap.syncrepl) : %s*" %
> eventlet.patcher.is_monkey_patched('ldap.syncrepl')
> print "*is_monkey_patched(ldap.ldapobject) : %s*" %
> eventlet.patcher.is_monkey_patched('ldap.ldapobject')

In your code you monkey patched the standard library modules, not the
ldap modules, so I think it's unsurprising that these return False.

As to why the ldap modules are blocking, python-ldap uses the OpenLDAP
client which is written in C and thus using its own blocking socket
code rather than the Python socket module. I don't believe it is
possible to "green" that. Therefore I think your only option for
concurrency with python-ldap is threading. It is possible to use
threads in conjunction with eventlet; see
http://eventlet.net/doc/threading.html
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: When is an int not an int? Who can explain this?

[Chris Angelico]

On Tue, Jan 19, 2016 at 3:28 AM, Charles T. Smith
 wrote:
> On Tue, 19 Jan 2016 03:19:59 +1100, Chris Angelico wrote:
>> Possibility #1: 'int' has been rebound.
>>
>> Possibility #2: 'type' has been rebound.
>>
>> I'd check them in that order.
>>
>> ChrisA
>
>
> But:
>
> (PDB)p 5 + 0
> 5

That shows that your literals haven't changed. They basically never
will (barring *really* crazy shenanigans).

> (PDB)class c (object): pass
> (PDB)type (c()) is c
> True

And this shows that the name 'type' probably hasn't changed. It's
still possible that it's not the normal 'type' type, but far more
likely that it's 'int' that got rebound, which is why I suggested
checking for that first.

> Okay, I think I understand it now:
>
> (PDB)type (int)
> 
>
> (PDB)type (float)
> 

And that's pretty strong evidence right there! So the next question
is... what got imported under the name 'int'?

int.__name__ will be a start. If that just returns the string 'int',
then try int.__package__ which might give a hint. Also, int.__file__
will tell you where it was loaded from, if it was indeed loaded from a
file.

Armed with that information, you should be able to track down what's
going on. It's curious, though, that you have a callable subclass of
module bound to the name int. Very curious indeed.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: When is an int not an int? Who can explain this?

[Ian Kelly]

On Mon, Jan 18, 2016 at 9:51 AM, Chris Angelico  wrote:
> On Tue, Jan 19, 2016 at 3:28 AM, Charles T. Smith
>> Okay, I think I understand it now:
>>
>> (PDB)type (int)
>> 
>>
>> (PDB)type (float)
>> 
>
> And that's pretty strong evidence right there! So the next question
> is... what got imported under the name 'int'?
>
> int.__name__ will be a start. If that just returns the string 'int',
> then try int.__package__ which might give a hint. Also, int.__file__
> will tell you where it was loaded from, if it was indeed loaded from a
> file.
>
> Armed with that information, you should be able to track down what's
> going on. It's curious, though, that you have a callable subclass of
> module bound to the name int. Very curious indeed.

What makes you think that it's a callable subclass? I don't see any
place in the posted transcript where int has been called.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: When is an int not an int? Who can explain this?

[Random832]

On Mon, Jan 18, 2016, at 11:51, Chris Angelico wrote:
> Armed with that information, you should be able to track down what's
> going on. It's curious, though, that you have a callable subclass of
> module bound to the name int. Very curious indeed.

He hasn't tried calling it. And there's no reason to assume it is a
subclass. The results he's shown imply that the object bound to the name
int is an instance of a type named module* (which is almost certainly,
but not definitely, the usual module type itself, and no more likely to
be a subclass of it than of anything else.)

*Strictly speaking, a type whose repr is ""
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: When is an int not an int? Who can explain this?

[Chris Angelico]

On Tue, Jan 19, 2016 at 4:00 AM, Ian Kelly  wrote:
>> Armed with that information, you should be able to track down what's
>> going on. It's curious, though, that you have a callable subclass of
>> module bound to the name int. Very curious indeed.
>
> What makes you think that it's a callable subclass? I don't see any
> place in the posted transcript where int has been called.

Oops, my bad. I thought he'd called int(5) in his original demo. So,
yeah, it's just a module. (Or, as Random832 points out, it's something
with that repr. But probably a module.) Strike that bit about it being
callable.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Question about function parameter copy

[Robert]

Hi,

I see the following line in a cython file (.pyx):




cdef inline dtype_t _logsumexp(dtype_t[:] X) nogil:
.

  fwdlattice[t, j] = _logsumexp(work_buffer) + framelogprob[t, j]

I find that [:] is about object copy, but I am not sure about its usage
here in a function parameter. Whether is it true that it makes a parameter
copy?
Why is it used in this way?


Thanks,
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question about function parameter copy

[Random832]

On Mon, Jan 18, 2016, at 13:44, Robert wrote:
> cdef inline dtype_t _logsumexp(dtype_t[:] X) nogil:
> .
> 
>   fwdlattice[t, j] = _logsumexp(work_buffer) + framelogprob[t, j]
> 
> I find that [:] is about object copy, but I am not sure about its usage
> here in a function parameter. Whether is it true that it makes a
> parameter
> copy?
> Why is it used in this way?

This is cython-specific syntax indicating that the parameter X is a
one-dimensional memory view (an array) whose elements are dtype_t. The
information you saw about object copying has to do with standard python
statements like "a_copy = a_original[:]", not examples like this one.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python.Exe Problem

[mateusz . nasciszewski]

Hello, I have the exact same issue.

I've installed Python 3.5 AMD64 from python.org 
(https://www.python.org/ftp/python/3.5.0/python-3.5.0a1-amd64.exe to be exact), 
and I've checked the installer MD5sum to be valid.
I've selected the advanced install to change installation directory and to 
precompile standard libs.
Upon finishing the installation I was displayed the message twice, the first 
time about python.exe, and the second about python35.dll, after which the 
installer finished without reporting other errors.

When launching Python 3 from the console via `py -3` I get the exact same 
dialogue box (twice: once for the exe and once for the dll), while `py -2` and 
`py -3.5-32` work with no issue (installed via seemingly the same method, but 
with the 32 bit installer instead).

I wanted to install an x64 version of python because I need additional memory 
to calculate on large numpy arrays.

Regards, -- Mateusz Naściszewski
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python.Exe Problem

[eryk sun]

On Mon, Jan 18, 2016 at 5:17 PM,   wrote:
>
> I've installed Python 3.5 AMD64 from python.org
> (https://www.python.org/ftp/python/3.5.0/python-3.5.0a1-amd64.exe to be 
> exact),

3.5.0a1 is the first alpha version, released 7 months before the final
version of 3.5.0. Remove it, and install 3.5.1 using one of the
following installers:

Web-based
https://www.python.org/ftp/python/3.5.1/python-3.5.1-amd64-webinstall.exe

Offline
https://www.python.org/ftp/python/3.5.1/python-3.5.1-amd64.exe
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python and Word OLE

[tdsperth]

On Friday, January 15, 2016 at 2:04:08 PM UTC+8, tdsp...@gmail.com wrote:
> Hi All
> 
> I am trying to change the active printer using win32com.client with the 
> following code 
> 
> 
> 
> wordapp = win32com.client.Dispatch('Word.Application')
> worddoc = wordapp.Documents.Open(mypathfile)
> print(wordapp.ActivePrinter)
> "\\server\queries"
> wordapp.ActivePrinter = "\\server\letters" //fails on this line
> 
> 
>   File "C:\Anaconda3\lib\site-packages\win32com\client\dynamic.py", line 544, 
> in __setattr__
> self._oleobj_.Invoke(entry.dispid, 0, invoke_type, 0, value)
> pywintypes.com_error: (-2147352567, 'Exception occurred.', (0, 'Microsoft 
> Word', 'There is a printer error.', 'wdmain11.
> chm', 24696, -2146823072), None)
> 
> If I don't try to change printers - it prints too the default printer.
> 
> Cheers
> 
> Colin

server\\letters - the extra back slashes were required.

Colin
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python.Exe Problem

[Mateusz Naściszewski]

W dniu wtorek, 19 stycznia 2016 00:42:20 UTC+1 użytkownik eryk sun napisał:
> On Mon, Jan 18, 2016 at 5:17 PM,   wrote:
> >
> > I've installed Python 3.5 AMD64 from python.org
> > (https://www.python.org/ftp/python/3.5.0/python-3.5.0a1-amd64.exe to be 
> > exact),
> 
> 3.5.0a1 is the first alpha version, released 7 months before the final
> version of 3.5.0. Remove it, and install 3.5.1 using one of the
> following installers:
> 
> Web-based
> https://www.python.org/ftp/python/3.5.1/python-3.5.1-amd64-webinstall.exe
> 
> Offline
> https://www.python.org/ftp/python/3.5.1/python-3.5.1-amd64.exe

Thank you, that seems to have helped. Not sure why I downloaded from an old 
download page.
The dialogue boxes don't appear and everything seems to be working.

Thanks again, -- Mateusz Naściszewski
-- 
https://mail.python.org/mailman/listinfo/python-list

Re[2]: Python.Exe Problem

[MRAB]

On 2016-01-18 23:17:10, mateusz.nasciszew...@gmail.com wrote:


Hello, I have the exact same issue.

I've installed Python 3.5 AMD64 from python.org 
(https://www.python.org/ftp/python/3.5.0/python-3.5.0a1-amd64.exe to be 
exact), and I've checked the installer MD5sum to be valid.
I've selected the advanced install to change installation directory and 
to precompile standard libs.
Upon finishing the installation I was displayed the message twice, the 
first time about python.exe, and the second about python35.dll, after 
which the installer finished without reporting other errors.


When launching Python 3 from the console via `py -3` I get the exact 
same dialogue box (twice: once for the exe and once for the dll), while 
`py -2` and `py -3.5-32` work with no issue (installed via seemingly 
the same method, but with the 32 bit installer instead).


I wanted to install an x64 version of python because I need additional 
memory to calculate on large numpy arrays.


Why did you want python-3.5.0a1-amd64.exe? That was the first alpha 
release of Python 3.5.0 (64-bit).


The latest release is Python 3.5.1.

--
https://mail.python.org/mailman/listinfo/python-list

Why generators take long time?

[Arshpreet Singh]

I was playing with Generators and found that using Generators time is bit more 
than list-comprehensions or I am doing it wrong?


Function with List comprehensions:

def sum_text(number_range):
return sum([i*i for i in xrange(number_range)])

%timeit sum_text(1)
1 loops, best of 3: 14.8 s per loop

Using generator Expressions:

def sum_text(number_range):

return sum((i*i for i in xrange(number_range))) 

%timeit sum_text(1)

1 loops, best of 3: 16.4 s per loop


-- 
https://mail.python.org/mailman/listinfo/python-list