[Cython] ImportError: DLL load failed: The specified module could not be found.
Hi cython list, I am having problems to distribute a python module for windows (written with cython). It compile ok with mingw (installed with but when I import the module in python I get this error ImportError: DLL load failed: The specified module could not be found. I guess that some where there is an error linking the DLL. Bout how to solve it to create a nice distribution (.exe) for my package? Thanks Adrian ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] cython.parallel tasks, single, master, critical, barriers
On 19 October 2011 06:01, Robert Bradshaw wrote: > On Fri, Oct 14, 2011 at 1:07 PM, mark florisson > wrote: >> On 14 October 2011 19:31, Robert Bradshaw >> wrote: >>> On Wed, Oct 12, 2011 at 7:55 AM, mark florisson >>> wrote: >> I ultimately feel things like that is more important than 100% coverage >> of >> the OpenMP standard. Of course, OpenMP is a lot lower-hanging fruit. > > +1 Prange handles the (corse-grained) SIMD case nicely, and a > task/futures model based on closures would I think flesh this out to > the next level of generality (and complexity). Futures are definitely nice. I suppose I think really like "inline futures", i.e. openmp tasks. I realize that futures may look more pythonic. However, as mentioned previously, I also see issues with that. When you submit a task then you expect a future object, which you might want to pass around. But we don't have the GIL for that. I personally feel that futures is something that should be done by a library (such as concurrent.futures in python 3.2), and inline tasks by a language. It also means I have to write an entire function or closure for perhaps only a few lines of code. I might also want to submit other functions that are not closures, or I might want to reuse my closures that are used for tasks and for something else. So what if my tasks contain more parallel constructs? e.g. what if I have a task closure that I return from my function that generates more tasks itself? Would you just execute them sequentially outside of the parallel construct, or would you simply disallow that? Also, do you restrict future "objects" to only the parallel section? Another problem is that you can only wait on tasks of your direct children. So what if I get access to my parent's future object (assuming you allow tasks to generate tasks), and then want the result of my parent? Or what if I store these future objects in an array or list and access them arbitrarily? You will only know at runtime which task to wait on, and openmp only has a static, lexical taskwait. I suppose my point is that without either a drastic rewrite (e.g., use pthreads instead of openmp) or quite a bit of contraints, I am unsure how futures would work here. Perhaps you guys have some concrete syntax and semantics proposals? >>> >>> It feels to me that OpenMP tasks took a different model of parallelism >>> and tried to force them into the OpenMP model/constraints, and so it'd >>> be even more difficult to fit them into a nice pythonic interface. >>> Perhaps to make progress on this front we need to have a concrete >>> example to look at. I'm also wondering if the standard threading >>> module (perhaps with overlay support) used with nogil functions would >>> be sufficient--locking is required for handling the queues, etc. so >>> the fact that the GIL is involved is not a big deal. It is possible >>> that this won't scale to as small of work units, but the overhead >>> should be minimal once your work unit is a sufficient size (which is >>> probably quite small) and it's already implemented and well >>> documented/used. >> >> It's all definitely possible with normal threads, but the thing you >> lose is convenience and conciseness. For big problems the programmer >> might sum up the courage and effort to implement it, but typically you >> will just stick to a serial version. This is really where OpenMP is >> powerful, you can take a simple sequential piece of code and make it >> parallel with minimal effort and without having to restructure, >> rethink and rewrite your algorithms. > > That is a very good point. > >> Something like concurrent.futures is definitely nice, but most people >> cannot afford to mandate python 3.2 for their users. >> >> The most classical examples I can think of for tasks are >> >> 1) independent code sections, i.e. two or more pieces of code that >> don't depend on each other which you want to execute in parallel >> 2) traversal of some kind of custom data structure, like a tree or a linked >> list >> 3) some kind of other producer/consumer model >> >> e.g. using with task syntax: >> >> cdef postorder_traverse(tree *t): # bullet 1) and 2) >> with task: >> traverse(t.left) >> with task: >> traverse(t.right) >> >> taskwait() # wait until we traversed our subtrees >> use(t.data) > > Is there an implicit parallel block here? Perhaps in the caller? Yes, it was implicit in my example. If you'd use that code, you'd call it from a parallel section. Depending on what semantics you'd define (see below), you'd call it either from one thread in the team, or with all of them. >> cdef list_traverse(linkedlist *L): # bullet 2) >> with nogil, parallel(): >> if threadid() == 0: >> while L.next: >> with task: >> do_s
Re: [Cython] cython.parallel tasks, single, master, critical, barriers
On 19 October 2011 19:19, mark florisson wrote: > On 19 October 2011 06:01, Robert Bradshaw > wrote: >> On Fri, Oct 14, 2011 at 1:07 PM, mark florisson >> wrote: >>> On 14 October 2011 19:31, Robert Bradshaw >>> wrote: On Wed, Oct 12, 2011 at 7:55 AM, mark florisson wrote: >>> I ultimately feel things like that is more important than 100% coverage >>> of >>> the OpenMP standard. Of course, OpenMP is a lot lower-hanging fruit. >> >> +1 Prange handles the (corse-grained) SIMD case nicely, and a >> task/futures model based on closures would I think flesh this out to >> the next level of generality (and complexity). > > Futures are definitely nice. I suppose I think really like "inline > futures", i.e. openmp tasks. I realize that futures may look more > pythonic. However, as mentioned previously, I also see issues with > that. When you submit a task then you expect a future object, which > you might want to pass around. But we don't have the GIL for that. I > personally feel that futures is something that should be done by a > library (such as concurrent.futures in python 3.2), and inline tasks > by a language. It also means I have to write an entire function or > closure for perhaps only a few lines of code. > > I might also want to submit other functions that are not closures, or > I might want to reuse my closures that are used for tasks and for > something else. So what if my tasks contain more parallel constructs? > e.g. what if I have a task closure that I return from my function that > generates more tasks itself? Would you just execute them sequentially > outside of the parallel construct, or would you simply disallow that? > Also, do you restrict future "objects" to only the parallel section? > > Another problem is that you can only wait on tasks of your direct > children. So what if I get access to my parent's future object > (assuming you allow tasks to generate tasks), and then want the result > of my parent? > Or what if I store these future objects in an array or list and access > them arbitrarily? You will only know at runtime which task to wait on, > and openmp only has a static, lexical taskwait. > > I suppose my point is that without either a drastic rewrite (e.g., use > pthreads instead of openmp) or quite a bit of contraints, I am unsure > how futures would work here. Perhaps you guys have some concrete > syntax and semantics proposals? It feels to me that OpenMP tasks took a different model of parallelism and tried to force them into the OpenMP model/constraints, and so it'd be even more difficult to fit them into a nice pythonic interface. Perhaps to make progress on this front we need to have a concrete example to look at. I'm also wondering if the standard threading module (perhaps with overlay support) used with nogil functions would be sufficient--locking is required for handling the queues, etc. so the fact that the GIL is involved is not a big deal. It is possible that this won't scale to as small of work units, but the overhead should be minimal once your work unit is a sufficient size (which is probably quite small) and it's already implemented and well documented/used. >>> >>> It's all definitely possible with normal threads, but the thing you >>> lose is convenience and conciseness. For big problems the programmer >>> might sum up the courage and effort to implement it, but typically you >>> will just stick to a serial version. This is really where OpenMP is >>> powerful, you can take a simple sequential piece of code and make it >>> parallel with minimal effort and without having to restructure, >>> rethink and rewrite your algorithms. >> >> That is a very good point. >> >>> Something like concurrent.futures is definitely nice, but most people >>> cannot afford to mandate python 3.2 for their users. >>> >>> The most classical examples I can think of for tasks are >>> >>> 1) independent code sections, i.e. two or more pieces of code that >>> don't depend on each other which you want to execute in parallel >>> 2) traversal of some kind of custom data structure, like a tree or a linked >>> list >>> 3) some kind of other producer/consumer model >>> >>> e.g. using with task syntax: >>> >>> cdef postorder_traverse(tree *t): # bullet 1) and 2) >>> with task: >>> traverse(t.left) >>> with task: >>> traverse(t.right) >>> >>> taskwait() # wait until we traversed our subtrees >>> use(t.data) >> >> Is there an implicit parallel block here? Perhaps in the caller? > > Yes, it was implicit in my example. If you'd use that code, you'd call > it from a parallel section. Depending on what semantics you'd define > (see below), you'd call it either from one thread in the team, or with > all of them. > >>> cdef list_traverse(linkedlist *L):
Re: [Cython] ImportError: DLL load failed: The specified module could not be found.
Adrian I'm a little unclear on the big picture here. Are you trying to distribute a module (a .pyd / .dll) that you or someone else can import from a .py script, or are you looking to compile a .exe that runs your cython code on execution? Just interpreting the error you're describing (ImportError: DLL load failed: could not be found), the dynamic linker couldn't find a library it needed. Most likely this is either a symptom of missing dependencies or a path problem. Here's my suggestions for diagnosing and fixing the problem: Missing Dependencies: One very simple way to confirm that all the dependencies of your cython module are available is to point the dependency walker utility[1] at it, and look for missing DLLs. Directory Structure: Is the .pyd file you built from your cython module in the PYTHONPATH (or your current working directory? If it's not, there's your issue. [1] http://www.dependencywalker.com/ Hope that helps! Best, -Alex ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] cython.parallel tasks, single, master, critical, barriers
On 19 October 2011 06:01, Robert Bradshaw wrote: > On Fri, Oct 14, 2011 at 1:07 PM, mark florisson > wrote: >> On 14 October 2011 19:31, Robert Bradshaw >> wrote: >>> On Wed, Oct 12, 2011 at 7:55 AM, mark florisson >>> wrote: >> I ultimately feel things like that is more important than 100% coverage >> of >> the OpenMP standard. Of course, OpenMP is a lot lower-hanging fruit. > > +1 Prange handles the (corse-grained) SIMD case nicely, and a > task/futures model based on closures would I think flesh this out to > the next level of generality (and complexity). Futures are definitely nice. I suppose I think really like "inline futures", i.e. openmp tasks. I realize that futures may look more pythonic. However, as mentioned previously, I also see issues with that. When you submit a task then you expect a future object, which you might want to pass around. But we don't have the GIL for that. I personally feel that futures is something that should be done by a library (such as concurrent.futures in python 3.2), and inline tasks by a language. It also means I have to write an entire function or closure for perhaps only a few lines of code. I might also want to submit other functions that are not closures, or I might want to reuse my closures that are used for tasks and for something else. So what if my tasks contain more parallel constructs? e.g. what if I have a task closure that I return from my function that generates more tasks itself? Would you just execute them sequentially outside of the parallel construct, or would you simply disallow that? Also, do you restrict future "objects" to only the parallel section? Another problem is that you can only wait on tasks of your direct children. So what if I get access to my parent's future object (assuming you allow tasks to generate tasks), and then want the result of my parent? Or what if I store these future objects in an array or list and access them arbitrarily? You will only know at runtime which task to wait on, and openmp only has a static, lexical taskwait. I suppose my point is that without either a drastic rewrite (e.g., use pthreads instead of openmp) or quite a bit of contraints, I am unsure how futures would work here. Perhaps you guys have some concrete syntax and semantics proposals? >>> >>> It feels to me that OpenMP tasks took a different model of parallelism >>> and tried to force them into the OpenMP model/constraints, and so it'd >>> be even more difficult to fit them into a nice pythonic interface. >>> Perhaps to make progress on this front we need to have a concrete >>> example to look at. I'm also wondering if the standard threading >>> module (perhaps with overlay support) used with nogil functions would >>> be sufficient--locking is required for handling the queues, etc. so >>> the fact that the GIL is involved is not a big deal. It is possible >>> that this won't scale to as small of work units, but the overhead >>> should be minimal once your work unit is a sufficient size (which is >>> probably quite small) and it's already implemented and well >>> documented/used. >> >> It's all definitely possible with normal threads, but the thing you >> lose is convenience and conciseness. For big problems the programmer >> might sum up the courage and effort to implement it, but typically you >> will just stick to a serial version. This is really where OpenMP is >> powerful, you can take a simple sequential piece of code and make it >> parallel with minimal effort and without having to restructure, >> rethink and rewrite your algorithms. > > That is a very good point. > >> Something like concurrent.futures is definitely nice, but most people >> cannot afford to mandate python 3.2 for their users. >> >> The most classical examples I can think of for tasks are >> >> 1) independent code sections, i.e. two or more pieces of code that >> don't depend on each other which you want to execute in parallel >> 2) traversal of some kind of custom data structure, like a tree or a linked >> list >> 3) some kind of other producer/consumer model >> >> e.g. using with task syntax: >> >> cdef postorder_traverse(tree *t): # bullet 1) and 2) >> with task: >> traverse(t.left) >> with task: >> traverse(t.right) >> >> taskwait() # wait until we traversed our subtrees >> use(t.data) > > Is there an implicit parallel block here? Perhaps in the caller? > >> cdef list_traverse(linkedlist *L): # bullet 2) >> with nogil, parallel(): >> if threadid() == 0: >> while L.next: >> with task: >> do_something(L.data) >> >> In the latter case we don't need a taskwait as we don't care about any >> particular order. Only one thread generates the tasks where the others >> just hit the barrier and see the tasks they can execu
