Re: [Cython] gilnanny
On 18 April 2011 22:26, Robert Bradshaw wrote: > On Mon, Apr 18, 2011 at 12:08 PM, mark florisson > wrote: >> Can I add a gilnanny to refnanny? I want to do a PyThreadState_Get() >> for every refnanny inc- and decref, so that it will issue a fatal >> error whenever reference counting is done without the gil, to make >> sure we never do any illegal things in nogil code blocks. > > Sounds like a good idea to me. Ok, cool :) >> While I'm at it, I also want to add a pystate.pxd to the cpython includes. > > Sure, corresponding to pystate.h? Have you looked into how stable this > API is (e.g. Py2 vs. Py3)? I haven't looked at all the functions, but most of them are documented in both Python 2 and Python 3. I'll be vigilant. > - Robert > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
[Cython] Hudson account
Can I get a Hudson account so I can setup my branches there? I can't seem to login using my trac credentials. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Hudson account
mark florisson, 19.04.2011 11:14: Can I get a Hudson account so I can setup my branches there? I've created one for you. Please tell me when you have changed your password, so that I can give you write access. Please be careful not to break too much, and note that it's easier to work at the file system level if you want to do mass duplication/adaptation of jobs. If you tell me what you need, I can set it up for you. It may actually be best to start from Viteks configured jobs, copy them and adapt their github URL and branch name. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Hudson account
On 19 April 2011 12:19, Stefan Behnel wrote: > mark florisson, 19.04.2011 11:14: >> >> Can I get a Hudson account so I can setup my branches there? > > I've created one for you. Please tell me when you have changed your > password, so that I can give you write access. Done. > Please be careful not to break too much, and note that it's easier to work > at the file system level if you want to do mass duplication/adaptation of > jobs. If you tell me what you need, I can set it up for you. All I want is to run the master branch (it has the 'with gil' statement') and the openmp branch. I also want the fusedtypes branch, which I'll create in a minute or two. > It may actually be best to start from Viteks configured jobs, copy them and > adapt their github URL and branch name. > > Stefan > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Hudson account
On 19 April 2011 12:31, mark florisson wrote: > On 19 April 2011 12:19, Stefan Behnel wrote: >> mark florisson, 19.04.2011 11:14: >>> >>> Can I get a Hudson account so I can setup my branches there? >> >> I've created one for you. Please tell me when you have changed your >> password, so that I can give you write access. > > Done. > >> Please be careful not to break too much, and note that it's easier to work >> at the file system level if you want to do mass duplication/adaptation of >> jobs. If you tell me what you need, I can set it up for you. > > All I want is to run the master branch (it has the 'with gil' > statement') and the openmp branch. I also want the fusedtypes branch, > which I'll create in a minute or two. Oh, as for the versions, I think at least 2.3, 2.7 and py3k would be a good idea. Does the hudson server have numpy installed? >> It may actually be best to start from Viteks configured jobs, copy them and >> adapt their github URL and branch name. >> >> Stefan >> ___ >> cython-devel mailing list >> cython-devel@python.org >> http://mail.python.org/mailman/listinfo/cython-devel >> > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Hudson account
mark florisson, 19.04.2011 12:31: On 19 April 2011 12:19, Stefan Behnel wrote: mark florisson, 19.04.2011 11:14: Can I get a Hudson account so I can setup my branches there? I've created one for you. Please tell me when you have changed your password, so that I can give you write access. Done. Ok, you should have write rights now. Please be careful not to break too much, and note that it's easier to work at the file system level if you want to do mass duplication/adaptation of jobs. If you tell me what you need, I can set it up for you. All I want is to run the master branch (it has the 'with gil' statement') and the openmp branch. I also want the fusedtypes branch, which I'll create in a minute or two. That's three branches in total, plus the builds in three CPython versions (2.4, 2.7 and Py3k), plus 2xC/C++ testing each. So, at least 3+9+18=30 new jobs that will compete with the others. I would expect that the three branches would rarely be changed all at the same time (except when you update them from the main branch), but that may still result in quite some additional delays due to the longer build queue. Vitek currently has one set of build jobs for himself and can change the branches at need. Question to the others: should we continue to do this, or set up a full set of build jobs for each git branch we want to test? Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Hudson account
On 19 April 2011 12:53, Stefan Behnel wrote: > mark florisson, 19.04.2011 12:31: >> >> On 19 April 2011 12:19, Stefan Behnel wrote: >>> >>> mark florisson, 19.04.2011 11:14: Can I get a Hudson account so I can setup my branches there? >>> >>> I've created one for you. Please tell me when you have changed your >>> password, so that I can give you write access. >> >> Done. > > Ok, you should have write rights now. > > >>> Please be careful not to break too much, and note that it's easier to >>> work >>> at the file system level if you want to do mass duplication/adaptation of >>> jobs. If you tell me what you need, I can set it up for you. >> >> All I want is to run the master branch (it has the 'with gil' >> statement') and the openmp branch. I also want the fusedtypes branch, >> which I'll create in a minute or two. > > That's three branches in total, plus the builds in three CPython versions > (2.4, 2.7 and Py3k), plus 2xC/C++ testing each. So, at least 3+9+18=30 new > jobs that will compete with the others. > > I would expect that the three branches would rarely be changed all at the > same time (except when you update them from the main branch), but that may > still result in quite some additional delays due to the longer build queue. Switching the branches would be fine with me, so 3 Python versions x 1 branch x 2xC/C++ = 6 jobs. Is it necessary to separate C from C++ testing? > Vitek currently has one set of build jobs for himself and can change the > branches at need. Question to the others: should we continue to do this, or > set up a full set of build jobs for each git branch we want to test? > > Stefan > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Hudson account
mark florisson, 19.04.2011 13:09: On 19 April 2011 12:53, Stefan Behnel wrote: mark florisson, 19.04.2011 12:31: All I want is to run the master branch (it has the 'with gil' statement') and the openmp branch. I also want the fusedtypes branch, which I'll create in a minute or two. That's three branches in total, plus the builds in three CPython versions (2.4, 2.7 and Py3k), plus 2xC/C++ testing each. So, at least 3+9+18=30 new jobs that will compete with the others. I would expect that the three branches would rarely be changed all at the same time (except when you update them from the main branch), but that may still result in quite some additional delays due to the longer build queue. Switching the branches would be fine with me, so 3 Python versions x 1 branch x 2xC/C++ = 6 jobs. 10 actually (1+3+6), but only the (6) test jobs are long running. The job tree first builds an sdist from github, then builds bdists from that in all CPython versions, then tests the result in the test jobs. Is it necessary to separate C from C++ testing? Not necessary, it just gives quicker feedback, right after the first test job is finished. If you test both in one job, it just runs twice as long. The problem is not so much the sheer number of jobs (we have several hundred Hudson jobs up at work). It's the time it takes to run the ones that get triggered, and especially long running jobs that fill up the pipeline. If you can live with a longer response time (usually just a couple of minutes), I'd suggest merging the test jobs, so that only one of them fills up the pipeline per build, instead of two. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Hudson account
On 19 April 2011 13:36, Stefan Behnel wrote: > mark florisson, 19.04.2011 13:09: >> >> On 19 April 2011 12:53, Stefan Behnel wrote: >>> >>> mark florisson, 19.04.2011 12:31: All I want is to run the master branch (it has the 'with gil' statement') and the openmp branch. I also want the fusedtypes branch, which I'll create in a minute or two. >>> >>> That's three branches in total, plus the builds in three CPython versions >>> (2.4, 2.7 and Py3k), plus 2xC/C++ testing each. So, at least 3+9+18=30 >>> new >>> jobs that will compete with the others. >>> >>> I would expect that the three branches would rarely be changed all at the >>> same time (except when you update them from the main branch), but that >>> may >>> still result in quite some additional delays due to the longer build >>> queue. >> >> Switching the branches would be fine with me, so 3 Python versions x 1 >> branch x 2xC/C++ = 6 jobs. > > 10 actually (1+3+6), but only the (6) test jobs are long running. > > The job tree first builds an sdist from github, then builds bdists from that > in all CPython versions, then tests the result in the test jobs. > I see, ok. >> Is it necessary to separate C from C++ testing? > > Not necessary, it just gives quicker feedback, right after the first test > job is finished. If you test both in one job, it just runs twice as long. > > The problem is not so much the sheer number of jobs (we have several hundred > Hudson jobs up at work). It's the time it takes to run the ones that get > triggered, and especially long running jobs that fill up the pipeline. > > If you can live with a longer response time (usually just a couple of > minutes), I'd suggest merging the test jobs, so that only one of them fills > up the pipeline per build, instead of two. Ok. So are you setting it up, or should I do it? > Stefan > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Hudson account
mark florisson, 19.04.2011 13:46: So are you setting it up Done. You now have your first red Hudson jobs. ;) Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Hudson account
On 19 April 2011 18:27, Stefan Behnel wrote: > mark florisson, 19.04.2011 13:46: >> >> So are you setting it up > > Done. You now have your first red Hudson jobs. ;) Thanks a lot for setting it up! Red means good right? :) > Stefan > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] gilnanny
Den 19.04.2011 07:08, skrev Stefan Behnel: Yes, that's what this is all about. https://github.com/markflorisson88/cython/commits/master Great :) Sturla ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
[Cython] libcpp.string operators
hi, i have been using a stub for the c++ in a lot of my work.
i decided to have a go at doing a proper string.pxd, pasted here:
https://gist.github.com/929604
other than the operators, it's mostly implemented with tests. but when
i try the operators
with this test:
def test_equal_operator(char *a, char *b):
"""
>>> test_equal_operator("asdf", "asdf")
True
"""
cdef string s = string(a)
cdef string t = string(b)
cdef bint same = t == s
return same
and this declaration in the pxd:
bint operator==(string&, string&)
i get the error below. any ideas what i'm doing wrong?
thanks,
-brent
=== Got errors: ===
152:23: Invalid types for '==' (string, string)
==
ERROR: runTest (__main__.CythonRunTestCase)
compiling (cpp) and running cpp_stl_string
--
Traceback (most recent call last):
File "runtests.py", line 569, in run
self.runCompileTest()
File "runtests.py", line 400, in runCompileTest
self.test_directory, self.expect_errors, self.annotate)
File "runtests.py", line 546, in compile
self.assertEquals(None, unexpected_error)
AssertionError: None != u"152:23: Invalid types for '==' (string, string)"
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] libcpp.string operators
On Tue, Apr 19, 2011 at 2:45 PM, Brent Pedersen wrote:
> hi, i have been using a stub for the c++ in a lot of my work.
> i decided to have a go at doing a proper string.pxd, pasted here:
>
> https://gist.github.com/929604
>
> other than the operators, it's mostly implemented with tests. but when
> i try the operators
> with this test:
>
> def test_equal_operator(char *a, char *b):
> """
> >>> test_equal_operator("asdf", "asdf")
> True
> """
> cdef string s = string(a)
> cdef string t = string(b)
> cdef bint same = t == s
> return same
>
> and this declaration in the pxd:
>
> bint operator==(string&, string&)
it seems:
bint operator==(string&)
is the correct syntax. i had copied the stuff from vector.pxd
>
> i get the error below. any ideas what i'm doing wrong?
> thanks,
> -brent
>
>
>
>
>
> === Got errors: ===
> 152:23: Invalid types for '==' (string, string)
>
>
> ==
> ERROR: runTest (__main__.CythonRunTestCase)
> compiling (cpp) and running cpp_stl_string
> --
> Traceback (most recent call last):
> File "runtests.py", line 569, in run
> self.runCompileTest()
> File "runtests.py", line 400, in runCompileTest
> self.test_directory, self.expect_errors, self.annotate)
> File "runtests.py", line 546, in compile
> self.assertEquals(None, unexpected_error)
> AssertionError: None != u"152:23: Invalid types for '==' (string, string)"
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] libcpp.string operators
On Tue, Apr 19, 2011 at 6:08 PM, Brent Pedersen wrote:
> On Tue, Apr 19, 2011 at 2:45 PM, Brent Pedersen wrote:
>> hi, i have been using a stub for the c++ in a lot of my work.
>> i decided to have a go at doing a proper string.pxd, pasted here:
>>
>> https://gist.github.com/929604
>>
>> other than the operators, it's mostly implemented with tests. but when
>> i try the operators
>> with this test:
>>
>> def test_equal_operator(char *a, char *b):
>> """
>> >>> test_equal_operator("asdf", "asdf")
>> True
>> """
>> cdef string s = string(a)
>> cdef string t = string(b)
>> cdef bint same = t == s
>> return same
>>
>> and this declaration in the pxd:
>>
>> bint operator==(string&, string&)
>
>
> it seems:
>
> bint operator==(string&)
>
> is the correct syntax. i had copied the stuff from vector.pxd
>
>>
>> i get the error below. any ideas what i'm doing wrong?
>> thanks,
>> -brent
>>
>>
>>
>>
>>
>> === Got errors: ===
>> 152:23: Invalid types for '==' (string, string)
>>
>>
>> ==
>> ERROR: runTest (__main__.CythonRunTestCase)
>> compiling (cpp) and running cpp_stl_string
>> --
>> Traceback (most recent call last):
>> File "runtests.py", line 569, in run
>> self.runCompileTest()
>> File "runtests.py", line 400, in runCompileTest
>> self.test_directory, self.expect_errors, self.annotate)
>> File "runtests.py", line 546, in compile
>> self.assertEquals(None, unexpected_error)
>> AssertionError: None != u"152:23: Invalid types for '==' (string, string)"
>>
>
i updated the gist with the operators, didn't do iterators.
https://gist.github.com/929604
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] GSoC Proposal - Reimplement C modules in CPython's standard library in Cython.
2011/4/17 Stefan Behnel > Arthur de Souza Ribeiro, 17.04.2011 20:07: > > Hi Stefan, about your first comment : "And it's better to let Cython know >> that this name refers to a function." in line 69 of encoder.pyx file I >> didn't understand well what does that mean, can you explain more this >> comment? >> > > Hmm, sorry, I think that was not so important. That code line is only used > to override the Python implementation with the implementation from the > external C accelerator module. To do that, it assigns either of the two > functions to a name. So, when that name is called in the code, Cython cannot > know that it actually is a function, and has to resort to Python calling, > whereas a visible c(p)def function that is defined inside of the same module > could be called faster. > > I missed the fact that this name isn't really used inside of the module, so > whether Cython knows that it's a function or not isn't really all that > important. > So, I don't have to be worried about this, right? > > I added another comment to this commit, though: > > > https://github.com/arthursribeiro/JSON-module/commit/e2d80e0aeab6d39ff2d9b847843423ebdb9c57b7#diff-4 > > > I saw your comment and what I understood of it is that the alias that are being attributed to the type names make code slower, I tried to compile in cython the same way that it was in python, but, there is something wrong with it. It says: Error compiling Cython file: ... def _make_iterencode(dict markers, _default, _encoder, _indent, _floatstr, _key_separator, _item_separator, bint _sort_keys, bint _skipkeys, bint _one_shot, ## HACK: hand-optimized bytecode; turn globals into locals ValueError=ValueError, dict=dict, float=float, ^ encoder.pyx:273:13: Empty declarator I turned that way because I think the user can maybe change what types are going to be used and cython do not allow do these things like python. (for reserved words) > > About the other comments, I think I solved them all, any problem with them >> or other ones, please tell me. I'll try to fix. >> > > It looks like you fixed a good deal of them. > > I actually tried to work with your code, but I'm not sure how you are > building it. Could you give me a hint on that? > I'm manually building them using setup.py files, for every module I create one and build manually, I don't think that's the best way to do it, but, to test things, that's the way I'm doing. > > Where did you actually take the original code from? Python 3.2? Or from > Python's hg branch? > > I took the original code from Python 3.2 > > > Note that it's not obvious from your initial commit what you actually >>> changed. It would have been better to import the original file first, >>> rename >>> it to .pyx, and then commit your changes. >>> >> >> I created a directory named 'Diff files' where I put the files generated >> by >> 'diff' command that i run in my computer, if you think it still be better >> if >> I commit and then change, there is no problem for me... >> > > Diff only gives you the final outcome. Committing on top of the original > files has the advantage of making the incremental changes visible > separately. That makes it clearer what you tried, and a good commit comment > will then make it clear why you did it. > > > > I think it's more important to get some performance >>> numbers to see how your module behaves compared to the C accelerator >>> module >>> (_json.c). I think the best approach to this project would actually be to >>> start with profiling the Python implementation to see where performance >>> problems occur (or to look through _json.c to see what the CPython >>> developers considered performance critical), and then put the focus on >>> trying to speed up only those parts of the Python implementation, by >>> adding >>> static types and potentially even rewriting them in a way that Cython can >>> optimise them better. >>> >> >> I've profilled the module I created and the module that is in Python 3.2, >> the result is that the cython module spent about 73% less time then >> python's >> > > That's a common mistake when profiling: the actual time it takes to run is > not meaningful. Depending on how far the two profiled programs differ, they > may interact with the profiler in more or less intensive ways (as is clearly > the case here), so the total time it takes for the programs to run can > differ quite heavily under profiling, even if the non-profiled programs run > at exactly the same speed. > > Also, I don't think you have enabled profiling for the Cython code. You can > do that by passing the "profile=True" directive to the compiler, or by > putting it at the top of the source files. That will add module-inner > function calls to the profiling output. Note, however, that enabling > profiling will slow down
Re: [Cython] GSoC Proposal - Reimplement C modules in CPython's standard library in Cython.
Arthur de Souza Ribeiro wrote: def _make_iterencode(dict markers, _default, _encoder, _indent, _floatstr, _key_separator, _item_separator, bint _sort_keys, bint _skipkeys, bint _one_shot, ## HACK: hand-optimized bytecode; turn globals into locals ValueError=ValueError, dict=dict, float=float, ^ encoder.pyx:273:13: Empty declarator You may need to choose something other than 'float' for the local name to avoid confusing the parser (it thinks you're about to declare a parameter of type 'float'). -- Greg ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
