Swig and distutils
Hi,
I want to write a setup.py for a module from a swig source. The
recommended way [1] does not work because it does not process the
generated Python file. Searching on Stackoverflow gives an (ugly)
workaround and the hint "this may be considered as a bug" [2].
However, I couldn't find a distutils bug or a discussion why this is not
changed (the problems seems to be already >5 years old). Is there a
rationale behind the current behauviour? And shouldn't that documented
in the Python distutils documentation? [1] is very misleading here then.
I have another problem with the current setup: I am using a directory
structure where setup.py resides outside of the current sources (since
it is generated by a CMakeFile) in the build directory. That means that
I have to use the "package_dir" argument of setup.py. This however has
two drawbacks:
1. It does not apply to ext_modules: how can I do that?
2. The Python code generated by swig will/should be placed in the build
dir (since it is generated), so they are not found when the package_dir
is set to the source directory. When I set package_dir to the build
directory, then the other Python source files are not found. How can I
specify both directories here?
One workaround would be to just call the setup function twice: First
with the existing Python source and the swig C sources, and then with
the Python sources generated from swig:
-8<
from distutils.core import setup, Extension
setup(name="my", version="1.0",
package_dir={"my": "../../src"}
py_modules=["my"],
ext_modules=[
Extension("_my_c", ["my_c.i"]) # specify source dir of my_c.i here?
])
setup(name="my", version="1.0", py_modules=["my_c"])
-8<
Although I didn't find any documentation that forbids calling the setup
function twice, I also did not find an example where this is done. Would
that work?
Best regards
Ole
[1]
https://docs.python.org/3.6/distutils/setupscript.html#extension-source-files
[2]
https://stackoverflow.com/questions/17666018/using-distutils-where-swig-interface-file-is-in-src-folder?rq=1
--
https://mail.python.org/mailman/listinfo/python-list
Re: doctest random output?
On Tue, 29 Aug 2017 12:25:45 +1000, Chris Angelico wrote:
> On Tue, Aug 29, 2017 at 11:53 AM, Steve D'Aprano
> wrote:
>> (1) Disable doctesting for that example, and treat it as just
>> documentation:
>>
>> def my_thing():
>> """blah blah blah
>>
>> >>> my_thing() #doctest:+SKIP
>> 4
>>
>> """
>
> For a lot of functions, this completely destroys the value of
> doctesting.
"The" value? Doc tests have two values: documentation (as examples of
use) and as tests. Disabling the test aspect leaves the value as
documentation untouched, and arguably is the least-worst result. You can
always write a unit test suite to perform more detailed, complicated
tests. Doc tests are rarely exhaustive, so you need unit tests as well.
>> (2) Monkey-patch the random module for testing. This is probably the
>> worst idea ever, but it's an idea :-)
>>
>> That makes for a fragile test and poor documentation.
>
> This highlights the inherent weakness of doctests. For proper unit
> testing, I would definitely recommend this. Maybe a hybrid of 1 and 2
> could be organized... hmm.
Doc tests should be seen as *documentation first* and tests second. The
main roll of the tests is to prove that the documented examples still do
what you say they do.
It makes for a horrible and uninformative help() experience to have
detailed, complex, exhaustive doc tests exercising every little corner
case of your function. That should go in your unit tests.
Possibly relevant: the unittest module has functionality to automatically
extract and run your library's doctests, treating them as unit tests. So
you can already do both.
>> (3) Write your functions to take an optional source of randomness, and
>> then in your doctests set them:
>>
>> def my_thing(randint=None):
>> """blah blah blah
>>
>> >>> my_thing(randint=lambda a,b: 4)
>> 4
>>
>> """
>> if randint is None:
>> from random import randint
>> ...
>
> Unless that would be useful for other reasons, not something I like
> doing. Having code in your core that exists solely (or even primarily)
> to make testing easier seems like doing things backwards.
I see your point, and I don't completely disagree. I'm on the fence about
this one. But testing is important, and we often write code to make
testing easier, e.g. pulling out a complex but short bit of code into its
own function so we can test it, using dependency injection, etc. Why
shouldn't we add hooks to enable testing? Not every function needs such a
hook, but some do.
See, for example, "Enemies of Test Driven Development":
https://jasonmbaker.wordpress.com/2009/01/08/enemies-of-test-driven-
development-part-i-encapsulation/
In Python, we have the best of both worlds: we can flag a method as
private, and *still* test it! So in a sense, Python's very design has
been created specifically to allow testing.
For a dissenting view, "Are Private Methods a Code Smell?":
http://carlosschults.net/en/are-private-methods-a-code-smell/
>> (4) Write your doctests to test the most general properties of the
>> returned results:
>>
>>
>> def my_thing(randint=None):
>> """blah blah blah
>>
>> >>> num = my_thing()
>> >>> isinstance(num, int) and 0 <= my_thing() <= 6
>> True
>>
>> """
>
> This is what I'd probably do, tbh.
Sometimes that's sufficient. Sometimes its not. It depends on the
function.
For example, imagine a function that returns a randomly selected prime
number. The larger the prime, the less likely it is to be selected, but
there's no upper limit. So you write:
>>> num = my_thing()
>>> isinstance(num, int) and 2 <= num
True
Not very informative as documentation, and a lousy test too.
> None of the options really appeal though. Personally, I'd probably
> either go with #4, or maybe something like this:
>
> def roll(sequence):
> """Roll a set of dice
>
> >>> from test_mymodule import * # ensure stable RNG
> >>> roll("d12 + 2d6 + 3")
> You roll d12: 8 You roll 2d6: 1, 6, totalling 7.
> You add a bonus of 3 For d12 + 2d6 + 3, you total: 18
> """
>
> and bury all the monkey-patching into test_mymodule.
Wait... are you saying that importing test_mymodule monkey-patches the
current library? And doesn't un-patch it afterwards? That's horrible.
Or are you saying that test_module has its own version of roll(), and so
you're using *that* version instead of the one in the library?
That's horrible too.
I think that once you are talking about monkey-patching things in order
to test them, you should give up on doc tests and use unittest instead.
At least then you get nice setUp and tearDown methods that you can use.
> It can have its own
> implementations of randint and whatever else you use. That way, at least
> there's only one line that does the messing around. I still don't like
> it though - so quite honestly, I'm most likely to go the route of "don't
> actually use doctests".
Are you saying do
Re: doctest random output?
On Tue, Aug 29, 2017 at 5:39 PM, Steven D'Aprano wrote: > On Tue, 29 Aug 2017 12:25:45 +1000, Chris Angelico wrote: > >> For a lot of functions, this completely destroys the value of >> doctesting. > > > "The" value? Doc tests have two values: documentation (as examples of > use) and as tests. Disabling the test aspect leaves the value as > documentation untouched, and arguably is the least-worst result. You can > always write a unit test suite to perform more detailed, complicated > tests. Doc tests are rarely exhaustive, so you need unit tests as well. You can have a docstring that isn't crafted to be runnable tests. The point about doc tests is that they're executable documentation - the point of them is to be tests, not just docs. You can always write your unit tests separately, and let your docstrings merely be documentation, and then none of this matters. > For example, imagine a function that returns a randomly selected prime > number. The larger the prime, the less likely it is to be selected, but > there's no upper limit. So you write: > >>>> num = my_thing() >>>> isinstance(num, int) and 2 <= num >True > > > Not very informative as documentation, and a lousy test too. Yes, but you could have some sort of primality test on it. >>> is_prime(my_thing()) True Even if all you have is a "probably prime" test, that would still make for better documentation AND better testing than no test at all. > Wait... are you saying that importing test_mymodule monkey-patches the > current library? And doesn't un-patch it afterwards? That's horrible. > > Or are you saying that test_module has its own version of roll(), and so > you're using *that* version instead of the one in the library? > > That's horrible too. My original plan was to have *a function in* that module that does the monkey-patching, but I seem to have not actually typed that part in... mea culpa. I agree that merely importing your test helpers shouldn't do the changes! Even with "import test_mymodule; test_mymodule.stable()" it's still just one extra line (better than the full version). Not un-patching it afterwards? Yes. Since part of its purpose is to seed the RNG with a fixed value, it's not really possible or practical to "undo" that, and so I wouldn't worry too much about an "afterwards" - after testing, you exit the interpreter, if you want to get back to normality. >> It can have its own >> implementations of randint and whatever else you use. That way, at least >> there's only one line that does the messing around. I still don't like >> it though - so quite honestly, I'm most likely to go the route of "don't >> actually use doctests". > > Are you saying don't use doctests for *this* problem, or don't use them > *at all*? For this and any other problem where doctesting is impractical. Because let's face it, laziness is a big thing. If it's too much hassle to make a docstring executable, I'm just not going to make it executable. Which has the unfortunate downside of allowing the docstrings to get out of sync with the code, but that's the cost. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
To mock/patch or not to, was Re: doctest random output?
Steven D'Aprano wrote:
> Wait... are you saying that importing test_mymodule monkey-patches the
> current library? And doesn't un-patch it afterwards? That's horrible.
There's something in the library, unittest.mock that makes this relatively
safe -- if not painless
with mock.patch("random.randint", side_effect=[42]) as randint:
self.assertEqual(my_module.my_thing(), 42)
randint.assert_called_once_with(1, 6)
and sometimes monkey-patching may be a necessary evil to verify that a
portion of the code that is buried a bit too deep is called as expected.
However, in this case it tests that the code is what it is rather than what
it does. Good tests would allow for replacing random.randint() with
random.randrange() or random.SystemRandom().randrange() and still succeed.
--
https://mail.python.org/mailman/listinfo/python-list
Re: doctest random output?
On 8/28/17, Leam Hall wrote:
> On 08/28/2017 11:40 AM, Dennis Lee Bieber wrote:
>
> ... a bunch of good stuff ...
>
> I'm (re-)learning python and just trying make sure my function works.
> Not at the statistical or cryptographic level. :)
>
> Thanks!
>
> Leam
> --
> https://mail.python.org/mailman/listinfo/python-list
>
I am not sure what is best practice but I would use sys.exit to
propagate failure (for better scripting possibility).
For example:
if __name__ == "__main__":
import doctest
import sys
sys.exit(doctest.testmod()[0])
But maybe I am wrong and non zero exit status is just for errors in code?
---
If you don't need something at scientific level (which is hard, see:
https://www.random.org/analysis/ ) you could probably use fact that
random sequences are "hard" to compress. For example something like
this could help ->
>>> import zlib
>>> A = [my_thing() for i in range(100)]
>>> 50 < len(zlib.compress(bytes(A))) < 70
True
But be careful!! Other randint parameters would need other limit values!
# for randint(1,6) you have distribution of lengths like this
collections.Counter(len(zlib.compress(bytes(random.randint(1,6) for i
in range(100 for j in range(10))
Counter({55: 1,
56: 46,
57: 834,
58: 7349,
59: 31035,
60: 42884,
61: 16434,
62: 1397,
63: 20})
# but for randint(1,16) you have distribution like this!
collections.Counter(len(zlib.compress(bytes(random.randint(1,16) for i
in range(100 for j in range(10))
Counter({71: 4,
72: 412,
73: 11291,
74: 27392,
75: 28293,
76: 29103,
77: 3296,
78: 209})
So maybe it help you, maybe not :)
--
https://mail.python.org/mailman/listinfo/python-list
Latency for API call to a Python Function
Hi,
I have an issue with one piece of code in Python that when my API using
Nginx/Uwsgi/EC2 AWS calls this function it causes latency. I need help to
figure out if there is I am doing something wrong with this particular
function. Rest of the code calls for DB are working fine and no issues. But
only this piece of code is causing latency.
def getvideos(self, listitem):
channelId = ""
returnVideos,dictData = {},{}
mode = 0
channelId = listitem[KEY_CHANNELID]
if KEY_CHANNELMODE in listitem :
mode = int(listitem[KEY_CHANNELMODE])
filteredVideos = []
if mode == 0:
filteredVideos = self.getFilteredVideos(listitem)
if mode == 1:
filteredVideos = self.getFocus(listitem)
if mode == 2:
filteredVideos = self.getFavourites(listitem)
newFilteredVideos = []
print "Returning videos : ",filteredVideos
if filteredVideos == None or len(filteredVideos) == 0:
print "\n\n\n\tWe are returning NO filteredVideos !"
#filteredVideos = list(set(filteredVideos))
dictData[KEY_SERVICE] = SERVICE_GETVIDEOS
dictData[KEY_CHANNELID] = str(channelId)
if len(filteredVideos) > 0 :
dictData[KEY_VIDEOS] = self.returnVideos(filteredVideos)
else :
dictData[KEY_VIDEOS] = {}
dictData[KEY_VIDEOS][KEY_STATUS] = 'No Videos'
returnVideos[TAG_IRON] = dictData
return json.dumps(returnVideos)
Looking forward,
Regards,
Shazia
--
https://mail.python.org/mailman/listinfo/python-list
Re: Latency for API call to a Python Function
On 8/29/2017 11:22 AM, [email protected] wrote: def getvideos(self, listitem): channelId = "" returnVideos,dictData = {},{} mode = 0 channelId = listitem[KEY_CHANNELID] if KEY_CHANNELMODE in listitem : mode = int(listitem[KEY_CHANNELMODE]) filteredVideos = [] ... Please make code readable by indenting properly, *with spaces*. Guessing at what you meant: def getvideos(self, listitem): channelId = "" returnVideos,dictData = {},{} mode = 0 channelId = listitem[KEY_CHANNELID] if KEY_CHANNELMODE in listitem : mode = int(listitem[KEY_CHANNELMODE]) filteredVideos = []. ... If you posted with tab indents, they disappeared, as is common with news/mail. -- Terry Jan Reedy -- https://mail.python.org/mailman/listinfo/python-list
Re: Latency for API call to a Python Function
On 2017-08-29 16:22, [email protected] wrote: Hi, I have an issue with one piece of code in Python that when my API using Nginx/Uwsgi/EC2 AWS calls this function it causes latency. I need help to figure out if there is I am doing something wrong with this particular function. Rest of the code calls for DB are working fine and no issues. But only this piece of code is causing latency. def getvideos(self, listitem): channelId = "" returnVideos,dictData = {},{} mode = 0 channelId = listitem[KEY_CHANNELID] if KEY_CHANNELMODE in listitem : mode = int(listitem[KEY_CHANNELMODE]) filteredVideos = [] if mode == 0: filteredVideos = self.getFilteredVideos(listitem) if mode == 1: filteredVideos = self.getFocus(listitem) if mode == 2: filteredVideos = self.getFavourites(listitem) newFilteredVideos = [] print "Returning videos : ",filteredVideos if filteredVideos == None or len(filteredVideos) == 0: print "\n\n\n\tWe are returning NO filteredVideos !" #filteredVideos = list(set(filteredVideos)) dictData[KEY_SERVICE] = SERVICE_GETVIDEOS dictData[KEY_CHANNELID] = str(channelId) if len(filteredVideos) > 0 : dictData[KEY_VIDEOS] = self.returnVideos(filteredVideos) else : dictData[KEY_VIDEOS] = {} dictData[KEY_VIDEOS][KEY_STATUS] = 'No Videos' returnVideos[TAG_IRON] = dictData return json.dumps(returnVideos) Looking forward, Regards, Shazia It's difficult to read the code because it's not indented correctly. At first glance, the latency might not be in this method itself, but in one of the other methods it calls. Another possibility is that somewhere you're using a list where a set or dict might be better, e.g. doing a search (it'll be a linear search on a list). -- https://mail.python.org/mailman/listinfo/python-list
Re: Latency for API call to a Python Function
You mean recursive linear search? Or is there a way I can test all my code with multiple files to see which function is causing issue? -- https://mail.python.org/mailman/listinfo/python-list
Re: IDLE won't install on manual installation
Hi! i want to ask my question again. I have now subscribed to the list. I'll just copy and past it here again: Hi! My name is Yusuf Mohammad. I work for a hospital in Norway and we plan to use python to program a mobile application. I have a admin account on my computer and i am trying to install python with the manual option. The reason so that it can be stored in the C:/Program files (x86) section so that the applicaton does not auto uninstall when the computer is turned off (don't know why it does that, but it's standard on our work computers): After installing Python with the manual option, i can't seem to find IDLE. Any suggestions? Kindly Yusuf Mohammad 2017-08-28 8:57 GMT+02:00 Yusuf Mohammad : > Hi! My name is Yusuf Mohammad. I work for a hospital in Norway and we plan > to use python to program a mobile application. > > I have a admin account on my computer and i am trying to install python > with the manual option. The reason so that it can be stored in the > C:/Program files (x86) section so that the applicaton does not auto > uninstall when the computer is turned off (don't know why it does that, but > it's standard on our work computers): > > After installing Python with the manual option, i can't seem to find IDLE. > Any suggestions? > > Kindly > > Yusuf Mohammad > -- https://mail.python.org/mailman/listinfo/python-list
Re: tkinter keypress events are a mess for numpad keys
On 29/08/2017 06:32, Terry Reedy wrote: > *The* documentation (for 8.6) is the tcl.tk/man doc set: > https://www.tcl.tk/man/tcl8.6/TkCmd/contents.htm > For the level of detail you are looking at, they are essential. > > The nmt docs for 8.5 are neither complete (intentionally not) nor always > correct nor > always up to date. The tk docs may also have errors, just as our do, but I > presume one > can file a report somewhere and possibly get a fix. Hi Terry thanks for pointing me to that resource. I'll have a look at https://www.tcl.tk/man/tcl8.6/TkCmd/keysyms.htm but I don't have high hopes because I already tried empirically to figure out the distinguishing attributes of the keypress event object, on various platforms (Mac OS, Linux, Windows)... Irmen. -- https://mail.python.org/mailman/listinfo/python-list
Re: Latency for API call to a Python Function
On 2017-08-29 17:08, [email protected] wrote: You mean recursive linear search? Or is there a way I can test all my code with multiple files to see which function is causing issue? You can just put some timing around the methods that might be a problem, writing to a logfile. Process a few files and then look at the logfile to see whether any of those methods took longer than expected. -- https://mail.python.org/mailman/listinfo/python-list
Re: doctest random output?
On Wed, Aug 30, 2017 at 1:39 AM, Stefan Ram wrote:
> Dennis Lee Bieber writes:
>>Testing randomness itself requires statistical tests...
>
> A perfectly random coin /can/ yield "heads" a thousand times
> in sequence (which is very unlikely, but possible).
>
> This behavior should fail nearly all statistical tests for
> randomness. Yet the generator was perfectly random.
>
> So the tests for randomness give correct answers only with
> a certain probability ("confidence"). Insofar the concept of
> randomness is "fuzzy" when defined as an observable
> property of an otherwise "black box".
>
> The tests in the OP test only what one can test with
> certainity, which might be reasonable.
>
> To gain confidence in a function providing sufficiently
> "random" results other measures might be added, such as
> a code review (view the generator as a "white box").
The point of unit testing (of which doctests are a form) is generally
that you test THIS function, without needing to test everything else.
Testing whether random.random() is "sufficiently random" is not the
point of the doctest. For a non-trivial example, consider my dice
roller; I don't have a Python function for it, but it's a feature of
my D&D MUD. You pass it a string that details the dice you want to
roll, and it rolls them:
>>> roll d20
You roll d20: 3
>>> roll d20 + 5
You roll d20: 14
You add a bonus of 5
For d20 + 5, you total: 19
>>> roll 3d6+d8 -2
You roll 3d6: 1, 5, 5, totalling 11.
You roll d8: 2
You add a bonus of -2
For 3d6+d8 -2, you total: 11
This is fine as documentation. The trouble is that, for testing, we
have to logically accept any integer from 1 to 20 as "correct", and
doctest doesn't support that. I don't care, in this test, whether the
dice roller is "fair" (that it has equal probability of returning each
value) - what I care about is whether, when you enter a particular
string of dice descriptions, you get back a proper pattern of rolls.
And I don't think doctest is flexible enough to handle this without
some sort of monkeypatching - unless you code your function to use
NOTHING other than random.random(), and then you can reliably just
seed the RNG.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
automatically set field values in django using function
I need t define some values of fields automatically in my model using function.
my function get input image path and calculate and I need that calculation
results to define my database fields in Django.
but first the upload image need to get the path in server and after i can to
define the values..
i will test this but myfunc cant understand the image path from upload image.
here the code :
def myview(request):
uploadimages = UploadImagesForm(request.POST or None, request.FILES or None)
if uploadimages.is_valid():
# Get the images that have been browsed
if request.FILES.get('multipleimages', None) is not None:
images = request.FILES.getlist('multipleimages')
for image in images:
instance = MyModel.objects.create(field1=request.user,
field2=image)
instance.field3 = myfunc(image)
instance.field4 = myfunc(image)
instance.save()
--
https://mail.python.org/mailman/listinfo/python-list
pip which should be installed with Python3.62 doesn't work in windows
I found many rookies like me asking similar question: why "pip" is not recognized by cmd? I install Python 3.62 64bit in Win10 and I'm sure I add PATH, the Python tutorial doesn't even mention about PATH. You can see the following picture showing the tutorial doesn't work. Please give a hand to the helpless rookies and me. [image: 內置圖片 1] tutorials url: https://packaging.python.org/tutorials/installing-packages/#creating-virtual-environments -- https://mail.python.org/mailman/listinfo/python-list
Re: pip which should be installed with Python3.62 doesn't work in windows
On 2017-08-29 20:10, Bear Light wrote: I found many rookies like me asking similar question: why "pip" is not recognized by cmd? I install Python 3.62 64bit in Win10 and I'm sure I add PATH, the Python tutorial doesn't even mention about PATH. You can see the following picture showing the tutorial doesn't work. This newsgroup is text-only. Please give a hand to the helpless rookies and me. [image: 內置圖片 1] tutorials url: https://packaging.python.org/tutorials/installing-packages/#creating-virtual-environments The best way is to use the Python "launcher" py.exe, which should be on the search path, so instead of, say: pip install regex you would write: py -3.6 -m pip install regex This ensures that it's installed into the correct version of Python (in this case v3.6). -- https://mail.python.org/mailman/listinfo/python-list
RE: pip which should be installed with Python3.62 doesn't work inwindows
Thanks for help but it doesn’t work. (cmd) C:\Users\user>py -3.6 -m pip install regex C:\Program Files\Python36\python.exe: No module named pip The picture that I attached showed similar result. Though you can see the option of installing pip during python3 installing, the command about pip doesn’t work. I can install pip manually, but according to the tutorials, it shouldn’t be necessary. What’s more, though people say you can find something like pip.py or pip3.py in the Python36/scripts folder, there is nothing in the scripts folder for me. Is it unusual? 寄件者: MRAB 傳送時間: 2017年8月30日 上午 05:07 收件者: [email protected] 主旨: Re: pip which should be installed with Python3.62 doesn't work inwindows On 2017-08-29 20:10, Bear Light wrote: > I found many rookies like me asking similar question: why "pip" is not > recognized by cmd? > I install Python 3.62 64bit in Win10 and I'm sure I add PATH, the Python > tutorial doesn't even mention about PATH. > You can see the following picture showing the tutorial doesn't work. This newsgroup is text-only. > Please give a hand to the helpless rookies and me. > > [image: 內置圖片 1] > tutorials url: > https://packaging.python.org/tutorials/installing-packages/#creating-virtual-environments > The best way is to use the Python "launcher" py.exe, which should be on the search path, so instead of, say: pip install regex you would write: py -3.6 -m pip install regex This ensures that it's installed into the correct version of Python (in this case v3.6). -- https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
