[Tutor] smtplib help required
Hi there gurus and everyone else. This is my first post to this group, and I'm turning here because I'm stumped and search engines are not helping. I've used smtplib for a few things already and while wanting to use it again today, I'm having weird things happen. Basically, my code looks like this: #!/usr/bin/env python import smtplib pass and when I run it, I get this: srv1:~/python# ./email.py Traceback (most recent call last): File "./email.py", line 3, in import smtplib File "/usr/local/lib/python2.6/smtplib.py", line 46, in import email.utils ImportError: No module named utils And I can also confirm that email.utils exists srv1:~/python# file /usr/local/lib/python2.6/email/utils.py /usr/local/lib/python2.6/email/utils.py: ASCII Java program text So can anyone steer me in the right direction? Oh, this first happened with the Debian distro release Python2.5, I then downloaded and compiled 2.6.5 and it's still happening. I'm very lost and would appreciate any help on the matter. -- James ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] smtplib help required
D'Oh! Of course! I feel like a right pillock now. Cheers for that though. -- James At Saturday, 24-04-2010 on 0:39 Jerry Hill wrote: On Fri, Apr 23, 2010 at 6:41 PM, James Chapman wrote: > Hi there gurus and everyone else. This is my first post to this group, and > I'm turning here because I'm stumped and search engines are not helping. > I've used smtplib for a few things already and while wanting to use it again > today, I'm having weird things happen. > > srv1:~/python# ./email.py > Traceback (most recent call last): > File "./email.py", line 3, in > import smtplib > File "/usr/local/lib/python2.6/smtplib.py", line 46, in > import email.utils > ImportError: No module named utils > You've named your script email.py, which is shadowing the email package from the standard library. -- Jerry ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] python telnet
traceback has:
child = winspawn('telnet 192.168.0.55:210')
When using telnet from CLI (on windows), you would type:
telnet 192.168.0.55 210
Note the space between the IP and port number and not a :colon.
Not sure this is your problem but probably worth mentioning.
--
James
At Saturday, 19/11/2011 on 11:44 Rayon wrote:
> I used the turn on turn off feature, and I can start a telnet session from
> the command line.
>
> -Original Message-
> From: tutor-bounces+evosweet=hotmail@python.org
> [mailto:tutor-bounces+evosweet=hotmail@python.org] On Behalf Of Steven
> D'Aprano
> Sent: 19 November 2011 07:04
> To: tutor@python.org
> Subject: Re: [Tutor] python telnet
>
> Rayon wrote:
> > I installed the telnet client but still the same error.
>
> How did you install it? Windows includes a telnet application, but it has to
> be enabled first:
>
> http://windows.microsoft.com/en-AU/windows-vista/Telnet-frequently-asked-que
> stions
>
> If you run "telnet" from the Windows shell (cmd.exe), what happens?
>
> Until you can successfully run telnet from Windows, there isn't any point
> trying to run it from Python.
>
>
> --
> Steven
>
> ___
> Tutor maillist - Tutor@python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/mailman/listinfo/tutor
>
> ___
> Tutor maillist - Tutor@python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/mailman/listinfo/tutor
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
[Tutor] Opening filenames with unicode characters
Hi there python list.
I'm trying to open a text file named "This is_a-test'FILE to Ensure$
that£ stuff^ works.txt" (without the quotes) but I'm struggling to
find a way to open it.
>>> filename = "This is_a-test'FILE to Ensure$ that£ stuff^
works.txt.js"
>>> open(filename)
Traceback (most recent call last):
File "", line 1, in
IOError: [Errno 2] No such file or directory: "This is_a-test'FILE to
Ensure$ that\x9c stuff^ works.txt.js"
>>> import codecs
>>> codecs.open(filename, 'r', 'utf-8')
Traceback (most recent call last):
File "", line 1, in
File "C:\Python27\lib\codecs.py", line 881, in open
file = __builtin__.open(filename, mode, buffering)
IOError: [Errno 2] No such file or directory: "This is_a-test'FILE to
Ensure$ that\x9c stuff^ works.txt.js"
>>> os.listdir(".")
["This is_a-test'FILE to Ensure$ that\xa3 stuff^ works.txt.js"]
>>> filename.decode()
Traceback (most recent call last):
File "", line 1, in
UnicodeDecodeError: 'ascii' codec can't decode byte 0x9c in position
35: ordinal not in range(128)
I've tried all sorts of encode and decode methods on the string
containing the file name but nothing seems to be working.
Any help would be appreciated.
James
PS: This is on 64bit Windows with ActivePython 2.7.3
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Opening filenames with unicode characters
Thanks Tim, while this works, I need the name to be stored in a variable as it's dynamic. In other words, how do I rewrite open(u"blah£.txt") to be filename = "blah£.txt" open(filename) At Thursday, 28/06/2012 on 18:39 Tim Golden wrote: On 28/06/2012 18:19, James Chapman wrote: > Hi there python list. > > I'm trying to open a text file named "This is_a-test'FILE to Ensure$ > that£ stuff^ works.txt" (without the quotes) but I'm struggling to > find a way to open it. Happily, you're using Windows, which makes this very much easier. Short Explanation Do this: open (u"blah £3.50.txt").read () # note the u- prefix Long Explanation: There's way too many levels of indirection between you and the disk to go into it all, but basically you're telling Python to create a (byte) string from those characters using its default encoding for code, which is UTF-8. UTF-8 maps "£" to the one-byte 0x9c. Somewhere in the layers between that string and the representation on disk, a mismatch occurs. TJG ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Opening filenames with unicode characters
Why can I not convert my existing byte string into a unicode string? In the mean time I'll create my original string as unicode and see if that solves my problem. >>> fileName = unicode(filename) Traceback (most recent call last): File "", line 1, in UnicodeDecodeError: 'utf8' codec can't decode byte 0x9c in position 35: invalid start byte -- James At Thursday, 28/06/2012 on 18:58 Jerry Hill wrote: On Thu, Jun 28, 2012 at 1:55 PM, James Chapman wrote: > Thanks Tim, while this works, I need the name to be stored in a variable as > it's dynamic. > > In other words, how do I rewrite > open(u"blah£.txt") > > to be > filename = "blah£.txt" > open(filename) You have to create a unicode-string, not a byte-string. So, like this: filename = u"blah£.txt" open(filename) -- Jerry ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Opening filenames with unicode characters
Informative thanks Jerry, however I'm not out of the woods yet. > Here's a couple of questions that you'll need to answer 'Yes' to > before you're going to get this to work reliably: > > Are you familiar with the differences between byte strings and unicode > strings? I think so, although I'm probably missing key bits of information. > Do you understand how to convert from one to the other, > using a particular encoding? No not really. This is something that's still very new to me. > Do you know what encoding your source > file is saved in? The name of the file I'm trying to open comes from a UTF-16 encoded text file, I'm then using regex to extract the string (filename) I need to open. However, all the examples I've been using here are just typed into the python console, meaning string source at this stage is largely irrelevant. > If your string is not coming from a source file, > but some other source of bytes, do you know what encoding those bytes > are using? > > Try the following. Before trying to convert filename to unicode, do a > "print repr(filename)". That will show you the byte string, along > with the numeric codes for the non-ascii parts. Then convert those > bytes to a unicode object using the appropriate encoding. If the > bytes are utf-8, then you'd do something like this: > unicode_filename = unicode(filename, 'utf-8') >>> print(repr(filename)) "This is_a-test'FILE to Ensure$ that\x9c stuff^ works.txt.js" >>> fileName = unicode(filename, 'utf-8') Traceback (most recent call last): File "", line 1, in UnicodeDecodeError: 'utf8' codec can't decode byte 0x9c in position 35: invalid start byte >>> fileName = unicode(filename, 'utf-16') >>> fileName u'\u6854\u7369\u6920\u5f73\u2d61\u6574\u7473\u4627\u4c49\u2045\u6f74\u4520\u736e\u7275\u2465\u7420\u6168\u9c74\u7320\u7574\u\u205e\u6f77\u6b72\u2e73\u7874\u2e74\u736a' So I now have a UTF-16 encoded string, but I still can't open it. >>> codecs.open(fileName, 'r', 'utf-16') Traceback (most recent call last): File "", line 1, in File "C:\Python27\lib\codecs.py", line 881, in open file = __builtin__.open(filename, mode, buffering) IOError: [Errno 2] No such file or directory: u'\u6854\u7369\u6920\u5f73\u2d61\u6574\u7473\u4627\u4c49\u2045\u6f74\u4520\u736e\u72 75\u2465\u7420\u6168\u9c74\u7320\u7574\u\u205e\u6f77\u6b72\u2e73\u7874\u2e74\u736a' I presume I need to perform some kind of decode operation on it to open the file but then am I not basically going back to my starting point? Apologies if I'm missing the obvious. -- James ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Opening filenames with unicode characters
Thanks to everyone who responded on this thread, your time is greatly
appreciated.
It appears however that my problem is related to the environment. I
sent my original email right before leaving work and have since been
working on a physical machine without any problems. I've copied some
of that code to my remote virtual machine where I'm doing the dev work
and the same example that works on my physical win7 machine fails on
my virtual win 2008 machine. Win2008 host platform is Linux with
VirtualBox.
The only remaining question is whether this is a one off issue,
whether it's related to the virtual machine or whether it's related to
Windows 2008. I guess I'll find out tomorrow.
Oh and Tim, you'll be happy to know that regex does not affect the
string in this case. Well, at least not the way I'm using it to
extract data.
--
James
At Thursday, 28/06/2012 on 21:17 Tim Golden wrote:
On 28/06/2012 20:48, James Chapman wrote:
> The name of the file I'm trying to open comes from a UTF-16 encoded
> text file, I'm then using regex to extract the string (filename) I
> need to open.
OK. Let's focus on that. For the moment -- although it might
well be very relevant -- I'm going to ignore the regex side
of things. It's always trying to portray things like this
because there's such confusion between what characters I
write to represent the data and the data represented by those
characters themselves!
OK, let's adopt a convention whereby I represent the data as
they kind of thing you'd see in a hex editor. This obviously
isn't how it appear in a a text file but hopefully it'll be
clear what's going on.
I have a filename £10.txt -- that is the characters:
POUND SIGN
DIGIT ONE
DIGIT ZERO
FULL STOP
LATIN SMALL LETTER T
LATIN SMALL LETTER X
LATIN SMALL LETTER T
I have -- prior to your getting there -- placed this in a text
file which I guarantee is UTF16-encoded. For the purposes of
illustration I shall do that in Python code here:
with open ("filedata.dat", "wb") as f:
f.write (u"£10.txt".encode ("utf16"))
The file is named "filedata.dat" and looks like this (per our
convention):
ff fe a3 00 31 00 30 00 2e 00 74 00 78 00 74 00
I now want to read the contents of the that file as a
filename and open the file in question. Here goes:
#
# Open the file and extract the data as a set of
# bytes into a Python (byte) string.
#
with open("filedata.dat", "rb") as f:
data = f.read()
#
# Convert the data into a unicode object by decoding
# the UTF16 bytes
#
filename = data.decode("utf16")
# filename is now a unicode object which, depending on
# what your console offers, will either display as
# £10.txt or as \xa310.txt or as something else.
#
# Open that file by passing the unicode object directly
# to Python's file-opening mechanism
#
ten_pound_txt = open (filename, "rb")
print ten_pound_txt.read () # whatever
ten_pound_txt.close ()
I don't know if that makes anything clearer for you, but at
least it gives you something to try out.
The business with the regex clouds the issue: regex can play
a little awkwardly with Unicode, so you'd have to show some
code if you need help there.
TJG
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] reading an input stream
Hi Richard There are a number of considerations you need to take into account here. Raw sockets is almost never the right solution, while a basic socket to socket connection is easy enough to program, handling failure and concurrency can very quickly make the solution a lot more complex than it needs to be, so perhaps you could supply more information? (I realise I'm venturing outside the realm of learning python, but I'm a pedant for doing things right). You said you need to read XML in from a socket connection. You've not mentioned what's generating the data? Is that data sent over HTTP in which case is this part of a SOAP or REST API? Is the data being generated by something you've written or a 3rd party software package? Is REST an option? Is there a reason to serialise to XML? (If I was performing the serialisation I would go with JSON if being human readable was a requirement. ) If the method of receiving that data is optional, have you considered using something like AMQP (RabbitMQ) which would eliminate your need to support concurrency? It would also handle failure well. James -- James On 29 December 2015 at 20:14, richard kappler wrote: > Sorry it took so long to respond, just getting back from the holidays. You > all have given me much to think about. I've read all the messages through > once, now I need to go trough them again and try to apply the ideas. I'll > be posting other questions as I run into problems. BTW, Danny, best > explanation of generators I've heard, well done and thank you. > > regards, Richard > > On Thu, Dec 24, 2015 at 4:54 PM, Danny Yoo wrote: > > > > I think what I need to do would be analogous to (pardon if I'm using > the > > > wrong terminology, at this poing in the discussion I am officially out > of > > > my depth) sending the input stream to a buffer(s) until the ETX for > that > > > message comes in, shoot the buffer contents to the parser while > accepting > > > the next STX + message fragment into the buffer, or something > analogous. > > > > Yes, I agree. It sounds like you have one process read the socket and > > collect chunks of bytes delimited by the STX markers. It can then > > send those chunks to the XML parser. > > > > > > We can imagine one process that reads the socket and spits out a list > > of byte chunks: > > > > chunks = readDelimitedChunks(socket) > > > > and another process that parses those chunks and does something with > them: > > > > for chunk in chunks: > > > > > > > > It would be nice if we could organize the program like this. But one > > problem is that chunks might not be finite! The socket might keep on > > returning bytes. If it keeps returning bytes, we can't possibly > > return a finite list of the chunked bytes. > > > > > > What we really want is something like: > > > > chunkStream = readDelimitedChunks(socket) > > for chunk in chunkStream: > > > > > > where chunkStream is itself like a socket: it should be something that > > we can repeatedly read from as if it were potentially infinite. > > > > > > We can actually do this, and it isn't too bad. There's a mechanism in > > Python called a generator that allows us to write function-like things > > that consume streams of input and produce streams of output. Here's a > > brief introduction to them. > > > > For example, here's a generator that knows how to produce an infinite > > stream of numbers: > > > > ## > > def nums(): > > n = 0 > > while True: > > yield n > > n += 1 > > ## > > > > What distinguishes a generator from a regular function? The use of > > "yield". A "yield" is like a return, but rather than completely > > escape out of the function with the return value, this generator will > > remember what it was doing at that time. Why? Because it can > > *resume* itself when we try to get another value out of the generator. > > > > Let's try it out: > > > > # > > > > >>> numStream = nums() > > >>> numStream.next() > > 0 > > >>> numStream.next() > > 1 > > >>> numStream.next() > > 2 > > >>> numStream.next() > > 3 > > >>> numStream.next() > > 4 > > # > > > > Every next() we call on a generator will restart it from where it left > > off, until it reaches its next "yield". That's how we get this > > generator to return an infinite sequence of things. > > > > > > That's how we produce infinite sequences. And we can write another > > generator that knows how to take a stream of numbers, and square each > > one. > > > > > > def squaring(stream): > > for n in stream: > > yield n > > > > > > > > Let's try it. > > > > > > > > > > >>> numStream = nums() > > >>> squaredNums = squaring(numStream) > > >>> squaredNums.next() > > 0 > > >>> squaredNums.next() > > 1 > > >>> squaredNums.next() > > 4 > > >>> squaredNums.next() > > 9 > > >>> squaredNums.next() >
Re: [Tutor] Question about the memory manager
If you read the comment that goes with the code snippet pasted in the original email it makes far more sense as the author is talking specifically about out of memory errors... "You already got excellent answers, I just wanted to add one more tip that's served me well over the years in a variety of language for the specific problem "how to cleanly diagnose, log, etc, out of memory errors?". Problem is, if your code gets control before enough objects have been destroyed and their memory recycled, memory might be too tight to do propert logging, gui work, etc, etc -- how do we ensure this doesn't happen? Answer: build an emergency stash so you know you can spend it in such emergencies: rainydayfund = [[] for x in xrange(16*1024)] # or however much you need def handle_exception(e): global rainydayfund del rainydayfund ... etc, etc ... " - Alex Martelli -- James ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Modularity
May I suggest: https://docs.python.org/2/tutorial/modules.html In particular: * https://docs.python.org/2/tutorial/modules.html#the-module-search-path * https://docs.python.org/2/tutorial/modules.html#packages Now the next bit of advice is likely to be controversial but I have good reasons for it. I like to import the top level module and use the full namespace in the code, that way, when I come back to it in 3 years time I know where each function call is coming from. For example, lets say I had the following package (stolen from the docs page linked above): sound/ Top-level package __init__.py Initialize the sound package formats/ Subpackage for file format conversions __init__.py wavread.py wavwrite.py aiffread.py aiffwrite.py auread.py auwrite.py ... effects/ Subpackage for sound effects __init__.py echo.py surround.py reverse.py ... filters/ Subpackage for filters __init__.py equalizer.py vocoder.py karaoke.py I myself would import as follows import sound Then in my code, the calls would look like: wave = sound.formats.waveread(someFile) aiffFile = sound.formats.aiffwrite(wave) auFile = sound.formats.auwrite(wave) If I did: from sound import formats.* Then the code would be wave = waveread(someFile) aiffFile = aiffwrite(wave) auFile = auwrite(wave) The problem with the latter is, which module supplies waveread? I have roughly 20 import statements where I'm importing something.* which one of those modules supplies the function waveread? A way around the above would be import sound.formats.waveread import sound.formats.aiffwrite import sound.formats.auwrite Code would then be wave = waveread(someFile) aiffFile = aiffwrite(wave) auFile = auwrite(wave) But what if we have various modules that implement a waveread function? Then we'd have to start using import sound.formats.waveread as sfwaveread import some.other.waveread Code is read more than it is written, so don't be lazy! Use the namespaces in your code. You make it clear what you're doing it you avoid clashing. Finally, if you absolutely must be lazy, then import like this: from sound import formats as sf wave = sf.waveread(someFile) aiffFile = sf.aiffwrite(wave) auFile = sf.auwrite(wave) There is nothing I hate more than being asked to change or fix someone elses code when that programmer is lazy and feels he/she can produce a solution quicker by reducing the amount of typing he/she has to do. Well, that's a lie, there are things I hate more but this come close! James ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Procedure to install dlib on windows?
>From one of the Python examples: # COMPILING/INSTALLING THE DLIB PYTHON INTERFACE # You can install dlib using the command: # pip install dlib # # Alternatively, if you want to compile dlib yourself then go into the dlib # root folder and run: # python setup.py install # or # python setup.py install --yes USE_AVX_INSTRUCTIONS It looks like the pip installer will install a pre-compiled lib with python API bindings for you. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Modularity
I should have re-read that last reply before hitting send. Apologies for the poor sentence construction! Something I forgot to highlight before which might be related to your initial question. If you have a file called sound.py which contained a class called WavFile, if you imported just sound like this: import sound Then your class constructor would be called like this: wavFile = sound.WavFile() Importing the module doesn't import the class, for that you'd do from sound import WavFile wavFile = WavFile() ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] Mocking with "mock" in unit testing
Hi all
I have a question regarding mocking in unit testing.
Let's assume I have the following class:
---
import subprocess
class Pinger(object):
def ping_host(self, host_to_ping):
cmd_string = 'ping %s' % (host_to_ping)
cmd_args = cmd_string.split()
proc = subprocess.Popen(cmd_args, shell=True)
proc.wait()
if proc.returncode != 1:
raise Exception('Error code was: %d' % (proc.returncode))
---
In my unittest I don't want to run the ping command, (It might not be
available on the build system) I merely want to check that a call to
subprocess.Popen is made and that the parameters are what I expect?
So far I have this, but it doesn't work and I suspect it's way off!!
---
import mock
import unittest
from tutor_q import Pinger
class Test_Pinger(unittest.TestCase):
def test_ping_host(self):
pinger = Pinger()
assert pinger
subprocess = mock.Mock()
subprocess.Popen.return_value = 0
subprocess.assert_called_once_with(['ping','localhost'])
pinger.ping_host('127.0.0.1')
if __name__ == '__main__':
unittest.main()
---
Can anyone point me in the right direction on how to mock up these
subprocess calls?
Thanks
James
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Mocking with "mock" in unit testing
Thanks eryksun.
There's a list for testing in python which I also posed the question to and
got pretty much the same answer as you provided.
The line if proc.returncode != 1 was a mistake. That 1 should have been a
zero.
As this question was just about mock and not really dealing with the bad
return code or exception handling or raising my final working example looks
like this:
pinger.py
import subprocess
class Pinger(object):
def ping_host(self, host_to_ping):
cmd_string = 'ping %s' % (host_to_ping)
cmd_args = cmd_string.split()
proc = subprocess.Popen(cmd_args, shell=True)
proc.wait()
if proc.returncode != 0:
raise Exception('Error code was: %d' % (proc.returncode))
if __name__ == '__main__':
PINGER = Pinger()
PINGER.ping_host('localhost')
test_pinger.py
import mockimport unittestimport pinger
class Test_Pinger(unittest.TestCase):
def test_ping_host_succeeds(self):
pinger = pinger.Pinger()
with mock.patch("pinger.subprocess") as subprocess:
subprocess.Popen.return_value.returncode = 0
pinger.ping_host('localhost')
subprocess.Popen.assert_called_once_with(['ping','localhost'],
shell=True)
def test_ping_host_fails_and_throws_exception(self):
pinger = pinger.Pinger()
with mock.patch('pinger.subprocess') as subprocess:
subprocess.Popen.return_value.returncode = 1
self.assertRaises(Exception, pinger.ping_host, 'localhost')
if __name__ == '__main__':
unittest.main()
--------
--
James
On 17 January 2014 01:05, eryksun wrote:
> On Thu, Jan 16, 2014 at 5:32 AM, James Chapman
> wrote:
> >
> > In my unittest I don't want to run the ping command, (It might not be
> > available on the build system) I merely want to check that a call to
> > subprocess.Popen is made and that the parameters are what I expect?
>
> You can mock `Popen` where it's accessed.
>
> @mock.patch('subprocess.Popen')
> def test_ping_host(self, Popen):
> Popen.return_value.returncode = 0
> pinger = Pinger()
> pinger.ping_host('127.0.0.1')
> Popen.assert_called_once_with(['ping','127.0.0.1'], shell=True)
>
> If the tutor_q module imported `Popen` into its namespace, then you'd
> patch tutor_q.Popen.
>
> > def ping_host(self, host_to_ping):
> > cmd_string = 'ping %s' % (host_to_ping)
> > cmd_args = cmd_string.split()
>
> Splitting on spaces doesn't work generally. Use `shlex.split`, or
> build the list manually.
>
> > proc = subprocess.Popen(cmd_args, shell=True)
>
> Maybe you really need the shell to process your command, but generally
> there's no reason to run the shell just to have it execute the command
> and wait. Plus it opens the door to security exploits.
>
> > proc.wait()
> > if proc.returncode != 1:
> > raise Exception('Error code was: %d' % (proc.returncode))
>
> A non-zero return code signals an error. When using `Popen` directly,
> you can be consistent with `check_call` and `check_output` if you
> raise a `CalledProcessError` in this case:
>
> retcode = proc.wait()
> if retcode:
> raise subprocess.CalledProcessError(retcode, cmd_args)
>
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Mocking with "mock" in unit testing
Erm...?
CPython yeah.
If I rename "pinger.py" to "tutor.py" and change the unittest it runs fine.
Why?
-
import mock
import unittest
import tutor
class Test_Pinger(unittest.TestCase):
def test_ping_host_succeeds(self):
pinger = tutor.Pinger()
with mock.patch("tutor.subprocess") as subprocess:
subprocess.Popen.return_value.returncode = 0
pinger.ping_host('localhost')
subprocess.Popen.assert_called_once_with(['ping','localhost'],
shell=True)
def test_ping_host_fails_and_throws_exception(self):
pinger = tutor.Pinger()
with mock.patch('tutor.subprocess') as subprocess:
subprocess.Popen.return_value.returncode = 1
self.assertRaises(Exception, pinger.ping_host, 'localhost')
if __name__ == '__main__':
unittest.main()
-
--
James
On 17 January 2014 10:50, eryksun wrote:
> On Fri, Jan 17, 2014 at 4:58 AM, James Chapman
> wrote:
> > import mock
> > import unittest
> > import pinger
> >
> > class Test_Pinger(unittest.TestCase):
> >
> > def test_ping_host_succeeds(self):
> > pinger = pinger.Pinger()
>
> Are you using CPython? That raises an UnboundLocalError. Take a look
> at the CPython bytecode:
>
> def test(self):
> pinger = pinger.Pinger()
>
> >>> dis.dis(test)
> 2 0 LOAD_FAST1 (pinger)
> 3 LOAD_ATTR0 (Pinger)
> 6 CALL_FUNCTION0
> 9 STORE_FAST 1 (pinger)
> 12 LOAD_CONST 0 (None)
> 15 RETURN_VALUE
>
> Notice LOAD_FAST(1), where fast local 1 is the local variable
> `pinger`. The value isn't assigned yet, and LOAD_FAST doesn't search
> globals and builtins. You need a unique name for the instance, such as
> `test_pinger`. Then the compiler knows to use LOAD_GLOBAL:
>
> def test(self):
> test_pinger = pinger.Pinger()
>
> >>> dis.dis(test)
> 2 0 LOAD_GLOBAL 0 (pinger)
> 3 LOAD_ATTR1 (Pinger)
> 6 CALL_FUNCTION0
> 9 STORE_FAST 1 (test_pinger)
> 12 LOAD_CONST 0 (None)
> 15 RETURN_VALUE
>
> Here's a version using a decorator instead:
>
> @mock.patch('pinger.subprocess')
> def test_ping_host_succeeds(self, subprocess):
> subprocess.Popen.return_value.returncode = 0
> test_pinger = pinger.Pinger()
> test_pinger.ping_host('localhost')
> subprocess.Popen.assert_called_once_with(['ping','localhost'],
> shell=True)
>
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Mocking with "mock" in unit testing
Ah of course!
pinger = pinger.Pinger() I should have noticed that myself. (I
renamed the file to pinger.py after establishing it all worked and then
didn't re-run).
ping = pinger.Pinger() works fine.
As for the syntax errors, those will be copy paste / different email client
errors.
--
James
On 17 January 2014 11:23, Steven D'Aprano wrote:
> On Fri, Jan 17, 2014 at 09:58:06AM +, James Chapman wrote:
>
> > As this question was just about mock and not really dealing with the bad
> > return code or exception handling or raising my final working example
> looks
> > like this:
>
> Your final *working* example? I don't think so. I can see at least two
> syntax errors, and a programming error.
>
> A word to the wise: code isn't working until you've actually seen it
> work :-)
>
>
>
> > pinger.py
> >
> >
> >
> > import subprocess
> > class Pinger(object):
>
> There is your first SyntaxError, a stray space ahead of the "class"
> keyword.
>
>
> > def ping_host(self, host_to_ping):
> > cmd_string = 'ping %s' % (host_to_ping)
> > cmd_args = cmd_string.split()
>
> This is not a programming error, but it is wasteful. First you join two
> strings: "ping", and the host. Then, having glued them together, you
> split them apart exactly where you glued them.
>
> Better to do something like:
>
> cmd_args = ["ping", host_to_ping]
>
> assuming that you know that host_to_ping is only a single word.
>
>
> > proc = subprocess.Popen(cmd_args, shell=True)
>
> Why shell=True?
>
> The documentation for subprocess.Popen says:
>
> The shell argument (which defaults to False) specifies whether to
> use the shell as the program to execute. If shell is True, it is
> recommended to pass args as a string rather than as a sequence.
>
> and also warns that shell=True can be a security hazard. Do you have a
> good reason for using it?
>
> http://docs.python.org/2/library/subprocess.html
>
>
>
> > proc.wait()
> > if proc.returncode != 0:
> > raise Exception('Error code was: %d' % (proc.returncode))
> >
> > if __name__ == '__main__':
> > PINGER = Pinger()
> > PINGER.ping_host('localhost')
> >
> >
> >
> >
> >
> > test_pinger.py
> >
> >
> >
> > import mockimport unittestimport pinger
> > class Test_Pinger(unittest.TestCase):
>
> And here you have two more SyntaxErrors: missing commas between
> arguments to import, and a stray space before the "class" again.
>
>
> > def test_ping_host_succeeds(self):
> > pinger = pinger.Pinger()
>
> And here is your programming error. Unlike some programming languages
> (Lua, I think) in Python you cannot use the same name to refer to both a
> global variable and a local variable inside the same function. In
> Python, either the variable is treated as a global, or as a local, but
> not both.
>
> The rules Python uses to decide are:
>
> * if you declare a name "global" inside the function, then it is
> treated as global inside that function;
>
> * otherwise, if you assign a value to the name *anywhere* inside
> the function, it is treated as local;
>
> * otherwise it is global.
>
> So in your example, you assign a value to the name "pinger", so it is
> treated as local. The offending line of code looks like this:
>
> pinger = pinger.Pinger()
>
> Your *intention* is to look up the global "pinger", which is a module,
> then create a pinger.Pinger() instance, then assign it to the local
> variable "pinger". But that's not what Python does. Instead, it tries to
> look up a local variable "pinger", finds it doesn't have one, and raises
> an UnboundLocalError exception.
>
> I recommend you change the name of the local variable "pinger" to
> something else, so it no longer clashes with the "pinger" module name.
>
>
> Hope this is of help to you,
>
>
>
> --
> Steven
> ___
> Tutor maillist - Tutor@python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Mocking with "mock" in unit testing
Really! import mock import unittest import pinger It should be three lines, but somehow it got all messed up, either through rich text formatting or copy paste. Being a bit pedantic now about import statements which are clearly unintentionally messed up. - Sent in plain text. -- James On 17 January 2014 15:32, Steven D'Aprano wrote: > > On Fri, Jan 17, 2014 at 08:35:04AM -0500, eryksun wrote: > > On Fri, Jan 17, 2014 at 6:23 AM, Steven D'Aprano > > wrote: > > > On Fri, Jan 17, 2014 at 09:58:06AM +, James Chapman wrote: > [...] > > >> import mockimport unittestimport pinger > > >> class Test_Pinger(unittest.TestCase): > > > > > > And here you have two more SyntaxErrors: missing commas between > > > arguments to import, and a stray space before the "class" again. > > > > This is also fine in the rich text version. BTW, the botched plain > > text conversion is missing line breaks, not commas. > > I don't understand this. If I'm interpreting you correctly, > > > import mockimport > unittestimport > pinger > > > would not be "fine". > > > -- > Steven > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] Unit testing infinite loops
Hello tutors
I've constructed an example which shows a problem I'm having testing a real
world program and would like to run it past you.
tutor_question.py
--
# -*- coding: utf-8 -*-
import sys
import threading
import time
class Time_Printer(threading.Thread):
def run(self):
for i in range(60):
print('%s - %s' % (self, time.ctime(time.time(
time.sleep(1)
class Infinite_Loop_Tutor_Question(object):
def start_A(self):
thread = Time_Printer()
thread.daemon = True
thread.start()
print('Started %s' % (thread))
def start_B(self):
thread = Time_Printer()
thread.daemon = True
thread.start()
print('Started %s' % (thread))
def run_forever(self):
self.start_A()
time.sleep(0.5)
self.start_B()
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
print("Caught Keyboard Interrupt...")
sys.exit(0)
if __name__ == '__main__':
infinite_loop = Infinite_Loop_Tutor_Question()
infinite_loop.run_forever()
--
In my example above, testing the everything but the run_forever method is
trivial.
So on to my question... The run_forever method essentially just fires up a
bunch of threads to serve various purposes and then waits for CTRL-C to
terminate the entire program. Testing this at the moment is very difficult
because the unit test ends up in the infinite loop. So, would a better idea
be to create an attribute, set it to True and then do
try:
while self.attribute:
time.sleep(1)
except KeyboardInterrupt:
...
My unit test could then set the attribute. However I'd still have the
problem of how I get from the unit test line that fires up the method to
the next line to change the attribute.
So how should the run_forever method be written so that it's testable, or
if it's testable as is, how would I test it?
And please, no comments about syntax, clean exits of threads, thread
communication, resources, or even the need for testing the run_forever
method. In my test I want to test that it makes the relevant calls and then
enters the infinite loop at which point I want to terminate it.
Thanks in advance, and hopefully there are no formatting issues this time.
--
James
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Unit testing infinite loops
Hmm...
Here is an example of how I'm currently trying to test it:
test_tutor_question.py
-
# -*- coding: utf-8 -*-
import unittest
import mock
from tutor_question import Infinite_Loop_Tutor_Question
class Test_Infinite_Loop_Tutor_Question(unittest.TestCase):
def test_run_forever(self):
with mock.patch('tutor_question.Infinite_Loop_Tutor_Question.start_A')
as start_A:
with
mock.patch('tutor_question.Infinite_Loop_Tutor_Question.start_B') as
start_B:
inf_loop = Infinite_Loop_Tutor_Question()
print start_A.call_count
print start_B.call_count
inf_loop.run_forever()
inf_loop.interrupt_main()
print start_A.call_count
print start_B.call_count
if __name__ == "__main__":
unittest.main()
-
As you can see if you run this, the test doesn't reach the lines below
inf_loop.run_forever().
So ideally, I'd need a way of injecting a keyboard interrupt into the
method, I could then check that the exception was handled and that the
start_A and start_B calls were made.
** Obviously the print lines will be substituted for some kind of
assert lines **
FYI I'm using CPython 2.7.
--
James
On 31 January 2014 12:57, eryksun wrote:
>
> On Fri, Jan 31, 2014 at 6:31 AM, James Chapman wrote:
> > try:
> > while self.attribute:
> > time.sleep(1)
> > except KeyboardInterrupt:
> > ...
> >
> > My unit test could then set the attribute. However I'd still have the
> > problem of how I get from the unit test line that fires up the method to the
> > next line to change the attribute.
>
> You could add a method that toggles the attribute, and use a
> threading.Timer to run it after a set interval.
>
> > if it's testable as is, how would I test it?
>
> CPython 2.3+ can interrupt the main thread from another thread using
> the built-in function `_thread.interrupt_main`:
>
> http://docs.python.org/3/library/_thread#_thread.interrupt_main
>
> >>> import _thread
> >>> _thread.interrupt_main()
> Traceback (most recent call last):
> File "", line 1, in
> KeyboardInterrupt
>
> It's also implemented in PyPy, but not in Jython.
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Unit testing infinite loops
Thanks Steven!
You've raised a few valid points, mostly that the run_forever method
should be broken up. I like the principle of a method doing just one
thing and for whatever reason I didn't apply that thinking to this
method as it's the master loop (even though it does nothing). So for
starters I'll fix that. Breaking everything up makes testing easier,
which in turn makes development easier.
On the note of what doesn't need a test...
The question of coverage always comes up when unit testing is
mentioned and I read an interesting blog article once about it. It
basically said: Assume you have 85% coverage on a program that
consists of 1,000 lines. That's 150 lines which are not tested. If
those lines are print lines, sleep lines, getters etc it's not really
a problem. But what happens when you scale that up. 1,000,000 lines of
code lets say (not unheard of, although in python that would be out of
this world big). You now end up with 150,000 lines of untested code.
While the percentage of code covered is high, there is _a_lot_ of code
there that isn't tested and a lot of room for mistakes to creep in. A
mistake on one of those 150,000 lines could break the build and
possibly cost you hours or even days tracking it down. If those lines
were tested however, your continuous integration build system would
hopefully highlight the fault.
In my experience testing works, saves time down the line, and makes
code easier to come back to.
--
James
On 31 January 2014 13:21, Steven D'Aprano wrote:
> On Fri, Jan 31, 2014 at 11:31:49AM +, James Chapman wrote:
>> Hello tutors
>>
>> I've constructed an example which shows a problem I'm having testing a real
>> world program and would like to run it past you.
> [...]
>> class Infinite_Loop_Tutor_Question(object):
>> def run_forever(self):
>> self.start_A()
>> time.sleep(0.5)
>> self.start_B()
>> try:
>> while True:
>> time.sleep(1)
>> except KeyboardInterrupt:
>> print("Caught Keyboard Interrupt...")
>> sys.exit(0)
> [...]
>> In my example above, testing the everything but the run_forever method is
>> trivial.
>>
>> So on to my question... The run_forever method essentially just fires up a
>> bunch of threads to serve various purposes and then waits for CTRL-C to
>> terminate the entire program. Testing this at the moment is very difficult
>> because the unit test ends up in the infinite loop. So, would a better idea
>> be to create an attribute, set it to True and then do
>>
>> try:
>> while self.attribute:
>> time.sleep(1)
>> except KeyboardInterrupt:
>> ...
>
>
> That probably won't hurt.
>
>
>> My unit test could then set the attribute. However I'd still have the
>> problem of how I get from the unit test line that fires up the method to
>> the next line to change the attribute.
>>
>> So how should the run_forever method be written so that it's testable, or
>> if it's testable as is, how would I test it?
>
> What are you trying to test? You don't just "test" a method, you test
> *something specific* about the method. So what specifically are you
> trying to test?
>
>
>> And please, no comments about syntax, clean exits of threads, thread
>> communication, resources, or even the need for testing the run_forever
>> method. In my test I want to test that it makes the relevant calls and then
>> enters the infinite loop at which point I want to terminate it.
>
> Ah, you see, now you have a problem. Consider this function:
>
> def long_calc():
> time.sleep(60*60*24*365)
> return 1
>
>
> How do I test that the function returns 1? As given, I can't
> really, not unless I wait a whole year for the sleep() to return. So
> what I can do is split the function into two pieces:
>
> def _sleep_a_year():
> time.sleep(60*60*24*365)
>
> def _do_calculation():
> return 1
>
> def long_calc():
> _sleep_a_year()
> return _do_calculation()
>
>
> Now I can unit-test the _do_calculation function, and long_calc() is now
> simple enough that I don't really need to unit-test it. (Unit testing
> should not be treated as a religion. You test what you can. Any testing
> is better than nothing, and if there are some parts of the program which
> are too hard to test automatically, don't test them automatically.)
>
> Or, I can monkey-patch the time.sleep function. Before running my
> test_long_calc unit-test, I do this:
>
> import time
> time.sleep = lambda n: None
>
&
[Tutor] When to use multiprocessing Managers?
Hello tutors I'm curious about managers and when to use them. For example, I see they offer a Queue() for sharing a Q between processes, but if I create a Q in the parent process and pass it down to child processes, then they can put messages into that Q just fine, and I presume the same thing for other objects available under the managers package. So unless the other process is on a different machine, is there a reason to use a manager? Does anyone have any use case examples or snippets I could look at even? Thanks in advance James ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] When to use multiprocessing Managers?
Thanks for the replies so far... As per the subject line, yes I'm talking about managers and any object a manager is capable of spawning. This is not version specific or code specific, it's more a discussion on when to use a manager and when it's not needed. >From the OReilly link sent by Steven...(http://broadcast.oreilly.com/2009/04/pymotw-multiprocessing-part-2.html) "The Manager is responsible for coordinating shared information state between all of its users. By creating the list through the manager, the list is updated in all processes when anyone modifies it. In addition to lists, dictionaries are also supported." In my case I have multiple threaded processing sending log messages to a Queue that was created by the parent process. The parent process then has a read thread to take messages out of the queue and write them to file. Whether I create the queue object like this: log_Q = multiprocessing.Queue() or whether I create it like this: multimanager = multiprocessing.Manager() log_Q = multimanager.Queue() seems to make no difference. I always get all the messages from all the threads in all the processes. Perhaps the manager would be important if I was writing to a Queue and expecting all threads to see that message? Although if I needed to command a thread to do something I'd probably have a separate class and separate thread for that purpose. James -- James On 26 February 2014 14:19, David Palao wrote: > 2014-02-25 11:52 GMT+01:00 James Chapman : >> Hello tutors >> >> I'm curious about managers and when to use them. >> For example, I see they offer a Queue() for sharing a Q between >> processes, but if I create a Q in the parent process and pass it down >> to child processes, then they can put messages into that Q just fine, >> and I presume the same thing for other objects available under the >> managers package. >> >> So unless the other process is on a different machine, is there a >> reason to use a manager? >> >> Does anyone have any use case examples or snippets I could look at even? >> >> Thanks in advance >> James >> ___ >> Tutor maillist - Tutor@python.org >> To unsubscribe or change subscription options: >> https://mail.python.org/mailman/listinfo/tutor > > Hello, > I asked myself the same question when I started using multiprocessing > time ago. So I was very happy when I saw the question by James. > > From my limited knowledge, I would say that a Manager can be useful > when processes are distributed across different hosts, or if the > exchange of information between processes is more complex than just a > couple of synchronization primitives. > > Best ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Python help
The answer lies in this page: http://docs.python.org/3.3/library/stdtypes.html#string-methods -- James On 28 February 2014 11:44, James Chapman wrote: > The answer lies in this page: > http://docs.python.org/3.3/library/stdtypes.html#string-methods > > > -- > James > > > On 28 February 2014 08:15, Ben Finney wrote: >> Tone Lite writes: >> >>> I am having trouble coming up with a solution to this exercise and any >>> help would be appreciated, thanks! I have attached the code below. >> >> The code you present appears to be the exercise. Are you asking for >> someone to write the solution for you? That isn't what we do here. We'll >> help you, but that doesn't mean we'll do your work for you. >> >> Please show us what you've tried, describe what it should be doing, and >> what is happening instead. If there is an error, please also show the >> complete error output. >> >> -- >> \ “He who allows oppression, shares the crime.” —Erasmus Darwin, | >> `\ grandfather of Charles Darwin | >> _o__) | >> Ben Finney >> >> ___ >> Tutor maillist - Tutor@python.org >> To unsubscribe or change subscription options: >> https://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] When to use multiprocessing Managers?
Thanks for the explanation.
Is there any reason or case you can think of where on a single system
you would use the manager (proxied) queue over the multiprocessing
(piped) queue?
Actually I can probably answer this myself...
The manager could potentially be extended to do some kind of data
validation / error checking before submitting to the Queue. This could
be important if the data going into the Queue was for example, user
generated.
Hmm, yeah I'd say question answered. Thanks eryksun.
--
James
On 1 March 2014 16:48, eryksun wrote:
> On Fri, Feb 28, 2014 at 6:31 AM, James Chapman wrote:
>>
>> log_Q = multiprocessing.Queue()
>
> This is a Queue from multiprocessing.queues. It uses system resources
> (e.g. a semaphore for the queue capacity) that can be shared with
> processes on the same machine.
>
> A value `put` in a queue.Queue is available immediately:
>
> >>> import queue
> >>> q1 = queue.Queue()
> >>> try: q1.put('value'); q1.get_nowait()
> ... except queue.Empty: 'empty'
> ...
> 'value'
>
> On the other hand, a Queue from multiprocessing.queues writes to a
> pipe using a background thread, so there can be a small delay:
>
> >>> import multiprocessing as mp
> >>> q2 = mp.Queue()
> >>> try: q2.put('value'); q2.get_nowait()
> ... except queue.Empty: 'empty'
> ...
> 'empty'
> >>> q2.get_nowait()
> 'value'
>
>> or whether I create it like this:
>>
>> multimanager = multiprocessing.Manager()
>> log_Q = multimanager.Queue()
>
> This is a queue.Queue wrapped by an AutoProxy. For example, its `get`
> method calls _callmethod('get', *args, **kwds), which connects to the
> manager, sends the request, and receives the result.
>
> The docs demonstrate using a manager with remote processes:
>
> http://docs.python.org/3/library/multiprocessing#using-a-remote-manager
>
> You can also proxy the Queue type from multiprocessing.queues. In that
> case, remote processes use a proxy, but local processes can use the
> queue directly.
>
>> Perhaps the manager would be important if I was writing to a Queue and
>> expecting all threads to see that message?
>
> Only 1 thread will `get` the message.
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] c++ on python
Perhaps I should look into Cython as I'm currently working on a
project that utilises a C API.
I've been finding that getting the data types to be exactly what the C
API is expecting to be the hardest part.
With the original question in mind, here's an example calling into a
C++ external C API:
(Works if compiled is VisualStudio. The DLL produced by MinGW-G++ didn't work).
---
// main.h
#ifndef __MAIN_H__
#define __MAIN_H__
#include
#define DLL_EXPORT __declspec(dllexport)
#ifdef __cplusplus
extern "C"
{
#endif
int DLL_EXPORT add(int a, int b);
#ifdef __cplusplus
}
#endif
#endif // __MAIN_H__
---
---
//main.cpp
#include "main.h"
// a sample exported function
int DLL_EXPORT add(int a, int b)
{
return(a + b);
}
extern "C" DLL_EXPORT BOOL APIENTRY DllMain(HINSTANCE hinstDLL, DWORD
fdwReason, LPVOID lpvReserved)
{
switch (fdwReason)
{
case DLL_PROCESS_ATTACH:
// attach to process
// return FALSE to fail DLL load
break;
case DLL_PROCESS_DETACH:
// detach from process
break;
case DLL_THREAD_ATTACH:
// attach to thread
break;
case DLL_THREAD_DETACH:
// detach from thread
break;
}
return TRUE; // succesful
}
---
---
# -*- coding: utf-8 -*-
# dll.py
import ctypes
class DllInterface(object):
dll_handle = None
def __init__(self, dll_file):
self.dll_handle = ctypes.WinDLL(dll_file)
def add_a_and_b(self, a=0, b=0):
return self.dll_handle.add(a, b)
if __name__ == '__main__':
dll_file = 'PythonDLL.dll'
external_lib = DllInterface(dll_file)
int_a = ctypes.c_int(1)
int_b = ctypes.c_int(2)
result = external_lib.add_a_and_b(int_a, int_b)
print(result)
---
--
James
--
James
On 13 March 2014 15:57, Stefan Behnel wrote:
> Alan Gauld, 12.03.2014 23:05:
>> On 12/03/14 16:49, Stefan Behnel wrote:
>>> Alan Gauld, 12.03.2014 10:11:
If it were a library then you would have to call
the individual C++ functions directly using
something like ctypes, which is usually more
complex.
>>>
>>> ctypes won't talk to C++, but Cython can do it quite easily.
>>
>> I thought it would work provided the interface functions
>> were declared as C functions? That might involve
>> writing a wrapper around it but that is usually
>> trivial if you have to compile the source anyway.
>
> The thing is: if you have to write your own wrapper anyway (trivial or
> not), then why not write it in Cython right away and avoid the intermediate
> plain C level?
>
> It's usually much nicer to work with object oriented code on both sides
> (assuming you understand the languages on both sides), than to try to
> squeeze them through a C-ish API bottleneck in the middle.
>
> Stefan
>
>
> ___
> Tutor maillist - Tutor@python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] mixing 64 bit and 32 bit
Depending on what you're doing you could run into problems. Areas that have been challenging for me in the past: * Loading 64bit compiled .dll in a 32bit Python environment and vice versa. * Reading registry entries. MS tried to be clever by introducing the Wow6432Node reg key. And I'm sure there has been a 3rd item that I am currently unable to recall. It involved py2exe. This might have been related to the registry issue mentioned above though. With all that said, if you know you're likely to have these kinds of issues it's pretty easy to write code that can deal with these types of problems and do one thing or another depending on the base OS. -- James On 19 March 2014 19:53, John Fabiani wrote: > Thanks > Johnf > > On 03/19/2014 11:01 AM, Reuben wrote: > > Hi John, > > The generated bytecodes will be different - but both version can run same > code without issues > > Regards, > Reuben > > On 19-Mar-2014 11:28 PM, "John Fabiani" wrote: >> >> Hi, >> >> At my office we have a mix of XP (32bit) and Window 7 (64 bit). I >> installed python 64 bit on the windows 7 machines and 32 bit on the XP >> machines. The question is can the different version run the same code >> without causing issues. Can the 64 bit use the same byte code as the 32 >> bit? It seems to be working but I thought best to check. >> >> Johnf >> ___ >> Tutor maillist - Tutor@python.org >> To unsubscribe or change subscription options: >> https://mail.python.org/mailman/listinfo/tutor > > > > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor > ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] How can I let the Python Console display more decimal precision?
You could just initialise your variables with float() >>> float(26)/float(12) 2.1665 >>> varA = float(26) >>> varB = float(12) >>> varA/varB 2.1665 And so on... In fact, you only need to initialise one variable with float for this to work: >>> varA = float(26) >>> varB = 12 >>> varA/varB 2.1665 This works in Python2 or Python3 without importing any extra libs. -- James On 12 June 2014 13:31, Steven D'Aprano wrote: > On Thu, Jun 12, 2014 at 08:48:25AM +0800, Marino David wrote: > > Hi All: > > > > I am a newbie at the Python. > > > > I type "26/12" in Python Console and get result of "2". > > > > It is obvious that the corresponding result should be 2... I > don't > > know why the Console only returns the integer part of true result. > Anyone > > can help me out? > > Try this instead: > > 26.0/12 > > and it will print a fractional number instead of an int: > > py> 26.0/12 > 2.1665 > > > What's going on? > > Back in the early mists of time, when Python first came out, Python's > creator Guido van Rossum decided that the / division operator should > behave like in the C programming language. In C, division of two > integers performs *integer division*, and drops the remainder, while > division of one or more floating point number keeps the remainder as a > fraction: > > 1/2 => 0 > 1/2.0 => 0.5 > > That was one of those decisions that seemed like a good idea at the > time, but turned out to be a mistake. But for backwards compatibility, > Python had to keep it until recently. > > In Python version 3, / now does calculator division, like you expect. > But in Python 2, you have to either convert one or both numbers to a > float, or you can put this at the top of your program: > > from __future__ import division > > Note that there are TWO underscores at the beginning and end of > "future". > > > If you want integer division, where the remainder is ignored, you can > use the // operator: > > py> 26.0//12 > 2.0 > > > > -- > Steven > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor > ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] While Loop and Threads
Multi-threading takes practice!
Are you using an event object to signal the thread should exit? I'm
guessing you're just using a bool which is why it does not work.
See: https://docs.python.org/3.4/library/threading.html#event-objects
I'm very short on time and the moment and therefore can't mock up a working
example. If I have time later/tomorrow and you haven't solved it or no one
else has commented I'll try and put something together.
IMO, move away from GTK until you get threading working as expected, then
add the additional layer. Solve one problem at a time.
James
--
James
On 13 July 2014 12:23, Oğuzhan Öğreden wrote:
> Hi,
>
> I've been practicing with multithreading and gtk for a while and recently
> have observed something I can't quite grasp.
>
> This is basically a timer with a settings window and a countdown window
> which is produced after setting_window passes necessary arguments to thread.
>
> I have a while loop which counts time, inside the thread, while variable
> "runner" is True, which looks like this:
>
> def run(self):
> cycle = ['work', 'break', 'long_break']
> cycle = cycle[:2]*self.workn+cycle[3]
> self.runner = True
> sec_count = timedelta(seconds=1)
>
> while self.runner == True:
> for i in cycle:
> self.setState(i)
> print('Start:', i, datetime.now())
>
> while self.end_time > datetime.now():
> time.sleep(1)
> self.duration -= sec_count
>
> self.count[self.state] += 1
>
> And I want countdown to stop when countdown_window receives delete event,
> thus I have a countdown_window.stop() function which is defined as follows:
>
> def stop(self, *args):
> pom.count[pom.state] -= 1
> pom.runner = False # refers to the runner of the thread.
> print(pom.runner) # prints False,
> Gtk.main_quit()
>
> What I expect to happen is that while loop to break, but it does not.
>
> Any suggestions?
>
> Best,
> Oğuzhan
>
> ___
> Tutor maillist - Tutor@python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>
>
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] While Loop and Threads
OK, so I mocked up an example now...
import time
import threading
g_threadStop = threading.Event()
def threadWorker(_arg1, _arg2):
print("Starting worker thread with args: %s, %s" % (_arg1, _arg2))
while(not g_threadStop.is_set()):
print("Thread running.")
time.sleep(1)
print("Thread exiting...")
def main():
t = threading.Thread(target=threadWorker, args = ("arg1", "arg2"))
t.start()
print("Main thread sleeping for 10 seconds...")
time.sleep(5)
print("Main thread setting g_threadStop")
g_threadStop.set()
time.sleep(3)
print("Main thread exiting...")
if __name__ == '__main__':
main()
--
James
On 14 July 2014 19:03, James Chapman wrote:
> Multi-threading takes practice!
>
> Are you using an event object to signal the thread should exit? I'm
> guessing you're just using a bool which is why it does not work.
>
> See: https://docs.python.org/3.4/library/threading.html#event-objects
>
> I'm very short on time and the moment and therefore can't mock up a
> working example. If I have time later/tomorrow and you haven't solved it or
> no one else has commented I'll try and put something together.
>
> IMO, move away from GTK until you get threading working as expected, then
> add the additional layer. Solve one problem at a time.
>
> James
>
> --
> James
>
>
> On 13 July 2014 12:23, Oğuzhan Öğreden wrote:
>
>> Hi,
>>
>> I've been practicing with multithreading and gtk for a while and recently
>> have observed something I can't quite grasp.
>>
>> This is basically a timer with a settings window and a countdown window
>> which is produced after setting_window passes necessary arguments to thread.
>>
>> I have a while loop which counts time, inside the thread, while variable
>> "runner" is True, which looks like this:
>>
>> def run(self):
>> cycle = ['work', 'break', 'long_break']
>> cycle = cycle[:2]*self.workn+cycle[3]
>> self.runner = True
>> sec_count = timedelta(seconds=1)
>>
>> while self.runner == True:
>> for i in cycle:
>> self.setState(i)
>> print('Start:', i, datetime.now())
>>
>> while self.end_time > datetime.now():
>> time.sleep(1)
>> self.duration -= sec_count
>>
>> self.count[self.state] += 1
>>
>> And I want countdown to stop when countdown_window receives delete event,
>> thus I have a countdown_window.stop() function which is defined as follows:
>>
>> def stop(self, *args):
>> pom.count[pom.state] -= 1
>> pom.runner = False # refers to the runner of the thread.
>> print(pom.runner) # prints False,
>> Gtk.main_quit()
>>
>> What I expect to happen is that while loop to break, but it does not.
>>
>> Any suggestions?
>>
>> Best,
>> Oğuzhan
>>
>> ___
>> Tutor maillist - Tutor@python.org
>> To unsubscribe or change subscription options:
>> https://mail.python.org/mailman/listinfo/tutor
>>
>>
>
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] While Loop and Threads
t;
> But let's now follow your advice and leave GTK out for a while and keep
> imagining I want:
>
>1. Command line output for the timer.
>2. Event signal handling without a delay.
>
> I have a felling that 'thread-way of thinking' would come up with another
> thread for counter. Is that correct? If yes, would there be an obvious
> advantage to either using GTK main loop or dealing with another thread?
>
> 2014-07-14 20:24 GMT+02:00 James Chapman :
>
> OK, so I mocked up an example now...
>>
>>
>>
>> import time
>> import threading
>>
>> g_threadStop = threading.Event()
>>
>> def threadWorker(_arg1, _arg2):
>> print("Starting worker thread with args: %s, %s" % (_arg1, _arg2))
>> while(not g_threadStop.is_set()):
>> print("Thread running.")
>> time.sleep(1)
>> print("Thread exiting...")
>>
>> def main():
>> t = threading.Thread(target=threadWorker, args = ("arg1", "arg2"))
>> t.start()
>> print("Main thread sleeping for 10 seconds...")
>> time.sleep(5)
>> print("Main thread setting g_threadStop")
>> g_threadStop.set()
>> time.sleep(3)
>> print("Main thread exiting...")
>>
>> if __name__ == '__main__':
>> main()
>>
>>
>>
>>
>>
>> --
>> James
>>
>>
>> On 14 July 2014 19:03, James Chapman wrote:
>>
>>> Multi-threading takes practice!
>>>
>>> Are you using an event object to signal the thread should exit? I'm
>>> guessing you're just using a bool which is why it does not work.
>>>
>>> See: https://docs.python.org/3.4/library/threading.html#event-objects
>>>
>>> I'm very short on time and the moment and therefore can't mock up a
>>> working example. If I have time later/tomorrow and you haven't solved it or
>>> no one else has commented I'll try and put something together.
>>>
>>> IMO, move away from GTK until you get threading working as expected,
>>> then add the additional layer. Solve one problem at a time.
>>>
>>> James
>>>
>>> --
>>> James
>>>
>>>
>>> On 13 July 2014 12:23, Oğuzhan Öğreden wrote:
>>>
>>>> Hi,
>>>>
>>>> I've been practicing with multithreading and gtk for a while and
>>>> recently have observed something I can't quite grasp.
>>>>
>>>> This is basically a timer with a settings window and a countdown window
>>>> which is produced after setting_window passes necessary arguments to
>>>> thread.
>>>>
>>>> I have a while loop which counts time, inside the thread, while
>>>> variable "runner" is True, which looks like this:
>>>>
>>>> def run(self):
>>>> cycle = ['work', 'break', 'long_break']
>>>> cycle = cycle[:2]*self.workn+cycle[3]
>>>> self.runner = True
>>>> sec_count = timedelta(seconds=1)
>>>>
>>>> while self.runner == True:
>>>> for i in cycle:
>>>> self.setState(i)
>>>> print('Start:', i, datetime.now())
>>>>
>>>> while self.end_time > datetime.now():
>>>> time.sleep(1)
>>>> self.duration -= sec_count
>>>>
>>>> self.count[self.state] += 1
>>>>
>>>> And I want countdown to stop when countdown_window receives delete
>>>> event, thus I have a countdown_window.stop() function which is defined as
>>>> follows:
>>>>
>>>> def stop(self, *args):
>>>> pom.count[pom.state] -= 1
>>>> pom.runner = False # refers to the runner of the thread.
>>>> print(pom.runner) # prints False,
>>>> Gtk.main_quit()
>>>>
>>>> What I expect to happen is that while loop to break, but it does not.
>>>>
>>>> Any suggestions?
>>>>
>>>> Best,
>>>> Oğuzhan
>>>>
>>>> ___
>>>> Tutor maillist - Tutor@python.org
>>>> To unsubscribe or change subscription options:
>>>> https://mail.python.org/mailman/listinfo/tutor
>>>>
>>>>
>>>
>>
>
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] While Loop and Threads
Apologies, I didn't answer your question.
while time_now < time_finish: # counter and count_until defined
somewhere above and
if g_threadStop.is_set() == False:
# return something
Thread methods are typically void, meaning they return nothing. At least
this is the case in other languages, so I would expect the return to fail.
Also, what would you be returning it to?
Perhaps just a return without a value to indicate exit. (Probably some
reading required here)
Exceptions should be raised when something occurs that you were not
expecting and not used as an exit mechanism for normal operation.
Where to place the timer is up to you. Another thread is an option, but is
it necessary? Overhead needs to be considered.
--
James
On 15 July 2014 14:14, James Chapman wrote:
> So if I understand this correctly, you want to start a thread and then
> stop it after a certain time period?
>
> Here's an adapted example that includes a timer. (I believe in learning by
> example where possible)
> With that said, if it were my code going into production I'd move the
> timer logic out of the thread and only signal the thread when it should
> exit. Keep the logic within the thread simple and try and make each
> method/function do one thing only. It makes for testable, reusable code.
>
> In my new example
> * The first thread should exit because the time limit is met.
> * The second thread should exit because the Event is set.
>
> Hope this helps...
>
> James
>
>
>
> import time
> import threading
>
> class ThreadExample(object):
>
> def __init__(self):
> self.thread1Stop = threading.Event()
> self.thread2Stop = threading.Event()
> self.timeFinish = time.time() + 12
>
> def threadWorkerOne(self, _arg1, _arg2):
> print("THREAD1 - started with args: %s, %s" % (_arg1, _arg2))
> while(not self.thread1Stop.is_set()):
> while(time.time() < self.timeFinish):
> print("THREAD1 - Running")
> if (self.thread1Stop.is_set()):
> break # Break out of the time while loop
> time.sleep(1)
> if time.time() > self.timeFinish:
> print("THREAD1 - Time limit exceeded")
> self.thread1Stop.set()
> print("THREAD1 - Exiting because thread1Stop is TRUE")
>
> def threadWorkerTwo(self, _arg1, _arg2):
> print("THREAD2 - started with args: %s, %s" % (_arg1, _arg2))
> while(not self.thread2Stop.is_set()):
> while(time.time() < self.timeFinish):
> print("THREAD2 - Running")
> if (self.thread2Stop.is_set()):
> break # Break out of the time while loop
> time.sleep(1)
> if time.time() > self.timeFinish:
> print("THREAD2 - Time limit exceeded")
> self.thread2Stop.set()
> print("THREAD2 - Exiting because thread2Stop is TRUE")
>
> def main(self):
> t1 = threading.Thread(target=self.threadWorkerOne, args = ("arg1",
> "arg2"))
> t1.start()
> t2 = threading.Thread(target=self.threadWorkerTwo, args = ("arg1",
> "arg2"))
> t2.start()
> print("MAIN - sleep(5)")
> time.sleep(5)
> print("MAIN - setting thread1Stop")
> self.thread1Stop.set()
> print("MAIN - sleep(10)")
> time.sleep(10)
> print("MAIN - exiting...")
>
> if __name__ == '__main__':
> runExample = ThreadExample()
> runExample.main()
>
>
>
>
>
>
>
> --
> James
>
>
> On 15 July 2014 11:09, Oğuzhan Öğreden wrote:
>
>> Thanks!
>>
>> I'll have a side question. If I implement this idea to my case,
>> threadWorker() would look like this:
>>
>>
>> def threadWorker(_arg1, _arg2):
>> print("Starting worker thread with args: %s, %s" % (_arg1, _arg2))
>> while g_threadStop.is_set() == False:
>> ## here comes a for loop:
>> while time_now < time_finish: # time_finish and counter
>> defined somewhere above and
>> print("Thread running.") # and expression of the
>> condition is intentionally (and necessarily, I believe)
>> counter += 1 # left out of for
>> expression.
>> print(counter)
>> time.sleep(1)
>> print("Thread exiting...")
>>
Re: [Tutor] Does the user need to install Python, when we deploy our c++ products using python?
Actually, after re-reading your original message I think I misunderstood and went off on a tangent. If you are embedding python into a C++ app, then you will need the dlls and all of the compiled python code. *.pyc. (Well most, not all). The exact location of these files in the final build I'm unsure of. -- James On 11 December 2014 at 11:39, James Chapman wrote: > > On 2 December 2014 at 20:28, gordon zhang > wrote: > >> >> >> I downloaded python 3.4.2 for c++ and create a vc++ project using python, >> but I have no idea what python dlls and other stuff needed to deploy the >> products. >> >> I know if we put Python34.dll and Python.dll in the folder of executable, >> it is not enough. What else do we need to put in the folder of >> executable?(the application does not run) >> >> If anyone knows please let me know. >> >> Thanks, Gordon >> > > Probably a little advanced for this list but the list is here to learn > right? So my own 2 pence worth... > > Are you referring to Python Tools for Visual Studio? > http://pytools.codeplex.com/ > > This allows you to build python projects in VS and then debug them using > the VS debugger, but you will need the debug files (.pdb) available on this > page: https://www.python.org/downloads/windows/ - This however is more > for extending python via C/C++ than building python apps or using python to > deploy apps. > > There are a number of tools out there, some already mentioned for bundling > python apps into exe's which then require the python dlls to be packaged in > an installer to be deployed with your newly built exe. I have done this in > the past and experience has taught me one thing. Don't do it! You end up > making your app way too complex, and it becomes very difficult to support. > Also, when you want to update the python environment you run into problems. > From someone who has been there, don't do it. Even if you have CI > infrastructure in place to make things easily repeatable, it still becomes > a nightmare. > > Rather find an automated way of deploying python independently and then > use an installer to install just your python code. That way all you support > is your code, you don't end up supporting 3rd party wrappers that don't > work properly. > > nullsoft scriptable installer is easy to use, has good documentation and > examples, and doesn't have a massively steep learning curve. > http://nsis.sourceforge.net/Main_Page > > > ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Does the user need to install Python, when we deploy our c++ products using python?
On 2 December 2014 at 20:28, gordon zhang wrote: > > > I downloaded python 3.4.2 for c++ and create a vc++ project using python, > but I have no idea what python dlls and other stuff needed to deploy the > products. > > I know if we put Python34.dll and Python.dll in the folder of executable, > it is not enough. What else do we need to put in the folder of > executable?(the application does not run) > > If anyone knows please let me know. > > Thanks, Gordon > Probably a little advanced for this list but the list is here to learn right? So my own 2 pence worth... Are you referring to Python Tools for Visual Studio? http://pytools.codeplex.com/ This allows you to build python projects in VS and then debug them using the VS debugger, but you will need the debug files (.pdb) available on this page: https://www.python.org/downloads/windows/ - This however is more for extending python via C/C++ than building python apps or using python to deploy apps. There are a number of tools out there, some already mentioned for bundling python apps into exe's which then require the python dlls to be packaged in an installer to be deployed with your newly built exe. I have done this in the past and experience has taught me one thing. Don't do it! You end up making your app way too complex, and it becomes very difficult to support. Also, when you want to update the python environment you run into problems. >From someone who has been there, don't do it. Even if you have CI infrastructure in place to make things easily repeatable, it still becomes a nightmare. Rather find an automated way of deploying python independently and then use an installer to install just your python code. That way all you support is your code, you don't end up supporting 3rd party wrappers that don't work properly. nullsoft scriptable installer is easy to use, has good documentation and examples, and doesn't have a massively steep learning curve. http://nsis.sourceforge.net/Main_Page ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] While Loop Help
While Alan has given you a far better solution, I feel someone should
mention the break statement as you will likely come across it a lot, and it
is quite an important flow control statement.
You could add a break statement to the else which would break out of the
while loop.
https://docs.python.org/3.4/reference/simple_stmts.html#break
Refactored to include a break
--
...
the_number = random.randint(1, 100)
win = false
tries = 0
guess = int(input("Take a guess: "))
while tries < 10:
guess = int(input("Take a guess: "))
tries += 1
if guess > the_number:
print("Lower...")
elif guess < the_number:
print("Higher...")
else:
win = true
break
if win:
print("You win")
else:
print("You fail! The number was {0}".format(the_number))
input("\n\nPress the enter key to exit.")
--
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Python 3.4.1 ImportError Linux
cd .. Terminal (~/lorem): python3 app/main.py import statement is relative to pwd. -- James On 16 December 2014 at 14:18, Juan Christian wrote: > > Python 3.4.1 > Fedora 21 Server > > My paths: > ~/lorem > ~/lorem/app > ~/lorem/core > > I want to execute: ~/lorem/app/main.py > > Terminal (~/lorem/app): python3 main.py > > Traceback (most recent call last): > File "app/main.py", line 5, in > from core.backpack import BackpackThread > ImportError: No module named 'core' > > > Why am I getting this? > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor > ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Python 3.4.1 ImportError Linux
> Further to my last email, here's some reading regarding Python Paths > > http://www.stereoplex.com/blog/understanding-imports-and-pythonpath ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Assistance with UnicodeDecodeError
>
> I am trying to scrap text from a website using Python 2.7 in windows 8 and
> i am getting this error *"**UnicodeDecodeError: 'charmap codec can't encode
> character u'\u2014 in position 11231 character maps to "*
>
>
For starters, move away from Python 2 unless you have a good reason to use
it. Unicode is built into Python 3 whereas it's an after thought in Python
2.
What's happening is that python doesn't understand the character set in use
and it's throwing the exception. You need to tell python what encoding to
use: (not all website are "utf-8")
Code example (using python 2.7):
>>> u = u'\u2014'
>>> print(u)
Traceback (most recent call last):
File "", line 1, in
File "c:\Python27\lib\encodings\cp850.py", line 12, in encode
return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character u'\u2014' in
position 0: character maps to
>>> s = u.encode("utf-8")
>>> print(s)
ÔÇö
I also strongly suggest you read:
https://docs.python.org/2/howto/unicode.html
There is much cursing to come. Unicode and especially multi-byte character
string processing is a nightmare!
Good luck ;-)
James
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Assistance with UnicodeDecodeError
Actually, it's more likely that the char you are grabbing is UTF-16 not
UTF-8 which is moving into the double byte...
* An assumption based on the following output:
>>> u = u'\u2014'
>>> s = u.encode("utf-16")
>>> print(s)
■¶
>>> s = u.encode("utf-32")
>>> print(s)
■ ¶
>>> s = u.encode("utf-16LE")
>>> print(s)
¶
>>> s = u.encode("utf-16BE")
>>> print(s)
¶
See https://en.wikipedia.org/wiki/Character_encoding to help with the
understanding of character encoding, code pages and why they are important.
James
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] mySQL and Python
One of my pet hates about this list... "This is a tutor list, your question is out of scope". Sure there might be better places to seek answers, and sure maybe the first responder doesn't know the answer, but that's not a reason to respond with that phrase. This list is a called python tutor, not python beginner, even if the large majority of the questions are beginner questions. The fact that people can ask any python related question is one of the things I like about it and wish that other languages had similar lists. Back to answer the original question... I recommend using the official MySQL connector because it's supported by MySQL and it's continuously developed, which means it won't stop working when you change Python versions, or MySQL versions, and it's documented. I've tried some other MySQL libs in the past that worked OK but were a nightmare when it came to supporting them due to changes in the environment. Download the connector from https://dev.mysql.com/downloads/connector/python/ The latest version should work just fine on Python 3.4 Documentation on how to use it is also available on the MySQL website: https://dev.mysql.com/doc/connector-python/en/connector-python-examples.html Hope this helps. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Topic focus of ‘python-tutor’ (was: mySQL and Python)
Long-ish reply, but please bear with me. To quote the list description "This list is for folks who want to ask questions regarding how to learn computer programming with the Python language and its standard library." While MySQL modules are not part of the standard library, consider the following scenario. A person, let's call him Xerxes, is interesting in learning to program in Python. So he does a whole lot of reading, asks a load of questions on this list, he learns the basics. He now thinks to himself, wow, it would be really cool if I could actually do something really useful with my new skill set, maybe he has a job where building an interface to a database would save him and his company a lot of effort, but as MySQL is not part of the standard library he's not really sure how to go about it. Since people on the tutor list have always been willing to help and offer assistance, he thinks, I'll ask there. He asks his question and almost immediately gets told that what he's asking is off topic for the list. Let's jump back to the list description for a minute there. "This list is for folks who want to *ask questions regarding how to learn computer programming with the Python language* and its standard library." Is the installation and usage of 3rd party modules not related to learning to program with python? By responding in that manner, you're discouraging the asking of questions. No one should *_EVER_ *be discouraged to ask a question they do not know the answer to. That response that I referred to in my initial response is very common on this list and I think we (collectively) could and should come up with a better way of pointing people in the direction of satisfactory resolution. I also don't think we should immediately point people else where when the question is outside our comfort zone. OO, inheritance and multi-byte string processing, to name a few, are part of learning to program but are easily more advanced topics than which MySQL module to use. Creating C data structures in python, while part of the standard library, is an example of a topic that is probably too advanced for this list, and even then, I managed to find someone on this list able to help with that topic. If we discourage people from asking more interesting questions then I suspect that many subscribers will stop subscribing and that knowledge will be lost. I subscribe to quite a few programming related mailing lists and let me tell you, this one is _BY_FAR_ the most willing to help. That question would very likely have been ignored on other "more advanced" lists. Finally, I realise I've offered some criticism but not a suggestion for an alternative, so here is an example of a response to a question that would be considered "outside the scope" of this list: Hi Xerxes Great question. I know there are many options when it comes to MySQL but I've not had any experience with any. Someone else on this list might be able to provide a satisfactory answer, but, as the main focus of this list is learning to program in python using the standard library you might not get an answer, and if you do, the answer might not be very knowledgeable. You could try asking your question on forum X or mailing list Y as these types of topics are often covered there. James ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Windows Memory Basics
We're heading into advanced territory here and I might get told off but...
Consider this C++ program for a second, it has a struct with different
types of variables which sit next to each other in memory. When you print
the byte values of the struct, you can see that there is no easy way to
know which byte value belongs to which variable unless you already know the
layout.
-
#include
typedef unsigned char BYTE;
typedef unsigned short WORD;
typedef unsigned long DWORD;
#pragma pack(1)
struct mem
{
char c; // 1 byte
WORD wn;// 2 byte
DWORD dwn; // 4 byte
int i; // 4 byte
BYTE b; // 1 byte
};
int main()
{
mem s;
s.c = 0xFF;
s.wn = 0xF;
s.dwn = 0x;
s.i = 0x;
s.b = 0xFF;
BYTE * memPointer = reinterpret_cast(&s);
for (int i = 0; i < sizeof(mem); i++)
printf("%02d [0x%08x] = %x \n", i, memPointer, *memPointer++);
return 0;
}
-
Prints
00 [0xecf0f789] = ff
01 [0xecf0f78a] = ff
02 [0xecf0f78b] = ff
03 [0xecf0f78c] = ff
04 [0xecf0f78d] = ff
05 [0xecf0f78e] = ff
06 [0xecf0f78f] = ff
07 [0xecf0f790] = ff
08 [0xecf0f791] = ff
09 [0xecf0f792] = ff
10 [0xecf0f793] = ff
11 [0xecf0f794] = ff
Packing can also come into play. If you change the packing to 2, you get
this:
00 [0x7d4ffcd9] = ff
01 [0x7d4ffcda] = cc
02 [0x7d4ffcdb] = ff
03 [0x7d4ffcdc] = ff
04 [0x7d4ffcdd] = ff
05 [0x7d4ffcde] = ff
06 [0x7d4ffcdf] = ff
07 [0x7d4ffce0] = ff
08 [0x7d4ffce1] = ff
09 [0x7d4ffce2] = ff
10 [0x7d4ffce3] = ff
11 [0x7d4ffce4] = ff
12 [0x7d4ffce5] = ff
13 [0x7d4ffce6] = cc
And if you change it to 4, you get this:
00 [0xf4f5fbf9] = ff
01 [0xf4f5fbfa] = cc
02 [0xf4f5fbfb] = ff
03 [0xf4f5fbfc] = ff
04 [0xf4f5fbfd] = ff
05 [0xf4f5fbfe] = ff
06 [0xf4f5fbff] = ff
07 [0xf4f5fc00] = ff
08 [0xf4f5fc01] = ff
09 [0xf4f5fc02] = ff
10 [0xf4f5fc03] = ff
11 [0xf4f5fc04] = ff
12 [0xf4f5fc05] = ff
13 [0xf4f5fc06] = cc
14 [0xf4f5fc07] = cc
15 [0xf4f5fc08] = cc
In other words, even if you have the source code for the program you want
to scan in memory, depending on the compiler settings the memory layout
could have changed, or rather not be what you expected due to packing and
alignment.
Probably not the answer you were hoping for but I hope this helps.
--
James
On 17 October 2017 at 01:02, Michael C
wrote:
> Hold on, supposed by using Openprocess and VirtualQueryEx, I have the
> locations of all the memory the application is using, wouldn't this to be
> true?
>
> Say, a 8 byte data is somewhere in the region i am scanning. Ok, I know by
> scanning it like this
> for n in range(start,end,1)
>
> will read into another variable and mostly nothing, but unless a variable,
> that is, one number, can be truncated and exist in multiple locations like
> this
>
> double = 12345678
>
> 123 is at x001
> 45 is at x005
> 678 is at x010
>
> unless a number can be broken up like that, wouldn't I, while use the silly
> 'increment by one' approach, actually luck out and get that value in it's
> actual position?
>
>
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] How to debug a memory leak in a wsgi application?
Why pymalloc? I presume this means you're using ctypes which means I have more questions. If you're allocating your own blocks of memory then you need to free them too. IE, does each call to pymalloc have a corresponding call to pyfree? Is the overhead of pythons built in malloc really a problem? Are you changing pointers before you've freed the corresponding block of memory? There are many ways to create a memory leak, all of them eliminated by letting python handle your memory allocations. But, back to your original question, check out "valgrind". HTH -- James On 6 December 2017 at 16:23, Etienne Robillard wrote: > Hi Alan, > > Thanks for the reply. I use Debian 9 with 2G of RAM and precompiled Python > 2.7 with pymalloc. I don't know if debugging was enabled for this build and > whether I should enable it to allow memory profiling with guppy... My > problem is that guppy won't show the heap stats for the uWSGI master > process. However I have partially resolved this issue by enabling > --reload-on-rss 200 for the uwsgi process. Previously, the htop utility > indicated a 42.7% rss memory usage for 2 uWSGI processes. I have restarted > the worker processes with SIGINT signal. Now my uwsgi command line looks > like: > > % uwsgi --reload-on-rss 200 --gevent 100 --socket localhost:8000 > --with-file /path/to/file.uwsgi --threads 2 --processes 4 --master > --daemonize /var/log/uwsgi.log > > My framework is Django with django-hotsauce 0.8.2 and werkzeug. The web > server is nginx using uWSGI with the gevent pooling handler. > > Etienne > > Le 2017-12-06 à 10:00, Alan Gauld via Tutor a écrit : > >> On 06/12/17 09:21, Etienne Robillard wrote: >> >>> Hi >>> >>> I think my wsgi application is leaking and I would like to debug it. >>> >>> What is the best way to profile memory usage in a running wsgi app? >>> >> This is probably a bit advanced for the tutor list, you might >> get a better response on the main Python list. >> >> But to get a sensible answer you need to provide more data: >> What OS and Python version? >> What toolset/framework are you using? >> What measurements lead you to suspect a memory leak? >> >> >> > -- > Etienne Robillard > tkad...@yandex.com > https://www.isotopesoftware.ca/ > > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor > ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] How to debug a memory leak in a wsgi application?
No, I'm saying you shouldn't need to make any kind of malloc calls manually. Python handles memory allocation and deallocation on your behalf. Why do you need to call pymalloc? Are you using ctypes? And if you are I presume this is then to make C-calls into a shared library? James -- James On 13 December 2017 at 18:30, Etienne Robillard wrote: > Hi James, > > Thank for your reply. Are you suggesting that under Linux the malloc() > glibc library call is more memory efficient than using pymalloc? > > Best regards, > > Etienne > > Le 2017-12-13 à 12:27, James Chapman a écrit : > > Why pymalloc? I presume this means you're using ctypes which means I have > more questions. > > If you're allocating your own blocks of memory then you need to free them > too. IE, does each call to pymalloc have a corresponding call to pyfree? > > Is the overhead of pythons built in malloc really a problem? > > Are you changing pointers before you've freed the corresponding block of > memory? > > There are many ways to create a memory leak, all of them eliminated by > letting python handle your memory allocations. > > But, back to your original question, check out "valgrind". > > HTH > > -- > James > > On 6 December 2017 at 16:23, Etienne Robillard wrote: > >> Hi Alan, >> >> Thanks for the reply. I use Debian 9 with 2G of RAM and precompiled >> Python 2.7 with pymalloc. I don't know if debugging was enabled for this >> build and whether I should enable it to allow memory profiling with >> guppy... My problem is that guppy won't show the heap stats for the uWSGI >> master process. However I have partially resolved this issue by enabling >> --reload-on-rss 200 for the uwsgi process. Previously, the htop utility >> indicated a 42.7% rss memory usage for 2 uWSGI processes. I have restarted >> the worker processes with SIGINT signal. Now my uwsgi command line looks >> like: >> >> % uwsgi --reload-on-rss 200 --gevent 100 --socket localhost:8000 >> --with-file /path/to/file.uwsgi --threads 2 --processes 4 --master >> --daemonize /var/log/uwsgi.log >> >> My framework is Django with django-hotsauce 0.8.2 and werkzeug. The web >> server is nginx using uWSGI with the gevent pooling handler. >> >> Etienne >> >> Le 2017-12-06 à 10:00, Alan Gauld via Tutor a écrit : >> >>> On 06/12/17 09:21, Etienne Robillard wrote: >>> >>>> Hi >>>> >>>> I think my wsgi application is leaking and I would like to debug it. >>>> >>>> What is the best way to profile memory usage in a running wsgi app? >>>> >>> This is probably a bit advanced for the tutor list, you might >>> get a better response on the main Python list. >>> >>> But to get a sensible answer you need to provide more data: >>> What OS and Python version? >>> What toolset/framework are you using? >>> What measurements lead you to suspect a memory leak? >>> >>> >>> >> -- >> Etienne Robillard >> tkad...@yandex.com >> https://www.isotopesoftware.ca/ >> >> ___ >> Tutor maillist - Tutor@python.org >> To unsubscribe or change subscription options: >> https://mail.python.org/mailman/listinfo/tutor >> > > > -- > Etienne Robillardtkadm30@yandex.comhttps://www.isotopesoftware.ca/ > > ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] How to debug a memory leak in a wsgi application?
Ah OK, now I understand why you mentioned pymalloc to begin with. I'm not familiar with uWSGI or cython. That said, why do you think it's uWSGI causing a leak? It seems unlikely. Python projects can grow in size if you're not dereferencing objects... (see https://f0rki.at/hunting-memory-leaks-in-python.html) If you use valgrind combined with python memory_profiler you should hopefully be able to get an idea as to where the leak is coming from. It's probably in your own code and leaks can be incredibly difficult to track down. Typically while reviewing your own code you end up skipping over the error time and time again because you become blind to your errors, so it might help to have someone else peer review it. These 2 links are a good starting point. https://github.com/KratosMultiphysics/Kratos/wiki/Checking-memory-usage-with-Valgrind https://github.com/pythonprofilers/memory_profiler One last note, if you are doing any of your own memory allocations, then make sure you're also freeing them: https://cython.readthedocs.io/en/latest/src/tutorial/memory_allocation.html But note, if you did this in cython: cdef double* data data = PyMem_Malloc(100 * sizeof(double)) data = PyMem_Malloc(100 * sizeof(double)) PyMem_Free(data) You would (probably, you would in C/C++) end up with a leak because you've changed the pointer. When you go to free it, only the 2nd allocation will be freed and you'll have no way of freeing the first. HTH ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] When do you know you're ready to start applying for jobs?
Why has no one mentioned Github/Gitlab? Set up a free account on either or both platforms, and start committing your code. When applying for jobs potential employers will often want to see what you're capable of even before inviting you for an interview, and many will ask for a github page to see your work and whether you're contributing to open source projects. They'll also want to see if your code is unit tested as they'll want to employ programmers who are happy and able to write unit tests and that understand the value of unit tests. It also goes to show that you understand and know how to use source control effectively as this will be a requirement for any software development company. https://about.gitlab.com/ https://github.com/ Gitlab offers better features than github and it's arguable a better git source control platform than github, that said, github has somehow become the defacto standard for open source projects. At the company where I work, we're also migrating to github enterprise. Demand for python programmers has grown over the years, reflected in the Tiobe index: https://www.tiobe.com/tiobe-index/ If you're able to write and understand what's happening on any of these projects (https://github.com/trending/python) then you're ready to start applying for jobs. Show off your skills via public git profiles and you should have something in no time, especially if you're not fussy! Programmers are in demand! Hope that helps and good luck. James ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] calling printf() in a C language DLL using ctypes.CDLL
A long time ago when I was working with Python and DLLs I slapped together a basic and ugly example. You can find it here: https://github.com/James-Chapman/python-code-snippets/ tree/master/DLL_C_funcs_w_callbacks The whole thing should load into Visual Studio. I can't guarantee that it works in its current state though. It was created mainly as a reminder to myself. -- James ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
