Re: [Tutor] reading an input stream

[eryk sun]

On Thu, Dec 24, 2015 at 9:13 PM, boB Stepp  wrote:
> AttributeError: 'generator' object has no attribute 'next'

The iterator protocol was added in Python 2.2 (circa 2001) as a
generalization for use in "for" loops, but the language didn't have
built-in next() at the time. Instead the method to get the next item
from an iterator was defined without double underscores. You'd simply
call it.next() to manually get the next item of iterator "it".

Python 3 added built-in next() and changed the method name to
"__next__". The built-in function was backported to 2.6 to have a
common idiom even though the method is still named "next" in Python 2.

The name change in Python 3  reflects that "__next__" is a special
method that's looked up on the type (in CPython it's the tp_iternext
field of the PyTypeObject). You can't simply add a bound next method
to an instance to make Python think it's an iterator. The same applies
in Python 2, but the name "next" doesn't suggest that this is the
case.

For example, let's start out with a normal Python 2 iterator that
simply iterates a count from some initial value.

class Iterator(object):
def __init__(self, start):
self.value = start - 1
def __iter__(self):
return self
def next(self):
self.value += 1
return self.value

>>> it = Iterator(0)
>>> it.next()
0
>>> next(it)
1

Now store the bound next method directly on the instance

>>> it.next = it.next
>>> it.next.__self__ is it
True

and remove the method from the class:

>>> del Iterator.next

The bound method still works:

>>> it.next()
2

But the interpreter doesn't look for "next" on the instance:

>>> next(it)
Traceback (most recent call last):
  File "", line 1, in 
TypeError: Iterator object is not an iterator

>>> for i in it:
... if i == 3: break
...
Traceback (most recent call last):
  File "", line 1, in 
TypeError: iter() returned non-iterator of type 'Iterator'

Since "next" is a special method, it should have the special name
"__next__". So let it be written. So let it be done... in Python 3.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Python 3.5.1 64 bit?

[eryk sun]

On Tue, Jan 5, 2016 at 5:50 PM, Alan Gauld  wrote:
> On 05/01/16 19:47, yehudak . wrote:
>> I'm using version 3.5 of Python and want to upgrade.
>
> 3.5 is the latest stream. If you are already using it why
> do you want to upgrade? Its easy to get caught up in a frenzy
> of chasing the latest dot release but unless there is a
> specific bug fix or feature you need there is rarely any
> good reason. Personally I haven't got around to moving
> from 3.4 yet and on my Windows  7 PC I'm still on v3.3.

Review the release notes [1] to determine whether you need to upgrade.
For example, 3.5.1 has the following fixes that are specific to
Windows:

Issue #19143: platform module now reads Windows version from
  kernel32.dll to avoid compatibility shims.
Issue #25112: py.exe launcher is missing icons
Issue #25118: Fix a regression of Python 3.5.0 in os.waitpid() on
  Windows.
Issue #25213: Restores requestedExecutionLevel to manifest to
  disable UAC virtualization.
Issue #25361: Disables use of SSE2 instructions in Windows 32-bit
  build
Issue #25450: Updates shortcuts to start Python in installation
  directory.

Issues 25213 and 25361 only affect 32-bit Python.

It also fixes several issues with the new installer:

Issue #25081: Makes Back button in installer go back to upgrade
  page when upgrading.
Issue #25089: Adds logging to installer for case where launcher is
  not selected on upgrade.
Issue #25091: Increases font size of the installer.
Issue #25102: Windows installer does not precompile for -O or -OO.
Issue #25126: Clarifies that the non-web installer will download
  some components.
Issue #25143: Improves installer error messages for unsupported
  platforms.
Issue #25163: Display correct directory in installer when using
  non-default settings.
Issue #25164: Changes default all-users install directory to match
  per-user directory.
Issue #25165: Windows uninstallation should not remove launcher if
  other versions remain
Issue #25715: Python 3.5.1 installer shows wrong upgrade path and
  incorrect logic for launcher detection.

[1]: https://docs.python.org/3.5/whatsnew/changelog.html

>> Also, when I upgrade, does the newer version over-ride the old one or I
>> have first to remove the older version?
>
> They generally leave the old version around - you may have
> a need for both (for example to test your codes compatibility
> with both versions).

On Windows, updating to a newer micro release (i.e. major.minor.micro)
replaces the previous version. However, if you have the 64-bit version
of 3.5.0 installed for all users, you can install the 32-bit version
of 3.5.1 for the current user without affecting the system
installation.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Python 3.5.1 64 bit?

[eryk sun]

On Wed, Jan 6, 2016 at 1:06 AM, yehudak .  wrote:
> What's wrong with upgrading to newer version?

It depends on how many systems and virtual environments that you're
upgrading. It shouldn't be an issue if it's just a new micro release
for your own development machine. If you come across a regression, you
can just downgrade to the previous version. That said, there's the old
adage that if something isn't broken (for your needs), then you
shouldn't 'fix' it.

Installing a new minor release (e.g. 3.5 to 3.6) is more work, since
you have to reinstall all packages and rebuild extension modules for
source packages. But it's worth it to stay current with the evolution
of the language. On Windows, make sure to set the PY_PYTHON3
environment variable to the version you want as the default "python3".
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Question about the memory manager

[eryk sun]

On Wed, Jan 13, 2016 at 2:01 AM, Albert-Jan Roskam
 wrote:
> I sometimes check the Task Manager to see how much RAM is in use. Is there a 
> Python way to do something similar?

Use the psutil module to solve this problem across platforms. For
Windows only, you can use ctypes to call GetProcessMemoryInfo. Here's
a function that returns the total working set size (including shared
pages) and the private working set.

import ctypes
from ctypes import wintypes
import collections

kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)
psapi = ctypes.WinDLL('psapi', use_last_error=True)

wintypes.SIZE_T = ctypes.c_size_t

class PROCESS_MEMORY_COUNTERS(ctypes.Structure):
_fields_ = (('cb', wintypes.DWORD),
('PageFaultCount', wintypes.DWORD),
('PeakWorkingSetSize', wintypes.SIZE_T),
('WorkingSetSize', wintypes.SIZE_T),
('QuotaPeakPagedPoolUsage',wintypes.SIZE_T),
('QuotaPagedPoolUsage',wintypes.SIZE_T),
('QuotaPeakNonPagedPoolUsage', wintypes.SIZE_T),
('QuotaNonPagedPoolUsage', wintypes.SIZE_T),
('PagefileUsage',  wintypes.SIZE_T),
('PeakPagefileUsage',  wintypes.SIZE_T))
def __init__(self, *args, **kwds):
super(PROCESS_MEMORY_COUNTERS, self).__init__(*args, **kwds)
self.cb = ctypes.sizeof(self)

class PROCESS_MEMORY_COUNTERS_EX(PROCESS_MEMORY_COUNTERS):
_fields_ = (('PrivateUsage', wintypes.SIZE_T),)

PPROCESS_MEMORY_COUNTERS = ctypes.POINTER(PROCESS_MEMORY_COUNTERS)

def check_bool(result, func, args):
if not result:
raise ctypes.WinError(ctypes.get_last_error())
return args

kernel32.GetCurrentProcess.restype = wintypes.HANDLE

psapi.GetProcessMemoryInfo.errcheck = check_bool
psapi.GetProcessMemoryInfo.argtypes = (wintypes.HANDLE,
   PPROCESS_MEMORY_COUNTERS,
   wintypes.DWORD)

MemoryUsage = collections.namedtuple('MemoryUsage', 'total private')

def get_memory_usage():
info = PROCESS_MEMORY_COUNTERS_EX()
psapi.GetProcessMemoryInfo(kernel32.GetCurrentProcess(),
   ctypes.byref(info),
   ctypes.sizeof(info))
if info.PrivateUsage:
return MemoryUsage(info.WorkingSetSize, info.PrivateUsage)
else:
# prior to Windows 7.
return MemoryUsage(info.WorkingSetSize, info.PagefileUsage)
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Question about the memory manager

[eryk sun]

On Thu, Jan 14, 2016 at 3:03 AM, Albert-Jan Roskam
 wrote:
>
> These two pages are quite nice. The author says the memory used by small 
> objects is
> never returned to the OS, which may be problematic for long running processes.

The article by Evan Jones discusses a patch to enable releasing unused
arenas (i.e. "how the problem was fixed"). Starting with 2.5, unused
arenas do get released back to the heap. Here's the diff in which Tim
Peters merged in a "heavily altered derivative" of Evan's patch [1].

Also, 2.7 and 3.3 bypass C malloc/free and the process heap to instead
use mmap/munmap on POSIX when available. This avoids the heap
high-water mark problem. Similarly, 3.4 switched to using
VirtualAlloc/VirtualFree on Windows. 3.4 also introduced the
PyObjectArenaAllocator and associated C API functions [2] to allow
modifying the default allocators.

[1]: https://hg.python.org/cpython/diff/685849bd905c/Objects/obmalloc.c
[2]: 
https://docs.python.org/3/c-api/memory.html#customize-pyobject-arena-allocator
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Simultaneous read and write on file

[eryk sun]

On Mon, Jan 18, 2016 at 10:01 AM, Anshu Kumar  wrote:
> I have read in documentation that wb+ mode is for writing and reading. Am i
> using wrong mode, should i use rb+ ?

Use w+ to create a new file, opened with read and write access. Use r+
to open an existing file with read and write access. Unlike w+, r+
does not truncate the file and will not create a new file.

To clarify what Alan said, the "+" modes do not 'lock' the file. On
Windows you may experience a sharing violation if another program
doesn't share write access. This isn't quite the same as a file lock.
All open references to a file have to agree on read, write, and delete
sharing. For example, when Notepad saves a file, it tries to open a
handle that shares only read access. If this open succeeds, Notepad
knows that it's the only writer. (It doesn't care if there are
multiple readers.) On the other hand, if some other program has the
file open with write or delete access, Notepad's open will fail with a
sharing violation. Then it displays a message box saying it can't save
the file because another process has it open, and it offers to save
the file using a different name.

FYI, in terms of POSIX open flags [1], the file mode gets mapped as follows:

  |   flags   |   no +   |   +
-
r |   | O_RDONLY | O_RDWR
w | O_CREAT, O_TRUNC  | O_WRONLY | O_RDWR
a | O_CREAT, O_APPEND | O_WRONLY | O_RDWR

Python 3 only:

x | O_CREAT, O_EXCL   | O_WRONLY | O_RDWR

In terms of Windows open dispositions and access modes [2], the file
mode gets mapped as follows:

  |  disposition  |  no + |+
---
r | OPEN_EXISTING | GENERIC_READ  | GENERIC_READ, GENERIC_WRITE
w | CREATE_ALWAYS | GENERIC_WRITE | GENERIC_READ, GENERIC_WRITE
a | OPEN_ALWAYS   | GENERIC_WRITE | GENERIC_READ, GENERIC_WRITE

Python 3 only:

x | CREATE_NEW| GENERIC_WRITE | GENERIC_READ, GENERIC_WRITE

O_APPEND entails an implicit seek to the end of a file before every
write. Note that Python uses the Windows CRT for file access, which
doesn't use the native FILE_APPEND_DATA access to implement O_APPEND.
Instead it opens the file for generic writing and manually updates the
file pointer. This may cause problems if the security descriptor only
allows appending to a file. In this case you'd have to call CreateFile
directly (e.g. via ctypes or win32api).

[1]: http://pubs.opengroup.org/onlinepubs/9699919799/functions/open.html
[2]: https://msdn.microsoft.com/en-us/library/aa363858
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Simultaneous read and write on file

[eryk sun]

On Mon, Jan 18, 2016 at 10:04 PM, Martin A. Brown  wrote:
> The above is a very handy chart.  Did you find this somewhere, eryk
> sun, or is this from your own knowledge and experience?

The mapping to POSIX open flags is from a table in the POSIX fopen
spec [1] as well as the Windows CRT docs for fopen (see the equivalent
oflag table) [2]. The flags for "x" mode can be verified in the Python
3 source [3]. The "x" mode flags are also mentioned in GNU's libc docs
for fopen [4] and open-time flags [5].

The mapping from POSIX open flags to Windows create disposition and
access modes is from the CRT source for the _open function. I can't
provide a source link, but the source code is distributed with MSVC.
For VS2015 it's in lowio/open.cpp, decode_access_flags() and
decode_open_create_flags().

[1]: http://pubs.opengroup.org/onlinepubs/9699919799/functions/fopen.html
[2]: https://msdn.microsoft.com/en-us/library/yeby3zcb
[3]: https://hg.python.org/cpython/file/v3.5.1/Modules/_io/fileio.c#l302
[4]: https://www.gnu.org/software/libc/manual/html_node/Opening-Streams.html
[5]: https://www.gnu.org/software/libc/manual/html_node/Open_002dtime-Flags.html
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] lc_ctype and re.LOCALE

[eryk sun]

On Thu, Jan 28, 2016 at 2:23 PM, Albert-Jan Roskam
 wrote:
> Out of curiosity, I wrote the throw-away script below to find a character 
> that is classified
> (--> LC_CTYPE) as digit in one locale, but not in another.

The re module is the wrong tool for this. The re.LOCALE flag is only
for byte strings, and in this case only ASCII 0-9 are matched as
decimal digits. It doesn't call the isdigit() ctype function. Using
Unicode with re.LOCALE is wrong. The current locale doesn't affect the
meaning of a Unicode character. Starting with 3.6 doing this will
raise an exception.

The POSIX ctype functions such as isalnum and isdigit are limited to a
single code in the range 0-255 and EOF (-1). For UTF-8, the ctype
functions return 0 in the range 128-255 (i.e. lead bytes and trailing
bytes aren't characters). Even if this range has valid characters in a
given locale, it's meaningless to use a Unicode value from the Latin-1
block, unless the locale uses Latin-1 as its codeset.

Python 2's str uses the locale-aware isdigit() function. However, all
of the locales on my Linux system use UTF-8, so I have to switch to
Windows to demonstrate two locales that differ with respect to
isdigit(). You could use PyWin32 or ctypes to iterate over all the
locales known to Windows, if it mattered that much to you.

The English locale (codepage 1252) includes superscript digits 1, 2, and 3:

>>> locale.setlocale(locale.LC_CTYPE, 'English_United Kingdom')
'English_United Kingdom.1252'
>>> [chr(x) for x in range(256) if chr(x).isdigit()]
['0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '\xb2', '\xb3', '\xb9']
>>> unicodedata.name('\xb9'.decode('1252'))
'SUPERSCRIPT ONE'
>>> unicodedata.name('\xb2'.decode('1252'))
'SUPERSCRIPT TWO'
>>> unicodedata.name('\xb3'.decode('1252'))
'SUPERSCRIPT THREE'

Note that using the re.LOCALE flag doesn't match these superscript digits:

>>> re.findall(r'\d', '0123456789\xb2\xb3\xb9', re.LOCALE)
['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']

The Windows Greek locale (codepage 1253) substitutes "Ή" for superscript 1:

>>> locale.setlocale(locale.LC_CTYPE, 'Greek_Greece')
'Greek_Greece.1253'
>>> [chr(x) for x in range(256) if chr(x).isdigit()]
['0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '\xb2', '\xb3']

>>> unicodedata.name('\xb9'.decode('1253'))
'GREEK CAPITAL LETTER ETA WITH TONOS'
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Surprised that print("a" "b") gives "ab"

[eryk sun]

On Sun, Mar 6, 2016 at 9:52 AM, Steven D'Aprano  wrote:
> On Sun, Mar 06, 2016 at 01:03:01AM -0600, boB Stepp wrote:
>
>> get your semantics point, but are there two string objects created in
>> both approaches or does the first in fact create only a single object?
>>  If the first truly only creates a single object, then it seems that
>> this is a more efficient approach.
>
> In practical terms, in CPython today, there is no difference between the
> two, or if there is any difference, it's undetectible: by the time the
> compiler has generated the byte-code, the concatenation has been
> performed:
>
> py> import dis
> py> code = compile("""
> ... a = 'ab' 'cd'
> ... b = 'ab' + 'cd'
> ... """, "", "exec")
> py> dis.dis(code)
>   2   0 LOAD_CONST   0 ('abcd')
>   3 STORE_NAME   0 (a)
>
>   3   6 LOAD_CONST   4 ('abcd')
>   9 STORE_NAME   1 (b)
>  12 LOAD_CONST   3 (None)
>  15 RETURN_VALUE

Note that when CPython's bytecode optimizer folds binary operations on
operands that are sized constants, such as strings, the size of the
resulting constant is limited to 20 items (e.g. characters in the case
of strings). In either case, the original constants are still stored
in the resulting code object (and when marshalled in .pyc files) for
simplicity since otherwise it would have to track when a constant is
no longer referenced by the final code.

>>> code = compile('"9876543210" + "0123456789";'
...'"4321098765" + "56789012345"',
...'', 'exec')
>>> code.co_consts
('9876543210', '0123456789', '4321098765', '56789012345',
 None, '98765432100123456789')
>>> dis.dis(code)
  1   0 LOAD_CONST   5 ('98765432100123456789')
  3 POP_TOP
  4 LOAD_CONST   2 ('4321098765')
  7 LOAD_CONST   3 ('56789012345')
 10 BINARY_ADD
 11 POP_TOP
 12 LOAD_CONST   4 (None)
 15 RETURN_VALUE

Note that "98765432100123456789" is listed after None in co_consts
because it was appended in the optimization stage.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Changing the interpreter prompt symbol from ">>>" to ???

[eryk sun]

On Sat, Mar 12, 2016 at 12:46 AM, boB Stepp  wrote:
> I did with the non-printing control character, but not with '\u25ba' !
>  So I had to go through some contortions after some research to get my
> Win7 cmd.exe and PowerShell to display the desired prompt using

The console is hosted by another process named conhost.exe. When
Python is running in the foreground, the cmd shell is just waiting in
the background until Python exits.

> '\u25ba' as the character with utf-8 encoding.  My new
> pythonstartup.py file (Which PYTHONSTARTUP now points to) follows:
>
> #!/usr/bin/env python3
>
> import os
> import sys
>
> os.system('chcp 65001')# cmd.exe and PowerShell require the code
> page to be changed.
> sys.ps1 = '\u25ba '  # I remembered to add the additional space.

chcp.com calls SetConsoleCP (to change the input codepage) and
SetConsoleOutputCP. You can call these functions via ctypes if you
need to separately modify the input or output codepages. For example:

>>> kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)
>>> sys.ps1 = '\u25ba '
Γû║ kernel32.SetConsoleOutputCP(65001)
1
►

UTF-8 in the console is generally buggy, but this example works since
the sys.ps1 prompt is written without buffering (discussed below).
However, Python still thinks the console is using the initial
codepage. To print non-ASCII characters, you'll either have to change
the codepage before starting Python or rebind sys.stdout and
sys.stderr.

► print(''.join(chr(x) for x in range(240, 256)))


Let's try to fix this:

► fd = os.dup(1)
► sys.stdout = open(fd, 'w', encoding='utf-8')
► print(''.join(chr(x) for x in range(240, 256)))
ðñòóôõö÷øùúûüýþÿ
ùúûüýþÿ
�þÿ

►

The above buggy output is in Windows 7. Codepage 65001 was only ever
meant for encoding text to and from files and sockets, via WinAPI
WideCharToMultiByte and MultiByteToWideChar. Using it in the console
is buggy because the console's ANSI API hard codes a lot of
assumptions that fall apart with UTF-8.

For example, if you try to paste non-ASCII characters into the console
using 65001 as the input codepage, Python will quit as if you had
entered Ctrl+Z. Also, in Windows 7 when you print non-ASCII characters
you'll get a trail of garbage written to the end of the print in
proportion to the number of non-ASCII characters, especially with
character codes that take more than 2 UTF-8 bytes. Also, with Python
2, the CRT's FILE buffering can split a UTF-8 sequence across two
writes. The split UTF-8 sequence gets printed as 2 to 4 replacement
characters. I've discussed these problems in more detail in the
following issue:

http://bugs.python.org/issue26345

Windows 10 fixes the problem with printing extra characters, but it
still has the problem with reading non-ASCII input as UTF-8 and still
can't handle a buffered writer that splits a UTF-8 sequence across two
writes. Maybe Windows 20 will finally get this right.

For the time being, programs that use Unicode in the console should
use the wide-character (UTF-16) API. Python doesn't support this out
of the box, since it's not designed to handle UTF-16 in the raw I/O
layer. The win-unicode-console package add this support.

https://pypi.python.org/pypi/win_unicode_console
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Changing the interpreter prompt symbol from ">>>" to ???

[eryk sun]

On Sun, Mar 13, 2016 at 3:14 AM, Albert-Jan Roskam
 wrote:
> I thought that utf-8 (cp65001) is by definition (or by design?) impossible
> for console output in windows? Aren't there "w" (wide) versions of functions
> that do accept utf-8?

The wide-character API works with the native Windows character
encoding, UTF-16. Except the console is a bit 'special'. A surrogate
pair (e.g. a non-BMP emoji) appears as 2 box characters, but you can
copy it from the console to a rich text application, and it renders
normally. The console also doesn't support variable-width fonts for
mixing narrow and wide (East Asian) glyphs on the same screen. If that
matters, there's a program called ConEmu that hides the console and
proxies its screen and input buffers to drive an improved interface
that has flexible font support, ANSI/VT100 terminal emulation, and
tabs. If you pair that with win_unicode_console, it's almost as good
as a Linux terminal, but the number of hoops you have to go through to
make it all work is too complicated.

Some people try to use UTF-8 (codepage 65001) in the ANSI API --
ReadConsoleA/ReadFile and WriteConsoleA/WriteFile. But the console's
UTF-8 support is dysfunctional. It's not designed to handle it.

In Windows 7, WriteFile calls WriteConsoleA, which decodes the buffer
to UTF-16 using the current codepage and returns the number of UTF-16
'characters' written instead of the number of bytes. This confuses
buffered writers. Say it writes a 20-byte UTF-8 string with 2 bytes
per character. WriteFile returns that it successfully wrote 10
characters, so the buffered writer tries to write the last 10 bytes
again. This leads to a trail of garbage text written after every
write.

When a program reads from the console using ReadFile or ReadConsoleA,
the console's input buffer has to be encoded to the target codepage.
It assumes that an ANSI character is 1 byte, so if you try to read N
bytes, it tries to encode N characters. This fails for non-ASCII
UTF-8, which has 2 to 4 bytes per character. However, it won't
decrease the number of characters to fit in the N byte buffer. In the
API the argument is named "nNumberOfCharsToRead", and they're sticking
to that literally. The result is that 0 bytes are read, which is
interpreted as EOF. So the REPL will quit, and input() will raise
EOFError.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Changing the interpreter prompt symbol from ">>>" to ???

[eryk sun]

On Tue, Mar 15, 2016 at 2:51 PM, Albert-Jan Roskam
 wrote:
>
> So windows uses the following (Western locales):
> console: cp437 (OEM codepage)
> "bytes": cp1252 (ANSI codepage)

The console defaults to the OEM codepage, but you can separately
switch the input and output to different codepages. This is an
exception to the rule, as otherwise the system codepage used in the
[A]NSI API is fixed when the system boots. Changing it requires
modifying the system locale and rebooting.

> unicode: utf-16-le (is 'mbcs' equivalent to utf-16-*?)

The native Unicode encoding of Windows is UTF-16LE. This is what gets
used in the kernel, device drivers, and filesystems. UTF-16 was
created to accommodate the early adopters of 16-bit Unicode, such as
Windows. When you call an ANSI API, such as CreateFileA, the bytes
argument(s) get decoded to UTF-16, and then it calls the corresponding
wide-character function, such as CreateFileW (or maybe a common
internal function, but that's an implementation detail). ANSI is a
legacy API, and it's moving towards deprecation and obsolescence. New
WinAPI functions are often created with only wide-character support.

MBCS (multibyte character set) refers to encoding that can be used in
the system locale for the [A]NSI API. While technically UTF-8 and
UTF-16 are multibyte encodings, they're not allowed in the legacy ANSI
API. That said, because the console is just plain weird, it allows
setting its input and output codepages to UTF-8, even though the
result is often buggy.

> Sheesh, so much room for errors. Why not everything utf-8, like in linux?

NT was developed before UTF-8 was released, so you're asking the NT
team to invent a time machine. Plus there's nothing really that
horrible about UTF-16. On the plus side, it uses only 2 bytes per
character for all characters in the BMP, whereas UTF-8 uses 3 bytes
per character for 61440 out of 63488 characters in the BMP (not
including the surrogate-pair block, U+D800-U+DFFF). On the plus side
for UTF-8, it encodes ASCII (i.e. ordinals less than 128) in a single
byte per character.

> Is cmd.exe that impopular that Microsoft does not replace it with something
> better?

cmd.exe is a shell, like powershell.exe or bash.exe, and a console
client application just like python.exe. cmd.exe doesn't host the
console, nor does it have anything to do with the console subsystem
other than being a client of it. When you run python.exe from cmd.exe,
all cmd does is wait for python to exit.

When you run a program that's flagged as a console application, the
system either attaches an inherited console if one exists, or opens
and attaches a new console. This window is hosted by an instance of
conhost.exe. It also implements the application's command-line
editing, input history buffer (e.g. F7), and input aliases, separately
for each attached executable. This part of the console API is
accessible from the command line via doskey.exe.

cmd.exe is a Unicode application, so codepages aren't generally of
much concern to it, except it defaults to encoding to the console
codepage when its built-in commands such as "dir" and "set" are piped
to another program. You can force it to use UTF-16 in this case by
running cmd /U. This is just for cmd's internal commands. What
external commands write to a pipe is up to them. For example, Python 3
defaults to using the ANSI codepage when stdio is a pipe. You can
override this via the environment variable PYTHONIOENCODING.

As to replacing the console, I doubt that will happen. Microsoft has
little incentive to invest in improving/replacing the console and
command-line applications. Windows administration has shifted to
PowerShell scripting and cmdlets.

> am I correct in saying that the use of codepages (with stupid differences
> such as latin-1 vs cp1252 as a bonus) are designed to hamper cross-
> platform compatibility (and force people to stick with windows)?

The difference between codepage 1252 and Latin-1 is historical.
Windows development circa 1990 was following a draft ANSI standard for
character encodings, which later became the ISO 8859-x encodings.
Windows codepages ended up deviating from the ISO standard. Plus the
'ANSI' API also supports MBCS codepages for East-Asian languages. I
can't speak to any nefarious business plans to hamper cross-platform
compatibility. But it seems to me that this is a matter of rushing to
get a product to market without having time to wait for a standards
committee, plus a good measure of arrogance that comes from having
market dominance.

> Strange. I would have thought it writes the first 10 bytes (5 characters)
> and that the remaining 10 bytes end up in oblivion.

Maybe a few more details will help to clarify the matter. In Windows
7, WriteFile basically calls WriteConsoleA, which makes a local
procedure call (LPC) to SrvWriteConsole in conhost.exe. This call is
flagged as to whether the buffer is ANSI or Unicode (UTF-16). If it's
ANSI, the console first decodes the buffer using

Re: [Tutor] Changing the interpreter prompt symbol from ">>>" to ???

[eryk sun]

On Tue, Mar 15, 2016 at 4:40 PM, Alan Gauld  wrote:
> On 15/03/16 19:51, Albert-Jan Roskam wrote:
>
>> Sheesh, so much room for errors. Why not everything utf-8, like in linux?
>> Is cmd.exe that impopular that Microsoft does not replace it with something 
>> better?
>
> It's replaced by Powershell.
> It's just that not many people use Powershell.
> Familiarity beats superiority.
>
> Whether Powershell actually handles display of text any better is
> another question. I don;t know the answer to that and am too lazy
> to fire up my Windows box to find out! :-(

powershell.exe is a standard console application. It works the same as
cmd.exe and python.exe.

OTOH, maybe you're referring to PowerShell Integrate Scripting
Environment (ISE), which is more like an IDE. It hides the console
window and sets stdout and stderr to pipes. Bizarrely, if you run
python.exe in this environment, stdin still uses the hidden console (I
made the console visible and could type into it and see the result in
ISE). It's pretty much useless for regular interactive console
applications. It's more focused on designing, testing and running
PowerShell scripts and cmdlets.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Changing the interpreter prompt symbol from ">>>" to ???

[eryk sun]

On Tue, Mar 15, 2016 at 7:45 PM, boB Stepp  wrote:
> On Tue, Mar 15, 2016 at 7:35 PM, boB Stepp  wrote:
>> Is the following result in PowerShell related to what Eryk has been 
>> discussing?
>>
>> Windows PowerShell
>> Copyright (C) 2009 Microsoft Corporation. All rights reserved.
>>
>> PS C:\Windows\system32> py
>> Python 3.5.1 (v3.5.1:37a07cee5969, Dec  6 2015, 01:54:25) [MSC v.1900
>> 64 bit (AMD64)] on w
>> in32
>> Type "help", "copyright", "credits" or "license" for more information.
>> Active code page: 65001
>> ► help(print)
>> Not enough memory.
>
> Apparently this is another consequence of setting the code page to 65001.  See
>
> https://bugs.python.org/issue19914

I added an analysis of this bug to issue 19914. It's not directly
related to a bug in the console, but occurs if the console is set to
certain codepages, including 65001. It's a bug in a library that's
used by Windows command-line utilities such as more.com and find.exe.
The bug doesn't exist in Windows 10.

> I am now wondering which is better:  Adopt the solution suggested
> earlier by Eryk, applying that Python monkey patch and using utf-16 or
> doing something more drastic, like installing Cygwin.  What are the
> cons in doing the latter?  Will I achieve Unicode harmony under
> Windows by using Cygwin?

If you don't need to support Windows, then you could develop in a
Linux VM. Unicode works very well in a UTF-8 terminal, which Python's
POSIX-oriented I/O system is designed for.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] (SIP installation problem) Best way to install Python 3 onto Windows 10

[eryk sun]

On Thu, Mar 17, 2016 at 6:36 AM, CMG Thrissur  wrote:
> Thank you for the link but on that page i see mention of SIP requirement, is
> that required for binary package also.

The .exe and .whl packages for PyQt are self-contained. You don't need
anything else, and you don't have to build anything from source.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] (SIP installation problem) Best way to install Python 3 onto Windows 10

[eryk sun]

On Wed, Mar 16, 2016 at 12:56 PM, Alan Gauld  wrote:
> On 16/03/16 12:46, CMG Thrissur wrote:
>
>> I  tried to install activestate on win 10 but due to my ignorance or
>> lack of knowlege i could n't get to install pyqt or sip. i tried it
>> through pip.

PyQt can't be easily installed from source using pip on Windows, and
the project doesn't distribute pre-built wheel packages. You probably
want an installer:

https://www.riverbankcomputing.com/software/pyqt/download
https://www.riverbankcomputing.com/software/pyqt/download5

Riverbank doesn't have a Python 3.5 build yet, but there's an
unofficial wheel available here:

http://www.lfd.uci.edu/~gohlke/pythonlibs/#pyqt4

Instructions to install wheels:

https://pip.pypa.io/en/latest/user_guide/#installing-from-wheels

>> Even on basic installation i am getting it hard to find installation
> of sip.
>
> I confess i'm not sure which SIP you mean? Google suggests a C/C++
> bindings generator which might make sense if that's what PyQt uses,
> although I'd expect all the bindings to be done for you.
> Are you sure you really need it to use PyQt?

The Qt bindings require the sip extension module at runtime, but
that's included. If you need SIP for your own C++ project, you can
learn more about it and download the source from Riverbank:

https://riverbankcomputing.com/software/sip/intro
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Fwd: Newbie trying to get pip run on windows 7

[eryk sun]

On Tue, Apr 19, 2016 at 11:26 PM, Gustavo Davis via Tutor
 wrote:
>  But for some reason after I made the changes and saved them they wont
>  run. I mean once I go to the file and right click on them and click run
>  the cmd prompt pops up for a moment and then it just closes down and the
>  pip module never runs in python.

When you run a .py script from Explorer, Windows creates a new console
for the script's standard input and output, but this console closes as
soon as the script exits (i.e. when the python.exe process exits).
Instead you can run the script from an existing cmd shell to inherit
the shell's console.

> ;C:\Python34\Scripts\pip

There's no "Scripts\pip" directory.

Only add fully qualified directories to PATH, separated by a
semicolon, without quotes, and with no spaces between entries. You can
reference another environment variable in PATH, but only if the
variable doesn't reference other environment variables.

Add ;.PY to PATHEXT to allow running "command" instead of "command.py".

> >>> import pyperclip
> Traceback (most recent call last):
>   File "", line 1, in 
> import pyperclip
> ImportError: No module named 'pyperclip'

With PATH set up correctly, you can install this module from a cmd
shell using "pip install pyperclip".

pip searches for packages on pypi.python.org. This should work fine
for pure-Python packages and those with a WHL or EGG that's built for
your version of Python. Some source-only packages require building
extension modules. If your system isn't configured to build
extensions, look for an unofficial WHL on Christoph Gohlke's site:

http://www.lfd.uci.edu/~gohlke/pythonlibs
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] int(1.99...99) = 1 and can = 2

[eryk sun]

On Sun, May 1, 2016 at 1:02 AM, boB Stepp  wrote:
>
> py3: 1.
> 2.0
> py3: 1.999
> 1.999
...
> It has been many years since I did problems in converting decimal to
> binary representation (Shades of two's-complement!), but I am under
> the (apparently mistaken!) impression that in these 0.999...999
> situations that the floating point representation should not go "up"
> in value to the next integer representation.

https://en.wikipedia.org/wiki/Double-precision_floating-point_format

A binary64 float has 52 signifcand bits, with an implicit integer
value of 1, so it's effectively 53 bits. That leaves 11 bits for the
exponent, and 1 bit for the sign.

The 11-bit exponent value is biased by 1023, i.e. 2**0 is stored as
1023. The minimum binary exponent is (1-1023) == -1022, and the
maximum binary exponent is (2046-1023) == 1023. A biased exponent of 0
signifies either signed 0 (mantissa is zero) or a subnormal number
(mantissa is nonzero). A biased exponent of 2047 signifies either
signed infinity (mantissa is zero) or a non-number, i.e. NaN
(mantissia is nonzero).

The largest finite value has all 53 bits set:

>>> sys.float_info.max
1.7976931348623157e+308

>>> sum(Decimal(2**-n) for n in range(53)) * 2**1023
Decimal('1.797693134862315708145274237E+308')

The smallest finite value has the 52 fractional bits unset:

>>> sys.float_info.min
2.2250738585072014e-308

>>> Decimal(2)**-1022
Decimal('2.225073858507201383090232717E-308')

The machine epsilon value is 2**-52:

>>> sys.float_info.epsilon
2.220446049250313e-16

>>> Decimal(2)**-52
Decimal('2.220446049250313080847263336E-16')

Your number is just shy of 2, i.e. implicit 1 plus a 52-bit fractional
value and a binary exponent of 0.

>>> sum(Decimal(2**-n) for n in range(53))
Decimal('1.999777955395075')

The next increment by epsilon jumps to 2.0. The 52-bit mantissa rolls
over to all 0s, and the exponent increments by 1, i.e. (1 + 0.0) *
2**1. Python's float type has a hex() method to let you inspect this:

>>> (1.9998).hex()
'0x1.fp+0'
>>> (2.0).hex()
'0x1.0p+1'

where the integer part is 0x1; the 52-bit mantissa is 13 hexadecimal
digits; and the binary exponent comes after 'p'. You can also parse a
float hex string using float.fromhex():

>>> float.fromhex('0x1.0p-1022')
2.2250738585072014e-308

>>> float.fromhex('0x1.fp+1023')
1.7976931348623157e+308
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Howto display progress as cmd is being executed via subprocess ?

[eryk sun]

On Mon, Jun 13, 2016 at 4:50 PM, Ramanathan Muthaiah
 wrote:
>
> subprocess.check_output([cmd], stderr=subprocess.STDOUT, shell=True)
> ...
> how to combine the progressbar and subprocess code snippets to show the
> progress as the cmd is being executed.

check_output waits for the process to exit, so you can't use it with a
progress bar. However, if the program writes its progress to stdout,
then you can use `p = subprocess.Popen(args, stdout=PIPE)`, and create
a thread to read p.stdout, parse the output, and update the progress
bar.

Many programs detect when a standard stream isn't an interactive TTY
and switch to full buffering. You need to avoid this if you're
tracking progress or otherwise need to interact with a child process.
If you're on a Unix system you can use the stdbuf utility program to
try to force the child to disable buffering or use line buffering. If
that doesn't work, or you're on Windows, maybe the program has a
command-line option or environment variable setting that forces
interactive mode, such as Python's `-i` option. If all else fails, try
pexpect or winpexpect instead of subprocess. This works by making a
program think it's connected to a terminal/console.

FYI, with shell=True, you should use a command line instead of an args list.

The POSIX implementation does the following:

if isinstance(args, (str, bytes)):
args = [args]
else:
args = list(args)

if shell:
args = ["/bin/sh", "-c"] + args

Using just [cmd] doesn't cause a problem, but [cmd, arg1, arg2] is
probably a mistake. This passes the arguments to the shell, which sets
them as $N parameters. For example:

>>> subprocess.call(['echo $0 $1', 'arg1', 'arg2'], shell=True)
arg1 arg2

The Windows implementation does the following:

if not isinstance(args, str):
args = list2cmdline(args)

if shell:
startupinfo.dwFlags |= _winapi.STARTF_USESHOWWINDOW
startupinfo.wShowWindow = _winapi.SW_HIDE
comspec = os.environ.get("COMSPEC", "cmd.exe")
args = '{} /c "{}"'.format (comspec, args)

Thus on Windows even [cmd] is potentially wrong with shell=True.
list2cmdline doesn't know how to escape special characters for
cmd.exe.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] capture output from a subprocess while it is being produced

[eryk sun]

On Tue, Jun 14, 2016 at 8:03 PM, Albert-Jan Roskam
 wrote:
>
> proc = Popen("ls -lF", cwd=cwd, shell=True, stdout=PIPE, stderr=PIPE)

Don't use shell=True if you can avoid it. "ls" isn't a shell command.
Use ['ls', '-lF'].

The child process probably buffers its output when stdout isn't a
terminal. A typical standard I/O buffer size is 4 KiB. Reading from
the pipe will block until the child flushes the buffer to the pipe. It
might fflush() at checkpoints, but otherwise this happens
automatically when the buffer is full or when the child exits
normally.

On Linux you can use stdbuf to try forcing a program's stdout to use
either no buffering (-o0) or line buffering (-oL) -- e.g. `stdbuf -o0
ls -lF`. That said, "ls" should use full buffering in a pipe. You need
to test using a long-running process with intermittent output. You
could use a Python script that sleeps for a random interval between
writes to stdout and stderr.

On Windows there's no program like stdbuf to control standard I/O
buffering. If the target is a Python script, you can force interactive
mode using the "-i" option or disable buffering using "-u" (but -u
doesn't work for the 2.x REPL on Windows, since it switches to binary
mode).

> #output, errors = proc.communicate() # "Wait for process to 
> terminate."
> output = proc.stdout.read()  # docs: "Use communicate() rather 
> than .stdin.write, .stdout.read or .stderr.read"
> errors = proc.stderr.read()

Start a dedicated thread to read from each file. Otherwise you can end
up with a deadlock. For example, the child could block while writing
to a full stderr pipe, while the parent blocks while reading from an
empty stdout pipe. Also, you should read by lines using readline(), or
a byte at a time using read(1).

> if errors:
> raise RuntimeError(str(errors))

Data on stderr doesn't necessarily indicate an error. It could simply
be logging, warnings, or debug output. An error is indicated by a
non-zero exit code. Use proc.wait().
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Why are expressions not allowed as parameters in function definition statements?

[eryk sun]

On Sun, Jun 19, 2016 at 4:04 AM, Danny Yoo  wrote:
>
>> def f((x,y), z):
> ...  print x, y, z
> ...
>>   f([1, 2], 3)
> 1 2 3

Note that Python 3 doesn't support tuple parameter unpacking. See PEP 3113:

https://www.python.org/dev/peps/pep-3113
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] iterators

[eryk sun]

On Mon, Jul 4, 2016 at 9:56 PM, Danny Yoo  wrote:
> So the extra trailing comma in a 1-tuple parenthesized expression is
> just there to make it different looking, to disambiguate it from the
> use of parentheses for expression grouping.

The comma is the distinguishing element of non-empty tuple literals.
It's always required in order to create a non-empty tuple, with or
without brackets. For example:

>>> 27,
(27,)

The parentheses are optional for a non-empty tuple, but they're often
required because a comma has low precedence. For example, with the
expression `x in y, z`, Python first evaluates `x in y`. If the goal
is to check whether x is in the the tuple `y, z`, then it has to be
written `x in (y, z)`.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Quote and double quotes opens up File Explorer in Windows 10

[eryk sun]

On Sat, Jul 9, 2016 at 2:22 PM, Ken G.  wrote:
> Hi: In gradually moving over my Python programs (2.7.6) to Windows 10 and
> using Geany 1.27 to modify or write up a program there, it is noted that
> whenever I typed in a quote (') or double quote ("), Windows 10 File
> Explorer opens up. I end up closing it by typing ALT-F4 to resume typing as
> before. Is there a way to prevent the File Explorer from opening up whenever
> I use a quote or double quote? Is there another key(s) that does weird thing
> in Windows 10? Thanks.

The single/double quote key isn't a standard hotkey in any version of
Windows. That would be insane. If this is just happening in Geany,
check its keybindings preferences. Otherwise, check that you're using
the right keyboard layout and that you don't have any programs (or
malware) that's registering a global hotkey.

For example, here's a simple script that registers the single quote
(US keyboard layout) as a global hotkey to open the desktop in
Explorer:

#! /usr/bin/python3
import os
import ctypes
from ctypes import wintypes

user32 = ctypes.WinDLL('user32')

WM_HOTKEY = 0x0312
MOD_NOREPEAT = 0x4000
VK_OEM_7 = 0xDE # US layout, single/double quote

hotkey_id = 1
if user32.RegisterHotKey(None, hotkey_id, MOD_NOREPEAT, VK_OEM_7):
print('Hotkey "\'" registered.')
msg = wintypes.MSG()
count = 0
while (count < 3 and
   user32.GetMessageW(ctypes.byref(msg), None, 0, 0)):
if msg.message == WM_HOTKEY and msg.wParam == hotkey_id:
count += 1
print('Hotkey received.')
os.startfile('shell:Desktop')
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] (no subject)

[eryk sun]

On Fri, Jul 22, 2016 at 7:38 AM, DiliupG  wrote:
> I am using Python 2.7.12 on Windows 10
>
> filename = u"මේක තියෙන්නේ සිංහලෙන්.txt"
> Unsupported characters in input

That error message is from IDLE. I'm not an expert with IDLE, so I
don't know what the following hack potentially breaks, but it does
allow entering the above Unicode filename in the interactive
interpreter.

Edit "Python27\Lib\idlelib\IOBinding.py". Look for the following
section on line 34:

if sys.platform == 'win32':
# On Windows, we could use "mbcs". However, to give the user
# a portable encoding name, we need to find the code page
try:
encoding = locale.getdefaultlocale()[1]
codecs.lookup(encoding)
except LookupError:
pass

Replace the encoding value with "utf-8" as follows:

# encoding = locale.getdefaultlocale()[1]
encoding = "utf-8"

When you restart IDLE, you should see that sys.stdin.encoding is now "utf-8".

IOBinding.encoding is used by ModifiedInterpreter.runsource in
PyShell.py. When the encoding is UTF-8, it passes the Unicode source
string directly to InteractiveInterpreter.runsource, where it gets
compiled using the built-in compile() function.

Note that IDLE uses the Tk GUI toolkit, which -- at least with Python
Tkinter on Windows -- is limited to the first 65,536 Unicode
characters, i.e the Basic Multilingual Plane. The BMP includes
Sinhalese, so your filename string is fine.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] (no subject)

[eryk sun]

On Mon, Jul 25, 2016 at 10:28 AM, Steven D'Aprano  wrote:
> I know that Linux and Mac OS X both use UTF-8 for filenames, and so support 
> all
> of Unicode. But Windows, I'm not sure. It might be localised to only use 
> Latin-1
> (Western European) or similar.

Windows filesystems (e.g. NTFS, ReFS, UDF, exFAT, FAT32) use Unicode
[1], i.e. UTF-16, as does the Windows wide-character API. Using 16-bit
wchar_t strings is a problem for C standard functions such as fopen,
which require 8-bit null-terminated strings, so the Windows C runtime
also supports wide-character alternatives such as _wfopen.

Python's os and io functions use Windows wide-character APIs for
unicode arguments, even in 2.x. Unfortunately some 2.x modules such as
subprocess use only the 8-bit API (e.g. 2.x Popen calls CreateProcessA
instead of CreateProcessW).

[1]: https://msdn.microsoft.com/en-us/library/ee681827#limits
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] (no subject)

[eryk sun]

On Tue, Jul 26, 2016 at 4:39 AM, DiliupG  wrote:
> I am reading in a list of file names with the following code.
>
> def get_filelist(self):
> ""
> app = QtGui.QApplication(sys.argv)
> a = QtGui.QFileDialog.getOpenFileNames()
>
> filelist = []
> if a:
> for name in a:
> a = unicode(name)
>
> filelist.append(a)
>
> return filelist
>
> mainprg.filelist = mainprg.get_filelist()
>
> filelist = mainprg.filelist
>
> for name in filelist:
> print name <  THIS IS WHERE THE PROBLEM IS

This is an output problem, which is unrelated to the IDLE input error
that you initially reported. But IDLE isn't (at least shouldn't be)
used for deployed applications. It's a development environment.

Generally if you're using Qt then you're not creating a console
application that prints to stdout, but instead the program outputs
text to a Qt widget such as a QTextEdit. That said, if you need to print
Unicode text to the Windows console, for whatever reason (maybe
debugging), then pip install and enable the win_unicode_console
module.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] command to determine of python 32 or 64 bit?

[eryk sun]

On Wed, Aug 10, 2016 at 4:27 AM, Garry Willgoose
 wrote:
> I have a number of binary libraries that are dependent on whether the 
> precompiled python
> distribution (eg. Enthought, ActiveState, etc)  in which they are installed 
> are compiled with
> 32 or 64 bit. Is there any reliable way to determine at run time whether a 
> python distribution
> is 32 or 64 bit? If I can determine at runtime then I can set the path for 
> loading of my libraries
> to something different depending on whether the python is 32 or 64 bit.

Use platform.architecture()[0]. Currently the value will be either
"32bit" or "64bit".
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Where is win32print in Windows 10 Pro

[eryk sun]

On Thu, Aug 11, 2016 at 2:44 PM, Joaquin Alzola
 wrote:
>
>>import win32print
>>ImportError: No module named win32print
>
> That module doesn't exist on your python path
>
> 'pywin32' is its canonical name.
>
> http://sourceforge.net/projects/pywin32/

I'm not certain what's meant in the above, but to be clear, PyWin32
installs most of its modules as top-level imports. So `import
win32print` is correct.

PyWin32 can be pip installed using the "pypiwin32" package:

https://pypi.python.org/pypi/pypiwin32

The difference compared to the executable installer is that the wheel
package doesn't automatically run the post-install script that copies
some DLLs to the System32 directory. Running the post-install script
isn't necessary if you just need to call Windows and COM APIs.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] __init__

[eryk sun]

On Tue, Aug 30, 2016 at 9:09 AM, Alan Gauld via Tutor  wrote:
> new() that sets up the memory then calls init(). So init is
> only used to initialise the object after it has been
> constructed.

__new__ and __init__ are called by the metaclass __call__ method.
__init_ is called if __new__ returns an instance of the class.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] @property for old style classes vs new style classes

[eryk sun]

On Thu, Sep 15, 2016 at 4:40 AM, monik...@netzero.net
 wrote:
> class GetSet():
>
> def __init__(self, value):
> self.attrval = value
>
> @property
> def var(self):
> print "getting the var attribute"
> return self.attrval
> @var.setter
> def var(self,value):
> print "setting the var attribute"
> self.attrval = value
>
> @var.deleter
> def var(self):
> print "deleting the var attribute"
> self.attrval = None
>
> me = GetSet(5)
> me.var = 1000

A classic instance has both a type (`instance`) and a __class__ (e.g.
GetSet). The type defines how getting, setting, and deleting
attributes works, and the hard-coded classic behavior for calling the
class __getattr__, __setattr__, and __delattr__ methods.

If the class doesn't define __setattr__ and __delattr__, the instance
type sets and deletes attributes directly in the instance dict. Unlike
new-style classes, a data descriptor defined by the class gets ignored
here.

Getting attributes also prefers the instance dict. However, to support
bound methods (e.g. __init__), it falls back on a class lookup and
calls the descriptor __get__ method if defined. Unintentionally, it
happens that this partially supports the property descriptor. But
since there's no data descriptor support, it's only useful for
read-only properties.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] @property for old style classes vs new style classes

[eryk sun]

On Thu, Sep 15, 2016 at 9:48 PM, monik...@netzero.net
 wrote:
> But why var is in class __dict__?
> Does @property make it a class attribute?

An instance of `property` is a data descriptor, i.e. it implements the
__get__, __set__, and __delete__ methods of the descriptor protocol. A
non-data descriptor only implements __get__. Descriptors are
fundamental to the way attribute lookup works in Python. See the
descriptor howto guide:

https://docs.python.org/3/howto/descriptor.html

The above document paints a picture of attribute access that's not
really accurate at times. Also, the example code that attempts to
demonstrate type.__getattribute__ in terms of object.__getattribute__
is wrong. (It fails to search the __mro__ of the class, only the
metaclass. Also, what it's trying to do is fundamentally impossible.
You can't know whether object.__getattribute___ already evaluated a
descriptor that it found in the metaclass.)

Here's an overview of how object.__getattribute__ works:

* If a data descriptor is found in the class, call its __get__
  method, passing it the instance and the class, and return the
  result.
* Else if the instance has a dict and the name is found in it,
  return the value from the instance dict.
* Else if a non-data descriptor is found in the class, call its
  __get__ method, passing it the instance and the class, and
  return the result.
* Else if a non-descriptor is found in the class, return it.
* Else raise AttributeError.

A class lookup searches the dicts of each class in the __mro__,
stopping on the first hit. Also, a type can define a __getattr__
fallback method, in which case the AttributeError raised by
__getattribute__ is ignored.

For type.__getattribute__, the class object in this case is handled as
an instance of the metaclass. The main difference compared to
object.__getattribute__ is in the second step, since the instance in
this case is a class. Instead of searching just the class dict, it
searches the dicts of each class in the __mro__. Also, if it finds a
descriptor in the class __mro__, it calls its __get__ method, passing
None for the instance.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Python IDLE for Windows 10, 64-bit

[eryk sun]

On Wed, Sep 21, 2016 at 3:14 PM, Aaron Rose  wrote:
> I'm looking for a link for Python IDLE for Windows 10, 64-bit.  Can't seem to 
> find
> the right link on python.org.  Could someone kindly point me in the right 
> direction?

Python's installer has an option to install IDLE. Here's the link for
the latest stable version of the web-based installer for x64:

https://www.python.org/ftp/python/3.5.2/python-3.5.2-amd64-webinstall.exe
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] help with Numpy

[eryk sun]

On Wed, Sep 21, 2016 at 2:53 PM, Paul Dentinger  wrote:
>
> I need Numpy and Scipy and may need matplotlib.  So I tried to call Numpy and 
> it
> can't be found.  So I go online and find Numpy and download it.

Read the following docs on Python packaging and pip:

https://pip.pypa.io
https://packaging.python.org

I recommend Christoph Gohlke's MKL builds of NumPy and SciPy:

http://www.lfd.uci.edu/~gohlke/pythonlibs
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Python IDLE for Windows 10, 64-bit

[eryk sun]

On Wed, Sep 21, 2016 at 9:21 PM, Alan Gauld via Tutor  wrote:
> On Windows it is usually installed by default at:
>
> %PYTHONDIR%/Lib/idelib/idle.bat

With Python 3, you can run the package as a script:

pythonw -m idlelib

Python 2:

pythonw -m idlelib.idle
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] help with Numpy

[eryk sun]

On Wed, Sep 21, 2016 at 10:35 PM, Oscar Benjamin
 wrote:
> I would have given the same advice a year or so ago. It's worth noting
> though that the situation with pip, wheel etc has improved significantly
> recently. It's now straightforward to pip install Numpy, scipy, matplotlib
> and many others on Windows, OSX and Linux. Try it!

There are NumPy and Matplotlib wheels for Windows on PyPI, but not for
SciPy. Building SciPy from source on Windows is not straightforward.
If you try to build with just VS 2015 installed, it fails because the
required LAPACK/BLAS libs aren't available, and there's no Fortran
compiler (Intel MKL is not free). For Windows users, I recommend
downloading the wheels for NumPy and SciPy from Gohlke's site -- or
just use a batteries-included distro as Alan recommended.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] using python shell program on windows

[eryk sun]

On Fri, Oct 14, 2016 at 4:52 PM, Peter Otten <__pete...@web.de> wrote:
>
> python -m pip install timesheet
>
> on the commandline should take care of the details. On my (Linux) machine
> this also installed a script that I can invoke on the command line with
>
> $ timesheet start foo bar
> Started task:
> Subject foo
> Title   bar
> Start   2016-10-14 18:49
> End
> Duration00:00
> $
>
> If that does not work on Windows there's probably a Windows user who knows
> what to try instead.

I haven't used this package before, but I can attest that the wheel
and its dependencies do install without error on Windows, and it
appears to be working.

It says it supports Python 3, but I discovered it has a raw_input call
in the implementation of its start command. So I suggest using Python
2 instead.

The installation creates a timesheet.exe script wrapper in Python's
"Scripts" folder. Unless you're using a virtual environment, you'll
have to add this directory to the PATH environment variable to run
timesheet from the command line.

Running timesheet warns about a missing configuration file,
"%LOCALAPPDATA%\timesheetrc" (expanded for your %LOCALAPPDATA%
directory). I created this as an empty file to avoid the warning.

If you need to back up or delete the timesheet database, it's located
at "%LOCALAPPDATA%\timesheet\timesheet.sqlite".
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Python 3.5 installation beside Python 2.3 on Windows 7

[eryk sun]

On Sun, Oct 23, 2016 at 1:39 AM, Pierre-Michel Averseng
 wrote:
> Le 22/10/2016 à 20:00, tutor-requ...@python.org a écrit :
>
>> Do I have to completely uninstall Python 2.7.1 and then install Python
>> 3.5.2? Or can I still install Python 3.5.2 keeping Python 2.7.1 untouched
>> on my laptop?
>
> For Windows (including XP, Vista, 7, and 8), Python comes as a
> self-installer MSI program file—simply double-click on its file icon, and 
> answer
> Yes or Next at every prompt to perform a default install. The default install
> includes Python’s documentation set and support for tkinter (Tkinter in Python
> 2.X) GUIs, shelve databases, and the IDLE development GUI. Python 3.3 and
> 2.7 are normally installed in the directories C:\Python33 and C:\Python27
> though this can be changed at install time.

3.5 doesn't support Windows XP. It also switched to using executable
installers that are built using the WiX toolset. The web installer
downloads up to 22 MSI packages on demand. Even the 'offline'
installer has to download the MSIs for the debug binaries, since
they're not commonly installed and including them would make the
offline installer substantially larger.

Also, the default installation targets have changed to
"%ProgramFiles[(x86)]%\Python35[-32]" for all-users installs and
"%LocalAppData%\Programs\Python\Python35[-32]" for per-user installs.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Q regarding external program calling

[eryk sun]

On Sun, Nov 6, 2016 at 9:09 AM, Alan Gauld via Tutor  wrote:
> On 06/11/16 01:44, Clayton Kirkwood wrote:
>> Looked all over, but haven't found the answer. If I have a (windows) program
>> which I wish to start, even shell scripts, and possibly capture the output
>> from, how do I do that?
>
> Others have already pointed you to the subprocess module.
> The documentation there includes several examples.
>
> On Windows it sometimes helps to start a program via the
> "start" command.

`start` is a cmd shell built-in command, so it requires passing
shell=True to subprocess.Popen. It allows more control over how a
process is started -- such as whether to use the current or a new
console, the window state, the process working directory, priority,
and CPU affinity -- and whether the shell should wait on the process.
Popen directly supports most of what `start` does, and otherwise you
can use os.startfile, which is generally safer than using the shell.

> start notepad.exe

Using `start` like this is typical in a batch file, to avoid waiting
for the command to exit. In contrast, Popen defaults to asynchronous
mode, and waiting requires the wait() or communicate() methods.

> start myfile.txt

`start` isn't generally necessary here. Both with and without `start`,
cmd tries to locate and execute (open) the file.

If the target is a batch file (i.e. .bat or .cmd), then cmd either
executes it in place or, if the `start` command is used, executes it
in a new shell via `cmd.exe /k`. Otherwise cmd first tries to execute
the target via CreateProcess(). If the latter fails, cmd tries
ShellExecuteEx(), using the default file association.

For example, the .py extension is usually associated with the ProgId
"Python.File". If the py launcher is installed, then the default
action for this ProgId is typically the template command
'C:\Windows\py.exe "%1" %*'. ShellExecuteEx substitutes the target
file and command-line arguments for the %1 and %* template parameters
and then calls CreateProcess.

The `start` command expands the set of possible targets to whatever
ShellExecuteEx supports, including directories and virtual targets.
For example:

* Opening directories in Explorer
* Opening `shell:` folders such as shell:UserProgramFiles
* Executing "App Paths" commands registered under
  [HKCU|HKLM]\Software\Microsoft\Windows\CurrentVersion\App Paths

> If your program has a GUI however, accessing the output
> becomes much more difficult and may not even be
> possible. In that case you may need to resort
> to driving it programmatically via its API using
> a tool like pywin32 or ctypes.

The best-case scenario is an app that exports a COM IDispatch
interface for scripting, which you can access via PyWin32's win32com
module.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Trouble Launching Python

[eryk sun]

On Mon, Dec 19, 2016 at 10:38 PM, Joseph Olugbohunmi via Tutor
 wrote:
> Hello,Good day, I installed Python 3.5.2 on my Windows 8.1 PC and then I 
> tried launching
> IDLE as well as the Interpreter but I got a message that 
> api-ms-win-crt-runtime-l1-1-0.dll was
> missing. I downloaded and installed that after which I got another message 
> that
> api-ms-win-crt-math-l1-1-0.dll was also missing, I got that, and then another 
> dll was missing
> and it goes on and on. Please what can I do?

The implementation of Python that you're using is written in C and
relies on the C runtime library to provide a measure of cross-platform
portability. The latest C runtime from Microsoft is called the
"Universal CRT". It's an operating system component. As such, you
should already have ucrtbase.dll and all of the API set DLLs, such as
api-ms-win-crt-runtime-l1-1-0.dll, assuming your system is up to date
via Windows Update. If for some reason your system doesn't have this
update, you can manually download and install it from the following
link:

https://www.microsoft.com/en-us/download/details.aspx?id=50410

Install either "Windows8.1-KB3118401-x64.msu" (64-bit) or
"Windows8.1-KB3118401-x86.msu" (32-bit), depending on whether your
version of Windows is 64-bit or 32-bit.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Trouble Launching Python

[eryk sun]

On Tue, Dec 20, 2016 at 7:12 PM, George Fischhof  wrote:
> 2016-12-19 23:38 GMT+01:00 Joseph Olugbohunmi via Tutor :
>
> this is a Microsoft visual c++ redistributable is missing from system.
> I found this stackoverflow link which seems to describing a solution
>
> http://stackoverflow.com/questions/33265663/
> api-ms-win-crt-runtime-l1-1-0-dll-is-missing-when-opening-microsoft-office-file

There is no reason in this case to install the VC++ redistributable
package. Python is written in C, so it only needs the C runtime, which
is now an OS component. I provided a link for the updated download for
the Universal C Runtime, KB3118401. The above Stack Overflow answer
mentions KB2999226, which is the old, outdated update from well over a
year ago.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Open a libreoffice calc file in Python

[eryk sun]

On Thu, Dec 22, 2016 at 4:50 AM,   wrote:
> BTW, the array form is Popen's default mode; sensibly you need to _ask_ to
> use a shell string with shell=True, because that is harder and more fragile.

Without shell=True, args as a string on POSIX is generally an error
because it will look for the entire string as the executable. The
exception is if the string has no command-line arguments (e.g.
Popen('ls')).

On Windows it's always acceptable to pass args as a string. For a
complex command line it may be easier to pass args as a list and let
Popen call list2cmdline to create a string that's properly quoted and
escaped. Here 'properly' assumes the target executable parses its
command line using VC++ rules. It may use custom rules, in which case
you have to pass args as a string.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Open a libreoffice calc file in Python

[eryk sun]

On Thu, Dec 22, 2016 at 4:20 PM, boB Stepp  wrote:
>
> Both you and Eryk seem to be speaking in terms of using
> subprocess.Popen() directly.  So I think I need some clarification.
> At 
> https://docs.python.org/3/library/subprocess.html#using-the-subprocess-module
> it says:
>
> "The recommended approach to invoking subprocesses is to use the run()
> function for all use cases it can handle. For more advanced use cases,
> the underlying Popen interface can be used directly.

The high-level functions such as `run` pass their positional arguments
and Popen keyword arguments to the Popen constructor. Everything
stated about handling args as a list or string and shell=True applies
the same to the high-level API.

> My current understanding as to why the subprocess module is preferred
> to using the older os.system() is to avoid shell injection attacks.
> So my assumption is that even when using "shell=True" with either
> run() or Popen(), this is avoided.  Is this true?

Using shell=True is insecure. The command-line string is passed
directly to the shell, the same as with os.system. Try to avoid using
shell=True as much as possible, especially when the command is based
on user input.

But the subprocess module has more to offer other than just avoiding
the shell. For example, it allows communicating directly with the
child process using standard I/O (stdin, stdout, stderr) pipes;
controlling inheritance of Unix file descriptors or Windows handles;
and creating a new Unix session or Windows process group.

On Windows, Popen also allows passing creationflags [1] (e.g.
CREATE_NEW_CONSOLE, CREATE_NO_WINDOW, DETACHED_PROCESS) and a subset
of the process startupinfo [2], including the standard handles and
wShowWindow.

[1]: https://msdn.microsoft.com/en-us/library/ms684863
[2]: https://docs.python.org/3/library/subprocess.html#subprocess.STARTUPINFO

> Are there subtleties as to when to use run() and when to use Popen()?

The high-level API executes a child process synchronously -- i.e.
write some (optional) input, read the output, close the standard I/O
files, and wait for the process to exit. For example, here's the
implementation of `run` in 3.5:

def run(*popenargs, input=None, timeout=None, check=False, **kwargs):
if input is not None:
if 'stdin' in kwargs:
raise ValueError(
'stdin and input arguments may not both be used.')
kwargs['stdin'] = PIPE
with Popen(*popenargs, **kwargs) as process:
try:
stdout, stderr = process.communicate(
input, timeout=timeout)
except TimeoutExpired:
process.kill()
stdout, stderr = process.communicate()
raise TimeoutExpired(process.args, timeout,
 output=stdout, stderr=stderr)
except:
process.kill()
process.wait()
raise
retcode = process.poll()
if check and retcode:
raise CalledProcessError(retcode, process.args,
 output=stdout, stderr=stderr)
return CompletedProcess(process.args, retcode, stdout, stderr)

In most cases the high-level API is all that you'll need. Rarely you
may need to execute a child process asynchronously and interact with a
long-running process. In that case call Popen directly.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] ReadableInt

[eryk sun]

On Fri, Dec 30, 2016 at 4:46 PM, Albert-Jan Roskam
 wrote:
>
> Remember I only intend to use it while debugging the given script.

Have you tried hooking sys.displayhook?

> Why does the call to str() below return '1' and not 'one'? Should I implement 
> __new__
> because int is immutable?

You can't reliably replace a built-int type with a late binding. The
concrete C API (in this case PyLong_FromLong) is called directly by
the compiler:

>>> str(1)

Breakpoint 0 hit
python35_d!PyAST_FromNodeObject:
`5ea92ad0 4c894c2420  mov qword ptr [rsp+20h],r9
  ss:0021`b39ef2d8=
  {python35_d!_PyParser_Grammar
   (`5ed84ad8)}

0:000> bp python35_d!PyLong_FromLong
0:000> g
Breakpoint 1 hit
python35_d!PyLong_FromLong:
`5e95fee0 894c2408mov dword ptr [rsp+8],ecx
  ss:0021`b39ee9e0=c19956a8

0:000> kc 0n15
Call Site
python35_d!PyLong_FromLong
python35_d!parsenumber
python35_d!ast_for_atom
python35_d!ast_for_atom_expr
python35_d!ast_for_power
python35_d!ast_for_expr
python35_d!ast_for_call
python35_d!ast_for_trailer
python35_d!ast_for_atom_expr
python35_d!ast_for_power
python35_d!ast_for_expr
python35_d!ast_for_testlist
python35_d!ast_for_expr_stmt
python35_d!ast_for_stmt
python35_d!PyAST_FromNodeObject

0:000> r rcx
rcx=0001

0:000> bd 0-1; g
'1'
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Best Python 3 module to create simple text editor in Windows command prompt?

[eryk sun]

On Fri, Feb 3, 2017 at 2:22 AM, boB Stepp  wrote:
> What would be the best Python 3 module to best assist me in controlling
> the command prompt window display, font colors, positioning the cursor
> dynamically, etc.?

Try using curses [1]. Christoph Gohlke has a port for Windows Python
based on the PDCurses library [2].

[1]: https://docs.python.org/3/howto/curses.html
[2]: http://www.lfd.uci.edu/~gohlke/pythonlibs/#curses
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] sort() method and non-ASCII

[eryk sun]

On Sun, Feb 5, 2017 at 3:52 AM, boB Stepp  wrote:
> Does the list sort() method (and other sort methods in Python) just go
> by the hex value assigned to each symbol to determine sort order in
> whichever Unicode encoding chart is being implemented?

list.sort uses a less-than comparison. What you really want to know is
how Python compares strings. They're compared by ordinal at
corresponding indexes, i.e. ord(s1[i]) vs ord(s2[i]) for i less than
min(len(s1), len(s2)).

This gets a bit interesting when you're comparing characters that have
composed and decomposed Unicode forms, i.e. a single code vs multiple
combining codes. For example:

>>> s1 = '\xc7'
>>> s2 = 'C' + '\u0327'
>>> print(s1, s2)
Ç Ç
>>> s2 < s1
True

where U+0327 is a combining cedilla. As characters, s1 and s2 are the
same. However, codewise s2 is less than s1 because 0x43 ("C") is less
than 0xc7 ("Ç"). In this case you can first normalize the strings to
either composed or decomposed form [1]. For example:

>>> strings = ['\xc7', 'C\u0327', 'D']
>>> sorted(strings)
['Ç', 'D', 'Ç']

>>> norm_nfc = functools.partial(unicodedata.normalize, 'NFC')
>>> sorted(strings, key=norm_nfc)
['D', 'Ç', 'Ç']

[1]: https://en.wikipedia.org/wiki/Unicode_equivalence#Normal_forms
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] sort() method and non-ASCII

[eryk sun]

On Sun, Feb 5, 2017 at 10:30 PM, boB Stepp  wrote:
> I was looking at http://unicode.org/charts/  Because they called them
> charts, so did I.  I'm assuming that despite this organization into
> charts, each and every character in each chart has its own unique
> hexadecimal code to designate each character.

Those are PDF charts (i.e. tables) for Unicode blocks:

https://en.wikipedia.org/wiki/Unicode_block

A Unicode block always has a multiple of 16 codepoints, so it's
convenient to represent the ordinal values in hexadecimal.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Test for type(object) == ???

[eryk sun]

On Sat, Feb 11, 2017 at 1:34 AM, boB Stepp  wrote:
> I was playing around with type() tonight.  If I type (pun intended), I get:
>
> py3: type(5)
> 

`type` is a metaclass that either creates a new class (given 3
arguments: name, bases, and dict) or returns a reference to the class
of an existing object. type(5) is the latter case, and it returns a
reference to the class `int`. What you see printed in the REPL is
repr(int), a string representation of the class object:

 >>> repr(int)
 ""

Speaking of classes and metaclasses, note that you can't call
int.__repr__(int) to get this representation, because the __repr__
special method of int is meant for instances of int such as int(5).
Instead you have to explicitly use __repr__ from the metaclass, i.e.
`type`:

>>> type(int)

>>> type.__repr__(int)
""

But just use built-in repr().
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Test for type(object) == ???

[eryk sun]

On Sat, Feb 11, 2017 at 4:32 AM, boB Stepp  wrote:
>
> This bit got me experimenting.  Since the integer "5" is an integer
> object instance, I am wondering why I can't do:
>
> py3: 5.__repr__()
>   File "", line 1
> 5.__repr__()
>  ^
> SyntaxError: invalid syntax
>
> , but I can do:
>
> py3: x = 5
> py3: x.__repr__()
> '5'

The parser sees "5." as a floating point number. You can use
parentheses to force it to parse 5 as an integer:

>>> (5).__repr__()
'5'

> But in class examples I've seen posted, I do not recall __repr__ ever
> being defined.  So I infer that most of the time I should not or need not
> do so, but under what circumstances should I do so?

Classes inherit a default __repr__ from `object`. For example:

>>> object.__repr__(5)
''

Define a custom __repr__ when it's useful for debugging to include
additional information. It also gets called for str() if you don't
implement a custom __str__.

> Another curiosity question.  If I type:
>
> py3: repr(int)
> ""
>
> I get what I expect, but if I do the same with my custom class, "boB",
> I instead get:
>
> py3: repr(boB)
> ""
>
> Why the difference between the Python class "int" and my custom class
> "boB"?  Did I not define __repr__() correctly?

The __repr__ defined by your class is used for instances of the class,
not for the class itself. In the above you're seeing the return value
from the metaclass __repr__ that's defined by `type`:

In CPython, type.__repr__ is implemented in C as type_repr in
Objects/typeobject.c. If it can determine the module and qualified
name of a class, then it uses the template "".
Otherwise it uses the template "" with the basic char
*tp_name from the PyTypeObject.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Test for type(object) == ???

[eryk sun]

On Sat, Feb 11, 2017 at 3:22 AM, boB Stepp  wrote:
>
> py3: help(repr)
> Help on built-in function repr in module builtins:
>
> repr(obj, /)
> Return the canonical string representation of the object.
>
> For many object types, including most builtins, eval(repr(obj)) == obj.
>
> Question:  What does the forward slash represent in this context?

It's from the function's __text_signature__.

>>> repr.__text_signature__
'($module, obj, /)'

It means "obj" is a positional-only argument that cannot be passed as a keyword.

>>> s = inspect.signature(repr)
>>> s.parameters['obj'].kind
<_ParameterKind.POSITIONAL_ONLY: 0>

CPython has a lot of built-in functions that don't allow keyword arguments.

>>> repr(obj=42)
Traceback (most recent call last):
  File "", line 1, in 
TypeError: repr() takes no keyword arguments

Look at open() for comparison.

>>> open.__text_signature__
"($module, /, file, mode='r', buffering=-1, encoding=None,\n
errors=None, newline=None, closefd=True, opener=None)"

All of its parameters can be passed as keyword arguments.

>>> s = inspect.signature(open)
>>> s.parameters['file'].kind
<_ParameterKind.POSITIONAL_OR_KEYWORD: 1>
>>> s.parameters['mode'].kind
<_ParameterKind.POSITIONAL_OR_KEYWORD: 1>

For example:

>>> open(file='spam', mode='w')
<_io.TextIOWrapper name='spam' mode='w' encoding='UTF-8'>
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Test for type(object) == ???

[eryk sun]

On Sat, Feb 11, 2017 at 6:22 AM, boB Stepp  wrote:
> On Fri, Feb 10, 2017 at 11:49 PM, eryk sun  wrote:
>
>> It's from the function's __text_signature__.
>>
>> >>> repr.__text_signature__
>> '($module, obj, /)'
>>
>> It means "obj" is a positional-only argument that cannot be passed as a 
>> keyword.
>
> [snip]
>
>> Look at open() for comparison.
>>
>> >>> open.__text_signature__
>> "($module, /, file, mode='r', buffering=-1, encoding=None,\n
>> errors=None, newline=None, closefd=True, opener=None)"
>
> In your comparison example, a forward slash appears again, but this
> time after "$module" in the location where "obj" previously appeared
> in the repr example.  Is it indicating a similar use here?

All parameters listed before the slash are position-only. Everything
after the slash can be passed as a keyword argument.

> Can you provide a link to "__text_signature__" in the docs?  I tried
> searching for it, but came up empty.

No, I can't. It was added in issue 19674.

http://bugs.python.org/issue19674

The '/' syntax for positional-only arguments is documented in PEP 457.

https://www.python.org/dev/peps/pep-0457

Also see the how-to for CPython's "Argument Clinic" preprocessor,
since (for now) positional-only arguments are only a concern for
built-in functions.

https://docs.python.org/3/howto/clinic.html
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Test for type(object) == ???

[eryk sun]

On Sat, Feb 11, 2017 at 7:35 AM, boB Stepp  wrote:
> Has this PEP been implemented yet?  I am running Python 3.5.2 and it
> appears not to work.  Also, in "What's New In Python 3.6"
> (https://docs.python.org/3/whatsnew/3.6.html) I did not see a mention
> of it.

You can see in the document header that PEP 457 is an Informational
document at the draft stage, as opposed to a Standards Track document
that's finalized. It's the best I could find as provisional
documentation of CPython's usage of '/' to mark positional-only
arguments of built-in functions. FYI, the PEP's author, Larry
Hastings, is also the author of Argument Clinic.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Python-list thread: int vs. float

[eryk sun]

On Sat, Feb 11, 2017 at 8:06 AM, Steven D'Aprano  wrote:
> Valid digits for integers include 0 through 9 in decimal

Note that Python 3 uses the Unicode database to determine the decimal
value of characters, if any. It's not limited to the ASCII decimal
digits 0-9. For example:

>>> s
'௧꘢୩'
>>> int(s)
123
>>> print(*(unicodedata.name(c) for c in s), sep='\n')
TAMIL DIGIT ONE
VAI DIGIT TWO
ORIYA DIGIT THREE
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Python-list thread: int vs. float

[eryk sun]

On Sat, Feb 11, 2017 at 8:53 PM, boB Stepp  wrote:
>
> I suspect Eryk had set a normal 's' as an identifier for the character
> code sequence that produces the non-ASCII output, but forgot to show
> us that step.  But I could be mistaken.

Sorry, I forgot to show the assignment of the string "௧꘢୩" to the variable s.

> Today I am working in Windows 7, not Linux Mint.  Of course when I
> attempted to copy and paste the non-ASCII sequence from Gmail into
> cmd.exe I got 3 rectangular boxes on the paste line, indicating
> cmd.exe could not translate those characters.

If you're running python.exe, it either inherits or allocates a
console, which is hosted by an instance of conhost.exe. At most you're
inheriting a console from cmd.exe. That's the extent of its
involvement. Think of conhost.exe as a terminal window, like GNOME
Terminal, etc. cmd.exe and powershell.exe are shells that use standard
I/O, much like Python's REPL -- or like running bash in a Unix
terminal.

The console window can handle characters in the basic multilingual
plane (BMP). Python 3.6 uses the console's wide-character (Unicode)
API, so it should have no problem reading the string "௧꘢୩" if typed or
pasted into the console. For example, if it's assigned to `s`, then
ascii(s) should show the correct \u literals:

>>> ascii(s)
"'\\u0be7\\ua622\\u0b69'"

Unfortunately, for rendering text the Windows console has limited font
support. It requires a strictly monospace font, among other criteria.
It won't switch automatically to other fonts when the current font
doesn't have a glyph for a character. It just displays an empty box
glyph. In Windows 7 a good font choice for the console window is
Consolas. It has pretty good coverage for Western languages, but not
for any of the characters in "௧꘢୩".

If you copy the empty-box glyphs from the console window to the
clipboard, it's actually copying the Unicode characters, which can be
pasted into a window that has better font support, such as notepad. So
in 3.6 this is just a superficial rendering issue that can be
addressed by using programs such as ConEmu that replace the standard
console window (it hides the console and hooks the API to write to its
own window).
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Exponential function

[eryk sun]

On Tue, Feb 14, 2017 at 11:01 PM, Alan Gauld via Tutor  wrote:
> To compute it if you don't know x in advance then yes,
> use something like
>
> value = 10**x
>
> But if you know the value in advance you can write it in
> a more compact form as:
>
> value = 1e5  # or 3e7 or whatever...

10**5 is an int and 1e5 is a float. This could lead to a significant
loss of precision depending on the calculation. For example:

>>> x = 10**5 * 1234567890123456789
>>> y = 1e5 * 1234567890123456789
>>> x - y
16777216.0

Replacing 10**5 with 10 is a compile-time optimization (constant
expression folding), so you needn't worry about manually optimizing
it. For example:

>>> compile('10**5', '', 'exec').co_consts
(10, 5, None, 10)

The compiled bytecode refers to the pre-computed constant:

>>> dis.dis(compile('10**5', '', 'exec'))
  1   0 LOAD_CONST   3 (10)
  3 POP_TOP
  4 LOAD_CONST   2 (None)
  7 RETURN_VALUE
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess.Popen / proc.communicate issue

[eryk sun]

On Thu, Mar 30, 2017 at 10:51 PM, Cameron Simpson  wrote:
> This suggests that .communicate uses Threads to send and to gather data
> independently, and that therefore the deadlock situation may not arise.

For Unix, communicate() uses select or poll. It uses threads on
Windows. Either way it avoids deadlocking.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] reading files in Python 3

[eryk sun]

On Thu, Mar 30, 2017 at 8:47 PM, Zachary Ware
 wrote:
> In this case, the problem is the bogus Unicode escape that you
> inadvertently included in your path: `\Us...`.  To fix it, either use a
> 'raw' string (`r"C:\Users\..."`) or use forward slashes rather than
> backslashes, which Windows is happy to accept.

Forward slash is acceptable when all you need to support is DOS-style
paths that are limited to DOS semantics and MAX_PATH. A raw string
works, except not for paths ending in backslash and not for unicode
strings in Python 2. A generic solution is to use a pathlib.Path() in
3.x, which normalizes the path to use backslash, or os.path.normpath()
in 2.x.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] reading files in Python 3

[eryk sun]

On Thu, Mar 30, 2017 at 8:45 PM, Mats Wichmann  wrote:
> Yeah, fun.  You need to escape the \ that the idiot MS-DOS people chose
> for the file path separator. Because \ is treated as an escape character.

The COMMAND.COM shell inherited command-line switches (options) that
use slash from TOPS-10 by way of CP/M, so using backslash in paths was
less ambiguous for the shell (e.g. dir/w could be intended to run "w"
in the "dir" subdirectory, or it could mean to run "dir" with the
option "/w"). The DOS kernel did support both slash and backslash in
file-system paths.

Also, C wasn't a common language on the PC back then. BASIC was.
Instead of using escapes in string literals, BASIC used addition at
runtime with the CHR$ function or predefined constants. Support for
hierarchical paths (DOS 2.0) came around at about the same time that C
was rising in popularity, so the pain came on slowly like boiling a
lobster.

The system designers who really have no excuse are the NT kernel
developers circa 1988-93. They were working in C on a system that
already required converting paths from the DOS namespace to a native
object namespace. They could have easily implemented the native object
system to use slash instead of backslash.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Count for loops

[eryk sun]

On Wed, Apr 12, 2017 at 3:40 AM, boB Stepp  wrote:
>
> I have to say I am surprised by this as well as the OP.  I knew that
> str() in general makes a nice printable representation

The single-argument str() constructor calls the object's __str__
method (or __repr__ if __str__ isn't defined). In Python 3,
float.__str__ and float.__repr__ behave the same. They use as many
digits as is required to round trip back to the float value exactly.
That's up to 17 digits.

>>> str(1.2345678901234567)
'1.2345678901234567'
>>> str(1.)
'1.0'

In Python 2, float.__str__ uses up to 12 digits:

>>> str(1.2345678901234567)
'1.23456789012'

> realize that using str() on an arbitrarily typed in "float-like" value
> would convert the typed in value to a normal float's max precision.

For a literal floating-point value in source code, the compiler
(component of the interpreter) first parses a NUMBER node:

>>> e = parser.expr('3.14159265358979323846264338327950288419716939')
>>> p = parser.st2list(e)
>>> while p[0] != token.NUMBER:
... p = p[1]
...
>>> p
[2, '3.14159265358979323846264338327950288419716939']

Next it transforms this concrete syntax tree into a abstract syntax tree:

>>> a = ast.parse('3.14159265358979323846264338327950288419716939',
...   mode='eval')
>>> ast.dump(a)
'Expression(body=Num(n=3.141592653589793))'

>>> a.body.n
3.141592653589793
>>> type(a.body.n)


You see above that the AST transforms the NUMBER node into a Num node
that references a float object. The value of the float is the closest
possible approximation of the source code literal as a
double-precision binary float.

If the compiler instead used Decimal objects, then it could retain all
of the literal's precision. Double-precision binary floats are fast
and efficient by virtue of hardware support, but that's not a
compelling reason in Python. Maybe some future version of Python will
switch to using Decimals for floating-point literals.

The final step is to compile this AST into bytecode and create a code
object. The float value is referenced in the code object's co_consts
attribute:

>>> c = compile(a, '', 'eval')
>>> c.co_consts
(3.141592653589793,)

The code in this case is simple; just load and return the constant value:

>>> dis.dis(c)
  1   0 LOAD_CONST   0 (3.141592653589793)
  3 RETURN_VALUE

>>> eval(c)
3.141592653589793
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Count for loops

[eryk sun]

On Wed, Apr 12, 2017 at 4:03 AM, boB Stepp  wrote:
>
> I have not used the decimal module (until tonight).  I just now played
> around with it some, but cannot get it to do an exact conversion of
> the number under discussion to a string using str().

Pass a string to the constructor:

>>> d = decimal.Decimal('3.14159265358979323846264338327950288419716939')
>>> str(d)
'3.14159265358979323846264338327950288419716939'

When formatting for printing, note that classic string interpolation
has to first convert the Decimal to a float, which only has 15 digits
of precision (15.95 rounded down).

>>> '%.44f' % d
'3.14159265358979311599796346854418516159057617'
>>> '%.44f' % float(d)
'3.14159265358979311599796346854418516159057617'

The result is more accurate using Python's newer string formatting
system, which allows types to define a custom __format__ method.

>>> '{:.44f}'.format(d)
'3.14159265358979323846264338327950288419716939'
>>> format(d, '.44f')
'3.14159265358979323846264338327950288419716939'
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] classproperty for Python 2.7 (read-only enough)

[eryk sun]

On Wed, Apr 19, 2017 at 10:19 AM, Peter Otten <__pete...@web.de> wrote:
> Steven D'Aprano wrote:
>
>> As I said, I haven't had a chance to try Peter's code, so it's possible
>> that he's solved all these problems. I'm judging by previous
>
> No, my simple code only "works" for read-only properties and only as long as
> you don't overwrite the property by assigning to the attribute. To disallow
> writing you can define a
>
> def __set__(*args):
> raise AttributeError
>
> method, but that would only be invoked for instances of the class, not the
> class itself. For the same reason the setter of a read/write class property
> following that design would only be invoked in an instance, not in the
> class.
>
> The other simple solution, defining a normal property in the metaclass,
> works for the class (as expected, remember that the class is an instance of
> the metaclass), but is invisible to the instance:
>
 class T(type):
> ... @property
> ... def foo(self): return 42
> ...
 class Foo:
> ... __metaclass__ = T
> ...
 Foo.foo
> 42
 Foo.foo = "bar"
> Traceback (most recent call last):
>   File "", line 1, in 
> AttributeError: can't set attribute
 Foo().foo
> Traceback (most recent call last):
>   File "", line 1, in 
> AttributeError: 'Foo' object has no attribute 'foo'

This is a bit awkward, but you can take advantage of property() being
a data descriptor. When the attribute is looked up on the class, the
metaclass __getattribute__ or __setattr__ first searches the metaclass
MRO. If it finds a data descriptor, then it uses it unconditionally.
This means you can add a foo property to the class as well and have it
chain to the metaclass property. For example:

class T(type):
@property
def foo(self):
return 42

class C(object):
__metaclass__ = T
@property
def foo(self):
return type(self).foo

>>> C.foo
42
>>> C.foo = 0
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: can't set attribute
>>> o = C()
>>> o.foo
42
>>> o.foo = 0
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: can't set attribute
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Sets question

[eryk sun]

On Thu, Apr 27, 2017 at 12:33 AM, Phil  wrote:
> Another question I'm afraid.
>
> If I want to remove 1 from a set then this is the answer:
>
> set([1,2,3]) - set([1])

You can also use set literals here, with the caveat that {} is
ambiguous, and Python chooses to make it an empty dict instead of a
set.

>>> {1, 2, 3} - {1}
{2, 3}

related operations:

>>> s = {1, 2, 3}
>>> s -= {1}
>>> s
{2, 3}
>>> s.remove(2)
>>> s
{3}

> Ideally, I would like {'1'} to become {1}. Try as I may, I have not
> discovered how to remove the '' marks. How do I achieve that?

The quotation marks are the representation of a string object (i.e.
str). If you want an integer object (i.e. int), you have to convert
the value of the string to an integer.

>>> x = '1'
>>> {1, 2, 3} - {int(x)}
{2, 3}
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Sets question

[eryk sun]

On Thu, Apr 27, 2017 at 1:34 AM, Phil  wrote:
> I did try {int(num)} but that resulted in an error that said something along
> the lines of int not being iterable. I'll have another look at that idea.

That exception indicates you probably used set(int(num)) instead of
either {int(num)} or set([int(num)]). The set() constructor needs an
iterable container such as a sequence (e.g. range, list, tuple, str).
An integer isn't iterable.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Difference between %f and %F string formatting?

[eryk sun]

On Thu, Apr 27, 2017 at 2:19 AM, Tim Peters  wrote:
> [boB Stepp ]
>
>>  I cannot find any discernible
>> difference between '%f' %  and '%F' %
>> .  Is there any or do they duplicate
>> functionality?  If the latter, why are there two ways of doing the
>> same thing?
>
> They differ only in the capitalization of the strings produced for
> NaNs and infinities (math.nan and math.inf in Python 3).
>
 "%f" % math.nan
> nan'
 "%F" % math.nan
> NAN'
 "%f" % math.inf
> inf'
 "%F" % math.inf
> INF'

In context this can be inferred from the docs, but I think f/F should
indicate the casing difference just like e/E, g/G, and x/X do.

https://docs.python.org/3/library/stdtypes.html#printf-style-string-formatting
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] [ctypes-users] Press ESC to exit()

[eryk sun]

On Mon, May 1, 2017 at 6:28 PM, Michael C
 wrote:
> Hi all, I found out that one way to press ESC to kill the script was to use
> my previous
> script language, AutoHotKey and this is how it works:
>
> AutoHotKey code
>  ## function that kills the window with title '*Python 3.6.1 Shell*'
>
> kill()
> {
> WinKill, *Python 3.6.1 Shell*
> }
>
> ## When ESC is pressed, runs the function 'kill'
> Esc::kill()
>
> Yes, that's the entire script.
> Is there a way to write it in Python on windows?

Instead of using a hotkey, you can set a global low-level keyboard
hook. Then optionally you can check for a particular foreground window
before handling the key. For example:

import ctypes
import win32con
import win32gui

from ctypes import wintypes

user32 = ctypes.WinDLL('user32', use_last_error=True)

VK_ESCAPE = 0x1B

LLKHF_EXTENDED  = 0x0001
LLKHF_LOWER_IL_INJECTED = 0x0002
LLKHF_INJECTED  = 0x0010
LLKHF_ALTDOWN   = 0x0020
LLKHF_UP= 0x0080

ULONG_PTR = wintypes.WPARAM
class KBDLLHOOKSTRUCT(ctypes.Structure):
_fields_ = (('vkCode',  wintypes.DWORD),
('scanCode',wintypes.DWORD),
('flags',   wintypes.DWORD),
('time',wintypes.DWORD),
('dwExtraInfo', ULONG_PTR))

HOOKPROC = ctypes.WINFUNCTYPE(wintypes.LPARAM, ctypes.c_int,
wintypes.WPARAM, wintypes.LPARAM)

user32.SetWindowsHookExW.restype = wintypes.HHOOK
user32.SetWindowsHookExW.argtypes = (
ctypes.c_int,   # _In_ idHook
HOOKPROC,   # _In_ lpfn
wintypes.HINSTANCE, # _In_ hMod
wintypes.DWORD) # _In_ dwThreadId

user32.CallNextHookEx.restype = wintypes.LPARAM
user32.CallNextHookEx.argtypes = (
wintypes.HHOOK,  # _In_opt_ hhk
ctypes.c_int,# _In_ nCode
wintypes.WPARAM, # _In_ wParam
wintypes.LPARAM) # _In_ lParam

def keyboard_hook(handler, hwnd=None):
@HOOKPROC
def hookfunc(nCode, wParam, lParam):
event = KBDLLHOOKSTRUCT.from_address(lParam)
if hwnd is not None and win32gui.GetForegroundWindow() == hwnd:
handler(event)
return user32.CallNextHookEx(hHook, nCode, wParam, lParam)

hHook = user32.SetWindowsHookExW(win32con.WH_KEYBOARD_LL, hookfunc,
None, 0)
if not hHook:
raise ctypes.WinError(ctypes.get_last_error())

win32gui.PumpMessages()

if __name__ == '__main__':
import msvcrt
import threading
import win32console

print('Press escape to quit')

def escape_handler(event):
if event.vkCode == VK_ESCAPE:
# kill the hook thread
win32gui.PostQuitMessage(0)
elif not (event.flags & LLKHF_UP):
print('Virtual Key Code: {:#04x} [{}]'.format(event.vkCode,
event.time))

t = threading.Thread(target=keyboard_hook, args=(escape_handler,
win32console.GetConsoleWindow()), daemon=True)
t.start()

# consume keyboard input
while t.is_alive():
if msvcrt.kbhit():
msvcrt.getwch()

The __main__ demo should be run as a console script. The keyboard hook
filters on the console window handle from GetConsoleWindow().
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] [ctypes-users] Press ESC to exit()

[eryk sun]

On Tue, May 2, 2017 at 3:03 AM, Michael C
 wrote:
> holy cow

The code for a global keyboard hook is a bit complex - mostly because
I had to use ctypes (properly instead of an unreliable hack). Normally
an application has one or more windows and a message loop, in which
case there's no need for such an awkward approach.

But you're asking to do this from a console application, which doesn't
own a window. The window procedure that gets keyboard input from the
system is in the console host process (conhost.exe). It processes
input as a sequence of events that it stores in its input buffer. Thus
you could use the input buffer instead of a global keyboard hook, but
that comes with its own set of problems. You're no longer getting
notified as soon as the character is typed in the window. Instead, the
escape-key monitor has to share the input buffer with the main
application.

Events can be read without removing them via PeekConsoleInput, or read
and removed via ReadConsoleInput. They can also be flushed (discarded)
via FlushConsoleInputBuffer. If your script doesn't read from the
console and the escape-key monitory only peeks the input buffer, it
will grow without bound. However, it shouldn't read or flush input
events that the main task needs, and the main task shouldn't read
events that causes the escape-key monitoring thread to miss the user
pressing escape. It needs an integrated design.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Hi all: How do I save a file in a designated folder?

[eryk sun]

On Tue, May 2, 2017 at 6:09 PM, Michael C
 wrote:
> screenshot.save("\test\missed.png")

You probably know that "\t" represents a tab in a string literal, but
there's something about working with a path that causes people to
overlook this. Windows won't overlook it. Control characters, i.e.
characters with an ordinal value below 32, are forbidden in Windows
file names.

Also, you're depending on whatever drive or UNC path is the current
directory. For example:

>>> os.chdir('C:\\')
>>> os.path.abspath('/test/missed.png')
'C:\\test\\missed.png'

>>> os.chdir('localhost\C$')
>>> os.path.abspath('/test/missed.png')
'localhost\\C$\\test\\missed.png'

A fully-qualified path must start with the drive or UNC share. For example:

>>> os.path.abspath('C:/test/missed.png')
'C:\\test\\missed.png'

>>> os.path.abspath('//localhost/C$/test/missed.png')
'localhost\\C$\\test\\missed.png'

In this case I'm using abspath() to normalize the path, which first
has Windows itself normalize the path via GetFullPathName(). This
shows what Windows will actually try to open or create, given that it
implements legacy DOS rules. For example, it ignores trailing spaces
and dots in the final path component:

>>> os.path.abspath('C:/test/missed.png ... ')
'C:\\test\\missed.png'

and DOS devices are virtually present in every directory:

>>> os.path.abspath('C:/test/nul.txt')
'.\\nul'
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Subtract a 1D Array from a 2D Array

[eryk sun]

On Sat, May 13, 2017 at 1:33 PM, Stephen P. Molnar
 wrote:
> I am using Python 3.6 in Anaconda3 Spyder IDE.
>
> I have two arrays:
>
> X   Y  Z
> a
> 0   0  0
>  2.059801   0  0
> -1.2031263.402953  0
> -1.108639   -1.5678532.715601
> -0.938564   -1.32733-2.299003
>a_c
> 0.42833750.91755 0.208299
>
> and want to subtract the value of the value of each column or the array a_c
> from the each value in the corresponding column of array c.
>
> Trying a_mm = a - a_c where a_mm is the new array results in the error
> message:
>
> ValueError: operands could not be broadcast together with shapes (3,5) (1,3)
>
> My search for a solution has not been successful.

You've presented array `a` as if it's 5x3, but the error says it's
3x5. If it were actually 5x3, there would be no problem:

import numpy as np

a = np.array([[0, 0, 0],
  [2.059801, 0, 0],
  [-1.203126, 3.402953, 0],
  [-1.108639, -1.567853, 2.715601],
  [-0.938564, -1.32733, -2.299003]])

a_c = np.array([[0.4283375, 0.91755, 0.208299]])

>>> a.shape
(5, 3)
>>> a_c.shape
(1, 3)
>>> a - a_c
array([[-0.4283375, -0.91755  , -0.208299 ],
   [ 1.6314635, -0.91755  , -0.208299 ],
   [-1.6314635,  2.485403 , -0.208299 ],
   [-1.5369765, -2.485403 ,  2.507302 ],
   [-1.3669015, -2.24488  , -2.507302 ]])

You can use the transpose of either array. For example, use `a.T` to
get the same number of columns, or use `a_c.T` to get the same number
of rows.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Python Image Library

[eryk sun]

On Wed, May 17, 2017 at 8:24 PM, Michael C
 wrote:
> from PIL import Image
>
> im = Image.open('pic.bmp')
> im.show()
>
> I ran this code and it not only opened the picture in paint, which is what
> I want, but it also opens a CMD.exe console window! how do I prevent that from
> happening?

You're probably running a .py script that's associated with py.exe or
python.exe. These executables create a new console (i.e. an instance
of the console host process, conhost.exe), if they don't inherit one
from their parent process. The new console defaults to creating a
visible window. Change the file extension to .pyw. This file extension
should be associated with pyw.exe or pythonw.exe, which will not
create a console.

FYI, the cmd shell is unrelated to that console window. Windows users
often confuse the cmd shell with the console window that it uses. I
suppose it's less confusing on Linux. Like cmd, bash will inherit the
parent's terminal/console; however, it doesn't automatically spawn a
terminal on Linux if the parent doesn't have one. (Getting that
behavior on Windows requires the DETACHED_PROCESS creation flag.)
Since Windows users typically run cmd.exe to get a command-line shell
for running programs, they associate cmd.exe with the console window.
It isn't obvious that other programs create consoles without any
associated instance of cmd.exe.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Python Image Library

[eryk sun]

On Wed, May 17, 2017 at 10:33 PM, Michael C
 wrote:
> On Wed, May 17, 2017 at 3:30 PM, eryk sun  wrote:
>
>> You're probably running a .py script that's associated with py.exe or
>> python.exe. These executables create a new console (i.e. an instance
>> of the console host process, conhost.exe), if they don't inherit one
>> from their parent process. The new console defaults to creating a
>> visible window. Change the file extension to .pyw. This file extension
>> should be associated with pyw.exe or pythonw.exe, which will not
>> create a console.
>>
>> FYI, the cmd shell is unrelated to that console window. Windows users
>> often confuse the cmd shell with the console window that it uses. I
>> suppose it's less confusing on Linux. Like cmd, bash will inherit the
>> parent's terminal/console; however, it doesn't automatically spawn a
>> terminal on Linux if the parent doesn't have one. (Getting that
>> behavior on Windows requires the DETACHED_PROCESS creation flag.)
>> Since Windows users typically run cmd.exe to get a command-line shell
>> for running programs, they associate cmd.exe with the console window.
>> It isn't obvious that other programs create consoles without any
>> associated instance of cmd.exe.
>
> Actually, that is the whole script! I didn't get used to have the cmd.exe
> window pop up at all, could it be something I did?

Changing the script's extension to .pyw didn't prevent the console
from appearing? Or did you not try it?

Also, to reiterate, the cmd shell doesn't create or own any window,
and unless something really weird is going on, there's no cmd.exe
instance associated with the console window that you're seeing. cmd
can use a console via standard I/O File handles, and usually does, but
not always. It's no different from python.exe, powershell.exe, or any
other console application. Really, there is no such thing as a "cmd
window", "python window", or "PowerShell window". Because these are
long-running shell processes (e.g. Python's REPL), people are inclined
to think in those terms, and that's fine, but would you call it a
"where.exe window", "chcp.com window", "doskey.exe window",
"whoami.exe window", "findstr.exe window"? I think not. It's clear
that these programs simply use the console; they don't own it. Neither
do shells, but in the case of shells and other long-running console
processes, we're loose with language for convenience -- as long as it
doesn't confuse our understanding of how things really work.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Python Image Library

[eryk sun]

On Thu, May 18, 2017 at 1:58 AM, Michael C
 wrote:
> when I run this, while it's called test.pyw, this pops up
>
> from PIL import Image
>
> im = Image.open('1.bmp')
> im.show()

Ok, I stand corrected. It's something weird, and most likely due to
Windows support here being an afterthought throw in just for coverage,
instead of something anyone actually cared about. The code does indeed
use os.system to show the file, which is just about the worst thing
one can do here for a Windows viewer.

>>> print(inspect.getsource(ImageShow.WindowsViewer))
class WindowsViewer(Viewer):
format = "BMP"

def get_command(self, file, **options):
return ('start "Pillow" /WAIT "%s" '
'&& ping -n 2 127.0.0.1 >NUL '
'&& del /f "%s"' % (file, file))

>>> print(inspect.getsource(ImageShow.Viewer.show_file))
def show_file(self, file, **options):
"""Display given file"""
os.system(self.get_command(file, **options))
return 1

The WindowsViewer class preferrably should override show_file to  call
ShellExecuteExW directly via ctypes, and wait (if possible) to delete
the temporary file. Or at least use subprocess.call with shell=True,
which will hide the console window. Note that being able to wait on
the image viewer is not guaranteed. For example, the image viewer in
Windows 10 is an app that cannot be waited on.

Since you already have a file on disk, you could skip PIL completely.
Just use os.startfile('1.bmp').
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] call key on_press event multiple times when key is held down

[eryk sun]

On Tue, Jul 4, 2017 at 8:50 AM, Carlton Banks  wrote:
> I am using pynput  for keyboard events

You could use an event that enables a recording loop. The on_press and
on_release callbacks of the Listener [1] would set() and clear() this
event, respectively. For example:

import threading
from pynput import keyboard

def main():
do_record = threading.Event()

def on_press(key):
if key == keyboard.Key.cmd_l:
do_record.set()

def on_release(key):
if key == keyboard.Key.cmd_l:
do_record.clear()

with keyboard.Listener(on_press=on_press,
   on_release=on_release) as listener:

do_record.wait()

frames = []

while do_record.is_set():
print('Started recording')
# record and append audio frame

print('Stopped recording')
listener.join()

[1]: http://pythonhosted.org/pynput/keyboard.html#pynput.keyboard.Listener
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Probably a silly question: how to install lxml?

[eryk sun]

On Thu, Jul 6, 2017 at 1:51 AM, Steven D'Aprano  wrote:
>
> I dunno, to me it sounds like it *didn't* work, not if it is printing
> red error messages at the end. What do they say?

lxml should install from a wheel (e.g.
lxml-3.8.0-cp36-cp36m-win_amd64.whl). There's nothing to build, so the
most likely reason for failing would be getting denied access to
modify site-packages.

> My guess is that you need Administrator privileges to install the
> packages. Or perhaps to tell pip to install as site packages in your
> home directory, but I'm not sure how to do that.

If you've installed Python for all users in the default location in
%ProgramFiles% or %ProgramFiles(x86), then you will need to use an
elevated command prompt to install to the system site-packages. A
standard user or UAC restricted administrator does not have the right
to modify this directory.

Alternatively you can install a package just for the current user with
pip's "--user" option.

You can also install packages in a virtual environment created with
the venv module, which is convenient for development.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Python __del__ method

[eryk sun]

On Tue, Jul 11, 2017 at 2:47 PM, Jia Yue Kee  wrote:
>
> Case 2: If I were to run the code in "Interactive Mode", the following output 
> will be obtained:
>
 x = Robot("Tik-Tok")
> Tik-Tok has been created!
 y = Robot("Jenkins")
> Jenkins has been created!
 z = x
 z
> <__main__.Robot object at 0x02D7E910>
 x
> <__main__.Robot object at 0x02D7E910>
 del x
 del z
 del y
> Robot has been destroyed
>
> My question being why is that "Robot has been destroyed" is only printed once 
> in Case 2
> (interactive mode) while it is printed out twice in Case 1 (script mode)?

The REPL (interactive mode) keeps a reference to the last non-None
result as builtins `_`, for convenient access to the last result. In
your case, it results from evaluating `x`. Thus you have a hidden
reference that's keeping that object alive. Enter anything except None
or _ to clear that reference.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Python __del__ method

[eryk sun]

On Wed, Jul 12, 2017 at 12:07 AM, eryk sun  wrote:
> On Tue, Jul 11, 2017 at 2:47 PM, Jia Yue Kee  
> wrote:
>>
>> Case 2: If I were to run the code in "Interactive Mode", the following 
>> output will be obtained:
>>
>>>>> x = Robot("Tik-Tok")
>> Tik-Tok has been created!
>>>>> y = Robot("Jenkins")
>> Jenkins has been created!
>>>>> z = x
>>>>> z
>> <__main__.Robot object at 0x02D7E910>
>>>>> x
>> <__main__.Robot object at 0x02D7E910>
>>>>> del x
>>>>> del z
>>>>> del y
>> Robot has been destroyed
>>
>> My question being why is that "Robot has been destroyed" is only printed 
>> once in Case 2
>> (interactive mode) while it is printed out twice in Case 1 (script mode)?
>
> The REPL (interactive mode) keeps a reference to the last non-None
> result as builtins `_`, for convenient access to the last result. In
> your case, it results from evaluating `x`. Thus you have a hidden
> reference that's keeping that object alive. Enter anything except None
> or _ to clear that reference.

Well, not just any statement. It has to evaluate to another object
except None or the Robot object that _ currently references. For
example, enter 42.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] pythonic ascii decoding!

[eryk sun]

On Mon, Jul 31, 2017 at 3:39 PM, bruce  wrote:
>
> So, is there a quick/dirty approach I can use to simply strip out the
> "non-ascii" chars. I know, this might not be the "best/pythonic" way,
> and that it might result in loss of some data/chars, but I can live
> with it for now.

Ignore or replace the non-ASCII characters. For example:

>>> print 's\xffp\xffa\xffm'.decode('ascii', 'ignore')
spam
>>> print 's\xffp\xffa\xffm'.decode('ascii', 'replace')
s�p�a�m
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] What is meaning of "/" in "pow(x, y, z=None, /)"?

[eryk sun]

On Wed, Aug 2, 2017 at 1:06 AM, boB Stepp  wrote:
> I had typed help(pow) in the interpreter and got:
>
> 
> py3: help(pow)
> Help on built-in function pow in module builtins:
>
> pow(x, y, z=None, /)
> Equivalent to x**y (with two arguments) or x**y % z (with three arguments)
>
> Some types, such as ints, are able to use a more efficient algorithm when
> invoked using the three argument form.
> 
>
> A quick scan of some of my Python books does not turn up the use of
> "/" as a function argument.  I have a nagging feeling I've read about
> this somewhere previously, but I cannot bring it to mind, and I have
> yet to stumble on a search that brings up an answer (Yet.).

We discussed this syntax several months ago:

https://mail.python.org/pipermail/tutor/2017-February/thread.html#110344
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] if __name__=='main' vs entry points: What to teach new comers?

[eryk sun]

On Wed, Aug 2, 2017 at 4:06 PM, Wolfgang Maier
 wrote:
> On 08/02/2017 04:57 PM, Steven D'Aprano wrote:
>
>> I have a module with a main() function and an "if __name__ == ..."
>> guard. Under what circumstances is that not sufficient, and I would want
>> console_scripts?
>
> If you install things using pip/setuptools and have defined a
> console_scripts entry point for it, then the corresponding wrapper
> script will be installed in whatever is considered the scripts directory
> at install time on that machine. With a bit of luck the entry point will
> thus be executable directly without any end-user intervention (like
> adding folders to $PATH and chmodding files).
> Personally, I always found it straightforward to write the wrapper
> script myself, then define this as a 'scripts' file in the package
> layout of my setup.py, but people's MMV.

For Windows, using frozen executables is preferred because
CreateProcess doesn't support shebangs. setuptools freezes entry-point
scripts with one of the following stub executables: console_scripts
(cli-32.exe, cli-64.exe) and gui_scripts (gui-32.exe, gui-64.exe).
Actually, it's better to create and install a wheel package, for which
pip uses the newer stubs from distlib: console_scripts (t32.exe,
t64.exe) and gui_scripts (w32.exe, w64.exe). Most Python 3
installations on Windows have at least two entry-point scripts:
pip.exe and easy_install.exe.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] setup.py "script" vs "console_scripts" Was: if __name__=='main' vs entry points: What to teach new comers?

[eryk sun]

On Thu, Aug 3, 2017 at 4:22 PM, Chris Warrick  wrote:
>
> Simple: `scripts` are legacy. `entry_points` are the new thing.
> There’s also a third approach: gui_scripts entry_points, which work
> the same way on Linux/*nix, but on Windows, it means that running your
> script by opening the created .exe files does not show a console
> window. Note that stdout/stderr do not work in that mode under
> Windows, which can lead to spurious application crashes.  (GUI-only
> processes cannot use stdout/stderr because they don’t have a console
> attached)

A Windows GUI executable doesn't automatically inherit or create a
console for standard I/O. (It has to manually call AttachConsole or
AllocConsole.) But it does automatically inherit the standard handle
values from the parent process (just the integer values, not the
handles themselves). If these handles are inheritable and the child is
created with handle inheritance, then standard I/O will work. For
example:

C:\Temp>echo spam | pythonw -c print(input()) >out.txt 2>&1
C:\Temp>type out.txt
spam

In the above example pythonw.exe is a GUI executable. The example
reads "spam" from stdin and prints it to stdout, which is redirected
to a file named "out.txt". For those who don't know, `>out.txt` is
shell syntax to redirect stdout to "out.txt", and `2>&1` redirects
stderr (2) to stdout (1).
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] setup.py "script" vs "console_scripts" Was: if __name__=='main' vs entry points: What to teach new comers?

[eryk sun]

On Fri, Aug 4, 2017 at 10:15 AM, Thomas Güttler
 wrote:
> Am 04.08.2017 um 02:50 schrieb Ben Finney:
>
>> Because Distutils implements only ‘scripts’, and that's not capable
>> enough for what people need so Setuptools implements entry points.
>>
>> In other words: One of them is in the standard library and does
>> something; the other is third-party and can do more.
>>
>> That answers why there are two. But maybe you wanted to ask some
>> underlying question?
>
> The underlaying question is: Imangine you are a newcomer. And there
> are two more choices. You need a guide like 'if unsure do x'. With
> other words: What is the sane default choice?

A newcomer should simply check for __name__ == "__main__". Learning
about packaging comes later.

> Chris wrote "Simple: `scripts` are legacy."
>
> You say it is the standard, and console_scripts is from a third party.

console_scripts and gui_scripts are installed by setuptools and pip,
which are developed under the umbrella of the Python Packaging
Authority [1]. The standard library also has the ensurepip [2] module
to install pip and setuptools. As third-party packages go, they're as
close to being 'standard' as you can get.

> For me "legacy" sound like "don't go this old way".

Legacy scripts aren't automatically created as console or GUI
executables when installed on Windows. Often Windows users associate
.py scripts with an editor, in which case legacy scripts aren't
executable from PATH, i.e. they have to be run as `python
legacy_script.py`, for example.

[1]: https://www.pypa.io
[2]: https://docs.python.org/3/library/ensurepip
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Difference(s) betweenPython 3 static methods with and without @staticmethod?

[eryk sun]

On Sun, Aug 6, 2017 at 11:35 PM, boB Stepp  wrote:
>
> I see no difference in result, whether I use the @staticmethod decorator or 
> not.

While a staticmethod and a function are both descriptors [1], a
staticmethod is basically a no-op descriptor. Its __get__ method
always returns its unbound callable (i.e. its __func__ attribute)
instead of returning a computed property or bound method.

Examples

>>> descr = vars(MyOtherClass)['my_other_method']
>>> descr


>>> descr.__func__


When accessed as an attribute of a class (i.e. the instance argument
passed to __get__ is None), a staticmethod returns just its __func__.

>>> descr.__get__(None, MyOtherClass)


That's effectively no different from a Python 3 function. But they
diverge when accessed as an attribute of an instance. In this case the
__get__ method of a staticmethod also simply returns its __func__,

>>> descr.__get__(MyOtherClass(), MyOtherClass)


whereas the __get__ method of a function returns a method that's bound
to the instance.

>>> func = vars(MyClass)['my_method']
>>> func


>>> func.__get__(MyClass(), MyClass)
>

[1]: https://docs.python.org/3/reference/datamodel.html#implementing-descriptors
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] How does len() compute length of a string in UTF-8, 16, and 32?

[eryk sun]

On Tue, Aug 8, 2017 at 3:20 AM, Cameron Simpson  wrote:
>
> As you note, the 16 and 32 forms are (6 + 1) times 2 or 4 respectively. This
> is because each encoding has a leading byte order marker to indicate the big
> endianness or little endianness. For big endian data that is \xff\xfe; for
> little endian data it would be \xfe\xff.

To avoid encoding a byte order mark (BOM), use an "le" or "be" suffix, e.g.

>>> 'Hello!'.encode('utf-16le')
b'H\x00e\x00l\x00l\x00o\x00!\x00'

Sometimes a data format includes the byte order, which makes using a
BOM redundant. For example, strings in the Windows registry use
UTF-16LE, without a BOM.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] setup.py "script" vs "console_scripts" Was: if __name__=='main' vs entry points: What to teach new comers?

[eryk sun]

On Wed, Aug 9, 2017 at 11:32 PM, Alan Gauld via Tutor  wrote:
> On 09/08/17 22:15, Steven D'Aprano wrote:
>>
>> This is the critical factor. How can you use *by default* something that
>> is *NOT* supplied by default?
>
> I have to agree with Steven here. Any mature language should
> ship with all the tools needed to create and distribute a
> finished program. It is to Python's shame that it currently
> fails that test because the recommended distribution framework
> is not part of the standard language package. (and the irony
> is that the tool for installing the recommended format (pip)
> is in the standard library. You can download other peoples
> stuff but you can't make your own in the same format.
> That's bad.

The standard library has ensurepip [1] to bootstrap bundled wheels for
pip and setuptools, which in turn vendor other packages such as
distlib. The wheels are already there, unless the distro or IT
department has removed them. (They're distributed with the official
Windows installers, at least.) If you want the legal details, you
could ask on the pypa-dev list [2]. I'm sure Nick Coghlan or Donald
Stufft would be happy to clarify the situation.

[1]: https://github.com/python/cpython/tree/v3.6.2/Lib/ensurepip
[2]: https://groups.google.com/forum/#!forum/pypa-dev
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] How does len() compute length of a string in UTF-8, 16, and 32?

[eryk sun]

On Fri, Aug 11, 2017 at 2:34 AM, Cameron Simpson  wrote:
>
> In files however, the default encoding for text files is 'utf-8': Python
> will read the file's bytes as UTF-8 data and will write Python string
> characters in UTF-8 encoding when writing.

The default encoding for source files is UTF-8. The default encoding
for text files is the locale's preferred encoding, i.e.
locale.getpreferredencoding(False). On Windows this is the locale's
ANSI codepage (e.g. 1252 in Western Europe). Also, the default newline
string in text files is os.linesep. On Windows this is "\r\n".
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Percentage of installations without setuptools (Was if __name__=='__main__' ...)

[eryk sun]

On Fri, Aug 11, 2017 at 6:27 PM, Alan Gauld via Tutor  wrote:
> On 11/08/17 19:13, Chris Warrick wrote:
>
>> False since Python 3.4/2.7.9. ensurepip installs Python on every new
>> Python install.
>
> Sorry Chris, that's not making sense? Do you mean ensurepip
> installs setuptools on every install? How does it do that if
> I don't have the internet connected? Does it wait for
> a connection then automatically do a download?

The Windows installer defaults to running ensurepip, which bundles
wheels for pip and setuptools. They may not be the latest versions.

> How would I tell if it is installed? Where do I look and
> for what? Because its not where I thought it would be
> - (in the libs)...

On Windows, look in "Lib\site-packages". In general you can `import
pip` and check pip.__file__. You should also have pip and pip3
commands. Run `pip3 show pip` to check the location.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] logging to cmd.exe

[eryk sun]

> cmd.exe can use cp65001 aka utf8???

CMD is a Unicode application that for the most part uses WinAPI
wide-character functions, including the console API functions (as does
Python 3.6+). There are a few exceptions. CMD uses the console
codepage when decoding batch files (line by line, so you can change
the codepage in the middle of a batch script), when writing output
from its internal commands (e.g. dir) to pipes and files (the /u
option overrides this), and when reading output from programs in a
`FOR /F` loop.

> Why does cmd.exe still use cp850?

In the above cases CMD uses the active console input or output
codepage, which defaults to the system locale's OEM codepage. If it's
not attached to a console (i.e. when run as a DETACHED_PROCESS), CMD
uses the ANSI codepage in these cases.

Anyway, you appear to be talking about the Windows console, which
people often confuse with CMD. Programs that use command-line
interfaces (CLIs) and text user interfaces (TUIs), such as classic
system shells, are clients of a given console or terminal interface. A
TUI application typically is tightly integrated with the console or
terminal interface (e.g. a curses application), while a CLI
application typically just uses standard I/O (stdin, stdout, stderr).
Both cmd.exe and python.exe are Windows console clients. There's
nothing special about cmd.exe in this regard.

Now, there are a couple of significant problems with using codepage
65001 in the Windows console.

Prior to Windows 8, WriteFile and WriteConsoleA return the number of
decoded wide characters written to the console, which is a bug because
they're supposed to return the number of bytes written. It's not a
problem so long as there's a one-to-mapping between bytes and
characters in the console's output codepage. But UTF-8 can have up to
4 bytes per character. This misleads buffered writers such as C FILE
streams and Python 3's io module, which in turn causes gibberish to be
printed after every write of a string that includes non-ASCII
characters.

Prior to Windows 10, with codepage 65001, reading input from the
console via ReadConsole or ReadConsoleA fails if the input has
non-ASCII characters. It gets reported as a successful read of zero
bytes. This causes Python to think it's at EOF, so the REPL quits (as
if Ctrl+Z had been entered) and input() raises EOFError.

Even in Windows 10, while the entire read doesn't fail, it's not much
better. It replaces non-ASCII characters with NUL bytes. For example,
in Windows 10.0.15063:

>>> os.read(0, 100)
abcαβγdef
b'abc\x00\x00\x00def\r\n'

Microsoft is gradually working on fixing UTF-8 support in the console
(well, two developers are working on it). They appear to have fixed it
at least for the private console APIs used by the new Linux subsystem
in Windows 10:

Python 3.5.2 (default, Nov 17 2016, 17:05:23)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> s = os.read(0, 100)
abcαβγdef
>>> s
b'abc\xce\xb1\xce\xb2\xce\xb3def\n'
>>> s.decode()
'abcαβγdef\n'

Maybe it's fixed in the Windows API in an upcoming update. But still,
there are a lot of Windows 7 and 8 systems out there, for which
codepage 65001 in the console will remain broken.

> I always thought 65001 was not a 'real' codepage, even though some locales 
> (e.g. Georgia) use it [1].

Codepage 65001 isn't used by any system locale as the legacy ANSI or
OEM codepage. The console allows it probably because no one thought to
prevent using it in the late 1990s. It has been buggy for two decades.

Moodle seems to have special support for using UTF-8 with Georgian.
But as far as Windows is concerned, there is no legacy codepage for
Georgian. For example:

import ctypes
kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)

LD_ACP = LOCALE_IDEFAULTANSICODEPAGE = 0x1004
acp = (ctypes.c_wchar * 6)()

>>> kernel32.GetLocaleInfoEx('ka-GE', LD_ACP, acp, 6)
2
>>> acp.value
'0'

A value of zero here means no ANSI codepage is defined [1]:

If no ANSI code page is available, only Unicode can be used for
the locale. In this case, the value is CP_ACP (0). Such a locale
cannot be set as the system locale. Applications that do not
support Unicode do not work correctly with locales marked as
"Unicode only".

Georgian (ka-GE) is a Unicode-only locale [2] that cannot be set as
the system locale.

[1]: https://msdn.microsoft.com/en-us/library/dd373761
[2]: https://msdn.microsoft.com/en-us/library/ms930130.aspx
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] logging to cmd.exe

[eryk sun]

On Tue, Sep 26, 2017 at 7:35 AM, Mats Wichmann  wrote:
> On 09/26/2017 05:22 AM, Albert-Jan Roskam wrote:
>
>> Rather than change your code can you change the codepage with the chcp
>> command?
>
> the way chcp takes effect is problematic for this:
>
> "Programs that you start after you assign a new code page use the new
> code page, however, programs (except Cmd.exe) that you started before
> assigning the new code page use the original code page. "

Some console applications only check the codepage at startup. If you
change it while the program is running, they'll encode/decode text for
the original codepage, but the console will decode/encode it for its
current codepage. That's called mojibake.

Prior to 3.6, at startup Python uses the input codepage for sys.stdin,
and the output codepage for sys.stdout and sys.stderr. You can of
course rebind sys.std* if you change the codepage via chcp.com or
SetConsoleCP() and SetConsoleOutputCP(). If you do change the
codepage, it's considerate to remember the previous value and restore
it in an atexit function.

> I think there's also a module you can use for pre-3.6, sorry too lazy to
> do a search.

It's win_unicode_console [1].

[1]: https://pypi.python.org/pypi/win_unicode_console
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] ctypes wintypes

[eryk sun]

On Tue, Oct 3, 2017 at 10:30 PM, Michael C
 wrote:
>
> I am trying to create SYSTEM_INFO structure  and MEMORY_BASIC_INFORMATION
> structure

First, avoid relying on constants, enumerations, and structures
published on MSDN. It's not always right. Get the SDK and use the
header files instead. MEMORY_BASIC_INFORMATION is defined in winnt.h,
and SYSTEM_INFO is defined in sysinfoapi.h.

MEMORY_BASIC_INFORMATION is simple. Don't worry about the
MEMORY_BASIC_INFORMATION32 and MEMORY_BASIC_INFORMATION64 versions.
Those are meant for a debugger that's reading this structure directly
from the memory of another process.

SYSTEM_INFO is a bit tricky, given the anonymous struct and union. I
prefer to nest the definitions, but you could flatten it as separate
definitions if you like. Refer to the docs for how to use _anonymous_:

https://docs.python.org/3/library/ctypes#ctypes.Structure._anonymous_

Here are the definitions. Please don't mindlessly copy and paste.
Recreate them on your own and use this example as a reference.

import ctypes
from ctypes.wintypes import WORD, DWORD, LPVOID

PVOID = LPVOID
SIZE_T = ctypes.c_size_t

# https://msdn.microsoft.com/en-us/library/aa383751#DWORD_PTR
if ctypes.sizeof(ctypes.c_void_p) == ctypes.sizeof(ctypes.c_ulonglong):
DWORD_PTR = ctypes.c_ulonglong
elif ctypes.sizeof(ctypes.c_void_p) == ctypes.sizeof(ctypes.c_ulong):
DWORD_PTR = ctypes.c_ulong

class MEMORY_BASIC_INFORMATION(ctypes.Structure):
"""https://msdn.microsoft.com/en-us/library/aa366775""";
_fields_ = (('BaseAddress', PVOID),
('AllocationBase',PVOID),
('AllocationProtect', DWORD),
('RegionSize', SIZE_T),
('State',   DWORD),
('Protect', DWORD),
('Type',DWORD))

PMEMORY_BASIC_INFORMATION = ctypes.POINTER(MEMORY_BASIC_INFORMATION)

class SYSTEM_INFO(ctypes.Structure):
"""https://msdn.microsoft.com/en-us/library/ms724958""";
class _U(ctypes.Union):
class _S(ctypes.Structure):
_fields_ = (('wProcessorArchitecture', WORD),
('wReserved', WORD))
_fields_ = (('dwOemId', DWORD), # obsolete
('_s', _S))
_anonymous_ = ('_s',)
_fields_ = (('_u', _U),
('dwPageSize', DWORD),
('lpMinimumApplicationAddress', LPVOID),
('lpMaximumApplicationAddress', LPVOID),
('dwActiveProcessorMask',   DWORD_PTR),
('dwNumberOfProcessors',DWORD),
('dwProcessorType', DWORD),
('dwAllocationGranularity', DWORD),
('wProcessorLevel',WORD),
('wProcessorRevision', WORD))
_anonymous_ = ('_u',)

LPSYSTEM_INFO = ctypes.POINTER(SYSTEM_INFO)
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] ctypes wintypes

[eryk sun]

On Thu, Oct 5, 2017 at 8:27 PM, Michael C
 wrote:
>
> How do I see the values of each field? This doesn't work.
>
> print(PMEMORY_BASIC_INFORMATION.Protect)

Create an instance of MEMORY_BASIC_INFORMATION and pass a pointer to
it via byref(). For example, the following queries the region of
memory of the VirtualQuery function itself.

kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)

MEM_COMMIT = 0x1000
PAGE_EXECUTE_READ = 0x20
PAGE_EXECUTE_WRITECOPY = 0x80

VirtualQuery = kernel32.VirtualQuery
VirtualQuery.restype = SIZE_T
VirtualQuery.argtypes = (LPVOID, PMEMORY_BASIC_INFORMATION, SIZE_T)

mbi = MEMORY_BASIC_INFORMATION()
VirtualQuery(VirtualQuery, ctypes.byref(mbi), ctypes.sizeof(mbi))

>>> mbi.AllocationBase == kernel32._handle
True
>>> mbi.AllocationProtect == PAGE_EXECUTE_WRITECOPY
True
>>> mbi.BaseAddress
140703181352960
>>> mbi.RegionSize
364544
>>> mbi.State == MEM_COMMIT
True
>>> mbi.Protect ==  PAGE_EXECUTE_READ
True
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] ctypes wintypes

[eryk sun]

On Fri, Oct 6, 2017 at 7:26 PM, Michael C
 wrote:
>
> I started out with what you gave me:
>
[...]
>
> I am trying to acquire "lpMinimumApplicationAddress" and
> "lpMaximumApplicationAddress" from system_info, so I did this,
>
>>code
> Kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)
> Kernel32.GetSystemInfo(LPSYSTEM_INFO)
> print(LPLPSYSTEM_INFO.lpMinimumApplicationAddress)

It's the same pattern as before. Create a SYSTEM_INFO instance, which
allocates the block of memory for the information, and pass
GetSystemInfo a pointer. For example:

kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)
kernel32.GetSystemInfo.restype = None
kernel32.GetSystemInfo.argtypes = (LPSYSTEM_INFO,)

sysinfo = SYSTEM_INFO()
kernel32.GetSystemInfo(ctypes.byref(sysinfo))

Here are the minimum and maximum addresses for a 64-bit process,
formatted in hexadecimal:

>>> hex(sysinfo.lpMinimumApplicationAddress)
'0x1'
>>> hex(sysinfo.lpMaximumApplicationAddress)
'0x7ffe'
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] ctypes wintypes

[eryk sun]

On Fri, Oct 6, 2017 at 7:43 PM, Michael C
 wrote:
> Sorry but I dont understand this line:
>
> mbi = MEMORY_BASIC_INFORMATION()
>
> This creates a instance of the class?

Yes, and this allocates sizeof(MEMORY_BASIC_INFORMATION) bytes at
addressof(mbi), which you pass to a function by reference via
byref(mbi).

> Also, I thought with VirtualQueryEx, what you need for it
> is a handle, which I acquire from this
> Process = Kernel32.OpenProcess(PROCESS_QUERY_INFORMATION|PROCESS_VM_READ,
> False, PID)

My example called VirtualQuery, not VirtualQueryEx. Internally
VirtualQuery calls VirtualQueryEx using the pseudo handle
(HANDLE)(-1), which refers to the current process.

> and then feed it to the function like so:
>
> VirtualQuery(Process, ctypes.byref(mbi), ctypes.sizeof(mbi))
>
> I know it doesn't work. But what are these lines for? They don't look like
> handle to me:
>
> VirtualQuery = kernel32.VirtualQuery
> VirtualQuery.restype = SIZE_T
> VirtualQuery.argtypes = (LPVOID, PMEMORY_BASIC_INFORMATION, SIZE_T)

In the above, I'm setting the function pointer's argtypes attribute to
the types of the 3 parameters that VirtualQuery takes: the target
address (i.e. LPVOID), a pointer to the buffer (i.e.
PMEMORY_BASIC_INFORMATION), and the size of the buffer (SIZE_T). This
is to allow ctypes to correctly check and convert arguments passed to
the function.

VirtualQueryEx has four parameters, starting with the handle to the
target process, hProcess. The remaining 3 are the same as
VirtualQuery.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] ctypes wintypes

[eryk sun]

On Fri, Oct 6, 2017 at 9:12 PM, Michael C
 wrote:
>
> How do I create a buffer, or rather, is a buffer just a variable?

A buffer is a block of memory for an I/O operation. For example, if
you need to read a 4-byte (32-bit) integer at an address in another
process, the 'buffer' could be ctypes.c_int32(). In general, to read
an arbitrary-sized block of memory, use ctypes.create_string_buffer()
to create a char array.

> How do I create a pointer to it?

Pass it byref().

> print('mbi.State: ',mbi.State)

Check whether mbi.State is MEM_COMMIT before trying to read it. If
it's MEM_FREE or MEM_RESERVE, then ReadProcessMemory will fail.

> buffer = ctypes.create_string_buffer(4)
> bufferSize = (ctypes.sizeof(buffer))
>
> ReadProcessMemory = Kernel32.ReadProcessMemory
>
> if ReadProcessMemory(Process, ctypes.byref(mbi), buffer, bufferSize, None):
> print('buffer is: ',buffer)
> else:
> print('something is wrong')

Don't print "something is wrong". You're capturing the thread's last
error value, so use it to raise an informative exception. For example:

if not success:
raise ctypes.WinError(ctypes.get_last_error())
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] ctypes wintypes

[eryk sun]

On Fri, Oct 6, 2017 at 10:06 PM, Michael C
 wrote:
> like this?
>
> buffer = ctypes.byref(ctypes.create_string_buffer(4))

No, the buffer is the array created by create_string_buffer, which you
pass byref(). In the following example I create a `test` buffer that
contains "spam", and I use the pseudo-handle from GetCurrentProcess
with ReadProcessMemory to read this buffer into a target `buffer`.
It's silly to do this in the current process, but it's just an
example.

import ctypes
from ctypes.wintypes import HANDLE, LPVOID

kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)

SIZE_T = ctypes.c_size_t
LPSIZE_T = ctypes.POINTER(SIZE_T)

kernel32.GetCurrentProcess.restype = HANDLE
kernel32.ReadProcessMemory.argtypes = (HANDLE, LPVOID,
LPVOID, SIZE_T, LPSIZE_T)

hProcess = kernel32.GetCurrentProcess()
test = ctypes.create_string_buffer(b'spam')
address = ctypes.addressof(test)
buffer = ctypes.create_string_buffer(4)
nread = SIZE_T()

success = kernel32.ReadProcessMemory(hProcess, address,
ctypes.byref(buffer), ctypes.sizeof(buffer),
ctypes.byref(nread))

if not success:
raise ctypes.WinError(ctypes.get_last_error())

print(buffer[:])
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] ctypes wintypes

[eryk sun]

On Fri, Oct 6, 2017 at 10:26 PM, Michael C
 wrote:
>
> base = mbi.BaseAddress
> buffer = ctypes.c_int32()
> buffer_pointer = ctypes.byref(buffer)
> ReadProcessMemory = Kernel32.ReadProcessMemory
>
> if ReadProcessMemory(Process, base, buffer_pointer, mbi.RegionSize, None):
> print('buffer is: ',buffer)
> else:
> raise ctypes.WinError(ctypes.get_last_error())

If you need to read RegionSize bytes, then you have to allocate a
buffer that's RegionSize bytes:

buffer = ctypes.create_string_buffer(mbi.RegionSize)

Or use a smaller buffer and loop until the total number of bytes read
is RegionSize.

Also, remember to check that the state is MEM_COMMIT. You cannot read
an address range that's free or reserved. It must be committed, i.e.
backed by physical storage.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] ctypes wintypes

[eryk sun]

On Fri, Oct 6, 2017 at 11:05 PM, Michael C
 wrote:
> For this read process memory, if I am trying compose a LPCVOID
> lpBaseAddress, am I not making a variable that equals to  mbi.BaseAddress,
> and then making a pointer pointing to it?
>
> start_address = mbi.BaseAddress
>  LPCVOID = ctypes.byref(start_address)

LPCVOID is a pointer type; don't use it as a variable name because
it's confusing to someone who's reading your code.

The `BaseAddress` field is an LPVOID, which is an alias for
ctypes.c_void_p. Simple C types such as c_void_p are automatically
converted to Python native types such as int, bytes, and str. It's
fine that mbi.BaseAddress is a Python int. With argtypes defined for
ReadProcessMemory, ctypes will convert the int back to a void pointer
for you automatically.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Why does os.path.realpath('test_main.py') give different results for unittest than for testing statement in interpreter?

[eryk sun]

On Sun, Jan 7, 2018 at 10:51 AM, Albert-Jan Roskam
 wrote:
>
> On Jan 7, 2018 09:08, Steven D'Aprano  wrote:
>>
>> realpath() returns the canonical path of the given filename. It doesn't
>> try to locate some actual existing file.
>
> I always thought that os.path.realpath is the Python equivalent of Linux
> realpath/readlink

In POSIX, os.path.realpath resolves a path that contains symlinks by
calling os.readlink in a loop until all links are resolved.

In Windows, os.path.realpath is an alias for os.path.abspath. This
decision predates NTFS support for junctions (2000) and symbolic links
(Vista). It's simply the case that no one has gotten around to
implementing realpath for Windows.

In Windows, os.path.abspath calls os.path._getfullpathname (WinAPI
GetFullPathName). Other than resolving the working directory, this is
a string transformation to return a canonical, fully-qualified path.
It doesn't touch the filesystem. Note that in the case of classic DOS
devices, the canonical path is a local-device path (i.e. "\\.\"
prefix). For example:

>>> print(os.path.abspath('C:/path/to/nul'))
\\.\nul

> maybe also when it's a hard link - in this case the function actually also 
> does
> something in Windows

A hard link shouldn't be resolved to another path by os.path.realpath.
It's already a real path.

Cross-platform Python 3 code that needs to resolve symbolic links (or
junctions) can use pathlib.Path.resolve(). For example:

>>> pathlib.Path('C:/Documents and Settings').resolve()
WindowsPath('C:/Users')

On Windows this relies on os.path._getfinalpathname (WinAPI
GetFinalPathNameByHandle). This function returns the final opened
filename, so the OS does all of the work to resolve the final path. In
non-strict mode pathlib uses a loop to reduce the path when it can't
be resolved to an existing file. (There are open issues for edge cases
in this code, but it should work as intended with regular files in
accessible directories.)

For those who are curious, WinAPI GetFinalPathNameByHandle is called
with a handle for a File object that's opened via CreateFile. It gets
the NT name of the volume (e.g, "\Device\HarddiskVolume1") and the
filesystem path (e.g. "\Users"). Then it asks the mount-point manager
to map this NT name back to a DOS name such as drive "C:"; or an NTFS
volume mountpoint such as "C:\Mount\VolumeName"; or a "Volume{GUID}"
name. (In the latter case, Python's _getfinalpathname and pathlib have
bugs and need improvement.)
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Why does os.path.realpath('test_main.py') give different results for unittest than for testing statement in interpreter?

[eryk sun]

On Tue, Jan 9, 2018 at 2:48 PM, Albert-Jan Roskam
 wrote:
>
> I think that it would be a great enhancement if os.realpath would return the 
> UNC path if
> given a mapped drive in Windows, if needed as extended path (prefixed with 
> "\\?\UNC\").
> That's something I use all the time, unlike symlinks, in Windows.

pathlib can do this for you, or call os.path._getfinalpathname. I
recently helped someone that wanted the reverse, to map the resolved
UNC path back to a logical drive:

https://bugs.python.org/issue32442

> And I would also welcome a convenience function in the os.path module that 
> does
> expanduser, expandvars, normpath, realpath, and maybe more.

pathlib implements expanduser, but it's still lame on Windows for a
user other than the current user. It assumes all user profiles are in
the same directory instead of directly getting the user's profile
path. Anyway, expanduser is of limited use.

We can't assume in general that a user's special folders (e.g.
Desktop, Documents) are in the default location relative to the
profile directory. Almost all of them are relocatable. There are shell
APIs to look up the current locations, such as SHGetKnownFolderPath.
This can be called with ctypes [1]. For users other than the current
user, it requires logging on and impersonating the user, which
requires administrator access.

[1]: https://stackoverflow.com/a/33181421/205580

You can log a user on without a password by calling LsaLogonUser to
request an MSV1 S4U (service for user) logon. The access token will
only be identification level, unless the script is running as a SYSTEM
service. But identification level is enough to query the location of a
user's known folders. I don't recommended calling LsaLogonUser via
ctypes if you have the option to write an extension module in C/C++,
but I did manage to get it working with ctypes [2]. For example:

>>> from security_lsa import logon_msv1_s4u
>>> from knownfolders import FOLDERID, get_known_folder_path

>>> logon_info = logon_msv1_s4u('Administrator')
>>> get_known_folder_path(FOLDERID.Desktop, logon_info.Token)
'C:\\Users\\Administrator\\Desktop'

[2]: https://stackoverflow.com/a/4322/205580

Also, please don't use expanduser to enable applications to spam the
profile directory with configuration files and directories. Use the
local and roaming application data directories.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Why does os.path.realpath('test_main.py') give different results for unittest than for testing statement in interpreter?

[eryk sun]

On Wed, Jan 10, 2018 at 12:59 PM, Albert-Jan Roskam
 wrote:
>
> I tried:
 from os.path import _getfullpathname
 _getfullpathname(r"H:")
> 'h:\\path\\to\\folder'
 import os
 os.getcwd()
> 'h:\\path\\to\\folder'
>
> I expected h:\ to be \\server\share\foo.

You called _getfullpathname (WinAPI GetFullPathName), not
_getfinalpathname (WinAPI GetFinalPathNameByHandle). GetFullPathName
works on the path as a string without touching the filesystem.
GetFinalPathNameByHandle reconstructs a path when given a handle to a
file or directory.

> The fact that the current working directory was returned was even more 
> unexpected.

"H:" or "H:relative/path" is relative to the working directory on
drive H:. The process only has one working directory, but
GetFullPathName also checks for environment variables such as "=H:".
The C runtime's _chdir function sets these magic variables, as does
Python's os.chdir function (we don't call C _chdir). WinAPI
SetCurrentDirectory does not set them. For example:

>>> os.chdir('Z:/Temp')
>>> win32api.GetEnvironmentVariable('=Z:')
'Z:\\Temp'
>>> os.path._getfullpathname('Z:relative')
'Z:\\Temp\\relative'

> Why would anybody *want* the drive letters? They are only useful because (a) 
> they save on
> keystrokes (b) they bypass the annoying limitation of the cd command on 
> windows, ie. it
> does not work with UNC paths.

Windows itself has no problem using a UNC path as the working
directory. That's a limit of the CMD shell.

SUBST drives can be used to access long paths. Since the substitution
occurs in the kernel, it avoids the MAX_PATH 260-character limit. Of
course, you can also use junctions and symlinks to access long paths,
or in Windows 10 simply enable long-path support.

> I know net use, pushd, subst. I use 'net use' for more or less permanent 
> drives and
> pushd/popd to get a temporary drive, available letter (cd nuisance).

`net.exe use` and CMD's PUSHD command (with a UNC path) both call
WinAPI WNetAddConnection2 to create a mapped network drive. The
difference is that net.exe can supply alternate credentials and create
a persistent mapping, while PUSHD uses the current user's credentials
and creates a non-persistent mapping.

If your account gets logged on with a UAC split token, the standard
and elevated tokens actually have separate logon sessions with
separate local-device mappings. You can enable a policy to link the
two logon sessions. Set a DWORD value of 1 named
"EnableLinkedConnections" in the key
"HKLM\Software\Microsoft\Windows\CurrentVersion\Policies\System", and
reboot.

subst.exe creates substitute paths using WinAPI DefineDosDevice.
Unlike the WNet API, this function doesn't use MPR (multiple provider
router) to create a direct link for the network provider (e.g.
\Device\LanmanRedirectory); doesn't create a linked connection when
EnableLinkedConnections is defined; and can't create a persistent
drive with stored credentials (though you can use an account logon
script for this). On the plus side, a drive mapped via subst.exe can
target any path.

> Interesting code! I have used the following, which uses SHGetFolderPath, ie. 
> without 'Known'.
> from win32com.shell import shell, shellcon
> desktop = shell.SHGetFolderPath(0, shellcon.CSIDL_DESKTOP, 0, 0)

SHGetFolderPath is usually fine, but still, it's outdated and
deprecated. win32com.shell doesn't wrap SHGetKnownFolderPath for some
reason, but you can still use the new known-folder API without ctypes.
Just create a KnownFolderManager instance. For example:

import pythoncom
from win32com.shell import shell

kfmgr = pythoncom.CoCreateInstance(shell.CLSID_KnownFolderManager, None,
pythoncom.CLSCTX_INPROC_SERVER, shell.IID_IKnownFolderManager)

desktop_path = kfmgr.GetFolder(shell.FOLDERID_Desktop).GetPath()

This doesn't work as conveniently for getting known folders of other
users. While the high-level SHGetKnownFolderPath function takes care
of loading the user profile and impersonating, we have to do this
ourselves when using a KnownFolderManager instance.

That said, to correct my previous post, you have to be logged on with
SeTcbPrivilege access (e.g. a SYSTEM service) to get and set other
users' known folders without their password. (If you have the password
you can use a regular logon instead of an S4U logon, and that works
fine.)

> Working with ctypes.wintypes is quite complex!

I wouldn't say ctypes is complex in general. But calling LsaLogonUser
is complex due to all of the structs that include variable-sized
buffers. And working with COM via ctypes is also complex, which is why
comtypes exists.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] When is and isn't "__file__" set?

[eryk sun]

On Fri, Jan 12, 2018 at 5:22 PM, Albert-Jan Roskam
 wrote:
> On Jan 11, 2018 03:47, Steven D'Aprano  wrote:
>>
>> Modules which are loaded from a .dll or .so binary file also should have
>> __file__ set.
>
> And .pyd? I would hope so

A .pyd is a Windows DLL (i.e. PE/COFF shared library). The .pyd type
is associated with the "Python.Extension" program identifier, which
sets a custom descriptive name and icon in Explorer. Sometimes a
Python extension module uses the .dll file extension instead of .pyd.
For example, from PyWin32:

>>> print(pywintypes.__file__)
C:\Program Files\Python36\pywintypes36.dll
>>> print(pythoncom.__file__)
C:\Program Files\Python36\pythoncom36.dll
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Virtual environment question

[eryk sun]

On Mon, Mar 12, 2018 at 12:44 AM, Jim  wrote:
>
> home = /usr/bin
> include-system-site-packages = false
[...]
> resp = opener.open(request, timeout=self._timeout)
>   File "/usr/lib/python3.5/urllib/request.py", line 466, in open

This is normal. Virtual environments are not isolated from the standard library.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Virtual environment question

[eryk sun]

On Tue, Mar 13, 2018 at 1:31 AM, Jim  wrote:
> On 03/12/2018 04:04 AM, eryk sun wrote:
>>
>> On Mon, Mar 12, 2018 at 12:44 AM, Jim  wrote:
>>>
>>> home = /usr/bin
>>> include-system-site-packages = false
>>
>> [...]
>>>
>>>  resp = opener.open(request, timeout=self._timeout)
>>>File "/usr/lib/python3.5/urllib/request.py", line 466, in open
>>
>> This is normal. Virtual environments are not isolated from the standard
>> library.
>
> Interesting. All I know about virtual environments is what I read on the
> net. I always see them recommended as a way to keep from messing up the
> default python, so I thought isolation was their purpose.

You're thinking of isolating packages that are installed in
site-packages, not the standard library. There's no point in copying
and recompiling the entire standard library in every virtual
environment.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor