Re: python 2.7.12 on Linux behaving differently than on Windows

2016-12-09 Thread Steve D'Aprano
On Fri, 9 Dec 2016 01:52 pm, Chris Angelico wrote:

> On Fri, Dec 9, 2016 at 12:34 PM, BartC  wrote:
>> With a case-sensitive file system, how do you search only for 'harry',
>> not knowing what combinations of upper and lower case have been used?
>> (It's a good thing Google search isn't case sensitive!)
> 
> This is handled by "case correlations", which aren't quite the same as
> case conversions. 

I think you mean to say, this *should* be handled by case correlations.

I expect that there is a lot of software on the planet that just blindly
converts the strings to both lower- or both uppercase and then checks
equality. And even more that assumes ASCII case conversions and just does
bit-twiddling to compare letters.

(By the way, case correlation... I've never come across that term before,
and googling doesn't find anything useful. Are you sure that's the right
term?)


> In Python, that's the .casefold() method. You 
> casefold the string "harry", and then casefold every file name that
> might potentially match, and see if they become the same. Taking my
> original example string:
> 
 "ßẞıİÅσςσ".casefold()
> 'ıi̇åσσσ'

Your example string is a good demonstration of mojibake. Or possibly a *bad*
demonstration of mojibake, since I cannot imagine any real words that would
generate that via encoding/decoding into the wrong character set :-)

I'm not really sure what point you were trying to make with Bart. Globbing
doesn't support case-insensitive matches on any platform I know of, so I
don't think this is really relevant.


> Any string that casefolds to that same string should be considered a
> match. This does NOT stop you from having multiple such files in the
> directory.

If you want to support multilingual string comparisons, you have to do more
than just casefold(). You need to normalise the strings, otherwise 'café'
and 'café' will be treated as distinct.

Frankly, I think that Apple HFS+ is the only modern file system that gets
Unicode right. Not only does it restrict file systems to valid UTF-8
sequences, but it forces them to a canonical form to avoid the é é gotcha,
and treats file names as case preserving but case insensitive.

Lastly, there's one last fly in the ointment for multilingual case
comparisons: Turkish i. Unfortunately, there's no clean way to do case
comparisons that works for "any arbitrary language".

Turkish, and one or two other languages, want dotless and dotted I to be
treated as distinct: ıI go together, and iİ go together. But other
languages want iI to go together, meaning that the standard case
conversions are lossy:

py> 'ıIiİ'.lower()
'ıiii'
py> 'ıIiİ'.upper()
'IIIİ'


Maybe it would have been better if the standard had kept ıİ together, and
iI, so at least the case conversion was lossless. Alas, too late now.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: python 2.7.12 on Linux behaving differently than on Windows

2016-12-09 Thread Steve D'Aprano
On Fri, 9 Dec 2016 04:52 pm, Marko Rauhamaa wrote:

> Random832 :
> 
>> On Thu, Dec 8, 2016, at 20:38, Dennis Lee Bieber wrote:
>>> In the original 8.3 scheme -- no files "contained" a dot
>>
>> Yes, but they do now, and the compatibility quirks persist.
> 
> When porting a Python program to Windows, I noticed the filename "aux"
> is not allowed in Windows. I suspect it's the same with "lst", "prn",
> "con" and what not.
> 
> In Linux, "." and ".." are taboo. 

No that's incorrect. It isn't that . and .. are forbidden, but they are
reserved: every single directory in Unix file systems have a . and ..
directory entry. So they are legitimate directory names -- they're just not
names you can use for your own files, as they are already in use.

> So is "".

That's one way of looking at things... I'm not sure that the empty string
counts as a file name. Its more of the LACK of a file name.

Besides, that would be ambiguous. Would "/home/steve/" mean my home
directory, or the file "" inside my home directory?




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: python 2.7.12 on Linux behaving differently than on Windows

2016-12-09 Thread Marko Rauhamaa
Steve D'Aprano :

> On Fri, 9 Dec 2016 04:52 pm, Marko Rauhamaa wrote:
>> In Linux, "." and ".." are taboo. 
>
> No that's incorrect. It isn't that . and .. are forbidden, but they are
> reserved: every single directory in Unix file systems have a . and ..
> directory entry. So they are legitimate directory names -- they're just not
> names you can use for your own files, as they are already in use.

Same difference.

>> So is "".
>
> That's one way of looking at things... I'm not sure that the empty
> string counts as a file name. Its more of the LACK of a file name.
>
> Besides, that would be ambiguous. Would "/home/steve/" mean my home
> directory, or the file "" inside my home directory?

If Python had been the standard shell when Unix came about, you could
have defined pathnames as lists of strings. Then, everything would be
unambigous and there wouldn't be any taboo or reserved names.

BTW, guile allows *any* characters in an identifier. Even the empty name
is a valid identifier:

   > (define #{}# "hello")
   > #{}#
   $1 = "hello"
   (symbol->string '#{}#)
   $2 = ""
   > (symbol->string '#{.}#)
   $4 = "."


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: python 2.7.12 on Linux behaving differently than on Windows

2016-12-09 Thread Chris Angelico
On Fri, Dec 9, 2016 at 6:41 PM, Steve D'Aprano
 wrote:
> On Fri, 9 Dec 2016 01:52 pm, Chris Angelico wrote:
>
>> On Fri, Dec 9, 2016 at 12:34 PM, BartC  wrote:
>>> With a case-sensitive file system, how do you search only for 'harry',
>>> not knowing what combinations of upper and lower case have been used?
>>> (It's a good thing Google search isn't case sensitive!)
>>
>> This is handled by "case correlations", which aren't quite the same as
>> case conversions.
>
> I think you mean to say, this *should* be handled by case correlations.
>
> I expect that there is a lot of software on the planet that just blindly
> converts the strings to both lower- or both uppercase and then checks
> equality. And even more that assumes ASCII case conversions and just does
> bit-twiddling to compare letters.

This is true. However, I would consider that to be buggy software, not
flawed design of file systems.

> (By the way, case correlation... I've never come across that term before,
> and googling doesn't find anything useful. Are you sure that's the right
> term?)

Hmm, now you mention it, I'm actually not sure where I got that term
from. But it doesn't matter what you call it; the point is that there
is a "case insensitive comparison equivalency" that is not the same as
merely converting the string to upper/lower. It's allowed to be lossy;
in fact, I would start the process with NFKC or NFKD normalization, to
eliminate any problems with ligatures and such.

>> In Python, that's the .casefold() method. You
>> casefold the string "harry", and then casefold every file name that
>> might potentially match, and see if they become the same. Taking my
>> original example string:
>>
> "ßẞıİÅσςσ".casefold()
>> 'ıi̇åσσσ'
>
> Your example string is a good demonstration of mojibake. Or possibly a *bad*
> demonstration of mojibake, since I cannot imagine any real words that would
> generate that via encoding/decoding into the wrong character set :-)

Heh. It's more of a "stress test" file name, picking up edge cases
from several languages. It's plausible for a file name to have one or
two of those characters in it, and for different file names to have
different selections from that set, and for a single file system to
have to cope with all of those examples.

> I'm not really sure what point you were trying to make with Bart. Globbing
> doesn't support case-insensitive matches on any platform I know of, so I
> don't think this is really relevant.

Windows does case insensitive matches. Has for as long as I've known it.

>> Any string that casefolds to that same string should be considered a
>> match. This does NOT stop you from having multiple such files in the
>> directory.
>
> If you want to support multilingual string comparisons, you have to do more
> than just casefold(). You need to normalise the strings, otherwise 'café'
> and 'café' will be treated as distinct.

IMO you should be able to NFC normalize file names before they get
stored. The only reason Linux file systems currently can't is backward
compat - the file names might not represent text. But they are
*names*. Fundamentally, they are supposed to be meaningful. Mandating
that they be UTF-8 byte streams representing Unicode text is not
unreasonable.

> Frankly, I think that Apple HFS+ is the only modern file system that gets
> Unicode right. Not only does it restrict file systems to valid UTF-8
> sequences, but it forces them to a canonical form to avoid the é é gotcha,
> and treats file names as case preserving but case insensitive.

Agreed. Other file systems and operating systems should pick this up.

> Lastly, there's one last fly in the ointment for multilingual case
> comparisons: Turkish i. Unfortunately, there's no clean way to do case
> comparisons that works for "any arbitrary language".

You may notice that my original string has a couple of examples of that :)

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: python 2.7.12 on Linux behaving differently than on Windows

2016-12-09 Thread eryk sun
On Fri, Dec 9, 2016 at 7:41 AM, Steve D'Aprano
 wrote:
> Frankly, I think that Apple HFS+ is the only modern file system that gets
> Unicode right. Not only does it restrict file systems to valid UTF-8
> sequences, but it forces them to a canonical form to avoid the é é gotcha,
> and treats file names as case preserving but case insensitive.

Windows NTFS doesn't normalize names to a canonical form. It also
allows lone surrogate codes, which is invalid UTF-16.

For case insensitive matches it converts to upper case, but the
conversion table it uses is extremely conservative. Here's a simple
function to convert a string to upper case using NT's runtime library
function RtlUpcaseUnicodeChar:

import ctypes

ntdll = ctypes.WinDLL('ntdll')

def upcase(s):
up = []
for c in s:
b = bytearray()
for c in memoryview(c.encode('utf-16le')).cast('H'):
c_up = ntdll.RtlUpcaseUnicodeChar(c)
b += c_up.to_bytes(2, 'little')
up.append(b.decode('utf-16le'))
return ''.join(up)

For example:

>>> upcase('abcd')
'ABCD'
>>> upcase('αβψδ')
'ΑΒΨΔ'
>>> upcase('ßẞıİÅσςσ')
'ßẞıİÅΣςΣ'

Attempting to create two files named 'ßẞıİÅσςσ' and 'ßẞıİÅΣςΣ' in the
same NTFS directory fails, as expected:

>>> s = 'ßẞıİÅσςσ'
>>> open(s, 'x').close()
>>> open(upcase(s), 'x').close()
Traceback (most recent call last):
  File "", line 1, in 
FileExistsError: [Errno 17] File exists: 'ßẞıİÅΣςΣ'

Note that Windows thinks standard case conversions of this name are all unique:

>>> open(s.upper(), 'x').close()
>>> open(s.lower(), 'x').close()
>>> open(s.casefold(), 'x').close()
>>> os.listdir()
['ıi̇åσσσ', 'SSẞIİÅΣΣΣ', 'ßßıi̇åσςσ', 'ßẞıİÅσςσ']
-- 
https://mail.python.org/mailman/listinfo/python-list


Best attack order for groups of numbers trying to destroy each other, given a victory chance for number to number attack.

2016-12-09 Thread skybuck2000
Hello,

(This problem is probably too computationally intensive to solve with Python, 
though Python + Cuda could be interesting, and also Python has some interesting 
expressive powers, so it could be interesting to see how Python programmers 
might be able to express this problem with Python code, so I am going to give 
this Python group a try... maybe it will surprise me ! :) At least now you have 
a nice computational problem for those boring rainy days (when the net is 
down?and the offline games bore you ;) LOL :))

There are two groups of numbers which try to destroy each other:

Group X = 1,2,3,4
Group Y = 1,2,3,4

There is a 4 by 4 victory table which describes the chance of a number 
destroying another number:

Victory table =
50,  3, 80, 70
90, 60, 20, 40
30, 90, 55, 65
75, 90, 98, 60

(Currently implemented as a chance by diving it by 100 and storing as 
floating point, but since these are subtracted only from 1.0 I guess they 
can be stored as integers instead, even bytes)

This leads to the following computational problem as far as I am concerned:

Each number has an attack order permutation as follows (factorial 4 = 
1x2x3x4 = 24)

1,2,3,4// 1
1,2,4,3// 2
1,3,2,4// 3
1,3,4,2// 4
1,4,2,3// 5
1,4,3,2// 6
2,1,3,4// 7
2,1,4,3// 8
2,3,1,4// 9
2,3,4,1// 10
2,4,1,3// 11
2,4,3,1// 12
3,1,2,4// 13
3,1,4,2// 14
3,2,1,4// 15
3,2,4,1// 16
3,4,1,2// 17
3,4,2,1// 18
4,1,2,3// 19
4,1,3,2// 20
4,2,1,3// 21
4,2,3,1// 22
4,3,1,2// 23
4,3,2,1   // 24

(These attack orders can be numbered from 1 to 24 or 0 to 23 and then it's 
attack order/permutation can be looked up to safe memory.)

Each number has it's own attack order and thus this leads to the following 
combinational computational problem:

All combinations of permutations in which order group X can attack Group Y 
and vice versa:

Group X = 24 x 24 x 24 x 24
Group Y = 24 x 24 x 24 x 24

So this is 24 possibility to the power of 8.

Final computation complexity at the very minimum is (24 to the power of 8) 
multiplied by roughly 4 attacks perhaps even 5 or 6 to finally destroy a 
group of numbers.

24 to the power of 8 = 110.075.314.176

I have already written a computer program which can solve this, however the 
computer program estimates it takes 19 hours on a 2.0 gigahertz AMD Athlon 
X2 core.

Using dual core could solve the problem over night, though I do not feel 
comfortable running my PC at night unattended.

So now I have the idea to make this program run when my computer is idling 
during the day, it should also be able to store it's progress so that it can 
continue after it was shutdown.

(Idea for now is to make it multi threaded and assign a low thread priority 
so it can run during the day when I use my computer and it's not doing much 
so it can use the reserve computational horse power).
(I still have to try these "idle/reverse" ideas to see which one works best 
without interrupting my web browsing or music listening too much ;))

My system has 4 GB of ram, so other ideas could be to store a data structure 
partially which could keep some computations so that it doesn't have to be 
done again... Though memory lookups might be a bit slow so not sure if that 
makes any sense.

I might also try GPU/Cuda since there seems to be lots of loops/reoccurences 
of the same computations that will happen over and over again... So maybe 
cuda can detect "same branch execution" and some "computations" and might 
speed it up, not sure about that.

Just the 8 index loops already cost a lot of instructions. Since there are 
only 24 permutation it would be enough to store it in 5 bits. Perhaps a 
rounded floating point which increases by 1/24 might be enough to trigger 
the 4th bit from incrementing when it actually needs to.
2x2x2x2x2 = 32  (it should increment bit 6 when the 5 bits reach 24).
So the idea here was to create 8 indexes from just 1 index being incremented 
to create the 8 combinations of indexes "instruction cheaply".
Not sure if this will work, just an idea I might try :)

Then those bits would still need to be extract and makes. So perhaps on 
older systems this is not efficient.

The 8 indexes need at least 3 instructions, 1 index increment, 1 
comparision, 1 jump.

The inner loop contains some while loops to increment attack index per 
number.

Each number has a "alive" variable which starts at 1.0 and is decreased 
everytime it's attacked.

Once a number is dead below 0.001 it's considered dead and can no longer 
attack.

(Since victory table was described as integers above this can also be read 
as: Alive starts at 100 and once it goes zero or negative it's dead).

Anyway the main reason why I wrote to this/these groups is that the numbers 
themselfes are not that larger and can fit in a byte, even a few bits.

Thus they will fit into SSE registers and such and I also know SSE has some 
permutations instructions.

I am not expert at SSE bu

Re: python 2.7.12 on Linux behaving differently than on Windows

2016-12-09 Thread Marko Rauhamaa
eryk sun :

> Windows NTFS doesn't normalize names to a canonical form. It also
> allows lone surrogate codes, which is invalid UTF-16.

Somewhat related, surrogate codes are invalid Unicode and shouldn't be
allowed in Unicode strings. However, Python does allow them.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: python 2.7.12 on Linux behaving differently than on Windows

2016-12-09 Thread Steve D'Aprano
On Fri, 9 Dec 2016 09:34 pm, Marko Rauhamaa wrote:

> Steve D'Aprano :
> 
>> On Fri, 9 Dec 2016 04:52 pm, Marko Rauhamaa wrote:
>>> In Linux, "." and ".." are taboo.
>>
>> No that's incorrect. It isn't that . and .. are forbidden, but they are
>> reserved: every single directory in Unix file systems have a . and ..
>> directory entry. So they are legitimate directory names -- they're just
>> not names you can use for your own files, as they are already in use.
> 
> Same difference.

Of course they are different.

'\0' (the ASCII null character) is not a legal file name. You can't open a
file with that name, or (more relevant to this point) have a directory with
that name.

'..' and '.' are legal file names, or to be more specific, legal directory
names. You can have directories with those names, and they can appear in
paths.

py> os.path.isdir('..')
True
py> os.path.isdir('\0')
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/local/lib/python3.3/genericpath.py", line 41, in isdir
st = os.stat(s)
TypeError: embedded NUL character


See the difference?



>>> So is "".
>>
>> That's one way of looking at things... I'm not sure that the empty
>> string counts as a file name. Its more of the LACK of a file name.
>>
>> Besides, that would be ambiguous. Would "/home/steve/" mean my home
>> directory, or the file "" inside my home directory?
> 
> If Python had been the standard shell 

Python isn't a shell.


> when Unix came about, 

Ah, a counter-factual.


> you could  
> have defined pathnames as lists of strings. Then, everything would be
> unambigous and there wouldn't be any taboo or reserved names.

I really don't think so.

So what path does [] represent? How do you distinguish between the root
directory / and no pathname at all in a syntactically correct, unambiguous
manner, using only lists?

What path does [[]] represent? How about [[], []]?

Here's a nice one for you to think about:

L = []; L.append(L)

What path does L represent? How would it be represented in the file system?

What about this "path"? [None, 23, float('NAN'), {}]

What's the difference between ['a', 'b', 'c'] and ['a', ['b', ['c']]] as
path names? If there's no difference, are you concerned at the redundancy?
Is there a canonical form for paths?

How do you represent the current and previous directory in a list without
reserving identifiers for them?

How do you distinguish between '/home/steve' and '/home/steve/' for those
programs that (rightly or wrongly) need to distinguish them?

How do you handle alternate data streams? Or versioning information, for
systems like VMS that track that?

What about file systems where you can have multiple roots? E.g. Windows, C
\file versus D:\file and classic Mac Floppy:file and HardDrive:file?


Do you really imagine that if Python were a shell, and required system
administrators to write things like:

mount ['dev', 'dvd'] ['mount', 'dvd']

instead of

mount /dev/dvd /mount/dvd

that it would have been a success?

Hint: https://www.jwz.org/doc/worse-is-better.html



> BTW, guile allows *any* characters in an identifier. Even the empty name
> is a valid identifier:
> 
>> (define #{}# "hello")
>> #{}#
>$1 = "hello"
>(symbol->string '#{}#)
>$2 = ""
>> (symbol->string '#{.}#)
>$4 = "."


And are Guile programs improved by having the empty string as an identify?

When was the last time you were coding and you thought "This program would
be so much easier to understand and maintain if only I could name this the
empty string"?

Does it simplify the process of one programmer talking to another programmer
about an algorithm to be able to use \n\n\n\0\n\n\n as an identifier?

Sometimes more is less.




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: python 2.7.12 on Linux behaving differently than on Windows

2016-12-09 Thread Marko Rauhamaa
Steve D'Aprano :

> On Fri, 9 Dec 2016 09:34 pm, Marko Rauhamaa wrote:
>> Steve D'Aprano :
>>> No that's incorrect. It isn't that . and .. are forbidden, but they
>>> are reserved: every single directory in Unix file systems have a .
>>> and .. directory entry. So they are legitimate directory names --
>>> they're just not names you can use for your own files, as they are
>>> already in use.
>> 
>> Same difference.
>
> Of course they are different.

You are not allowed to name your next file "." or ".." or "1/8".

>> you could have defined pathnames as lists of strings. Then,
>> everything would be unambigous and there wouldn't be any taboo or
>> reserved names.
>
> I really don't think so.
>
> So what path does [] represent?

That's the root directory.

> How do you distinguish between the root directory / and no pathname at
> all in a syntactically correct, unambiguous manner, using only lists?

I don't know what you mean.

> What path does [[]] represent? How about [[], []]?

Those are not lists of strings (see above).

> Here's a nice one for you to think about:
>
> L = []; L.append(L)
>
> What path does L represent?

That's not a list of strings.

> How would it be represented in the file system?
>
> What about this "path"? [None, 23, float('NAN'), {}]

That's not a list of strings.

> What's the difference between ['a', 'b', 'c'] and ['a', ['b', ['c']]] as
> path names?

Only the first one is a legal pathname.

> How do you represent the current and previous directory in a list
> without reserving identifiers for them?

My idea covered absolute pathnames only.

> How do you distinguish between '/home/steve' and '/home/steve/' for those
> programs that (rightly or wrongly) need to distinguish them?

I wouldn't. That distinction would have to be made some other way.

> How do you handle alternate data streams? Or versioning information, for
> systems like VMS that track that?

I am only talking about pathnames.

> What about file systems where you can have multiple roots? E.g. Windows, C
> \file versus D:\file and classic Mac Floppy:file and HardDrive:file?

I would only have one root.

> Do you really imagine that if Python were a shell, and required system
> administrators to write things like:
>
> mount ['dev', 'dvd'] ['mount', 'dvd']
>
> instead of
>
> mount /dev/dvd /mount/dvd
>
> that it would have been a success?

Dunno. I've been burned by bash so often that solid ground is what I
yearn for most.

>> BTW, guile allows *any* characters in an identifier. Even the empty name
>> is a valid identifier:
>
> And are Guile programs improved by having the empty string as an
> identify?

Point is, it's not at all impossible to remove the limitations on what
can go in names, filenames or otherwise.

> When was the last time you were coding and you thought "This program
> would be so much easier to understand and maintain if only I could
> name this the empty string"?

Having to be on constant lookout for corner cases is the problem.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: python 2.7.12 on Linux behaving differently than on Windows

2016-12-09 Thread Chris Angelico
On Sat, Dec 10, 2016 at 5:40 AM, Marko Rauhamaa  wrote:
>> How do you represent the current and previous directory in a list
>> without reserving identifiers for them?
>
> My idea covered absolute pathnames only.

Well, you're going to need to cope with relative pathnames somehow. I
suppose you could do:

[RELATIVE, PARENT, PARENT, "subdir", "file"]

but I'd rather use "../../subdir/file", myself.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: CLP stats: last 500 posts

2016-12-09 Thread Jon Ribbens
On 2016-12-09, DFS  wrote:
> import sys as y,nntplib as t,datetime as d
> s=''
> g=y.argv[1]
> n=t.NNTP(s,119,'','')
> r,a,b,e,gn=n.group(g)
> def printStat(st,hd,rg):
>   r,d=n.xhdr(st,'%s-%s'%rg)
>   p=[]
>   for i in range(len(d)):
>   v=d[i][1]
>   if st=='Subject':v=v[4:] if v[:3]=='Re:' else v
>   p.append(v)
>   x=[(i,p.count(i)) for i in set(p)]
>   x.sort(key=lambda s:(-s[1],s[0].lower()))
>   print('Posts  %s %s'%(len(set(p)),hd))
>   for v in x: print(' %s %s'%(v[1],v[0]))
>   print
> print 'As of '+d.datetime.now().strftime("%I:%M%p %B %d, %Y") + '\n'
> m=(int(e)-int(y.argv[3])+1,int(e))
> printStat("From","Posters",m)
> printStat("Subject","Subjects",m)
> printStat("User-Agent","User-Agents",m)
> n.quit()

Was there ever an "International Obfuscated Python Code Contest"? ;-)
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: CLP stats: last 500 posts

2016-12-09 Thread Chris Angelico
On Sat, Dec 10, 2016 at 8:34 AM, Jon Ribbens  wrote:
>
> Was there ever an "International Obfuscated Python Code Contest"? ;-)

I don't know, but if so, here's my entry:

print(*([0,"Fizz","Buzz","Fizzbuzz"][[3,0,0,1,0,2,1,0,0,1,2,0,1,0,0][i%15]]or
i for i in range(1,51)))

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list


Running python from pty without prompt

2016-12-09 Thread space . ship . traveller
Hello.

I'm working on a script runner for Atom.

https://github.com/ioquatix/script-runner

We are trying to understand how to make python work well. I'll use a comparison 
to the ruby executable because it's convenient to explain the problem.

When you invoke `ruby` from a pty, you get no output (as opposed to `irb`, 
interactive ruby [shell]). You can write a script to stdin, and send Ctrl-D 
(EOT / 0x04). Then, ruby will execute the script. stdin is not closed so 
programs that expect interactive input will work correctly.

When we run python in the same way, we get an output prompt. As soon as a 
function like `input` is called, the program is interrupted.

I'm happy to hear feedback about how this should work. Perhaps our expectations 
are wrong, or what we are doing is wrong.

One option which we've been considering is saving the input as a file and 
executing that. But, it's not as clean compared to simply writing a string to 
stdin of a running interpreter and then 0x04.

Thanks for any and all help.

Kind regards,
Samuel
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Running python from pty without prompt

2016-12-09 Thread Michael Torrie
On 12/09/2016 04:11 PM, [email protected] wrote:
> Hello.
> 
> I'm working on a script runner for Atom.
> 
> https://github.com/ioquatix/script-runner
> 
> We are trying to understand how to make python work well. I'll use a
> comparison to the ruby executable because it's convenient to explain
> the problem.
> 
> When you invoke `ruby` from a pty, you get no output (as opposed to
> `irb`, interactive ruby [shell]). You can write a script to stdin,
> and send Ctrl-D (EOT / 0x04). Then, ruby will execute the script.
> stdin is not closed so programs that expect interactive input will
> work correctly.
> 
> When we run python in the same way, we get an output prompt. As soon
> as a function like `input` is called, the program is interrupted.
> 
> I'm happy to hear feedback about how this should work. Perhaps our
> expectations are wrong, or what we are doing is wrong.

Not sure I understand the issue here.  You can pipe a script to Python
and it runs it without any immediate-mode prompt.  I think python only
shows the REPL prompt if you are attached to a pty.

But if the script is piped into Python or ruby, I don't know how we
could expect raw_input() or input() to function.

> One option which we've been considering is saving the input as a file
> and executing that. But, it's not as clean compared to simply writing
> a string to stdin of a running interpreter and then 0x04.

An intermediate file is unnecessary as you can pipe a script into Python.
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Running python from pty without prompt

2016-12-09 Thread Michael Torrie
On 12/09/2016 04:11 PM, [email protected] wrote:
> When you invoke `ruby` from a pty, you get no output (as opposed to
> `irb`, interactive ruby [shell]). You can write a script to stdin,
> and send Ctrl-D (EOT / 0x04). Then, ruby will execute the script.
> stdin is not closed so programs that expect interactive input will
> work correctly.
> 
> When we run python in the same way, we get an output prompt. As soon
> as a function like `input` is called, the program is interrupted.

Nevermind my other post. I understand what you are saying.
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Running python from pty without prompt

2016-12-09 Thread Samuel Williams
Just in case it's not clear, this is running on a (virtual) PTY.
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: CLP stats: last 500 posts

2016-12-09 Thread Steve D'Aprano
On Sat, 10 Dec 2016 08:07 am, DFS wrote:

> 
> As of 04:04PM December 09, 2016
> 
> Posts  85 Posters
[...]


Interesting stats, but couldn't you have post-processed the results to avoid
including the defamatory spam posts?

Your post is likely to be removed from the official web archive as it
contains defamatory material.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Running python from pty without prompt

2016-12-09 Thread Steve D'Aprano
On Sat, 10 Dec 2016 10:11 am, [email protected] wrote:

> Hello.
> 
> I'm working on a script runner for Atom.
> 
> https://github.com/ioquatix/script-runner
> 
> We are trying to understand how to make python work well. I'll use a
> comparison to the ruby executable because it's convenient to explain the
> problem.

Could you show a small, simple demonstration of both the Ruby code that
works they way you want, and the Python code that behaves differently?

Preferably one that runs straight from vanilla Python without any
third-party libraries, including your script-runner.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Running python from pty without prompt

2016-12-09 Thread Michael Torrie
On 12/09/2016 06:43 PM, Steve D'Aprano wrote:
> On Sat, 10 Dec 2016 10:11 am, [email protected] wrote:
> 
>> Hello.
>>
>> I'm working on a script runner for Atom.
>>
>> https://github.com/ioquatix/script-runner
>>
>> We are trying to understand how to make python work well. I'll use a
>> comparison to the ruby executable because it's convenient to explain the
>> problem.
> 
> Could you show a small, simple demonstration of both the Ruby code that
> works they way you want, and the Python code that behaves differently?
> 
> Preferably one that runs straight from vanilla Python without any
> third-party libraries, including your script-runner.

I imagine they want to feed Python a combination of a script and also
standard-in input in the same stream. Something like:

$ python << EOF
a = input("Enter your name: ")
print a
^D
bob
EOF

Where ^D is a literal control character (ctrl-v, control-d on the
terminal, but does not actually close the stream or signify the end).
This would be piped into Python's standard-in where it would feed python
both the script to run, and also input to feed the script.  Apparently
ruby can do this.

Did I understand this correctly, space.ship.traveller?

-- 
https://mail.python.org/mailman/listinfo/python-list


Is there a replement for bsddb in python3?

2016-12-09 Thread clvanwall



I have been looking for a simple indexed file database and Berkley-db was what 
I waa used to.  I found that bsddb module was removed from Python3.  Is there a 
replacement for it?
John  Van Walleghen Sent from my Galaxy Tab® A
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Is there a replement for bsddb in python3?

2016-12-09 Thread Paul Rubin
clvanwall  writes:
> I found that bsddb module was removed from Python3.  Is there a
> replacement for it?

Try "dbm" which has a few options for the underlying engine.
Alternatively, try sqlite3.
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Is there a replement for bsddb in python3?

2016-12-09 Thread Ben Finney
clvanwall  writes:

> I found that bsddb module was removed from Python3.  Is there a
> replacement for it?

The ‘dbm’ library is what you need now, I believe
https://docs.python.org/3/library/dbm.html>.

dbm is a generic interface to variants of the DBM database […]
There is a third party interface to the Oracle Berkeley DB.

-- 
 \   “Give a man a fish, and you'll feed him for a day; give him a |
  `\religion, and he'll starve to death while praying for a fish.” |
_o__)   —Anonymous |
Ben Finney

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: CLP stats: last 500 posts

2016-12-09 Thread Terry Reedy

On 12/9/2016 8:39 PM, Steve D'Aprano wrote:

On Sat, 10 Dec 2016 08:07 am, DFS wrote:



As of 04:04PM December 09, 2016

Posts  85 Posters

[...]


Interesting stats, but couldn't you have post-processed the results to avoid
including the defamatory spam posts?

Your post is likely to be removed from the official web archive as it
contains defamatory material.


Reading the news.gmane.org mirror, I never received it.

--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list


% string formatting - what special method is used for %d?

2016-12-09 Thread Veek M
When we do:

print '%s %d' % ('hello', 10)

what special method is being invoked internally within the string-
format-specifier? 

format() invokes format__
print invokes __str__

I'm basically trying to make sense of:

raise TypeError('urkle urkle %s' % list(dictionary))
<=> raise TypeError('urkle urkle %s' % [ key1, val1, key2, val2 ]

So, the % operator reads the format specifier and notices %s and 
therefore calls __str__ in the list class to figure out how to represent 
[ key1, val1, key2, val2 ].

However what if I use %d? How do the other format specs work?
-- 
https://mail.python.org/mailman/listinfo/python-list