Re: 'str' object has no attribute 'intersection' and unhashable set
meInvent bbird writes:
> ...
> is there any algorithm tutorials or books or external library that
> can have a better result for finding repeated lines as group in grouping
> application
I do not yet understand your problem (and still am not ready to
look at your code).
When I need grouping for elements based on some feature ("F(elem)") I usually
use:
groups = {}
for elem in elements:
feature = F(elem)
fg = groups.get(feature)
if fg is None: groups[feature] = fg = []
fg.append(elem)
After this look, the dict "groups" maps features to the list
of elements with the corresponding feature.
For this to work, the features must be hashable.
If they are not naturally, I derive a consistent (same feature
leads to same representation) hashable representation for them
and use that for the grouping.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Assignment versus binding
Chris Angelico wrote:
There's one other consideration. With Python functions, you often want
to run a function for its side effects and ignore its return value.
You can't just ignore a return value in Haskell. The
return value is always going *somewhere*. If you leave
off an argument, the result is going to be a function
instead of whatever type the function was meant to
return. Since Haskell is statically-typed, this will
give you a type error at some point.
print("Heading")
print("===")
print
You're talking about I/O here, which complicates things
somewhat. Essentially, the equivalent of 'print' is
going to be a function that takes the value to be
printed and returns another function of a specific
type. If it's not the right type, it won't fit into
the machinery that executes I/O operations. So leaving
the argument off the print would lead to a type error,
because you'd be trying to plug the print function
itself into the I/O machine, and it's not the right
type.
--
Greg
--
https://mail.python.org/mailman/listinfo/python-list
Re: Assignment versus binding
Rustom Mody wrote: It is fair to say that Haskell's variadic function support is poorer than C's [Collecting into a list fails for different types If you want an arbitrary number of args of unrelated types, you would need to define a wrapper type that can encapsulate all the ones you're interested in. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: static, class and instance methods (Reposting On Python-List Prohibited)
Lawrence D’Oliveiro wrote:
Every function is already a descriptor.
Which you can see with a simple experiment:
>>> def f(self):
... print("self =", self)
...
>>> g = f.__get__(17, None)
>>> g
>>> g()
self = 17
--
Greg
--
https://mail.python.org/mailman/listinfo/python-list
Re: Assignment versus binding
Paul Rubin wrote: It's useful to write some Python things in monadic style, but monads make the most sense as type operators, which don't map onto Python that well. There probably isn't much point in using the monadic style in Python, since the main reason for it is to express stateful processes in a functional framework, and Python already has more direct ways of expressing stateful processes. Also, if you translate monadic code directly from Haskell to Python, you get something which is... not all that pleasant to use. You'll see that when I've posted my Python-based monads essay... -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: static, class and instance methods (Reposting On Python-List Prohibited)
"Lawrence D’Oliveiro" a écrit dans le message de news:[email protected]... On Thursday, October 6, 2016 at 7:54:08 PM UTC+13, ast wrote: But there is no decorator, why ? Is python doing the conversion of funct2 to a descriptor itself, behind the scene ? Every function is already a descriptor. So you get the instance-method behaviour automatically. * static methods are decorated too This is to turn off the instance-method behaviour. I do not understand your answer. Consider this function: def add(a, b): return a+b You say that a function is always stored as a descriptor object, so when I execute sum = f(4, 6) from which class it is supposed to come from ? -- https://mail.python.org/mailman/listinfo/python-list
Re: Assignment versus binding
Ben Bacarisse wrote: import Control.Concurrent spam io = do x <- io; print x; print (x+1) main = spam (do threadDelay (2*10^6); return 1) It matches the Python in that the delay happens once. To get the behaviour being hinted at (two delays) you need to re-bind the IO action: spam io = do x <- io; print x; x <- io; print (x+1) Yep. What's actually happening here is that spam is being passed a function, and that function takes an extra implicit argument representing "the state of the world". The 'x <- io' part is syntactic sugar for something that calls io with the current state of the world, and then evaluates all the stuff after the semicolon with the new state of the world. So the first version only evaluates io once, and the second one evaluates it twice with different arguments (the states of the world at two different times). (At least *conceptually* that's what happens. The Haskell interpreter probably doesn't actually carry the state of the whole world around inside it. :-) -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: any json data validation library recommendation? (Reposting on Python-Lists Prohitibited)
On Thursday, October 6, 2016 at 7:20:58 AM UTC+1, Lawrence D’Oliveiro wrote: > What action are users supposed to take on such errors, other than include > them in a bug report? By users I mean API users (developers). Most common action would be to add a missing field, correct typo or change value type. Failing validation is usually user error, and rarely leads to reporting a bug. -- https://mail.python.org/mailman/listinfo/python-list
Re: static, class and instance methods (Reposting On Python-List Prohibited)
ast wrote: > > "Lawrence D’Oliveiro" a écrit dans le message de > news:[email protected]... >> On Thursday, October 6, 2016 at 7:54:08 PM UTC+13, ast wrote: >>> But there is no decorator, why ? Is python doing the conversion >>> of funct2 to a descriptor itself, behind the scene ? >> >> Every function is already a descriptor. So you get the instance-method >> behaviour automatically. >> >>> * static methods are decorated too >> >> This is to turn off the instance-method behaviour. > > > I do not understand your answer. > Consider this function: > > def add(a, b): > return a+b > > You say that a function is always stored as > a descriptor object, so when I execute > > sum = f(4, 6) > > from which class it is supposed to come from ? It doesn't matter. You can invoke it in the usual way >>> def add(a, b): return a + b ... >>> add(2, 3) 5 and even put it into an instance: >>> class A: pass ... >>> a = A() >>> a.foo = add >>> a.foo(3, 4) 7 Only when you put it into a class the first argument will be the instance >>> A.bar = add >>> a.bar(5, 6) Traceback (most recent call last): File "", line 1, in TypeError: add() takes 2 positional arguments but 3 were given because under the hood add.__get__() is invoked. Doing it manually: >>> add.__get__(a)(5, 6) Traceback (most recent call last): File "", line 1, in TypeError: add() takes 2 positional arguments but 3 were given Unlike Python functions those implemented in C don't support the descriptor protocol out of the box, so >>> A.baz = sum >>> a.baz([1,2,3]) 6 does not pass the instance a as an argument. -- https://mail.python.org/mailman/listinfo/python-list
Re: working with ctypes and complex data structures
On 04-Oct-16 04:48, eryk sun wrote:
On Mon, Oct 3, 2016 at 9:27 PM, Michael Felt wrote:
int perfstat_subsystem_total(
perfstat_id_t *name,
perfstat_subsystem_total_t *userbuff,
int sizeof_struct,
int desired_number);
...
+79 class cpu_total:
+80 def __init__(self):
+81 __perfstat__ = CDLL("libperfstat.a(shr_64.o)")
Thanks for the model below - a lot to think about. Some was expected,
some is a gift!
Python is "a lot to learn". This helps a great deal.
Move loading the library and defining function prototypes to either
the module or class body. Also, don't use dunder names. The convention
for a private attribute is a single underscore. For type safety,
define each function's argtypes (and restype where required, the
default is c_int). For a more Pythonic interface, define a ctypes
errcheck function that encapsulates raising an appropriate exception
when a C call fails. For example:
import ctypes
# If the following types are used generally, define them at
# the module level, else define them in the class namespace
# that uses them.
perfstat = ctypes.CDLL("libperfstat.a(shr_64.o)")
class perfstat_id_t(ctypes.Structure):
pass
IDENTIFIER_LENGTH = 64
class time64_t(ctypes._SimpleCData):
_type_ = ctypes.c_int64._type_
time_t = time64_t
class perfstat_cpu_total_t(ctypes.Structure):
_fields_ = (("ncpus", ctypes.c_int),
("ncpus_cfg", ctypes.c_int),
("description", ctypes.c_char * IDENTIFIER_LENGTH),
("buffer1", ctypes.c_ulonglong * 15),
("lbolt", time_t),
("loadavg", ctypes.c_ulonglong * 3),
("buffer2", ctypes.c_ulonglong * 29),
("ncpus_high", ctypes.c_int),
("puser", ctypes.c_ulonglong),
("psys",ctypes.c_ulonglong),
("pidle", ctypes.c_ulonglong),
("pwait", ctypes.c_ulonglong),
("buffer3", ctypes.c_ulonglong * 12))
def _perfstat_errcheck(result, func, args):
if result == -1:
# get error info and
raise SomeException()
return args
class CPUTotal(object):
# These may be defined here or just referenced here.
_perfstat = perfstat
_perfstat_id_t = perfstat_id_t
_perfstat_cpu_total_t = perfstat_cpu_total_t
_cpu_total = _perfstat.perfstat_cpu_total
_cpu_total.errcheck = _perfstat_errcheck
_cpu_total.argtypes = (
ctypes.POINTER(_perfstat_id_t),
ctypes.POINTER(_perfstat_cpu_total_t),
ctypes.c_int, ctypes.c_int)
def __init__(self):
self._tcpu = self._perfstat_cpu_total_t()
self._cpu_total(None,
ctypes.byref(self._tcpu),
ctypes.sizeof(self._tcpu), 1)
--
https://mail.python.org/mailman/listinfo/python-list
Re: I am comfused about the name behavior of Python in recursion
On Wed, 05 Oct 2016 17:30:22 -0700, 380162267qq wrote: > Google told me Python name is a label attaching to the object. > But in this recursive function,the name 'a' will point to different > number object. > > def rec(a): > a+=1 if a<10: > rec(a) > print(a) > > rec(0) gives me 101 normally.Why it works? Because of the stack > memory management? > > Thank you google local & global variables (this relates to many languages not just python) -- Excerpts From The First Annual Nerd Bowl (#1) JOHN SPLADDEN: Hi, and welcome to the first annual Nerd Bowl in sunny Silicon Valley. BRYANT DUMBELL: We're coming to you live from the Transmeta Dome to watch the battle between the North Carolina Mad Hatters and the Michigan Portalbacks as they compete for the coveted Linus Torvalds Trophy. SPLADDEN: This is shaping up to be one hell of a match. The Mad Hatters -- sponsored by Linux distributor Red Hat -- have been on fire the past month. But the Andover.Net sponsored Michigan Portalbacks are on a tear as well, thanks in part to the stellar performance of Rob "Taco Boy" Malda. DUMBELL: Taco Boy is quite a star, John. Last week at the Kernelbowl he blew away the Transmeta Secret Agents when he scored 51 points singlehandedly in the Flying CompactDiscus round. SPLADDEN: But then Mad Hatter's Alan Cox was voted this season's Most Valuable Hacker in the Eastern Division. So, this game is going to be quite a show. -- https://mail.python.org/mailman/listinfo/python-list
Re: Assignment versus binding
On Thursday, 6 October 2016 14:17:06 UTC+5:30, Gregory Ewing wrote: > Paul Rubin wrote: > > It's useful to write some Python things in monadic style, but monads > > make the most sense as type operators, which don't map onto Python that > > well. > > There probably isn't much point in using the monadic style > in Python, since the main reason for it is to express > stateful processes in a functional framework, and Python > already has more direct ways of expressing stateful processes. Agreed > > Also, if you translate monadic code directly from Haskell > to Python, you get something which is... not all that > pleasant to use. Of course ;-) “Pleasant to use” is not my primary need here. > You'll see that when I've posted my Python-based monads essay... The reason I would like this essay is that I want to teach the notion of first-classing of computation, if possible without the learning curve of Haskell syntax, type-system etc There is a well-known joke in the Haskell space that “Haskell is an excellent imperative language” I guess this is because monads can first-class arbitrary imperative (computational) notions. Since in python the two worlds of purely functional and imperative meet being able to see this in python would be neat! - Anuradha -- https://mail.python.org/mailman/listinfo/python-list
Re: User Interface Suggestions? (newbie)
On Wed, 05 Oct 2016 14:33:43 -0700, Beverly Howard wrote: >>> if it is a pi controlling the system I would tend towards controlling >>> it > from a web page via the network. to keep it updating would require AJAX > style programming of the web page. << > > Thanks. I am interested in eventually doing that, but it seems that > learning how to do it on a local console first would be to my > advantage... especially during initial testing stages. > > fwiw, this project is to transfer (actually re-create) a basic program > that I wrote for a Tandy Model 100 portable back in the early 80's to > control ceramic kilns which get to over 2,000 degrees F. > > Worked perfectly until a few years ago when there were no longer any > Tandy Model 100s available at any cost ;-) (In case anyone is > interested in fossilized projects, see > http://bevhoward.com/kiln/KilnCtrl.htm) > > Thanks again for the response and pointers, > Beverly Howard you may also want to look at the pi newsgroup as well comp.sys.raspberry- pi for any non python queries (& I am sure they would be interested in the python ones as well) -- Two brothers, Mort and Bill, like to sail. While Bill has a great deal of experience, he certainly isn't the rigger Mort is. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python 3.5 amd64 and win32service
On Thu, Oct 6, 2016 at 5:23 AM, Nagy László Zsolt wrote: > "C:\Users\Laci\AppData\Local\Programs\Python\Python35\lib\site-packages\win32\PythonService.exe" I wanted you to run the above executable, not python.exe. If it fails you'll get more information about why it's failing when run directly then when the service controller runs it. Since you're using a per-user installation of Python 3.5, possibly it can't find python35.dll. -- https://mail.python.org/mailman/listinfo/python-list
Re: User Interface Suggestions? (newbie)
On 05/10/2016 23:12, Akira Li wrote:
Beverly Howard writes:
...snip...
A primary question would be, "What are options for building a display
that would update displayed values without scrolling?"
To rewrite only the last character, you could use '\b':
import os
import itertools
import time
for c in map(str.encode, itertools.cycle('\-/|')):
os.write(1, b'\b'+c)
time.sleep(.3)
To rewrite only the last line, you could use '\r':
for i in range(1, 101):
time.sleep(.1)
print('\r{:4.0%}'.format(i/100), flush=True, end='')
To control the full screen in the terminal e.g., to change the position,
you could use *blessings* module [1]:
from blessings import Terminal # $ pip install blessings
line = 'Line in the middle of the terminal.'
term = Terminal()
with term.hidden_cursor(), term.fullscreen():
x = (term.width - len(line)) // 2
y = (term.height - 1) // 2
with term.location(x, y):
print(term.bold_white_on_black(line))
with term.location(0, term.height - 1):
input('press to exit..')
For interactive command-line applications from a simple prompt to full
screen terminal applications, you could use *prompt_toolkit* module [2].
My first goal is to build a program that would interface with a
Raspberry Pi to control a heating process.
For flexibility, you could split your program into a server and a client
that controls it (text, GUI, web).
All this advice seems to be getting out of hand, with suggestions of
'curses' and 'blessings' and using GUI. I've tried 'ncurses' elsewhere
and it was over the top for what I wanted to do.
The OP wants to runs on Pi which I think runs Linux.
So all they are asking is, is there a way of randomly positioning the
cursor within the terminal window so that the next output is at that
position. Something like an escape sequence. Terminal screens have been
around a long time, you'd think someone would have had such a
requirement before!
I'd quite like to know too. However I've just tried a test sequence
("[P1d" to move the cursor to row 1) and it didn't work. If there's
reason why something so basic won't work (hence the need for curses etc)
then that would be useful to know too. (And how does curses manage it?!)
--
Bartc
--
https://mail.python.org/mailman/listinfo/python-list
Re: static, class and instance methods
On Thu, 6 Oct 2016 05:53 pm, ast wrote: [...] > * For instance methods, there is no decorator: > > def funct2(self, a, b): > ... > > self is automatically filled with the instance when we call > funct2 from an instance and not filled if funct2 is called from > a class. > But there is no decorator, why ? Is python doing the conversion > of funct2 to a descriptor itself, behind the scene ? Functions don't need to be decorated because they are already descriptors. Descriptors have __get__, __set__ or __delete__ methods. In the case of methods, it is __get__ which does the work. Normally this happens automatically, but we can do it by hand: py> class X(object): ... def func(self): ... pass ... py> x = X() py> vars(X)['func'] py> vars(X)['func'].__get__(x, X) > py> x.func > > * static methods are decorated too > > @staticmethod > def funct3(a, b): > ... > > The 1st argument is not supposed to be automatically filled > So what the decorator used for ? > Just to distinguish funct3 from an instance method ? Correct. -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: Assignment versus binding
On 06/10/2016 00:27, Chris Angelico wrote: On Thu, Oct 6, 2016 at 9:45 AM, BartC wrote: Small languages are perfectly viable: the JIT version of Lua, for example, is only about 225KB, and is very fast. If I wanted to send you program.lua, and you didn't have Lua, I only need to add luajit.exe and lus51.dll (for Windows). With my system, it would be program.q and r.exe, even for multi-module apps. See above about how restricted the stdib is. Lua, on its own, is not a full-featured language for writing general-purpose applications. It's designed to be embedded in something else, and it's great at that, but it doesn't have all the features of a modern apps language. Some languages don't come with their own libraries but are expected to hook into third party ones. OpenGL for example, which on Windows existed as a language-neutral set of DLL files (iirc). A language might provide 'bindings' as a convenience, or perhaps an easy-to-use set of wrappers. It's just a lower-level way of doing things compared to what you might be used to. You could write an Ook interpreter in a handful of bytes of code, but it's not what you would write an app in. Since you're talking about Windows, I grabbed file sizes for Windows installers; for Pike, the .msi file is a 24MB download, and Python clocks in variously at 18MB (32-bit 2.7.12) up to 28MB (64-bit 3.5.2 full installation). Those kinds of figures are pretty reasonable for full-featured language interpreters. You get enough for it to actually be usable as-is, but you're not getting a full C dev environment. And those are compressed download sizes. I started creating interpreters when they had to run within a tiny corner of 640KB main memory, and they were a tremendously useful way of helping implement applications by using scripts. (Scripts also acted as overlays resident on floppies and loaded as needed.) It also meant users could instantly create and run their own scripts (without using huge, sprawling and, in 1980s, incredibly slow compilers). You can't just dismiss something as useless just because it's relatively small. (I've been using my interpreters to run compilers. What specialist libraries does a compiler need other than file i/o? None!) -- Bartc -- https://mail.python.org/mailman/listinfo/python-list
Is it possible to use 'groupby' asynchronously?
Hi all I have used itertools.groupby before, and I love it. I used it to process a csv file and 'break' on change of a particular field. It worked very well. Now I want to use it to process a database table. I can select the rows in the desired sequence with no problem. However, I am using asyncio, so I am reading the rows asynchronously. My 'reader' class has __aiter__() and __anext__() defined. If I pass the reader to groupby, I get the error message 'object is not iterable'. Before I spend hours trying to figure it out, can anyone confirm if this is doable at all, or is groupby not designed for this. Thanks Frank Millman -- https://mail.python.org/mailman/listinfo/python-list
Re: User Interface Suggestions? (newbie)
BartC wrote:
> On 05/10/2016 23:12, Akira Li wrote:
>> Beverly Howard writes:
>>
>>> ...snip...
>>> A primary question would be, "What are options for building a display
>>> that would update displayed values without scrolling?"
>>
>> To rewrite only the last character, you could use '\b':
>>
>> import os
>> import itertools
>> import time
>> for c in map(str.encode, itertools.cycle('\-/|')):
>> os.write(1, b'\b'+c)
>> time.sleep(.3)
>>
>> To rewrite only the last line, you could use '\r':
>>
>> for i in range(1, 101):
>> time.sleep(.1)
>> print('\r{:4.0%}'.format(i/100), flush=True, end='')
>>
>> To control the full screen in the terminal e.g., to change the position,
>> you could use *blessings* module [1]:
>>
>> from blessings import Terminal # $ pip install blessings
>>
>> line = 'Line in the middle of the terminal.'
>> term = Terminal()
>> with term.hidden_cursor(), term.fullscreen():
>> x = (term.width - len(line)) // 2
>> y = (term.height - 1) // 2
>> with term.location(x, y):
>> print(term.bold_white_on_black(line))
>>
>> with term.location(0, term.height - 1):
>> input('press to exit..')
>>
>> For interactive command-line applications from a simple prompt to full
>> screen terminal applications, you could use *prompt_toolkit* module [2].
>>
>>> My first goal is to build a program that would interface with a
>>> Raspberry Pi to control a heating process.
>>>
>>
>> For flexibility, you could split your program into a server and a client
>> that controls it (text, GUI, web).
>
> All this advice seems to be getting out of hand, with suggestions of
> 'curses' and 'blessings' and using GUI. I've tried 'ncurses' elsewhere
> and it was over the top for what I wanted to do.
>
> The OP wants to runs on Pi which I think runs Linux.
>
> So all they are asking is, is there a way of randomly positioning the
> cursor within the terminal window so that the next output is at that
> position. Something like an escape sequence. Terminal screens have been
> around a long time, you'd think someone would have had such a
> requirement before!
>
> I'd quite like to know too. However I've just tried a test sequence
> ("[P1d" to move the cursor to row 1) and it didn't work. If there's
> reason why something so basic won't work (hence the need for curses etc)
> then that would be useful to know too. (And how does curses manage it?!)
Perhaps you picked the wrong escape sequence? I tried ;H
taken from , and it
seems to work (the test system is not a Pi though):
$ python3
Python 3.4.3 (default, Sep 14 2016, 12:36:27)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> for i in range(20, 10, -1):
... print("\033[{0};5HHello from line {0}".format(i), end="")
...
Hello from line 11>>>
Hello from line 12
Hello from line 13
Hello from line 14
Hello from line 15
Hello from line 16
Hello from line 17
Hello from line 18
Hello from line 19
Hello from line 20
--
https://mail.python.org/mailman/listinfo/python-list
Re: User Interface Suggestions? (newbie)
On 06/10/2016 13:38, Peter Otten wrote:
> BartC wrote:
>> All this advice seems to be getting out of hand, with suggestions of
>> 'curses' and 'blessings' and using GUI. I've tried 'ncurses' elsewhere
>> and it was over the top for what I wanted to do.
>>
>> The OP wants to runs on Pi which I think runs Linux.
>>
>> So all they are asking is, is there a way of randomly positioning the
>> cursor within the terminal window so that the next output is at that
>> position. Something like an escape sequence. Terminal screens have been
>> around a long time, you'd think someone would have had such a
>> requirement before!
>>
>> I'd quite like to know too. However I've just tried a test sequence
>> ("[P1d" to move the cursor to row 1) and it didn't work. If there's
>> reason why something so basic won't work (hence the need for curses etc)
>> then that would be useful to know too. (And how does curses manage it?!)
>
> Perhaps you picked the wrong escape sequence? I tried ;H
> taken from , and it
> seems to work (the test system is not a Pi though):
>
For lightweight stuff like this, colorama is useful (basically wraps the
ANSI sequences):
https://pypi.python.org/pypi/colorama
TJG
--
https://mail.python.org/mailman/listinfo/python-list
Re: User Interface Suggestions? (newbie)
On 10/05/2016 11:33 PM, Chris Angelico wrote:
On Thu, Oct 6, 2016 at 8:19 AM, Beverly Howard wrote:
Thanks for the responses... appreciated.
print("value value data data data", end="\r") <<
That makes sense, but it also seems to suggest that there is no other way to
position the cursor prior to printing.
For example, if that line is halfway down the screen, is there any way to
position the cursor two lines above it?
fwiw, I am open to creating functions if there are positioning options, both to
meet the need, but, more importantly, to learn.
What I showed you was the very simplest way of doing things. If you
want to move the cursor around, I would recommend the 'curses'
library:
https://docs.python.org/3/library/curses.html
https://docs.python.org/3/howto/curses.html
There are other ways, too; with more info on what you're trying to
accomplish, we could better advise. It might be that a GUI will serve
you well, particularly if you have several pieces of information that
you want to update.
ChrisA
Since someone mentioned curses, I'll add that I've used npyscreen (built
on top of ncurses), successfully, creating small guis with very few code
lines.
If you like oldschool gui, it's a must.
http://npyscreen.readthedocs.io/introduction.html
jm
--
https://mail.python.org/mailman/listinfo/python-list
Re: User Interface Suggestions? (newbie)
On 06/10/2016 13:38, Peter Otten wrote:
BartC wrote:
All this advice seems to be getting out of hand, with suggestions of
'curses' and 'blessings' and using GUI. I've tried 'ncurses' elsewhere
and it was over the top for what I wanted to do.
The OP wants to runs on Pi which I think runs Linux.
So all they are asking is, is there a way of randomly positioning the
cursor within the terminal window so that the next output is at that
position. Something like an escape sequence. Terminal screens have been
around a long time, you'd think someone would have had such a
requirement before!
I'd quite like to know too. However I've just tried a test sequence
("[P1d" to move the cursor to row 1) and it didn't work. If there's
reason why something so basic won't work (hence the need for curses etc)
then that would be useful to know too. (And how does curses manage it?!)
Perhaps you picked the wrong escape sequence? I tried ;H
taken from , and it
seems to work (the test system is not a Pi though):
Hey, that worked!
(Maybe I'll finally be able to port my editor and IDE to Linux after all.)
Maybe that can help the OP too. These sequences can trivially be wrapped
with easy-to-use functions such as setpos(row,column). (Ideally there
needs to be a way to get terminal size in rows and columns too.)
--
Bartc
--
https://mail.python.org/mailman/listinfo/python-list
Re: Copying a compiled Python from one system to another
On 2016-10-02 00:25, Steve D'Aprano wrote: > On Sun, 2 Oct 2016 01:58 pm, Chris Angelico wrote: >> Hmm, I've possibly missed something here, which may indicate a >> problem. Why can't your existing machines build? Is it because they >> have too-old versions of tools, and if so, which? > Yes, this. You need gcc 4.8 or better to build CPython 3.6, and the most > recent any of my systems support is 4.4. I'm using a gcc 4.2 to build 3.6. -- https://mail.python.org/mailman/listinfo/python-list
Re: static, class and instance methods (Reposting On Python-List Prohibited)
On Thu, 6 Oct 2016 08:03 pm, ast wrote: > Consider this function: > > def add(a, b): > return a+b > > You say that a function is always stored as > a descriptor object, so when I execute > > sum = f(4, 6) > > from which class it is supposed to come from ? It doesn't. The descriptor protocol only gets called by classes, so when you call a function directly, the special __get__ method isn't used. -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: Copying a compiled Python from one system to another
On Fri, 7 Oct 2016 01:36 am, Ned Deily wrote: > On 2016-10-02 00:25, Steve D'Aprano wrote: >> On Sun, 2 Oct 2016 01:58 pm, Chris Angelico wrote: >>> Hmm, I've possibly missed something here, which may indicate a >>> problem. Why can't your existing machines build? Is it because they >>> have too-old versions of tools, and if so, which? >> Yes, this. You need gcc 4.8 or better to build CPython 3.6, and the most >> recent any of my systems support is 4.4. > > I'm using a gcc 4.2 to build 3.6. How on earth do you do that? What magic incantations are you using? I understand that Python 3.6 is now using C99 features that aren't available before gcc 4.8. -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: User Interface Suggestions? (newbie)
On Thu, 6 Oct 2016 10:03 pm, BartC wrote:
> I'd quite like to know too. However I've just tried a test sequence
> ("[P1d" to move the cursor to row 1) and it didn't work. If there's
> reason why something so basic won't work (hence the need for curses etc)
> then that would be useful to know too. (And how does curses manage it?!)
I believe that curses uses a large database of terminal types and associated
escape sequences. So when you start it up, it determines what terminal type
you have (that's black magic itself) and then works out what escape
sequences you need.
--
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.
--
https://mail.python.org/mailman/listinfo/python-list
Re: static, class and instance methods (Reposting On Python-List Prohibited)
"Steve D'Aprano" a écrit dans le message de news:[email protected]... On Thu, 6 Oct 2016 08:03 pm, ast wrote: Consider this function: def add(a, b): return a+b You say that a function is always stored as a descriptor object, so when I execute sum = f(4, 6) from which class it is supposed to come from ? It doesn't. The descriptor protocol only gets called by classes, so when you call a function directly, the special __get__ method isn't used. yes, it is clear, thanks to all (I didn't know that functions were descriptors with a __get__ method) -- https://mail.python.org/mailman/listinfo/python-list
BeautifulSoup help !!
So I've just started up with python and an assignment was given to me by a
company as an recruitment task.
I need to web scrap the coupons of all the websites available on
http://www.couponraja.in and export it to csv format.
The details which I need to be present in the csv are the coupon title , vendor
, validity , description/detail , url to the vendor , image url of the coupon.
I have gone through many tutorials on beautifulsoup and have a beginners
understanding of using it. Wrote a code as well , but the problem Im facing
here is when i collect info from the divs which contains all those info , Im
getting it in with all the html tags and the info is clustered.
Code m using :
import requests
from bs4 import BeautifulSoup
url = "https://www.couponraja.in/amazon";
r = requests.get(url)
soup = BeautifulSoup(r.content)
g_data = soup.find_all("div", {"class": "nw-offrtxt"})
for item in g_data:
print item.contents
also will need help on how to export the info to csv format , I just know I
need to import csv then write the information to a csv file.
But not getting through on how to achieve that.
Any help will be appreciated.
--
https://mail.python.org/mailman/listinfo/python-list
Re: BeautifulSoup help !!
On Thu, 06 Oct 2016 08:22:05 -0700, desolate.soul.me wrote: > So I've just started up with python and an assignment was given to me by > a company as an recruitment task. > so by your own admission you have just started with python yet you consider your self suitable for employment? -- "Unibus timeout fatal trap program lost sorry" - An error message printed by DEC's RSTS operating system for the PDP-11 -- https://mail.python.org/mailman/listinfo/python-list
Re: BeautifulSoup help !!
On Thursday, October 6, 2016 at 9:00:21 PM UTC+5:30, alister wrote: > On Thu, 06 Oct 2016 08:22:05 -0700, desolate.soul.me wrote: > > > So I've just started up with python and an assignment was given to me by > > a company as an recruitment task. > > > so by your own admission you have just started with python yet you > consider your self suitable for employment? > > > -- > "Unibus timeout fatal trap program lost sorry" > - An error message printed by DEC's RSTS operating system for the PDP-11 yup ... training will be provided further , all they want to confirm is atleast I have basic knowledge of how the language works and they wont have to tell me how to install python on the system or any further extensions to it. Im not quite as to why they provided me with this assignment when my cv clearly states that im good with .net and not python. I hope you could help here and not just trolling around. -- https://mail.python.org/mailman/listinfo/python-list
Re: Is it possible to use 'groupby' asynchronously?
On Thu, Oct 6, 2016 at 11:09 PM, Frank Millman wrote: > Hi all > > I have used itertools.groupby before, and I love it. I used it to process a > csv file and 'break' on change of a particular field. It worked very well. > > Now I want to use it to process a database table. I can select the rows in > the desired sequence with no problem. However, I am using asyncio, so I am > reading the rows asynchronously. > > Before I spend hours trying to figure it out, can anyone confirm if this is > doable at all, or is groupby not designed for this. Most of itertools is going to assume synchronous iterables. However, the docs give pure-Python equivalents for quite a few: https://docs.python.org/3/library/itertools.html#itertools.groupby You may be able to tweak that into an actual "itertools.agroupby" or "aitertools.groupby". ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: BeautifulSoup help !!
On Fri, Oct 7, 2016 at 2:50 AM, Navneet Siddhant wrote: > On Thursday, October 6, 2016 at 9:00:21 PM UTC+5:30, alister wrote: >> On Thu, 06 Oct 2016 08:22:05 -0700, desolate.soul.me wrote: >> >> > So I've just started up with python and an assignment was given to me by >> > a company as an recruitment task. >> > >> so by your own admission you have just started with python yet you >> consider your self suitable for employment? > > yup ... training will be provided further , all they want to confirm is > atleast I have basic knowledge of how the language works and they wont have > to tell me how to install python on the system or any further extensions to > it. Im not quite as to why they provided me with this assignment when my cv > clearly states that im good with .net and not python. > > I hope you could help here and not just trolling around. It's not trolling around, it's a very serious question. You are asking for assistance with something that was assigned to you *as a recruitment task*. Were you told that asking for help was a legitimate solution? If you need to ask for help with this task, will you also need to ask for help with everything you'll be doing once employed? Maybe the right thing to do is dedicate a few solid hours (an evening or afternoon or something) to just messing around with Python and bs4. Learn the stuff you're trying to get hired for. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: BeautifulSoup help !!
On Thu, 06 Oct 2016 08:50:25 -0700, Navneet Siddhant wrote: > On Thursday, October 6, 2016 at 9:00:21 PM UTC+5:30, alister wrote: >> On Thu, 06 Oct 2016 08:22:05 -0700, desolate.soul.me wrote: >> >> > So I've just started up with python and an assignment was given to me >> > by a company as an recruitment task. >> > >> so by your own admission you have just started with python yet you >> consider your self suitable for employment? >> >> >> -- >> "Unibus timeout fatal trap program lost sorry" >> - An error message printed by DEC's RSTS operating system for the >> PDP-11 > > > yup ... training will be provided further , all they want to confirm is > atleast I have basic knowledge of how the language works and they wont > have to tell me how to install python on the system or any further > extensions to it. Im not quite as to why they provided me with this > assignment when my cv clearly states that im good with .net and not > python. > > I hope you could help here and not just trolling around. well if this is just a start of training it is a bit different, we dont like writing code for homework assignments (even less for comercial assignments) but can give some general pointers. you code currently gets the div from the soup object of the page & as you say it contains all of the HTML (it is also a soup object) what you need to do is use another find_all to extract the elements that contain the information you require. depending on the page layout you may need to drill down multiple levels & combine multiple different elements to get what you require performing the tasks manually with the page html source before automating is the usual approach. -- It turned out that the worm exploited three or four different holes in the system. From this, and the fact that we were able to capture and examine some of the source code, we realized that we were dealing with someone very sharp, probably not someone here on campus. -- Dr. Richard LeBlanc, associate professor of ICS, in Georgia Tech's campus newspaper after the Internet worm. -- https://mail.python.org/mailman/listinfo/python-list
Re: Copying a compiled Python from one system to another
On 2016-10-06 11:04, Steve D'Aprano wrote: > On Fri, 7 Oct 2016 01:36 am, Ned Deily wrote: >> On 2016-10-02 00:25, Steve D'Aprano wrote: >>> On Sun, 2 Oct 2016 01:58 pm, Chris Angelico wrote: Hmm, I've possibly missed something here, which may indicate a problem. Why can't your existing machines build? Is it because they have too-old versions of tools, and if so, which? >>> Yes, this. You need gcc 4.8 or better to build CPython 3.6, and the most >>> recent any of my systems support is 4.4. >> >> I'm using a gcc 4.2 to build 3.6. > > How on earth do you do that? What magic incantations are you using? > > I understand that Python 3.6 is now using C99 features that aren't available > before gcc 4.8. 3.6 is only using a subset of the C99 features that were proposed in the earlier python-dev discussion. In this case, I am using the Apple version of GNU gcc-4.2 that shipped with Xcode 3 on OS X 10.5 and 10.6. I haven't tried it with other versions of gcc-4.2 but it's worth trying. -- https://mail.python.org/mailman/listinfo/python-list
Re: BeautifulSoup help !!
On Fri, 7 Oct 2016 02:30 am, alister wrote: > On Thu, 06 Oct 2016 08:22:05 -0700, desolate.soul.me wrote: > >> So I've just started up with python and an assignment was given to me by >> a company as an recruitment task. >> > so by your own admission you have just started with python yet you > consider your self suitable for employment? What's your problem Alister? Do you think that junior devs aren't allowed to ask for help? Desolate.Soul.Me has either applied for a job, and their interview test is "do this task using Python", or he's been accepted in a new job, and the same applies. Whether it's a learning exercise, a test of skill + initiative, or actual work given to a junior developer, Desolate.Soul.Me is perfectly entitled to ask for help. This isn't some artificially constrained academic homework, with stupidly strict and hypocritical rules about so-called "plagiarism". This is the real world where you take all the help you can get and you shouldn't feel ashamed for asking for help. ESPECIALLY in the open source world, including Python, where one of the community values is to share expertise. My own employer has hired plenty of junior developers and given them relatively minor tasks to do as a learning exercise. We're not going to trust a junior developer with a critical piece of code, but we might say: "Scrape this website. Use Python. Here's the Python For Beginners book. Here's the Python documentation, and a few more forums where you can ask for help. If you get stuck, and aren't getting useful answers from the forums, you can ask Lisa. But not today, as she's busy doing a critical release and can't be disturbed." P.S. Desolate.Soul.Me, you might be taken a bit more seriously if you give a name, or at least a moniker or nick-name which is easier for others to refer to you by. It doesn't have to be your birthname, or legal name. What do your friends and workmates call you? I don't know Beautiful Soup, so I'm afraid I can't help. -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: BeautifulSoup help !!
+1 at Steve On 6 Oct 2016 19:17, "Steve D'Aprano" wrote: > On Fri, 7 Oct 2016 02:30 am, alister wrote: > > > On Thu, 06 Oct 2016 08:22:05 -0700, desolate.soul.me wrote: > > > >> So I've just started up with python and an assignment was given to me by > >> a company as an recruitment task. > >> > > so by your own admission you have just started with python yet you > > consider your self suitable for employment? > > What's your problem Alister? Do you think that junior devs aren't allowed > to > ask for help? > > Desolate.Soul.Me has either applied for a job, and their interview test > is "do this task using Python", or he's been accepted in a new job, and the > same applies. > > Whether it's a learning exercise, a test of skill + initiative, or actual > work given to a junior developer, Desolate.Soul.Me is perfectly entitled > to > ask for help. > > This isn't some artificially constrained academic homework, with stupidly > strict and hypocritical rules about so-called "plagiarism". This is the > real world where you take all the help you can get and you shouldn't feel > ashamed for asking for help. ESPECIALLY in the open source world, including > Python, where one of the community values is to share expertise. > > My own employer has hired plenty of junior developers and given them > relatively minor tasks to do as a learning exercise. We're not going to > trust a junior developer with a critical piece of code, but we might say: > > "Scrape this website. Use Python. Here's the Python For Beginners > book. Here's the Python documentation, and a few more forums where > you can ask for help. If you get stuck, and aren't getting useful > answers from the forums, you can ask Lisa. But not today, as she's > busy doing a critical release and can't be disturbed." > > > P.S. Desolate.Soul.Me, you might be taken a bit more seriously if you > give a > name, or at least a moniker or nick-name which is easier for others to > refer to you by. It doesn't have to be your birthname, or legal name. What > do your friends and workmates call you? > > > I don't know Beautiful Soup, so I'm afraid I can't help. > > > > -- > Steve > “Cheer up,” they said, “things could be worse.” So I cheered up, and sure > enough, things got worse. > > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: BeautifulSoup help !!
On Thursday, October 6, 2016 at 9:42:47 PM UTC+5:30, Steve D'Aprano wrote: > On Fri, 7 Oct 2016 02:30 am, alister wrote: > > > On Thu, 06 Oct 2016 08:22:05 -0700, desolate.soul.me wrote: > > > >> So I've just started up with python and an assignment was given to me by > >> a company as an recruitment task. > >> > > so by your own admission you have just started with python yet you > > consider your self suitable for employment? > > What's your problem Alister? Do you think that junior devs aren't allowed to > ask for help? > > Desolate.Soul.Me has either applied for a job, and their interview test > is "do this task using Python", or he's been accepted in a new job, and the > same applies. > > Whether it's a learning exercise, a test of skill + initiative, or actual > work given to a junior developer, Desolate.Soul.Me is perfectly entitled to > ask for help. > > This isn't some artificially constrained academic homework, with stupidly > strict and hypocritical rules about so-called "plagiarism". This is the > real world where you take all the help you can get and you shouldn't feel > ashamed for asking for help. ESPECIALLY in the open source world, including > Python, where one of the community values is to share expertise. > > My own employer has hired plenty of junior developers and given them > relatively minor tasks to do as a learning exercise. We're not going to > trust a junior developer with a critical piece of code, but we might say: > > "Scrape this website. Use Python. Here's the Python For Beginners > book. Here's the Python documentation, and a few more forums where > you can ask for help. If you get stuck, and aren't getting useful > answers from the forums, you can ask Lisa. But not today, as she's > busy doing a critical release and can't be disturbed." > > > P.S. Desolate.Soul.Me, you might be taken a bit more seriously if you give a > name, or at least a moniker or nick-name which is easier for others to > refer to you by. It doesn't have to be your birthname, or legal name. What > do your friends and workmates call you? > > > I don't know Beautiful Soup, so I'm afraid I can't help. > > > > -- > Steve > “Cheer up,” they said, “things could be worse.” So I cheered up, and sure > enough, things got worse. Yes ... Im allowed to take help as this group and python forums and other links were suggested to me by them. Have been trying since morning not getting the desired results , posted here as last resort .. and Im ok if I dont find what m seeking. Though m getting close , lets see what I could achieve. -- https://mail.python.org/mailman/listinfo/python-list
Re: BeautifulSoup help !!
On Thursday, October 6, 2016 at 9:57:46 PM UTC+5:30, Navneet Siddhant wrote: > On Thursday, October 6, 2016 at 9:42:47 PM UTC+5:30, Steve D'Aprano wrote: > > On Fri, 7 Oct 2016 02:30 am, alister wrote: > > > > > On Thu, 06 Oct 2016 08:22:05 -0700, desolate.soul.me wrote: > > > > > >> So I've just started up with python and an assignment was given to me by > > >> a company as an recruitment task. > > >> > > > so by your own admission you have just started with python yet you > > > consider your self suitable for employment? > > > > What's your problem Alister? Do you think that junior devs aren't allowed to > > ask for help? > > > > Desolate.Soul.Me has either applied for a job, and their interview test > > is "do this task using Python", or he's been accepted in a new job, and the > > same applies. > > > > Whether it's a learning exercise, a test of skill + initiative, or actual > > work given to a junior developer, Desolate.Soul.Me is perfectly entitled to > > ask for help. > > > > This isn't some artificially constrained academic homework, with stupidly > > strict and hypocritical rules about so-called "plagiarism". This is the > > real world where you take all the help you can get and you shouldn't feel > > ashamed for asking for help. ESPECIALLY in the open source world, including > > Python, where one of the community values is to share expertise. > > > > My own employer has hired plenty of junior developers and given them > > relatively minor tasks to do as a learning exercise. We're not going to > > trust a junior developer with a critical piece of code, but we might say: > > > > "Scrape this website. Use Python. Here's the Python For Beginners > > book. Here's the Python documentation, and a few more forums where > > you can ask for help. If you get stuck, and aren't getting useful > > answers from the forums, you can ask Lisa. But not today, as she's > > busy doing a critical release and can't be disturbed." > > > > > > P.S. Desolate.Soul.Me, you might be taken a bit more seriously if you give a > > name, or at least a moniker or nick-name which is easier for others to > > refer to you by. It doesn't have to be your birthname, or legal name. What > > do your friends and workmates call you? > > > > > > I don't know Beautiful Soup, so I'm afraid I can't help. > > > > > > > > -- > > Steve > > “Cheer up,” they said, “things could be worse.” So I cheered up, and sure > > enough, things got worse. > > Yes ... Im allowed to take help as this group and python forums and other > links were suggested to me by them. Have been trying since morning not > getting the desired results , posted here as last resort .. and Im ok if I > dont find what m seeking. Though m getting close , lets see what I could > achieve. Btw my institute has referred me for this job in the company who has further emailed me with the assignment to complete after which I will have face to face interviews. Im still wondering as why I was told to do a python assignment when nowhere in my CV i have mentioned about python nor it was taught to us in the curriculum -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and ssh for remote login
On 6 Oct 2016 04:56, "Michael Torrie" wrote: > > On 10/05/2016 11:46 AM, Noah wrote: > > Hello folk, > > > > I would like to use a python script to ssh into a server using a username > > and password and perhaps ass port. > > > > Any ideas on how to script that. > > If paramiko doesn't fit your needs, traditionally this sort of work was > done with the pexpect module for drying a TTY. There is a submodule of > pexpect called pxssh for automating things. > Hi Micheal Thank youn for your suggestion. I played around with paramiko today and the results are owesome. > http://pexpect.readthedocs.io/en/stable/api/pxssh.html > > Note that pexpect uses your normal ssh binary. Paramiko is a complete > implementation of the ssh protocol in python. Both modules are useful > and fill certain needs. So i am going to also try pexpect and everything pexpect and i will let you know. Thank you so much Noah -- https://mail.python.org/mailman/listinfo/python-list
Re: Is it possible to use 'groupby' asynchronously?
On Thu, Oct 6, 2016 at 9:57 AM, Chris Angelico wrote: > On Thu, Oct 6, 2016 at 11:09 PM, Frank Millman wrote: >> Hi all >> >> I have used itertools.groupby before, and I love it. I used it to process a >> csv file and 'break' on change of a particular field. It worked very well. >> >> Now I want to use it to process a database table. I can select the rows in >> the desired sequence with no problem. However, I am using asyncio, so I am >> reading the rows asynchronously. >> >> Before I spend hours trying to figure it out, can anyone confirm if this is >> doable at all, or is groupby not designed for this. > > Most of itertools is going to assume synchronous iterables. However, > the docs give pure-Python equivalents for quite a few: > > https://docs.python.org/3/library/itertools.html#itertools.groupby > > You may be able to tweak that into an actual "itertools.agroupby" or > "aitertools.groupby". There is already a third-party "aitertools" library, so there should be no need to write one from scratch. https://pypi.python.org/pypi/aitertools/0.1.0 -- https://mail.python.org/mailman/listinfo/python-list
Re: BeautifulSoup help !!
On Fri, 7 Oct 2016 03:00 am, Chris Angelico wrote: > You are asking > for assistance with something that was assigned to you *as a > recruitment task*. Were you told that asking for help was a legitimate > solution? Why should he need to be told that? Asking for help *is* a legitimate solution, just as legitimate as Reading The Fine Manual, searching on Google, asking your work peers for mentoring, or doing a training course. What's next? Are we going to start dumping on people for reading the manual? "If you can't intuit the right method calls, you're cheating"? "What, you had to *study* to learn the language? With a teacher and everything? Cheater!" Asking questions and sharing expertise is what we're here for, and it is an absolutely critical part of open-source communities. The idea that people cannot ask for help because they are working, or being interviewed for work, should be anathema to us all. Schooling is slightly different in that (1) whether we like it or not, many schools enforce ridiculously stupid and strict rules against plagiarism, so bizarrely the best way we can help students is to refuse to help them[1]; and (2) when people are learning, making their own mistakes and wrong turns is often a good learning exercise. For students, it is the learning process itself which is more important than the solution -- and the less experienced the student, the more that is the case. But heaven forbid that students copy recipes from the documentation, from the ActiveState Cookbook, or Stackoverflow without attribution like professionals do. The current anti-plagiarism (so-called plagiarism) climate goes directly against the principles of free and open code and information. Requiring every trivial thought and phrase to be wholly original[2] or else paid for (whether paid for in money or in credit/ reputation points) is a direct assault against the intellectual Commons. [1] The negative effects of an accusation of plagiarism for few lines of allegedly copied code or text may be far, far worse than if the student actually learns nothing. If you learn nothing, there's at least a chance that your teacher will grade on a curve and you'll squeeze in a passing grade, but allegedly plagiarise, even *your own work*[3] and you may lose all your academic future. [2] An impossibility. [3] Quite possibly the stupidest thing that has come out of academia since post-modernism. -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
segfault using shutil.make_archive
I need to zip up a directory that's about 400mb. I'm using shutil.make_archive and I'm getting this response: Segmentation fault: 11 (core dumped) The code is straightforward (and works on other, smaller dirs): shutil.make_archive(os.path.join(zip_dir, zname), 'zip', tgt_dir) I guess I could drop the convenience of make_archive and use zipfile but that seems to be exactly what make_archive does. I'm on FreeBSD, python2.7. Anyone seen this before? thanks, --tim -- https://mail.python.org/mailman/listinfo/python-list
Re: User Interface Suggestions? (newbie)
On 2016-10-06, BartC wrote:
> All this advice seems to be getting out of hand, with suggestions of
> 'curses' and 'blessings' and using GUI. I've tried 'ncurses' elsewhere
> and it was over the top for what I wanted to do.
>
> The OP wants to runs on Pi which I think runs Linux.
It can run Linux. Whether the OP's is or not, I don't know.
> So all they are asking is, is there a way of randomly positioning
> the cursor within the terminal window so that the next output is at
> that position.
Yes.
> Something like an escape sequence. Terminal screens have been around
> a long time, you'd think someone would have had such a requirement
> before!
Yes, they did. They wrote the curses library.
> I'd quite like to know too. However I've just tried a test sequence
> ("[P1d" to move the cursor to row 1) and it didn't work.
That doesn't look like an ANSI terminal escape sequence to me -- but I
have no idea what terminal type you're using, so I can't tell you
whether it's right or not.
The ANSI seqeunce to move the cursor to row 1, column 1, is
'[1;1m'. Both positions default to 1, so '[m' does the same
thing.
> If there's reason why something so basic won't work (hence the need
> for curses etc) then that would be useful to know too. (And how does
> curses manage it?!)
Curses knows how to deal with various different terminal types and it
also knows about the intracacies of the Unix tty API.
If you want to, you can just assume your terminal uses ANSI escape
sequences:
https://en.wikipedia.org/wiki/ANSI_escape_code
That will mostly work on most terminals you run into these days.
If you want to go one step further, you can use the terminfo library
to deal with different terminal types, but I have no idea how to use
it without ncurses. If people care about their programs working with
different terminal types, they use ncurses.
--
Grant Edwards grant.b.edwardsYow! ! Up ahead! It's a
at DONUT HUT!!
gmail.com
--
https://mail.python.org/mailman/listinfo/python-list
A newbie doubt on methods/functions calling
Hi I just started learning python. Is there any way to call functions in different way ? Rather calling obj.function(arg1, arg2) I would like to call like below "obj function arg1 arg2" this function is part of a class. class myClass: def function(arg1, arg2): # do something Is it possible to do in python ? May be not directly but using some other methods. Regards, Puneet -- https://mail.python.org/mailman/listinfo/python-list
Re: BeautifulSoup help !!
On Thursday, October 6, 2016 at 8:52:18 PM UTC+5:30, Navneet Siddhant wrote:
> So I've just started up with python and an assignment was given to me by a
> company as an recruitment task.
>
> I need to web scrap the coupons of all the websites available on
> http://www.couponraja.in and export it to csv format.
> The details which I need to be present in the csv are the coupon title ,
> vendor , validity , description/detail , url to the vendor , image url of the
> coupon.
>
> I have gone through many tutorials on beautifulsoup and have a beginners
> understanding of using it. Wrote a code as well , but the problem Im facing
> here is when i collect info from the divs which contains all those info , Im
> getting it in with all the html tags and the info is clustered.
>
> Code m using :
>
> import requests
> from bs4 import BeautifulSoup
>
> url = "https://www.couponraja.in/amazon";
> r = requests.get(url)
> soup = BeautifulSoup(r.content)
> g_data = soup.find_all("div", {"class": "nw-offrtxt"})
> for item in g_data:
> print item.contents
>
> also will need help on how to export the info to csv format , I just know I
> need to import csv then write the information to a csv file.
> But not getting through on how to achieve that.
>
> Any help will be appreciated.
Thanx for the support Steve . N yes you can call me Sid .. :) . . ..
I guess I shouldnt have mentioned as this was a recruitment task. If needed I
can post a screenshot of the mail I got which says I can take help from
anywhere possible as long as the assignment is done. Wont be simply copying
pasting the code as question related to the same will be asked in the interview.
I just need a proper understanding as to what I need to do to get the results.
Also how to export the result to csv format.
--
https://mail.python.org/mailman/listinfo/python-list
Re: User Interface Suggestions? (newbie)
On 2016-10-06, Steve D'Aprano wrote:
> On Thu, 6 Oct 2016 10:03 pm, BartC wrote:
>
>> I'd quite like to know too. However I've just tried a test sequence
>> ("[P1d" to move the cursor to row 1) and it didn't work. If there's
>> reason why something so basic won't work (hence the need for curses etc)
>> then that would be useful to know too. (And how does curses manage it?!)
>
> I believe that curses uses a large database of terminal types and
> associated escape sequences. So when you start it up, it determines
> what terminal type you have (that's black magic itself) and then
> works out what escape sequences you need.
Nothing magic, it just looks at the 'TERM' enviroment varible and
looks up that value in the terminfo database.
--
Grant Edwards grant.b.edwardsYow! I wonder if there's
at anything GOOD on tonight?
gmail.com
--
https://mail.python.org/mailman/listinfo/python-list
Re: BeautifulSoup help !!
On Fri, Oct 7, 2016 at 3:38 AM, Steve D'Aprano wrote: > On Fri, 7 Oct 2016 03:00 am, Chris Angelico wrote: > >> You are asking >> for assistance with something that was assigned to you *as a >> recruitment task*. Were you told that asking for help was a legitimate >> solution? > > Why should he need to be told that? Asking for help *is* a legitimate > solution, just as legitimate as Reading The Fine Manual, searching on > Google, asking your work peers for mentoring, or doing a training course. Let me clarify a bit. Is "asking someone else to write your code" acceptable? I would say no, in the same way that getting someone else to do your job interview for you is inappropriate. You're right, though, "asking for help" was too broad a description for the situation. Apologies to the OP; of course you can ask questions and get help. You just need to figure out where the boundaries are between "stuff you know and are being hired for", "stuff you don't know and are being hired for, and are learning on the job", and "stuff you don't know and don't need to know, and can get someone else to do for you". The boundary between the first two is not overly important, but if there's not much in the first category, you're going to be very much in over your head. It's the boundary between the latter two that's more important, and which determines the kind of help you're asking for. But please don't accuse us of "trolling around" when you're asked this kind of thing. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: BeautifulSoup help !!
On Fri, Oct 7, 2016 at 4:00 AM, Navneet Siddhant wrote: > I guess I shouldnt have mentioned as this was a recruitment task. If needed I > can post a screenshot of the mail I got which says I can take help from > anywhere possible as long as the assignment is done. Wont be simply copying > pasting the code as question related to the same will be asked in the > interview. > I just need a proper understanding as to what I need to do to get the results. > Also how to export the result to csv format. A screenshot isn't necessary - we trust you to not flat-out lie to us. (And if you did, well, this is a public list, so your deception would burn you pretty thoroughly once someone finds out.) Stating that you can take help from anywhere would have been a good clarification. Anyway. One of the annoying facts of the real world is that web scraping is *hard*. We have awesome tools like Beautiful Soup that save us a lot of hassle, but ultimately, you have to look at the results and figure out which parts are "interesting" (that is, the parts that have the data you want, or tell you about its structure, or something like that). I strongly recommend messing with bs4 at the interactive prompt; basically, just play around with everything you get hold of. Eventually, you want to be building up a series of rows, where each row is a list of column values; you write a row to the CSV file, and your job's done. Most likely, you're going to have some sort of primary loop - outside of that loop you have bs4 navigation to get you the main block of info, and inside it, you parse through some wad of stuff to find the truly interesting info, and at the bottom of the loop, you write something to the CSV file. Hope that's of some help! ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: A newbie doubt on methods/functions calling
[email protected] wrote: > Hi > > I just started learning python. Is there any way to call functions in > different way ? > > Rather calling obj.function(arg1, arg2) I would like to call like below > > "obj function arg1 arg2" How would the machine reading the above know that you didn't mean obj(function, arg1, arg2) or obj.function.arg1(arg2) ? > this function is part of a class. > > class myClass: > def function(arg1, arg2): > # do something > > Is it possible to do in python ? May be not directly but using some other > methods. No, that would be a different language, and if someone were to implement it this would make you a newbie in two languages -- hardly an improvement ;) -- https://mail.python.org/mailman/listinfo/python-list
Re: A newbie doubt on methods/functions calling
On Fri, 7 Oct 2016 04:06 am, [email protected] wrote: > Hi > > I just started learning python. Is there any way to call functions in > different way ? > > Rather calling obj.function(arg1, arg2) I would like to call like below > > "obj function arg1 arg2" No. This will be a syntax error. > this function is part of a class. > > class myClass: > def function(arg1, arg2): > # do something > > Is it possible to do in python ? May be not directly but using some other > methods. The only way to do this will be to write your own pre-processor, which will parse your source code, and translate it from your language to valid Python. That's a lot of work for very little value -- I recommend you just learn the Python syntax rather than trying to force it to be something it is not. -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: BeautifulSoup help !!
I guess I will have to extract data from multiple divs as only extracting data from the parent div which has other divs in it with the different data is coming up all messed up. Will play around and see if I could get through it. Let me clarify once again I dont need complete code , a resource where I could find more info about using Beautifulsoup will be appreciated. Also do I need some kind of plugin etc to extract data to csv ? or it is built in python and I could simply import csv and write other commands needed ?? -- https://mail.python.org/mailman/listinfo/python-list
Re: segfault using shutil.make_archive
On Thu, Oct 6, 2016, at 12:46, Tim wrote: > I need to zip up a directory that's about 400mb. > I'm using shutil.make_archive and I'm getting this response: > > Segmentation fault: 11 (core dumped) > > The code is straightforward (and works on other, smaller dirs): Are you able to make a test case that reproduces it reliably without any specifics about what data you are using (e.g. can you generate a directory full of blank [or /dev/urandom if the compressed size is a factor] files that causes this)? Is it a large number of files or a large total size of files? -- https://mail.python.org/mailman/listinfo/python-list
Re: segfault using shutil.make_archive
On Thu, Oct 6, 2016, at 13:45, Random832 wrote: > On Thu, Oct 6, 2016, at 12:46, Tim wrote: > > I need to zip up a directory that's about 400mb. > > I'm using shutil.make_archive and I'm getting this response: > > > > Segmentation fault: 11 (core dumped) > > > > The code is straightforward (and works on other, smaller dirs): > > Are you able to make a test case that reproduces it reliably without any > specifics about what data you are using (e.g. can you generate a > directory full of blank [or /dev/urandom if the compressed size is a > factor] files that causes this)? Is it a large number of files or a > large total size of files? Also consider passing a logging object to make_archive and see if it prints out anything interesting before the segfault. import shutil import logging import sys logger = logging.getLogger() logger.setLevel(logging.DEBUG) logger.addHandler(logging.StreamHandler(sys.stderr)) shutil.make_archive(..., logger=logger) -- https://mail.python.org/mailman/listinfo/python-list
Re: BeautifulSoup help !!
On 10/06/2016 11:34 AM, Navneet Siddhant wrote:
> I guess I will have to extract data from multiple divs as only
> extracting data from the parent div which has other divs in it with
> the different data is coming up all messed up. Will play around and
> see if I could get through it. Let me clarify once again I dont need
> complete code , a resource where I could find more info about using
> Beautifulsoup will be appreciated. Also do I need some kind of
> plugin etc to extract data to csv ? or it is built in python and I
> could simply import csv and write other commands needed ??
Writing CSV files from Python is relatively easy. Though you may need
to understand basic Python data types first, such as lists and dicts.
The module itself is mostly documented here:
https://docs.python.org/3/library/csv.html
And there are numerous examples of its use:
https://pymotw.com/2/csv/
https://www.getdatajoy.com/examples/python-data-analysis/read-and-write-a-csv-file-with-the-csv-module
To name but two of the first of many google search results.
Sounds to me like you need to spend some time learning basic Python data
types and how to iterate through them (lists and dicts mainly). BS4
uses both lists and dicts for nearly everything. An hour or two should
be enough to get a handle on this. The nice thing about Python is you
can build things and run them in an incremental, and mostly interactive
way. Regularly print out things so you can see what structure the data
has. BS4 is very good about string representations of all its
structures you can print them out without knowing anything about them.
For example, if you were searching for a tag:
results = soup.find('a', attrs = {'data-search-key': "name" })
you can just do:
print (results)
And easily see how things are nested. Then you can use that to drill
down using list indexing to get just the part you need.
I suspect if they hire you and you work more on Python you'll grow to
really like it.
I suppose that Alister expressed consternation because much of what you
ask can be solved with some good old fashioned searching of google and
is no different from learning any language, including C# which you
already know. Python is not C# of course, but the basic principles
behind programming are nearly universally-applied to nearly all
programming languages. For a programmer, it shouldn't be too hard to
move from language to language. Except for some of the more idiomatic
things about a language, many aspects are syntactically equivalent.
--
https://mail.python.org/mailman/listinfo/python-list
Re: User Interface Suggestions? (newbie)
On Wednesday, 5 October 2016 14:10:21 UTC+1, Beverly Howard wrote: > I'm new to Python, but have three decades of experience with FoxPro and VFP > plus I started programming in Basic and still comfortable with that. > > I have spent some time with Python and am now fairly familiar with the syntax > and functions, but building a user interface has me stopped. > > A primary question would be, "What are options for building a display that > would update displayed values without scrolling?" > > My first goal is to build a program that would interface with a Raspberry Pi > to control a heating process. > > Thanks in advance, > Beverly Howard I've had great results using glade and python3. Glade provides a graphical tool for designing GUI windows. Once the GUI is designed, glade writes out an XML file which can be read by either C or Python programs. Reading the XML from python3 requires that you import gi.repository. The python code after that is straightforward. (See http://python-gtk-3-tutorial.readthedocs.io/en/latest/builder.html) -- https://mail.python.org/mailman/listinfo/python-list
Re: segfault using shutil.make_archive
On Thursday, October 6, 2016 at 2:04:20 PM UTC-4, Random832 wrote: > On Thu, Oct 6, 2016, at 13:45, Random832 wrote: > > On Thu, Oct 6, 2016, at 12:46, Tim wrote: > > > I need to zip up a directory that's about 400mb. > > > I'm using shutil.make_archive and I'm getting this response: > > > > > > Segmentation fault: 11 (core dumped) > > > > > > The code is straightforward (and works on other, smaller dirs): > > > > Are you able to make a test case that reproduces it reliably without any > > specifics about what data you are using (e.g. can you generate a > > directory full of blank [or /dev/urandom if the compressed size is a > > factor] files that causes this)? Is it a large number of files or a > > large total size of files? > > Also consider passing a logging object to make_archive and see if it > prints out anything interesting before the segfault. > > import shutil > import logging > import sys > logger = logging.getLogger() > logger.setLevel(logging.DEBUG) > logger.addHandler(logging.StreamHandler(sys.stderr)) > shutil.make_archive(..., logger=logger) Interesting. I tried the above in an interactive python session and never saw the problem at all. So then I added the logger to my program code and watched it go. Using the program, I still get the segfault. It happens on the same file; I removed it and then it happens on the next file in the list. It looks like it (python process) has no more space, but the machine has plenty of disk space and available memory. I didn't think about the number of files, I'll look into that. For now, I'm just using subprocess and the zip command, which is working. thanks, --Tim -- https://mail.python.org/mailman/listinfo/python-list
Question on multiple Python users in one application
Hello. Please pardon a newbie question. I have a Windows multi-user program that can be used by an arbitrary number of users at once. They can work independently, or they can share data at the file or even variable level if they want. I want to give them the ability to write Python programs within this environment. So I intend to embed CPython access in the program. The basic embedding of CPython seems straight forward. But since I have multiple users, each needs their own Python sandbox, so if they all compile programs with variable 'spam', it doesn't collide. Of course they can all have different programs running at the same time too. I think I need to do some behind-the-scenes thread management and possibly namespace management to allow this to work correctly. But this is where I'm confused, and not sure where to start. I know how to tell my users apart. The program is multi-threaded, but there is not a direct correspondence between a user and a thread; a single user can switch threads, but I know when this happens. Can someone please suggest what I should be looking at and doing to be able to effectively run multiple independent Pythons in a single program? Thank you for your help! Loren -- https://mail.python.org/mailman/listinfo/python-list
Re: Question on multiple Python users in one application
"Jolly Good Spam" writes: > Can someone please suggest what I should be looking at and doing to be > able to effectively run multiple independent Pythons in a single > program? Put each Python in a separate process and communicate by IPC. -- https://mail.python.org/mailman/listinfo/python-list
Re: Question on multiple Python users in one application
On 06/10/16 22:11, Paul Rubin wrote: "Jolly Good Spam" writes: Can someone please suggest what I should be looking at and doing to be able to effectively run multiple independent Pythons in a single program? Put each Python in a separate process and communicate by IPC. Loren says that this is to be CPython embedded into an existing application (in which each user can be identified somehow). We need to understand first what the process/threading/per-user model of the existing application is. E. -- https://mail.python.org/mailman/listinfo/python-list
Re: Question on multiple Python users in one application
On Fri, Oct 7, 2016 at 7:59 AM, Jolly Good Spam wrote: > I have a Windows multi-user program that can be used by an arbitrary number > of users at once. They can work independently, or they can share data at the > file or even variable level if they want. I want to give them the ability to > write Python programs within this environment. So I intend to embed CPython > access in the program. Okay. Before you go one micron further, answer this critical question: *Do you trust your users?* Would you permit your users to have complete access to the computer that this program is running on? If they're all people in the same company, running something on the company's own server, you're fine. But if there's even the slightest chance that a malicious user will be on this system, you MUST NOT permit arbitrary code. CPython is *not* a secured environment. So, on to the specifics. > The basic embedding of CPython seems straight forward. But since I have > multiple users, each needs their own Python sandbox, so if they all compile > programs with variable 'spam', it doesn't collide. Of course they can all have > different programs running at the same time too. You want them to be able to share data, even at the level of a single variable. That strongly suggests using the same CPython embed for all your users. You can avoid name collisions simply by giving each user a module; one person's "spam" doesn't collide with another user's "spam" any more than math.log collides with logging.log. However, this is *not* protecting one user from another - it just protects against accidents. (I could easily reach into someone else's module by typing "fred.spam = 123".) So the question really becomes: How independent should the Pythons be? Sharing data is far easier if they're less isolated, but then it's harder to multithread. And above all, the security question. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Question on multiple Python users in one application
(Apologies for the broken threading on this reply, I'm just getting the list access set up.) Put each Python in a separate process and communicate by IPC. I thought of that, but the multi-user program is a virtual machine implementation, and the programs running on the machine have resources like variables, arrays, files, and databases that the users are likely to want to access from Python as native Python objects. It's unlikely or impossible that they would or could pass all objects as command line parameters to a Python program. While it is certianly possible to marshall every variable access through IPC, it isn't real efficient. I would much perfer to avoid this if I possibly can. Thanks, Loren -- https://mail.python.org/mailman/listinfo/python-list
Re: Question on multiple Python users in one application
"Loren Wilton" writes: > While it is certianly possible to marshall every variable access > through IPC, it isn't real efficient. I would much perfer to avoid > this if I possibly can. Maybe you could use Python proxy objects accessing a shared memory segment. Though as Chris Angelico mentions, you may have a security issue depending on what the users are up to. Can you say what the application is for? Does the user language have to be Python? What you're describing might be easier with Erlang. -- https://mail.python.org/mailman/listinfo/python-list
Re: Question on multiple Python users in one application
On 06/10/16 22:40, Loren Wilton wrote: the multi-user program is a virtual machine implementation That's not relevant (unless you mean each user is running in their own VM, in which case you _really_ need to describe your execution environment). BTW, you _really_ need to describe your execution environment ;) and the programs running on the machine have resources like variables, arrays, files, and databases Python variables? Python arrays (lists? tuples?)? Or some other sort of "variables" and "arrays" that exist outside of Python? You _really_ need to describe your execution environment ;) E ;) -- https://mail.python.org/mailman/listinfo/python-list
Re: Question on multiple Python users in one application
Okay. Before you go one micron further, answer this critical question: *Do you trust your users?* The basic answer is yes. This program amounts to a virtual machine implementation of a multi-processor and multi-user mainframe computer. It runs on a Windows box of its own. It has an IO subsystem of its own with a pretty hefty firewall machine in front of it before it gets to internet access. The Windows OS doesn't have network access at all. That may sound like a joke to you, but some of the biggest companies in the world have this machine running in their back rooms. Would you permit your users to have complete access to the computer that this program is running on? If they're all people in the same company, running something on the company's own server, you're fine. But if there's even the slightest chance that a malicious user will be on this system, you MUST NOT permit arbitrary code. CPython is *not* a secured environment. This is understodd and accepted. I should say at the moment this is at the feasibility and experimental stage. It would be possible to firewall this into a separate application to protect the VM data, but the machine users would still have access to the PC side thru CPython. It would be possible to go a step farther and put Python in a PC of its own connected thru Infiniband. We've done this with Java, but the results are painful. The goal at the moment is to see if I can make it appear that I have Python running more or less natively on the virtual machine. This means that Python will have direct access to user variables, files, and direct database access for the mainframe databases. Since the mainframe data format doesn't match that of the commmon Python objects, I'll either have to subclass the Python objects or make my own objects similar to the equivalent Python objects for things like numbers and strings. That is a project for tomorrow, not today. It seems straight-forward, just a lot of work. You want them to be able to share data, even at the level of a single variable. Yes. That strongly suggests using the same CPython embed for all your users. That was the conclusion I'd initially come to, yes. You can avoid name collisions simply by giving each user a module; one person's "spam" doesn't collide with another user's "spam" any more than math.log collides with logging.log. However, this is *not* protecting one user from another - it just protects against accidents. (I could easily reach into someone else's module by typing "fred.spam = 123".) This particular mainframe architecture is odd, by today's machine standards. It is a stack machine. The stack is not a simple linear stack as far as addressing is concerned, but more of a tree structure. All user programs have stacks that are rooted at D0 in the operating system, at D1 in their own code file, and at higher levels in various nested program procedures. A new process can be created by forking at any level, and the new process shares all data with the original program from that level outward. It is possible for one stack to export procedure entry points that can be called by other stacks. Variables, arrays, files, etc, can be passed fron one stack (process) to another stack (process). So the question really becomes: How independent should the Pythons be? Sharing data is far easier if they're less isolated, but then it's harder to multithread. The desirable goal is to make it look like Python is a native language running on this machine, but also giving access to existing Python libraries running on the PC that contains this virtual machine. Effectively this is extending the boundaries of the virtual machine to include processor chips of a different architecture within the scope of the VM. This has security implications, but at the moment they are (at least partially) understood and accepted. And above all, the security question. See last sentence above. Thanks! Loren -- https://mail.python.org/mailman/listinfo/python-list
Re: Question on multiple Python users in one application
We need to understand first what the process/threading/per-user model of the existing application is. The program is a virtual machine for an old mainframe architecture. You could think of a VM running a Linux implementaiton as a kind of crude mental reference model. The emulated mainframe is a multi-processor machine. We use separate threads to represent each physical processor, and those threads are assigned to separate real Intel processors underneath the program, in Windows. A given user program can have multiple threads. Each thread can be running or waiting for something. If the user thread is running it will be on one of the virtual processors, so will be on some one of the processor threads in the application. The user thread can be interrupted by an IO completion interrupt or timer interrupt or give up control in some way, and be kicked off the virtual processor. When next run it might be on some other virtual processor, and thus on some other thread in the VM application. Again, this is just like a normal user program running in say Linux. So the VM application has many "processor" threads, but a user does not correspond (except transiently) to any given processor thread. The VM also has a number of other threads for various purposes. It would be possible to make a dedicated thread for each user that started a Python program. This could get messy to manage, and there is always the possibility of runnnig out of threads, but it might be a simple way to keep things separated. I think that might allpw Python to use thread local storage to some advantage, if I've understood the docs thatI've read correctly. Loren -- https://mail.python.org/mailman/listinfo/python-list
Re: Question on multiple Python users in one application
the multi-user program is a virtual machine
implementation
and the programs running on the machine have resources
like variables, arrays, files, and databases
Python variables? Python arrays (lists? tuples?)? Or some other sort of
"variables" and "arrays" that exist outside of Python?
You _really_ need to describe your execution environment ;)
http://bitsavers.informatik.uni-stuttgart.de/pdf/burroughs/B6500_6700/1035441_B6500_B7500_Stack_Mechanism_1968.pdf
The linked paper By Erv Hauck and Ben Dent describes what our ancestor
machine looked like in 1968, and is about the most recent documentation
commonly available on the web. Its been a few years since then, and the
machine has changed a little over time, but the basics of the word format
and stack mechanism and program structure are basically unchanged.
Since the machine is no longer represented by hardware, but instead by an
emulator running on Windows, we have some additional capabilities. For
instance, instead of writing a Python interpreter that will run inside the
virtual machine structure (effectively being a normal B6500 program, in
terms of the referenced document) I can use a Python interpreter (CPython)
that is an existing DLL running in the WIndows environment, and do a
procedure call from my B6500 environment into the Python environment.
Effectively this is a call from one stack-based procedure using one bytecode
to another stack-based procedure using a different bytecode. The interpreter
for the B6500 bytecode will realize what it is calling, fabricate an
appropriate call into CPython (possibly switching to a different thread if
that is necessary) and then let the Python code do its thing. Eventually the
procedure call will (probably) return, and the B6500 code will resume
execution. Basically there is a fence where we swich between interpreters on
a procedure call or return boundary.
The Python code is running as (I hope) a native Windows DLL, so should be
able to access any existing Python libraries that exist on the WIndows
machine. Obviously this Python code will be using Windows-shaped data
objects like integers, floats, and strings.
The B6500 code also has integers, floats, doubles, and strings, but they all
look different from what Windows data looks like. Since Python accesses
things as objects and objects have accessor methods, I am reasonably sure I
can either subclass existing Python objects to access B6500 data, or simply
make new objects to access that data. Thus the Python code, even though it
is running in its own interpreter, will be able to access resources within
the virtual machine environment.
Does that help? I can expound more if needed, but I don't want to bloat the
newsgroup with irrelevent stuff.
I don't think my main concern here is being able to call the CPython
interpreter routines, but instead it is to be able to provide separate
sandboxes for the various programs ("stacks", in B6500 terminology) that
might have their own Python sessions or programs.
One possible scenario is a B6500 user, sitting at a terminal, and typing
"run python". That should get him a copyright notice and a >>> prompt, and
he should be able to carry on just as though he was sitting at a Windows
command line. The guy at the terminal in the next office should be able to
be doing the same thing at the same time.
Loren
--
https://mail.python.org/mailman/listinfo/python-list
Re: Question on multiple Python users in one application
"Loren Wilton" writes:
> I don't think my main concern here is being able to call the CPython
> interpreter routines, but instead it is to be able to provide separate
> sandboxes for the various programs ("stacks", in B6500 terminology)
> that might have their own Python sessions or programs.
This is much easier to do with Lua than CPython. I haven't looked at
Micropython much yet.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Question on multiple Python users in one application
"Loren Wilton" writes:
I don't think my main concern here is being able to call the CPython
interpreter routines, but instead it is to be able to provide separate
sandboxes for the various programs ("stacks", in B6500 terminology)
that might have their own Python sessions or programs.
This is much easier to do with Lua than CPython. I haven't looked at
Micropython much yet.
Unfortunately the requirement is to be able to run Python because of its
popularity and the size of the existing code base of available libraries.
Other languages would be interesting at some point, but only if there is
some real business use for the language. Unless banks and clearing houses
are commonly using Lua, it might be a hard sell. They are using Python.
Loren
--
https://mail.python.org/mailman/listinfo/python-list
Re: Question on multiple Python users in one application
On Fri, Oct 7, 2016 at 9:09 AM, Loren Wilton wrote: >> Okay. Before you go one micron further, answer this critical question: >> >> *Do you trust your users?* > > > The basic answer is yes. This program amounts to a virtual machine > implementation of a multi-processor and multi-user mainframe computer. It > runs on a Windows box of its own. It has an IO subsystem of its own with a > pretty hefty firewall machine in front of it before it gets to internet > access. The Windows OS doesn't have network access at all. That may sound > like a joke to you, but some of the biggest companies in the world have this > machine running in their back rooms. Good. And no, that's most definitely not a joke; having a VM with no network access is a very smart way to manage security. So you don't have to worry about someone deliberately trying to mess you around. This saves a *TON* of trouble. > The goal at the moment is to see if I can make it appear that I have Python > running more or less natively on the virtual machine. This means that Python > will have direct access to user variables, files, and direct database access > for the mainframe databases. Since the mainframe data format doesn't match > that of the commmon Python objects, I'll either have to subclass the Python > objects or make my own objects similar to the equivalent Python objects for > things like numbers and strings. That is a project for tomorrow, not today. > It seems straight-forward, just a lot of work. That's the inherent work of trying to make one thing talk to another. Most likely, you can ignore most of the differences, and just make sure your script writers are aware, for instance, that integer wrap-around doesn't exist in Python. Unless someone's been deliberately exploiting those kinds of features, that's unlikely to cause problems. >> You can avoid name collisions simply by giving each user a >> module; one person's "spam" doesn't collide with another user's "spam" >> any more than math.log collides with logging.log. However, this is >> *not* protecting one user from another - it just protects against >> accidents. (I could easily reach into someone else's module by typing >> "fred.spam = 123".) > > > This particular mainframe architecture is odd, by today's machine standards. > It is a stack machine. The stack is not a simple linear stack as far as > addressing is concerned, but more of a tree structure. All user programs > have stacks that are rooted at D0 in the operating system, at D1 in their > own code file, and at higher levels in various nested program procedures. A > new process can be created by forking at any level, and the new process > shares all data with the original program from that level outward. It is > possible for one stack to export procedure entry points that can be called > by other stacks. Variables, arrays, files, etc, can be passed fron one stack > (process) to another stack (process). > >> So the question really becomes: How independent should the Pythons be? >> Sharing data is far easier if they're less isolated, but then it's >> harder to multithread. > > > The desirable goal is to make it look like Python is a native language > running on this machine, but also giving access to existing Python libraries > running on the PC that contains this virtual machine. Effectively this is > extending the boundaries of the virtual machine to include processor chips > of a different architecture within the scope of the VM. This has security > implications, but at the moment they are (at least partially) understood and > accepted. Hmm. Okay, so you have some pretty fundamental architectural differences to deal with. I think what you're doing is going to be possible, but either (a) this is a large and ongoing job, or (b) Python won't feel truly native - it'll be a bridging system, and people will need to understand both ends. Sounds like it'd be a good fun project, though. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Question on multiple Python users in one application
On 10/06/2016 04:47 PM, Loren Wilton wrote:
> The Python code is running as (I hope) a native Windows DLL, so should be
> able to access any existing Python libraries that exist on the WIndows
> machine. Obviously this Python code will be using Windows-shaped data
> objects like integers, floats, and strings.
>
> The B6500 code also has integers, floats, doubles, and strings, but they all
> look different from what Windows data looks like. Since Python accesses
> things as objects and objects have accessor methods, I am reasonably sure I
> can either subclass existing Python objects to access B6500 data, or simply
> make new objects to access that data. Thus the Python code, even though it
> is running in its own interpreter, will be able to access resources within
> the virtual machine environment.
So I take it that currently users access the software running in the
virtual mainframe over telnet or some form of serial link and that they
interact with it in a text terminal? This point is fairly important,
because if it's true, then you really don't have any in-band way of
clients talking to some Python process(es) running on the machine
hosting the virtual mainframe.
>
> Does that help? I can expound more if needed, but I don't want to bloat the
> newsgroup with irrelevent stuff.
It's a start. This is fairly interesting. I love the welding of old and
new technologies in innovative ways. Please continue.
> I don't think my main concern here is being able to call the CPython
> interpreter routines, but instead it is to be able to provide separate
> sandboxes for the various programs ("stacks", in B6500 terminology) that
> might have their own Python sessions or programs.
You'll just have to implement Python support for these frames of
reference. Python is not going to be running on the virtual hardware;
it's running on Windows. All you need to do is provide a way for the
remote end users to talk to the Python server running on the windows
machine would define sessions for the users and manage the interaction
with the virtual mainframe. No matter how you cut it, Python is
separate from the users and separate from the mainframe since you can't
run Python on the mainframe itself. The only thing that makes sense is
to develop some kind of Python-based server architecture that clients
can talk to to do things via Python on behalf of the user.
At least that's the way I see it pending an answer to the question of
how users interact with this mainframe and its programs.
--
https://mail.python.org/mailman/listinfo/python-list
Re: A newbie doubt on methods/functions calling
On 2016-10-06, Steve D'Aprano wrote: > The only way to do this will be to write your own pre-processor, which will > parse your source code, and translate it from your language to valid > Python. That's a lot of work for very little value -- I recommend you just > learn the Python syntax rather than trying to force it to be something it > is not. [Cue the decades-old story about the elaborate set of C macros that I once saw somebody using so he could write a C program that looked like some flavor of structured BASIC.] -- Grant Edwards grant.b.edwardsYow! Don't SANFORIZE me!! at gmail.com -- https://mail.python.org/mailman/listinfo/python-list
Re: Question on multiple Python users in one application
On Fri, Oct 7, 2016 at 9:47 AM, Loren Wilton wrote:
> I don't think my main concern here is being able to call the CPython
> interpreter routines, but instead it is to be able to provide separate
> sandboxes for the various programs ("stacks", in B6500 terminology) that
> might have their own Python sessions or programs.
>
> One possible scenario is a B6500 user, sitting at a terminal, and typing
> "run python". That should get him a copyright notice and a >>> prompt, and
> he should be able to carry on just as though he was sitting at a Windows
> command line. The guy at the terminal in the next office should be able to
> be doing the same thing at the same time.
Ah, that probably means you want separate interpreters, then. My
previous idea of just having separate modules wouldn't isolate well
enough to feel properly comfortable (for instance, the interactive
interpreter uses "_" in the builtins to store your last result, and if
you have multiple users sharing the builtins, they'd be trampling over
each other's underscores). But if you accept that this is a shared
environment, with the consequences thereof, you could have easy and
convenient sharing - and might even be able to have a tiny hack around
the interactive interpreter to make it use a different module name,
instead of __main__. So you could do something like this:
User "Fred"
run python
Python 3.6, yada yada
Type "help" yada yada
>>> spam = 1234
User "Joe"
run python
Python 3.6, yada yada
Type "help" yada yada
>>> spam = 4321
>>> import fred
>>> print(fred.spam)
1234
>>> print(spam)
4321
>>> fred.ham = 2
User "Fred"
>>> print(spam)
1234
>>> print(ham)
2
So they would be *visibly* sharing state. This might be acceptable to
you and your users. It would be pretty easy to code up (with the
possible exception of the interactive mode).
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: A newbie doubt on methods/functions calling
[Cue the decades-old story about the elaborate set of C macros that I once saw somebody using so he could write a C program that looked like some flavor of structured BASIC.] I once wrote a set pf C defines so that I could compile Pascal with a C compiler without having to change the Pascal source code. Fortunately that was about 40 years ago, and I've long since lost the source for those macros. The goal here isn't to make one thing look like another that it isn't. The goal is to "write a Python environment for a mainframe", ideally without having to do all of the writing to do it from scratch. Lots of people seem to have written Python interpeters in various languages. I guess I could sit down and write one in Algol or NEWP and have a native Python environment. But what good would it do me? This machine isn't something at a university where the first-year hackers write their 5-line programs with no IO. Everything I read says the strength of Python is that there are many existing 3rd party libraries that do lots of useful things. Since a lot of them are distributed as binaries, they would not work in this mainframe environment. So I don't want to WRITE a Python interpreter for the actual mainframe environment. I want to use an interpreter for an existing environment (Windows) where there are already a lot of existing libraries. But since a lot of the data to be analyzed is on the mainframe environment, and the results would be wanted there too, I need to extend the Python data access to the mainframe environment. Loren -- https://mail.python.org/mailman/listinfo/python-list
Re: A newbie doubt on methods/functions calling
On 06/10/2016 18:06, [email protected] wrote: Hi I just started learning python. Is there any way to call functions in different way ? Rather calling obj.function(arg1, arg2) I would like to call like below "obj function arg1 arg2" As has been pointed out, it's difficult to tell whether a space should mean a "." or "(". Or, as is also possible, "=" or "+" or any other kind of operator. But even sticking only with "." and "(", then something like: a b c d e f g could be interpreted in a myriad different ways. Python requires that this is determined at 'compile' time, but information about what each name is (class, instance, function, attribute etc), which might help resolve the ambiguity, isn't available until run-time. this function is part of a class. class myClass: def function(arg1, arg2): # do something Is it possible to do in python ? May be not directly but using some other methods. It would be hard without extra clues, or extra restrictions on what is possible. But even if a translator could figure out what is what, a human reader will have trouble. What's the problem with typing "." and "(" anyway; keyboard problems? (If so then keep away from languages such as C and C++ because they have a /lot/ more punctuation!) Or used to a language that doesn't require (,) around function arguments? -- Bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: Question on multiple Python users in one application
So I take it that currently users access the software running in the
virtual mainframe over telnet or some form of serial link and that they
interact with it in a text terminal? This point is fairly important,
because if it's true, then you really don't have any in-band way of
clients talking to some Python process(es) running on the machine
hosting the virtual mainframe.
Um... Yes and no.
Consider a Linux web server environment that happens to have Python
installed.
A programmer in the organization that ownns the server can probably log on
with an SSH session and type Python and then have an interactive Python
session where does simple computations or database accesses, or where he
runs existing Python scripts.
But that isn't how the machine will normally run.
Normally the web pages might have queries or other items that run bits of
Python code, or Python scripts, some of which might access or even update a
database. Normally there might be nightly cron jobs, some of which might be
Pyton scripts.
then you really don't have any in-band way of
clients talking to some Python process(es) running on the machine
hosting the virtual mainframe.
I'm not clear on what you mean by "in band way". If you mean that there is
something on the Windows side of the machine that might wake up and decide
it wants to talk to Python, then no, that can't happen.
But mainframes these days are generally not interactive in the sense of a
Telent terminal. The programming machines have something like that, but the
production machines usualy don't. There may be terminals, but they are
probably automatic teller machines, or airline reservation terminals. These
don't interact with a shell in the Unix sense, they interact with a series
of predefined transaction commands that control program access, or maybe are
accessed by a single dedicated program. Even web pages, if well written, can
only access fixed assets.
I don't think my main concern here is being able to call the CPython
interpreter routines, but instead it is to be able to provide separate
sandboxes for the various programs ("stacks", in B6500 terminology) that
might have their own Python sessions or programs.
You'll just have to implement Python support for these frames of
reference. Python is not going to be running on the virtual hardware;
it's running on Windows. All you need to do is provide a way for the
remote end users to talk to the Python server running on the windows
machine would define sessions for the users and manage the interaction
with the virtual mainframe. No matter how you cut it, Python is
separate from the users and separate from the mainframe since you can't
run Python on the mainframe itself. The only thing that makes sense is
to develop some kind of Python-based server architecture that clients
can talk to to do things via Python on behalf of the user.
If I need to write these sort of hooks, I'm new enough to Python to not have
much of a clue where to start, and would appreciate all the help I can get.
I've read that Python supports 'threads', and I'd assumed (maybe
incorrectly) that these were somewhat separate environments that could be
operating concurrently (modulo the GC lock). I assume that data can be
shared between the threads, and probably will share automatically if the
name of the item (or the reference) matches. But I'd also assumed that I
might be able to do something like start a thread and tell it that it is
working in "module user_42" or some such, so that by default it's data would
not be global.
Assuming that I'm not completely off-base about using threads, I don't
really understand how to create a thread or destroy it. I'm also not
completely clear on how much useful data there is that is thread local, and
how much in the way of Python objects goes in the gloabl heap.
Would using threads help me do what I want? Or am I completely off base? Or
is there some other approach that I've missed that might be better?
Loren
--
https://mail.python.org/mailman/listinfo/python-list
Re: Question on multiple Python users in one application
On 10/06/2016 06:03 PM, Loren Wilton wrote: >> So I take it that currently users access the software running in the >> virtual mainframe over telnet or some form of serial link and that they >> interact with it in a text terminal? This point is fairly important, >> because if it's true, then you really don't have any in-band way of >> clients talking to some Python process(es) running on the machine >> hosting the virtual mainframe. > > Um... Yes and no. > > Consider a Linux web server environment that happens to have Python > installed. > > A programmer in the organization that ownns the server can probably log on > with an SSH session and type Python and then have an interactive Python > session where does simple computations or database accesses, or where he > runs existing Python scripts. > > But that isn't how the machine will normally run. Please let us know how the machine normally runs and how your clients interact with it. We're still very much working in the dark here since we can't even determine how users or clients interact with the mainframe and the kinds of general tasks they do. You mention a web interface. How do you envision your clients invoking Python? > Normally the web pages might have queries or other items that run bits of > Python code, or Python scripts, some of which might access or even update a > database. Normally there might be nightly cron jobs, some of which might be > Pyton scripts. Well that last bit is comparatively easier and doesn't require any threading or multi-user capability in python itself. The script runs, talks to the mainframe over some protocol you've apparently established, retrieves some data and performs some sort of command. > >> then you really don't have any in-band way of >> clients talking to some Python process(es) running on the machine >> hosting the virtual mainframe. > > I'm not clear on what you mean by "in band way". If you mean that there is > something on the Windows side of the machine that might wake up and decide > it wants to talk to Python, then no, that can't happen. I mean that if you client is talking to the mainframe over some kind of protocol already, then you can't really add Python to the mix directly since it does not run on the mainframe. But like I said, we don't know how your clients are using the mainframe and its program, so I can't say what the "band" actually is precisely. But in other terms of reference, if you have a web server that lets a client do something with the mainframe, then using the browser would be considered "in band" since command and control functions through the same interface. If you wanted to add some sort of python console, you'd have to either add something to the protocol to allow this interaction, or do it via some other out-of-band way, such as ssh-ing and running a script. > But mainframes these days are generally not interactive in the sense of a > Telent terminal. The programming machines have something like that, but the > production machines usualy don't. There may be terminals, but they are > probably automatic teller machines, or airline reservation terminals. These > don't interact with a shell in the Unix sense, they interact with a series > of predefined transaction commands that control program access, or maybe are > accessed by a single dedicated program. Even web pages, if well written, can > only access fixed assets. How do the users interact with these predefined transaction commands? There must be some protocol you've implemented to allow this over a remote connection. >> You'll just have to implement Python support for these frames of >> reference. Python is not going to be running on the virtual hardware; >> it's running on Windows. All you need to do is provide a way for the >> remote end users to talk to the Python server running on the windows >> machine would define sessions for the users and manage the interaction >> with the virtual mainframe. No matter how you cut it, Python is >> separate from the users and separate from the mainframe since you can't >> run Python on the mainframe itself. The only thing that makes sense is >> to develop some kind of Python-based server architecture that clients >> can talk to to do things via Python on behalf of the user. > > If I need to write these sort of hooks, I'm new enough to Python to not have > much of a clue where to start, and would appreciate all the help I can get. Well this is an advanced Python topic for sure. But it's one you'll have to delve into eventually it sounds like. Google for information on embedding Python. > I've read that Python supports 'threads', and I'd assumed (maybe > incorrectly) that these were somewhat separate environments that could be > operating concurrently (modulo the GC lock). I assume that data can be > shared between the threads, and probably will share automatically if the > name of the item (or the reference) matches. But I'd also assumed that
Re: Question on multiple Python users in one application
Ah, that probably means you want separate interpreters, then. I hope not, unless I can get separate simultaneous interpreters out of one CPython.dll in one Windows process space. I'm guessing that I can't, but since I really don't know what I'm doing with Python yet, maybe I'm wrong there. (for instance, the interactive interpreter uses "_" in the builtins to store your last result, and if you have multiple users sharing the builtins, they'd be trampling over each other's underscores). I'm very new to Python and I suspect I don't completely understand what you mean by 'builtins' in this instance. Does Python have a non-module-scope namespace that contains things like _ ? Could you give me a pointer to any docs on this? If it does, do you have an opinion on how hard it might be to wrap it in some form of module (or "super module"?) scope? But if you accept that this is a shared environment, with the consequences thereof, you could have easy and convenient sharing - and might even be able to have a tiny hack around the interactive interpreter to make it use a different module name, instead of __main__. So you could do something like this: User "Fred" etc. So they would be *visibly* sharing state. This might be acceptable to you and your users. It would be pretty easy to code up (with the possible exception of the interactive mode). What you show in example is just about exactly what I want. Each user by default will have his own auto-generated (or maybe overridable by the user) namespace, so normally can work independently. But given that he can get the namespace handle for some other user (which might not be an interactive user, it might be a background process) then they can share data. The worry I have is any implicitly global variables, like underscore, and whatever else may exist in this global namespace. I'd really want some way to wrap those and make them unique for each user. I'm guessing that may mean hacking the code for CPython. I'd rather not do that, unless people would like the code back so I don't have my own unique branch that would break on each new release. But if that is what is needed, it is certainly something I could do. One concern I have is if two users both "import widget". Are they now sharing the widget namespace? I suspect they are, and that is probably undesirable. Or does Python have a hierarchical namespace concept, so that we would have fred.widget.ham and joe.widget.ham after fred and joe both import widget? Thanks again! Loren -- https://mail.python.org/mailman/listinfo/python-list
Re: A newbie doubt on methods/functions calling
Oops, apologies for replying to the wrong thread! Loren -- https://mail.python.org/mailman/listinfo/python-list
Re: Question on multiple Python users in one application
On Fri, Oct 7, 2016 at 11:03 AM, Loren Wilton wrote: > I've read that Python supports 'threads', and I'd assumed (maybe > incorrectly) that these were somewhat separate environments that could be > operating concurrently (modulo the GC lock). I assume that data can be > shared between the threads, and probably will share automatically if the > name of the item (or the reference) matches. But I'd also assumed that I > might be able to do something like start a thread and tell it that it is > working in "module user_42" or some such, so that by default it's data would > not be global. > > Assuming that I'm not completely off-base about using threads, I don't > really understand how to create a thread or destroy it. I'm also not > completely clear on how much useful data there is that is thread local, and > how much in the way of Python objects goes in the gloabl heap. > Threads don't really have much to do with namespacing and data sharing. The two are broadly orthogonal. Basically, threads in Python exist so that one function call can be running code while another one is blocked (eg on I/O). Other than that, they're just two "current stack location" markers, but they use the same namespaces, the same globals, etc. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Question on multiple Python users in one application
No idea as it's still not clear what you want to accomplish exactly.
You are providing a lot of details, just not concrete ones that show how
things are currently done and how you want to change that by adding
Python to the mix.
Hum, this could take a lot of pages of text to describe how the existing
environment works in detail, and some of it is considered proprietary so I
couldn't say anyway. But I can try to give you some general concepts. Then
once I have a description of the existing environment, I can say how I would
like to extend it to handle Python. Also see my reply a few moments ago to
Chris, where he siggested how to users might interact if they were sitting
at terminals.
Viruses are common today on commodity machines and operating systems.
Mainframes in general avoid this by obscurity. Most people don't believe
that mainframes still exist (this is what they have been taught in school)
so they don't think of hacking them. Also, most people have probably never
heard of a mainframe other than an IBM 370 with AIX, so have no idea what
the environment looks like, so they can't hack it. Just to make life harder
for the hackers, mainframes these days usually don't have anything
resembling interactive shell access that you can physically reach without a
badge to open a locked door or two. They typically don't access the internet
directly. They are a backend database machine with a frontend web server and
a firewall between it and the internet.
The general way that banking and airline networks work is that they are
private networks. They may be phone lines carrying RS-232 or similar
traffic. They may be some sort of packet network that may or may not be IP
based, and quite possibly does not use public network wires at all. When it
comes to how terminals access the mainframes, there is a routing program on
the mainframe that knows the name and address of every possible terminal
that can access the system. A transaction is usually something like a 6 or 8
character uppercase word followed by positional character data. The routing
program gets this message, looks up the terminal name, makes sure it is
allwed to do this trnasaction, looks up th eprogram that will handle the
transaction, and sends the termianl name and input data to that program. The
program does whatever it needs to do, and sends the terminal name and a
screenload of data back to the routing program. The routing program forwards
the response data to the terminal.
Web sites like Orbitz have web forms that make search or transaction
requests. The web server takes the information from the web page, transforms
it into one of these six character (or so) transaction requests, and
pretends it is a terminal, and sends the 'screen data' to the mainframe. The
mainframe processes the request from the "terminal" and sends back a
screenload of data. The reply may be more than one screen, so there may be
more transactions back and forth to get all the data. Then the CGI program
running on the web server formats this into the web page response.
Some mainframes may also be web servers. In this case the web server program
on the mainframe will interact using the same text commands with the backend
programs that handle the terminals, in many cases.
Mainframes can have interactive terminals, possibly thousands of them. This
is rare these days except for the programming machines. But the transaction
termainls could be used as interactive terminals, by running a transaction
that connects them to a PYTHON application. So there could be many users
(thousands of them) all doing independent things in Python at once. There
could be web page transaction servers that are talking to back-end programs
that invoke Python commands or scripts. In fact the web page handler could
appear to be a Python interpreter running a different script for each web
page request. There are also nightly, weekly, monthly, year-end, and ad-hoc
reports that are run on mainframes, typically to gether from the databases.
Any or all of these reports could be Python scripts, or could be programs
that somehow invoked snippits of Python code.
Ok, that is a user-level operational view of a typical mainframe. Now for
some lower level details of the particular mainframe of interest.
Our mainframe is emulated in a single Windows program. Any given instance
can have from 1 to 32 processors, and up to about 100 GW of RAM, and tens of
thousands of disk drives online. It has an operating system of its own that
can run tens of thousands of independent processoes ("stacks") at the same
time. The OS does all of the sort of scheduling, IO, and communications
things that an OS normally does. There are hundreds of system libraries
available that programs can use to perform common functions.
User programs don't have separate address spaces in the sense that you
understand. Memory isn't paged. Effectively memory consists of objects that
h
Re: Question on multiple Python users in one application
On Fri, Oct 7, 2016 at 11:22 AM, Loren Wilton wrote: >> Ah, that probably means you want separate interpreters, then. > > > I hope not, unless I can get separate simultaneous interpreters out of one > CPython.dll in one Windows process space. I'm guessing that I can't, but > since I really don't know what I'm doing with Python yet, maybe I'm wrong > there. It is possible to do that, but based on the info you give in this post, I withdraw the recommendation for separate interpreters. >> (for instance, the interactive >> interpreter uses "_" in the builtins to store your last result, and if >> you have multiple users sharing the builtins, they'd be trampling over >> each other's underscores). > > > I'm very new to Python and I suspect I don't completely understand what you > mean by 'builtins' in this instance. Does Python have a non-module-scope > namespace that contains things like _ ? Could you give me a pointer to any > docs on this? In Python, there are three [1] namespaces: function locals, module globals, and built-ins. Function locals are exactly what you'd expect: one invocation of a function has one set of locals, and another invocation (incl recursion) has a completely separate set. Module-level names are called "globals", but they're not process-wide like in C; code in a different module has separate globals. Built-ins are for "standard" names, like range(), str(), len(), and so on. They are shared among all modules in a process. [1] There's also the class namespace, used while you're building a class. Amongst our namespaces are such diverse elements as... > If it does, do you have an opinion on how hard it might be to wrap it in > some form of module (or "super module"?) scope? I wouldn't. It's generally immutable. What I'd advise is having some way to terminate Python and restart it (which will take care of any serious messes), and for the rest, just leave it shared. >> But if you accept that this is a shared >> environment, with the consequences thereof, you could have easy and >> convenient sharing - and might even be able to have a tiny hack around >> the interactive interpreter to make it use a different module name, >> instead of __main__. So you could do something like this: >> >> User "Fred" >>etc. >> >> So they would be *visibly* sharing state. This might be acceptable to >> you and your users. It would be pretty easy to code up (with the >> possible exception of the interactive mode). > > > What you show in example is just about exactly what I want. Each user by > default will have his own auto-generated (or maybe overridable by the user) > namespace, so normally can work independently. But given that he can get the > namespace handle for some other user (which might not be an interactive > user, it might be a background process) then they can share data. That would align very nicely with modules in Python. > The worry I have is any implicitly global variables, like underscore, and > whatever else may exist in this global namespace. I'd really want some way > to wrap those and make them unique for each user. I'm guessing that may mean > hacking the code for CPython. I'd rather not do that, unless people would > like the code back so I don't have my own unique branch that would break on > each new release. But if that is what is needed, it is certainly something I > could do. It's probably only the underscore, and that's only for interactive work. Seems to me it's not going to be too big a deal. > One concern I have is if two users both "import widget". Are they now > sharing the widget namespace? I suspect they are, and that is probably > undesirable. Or does Python have a hierarchical namespace concept, so that > we would have fred.widget.ham and joe.widget.ham after fred and joe both > import widget? Yes, they would. An import would be used in two ways: 1) Standard library modules, or third-party extension modules. It's 99.999% likely that sharing these won't be a problem. 2) Accessing another user's namespace in order to share data. Sharing these modules is the very mechanic of data sharing. Your users would have to be aware that they're sharing state. Off the top of my head, I can think of very few consequences of this, most of them having workarounds (for instance, they'd share PRNG state, but you can explicitly construct an RNG and have your own dedicated state; also, the decimal module has shared contexts, so you'd have to be careful if you ever want to change its parameters). The only real requirement would be a Python that doesn't do the usual thing with __main__, but instead has a special module name. This would be easy to do for script running, but might require a little more work for interactive mode. Worst case, you roll your own REPL using exec(). It wouldn't be too hard. I think this is doable. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Question on multiple Python users in one application
On Fri, Oct 7, 2016 at 12:21 PM, Loren Wilton wrote:
> The VM is a program running on Windows. The mainframe object code is just
> byte code. The VM has routines to handle every operator and each kind of
> Descriptor that they might validly use. One form of Descriptor is only used
> by the CALL instruction. It allows an "internal call" to a procedure that is
> part of the VM. Assuming that the VM linked to CPython, I could use one
> form of this to pass a text line to PyRun_SimpleString. Then I cna write a
> trivial program that will read from a terminal file and pass any received
> text to SimpleString, and somehow route any response back to the terminal
> file.
The source code for SimpleString looks like this:
int
PyRun_SimpleStringFlags(const char *command, PyCompilerFlags *flags)
{
PyObject *m, *d, *v;
m = PyImport_AddModule("__main__");
if (m == NULL)
return -1;
d = PyModule_GetDict(m);
v = PyRun_StringFlags(command, Py_file_input, d, d, flags);
if (v == NULL) {
PyErr_Print();
return -1;
}
Py_DECREF(v);
return 0;
}
If that had been your original plan, it's dead simple to enhance it to
use per-user module names. Just do this same work, but substitute a
different module name right at the beginning! Other
extremely-high-level interface functions are similar. You should have
no trouble making this embed work; and then it's just a matter of
figuring out the bridge code between that and the rest of the VM.
(Okay, so it's probably not going to be as easy as that made it sound,
but still. Definitely plausible.)
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Question on multiple Python users in one application
"Loren Wilton" writes: > I've read that Python supports 'threads', and I'd assumed (maybe > incorrectly) that these were somewhat separate environments that could > be operating concurrently (modulo the GC lock). I assume that data can > be shared between the threads, Threads all run in the same address space. Data "can be" shared is an understatement--it IS shared, so threads can interfere with each other very easily. You have to be quite careful to prevent that, and thread programming has a reputation for being difficult for that reason. > Would using threads help me do what I want? I don't think Python threads are the answer. You want a separate interpreter per user, which is annoying to do with CPython. Do you have concrete performance expectations about the sharing of data between interpreter sessions? Those old mainframes were very slow compared to today's machines, even counting Python's interpretation overhead. -- https://mail.python.org/mailman/listinfo/python-list
Re: A newbie doubt on methods/functions calling
"Loren Wilton" writes: > strength of Python is that there are many existing 3rd party libraries > that do lots of useful things. Since a lot of them are distributed as > binaries, they would not work in this mainframe environment. Python libraries are usually available as source, either in Python or in C. -- https://mail.python.org/mailman/listinfo/python-list
Re: Question on multiple Python users in one application
On Fri, Oct 7, 2016 at 12:52 PM, Paul Rubin wrote: > "Loren Wilton" writes: >> I've read that Python supports 'threads', and I'd assumed (maybe >> incorrectly) that these were somewhat separate environments that could >> be operating concurrently (modulo the GC lock). I assume that data can >> be shared between the threads, > > Threads all run in the same address space. Data "can be" shared is an > understatement--it IS shared, so threads can interfere with each other > very easily. You have to be quite careful to prevent that, and thread > programming has a reputation for being difficult for that reason. To be fair, the sharing of data between threads is no different from other concerns about global state. The only thing threads change is that you can have multiple functions doing stuff at once. state = [1,2,3] def func(): state[1] += 5 # do work state[1] -= 5 This function would probably work in most single-threaded environments. (Proper use of try/finally would make it safer.) This "patch, operate, unpatch" model becomes dangerous in a threaded system... but it also becomes dangerous in any case of recursion. Or just calling a function from inside func() that cares about state. Or anything like that. Mutable globals must be strictly managed, Alice; unproductive ones should be *eliminated*. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Question on multiple Python users in one application
On 10/06/2016 07:48 PM, Chris Angelico wrote: > If that had been your original plan, it's dead simple to enhance it to > use per-user module names. Just do this same work, but substitute a > different module name right at the beginning! Other > extremely-high-level interface functions are similar. You should have > no trouble making this embed work; and then it's just a matter of > figuring out the bridge code between that and the rest of the VM. I still don't understand why this has to be in the same process space as the VM. Wouldn't it be a lot simpler to create a simple RPC layer (all localhost of course) to interface between the VM and a Python server that spins up multiple processes or sessions? Kind of like how Python for a web server would work. If a request comes in-band from the client to the VM that necessitates handing off to Python, the RPC layer would do this. Just trying to think architecturally here. -- https://mail.python.org/mailman/listinfo/python-list
Re: A newbie doubt on methods/functions calling
Well I jump from TCL to Python. And found that it was very convenient to use Procs there. So I was looking for that luxury in Python. I am not trying to reinvent the wheel. I was just curious to know if there is any possibility to create a caller function in my way (TCL) where I can call python function/proc inside myProc. -- https://mail.python.org/mailman/listinfo/python-list
Re: Question on multiple Python users in one application
From: "Paul Rubin" I don't think Python threads are the answer. You want a separate interpreter per user, which is annoying to do with CPython. Do you have concrete performance expectations about the sharing of data between interpreter sessions? Those old mainframes were very slow compared to today's machines, even counting Python's interpretation overhead. Without separate context it sounds like Python threads are moderately useless except in special circumstances, and this certainly isn't one of those circumstances. As for performance, especially with shared data, it is an interesting question, but one not easily answered at the moment. The VM does a lot of interesting JIT type stuff, so gets performance that easily exceeds the most recent hardware implementation of the machine, which was only made a few years ago. Yes, a raw PC can do things faster (obviously). But people like the mainframe environment for the same reason people like Python over C: it does what they want, easier. They are willing to pay some performance for this -- but only some. They still have network response times to meet and need to get the monthend batch processing done before midnight. In our case, shared data is no slower to access than process-unique data. Well, maybe a little slower if you have to acquire and release a lock, because that takes a few nanoseconds or so. So I'd prefer that Python shared data access was not a lot slower than non-shared access. My impression is that all data is in a common heap, so a lock is required to access anything. I'd think that makes all data access equally slow or equally fast. I haven't found much data on multiple separate interpreters within a single process space. What exactly do you get that is unique to each interpreter? Separate heaps? Seperate GC locks? Separate process threads? I would guess maybe separate builtins. Does multiple interpreters make it harder to share data between the interpreters? Loren -- https://mail.python.org/mailman/listinfo/python-list
Re: A newbie doubt on methods/functions calling
On Thu, Oct 6, 2016, at 19:27, Loren Wilton wrote: > So I don't want to WRITE a Python interpreter for the actual mainframe > environment. I want to use an interpreter for an existing environment > (Windows) where there are already a lot of existing libraries. But > since a lot of the data to be analyzed is on the mainframe > environment, and the results would be wanted there too, I need to > extend the Python data access to the mainframe environment. Honestly, the best implementation strategy I can think of is to first implement a Python interpreter for the actual mainframe environment. Then invent an RPC layer that can semi-transparently bridge the two for when you want to call a module that only exists in the Windows environment (or call _from_ such a module back to an object/module that only exists in the mainframe environment), whether it's because it requires a Windows library or because you want the Windows python interpreter to do the heavy lifting because it's faster. Do you have C in the mainframe environment? -- https://mail.python.org/mailman/listinfo/python-list
Re: Question on multiple Python users in one application
To be fair, the sharing of data between threads is no different from other concerns about global state. The only thing threads change is that you can have multiple functions doing stuff at once. state = [1,2,3] def func(): state[1] += 5 # do work state[1] -= 5 It occurs to me that threads must have some state of their own. For instance, if I have two thread and they are both in that same function above, one thread needs to know that it is currently on the bytecode for the first statement, and the other needs to know that it is in the bytecode for some of the # do work. In other words, each thread has a current instruction pointer. If the thread has a current instruction pointer, then it must also know what function it is currently in, and therfore have a pointer (or some such) to the current function locals: def func1(): ham = 0.0; ham += 1;// thread 1 here def func2(): ham = 0.0; ham -= 1;// thread 2 here In my C++ understanding of threads, those manipulations on ham won't interact. I would hope this is also true in Python? If I'm right that a thread knows what function it is in, and therefore has function-level state access to the current function, then I'd think that it also has to have module level state. After all, functions occur inside modules, or at least they can: spam.py: ham = 0.0; def func(): ham += 1;// thread 1 here eggs.py: ham = 0.0; def func(): ham -= 1;// thread 2 here Here I would again hope that the simultaneous threads would not cause collisions on ham, since (I think?) there should be spam.ham and eggs.ham, which would be two separate things. The worry I can see is if the CPython interpreter has taken some shortcuts and has global variables for the pointers to the module globals and the funciton locals. In that case, if two threads aren't in the same function I can see all hell breaking loose pretty easily. Loren -- https://mail.python.org/mailman/listinfo/python-list
Re: Question on multiple Python users in one application
I still don't understand why this has to be in the same process space as the VM. Wouldn't it be a lot simpler to create a simple RPC layer (all localhost of course) to interface between the VM and a Python server that spins up multiple processes or sessions? Kind of like how Python for a web server would work. If a request comes in-band from the client to the VM that necessitates handing off to Python, the RPC layer would do this. Assuming that all Python usages are single procedure calls, then all of the parameters can be marshalled and passed to another process. I suppose it may or may not be possible to modify the internals of structured parameters (arrays, lists, dictionaries, etc) and return the modified values, depending on how marshalling works. And it may or may not be fast. If the Python code wants to make reverse calls back into the VM environment (and it may well want to do this, for instance for database access) then it is still possible but starts to get trickier, and again possibly slower. Depending on use cases, any marshalling slowness might not matter at all. Or it might matter a great deal. Unfortunately I don't yet have any clearly defined use cases from the people that should be thinking about that. So at the moment I'm trying to follow the original concept spec as closely as possible, which is "integrate Python in the VM environment." Loren -- https://mail.python.org/mailman/listinfo/python-list
Re: A newbie doubt on methods/functions calling
Let me rephrase my question in other way. class myClass: def __init__(self, var): self.var = var myObj = myClass(abc) # I am calling instance with function name and arguments myObj func1 arg1 arg2 Can i associate any function like __init__ with instance ? Means if I just use instance anywhere as above, it calls that function in the class and take everything as argument which I mentioned after instance? Something like below class myClass: def __call__(self, args): # Then I parse the args here and call function internally self.func1(arg1, arg2) -- https://mail.python.org/mailman/listinfo/python-list
Re: A newbie doubt on methods/functions calling
Honestly, the best implementation strategy I can think of is to first implement a Python interpreter for the actual mainframe environment. Then invent an RPC layer that can semi-transparently bridge the two for when you want to call a module that only exists in the Windows environment (or call _from_ such a module back to an object/module that only exists in the mainframe environment), whether it's because it requires a Windows library or because you want the Windows python interpreter to do the heavy lifting because it's faster. Do you have C in the mainframe environment? Essentially the answer is "no". There is a thing that calls itself a C compiler, but it can only compile truely trivial programs, and then only after a great deal of manual hacking on the source code to change the syntax to what this compiler likes. There is no equivalent of "make". I'd be better off starting with a Python interpreter in JaveScript or the like, if I wanted to do a transliteration that would actually work. I'm trying to avoid this, it would be months of work. Loren -- https://mail.python.org/mailman/listinfo/python-list
Re: static, class and instance methods (Reposting On Python-List Prohibited)
"Gregory Ewing" a écrit dans le message de news:[email protected]... Lawrence D’Oliveiro wrote: Every function is already a descriptor. Which you can see with a simple experiment: >>> def f(self): ... print("self =", self) ... I thought yesterday that every thing was clear. But I have a doubt now with the following line: >>> g = f.__get__(17, None) The signature of __get__ method is __get__ (self, inst, owner) so once again the first parameter is filled automatically. Is method __get__ itself a descriptor with an attribute __get__ to perform the operation ? Hum, it would be endless ... vars(f.__get__) TypeError: vars() argument must have __dict__ attribute no, so how does it work here ? >>> g >>> g() self = 17 -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Question on multiple Python users in one application
On Fri, Oct 7, 2016 at 3:06 PM, Loren Wilton wrote: > If the thread has a current instruction pointer, then it must also know what > function it is currently in, and therfore have a pointer (or some such) to > the current function locals: > > def func1(): >ham = 0.0; >ham += 1;// thread 1 here > > def func2(): >ham = 0.0; >ham -= 1;// thread 2 here > > In my C++ understanding of threads, those manipulations on ham won't > interact. I would hope this is also true in Python? > > If I'm right that a thread knows what function it is in, and therefore has > function-level state access to the current function, then I'd think that it > also has to have module level state. After all, functions occur inside > modules, or at least they can: > > spam.py: > ham = 0.0; > def func(): >ham += 1;// thread 1 here > > eggs.py: > ham = 0.0; > def func(): >ham -= 1;// thread 2 here > > Here I would again hope that the simultaneous threads would not cause > collisions on ham, since (I think?) there should be spam.ham and eggs.ham, > which would be two separate things. Correct on both counts. Threads executing different functions cannot trample on each other's locals. Even if they're executing different invocations of the same function, they won't trample on locals. With your module example (assuming you add "global ham" to each function), they would be fine too; however, if two threads are in two invocations of the same function, it's possible for them to overwrite each other's change. (It'd require pinpoint timing, but it could happen.) So if you're going to use threads, you have to not use mutable global state. That's honestly not a difficult rule to follow; most programmers should have a healthy caution around mutable global state anyway, regardless of threads. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: segfault using shutil.make_archive
Tim writes: > I need to zip up a directory that's about 400mb. > I'm using shutil.make_archive and I'm getting this response: > > Segmentation fault: 11 (core dumped) > > The code is straightforward (and works on other, smaller dirs): > > shutil.make_archive(os.path.join(zip_dir, zname), 'zip', tgt_dir) > > I guess I could drop the convenience of make_archive and use zipfile but that > seems to be exactly what make_archive does. >From your description (above and in other messages of this thread), I guess, that the cause is a (running out of) memory problem. Memory allocations are frequent (in many places) and they fail rarely. Therefore, there is quite a high probability that some of those allocations fail to check that the allocation has been successful. If this happens (and the allocation fails), then a "SIGSEGV" is almost certain. I would try to move the task on a system with a larger address space (more RAM, more swap space, maybe a 64 bit system). It may succeed there. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python 3.5 amd64 and win32service
>> "C:\Users\Laci\AppData\Local\Programs\Python\Python35\lib\site-packages\win32\PythonService.exe" > I wanted you to run the above executable, not python.exe. If it fails > you'll get more information about why it's failing when run directly > then when the service controller runs it. Since you're using a > per-user installation of Python 3.5, possibly it can't find > python35.dll. This is something I cannot understand. This is what happens when I start it from the command line, as a normal user: C:\>"C:\Users\Laci\AppData\Local\Programs\Python\Python35\lib\site-packages\win32\PythonService.exe" -debug TestService Debugging service TestService - press Ctrl+C to stop. Info 0x40001002 - The TestService service has started. *Meanwhile, the service manager shows that the service is not started*. Then I press Ctrl+C several times and I see this: C:\>"C:\Users\Laci\AppData\Local\Programs\Python\Python35\lib\site-packages\win32\PythonService.exe" -debug TestService Debugging service TestService - press Ctrl+C to stop. Info 0x40001002 - The TestService service has started. Stopping debug service. Stopping debug service. Stopping debug service. Stopping debug service. Stopping debug service. But the process is not closed. I had to terminate pythonservice.exe from task manager. Next thing I tried is to start it as an administrator: there is no difference. On the console it says that it has been started, but in service manager it appears to be stopped. I got no messages about missing DLL files at all. The very same MWE works perfectly on a new virtual machine with Python 3.5.2 amd64 + pypiwin32. Any idea what could be wrong? -- https://mail.python.org/mailman/listinfo/python-list
