Re: [Tutor] Interpreter pasting Question
On 28/12/2018 05:58, Avi Gross wrote: > This is a serious question. I have tried things and searched and remain > stumped. It is about python and perhaps just the interpreter. > > Copying and pasting multiple lines into the interpreter fails in mysterious > ways, unless they are a logical single entity. That will depend on the interpreter. Which one are you using? > Is there a way to change this behavior, or perhaps an editor/environment > that feeds multiple lines more carefully to the interpreter? IDLE does what you say but IDLEX accepts multiple lines. The vanilla command line >>> accepts multiple lines (on Linux at least) PyCharm has multiple options for how it processes pasted lines/blocks. There are even IDEs that will evaluate/execute selected text without the need to paste. It all depends on the tool. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] dangerous class neighborhood
That was nice 😊 I did not know the python-list. -Original Message- From: Tutor On Behalf Of Steven D'Aprano Sent: Friday, December 28, 2018 5:56 AM To: tutor@python.org Subject: Re: [Tutor] dangerous class neighborhood On Thu, Dec 27, 2018 at 09:48:02PM -0500, Avi Gross wrote: > Sometimes when I post something I get back comments and evaluate them > and learn quite a bit. I then reply and debate every little point and > it can continue for a few rounds. I think you sent this to the wrong mailing list. The original post and discussion was on Python-List, but you replied to Tutor. -- Steve ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] decomposing a problem
> On Dec 27, 2018, at 3:32 PM, Avi Gross wrote: > > [Mark Lawrence please press DELETE now in case the rest of this message is > all about you.] > [[If that is not working, if on Windows, try Control-ALT-DELETE as that will > really get rid of my message.]] > Hi Avi, Mark doesn’t have a basis for complaining, of course, as he can simply not read your posts. > Back to replying to Steven, > > Of course I want to be corrected when wrong. > > I think everyone here knows I tend to be quite expansive in my thoughts and > sometimes to the point where they suggest I am free-associating. I am trying > to get to the point faster and stay there. Since you are expressing interest, I'll give some thoughts. I think it’s important not only for writing, but for economy of thinking to use fewer words and simpler concepts, and it can make us better programmers and teachers. Previously, when I worked alone as a programmer, I was stuck in overcomplicated ways of thinking. It’s “getting out there” and interacting with people that rejuvenated my thinking, and I’ll be forever grateful. One form of practice at this is to edit my posts for brevity. Here’s a link about brevity in writing: http://copymatter.com/embracing-brevity/ It helps me as well that I tutor students in math and computer science regularly, because it forces me to get more simple and concrete. A student is a “feedback device” — when I’m doing better, I can read the results in their expression and their understanding. I think it’s important both to have something you are aiming for (a sense of what level of brevity you’d like to achieve) and a feedback mechanism that helps you to know if you are succeeding. Take or leave these thoughts as you see fit. Mike ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Interpreter pasting Question
Steven,
Thanks. You are right. I am no longer going to talk IDLE worship!
When I open up a command window and run what I assumed was the same version
of python, no problems. Should have tried that but frankly that means I am
stuck with having to then do other things differently like reloading an
edited file. Oh well.
It sounds like the problem is that IDLE is (invisibly) intercepting what I
type or paste and then passing it on to a slaved interpreter. There likely
are other side effects and another editor/environment may not have this
issue. I will try some.
As one of my programming styles includes lots of hands-on incremental
analysis of bits and pieces to get them working before combining them, I
have no time for an idyllic experience.
OK, just kidding. Nice to have choices and I particularly want to move on to
using notebooks in which I can have batches of code I can run inline and
even interleave bits of multiple languages.
Yes, I am aware that my experiments with exec are not testing the same
thing. They do indicate ways to get around the issue even using IDLE and I
am now thinking of ways to do more dynamic things by building code and then
executing it. So, not really time wasted. The fact that python relies on
indentation means you can have problems if things are not aligned just
right.
Here is my work-around if I happen to be using IDLE and want to copy in
something huge from say a web page.
Type this:
Cmd = """
Do the paste.
Type on a line by itself:
"""
Now type
exec(Cmd)
May not work all the time, but perhaps a reasonable work around.
-Original Message-
From: Tutor On Behalf Of
Steven D'Aprano
Sent: Friday, December 28, 2018 1:58 AM
To: tutor@python.org
Subject: Re: [Tutor] Interpreter pasting Question
On Fri, Dec 28, 2018 at 12:58:00AM -0500, Avi Gross wrote:
[...]
> Copying and pasting multiple lines into the interpreter fails in
> mysterious ways, unless they are a logical single entity.
>
> Is there a way to change this behavior, or perhaps an
> editor/environment that feeds multiple lines more carefully to the
interpreter?
Which interpreter are you using?
If it is the regular Python REPL (Read Eval Print Loop), then pasting
multiple lines should work, with some limitations. For example, I can
paste:
x = 5
y = 6
as two lines, and it works fine under Linux. I see no reason why it should
be different under Windows.
If you just run "python" in the Windows shell (cmd.exe or whatever its
called), you should get an interactive interpreter. What happens when you
paste multiple lines in that?
[...]
> When copied in looks like this:
>
> >>> X=5
>
> Y=6
>
> SyntaxError: multiple statements found while compiling a single
> statement
That looks like a bug. Are you using IDLE? Perhaps it has been fixed in
newer versions of Python, and if not, you should report it as a bug on the
bugtracker
https://bugs.python.org/
but:
(1) try searching for similar bug reports first;
(2) read this first:
http://www.sscce.org/
(3) and do try to keep your bug report short and to the point.
It looks like others have this problem with IDLE too:
https://duckduckgo.com/?q=idle+paste+multiple+lines
IPython/Jupyter allows pasting of multiple lines; I expect that bpython will
too.
https://ipython.org/
https://bpython-interpreter.org/
But as I said, the vanilla Python REPL ought to work. How are you starting
the interpreter? My guess is that you're using IDLE.
[...]
> Python has an eval() and an exec() and I would assume the latter would be
a
> way to see what works.
No, exec() executes Python code, it doesn't try to simulate a REPL.
> Here are three lines using \n:
>
> >>> exec("x=6\ny=7\nprint(x+y)")
> 13
>
>
> That seems to work. Again, if I copied and pasted the same as three lines,
> it fails.
Without knowing the system you are using, and how you copy and paste, it
is hard to comment except to say "Works for me". Here are three lines:
x = 6
y = 7
print(x + y)
If I select those three lines with the mouse, copy, then paste into a
standard Python interactive interpreter, I get this:
py> x = 6
py> y = 7
py> print(x + y)
13
exactly as expected. But if I do it in IDLE, I get this:
>>> x = 6
y = 7
print(x + y)
SyntaxError: multiple statements found while compiling a single
statement
>>>
That *really* sounds like a bug to me. But perhaps I just don't
understand IDLE.
--
Steve
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Interpreter pasting Question
On 12/28/18 10:13 AM, Avi Gross wrote: > As one of my programming styles includes lots of hands-on incremental > analysis of bits and pieces to get them working before combining them, I > have no time for an idyllic experience. don't presume to know what environment will actually work for you, but that working model sounds somewhat like Jupyter, which has a "notebook" for recording live code snippets, commentary, etc. - and a way to clip those bits and pieces into your combination. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Interpreter pasting Question
On 28/12/2018 17:13, Avi Gross wrote: > It sounds like the problem is that IDLE is (invisibly) intercepting what I > type or paste and then passing it on to a slaved interpreter. There likely > are other side effects and another editor/environment may not have this > issue. I will try some. Do not confuse the interactive prompt with the interpreter. The prompt is unique to the tool and the interpreter is, ultimately, used by the tool in the background. But each tool handles the input differently before invoking the interpreter. > As one of my programming styles includes lots of hands-on incremental > analysis of bits and pieces to get them working before combining them, I > have no time for an idyllic experience. Remember tat IDLE is not intended as an industrial strength IDE like Eclipse, Netbeans, VS or Blackadder etc it was intended to be an easy to learn tool for beginners. There are several tools that do what you want better than IDLE - including IDLEX. (And if things go to plan many IDLEX features will appear in the 3.7/3.8 versions of IDLE.) But even IDLEX is not the last word in toolsets. There are many alternatives including IPython and Jupyter etc. > OK, just kidding. Nice to have choices and I particularly want to move on to > using notebooks in which I can have batches of code I can run inline and > even interleave bits of multiple languages. Sounds like you should take a look at IPython/Jupyter > Now type > > exec(Cmd) > > May not work all the time, but perhaps a reasonable work around. The biggest issue is that it requires the environment to pre-exist - variables initialised sensibly, etc. But thats true of any tool that offers that kind of eval/exec function. But mostly you can avoid all of that scaffolding with a better toolset. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] decomposing a problem
Steve, I am going to just respond to one part of your message and will snip the rest. I am not is disagreement with most of what you say and may simply stress different aspects. I will say that unless I have reason to, I don't feel a need to test speeds for an academic discussion. Had this been a real project, sure. Even then, if it will need to run on multiple machines using multiple incarnations of python, the results will vary, especially if the data varies too. You suggest that discussions backed by real data are better. Sure. But when a discussion is abstract enough, then I think it perfectly reasonable to say "may be faster" to mean that until you try it, there are few guarantees. Many times a method seems superior until you reach a pathological case. One sorting algorithm is fast except when the data is already almost fully sorted already. So why do I bother saying things like MAY? It seems to be impossible to please everybody. There are many things with nuance and exceptions. When I state things one way, some people (often legitimately) snipe. When I don't insist on certainty, other have problem with that. When I make it short, I am clearly leaving many things out. When I go into as much detail as I am aware of, I get feedback that it is too long or boring or it wanders too much. None of this is a problem as much as a reality about tradeoffs. So before I respond, here is a general statement. I am NOT particularly interested in much of what we discuss here from a specific point of view. Someone raises a question and I think about it. They want to know of a better way to get a random key from a dictionary. My thought is that if I needed that random key, maybe I would not have stored it in a dictionary in the first place. But, given that the data is in a dictionary, I wonder what could be done. It is an ACADEMIC discussion with a certain amount of hand waving. Sometimes I do experiment and show what I did. Other times I say I am speculating and if someone disagrees, fine. If they show solid arguments or point out errors on my part or create evidence, they can change my mind. You (Steve) are an easy person to discuss things with but there are some who are less. People who have some idea of my style and understand the kind of discussion I am having at that point and who let me understand where they are coming from, can have a reasonable discussion. The ones who act like TV lawyers who hear that some piece of evidence has less than one in a quadrillion chance of happening then say BUT THERE IS A CHANCE so reasonable doubt ... are hardly worth debating. You replied to one of my points with this about a way to partition data: --- The obvious solution: keys = list(mydict.keys()) random.shuffle(keys) index = len(keys)*3//4 training_data = keys[:index] reserved = keys[index:] --- (In the above, "---" is not python but a separator!) That is indeed a very reasonable way to segment the data. But it sort of makes my point. If the data is stored in a dictionary, the way to access it ended up being to make a list and play with that. I would still need to get the values one at a time from the dictionary such as in the ways you also show and I omit. For me, it seems more natural in this case to simply have the data in a data frame where I have lots of tools and methods available. Yes, underneath it all providing an array of indices or True/False Booleans to index the data frame can be slow but it feels more natural. Yes, python has additional paradigms I may not have used in R such as list comprehensions and dictionary comprehensions that are conceptually simple. But I did use the R-onic (to coin a phrase nobody would ironically use) equivalents that can also be powerful and I need not discuss here in a python list. Part of adjusting to python includes unlearning some old habits and attitudes and living off this new land. [[Just for amusement, the original R language was called S so you might call its way of doing things Sonic.]] I see a balance between various ways the data is used. Clearly it is possible to convert it between forms and for reasonable amounts of data it can be fast enough. But as you note, at some point you can just toss one representation away so maybe you can not bother using that in the first place. Keep it simple. In many real life situations, you are storing many units of data and often have multiple ways of indexing the data. There are representations that do much of the work for you. Creating a dictionary where each item is a list or other data structure can emulate such functionality and even have advantages but if your coding style is more comfortable with another way, why bother unless you are trying to learn other ways and be flexible. As I have mentioned too many times, my most recent work was in R and I sometimes delight and other times groan at the very different ways some things are done when using specific modules or libraries. But even within a language and environment
Re: [Tutor] decomposing a problem
On 28/12/2018 20:34, Avi Gross wrote: > So before I respond, here is a general statement. I am NOT particularly > interested in much of what we discuss here from a specific point of view. > Someone raises a question and I think about it. They want to know of a > better way to get a random key from a dictionary. My thought is that if I > needed that random key, maybe I would not have stored it in a dictionary in > the first place. But, given that the data is in a dictionary, I wonder what > could be done. It is an ACADEMIC discussion with a certain amount of hand > waving. But you need to apply real world constraints. The choice of data type is intrinsic to the language in use. (The same is true of control structures - loops, decision points etc - but that is less pertinent here.) If you program in C pretty much all you get is the array. Everything else (including struct/union/typedef) is hand crafted by the programmer. If its Lisp then you get the list and anything else is coded (or simulated in code) by hand. In Python we have lists, tuples, dictionaries and sets. Anything else, including subclassing the basic types, is down to the programmer (or finding a module already created by somebody else). In Smalltalk we have over a hundred basic collection types to choose from. And the choice to subclass any of them. So when you address a problem in any given language the available solutions must be constrained by whatever the language in question offers. Wishing for non-existent data structures that may exist elsewhere is simply to request a new feature to be designed and programmed in the language at hand. That may be the best solution depending on the nature of the problem but we need to recognise the nature of the request. It is still a new feature. Of course we can learn a great deal by comparing features on one language against another but in terms of solving a specific request we need specific answers too. > --- > The obvious solution: > > keys = list(mydict.keys()) > random.shuffle(keys) > index = len(keys)*3//4 > training_data = keys[:index] > reserved = keys[index:] > --- > For me, it seems more natural in this case to simply have the data in a data > frame where I have lots of tools and methods available. But only if such a data frame exists. In Python it does not (at least, not in the standard library). So any reference to such a non existent structure is in effect a work request for someone to build one. To do so requires a specification or design that the OP can follow or better still a prototypical template. It also assumes a much higher level of skill than the original request and the "obvious solution". > but if your coding style is more comfortable with another way, why bother > unless you are trying to learn other ways and be flexible. If your current language does not support the structure you desire you have three choices: 1) change your programming language or 2) build the missing feature or 3) find a workaround using the structures available. Most people opt for #3. (Although all may be valid options depending on the circumstance) -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] decomposing a problem
On Fri, Dec 28, 2018 at 03:34:19PM -0500, Avi Gross wrote:
[...]
> You replied to one of my points with this about a way to partition data:
>
> ---
> The obvious solution:
>
> keys = list(mydict.keys())
> random.shuffle(keys)
> index = len(keys)*3//4
> training_data = keys[:index]
> reserved = keys[index:]
> ---
>
> (In the above, "---" is not python but a separator!)
>
> That is indeed a very reasonable way to segment the data. But it sort of
> makes my point. If the data is stored in a dictionary, the way to access it
> ended up being to make a list and play with that. I would still need to get
> the values one at a time from the dictionary such as in the ways you also
> show and I omit.
Yes? How else do you expect to get the value given a key except by
looking it up?
> For me, it seems more natural in this case to simply have the data in
> a data frame where I have lots of tools and methods available.
I'm not sure if your understanding of a data frame is the same as my
understanding. Are you talking about this?
http://www.r-tutor.com/r-introduction/data-frame
In other words, a two-dimensional array of some sort?
Okay, you have your data frame. Now what? How do you solve the problem
being asked? I'm not interested in vague handwaving that doesn't solve
anything. You specified data in a key:value store, let's say like this:
mydict = {'spam': 25, 'ham': 2, 'eggs': 7, 'cheddar': 1, 'brie': 14,
'aardvark': 3, 'argument': 11, 'parrot': 16}
Here it is as a data frame:
df = [['spam', 'ham', 'eggs', 'cheddar', 'brie', 'aardvark', 'argument',
'parrot'],
[25, 2, 7, 1, 14, 3, 11, 16]]
Now what? How do you randomly split that into randomly selected set of
training data and reserved data?
Feel free to give an answer in terms of R, provided you also give an
answer in terms of Python. Remember that unlike R, Python doesn't have a
standard data frame type, so you are responsible for building whatever
methods you need.
--
Steve
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] decomposing a problem
I will answer this question then head off on vacation.
I said that "I" have lots of experience dealing with data in dataframes.
Most of it is not in Python, obviously. It works well for some kinds of
data, not others. Let me not diverge from the purpose of the group by saying
a version of it is commonly used in the python module pandas. Python has
been extended in many ways including arrays in many dimensions in numpy.
There are also matrix types that are 2-D or more but all the above can only
hold one kind of data at a time. In real life, you have tabular data where
each column can be of any type.
There are quite a few implementations out there. Yes, your pointer to R
shows a basic version. But I note python has its own versions too.
Implementations may have some nice tools for taking subsets of rows or
columns, modifying existing structures and adding additional rows/columns
and iterating over them and generating all kinds of logical subsets. The
implementation may be as sort of a list of vectors of the same length. Lots
of flexibility if your data fits, not so much otherwise. It is one serious
reason R was used by so many for purposes like statistics and graphics but
python has been extended and now more and more people are using it for such
things as well as so many other things that R is not designed for.
Just one point since it may be relevant. You can index a data.frame in ways
not that different than objects in python and one is to generate a condition
like "variable > 5" (or much more complex) that generates a Boolean vector
that can be used to take a subset of all rows into a copy for one purpose,
such as training in machine learning and simply negating the vector gets the
remaining ones. You can do similar things in python. My recent expertise and
experience relied heavily on the data structures R uses, where everything is
a vector and most operations are vectorized. In python, I am learning to do
things in whatever way works and that has been interesting too.
You asked about the dictionary. R has things sort of similar but not quite.
It has environments that most people would not use as a dictionary. So in R,
I might have made a data.frame to hold key/value columns albeit keys would
not be forced to be unique. Thus selecting random keys would just mean
selecting a random number in the range of the number of rows and taking that
item in a key column.
What you call a data.frame is not a data.frame. It is a list of two lists.
If you will pardon my putting python code here, it might look like this and
note numpy and pandas and other mathematical/scientific modules have plenty
of methods available. No real need for me to show you in R, unless offline.
import numpy as np
import pandas as pd
Your two forms of data will now be instantiated and then put into the
alternate formats to show some things.
>>> mydict = {'spam': 25, 'ham': 2, 'eggs': 7, 'cheddar': 1, 'brie': 14,
'aardvark': 3, 'argument': 11, 'parrot': 16}
>>> df = [['spam', 'ham', 'eggs', 'cheddar', 'brie', 'aardvark', 'argument',
'parrot'],
[25, 2, 7, 1, 14, 3, 11, 16]]
>>> mydict
{'spam': 25, 'ham': 2, 'eggs': 7, 'cheddar': 1, 'brie': 14, 'aardvark': 3,
'argument': 11, 'parrot': 16}
>>> df
[['spam', 'ham', 'eggs', 'cheddar', 'brie', 'aardvark', 'argument',
'parrot'], [25, 2, 7, 1, 14, 3, 11, 16]]
There are many ways to create a data.frame from lists, vectors,
dictionaries, ...
Your data is a bit skimpy for showing much but it can easily be imported:
>>> df_from_list = pd.DataFrame(data=df)
>>> df_from_list
01 23 4 5 6 7
0 spam ham eggs cheddar brie aardvark argument parrot
1252 7114 311 16
I would prefer to import this as columns:
I will first convert your list of 2 lists to a vertical format more suitable
in such tables:
>>> newdf = list(zip(df[0], df[1]))
>>> newdf
[('spam', 25), ('ham', 2), ('eggs', 7), ('cheddar', 1), ('brie', 14),
('aardvark', 3), ('argument', 11), ('parrot', 16)]
>>> vert_df = pd.DataFrame(newdf, columns=["FOOD", "AMOUNT"])
>>> vert_df
FOOD AMOUNT
0 spam 25
1 ham 2
2 eggs 7
3 cheddar 1
4 brie 14
5 aardvark 3
6 argument 11
7parrot 16
It does not format well here but you see the two columns as well as a sort
of index.
You can take subsets of rows:
>>> vert_df[2:4]
FOOD AMOUNT
2 eggs 7
3 cheddar 1
You can select by conditions:
>> vert_df[vert_df['AMOUNT'] >= 7]
FOOD AMOUNT
0 spam 25
2 eggs 7
4 brie 14
6 argument 11
7parrot 16
And so on. The point is you can often read in data and manipulate it. R has
multiple sets of tools including one in what they call the tidyverse. In
English, given such a data structure with any
Re: [Tutor] decomposing a problem
On Fri, Dec 28, 2018 at 10:39:53PM -0500, Avi Gross wrote: > I will answer this question then head off on vacation. You wrote about 140 or more lines, but didn't come close to answering the question: how to randomly split data from a dictionary into training data and reserved data. -- Steve ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
