[Tutor] look back comprehensively
I have been thinking about the thread we have had where the job seemed to be
to read in a log file and if some string was found, process the line before
it and generate some report. Is that generally correct?
The questioner suggested they needed both the entire file as one string but
also as a list of strings. Suggestions were made, IF SO, to read the entire
file twice, once as a whole and once as lines. Another suggestion was to
read either version ONCE and use cheaper Python methods to make the second
copy.
We also looked at a similar issue about using a buffer to keep the last N
lines.
I thought of another tack that may be in between but allow serious
functionality.
OUTLINE:
Use just the readlines version to get a list of strings representing each
line. Assuming the searched text is static, no need for regular expressions.
You can ask if
'something' in line
But if you find it, you may not have an index so a use of enumerate (or zip)
to make a tuple might be of use. You can do a list comprehension on an
enumerate object to get both the indexes where the requested 'something' was
found and also optionally the line contents (ignored) and be able to use the
indices to look at the index (or more) before.
Here is an example using a fake program where I create the four lines and
generate only the tuples needed for further processing, or just the indices
for the line ABOVE where it was found. It can be done with simple list
comprehensions or made into a generator expression.
-CODE-
"""
Sample code showing how to read a (simulated) file
and search for a fixed string and return the item
number in a list of strings for further processing
including of earlier lines.
"""
# Make test data without use of file
fromfile = str1 = """alpha line one
beta line two
gamma line three
alphabet line four"""
lines= fromfile.split('\n')
print("RAW data: ", lines) # just for illustration
errors = [(index,line)
for (index, line) in enumerate(lines)
if 'bet' in line]
just_indices = [index - 1
for (index, line) in enumerate(lines)
if 'bet' in line]
from pprint import pprint
print("ERROR tuples:") # just for illustration
pprint(errors)
print("Just error indices:")
pprint(just_indices)
-END-CODE-
-OUTPUT-
RAW data: ['alpha line one', 'beta line two', 'gamma line three', 'alphabet
line four']
ERROR tuples:
[(1, 'beta line two'), (3, 'alphabet line four')]
Just error indices:
[0, 2]
-END-OUTPUT-
Again, this did two ways, and only one is needed. But the next step would be
to iterate over the results and process the earlier line to find whatever it
is you need to report. Many ways to do that such as:
for (index, ignore) in errors
or
for index in just_indices
You can use a regular expression on a line at a time. And so on.
Again, other methods mentioned work fine, and using a deque to store earlier
lines in a limited buffer while not reading the entire file into memory
would also be a good way.
Warning: the above assumes the text found will never be in the zeroeth line.
Otherwise, you need to check as accessing line -1 may actually return the
last line!
As stated many times, there seem to be an amazing number of ways to do
anything. As an example, I mentioned using zip above. One obvious method is
to zip it with a range statement making it look just like enumerate. A more
subtle one would be to make a copy of the set of lines the same length but
each line content shifted by one. Zip that to the original and you get a
tuple with (line 0, null) then (line 1, line 0) up to (line n, line n-1)
Yes, that doubles memory use but you can solve so much more in one somewhat
more complicated list comprehension. Anyone want to guess how?
If you recall, some regular expression matches something on the previous
line. Let us make believe you wrote a function called do_the_match(line, re)
that applies the regular expression to the line of text and returns the
matching text or perhaps an empty string if not found. So if you define that
function then the following code will work.
First, make my funny zip:
I make lines_after as copy of lines shifted over one. Or more exactly
circularly permuted by one. The goal is to search in line_after and if
found, do the regular expression match in line before.
>>> lines_after = lines[1:]
>>> lines_after.append(lines[0])
>>> lines_after
['beta line two', 'gamma line three', 'alphabet line four', 'alpha line
one']
>>> lines
['alpha line one', 'beta line two', 'gamma line three', 'alphabet line
four']
>>> list(zip(lines_after, lines))
[('beta line two', 'alpha line one'), ('gamma line three', 'beta line two'),
('alphabet line four', 'gamma line three'), ('alpha line one', 'alphabet
line four')]
So the list comprehension looks something like this:
matches = [ do_theMatch(line, re)
for (line_after, line)
[Tutor] Request to join subscriber list
Sir I am really excited to learn about programming languages and as python becoming very famous these days i just wanted a list of all commands of it. Please if you are able to send it ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] All of a sudden code started throwing errors
Dear Python Experts
Could you please let me know why am I seeing the following errors as before
this it was working consistently?
root:~/qa/robot_tests# python3 -m robot --variable
SIGNAL_LEVEL_THRESHOLD:-60 wifi_testing.robot
==
Wifi Testing :: This is the Maschine Native OS Build Wi-Fi Test.
==
Initialize Wi-Fi Module |
PASS |
--
Enable Wi-Fi Module |
PASS |
--
Connect Wi-Fi Module to SSID |
PASS |
--
Check for Wi-Fi Connectivity |
PASS |
--
Log Wi-Fi Module IP |
PASS |
--
Get Gateway IP and Check Whether Wi-Fi is Pingable|
PASS |
--
Check for Wi-Fi Signal Strength with Threshold|
FAIL |
'-68 >= -60' should be true.
--
Wifi Testing :: This is the Maschine Native OS Build Wi-Fi Test. |
FAIL |
7 critical tests, 6 passed, 1 failed
7 tests total, 6 passed, 1 failed
==
Output: /home/root/qa/robot_tests/output.xml
Log: /home/root/qa/robot_tests/log.html
Report: /home/root/qa/robot_tests/report.html
root:~/qa/robot_tests# cat /proc/net/wireless
Inter-| sta-| Quality| Discarded packets | Missed
| WE
face | tus | link level noise | nwid crypt frag retry misc | beacon
| 22
wlp1s0: 44. -66. -2560 0 0 0 90
Errors:
=
root:~/qa/robot_tests# python3 -m robot --variable
SIGNAL_LEVEL_THRESHOLD:-70 wifi_testing.robot
Traceback (most recent call last):
File "/usr/lib/python3.5/runpy.py", line 183, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "/usr/lib/python3.5/runpy.py", line 142, in _get_module_details
return _get_module_details(pkg_main_name, error)
File "/usr/lib/python3.5/runpy.py", line 109, in _get_module_details
__import__(pkg_name)
File "/usr/lib/python3.5/site-packages/robot/__init__.py", line 41, in
from robot.rebot import rebot, rebot_cli
File "/usr/lib/python3.5/site-packages/robot/rebot.py", line 40, in
from robot.conf import RebotSettings
File "/usr/lib/python3.5/site-packages/robot/conf/__init__.py", line 27,
in
from .settings import RobotSettings, RebotSettings
File "/usr/lib/python3.5/site-packages/robot/conf/settings.py", line 33,
in
class _BaseSettings(object):
File "/usr/lib/python3.5/site-packages/robot/conf/settings.py", line 46,
in _BaseSettings
'OutputDir': ('outputdir', abspath('.')),
File "/usr/lib/python3.5/site-packages/robot/utils/robotpath.py", line
84, in abspath
return normpath(_abspath(path), case_normalize)
File "/usr/lib/python3.5/posixpath.py", line 362, in abspath
cwd = os.getcwd()
FileNotFoundError: [Errno 2] No such file or directory
root:~/qa/robot_tests# cat /proc/net/wireless
Inter-| sta-| Quality| Discarded packets | Missed
| WE
face | tus | link level noise | nwid crypt frag retry misc | beacon
| 22
wlp1s0: 45. -65. -2560 0 0 0 120
root:~/qa/robot_tests# python3 -m robot wifi_testing.robot
Traceback (most recent call last):
File "/usr/lib/python3.5/runpy.py", line 183, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "/usr/lib/python3.5/runpy.py", line 142, in _get_module_details
return _get_module_details(pkg_main_name, error)
File "/usr/lib/python3.5/runpy.py", line 109, in _get_module_details
__import__(pkg_name)
File "/usr/lib/python3.5/site-packages/robot/__init__.py", line 41, in
from robot.rebot import rebot, rebot_cli
File "/usr/lib/python3.5/site-packages/robot/rebot.py", line 40, in
from robot.conf import RebotSettings
File "/usr/lib/python3.5/site-packages/robot/conf/__init__.py", line 27,
in
from .settings import RobotSettings, RebotSettings
File "/usr/lib/python3.5/site-packages/robot/conf/settings.py", line 33,
in
class _BaseSettings(object):
File "/usr/lib/python3.5/site-packages/robot/conf/settings.py", line 46,
in _BaseSettings
'OutputDir': ('outputdi
Re: [Tutor] best way to dynamically set class variables?
I really like the idea of using Python to generate another .py file with
the dynamic definitions taken into account.
That way you separate the "hackery" into a single function or module,
document how it works, and have it also annotate the dynamically generated
python source containing the class definitions as you need them with top of
file doc-string stating "this file was dynamically generated by xxx - don't
modify by hand".
That way your code is not confusing, you isolate the trick and the
resulting source, and you have full source files with all class attributes
that developers can look at and source control, rather than looking at a
weird looking class definition. It's just a simple python file after all,
no tricks.
On Fri, Nov 9, 2018 at 11:50 PM Albert-Jan Roskam
wrote:
>
>
> On 10 Nov 2018 01:03, Steven D'Aprano wrote:
>
> On Thu, Nov 08, 2018 at 11:34:35AM -0500, Avi Gross wrote:
> > An interesting discussion that is outside the scope of a group like this
> is
> > HOW malicious things can be done and perhaps how to avoid them.
> >
>
> Isn't the rule, simply:
>
> this_is_stupid = eval(input("please enter malicious code: "))
>
> ... and other uses range from 'code smell' to 'elegant' (where namedtuple
> is an example of the latter)
>
>
> ___
> Tutor maillist - Tutor@python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>
--
*Pablo Lucena*
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
[Tutor] Cannot find reference 'bluetoothctl' in 'sh.py' less... (Ctrl+F1)
Dear Python Experts,
As am newbie to python, I am planning to automate BT functionality test
using Bluez "bluetoothctl" utility by writing python wrapper and robot
framework integrated with Jenkins
I came across the below link:
https://www.reddit.com/r/raspberry_pi/comments/4bxu2o/bluetoothctl_in_python_program/
In the above link, I saw the below code snippet
*code:*
*from sh import bluetoothctl*
*mac = "AA:BB:CC:DD:EE"*
*bluetoothctl("connect",mac)*
And firstly I wanted to verify BT functionality with my PC and the
bluetooth device (basically turned on BT option in my phone and trying to
discover my phone as a BT device) And I have installed the below packages
in my ubuntu 18.04 desktop PC
$* pip3 install sh*
Collecting sh
Downloading
https://files.pythonhosted.org/packages/4a/22/17b22ef5b049f12080f5815c41bf94de3c229217609e469001a8f80c1b3d/sh-1.12.14-py2.py3-none-any.whl
Installing collected packages: sh
Successfully installed sh-1.12.14
*$ pip3 install bluetoothctl*
*Collecting bluetoothctl*
* Could not find a version that satisfies the requirement bluetoothctl
(from versions: )*
*No matching distribution found for bluetoothctl*
$ *pip3 install pexpect*
Collecting pexpect
Downloading
https://files.pythonhosted.org/packages/89/e6/b5a1de8b0cc4e07ca1b305a4fcc3f9806025c1b651ea302646341222f88b/pexpect-4.6.0-py2.py3-none-any.whl
(57kB)
100% || 61kB 1.5MB/s
Collecting ptyprocess>=0.5 (from pexpect)
Downloading
https://files.pythonhosted.org/packages/d1/29/605c2cc68a9992d18dada28206eeada56ea4bd07a239669da41674648b6f/ptyprocess-0.6.0-py2.py3-none-any.whl
Installing collected packages: ptyprocess, pexpect
Successfully installed pexpect-4.6.0 ptyprocess-0.6.0
$
When I try to paste the below code on pycharm and try to point on the word
"bluetoothctl" in the beginning of the line "*from sh import bluetoothctl*"
*from sh import bluetoothctl*
*mac = "your bluetooth mac"*
*bluetoothctl("connect", mac)*
In the pycharm, I see the below error message :
*Cannot find reference 'bluetoothctl' in 'sh.py' less... (Ctrl+F1) *
*Inspection info: This inspection detects names that should resolve but
don't. Due to dynamic dispatch and duck typing, this is possible in a
limited but useful number of cases. Top-level and class-level items are
supported better than instance items.*
Could you please help me to resolve the above issue, like why am I seeing
the above issue it seems to be some importing "bluetoothhctl" module issue
(sorry if my understanding is wrong)
Kindly do the needful
Many Thanks in advance
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Request to join subscriber list
On 11/14/18 6:20 AM, Dev Pratap Singh wrote: > Sir > I am really excited to learn about programming languages and as python > becoming very famous these days i just wanted a list of all commands of it. > Please if you are able to send it these are the "official" documentation links: https://docs.python.org/3/ ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] (no subject)
I need help creating a weighted GPA calculator. I attempted to create one but I was unable to incorporate different leveled classes (College Prep, Honors, and AP). How can I incorporate those leveled classes into the calculator in order to get a final GPA ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] All of a sudden code started throwing errors
> On 14 Nov 2018, at 13:51, Peter Otten <__pete...@web.de> wrote: > > Diagnosis: you managed to remove your current working directory, probably > because you used os.chdir() to switch to a temporary directory and then > forgot to switch back. > > Solution: don't do that ;) > > I recommend that you avoid chdir() in your code and instead always include > the directory into the filename. A pattern I find myself using frequently, when I do need to “cd”, is to write a tiny context manager: ``` import os from contextlib import context manager @contextmanager def pushd(path): old_dir = os.getcwd() os.chdir(path) try: yield finally: os.chdir(old_dir) ``` (I tend to just copy this into projects where I need it (or write it again), as a whole dependency for something so tiny seems like it would be overkill :) -- Matt Wheeler http://funkyh.at ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] GPA calculator
On 14Nov2018 13:51, Whitney Eichelberger wrote: I need help creating a weighted GPA calculator. I attempted to create one but I was unable to incorporate different leveled classes (College Prep, Honors, and AP). How can I incorporate those leveled classes into the calculator in order to get a final GPA Please always include your code, and some sample data and the output, and explain what isn't as you want it. Past that into your message - this list does not do attachments. Also, please explain terms like "GPA". I imagine it may mean "Grade Point Average", but really it might mean almost anything. On the tutor list we're happy to help with code by expalining problems and suggesting solutions, but we don't write things outright - you need to do that yourself in order to learn. Finally, I put a better description in the Subject: line above. It is useful so that people know what various discussions are about. Cheers, Cameron Simpson ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] GPA calculator
> On Nov 14, 2018, at 14:50, Cameron Simpson wrote: > > On 14Nov2018 13:51, Whitney Eichelberger > wrote: >> I need help creating a weighted GPA calculator. I attempted to create one >> but I was unable to incorporate different leveled classes (College Prep, >> Honors, and AP). How can I incorporate those leveled classes into the >> calculator in order to get a final GPA > > Please always include your code, and some sample data and the output, and > explain what isn't as you want it. > > Past that into your message - this list does not do attachments. > > Also, please explain terms like "GPA". I imagine it may mean "Grade Point > Average", but really it might mean almost anything. One assumes you are talking about school, but regardless, you need to be able to explain your goal before you can code it. For example, do you know how to manually calculate a weighted GPA? Do you know how your school does the weighting (.5 point, full point, etc)? This link has a good overview of the process/steps you probably need to work through. https://www.wikihow.com/Calculate-Your-Weighted-GPA let us know what you have tried so far and where you get stuck, and we will be better able to help. — David Rock da...@graniteweb.com ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Cannot find reference 'bluetoothctl' in 'sh.py' less... (Ctrl+F1)
On Wed, Nov 14, 2018 at 04:15:57PM +0100, srinivasan wrote: [...] > $* pip3 install sh* You don't need to show every command you ran, if they worked successfully. We're volunteers, giving our own efforts for free, not being paid by the hour. The more irrelevant detail you drop in our laps, the fewer of us will bother reading it all. Cut out anything not relevant to your actual problem. Please read this: http://sscce.org/ which will help you debug your own problems, and ask better questions when you can't debug them yourself. However, error messages are important: I missed this the first two times I read your post, and only spotted it by accident: > *$ pip3 install bluetoothctl* > *Collecting bluetoothctl* > * Could not find a version that satisfies the requirement bluetoothctl > (from versions: )* > *No matching distribution found for bluetoothctl* You tried to install a package which doesn't exist, or pip cannot find it. Since it is not installed, naturally later on you will be unable to import that package. > When I try to paste the below code on pycharm and try to point on the word > "bluetoothctl" in the beginning of the line "*from sh import bluetoothctl*" > > *from sh import bluetoothctl* [...] > In the pycharm, I see the below error message : > > *Cannot find reference 'bluetoothctl' in 'sh.py' less... (Ctrl+F1) * > *Inspection info: This inspection detects names that should resolve but > don't. Due to dynamic dispatch and duck typing, this is possible in a > limited but useful number of cases. Top-level and class-level items are > supported better than instance items.* That means that PyCharm's linter cannot find any name called "bluetoothctl" in the sh.py package. Because PyCharm doesn't know what sort of name bluetoothctl is (a subpackage, a module, a class, a function, a variable...) it cannot be more specific than saying it can't resolve the reference. In some (rare!) cases, that's not actually a missing name. It just means that PyCharm isn't clever enough to determine where the name is, but if you actually *run the code* the Python interpreter will find it and the code will run correctly. But this is not the case this time, since you don't actually have bluetoothctl installed, so it cannot be imported. https://duckduckgo.com/?q=bluetoothctl+python -- Steve ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] look back comprehensively
On Tue, Nov 13, 2018 at 11:59:24PM -0500, Avi Gross wrote: > I have been thinking about the thread we have had where the job seemed to be > to read in a log file and if some string was found, process the line before > it and generate some report. Is that generally correct? If that description is correct, then the solution is trivial: iterate over the file, line by line, keeping the previous line: previous_line = None for current_line in file: process(current_line, previous_line) previous_line = current_line No need for complex solutions, or memory-hungry solutions that require reading the entire file into memory at once (okay for, say, a million lines, but not if your logfile is 2GB in size). If you need the line number: previous_line = None for line_num, current_line in enumerate(file, 1): process(current_line, previous_line) previous_line = current_line > Use just the readlines version to get a list of strings representing each > line. That requires reading the entire file into memory at once. That may be acceptable if your file is guaranteed to be small, but since log files can grow big enough to fill hard drives, that might not be a good assumption for serious production-quality scripts. -- Steve ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
