Re: How to run a module before install?

[Peter Otten]

jf...@ms4.hinet.net wrote:

> I had download a module which has the following directory structure:
> 
> [XL-Sudoku-Solver_master]
>   |__[tests]
>   |__[xl_sudoku_solver]
>   |__setup.py   |__ __init__.py
>   |__problem1.txt   |__ __main__.py
>   |__README.md  |__ ...
>   |__ ...
> 
> The setup.py file has something like this:
> -
> ...
> setup(
> name="xl-sudoku-solver",
> version="0.0.1.post3",
> packages=['xl_sudoku_solver'],
> entry_points={
> "console_scripts": [
> "xl-sudoku-solver = xl_sudoku_solver.__main__:main"
> ]
> },
> ...
> 
> 
> and the __main__.py has:
> 
> ...
> from . import Solver, load_from_file, load_from_input, load_from_string
> 
> def main():
> ...
> 
> if __name__ == '__main__':
> main()
> 
> 
> The README.md suggest the way of running it:
> 
> pip install xl-sudoku-solver
> $ xl-sudoku-solver --time -f problem1.txt
> 
> But I like to run it before install. Is it possible? and how? I had try
> two ways below but niether works:
> 
> 1) h:\Temp\XL-Sudoku-Solver-master>py xl_sudoku_solver --time -f
> problem1.txt Traceback (most recent call last):
>   File "C:\Python34\lib\runpy.py", line 170, in _run_module_as_main
> "__main__", mod_spec)
>   File "C:\Python34\lib\runpy.py", line 85, in _run_code
> exec(code, run_globals)
>   File "xl_sudoku_solver\__main__.py", line 5, in 
> from . import Solver, load_from_file, load_from_input,
> load_from_string
> SystemError: Parent module '' not loaded, cannot perform relative import
> 
> 2) h:\Temp\XL-Sudoku-Solver-master>py
> Python 3.4.4 (v3.4.4:737efcadf5a6, Dec 20 2015, 19:28:18) [MSC v.1600 32
> bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for
> more information.
 import xl_sudoku_solver

> 
> Can't find the name "main", What to do?

With h:\Temp\XL-Sudoku-Solver-master as the current working directory try

h:\Temp\XL-Sudoku-Solver-master>py -m xl_sudoku_solver


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to run a module before install?

[jfong]

Peter Otten於 2019年1月14日星期一 UTC+8下午4時04分33秒寫道：
> jf...@ms4.hinet.net wrote:
> 
> > I had download a module which has the following directory structure:
> > 
> > [XL-Sudoku-Solver_master]
> >   |__[tests]
> >   |__[xl_sudoku_solver]
> >   |__setup.py   |__ __init__.py
> >   |__problem1.txt   |__ __main__.py
> >   |__README.md  |__ ...
> >   |__ ...
> > 
> > The setup.py file has something like this:
> > -
> > ...
> > setup(
> > name="xl-sudoku-solver",
> > version="0.0.1.post3",
> > packages=['xl_sudoku_solver'],
> > entry_points={
> > "console_scripts": [
> > "xl-sudoku-solver = xl_sudoku_solver.__main__:main"
> > ]
> > },
> > ...
> > 
> > 
> > and the __main__.py has:
> > 
> > ...
> > from . import Solver, load_from_file, load_from_input, load_from_string
> > 
> > def main():
> > ...
> > 
> > if __name__ == '__main__':
> > main()
> > 
> > 
> > The README.md suggest the way of running it:
> > 
> > pip install xl-sudoku-solver
> > $ xl-sudoku-solver --time -f problem1.txt
> > 
> > But I like to run it before install. Is it possible? and how? I had try
> > two ways below but niether works:
> > 
> > 1) h:\Temp\XL-Sudoku-Solver-master>py xl_sudoku_solver --time -f
> > problem1.txt Traceback (most recent call last):
> >   File "C:\Python34\lib\runpy.py", line 170, in _run_module_as_main
> > "__main__", mod_spec)
> >   File "C:\Python34\lib\runpy.py", line 85, in _run_code
> > exec(code, run_globals)
> >   File "xl_sudoku_solver\__main__.py", line 5, in 
> > from . import Solver, load_from_file, load_from_input,
> > load_from_string
> > SystemError: Parent module '' not loaded, cannot perform relative import
> > 
> > 2) h:\Temp\XL-Sudoku-Solver-master>py
> > Python 3.4.4 (v3.4.4:737efcadf5a6, Dec 20 2015, 19:28:18) [MSC v.1600 32
> > bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for
> > more information.
>  import xl_sudoku_solver
> 
> > 
> > Can't find the name "main", What to do?
> 
> With h:\Temp\XL-Sudoku-Solver-master as the current working directory try
> 
> h:\Temp\XL-Sudoku-Solver-master>py -m xl_sudoku_solver

Great! it works. Thank you, peter.

By the way, can you explain what these two command difference means? with or 
without a "-m". Or pointing where the document is. Thanks ahead:-)

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Email blast management?

[Hartmut Goebel]

Am 09.01.19 um 18:59 schrieb Chris Angelico:
> TBH, I'd recommend using a service like MailChimp, avoiding the entire
> Python question at all.

But for this you need to hand over the addresses to a third-party (the
service provider). This does not respect the privacy of the customers.

-- 
Schönen Gruß
Hartmut Goebel
Dipl.-Informatiker (univ), CISSP, CSSLP, ISO 27001 Lead Implementer
Information Security Management, Security Governance, Secure Software
Development

Goebel Consult, Landshut
http://www.goebel-consult.de

Blog: https://www.goe-con.de/blog/deshalb-tr-069-ausschalten
Kolumne:
https://www.goe-con.de/hartmut-goebel/cissp-gefluester/2010-01-hinterturen-allen-ortes


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Email blast management?

[Chris Angelico]

On Mon, Jan 14, 2019 at 10:39 PM Hartmut Goebel
 wrote:
>
> Am 09.01.19 um 18:59 schrieb Chris Angelico:
> > TBH, I'd recommend using a service like MailChimp, avoiding the entire
> > Python question at all.
>
> But for this you need to hand over the addresses to a third-party (the
> service provider). This does not respect the privacy of the customers.
>

It's a whole lot more respectful than keeping your own database of
email addresses and then having it compromised some day. MailChimp has
a detailed privacy policy [1]; does your handrolled service offer the
same?

ChrisA


[1] https://mailchimp.com/legal/privacy/
-- 
https://mail.python.org/mailman/listinfo/python-list

get the terminal's size

[Alex Ternaute]

Hi there,

I want to know the number of columns of the terminal where python2 writes 
it's outputs.

In a terminal, I type
$ echo $COLUMNS
100

But in Python, os.getenv("COLUMNS") gets nothing.
It gets nothing as well if I try to read the output of "echo $COLUMNS" 
from a subprocess.

I feel that I'm missing something but what ?

Looking on the internet for a hint, I see that python3 has an  
os.get_terminal_size(). 
Please, is there something similar for python2 ?

Cheers
-- 
Alex
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to run a module before install?

[Peter Otten]

jf...@ms4.hinet.net wrote:

>> h:\Temp\XL-Sudoku-Solver-master>py -m xl_sudoku_solver
> 
> Great! it works. Thank you, peter.
> 
> By the way, can you explain what these two command difference means? with
> or without a "-m". Or pointing where the document is. Thanks ahead:-)
> 

https://docs.python.org/dev/using/cmdline.html#cmdoption-m

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: get the terminal's size

[Peter Otten]

Alex Ternaute wrote:

> Hi there,
> 
> I want to know the number of columns of the terminal where python2 writes
> it's outputs.
> 
> In a terminal, I type
> $ echo $COLUMNS
> 100
> 
> But in Python, os.getenv("COLUMNS") gets nothing.
> It gets nothing as well if I try to read the output of "echo $COLUMNS"
> from a subprocess.
> 
> I feel that I'm missing something but what ?

$ python -c 'import os; print os.environ["COLUMNS"]'
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib/python2.7/UserDict.py", line 23, in __getitem__
raise KeyError(key)
KeyError: 'COLUMNS'
$ export COLUMNS
$ python -c 'import os; print os.environ["COLUMNS"]'
157

If you see similar output consider adding

export COLUMNS LINES

to your .bashrc or equivalent.

> Looking on the internet for a hint, I see that python3 has an
> os.get_terminal_size().
> Please, is there something similar for python2 ?
> 
> Cheers


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: get the terminal's size

[Alex Ternaute]

Hi,

Peter Otten :

>> In a terminal, I type $ echo $COLUMNS 100
>> But in Python, os.getenv("COLUMNS") gets nothing.
>> I feel that I'm missing something but what ?

> $ export COLUMNS 

Thank you very much !
-- 
Aelx
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: get the terminal's size

[Thomas Jollans]

On 14/01/2019 12.57, Alex Ternaute wrote:
> Hi there,
> 
> I want to know the number of columns of the terminal where python2 writes 
> it's outputs.
> 
> In a terminal, I type
> $ echo $COLUMNS
> 100
> 
> But in Python, os.getenv("COLUMNS") gets nothing.
> It gets nothing as well if I try to read the output of "echo $COLUMNS" 
> from a subprocess.
> 
> I feel that I'm missing something but what ?
> 
> Looking on the internet for a hint, I see that python3 has an  
> os.get_terminal_size(). 

Use that then.

> Please, is there something similar for python2 ?

I suspect there is some solution in the curses module...

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Email blast management?

[Hartmut Goebel]

Am 14.01.19 um 12:47 schrieb Chris Angelico:
> It's a whole lot more respectful than keeping your own database of
> email addresses and then having it compromised some day.

This assumes that one would not *keep* a list of customers in in company.

-- 
Schönen Gruß
Hartmut Goebel
Dipl.-Informatiker (univ), CISSP, CSSLP, ISO 27001 Lead Implementer
Information Security Management, Security Governance, Secure Software
Development

Goebel Consult, Landshut
http://www.goebel-consult.de

Blog: https://www.goe-con.de/blog/chatsecure-ist-tot-lang-lebe-chatsecure
Kolumne:
https://www.goe-con.de/hartmut-goebel/cissp-gefluester/2011-02-fleisige-datensammler-fur-lukratives-geschaeftsmodell-gesucht


-- 
https://mail.python.org/mailman/listinfo/python-list

python package management confusion

[dcs3spp via Python-list]

Hi,

I am a newbie completely confused with python package management.

I have a setup.py file (listed below) and have setup pip and setup.cfg to 
install my own dependencies  from a local devpi repository.

Can setup.py reference a git repository so that I can install from that url?

Is this possible in newer versions of setuptools? Does anyone have any 
examples? 

Completely confused with managing packages in python


Kind regards

dcs3spp

*** setup.py **

import os
import sys

from setuptools import setup, find_packages

here = os.path.abspath(os.path.dirname(__file__))
with open(os.path.join(here, 'README.md')) as f:
README = f.read()
with open(os.path.join(here, 'CHANGES.md')) as f:
CHANGES = f.read()

requires = [
'cryptography',
'odfpy',
'PyJWT',
'pycrypto',
'pyramid',
'pyramid_core',  # currently installed from devpi...can setup.py use 
git+ssh url like pip requirements.txt file?
'pyramid_debugtoolbar',
'pyramid_tm',
'requests==2.18.4',
'SQLAlchemy',
'transaction',
'zope.sqlalchemy',
'waitress',
'psycopg2-binary',
'python-dateutil',
'uwsgi',
'marshmallow-sqlalchemy',
]

setup_requires = [
'pytest-runner',
]

tests_require = [
'boto3',
'lovely-pytest-docker',
'pytest',
'pytest-cov',
'tuspy',
'WebTest >= 1.3.1',
]

setup(name='api',
  version='0.0',
  description='api',
  long_description=README + '\n\n' + CHANGES,
  classifiers=[
  "Programming Language :: Python",
  "Framework :: Pyramid",
  "Topic :: Internet :: WWW/HTTP",
  "Topic :: Internet :: WWW/HTTP :: WSGI :: Application",
  ],
  author='simon pears',
  author_email='roughlea-mu...@outlook.com',
  url='',
  keywords='web wsgi bfg pylons pyramid',
  packages=find_packages('src'),
  package_dir={'': 'src'},
  include_package_data=True,
  zip_safe=False,
  extras_require={
  'testing': tests_require,
  },
  install_requires=requires,
  setup_requires=setup_requires,
  tests_require=tests_require,
  test_suite='tests',
  entry_points="""\
  [paste.app_factory]
  main = api:main
  [console_scripts]
  initialize_api_db = api.scripts.initializedb:main
  """,
  )

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: get the terminal's size

[Grant Edwards]

On 2019-01-14, Peter Otten <__pete...@web.de> wrote:
>
>> I want to know the number of columns of the terminal where python2 writes
>> it's outputs.
>> 
>> In a terminal, I type
>> $ echo $COLUMNS
>> 100
>> 
>> But in Python, os.getenv("COLUMNS") gets nothing.
>> It gets nothing as well if I try to read the output of "echo $COLUMNS"
>> from a subprocess.
[...]
> If you see similar output consider adding
>
> export COLUMNS LINES
>
> to your .bashrc or equivalent.

That will tell you the terminal size at the time Python was started.

If the terminal size has changed while Python was running, those
environment variables will be wrong.  You need to use the TIOCGWINSZ
ioctl call:

http://www.delorie.com/djgpp/doc/libc/libc_495.html

And to detect the size changes (so you know _when_ you need to do the
above), you need to attach a signal handler for the WINCH signal.

-- 
Grant Edwards   grant.b.edwardsYow! Should I do my BOBBIE
  at   VINTON medley?
  gmail.com

-- 
https://mail.python.org/mailman/listinfo/python-list

ANN: Wing Python IDE 6.1.3 released

[Wingware]

Hi,

We've just released Wing 6.1.3 , 
which improves management of the Python Shell when the project 
environment changes, adds 2FA card selector capability in remote host 
configuration , improves 
support for virtualenv  and 
PEP 8 reformatting , updates the 
How-To for Autodesk Maya , 
improves auto-completion in regex.py and some other third party modules, 
streamlines remote agent installation, and makes about 30 other 
improvements. See the change log 
 for detailsFor 
details, see https://wingware.com/pub/wingide/6.1.3/CHANGELOG.txt


Download Now 

About Wing

Wingware's family of cross-platform 
 Python IDEs make 
Python development easier, with powerful integrated editing, debugging, 
unit testing, and project management features. Wing runs on Windows, 
Linux, and OS X, and can be used to develop any kind of Python code for 
web, desktop, scientific, data analysis, embedded scripting, and other 
applications.


Version 6 introduces many new features, including improved 
multi-selection , 
much easier remote development 
, debugging from the Python 
Shell , recursive 
debugging , PEP 484 
and 526 type hinting 
, PEP 8 
reformatting , support for Python 3.6 
and 3.7, ability to create a new virtualenv 
 from the New Project 
dialog, improved VI mode, support for Vagrant 
, Jupyter 
, Django 
 1.10+ and 2.0, and Windows 
Subsystem for Linux , improved 
support for matplotlib , 
easier Raspberry Pi  
development, optimized debugger, OS X full screen mode, One Dark color 
palette, Russian localization (thanks to Alexandr Dragukin), expanded 
free product line, and much more. For details, see What's New in Wing 
Version 6 .


Wing 6 works with Python versions 2.5 through 2.7 and 3.2 through 3.7, 
including also Anaconda, ActivePython, EPD, Stackless, and others 
derived from the CPython implementation.


Wing Pro 
 requires purchasing 
 or upgrading 
 a license, or obtaining a 30-day 
trial at startup. Wing 101  and 
Wing Personal  are free 
versions that omit some features .


For more product information, please visit wingware.com 



Upgrading

You can try Wing 6 without removing older versions. Wing 6 will read and 
convert your old preferences, settings, and projects. Projects should be 
saved to a new name since previous versions of Wing cannot read Wing 6 
projects.


See also Migrating from Older Versions 
 and Upgrading 
.


Links

Release notice: https://wingware.com/news/2019-01-11
Downloads and Free Trial: https://wingware.com/downloads
Buy: https://wingware.com/store/purchase
Upgrade: https://wingware.com/store/upgrade

Questions?  Don't hesitate to email us at supp...@wingware.com.

Thanks,

--

Stephan Deibel
Wingware | Python IDE

The Intelligent Development Environment for Python Programmers

wingware.com

--
https://mail.python.org/mailman/listinfo/python-list

RE: Python read text file columnwise

[Schachner, Joseph]

About the original question:   If I were you, I would put the 3 numbers into a 
list (or a tuple, if you don't need to modify them) and put this into a 
dictionary.  The key would be the date & time string.

Then, if you need to find a particular entry you can look it up by date and 
time.  But I suspect, since you want column access, you won't need to do that.  
You can iterate through the entries in the dictionary easily and extract the 
data from a column, or from all the columns, if that’s what you want.

for entry in mydict:
value = entry.datalist[0]   # I hope I have the syntax correct


Now, what you do with value is up to you.  I think personally rather than 
building a list I would make a generator function.  A generator uses a "yield" 
statement to return a value, and it waits in that state.  The next time you 
call it it continues and returns the next value.  Kind of useful when the 
alternative is making and passing around huge lists.

--- Joseph S.

-Original Message-
From: DL Neil  
Sent: Saturday, January 12, 2019 4:48 PM
To: python-list@python.org
Subject: Re: Python read text file columnwise

On 12/01/19 1:03 PM, Piet van Oostrum wrote:
> shibashib...@gmail.com writes:
> 
>> Hello
>>>
>>> I'm very new in python. I have a file in the format:
>>>
>>> 2018-05-31  16:00:0028.90   81.77   4.3
>>> 2018-05-31  20:32:0028.17   84.89   4.1
>>> 2018-06-20  04:09:0027.36   88.01   4.8
>>> 2018-06-20  04:15:0027.31   87.09   4.7
>>> 2018-06-28  04.07:0027.87   84.91   5.0
>>> 2018-06-29  00.42:0032.20   104.61  4.8
>>
>> I would like to read this file in python column-wise.
>>
>> I tried this way but not working 
>>event_list = open('seismicity_R023E.txt',"r")
>>  info_event = read(event_list,'%s %s %f %f %f %f\n');


To the OP:

Python's standard I/O is based around data "streams". Whilst there is a concept 
of "lines" and thus an end-of-line character, there is not the idea of a 
record, in the sense of fixed-length fields and thus a defining and distinction 
between data items based upon position.

Accordingly, whilst the formatting specification of strings and floats might 
work for output, there is no equivalent for accepting input data. 
Please re-read refs on file, read, readline, etc.


> Why would you think that this would work?

To the PO:

Because in languages/libraries built around fixed-length files this is 
how one specifies the composition of fields making up a record - a data 
structure which dates back to FORTRAN and Assembler on mainframes and 
other magtape-era machines.

Whilst fixed-length records/files are, by definition, less flexible than 
the more free-form data input Python accepts, they are more efficient 
and faster in situations where the data (format) is entirely consistent 
- such as the OP is describing!


-- 
Regards =dn

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: get the terminal's size

[Peter Otten]

Grant Edwards wrote:

os.environ["COLUMNS"]

> [...] will tell you the terminal size at the time Python was started.

I admit that none of my scripts is ambitious enough to try and track changes 
in terminal size. 

But still, Grant's post prompted me to reread the doc and source of 
shutil.get_terminal_size(), and I think I should warn you that the 
environment variables take precedence over ioctl()-based detection in 
os.get_terminal_size().

I'll keep the exports for now because even for a pipe

$ python3 -c 'import shutil; print(shutil.get_terminal_size())' | cat
os.terminal_size(columns=80, lines=24)
$ export LINES COLUMNS
$ python3 -c 'import shutil; print(shutil.get_terminal_size())' | cat
os.terminal_size(columns=137, lines=42)

the current terminal is usually my final target.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: get the terminal's size

[Grant Edwards]

On 2019-01-14, Peter Otten <__pete...@web.de> wrote:
> Grant Edwards wrote:
>
> os.environ["COLUMNS"]
>
>> [...] will tell you the terminal size at the time Python was started.
>
> I admit that none of my scripts is ambitious enough to try and track
> changes in terminal size.
>
> But still, Grant's post prompted me to reread the doc and source of
> shutil.get_terminal_size(), and I think I should warn you that the
> environment variables take precedence over ioctl()-based detection
> in os.get_terminal_size().

For non-interactive programs, the environment variable approach is
usually good enough.  For interactive programs you use ncurses, slang,
newt, or somesuch library, and they deal with tracking the window size
for you (mostly).

-- 
Grant Edwards   grant.b.edwardsYow! My haircut is totally
  at   traditional!
  gmail.com

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: get the terminal's size

[Alex Ternaute]

Hi Thomas

>> Looking on the internet for a hint, I see that python3 has an
>> os.get_terminal_size().
> Use that then.

Up to now I wanted to keep compatibility with a big bunch of code in 
Python2 that I do no maintain by myself.

Well, I saw that get_terminal_size() follows the windows resizings.
Whenever I consider forking to Python3, this would be my 1st step.
 
>> Please, is there something similar for python2 ?
> I suspect there is some solution in the curses module...

I did  dir(curse) but I could not see if something goes this way.

Bye
-- 
Alex
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: get the terminal's size

[Alex Ternaute]

Hi,

Grant Edwards :

>>export COLUMNS LINES
> That will tell you the terminal size at the time Python was started.

Ok, I think tracking these changes in real time is not worth the work to 
be done using Python2.

I think at last I'll rewrite this (little) programe in Python3 in order to 
use get_terminal_size().

Bye
-- 
Alex
 

-- 
https://mail.python.org/mailman/listinfo/python-list

RE: get the terminal's size

[Schachner, Joseph]

Note sure why you couldn't capture $ echo $COLUMNS from a subprocess call.  
But, how about this (found on the web):

from win32api import GetSystemMetrics
 
print "Width =", GetSystemMetrics(0)
print "Height =", GetSystemMetrics(1)

-Original Message-
From: Alex Ternaute  
Sent: Monday, January 14, 2019 6:58 AM
To: python-list@python.org
Subject: get the terminal's size

Hi there,

I want to know the number of columns of the terminal where python2 writes it's 
outputs.

In a terminal, I type
$ echo $COLUMNS
100

But in Python, os.getenv("COLUMNS") gets nothing.
It gets nothing as well if I try to read the output of "echo $COLUMNS" 
from a subprocess.

I feel that I'm missing something but what ?

Looking on the internet for a hint, I see that python3 has an 
os.get_terminal_size(). 
Please, is there something similar for python2 ?

Cheers
--
Alex

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: get the terminal's size

[Grant Edwards]

On 2019-01-14, Schachner, Joseph  wrote:

> Note sure why you couldn't capture $ echo $COLUMNS from a subprocess
> call.

You can. But, the subprocess is going to inherit the value from the
Python program's environment, so it's just pointless complexity.

-- 
Grant Edwards   grant.b.edwardsYow! I represent a
  at   sardine!!
  gmail.com

-- 
https://mail.python.org/mailman/listinfo/python-list

RE: get the terminal's size

[Schachner, Joseph]

I just tested the fix I proposed, in Python 2.7.13

Code:
from win32api import GetSystemMetrics

def main():
print "Width =", GetSystemMetrics(0)
print "Height =", GetSystemMetrics(1)

if __name__ == '__main__':
main()

Result:
Width = 1536
Height = 864

-Original Message-
From: Alex Ternaute  
Sent: Monday, January 14, 2019 6:58 AM
To: python-list@python.org
Subject: get the terminal's size

Hi there,

I want to know the number of columns of the terminal where python2 writes it's 
outputs.

In a terminal, I type
$ echo $COLUMNS
100

But in Python, os.getenv("COLUMNS") gets nothing.
It gets nothing as well if I try to read the output of "echo $COLUMNS" 
from a subprocess.

I feel that I'm missing something but what ?

Looking on the internet for a hint, I see that python3 has an 
os.get_terminal_size(). 
Please, is there something similar for python2 ?

Cheers
--
Alex

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: get the terminal's size

[Bob van der Poel]

On Mon, Jan 14, 2019 at 4:57 AM Alex Ternaute  wrote:

> Hi there,
>
> I want to know the number of columns of the terminal where python2 writes
> it's outputs.
>
> In a terminal, I type
> $ echo $COLUMNS
> 100
>
> But in Python, os.getenv("COLUMNS") gets nothing.
> It gets nothing as well if I try to read the output of "echo $COLUMNS"
> from a subprocess.
>
> I feel that I'm missing something but what ?
>
> Looking on the internet for a hint, I see that python3 has an
> os.get_terminal_size().
> Please, is there something similar for python2 ?
>
> Cheers
> --
> Alex
> --
> https://mail.python.org/mailman/listinfo/python-list
>

try this:


http://stackoverflow.com/questions/566746/how-to-get-console-window-width-in-python


-- 

 Listen to my FREE CD at http://www.mellowood.ca/music/cedars 
Bob van der Poel ** Wynndel, British Columbia, CANADA **
EMAIL: b...@mellowood.ca
WWW:   http://www.mellowood.ca
-- 
https://mail.python.org/mailman/listinfo/python-list

Tracemalloc overhead when profiling

[Juris __]

Hi,

I was looking for a way to profile memory usage for some script which 
deals with log message parsing. Looking through Python's stdlib I 
stumbled upon tracemalloc module. So I tried my hand on profiling my 
script. A few things I noticed that I am not 100% sure I can explain.

Tracemalloc memory overhead when tracing seems somewhere 3x-4x. Is that 
expected? The dumb example that demonstrates behavior:

---8<---
# memprof.py
import tracemalloc

def expensive():
 return [str(x) for x in range(1_000_000)]

if __name__ == '__main__':

 if not tracemalloc.is_tracing():
 tracemalloc.start()

 snapshot1 = tracemalloc.take_snapshot()

 _ = expensive()

 snapshot2 = tracemalloc.take_snapshot()
 tracemalloc.stop()

 for stat in snapshot2.compare_to(snapshot1, key_type="lineno"):
 print(stat)
---8<---


Script output with naive GNU time program profiling:

$ /usr/bin/time python3.7 memprof.py
memprof.py:6: size=60.6 MiB (+60.6 MiB), count=101 (+101), 
average=64 B
...snip...
1.40user 0.10system 0:01.51elapsed 99%CPU (0avgtext+0avgdata 
280284maxresident)k
0inputs+0outputs (0major+62801minor)pagefaults 0swaps


Same script but without actually tracing with tracemalloc:

$ /usr/bin/time python3.7 memprof.py
0.26user 0.03system 0:00.29elapsed 100%CPU (0avgtext+0avgdata 
72316maxresident)k
0inputs+0outputs (0major+17046minor)pagefaults 0swaps


So, when not tracing with tracemalloc memory used by script is 72MiB 
(credible since tracemalloc reports 60.6MiB allocated in hot spot). But 
then when tracemalloc is tracing script uses almost 4x memory e.g. 280MiB.

Is this expected? Any other tools for memory profiling you can recommend?

Running Python 3.7.2 on x86_64 Linux system.

BR,
Juris
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Email blast management?

[Chris Angelico]

On Tue, Jan 15, 2019 at 12:53 AM Hartmut Goebel
 wrote:
>
> Am 14.01.19 um 12:47 schrieb Chris Angelico:
> > It's a whole lot more respectful than keeping your own database of
> > email addresses and then having it compromised some day.
>
> This assumes that one would not *keep* a list of customers in in company.
>

Honestly, I have no idea what you're moaning about. We don't have
nearly enough details here to say what would be _the best_ way to do
things, but I stand by my statement that services like MailChimp and
Mailman are worth looking into.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

ANN: Creating GUI Applications with wxPython

[Mike Driscoll]

Hi,

I just thought I would let you all know that I am working on my 2nd wxPython 
book, "Creating GUI Applications with wxPython". This one will be about 
actually creating small runnable applications instead of just recipes like my 
Cookbook did. I hope to have 8-10 working applications included with the book.

You can read more about it here if you are interested: 
https://www.blog.pythonlibrary.org/2019/01/14/creating-gui-applications-with-wxpython-kickstarter/

Feel free to ask me questions about it too.

Thanks,
Mike
-- 
https://mail.python.org/mailman/listinfo/python-list

AssertionError without traceback?

[Israel Brewster]

I have a flask application deployed on CentOS 7 using Python 3.6.7 and uwsgi 
2.0.17.1, proxied behind nginx. uwsgi is configured to listed on a socket in 
/tmp. The app uses gevent and the flask_uwsgi_websockets plugin as well as 
various other third-party modules, all installed via pip in a virtualenv. The 
environment was set up using pip just a couple of days ago, so everything 
should be fully up-to-date. The application *appears* to be running properly 
(it is in moderate use and there have been no reports of issues, nor has my 
testing turned up any problems), however I keep getting entries like the 
following in the error log:

AssertionError
2019-01-14T19:16:32Z  failed with 
AssertionError

There is no additional information provided, just that. I was running the same 
app (checked out from a GIT repository, so exact same code) on CentOS 6 for 
years without issue, it was only since I moved to CentOS 7 that I've seen the 
errors. I have not so far been able to correlate this error with any specific 
request. Has anyone seen anything like this before such that you can give me 
some pointers to fixing this? As the application *appears* to be functioning 
normally, it may not be a big issue, but it has locked up once since the move 
(no errors in the log, just not responding on the socket), so I am a bit 
concerned.
---
Israel Brewster
Systems Analyst II
5245 Airport Industrial Rd
Fairbanks, AK 99709
(907) 450-7293
---

[cid:05a3a602-0c27-4749-91b8-096a5857d984@flyravn.com]



[cid:bbc82752-6db4-44cf-b919-421ed304e1d1@flyravn.com]







-- 
https://mail.python.org/mailman/listinfo/python-list

sampling from frequency distribution / histogram without replacement

[duncan smith]

Hello,
  Just checking to see if anyone has attacked this problem before
for cases where the population size is unfeasibly large. i.e. The number
of categories is manageable, but the sum of the frequencies, N,
precludes simple solutions such as creating a list, shuffling it and
using the first n items to populate the sample (frequency distribution /
histogram).

I note that numpy.random.hypergeometric will allow me to generate a
sample when I only have two categories, and that I could probably
implement some kind of iterative / partitioning approach calling this
repeatedly. But before I do I thought I'd ask if anyone has tackled this
before. Can't find much on the web. Cheers.

Duncan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: get the terminal's size

[Cameron Simpson]

On 14Jan2019 17:16, Alex Ternaute  wrote:

Looking on the internet for a hint, I see that python3 has an
os.get_terminal_size().

Use that then.


Up to now I wanted to keep compatibility with a big bunch of code in
Python2 that I do no maintain by myself.

Well, I saw that get_terminal_size() follows the windows resizings.
Whenever I consider forking to Python3, this would be my 1st step.


Please, is there something similar for python2 ?

I suspect there is some solution in the curses module...


I did  dir(curse) but I could not see if something goes this way.


My cs.tty module (on PyPI) has a ttysize function:

 https://pypi.org/project/cs.tty/

which just parses the output of the stty command.

Personally I resist using the environment variables; they're (a) not 
exports by default because they're "live" and (b) then don't track 
changes if they are exported and (c) rely on the shell providing them. I 
just don't trust them.


If you don't want the cs.tty module, the ttysize code is just this:

   WinSize = namedtuple('WinSize', 'rows columns')

   def ttysize(fd):
 ''' Return a (rows, columns) tuple for the specified file descriptor.

 If the window size cannot be determined, None will be returned
 for either or both of rows and columns.

 This function relies on the UNIX `stty` command.
 '''
 if not isinstance(fd, int):
   fd = fd.fileno()
 P = Popen(['stty', '-a'], stdin=fd, stdout=PIPE, universal_newlines=True)
 stty = P.stdout.read()
 xit = P.wait()
 if xit != 0:
   return None
 m = re.compile(r' rows (\d+); columns (\d+)').search(stty)
 if m:
   rows, columns = int(m.group(1)), int(m.group(2))
 else:
   m = re.compile(r' (\d+) rows; (\d+) columns').search(stty)
   if m:
 rows, columns = int(m.group(1)), int(m.group(2))
   else:
 rows, columns = None, None
 return WinSize(rows, columns)

Hope this helps.

Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

Re: sampling from frequency distribution / histogram without replacement

[Gregory Ewing]

duncan smith wrote:

Hello,
  Just checking to see if anyone has attacked this problem before
for cases where the population size is unfeasibly large.


The fastest way I know of is to create a list of cumulative
frequencies, then generate uniformly distributed numbers and
use a binary search to find where they fall in the list.
That's O(log n) per sample in the size of the list once it's
been set up.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: get the terminal's size

[John Doe]

On 2019-01-14, Bob van der Poel  wrote:
> try this:
>
>
> http://stackoverflow.com/questions/566746/how-to-get-console-window-width-in-python
>

and have a look at this one too:
https://stackoverflow.com/questions/1396820/apt-like-column-output-python-library/1446973#1446973
>
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: Email blast management?

[Avi Gross]

There is an old saying about getting what you paid for. Python can be free
but applications have costs.

Chris makes some valid points when saying there are existing solutions that
may be worth considering.

If someone wants to know about commercial products that do approximately
what they need and provide some level of service, they can buy it. If they
want it for free, they can create whatever they want using their own code
augmented by any free code they are legally allowed to incorporate.

But free is often not quite free. Look at the complaints about Google and
Facebook who try to make a buck (or a billion) by looking at the data you
generate and helping target advertisers or worse? Look at how the data
sometimes gets out, legally or otherwise? Some people now choose to switch
browsers after they keep seeing ads follow them everywhere for items they
once LOOKED AT and did not even buy.

Is there absolute security if you roll your own app, using python or
anything else? I seriously doubt it. Python probably is not ideal in the
sense that your source code is often readable if someone breaks into your
machine. If you were to say encrypt/decrypt some things so items are never
in plaintext on disk, your code may allow them to see what methods are used
and, if you are careless enough, may even expose the key you use.

I was involved in creating and maintaining a fairly early email product
targeted at businesses quite a few years ago. Part of my responsibility at
one point was to READ a subclass of the mail. Messages that made it into the
system but ran into problems would often end up in a junkmail category and
we needed to examine them to find out what went wrong and file modification
requests. In addition, if we could figure out how to "fix" a message and
sent it onward for delivery, we tried. An example of such an error was when
we added a heterogeneous set of machines in the worldwide network of
different types, a message that fit in memory on one might fail when passed
to/through another kind. We had to adjust the maximum size allowed to what
would fit anywhere.

The point is that someone like me with the root password could read
ANYTHING. All logs were equally available so making a list of all email
addresses or a graph showing communication chains was possible. Many
applications may be vulnerable to just one bad employee given such access.
Some may be vulnerable if just one machine in an intranet is corrupted in
some way. Again, this is not just about email but credit card use, browsing
history, etc.

But I suggest that unless you hire very experienced people to roll your own,
you risk being even less secure than with a more trusted commercial product.
Of course, if you want truly unique features, that may be a reason to have
your own.

The above is some thoughts and is not to be attacked as a suggestion to
waste money buying specific products. I am not selling anything just
reflecting on a wider issue. There seems to be very little absolute safety
in the cyber world just as there really isn't in the physical world. You
take your chances but hopefully with some idea of the risks. Someday
anything you encrypt today may become easily readable unless you use fairly
elaborate one-time pads.


-Original Message-
From: Python-list  On
Behalf Of Chris Angelico
Sent: Monday, January 14, 2019 10:25 AM
To: Python 
Subject: Re: Email blast management?

On Tue, Jan 15, 2019 at 12:53 AM Hartmut Goebel 
wrote:
>
> Am 14.01.19 um 12:47 schrieb Chris Angelico:
> > It's a whole lot more respectful than keeping your own database of 
> > email addresses and then having it compromised some day.
>
> This assumes that one would not *keep* a list of customers in in company.
>

Honestly, I have no idea what you're moaning about. We don't have nearly
enough details here to say what would be _the best_ way to do things, but I
stand by my statement that services like MailChimp and Mailman are worth
looking into.

ChrisA
--
https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: sampling from frequency distribution / histogram without replacement

[duncan smith]

On 14/01/2019 22:59, Gregory Ewing wrote:
> duncan smith wrote:
>> Hello,
>>   Just checking to see if anyone has attacked this problem before
>> for cases where the population size is unfeasibly large.
> 
> The fastest way I know of is to create a list of cumulative
> frequencies, then generate uniformly distributed numbers and
> use a binary search to find where they fall in the list.
> That's O(log n) per sample in the size of the list once it's
> been set up.
> 

That's the sort of thing I've been thinking about. But once I'd found
the relevant category I'd need to reduce its frequency by 1 and
correspondingly update the cumulative frequencies. Alternatively, I
could add an extra step where I selected a unit from the relevant
category with probability equal to the proportion of non-sampled units
from the category. I could maybe set up an alias table and do something
similar.

The other thing I was thinking about was iterating through the
categories (ideally from largest frequency to smallest frequency),
generating the numbers to be sampled from the current category and the
remaining categories (using numpy.random.hypergeometric). With a few
large frequencies and lots of small frequencies that could be quite
quick (on average). Alternatively I could partition the categories into
two sets, generate the number to be sampled from each partition, then
partition the partitions etc. binary search style.

I suppose I'll try the both the alias table + rejection step and the
recursive partitioning approach and see how they turn out. Cheers.

Duncan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: get the terminal's size

[eryk sun]

On 1/14/19, Schachner, Joseph  wrote:
> I just tested the fix I proposed, in Python 2.7.13
>
> Code:
> from win32api import GetSystemMetrics
>
> def main():
> print "Width =", GetSystemMetrics(0)
> print "Height =", GetSystemMetrics(1)

That gets the monitor size, i.e:

SM_CXSCREEN (0)
The width of the screen of the primary display monitor, in pixels.

SM_CYSCREEN (1)
The height of the screen of the primary display monitor, in pixels.

The console's visible window is a rectangular view on its active
screen buffer. We have to query the screen-buffer information to
obtain the coordinates of this rectangle (right, left, bottom, top).
Python's os.get_terminal_size does this, but I dislike the fact that
it uses the process standard handles instead of C file descriptors and
only supports 0, 1, and 2.

Here's a version that defaults to opening the console's active screen
buffer, "CONOUT$". Otherwise it uses msvcrt.get_osfhandle to get the
handle for the fd argument instead of hard-mapping 0, 1, and 2 to the
process standard handles. For symmetry, I've also added POSIX code
that switches the default to opening "/dev/tty" instead of using
stdout.

import os
import ctypes
from collections import namedtuple

terminal_size = namedtuple('terminal_size', 'columns lines')

if os.name == 'posix':

import tty
import fcntl

class winsize(ctypes.Structure):
_fields_ = (('ws_row', ctypes.c_ushort),
('ws_col', ctypes.c_ushort),
('ws_xpixel', ctypes.c_ushort),
('ws_ypixel', ctypes.c_ushort))

def get_terminal_size(fd=None):
"""Return the size of the terminal window as (columns, lines).

The optional argument fd specifies which file descriptor should
be queried. An OSError is raised if the file descriptor is not
connected to a terminal (Unix) or a console screen buffer
(Windows). The default behavior is to open and query the process
controlling terminal (i.e. Unix "/dev/tty") or active console
screen buffer (i.e. Windows "CONOUT$").
"""
w = winsize()
if fd is None:
fd_used = os.open('/dev/tty', os.O_RDWR)
else:
fd_used = fd
try:
fcntl.ioctl(fd_used, tty.TIOCGWINSZ, w)
finally:
if fd is None:
os.close(fd_used)
return terminal_size(w.ws_col, w.ws_row)

elif os.name == 'nt':

import msvcrt
from ctypes import wintypes
kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)

class CONSOLE_SCREEN_BUFFER_INFO(ctypes.Structure):
_fields_ = (('dwSize', wintypes._COORD),
('dwCursorPosition', wintypes._COORD),
('wAttributes', wintypes.WORD),
('srWindow', wintypes.SMALL_RECT),
('dwMaximumWindowSize', wintypes._COORD))

kernel32.GetConsoleScreenBufferInfo.argtypes = (
wintypes.HANDLE, ctypes.POINTER(CONSOLE_SCREEN_BUFFER_INFO))

def get_terminal_size(fd=None):
"""Return the size of the terminal window as (columns, lines).

The optional argument fd specifies which file descriptor should
be queried. An OSError is raised if the file descriptor is not
connected to a terminal (Unix) or a console screen buffer
(Windows). The default behavior is to open and query the process
controlling terminal (i.e. Unix "/dev/tty") or active console
screen buffer (i.e. Windows "CONOUT$").
"""
csbi = CONSOLE_SCREEN_BUFFER_INFO()
w = csbi.srWindow
if fd is None:
fd_used = os.open('CONOUT$', os.O_RDWR)
else:
fd_used = fd
try:
h = msvcrt.get_osfhandle(fd_used)
if not kernel32.GetConsoleScreenBufferInfo(h,
ctypes.byref(csbi)):
raise ctypes.WinError(ctypes.get_last_error())
finally:
if fd is None:
os.close(fd_used)
return terminal_size(w.Right - w.Left + 1, w.Bottom - w.Top + 1)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: sampling from frequency distribution / histogram without replacement

[Spencer Graves]

On 2019-01-14 18:40, duncan smith wrote:

On 14/01/2019 22:59, Gregory Ewing wrote:

duncan smith wrote:

Hello,
   Just checking to see if anyone has attacked this problem before
for cases where the population size is unfeasibly large.

The fastest way I know of is to create a list of cumulative
frequencies, then generate uniformly distributed numbers and
use a binary search to find where they fall in the list.
That's O(log n) per sample in the size of the list once it's
been set up.


That's the sort of thing I've been thinking about. But once I'd found
the relevant category I'd need to reduce its frequency by 1 and
correspondingly update the cumulative frequencies. Alternatively, I
could add an extra step where I selected a unit from the relevant
category with probability equal to the proportion of non-sampled units
from the category. I could maybe set up an alias table and do something
similar.

The other thing I was thinking about was iterating through the
categories (ideally from largest frequency to smallest frequency),
generating the numbers to be sampled from the current category and the
remaining categories (using numpy.random.hypergeometric). With a few
large frequencies and lots of small frequencies that could be quite
quick (on average). Alternatively I could partition the categories into
two sets, generate the number to be sampled from each partition, then
partition the partitions etc. binary search style.

I suppose I'll try the both the alias table + rejection step and the
recursive partitioning approach and see how they turn out. Cheers.



  R has functions "sample" and "sample.int";  see 
"https://www.rdocumentation.org/packages/base/versions/3.5.2/topics/sample";. 
You can call R from Python, 
"https://sites.google.com/site/aslugsguidetopython/data-analysis/pandas/calling-r-from-python";. 




  These are in the "base" package.  I believe they have been an 
important part of the base R language almost since its inception and 
have been used extensively.  You'd have to work really hard to do 
better, in my judgment.



      Spencer Graves


DISCLAIMER:  I'm primarily an R guy and only use Python when I can't 
find a sensible way to do what I want in R.


Duncan


--
https://mail.python.org/mailman/listinfo/python-list

3 random numbers

[caigy84]

So I was given this question to be solved in Python 3 : Pick any 3 random 
ascending numbers and write out a loop function that prints out all 3 numbers. 
This was the code and solution presented to me. Can anyone understand it and 
explain it to me please ? I have looked at it but cannot see how it was derived 
and why the correct solution was printed. Thanks alot !

# any 3 ascending numbers , counter must start at 0. 
# 400 467 851
i = 0
x = 400
while x < 852:
print(x)
if i > 0:
x = x + ((i + 4) * 67) + (i * 49)
else:
x = x + 67
i = i + 1
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: 3 random numbers

[Spencer Graves]

On 2019-01-14 23:29, caig...@gmail.com wrote:

So I was given this question to be solved in Python 3 : Pick any 3 random 
ascending numbers and write out a loop function that prints out all 3 numbers. 
This was the code and solution presented to me. Can anyone understand it and 
explain it to me please ? I have looked at it but cannot see how it was derived 
and why the correct solution was printed. Thanks alot !

# any 3 ascending numbers , counter must start at 0.
# 400 467 851
i = 0
x = 400
while x < 852:
print(x)
if i > 0:
x = x + ((i + 4) * 67) + (i * 49)
else:
x = x + 67
i = i + 1



  This sounds like a homework problem for a class.  I don't know 
how this list treats such questions, but I suspect answering such 
questions may be discouraged.



  Hint:  Read the documentation on "while" and then trace the 
iterations.



      Spencer



--
https://mail.python.org/mailman/listinfo/python-list

Re: 3 random numbers

[Cameron Simpson]

On 14Jan2019 21:29, caig...@gmail.com  wrote:

So I was given this question to be solved in Python 3 : Pick any
3 random ascending numbers and write out a loop function that prints
out all 3 numbers. This was the code and solution presented to me.
Can anyone understand it and explain it to me please ? I have looked
at it but cannot see how it was derived and why the correct solution
was printed. Thanks alot !


Indentation is vital in Python; it is used to delineate the start and 
end of loops and control structures etc (where other languages might use 
curly braces). So because your pasted code has no indentation we must 
guess. I'm going to reindent to express what I'm guessing:


   # any 3 ascending numbers , counter must start at 0.
   # 400 467 851
   i = 0
   x = 400
   while x < 852:
 print(x)
 if i > 0:
   x = x + ((i + 4) * 67) + (i * 49)
 else:
   x = x + 67
 i = i + 1

The other thing is that I do not think you have described the problem 
correctly, or if that _is_ how it was expressed to you then it is a 
terrible terrible problem description.


All the code above does is (a) start "x" at 400, which is your first 
arbitrary value and (b) stop the loop when "x" >= 852, which _prevents_ 
it printing any numbers above your last arbitrary value (851).


The bit in the middle is just contrived.

The important things about the code seem to be:

"x" is always increased. This ensures that the loop will finish, because 
"x" starts below the limit (852) and always gets closer to it, and 
eventually exceeds it ==> loop exits.


The first pass through the loop when "x" == 0 just adds 67 to it, 
getting from 400 to 467, your second arbitrary number.


So this isn't some clever bit of code that does something to generate 3 
values, it is a loop that runs three times to get exactly the 3 values 
in your comment.


I suspect we're missing the larger picture here.

Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

Re: python package management confusion

[dieter]

dcs3spp via Python-list  writes:
> I am a newbie completely confused with python package management.
>
> I have a setup.py file (listed below) and have setup pip and setup.cfg to 
> install my own dependencies  from a local devpi repository.
>
> Can setup.py reference a git repository so that I can install from that url?

I doubt it:
A primary goal of the Python package management is to allow users
to easily install prepackaged components (published in a repository
like PyPI) - not to integrate transparently with source code control
systems.

I approach development of Python components by checking them out
of the source code control system and then use "setuptool"'s
"python setup.py develop" in their source tree to integrate
the component in a virtual env for testing and developping purposes.

Another option (I also use occasioanally) would be to use
"zc.buildout". By itself, it does not integrate with "git"
but there is an extension (I think "mr.developer" or similarly spelled)
which allows to directly reference "git" repositories as
source for so called "development eggs").


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Tracemalloc overhead when profiling

[dieter]

Juris __  writes:
> I was looking for a way to profile memory usage for some script which 
> deals with log message parsing. Looking through Python's stdlib I 
> stumbled upon tracemalloc module. So I tried my hand on profiling my 
> script. A few things I noticed that I am not 100% sure I can explain.
>
> Tracemalloc memory overhead when tracing seems somewhere 3x-4x. Is that 
> expected?

Yes.

One of the API functions of "tracemalloc" is "get_object_traceback"
which allows you to ask for each allocated Python object where
is has been allocated. Your example below constructs a large
number of small objects and the "where allocated" information for
each of those objects is far bigger than the object itself.

> The dumb example that demonstrates behavior:
>
> ---8<---
> # memprof.py
> import tracemalloc
>
> def expensive():
>  return [str(x) for x in range(1_000_000)]
>
> if __name__ == '__main__':
>
>  if not tracemalloc.is_tracing():
>  tracemalloc.start()
>
>  snapshot1 = tracemalloc.take_snapshot()
>
>  _ = expensive()
>
>  snapshot2 = tracemalloc.take_snapshot()
>  tracemalloc.stop()
>
>  for stat in snapshot2.compare_to(snapshot1, key_type="lineno"):
>  print(stat)
> ---8<---
>
>
> Script output with naive GNU time program profiling:
>
> $ /usr/bin/time python3.7 memprof.py
> memprof.py:6: size=60.6 MiB (+60.6 MiB), count=101 (+101), 
> average=64 B
> ...snip...
> 1.40user 0.10system 0:01.51elapsed 99%CPU (0avgtext+0avgdata 
> 280284maxresident)k
> 0inputs+0outputs (0major+62801minor)pagefaults 0swaps
>
>
> Same script but without actually tracing with tracemalloc:
>
> $ /usr/bin/time python3.7 memprof.py
> 0.26user 0.03system 0:00.29elapsed 100%CPU (0avgtext+0avgdata 
> 72316maxresident)k
> 0inputs+0outputs (0major+17046minor)pagefaults 0swaps
>
>
> So, when not tracing with tracemalloc memory used by script is 72MiB 
> (credible since tracemalloc reports 60.6MiB allocated in hot spot). But 
> then when tracemalloc is tracing script uses almost 4x memory e.g. 280MiB.
>
> Is this expected? Any other tools for memory profiling you can recommend?
>
> Running Python 3.7.2 on x86_64 Linux system.
>
> BR,
> Juris

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: AssertionError without traceback?

[dieter]

Israel Brewster  writes:
> I have a flask application deployed on CentOS 7 using Python 3.6.7 and uwsgi 
> 2.0.17.1, proxied behind nginx. uwsgi is configured to listed on a socket in 
> /tmp. The app uses gevent and the flask_uwsgi_websockets plugin as well as 
> various other third-party modules, all installed via pip in a virtualenv. The 
> environment was set up using pip just a couple of days ago, so everything 
> should be fully up-to-date. The application *appears* to be running properly 
> (it is in moderate use and there have been no reports of issues, nor has my 
> testing turned up any problems), however I keep getting entries like the 
> following in the error log:
>
> AssertionError
> 2019-01-14T19:16:32Z  failed with 
> AssertionError

I would try to find out where the log message above has been generated
and ensure it does not only log the information above but also the
associated traceback.

I assume that the log comes from some framework -- maybe "uwsgi"
or "gevent". It is a weakness to log exceptions without the
associated traceback.

> There is no additional information provided, just that. I was running the 
> same app (checked out from a GIT repository, so exact same code) on CentOS 6 
> for years without issue, it was only since I moved to CentOS 7 that I've seen 
> the errors. I have not so far been able to correlate this error with any 
> specific request. Has anyone seen anything like this before such that you can 
> give me some pointers to fixing this? As the application *appears* to be 
> functioning normally, it may not be a big issue, but it has locked up once 
> since the move (no errors in the log, just not responding on the socket), so 
> I am a bit concerned.
> ---
> Israel Brewster
> Systems Analyst II
> 5245 Airport Industrial Rd
> Fairbanks, AK 99709
> (907) 450-7293
> ---
>
> [cid:05a3a602-0c27-4749-91b8-096a5857d984@flyravn.com]
>
>
>
> [cid:bbc82752-6db4-44cf-b919-421ed304e1d1@flyravn.com]

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: python package management confusion

[Chris Angelico]

On Tue, Jan 15, 2019 at 6:18 PM dieter  wrote:
>
> dcs3spp via Python-list  writes:
> > I am a newbie completely confused with python package management.
> >
> > I have a setup.py file (listed below) and have setup pip and setup.cfg to 
> > install my own dependencies  from a local devpi repository.
> >
> > Can setup.py reference a git repository so that I can install from that url?
>
> I doubt it:
> A primary goal of the Python package management is to allow users
> to easily install prepackaged components (published in a repository
> like PyPI) - not to integrate transparently with source code control
> systems.

You can use pip to install from a git repo, but I don't know the details.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list