Getting rid of virtual environments with a better dependency system
Hello all, I don't know if this suggestion is missing some point, or it's part of something already proposed before. In a professional environment, we've came to a point in which most people use virtual environments or code environments to avoid "polluting a global environment". However, I think that's a problem with the default behaviour of the module management in Python. A nice default behaviour would be to search for a requirements.txt file in the same directory as __file__, and use the newest version of every module that matches the constraints. If no requirements where given, the newest version already installed could be used. That would require a structure that allows multiple versions of the same module to be downloaded. I already anticipate some problems: increased disk usage for people that are not using virtual environments, the possibility of breaking changes for scripts with no constraints over a module (so that if a different module download a newer version, both would be using it), and of course the hassle of a completely new default behaviour that would require a transition in many codebases. But still, I believe it would pay off in terms of time saved in environment installing and switching. Also, I think it's a good step in the path to integrating pip as something closer to the Python core. What's your opinion, is the effort required too big for the returns? Do you think other problems may arise? -- https://mail.python.org/mailman/listinfo/python-list
Getting rid of virtual environments with a better dependency system
Hello all, I don't know if this suggestion is missing some point, or it's part of something already proposed. In a professional environment, we've came to a point in which most people use virtual environments or conda environments to avoid "polluting a global environment". However, I think that's a problem with the default behaviour of the module management in Python. A nice default behaviour would be to search for a requirements.txt file in the same directory as __file__, and use the newest version of every module that matches the constraints. If no requirements where given, the newest version already installed could be used. That would require allowing multiple versions of the same module to be downloaded. I already anticipate some problems: increased disk usage for people that are not using virtual environments, the possibility of breaking changes for scripts with no constraints over a module (so that if a different module download a newer version, both would be using it), and of course the hassle of a completely new default behaviour that would require a transition in many codebases. That there are other solutions to the problem, such as forcing the usage of semantic versioning, but that's a bit utopic. But still, I believe it would pay off in terms of time saved in environment installing and switching. Also, it's a good step in the path to integrating pip as something closer to the Python core. What's your opinion, is the effort required too big for the returns? Do you think other problems may arise? -- https://mail.python.org/mailman/listinfo/python-list
Re: Getting rid of virtual environments with a better dependency system
On Wed, Nov 11, 2020 at 10:06 PM j c wrote: > > Hello all, > > I don't know if this suggestion is missing some point, or it's part of > something already proposed. > > In a professional environment, we've came to a point in which most people use > virtual environments or conda environments to avoid "polluting a global > environment". > > However, I think that's a problem with the default behaviour of the module > management in Python. A nice default behaviour would be to search for a > requirements.txt file in the same directory as __file__, and use the newest > version of every module that matches the constraints. If no requirements > where given, the newest version already installed could be used. That would > require allowing multiple versions of the same module to be downloaded. > This would stop venvs from providing the isolation that they are supposed to, and instead would just create yet another way to invoke dependency hell. No thank you. A virtual environment isn't just a way to install different versions of modules. It's way WAY more than that, and if you need to have too many different versions around, you have bigger problems to deal with. (As a simple thought experiment to prove the problem with your proposal: what happens with your dependencies' dependencies, and what if they conflict? At what point would that be detected?) ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Problem with rearanging list with paired letters next to each others
On 2020-11-11, MRAB wrote: >> > Points to note in your first code: > > 1. Modifying a list while iterating over it is a bad idea. > > 2. You're modifying the list that you're passing in and also returning > it. That's a bad idea. Either modify it in place or modify and return a > copy. > > 3. The line: > > m = (letter_list.index(x)) > > returns the position of the first occurrence of x. > > Points to note in your second code: > > 1. (See above) > > 2. (See above) > > 3. (See above) > > 4. 'i' goes from 0 to len(letter_list)-1, so i+1 goes from 1 to > len(letter_list), but the maximum index permitted by letter_list is > len(letter_list)-1, hence letter_list[i+1] will raise IndexError on the > last iteration. > > 5. 'i' goes from 0 to len(letter_list)-1, so i-1 goes from -1 to > len(letter_list)-2. letter_list[-1] returns the last (final) letter in > the list, and it's treated as a true. > > 6. This: > > x != letter_list[i+1] and letter_list[i-1] > > is checking whether: > > x != letter_list[i+1] > > is true and also whether: > > letter_list[i-1] > > is true. I see now I overcomplicated it, what is a good idea then? -- https://mail.python.org/mailman/listinfo/python-list
Re: constant folding - why not more
11.11.20 02:46, Skip Montanaro пише: > I can think of two reasons. One, this kind of comparison will almost never > appear in production code (maybe in unit tests?). Unlike the C family of > languages, Python doesn't have a macro processor which would give symbolic > names to numeric constants or string literals. Code generators might > conceivably generate constant comparisons, but they might be able to easily > do constant folding of comparisons themselves. > > Two, given that this sort of construct will almost never be found in the > wild, folding constant comparisons in the compiler would increase the > maintenance burden of the compiler (just slightly, but still...) with no > clear benefit. I concur with Skip. Expressions like 2**32-1 or 1/3 or b'A'[0] are pretty common, so it makes sense to evaluate them at compile time. But comparison and boolean operators are newer used with constants in production code. -- https://mail.python.org/mailman/listinfo/python-list
Need help in installing numpy
Dear Python Community, I am new to python. I sucessfully installed python 3.9 from python.org on my windows 10 pc. I checked it by typing 'py' in the windows cmd prompt and the system indeed responds with the version number, then the >>> prompt, and I can run simple programs without any problem. BUT, if I go back to the cmd prompt and type 'python --version', I simply get the cmd prompt again with no outputs whatsoever. I want to install the 'numpy' package. But when I type on the cmd prompt 'pip install numpy', I get the response:pip : The term 'pip' is not recognized as the name of a cmd let ... etc. I thought that pip is now a part of later versions of python versions 3.x. If so, what did I do wrong? I know this is a very elementary question for this site, but I would appreciate any help. Thanks, -- https://mail.python.org/mailman/listinfo/python-list
Re: Changing strings in files
10.11.20 22:40, Dennis Lee Bieber пише: > Testing for extension in a list of exclusions would be much faster than > scanning the contents of a file, and the few that do get through would have > to be scanned anyway. Then the simplest method should work: read the first 512 bytes and check if they contain b'\0'. Chance that a random sequences of bytes does not contain NUL is (1-1/256)**512 = 0.13. So this will filter out 87% of binary files. Likely6 more, because binary files usually have some structure, and reserve fixed size for integers. Most integers are much less than the maximal value, so higher bits and bytes are zeroes. You can also decrease the probability of false results by increasing the size of tested data or by testing few other byte values (b'\1', b'\2', etc). Anything more sophisticate is just a waste of your time. -- https://mail.python.org/mailman/listinfo/python-list
Re: Changing strings in files
10.11.20 11:07, Manfred Lotz пише: > Perhaps better. I like to use os.scandir this way > > def scantree(path: str) -> Iterator[os.DirEntry[str]]: > """Recursively yield DirEntry objects (no directories) > for a given directory. > """ > for entry in os.scandir(path): > if entry.is_dir(follow_symlinks=False): > yield from scantree(entry.path) > > yield entry > > Worked fine so far. I think I coded it this way because I wanted the > full path of the file the easy way. If this simple code works to you, that's fine, use it. But there are some pitfalls: 1. If you add or remove entries while iterate the directory, the behavior is not specified. It depends on the OS and file system. If you do not modify directories -- all okay. 2. As Cameron said, this method holds open file descriptors for all parent directories. It is usually not problem, because the maximal depth of directories is usually less than the limit of open files. But it can be problem if you process several trees in parallel. The common mistake also to not handle symlinks, but your code does it correctly. -- https://mail.python.org/mailman/listinfo/python-list
Re: Need help in installing numpy
On 2020-11-10 23:32, adelamsaleh--- via Python-list wrote: Dear Python Community, I am new to python. I sucessfully installed python 3.9 from python.org on my windows 10 pc. I checked it by typing 'py' in the windows cmd prompt and the system indeed responds with the version number, then the >>> prompt, and I can run simple programs without any problem. BUT, if I go back to the cmd prompt and type 'python --version', I simply get the cmd prompt again with no outputs whatsoever. I want to install the 'numpy' package. But when I type on the cmd prompt 'pip install numpy', I get the response:pip : The term 'pip' is not recognized as the name of a cmd let ... etc. I thought that pip is now a part of later versions of python versions 3.x. If so, what did I do wrong? I know this is a very elementary question for this site, but I would appreciate any help. Using the Python launcher: py --version and: py -m pip install numpy -- https://mail.python.org/mailman/listinfo/python-list
Re: Getting rid of virtual environments with a better dependency system
On Wed, Nov 11, 2020 at 3:00 AM j c wrote: > Hello all, > > I don't know if this suggestion is missing some point, or it's part of > something already proposed before. > > In a professional environment, we've came to a point in which most people > use virtual environments or code environments to avoid "polluting a global > environment". > I think it'd be a good idea to have a directory (hierarchy) for each python application, and make pip (or similar tool) download to that directory - and then modify the _application's_ sys.path to include that directory at the beginning. This is what I've done with backshift ( https://stromberg.dnsalias.org/~strombrg/backshift/). It works well, without a need for a virtual environment, while still giving dependency isolation. But it's not as automatic as I'd like - I've had to manually specify what dependencies to put in the directory. I use virtual environments sometimes, but I always cringe slightly when doing so. There should be a better way. This is one of the chief benefits of languages like C, C++ and Rust over Python - they don't require you to source something before you can run an app written in them. -- https://mail.python.org/mailman/listinfo/python-list
Re: Problem with rearanging list with paired letters next to each others
On 12/11/2020 00:45, Bischoop wrote: I see now I overcomplicated it, what is a good idea then? This is a variation on a data-compression technique called RLE=Run-Length Encoding (excepting that in this case there is no need to count the repetitions). Web.Ref: https://stackabuse.com/run-length-encoding/ -- Regards =dn -- https://mail.python.org/mailman/listinfo/python-list
Re: Getting rid of virtual environments with a better dependency system
On Thu, Nov 12, 2020 at 4:35 AM Dan Stromberg wrote: > > On Wed, Nov 11, 2020 at 3:00 AM j c wrote: > > > Hello all, > > > > I don't know if this suggestion is missing some point, or it's part of > > something already proposed before. > > > > In a professional environment, we've came to a point in which most people > > use virtual environments or code environments to avoid "polluting a global > > environment". > > > I think it'd be a good idea to have a directory (hierarchy) for each python > application, and make pip (or similar tool) download to that directory - > and then modify the _application's_ sys.path to include that directory at > the beginning. > > This is what I've done with backshift ( > https://stromberg.dnsalias.org/~strombrg/backshift/). It works well, > without a need for a virtual environment, while still giving dependency > isolation. But it's not as automatic as I'd like - I've had to manually > specify what dependencies to put in the directory. > Can you elaborate on exactly how this is different? When you create a venv, it creates some symlinks and such that basically mean you get that - a directory for the application, pip installs into it, and then when you run that Python binary, it'll automatically have sys.path contain the appropriate directory. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Getting rid of virtual environments with a better dependency system
On Wed, Nov 11, 2020 at 10:38 AM Chris Angelico wrote: > On Thu, Nov 12, 2020 at 4:35 AM Dan Stromberg wrote: > > > > On Wed, Nov 11, 2020 at 3:00 AM j c wrote: > > > > > Hello all, > > > > > > I don't know if this suggestion is missing some point, or it's part of > > > something already proposed before. > > > > > > In a professional environment, we've came to a point in which most > people > > > use virtual environments or code environments to avoid "polluting a > global > > > environment". > > > > > I think it'd be a good idea to have a directory (hierarchy) for each > python > > application, and make pip (or similar tool) download to that directory - > > and then modify the _application's_ sys.path to include that directory at > > the beginning. > > > > This is what I've done with backshift ( > > https://stromberg.dnsalias.org/~strombrg/backshift/). It works well, > > without a need for a virtual environment, while still giving dependency > > isolation. But it's not as automatic as I'd like - I've had to manually > > specify what dependencies to put in the directory. > > > > Can you elaborate on exactly how this is different? When you create a > venv, it creates some symlinks and such that basically mean you get > that - a directory for the application, pip installs into it, and then > when you run that Python binary, it'll automatically have sys.path > contain the appropriate directory. > Maybe it's not that different. I do get a .../bin/backshift though, which is a bash script that knows how to start up python on the main module. So the user need not source something at install time or at run time. The install process is ./configure and make install, where ./configure is Not autoconf, but acts sort of like an autoconf script. -- https://mail.python.org/mailman/listinfo/python-list
Re: Getting rid of virtual environments with a better dependency system
On Wednesday, 11 November 2020 at 12:22:24 UTC+1, Chris Angelico wrote: > On Wed, Nov 11, 2020 at 10:06 PM j c wrote: > > > > Hello all, > > > > I don't know if this suggestion is missing some point, or it's part of > > something already proposed. > > > > In a professional environment, we've came to a point in which most people > > use virtual environments or conda environments to avoid "polluting a global > > environment". > > > > However, I think that's a problem with the default behaviour of the module > > management in Python. A nice default behaviour would be to search for a > > requirements.txt file in the same directory as __file__, and use the newest > > version of every module that matches the constraints. If no requirements > > where given, the newest version already installed could be used. That would > > require allowing multiple versions of the same module to be downloaded. > > > This would stop venvs from providing the isolation that they are > supposed to, and instead would just create yet another way to invoke > dependency hell. No thank you. > > A virtual environment isn't just a way to install different versions > of modules. It's way WAY more than that, and if you need to have too > many different versions around, you have bigger problems to deal with. > > (As a simple thought experiment to prove the problem with your > proposal: what happens with your dependencies' dependencies, and what > if they conflict? At what point would that be detected?) > > ChrisA How can this behaviour turn into dependency hell? Every dependency use the specified version if any, otherwise the most recent one, which also applies to second order dependencies. In case of conflict, the first version would be imported, which is currently the default behaviour. The main difference is that this approach would be able to generate a warning before running the script with no need for pipdeptree. -- https://mail.python.org/mailman/listinfo/python-list
Re: Getting rid of virtual environments with a better dependency system
On Thu, Nov 12, 2020 at 10:44 AM Dan Stromberg wrote: > > > On Wed, Nov 11, 2020 at 10:38 AM Chris Angelico wrote: >> >> On Thu, Nov 12, 2020 at 4:35 AM Dan Stromberg wrote: >> > >> > On Wed, Nov 11, 2020 at 3:00 AM j c wrote: >> > >> > > Hello all, >> > > >> > > I don't know if this suggestion is missing some point, or it's part of >> > > something already proposed before. >> > > >> > > In a professional environment, we've came to a point in which most people >> > > use virtual environments or code environments to avoid "polluting a >> > > global >> > > environment". >> > > >> > I think it'd be a good idea to have a directory (hierarchy) for each python >> > application, and make pip (or similar tool) download to that directory - >> > and then modify the _application's_ sys.path to include that directory at >> > the beginning. >> > >> > This is what I've done with backshift ( >> > https://stromberg.dnsalias.org/~strombrg/backshift/). It works well, >> > without a need for a virtual environment, while still giving dependency >> > isolation. But it's not as automatic as I'd like - I've had to manually >> > specify what dependencies to put in the directory. >> > >> >> Can you elaborate on exactly how this is different? When you create a >> venv, it creates some symlinks and such that basically mean you get >> that - a directory for the application, pip installs into it, and then >> when you run that Python binary, it'll automatically have sys.path >> contain the appropriate directory. > > > Maybe it's not that different. > > I do get a .../bin/backshift though, which is a bash script that knows how to > start up python on the main module. So the user need not source something at > install time or at run time. > > The install process is ./configure and make install, where ./configure is Not > autoconf, but acts sort of like an autoconf script. > Ah, fair enough. Did you know that, with a vanilla venv, you can actually just run a Python interpreter from within there, without sourcing anything? Very handy for automated scripts. rosuav@sikorsky:~$ mustard-mine/env/bin/python3 -c 'import sys; print(sys.path)' ['', '/usr/local/lib/python38.zip', '/usr/local/lib/python3.8', '/usr/local/lib/python3.8/lib-dynload', '/home/rosuav/mustard-mine/env/lib/python3.8/site-packages'] ChrisA -- https://mail.python.org/mailman/listinfo/python-list
io.TextIOWrapper.readlines
Hi folks, import io with io.open(filename, ‘r’) as fd: lines = fd.readlines(hint=1000) for line in lines: # do something I was just looking at the io module. io.open(‘file’, ‘r') returns an io.TextIOWrapper object, which has the io.TextIOWrapper.readlines(hint=-1/) method. >>> help(io.TextIOWrapper.readlines) readlines(self, hint=-1, /) Return a list of lines from the stream. hint can be specified to control the number of lines read: no more lines will be read if the total size (in bytes/characters) of all lines so far exceeds hint. I haven’t verified this, but that looks like it isn’t reading the entire file. With hint=1000, the method returns as many complete lines that consume less than 1000 bytes of the stream. I’m lazy. Didn’t test it. Seems like only 1000 bytes would be read from the file, rather than the entire file? The builtin ‘open’ function is defined in the io streams module. I presume the builtin open(‘file’, ‘r’) returns an io.TextIOWrapper object. And maybe the readlines method just isn’t documented? Just curious and lazy. thanks, Karen -- https://mail.python.org/mailman/listinfo/python-list
Re: io.TextIOWrapper.readlines
On 11Nov2020 21:26, Karen Shaeffer wrote:
>import io
>
>with io.open(filename, ‘r’) as fd:
> lines = fd.readlines(hint=1000)
> for line in lines:
> # do something
>
>
>I was just looking at the io module. io.open(‘file’, ‘r') returns an
>io.TextIOWrapper object, which has the io.TextIOWrapper.readlines(hint=-1/)
>method.
>
>
help(io.TextIOWrapper.readlines)
>readlines(self, hint=-1, /)
>Return a list of lines from the stream.
>
>hint can be specified to control the number of lines read: no more
>lines will be read if the total size (in bytes/characters) of all
>lines so far exceeds hint.
>
>I haven’t verified this, but that looks like it isn’t reading the entire file.
>With hint=1000, the method returns as many complete lines that consume less
>than 1000 bytes of the stream. I’m lazy. Didn’t test it. Seems like only 1000
>bytes would be read from the file, rather than the entire file?
Test it. To my eye, I also think that is what it means.
>The builtin ‘open’ function is defined in the io streams module. I presume the
>builtin open(‘file’, ‘r’) returns an io.TextIOWrapper object.
Test it:
>>> open('/dev/null','r')
<_io.TextIOWrapper name='/dev/null' mode='r' encoding='UTF-8'>
> And maybe the readlines method just isn’t documented?
It is documented though.
The docs for io.TextIOWrapper start with this (from the 3.8.0 docs I
have handy):
A buffered text stream over a BufferedIOBase binary stream. It
inherits TextIOBase.
The docs for io.TextIOBase start with this:
Base class for text streams. This class provides a character and
line based interface to stream I/O. It inherits IOBase.
And the docs for IOBase has the readlines method with the signature you
describe from its help.
Nte that if I look at the readlines from TextIOWrapper it says:
>>> open('/dev/null','r').readlines
which probably means TextIOWrapper is a C implementation (for
performance), not doing the plain old class inheritance you'd get from a
pure Python implementation.
Cheers,
Cameron Simpson
--
https://mail.python.org/mailman/listinfo/python-list
