Re: Suggestions on mechanism or existing code - maintain persistence of file download history
jkn, > MRAB's scheme does have the disadvantages to me that Chris has pointed > out. Nothing that can't be countered by keeping copies of the last X number of to-be-dowloaded-URLs files. As for rewriting every time, you will /have/ to write something for every action (and flush the file!), if you think you should be able to ctrl-c (or worse) out of the program. But, you could opt to write this sessions successfully downloaded URLs to a seperate file, and only merge that with the origional one program start. That together with an integrity check of the seperate file (eventually on a line-by-line (URL) basis) should make the origional files corruption rather unlikely. A database /sounds/ good, but what happens when you ctrl-c outof a non-atomic operation ? How do you fix that ?IOW: Databases can be corrupted for pretty-much the same reason as for a simple datafile (but with much worse consequences). Also think of the old adagio: "I had a problem, and than I thought I could use X. Now I have two problems..." - with X traditionally being "regular expressions". In other words: do KISS (keep it ) By the way: The "just write the URLs in a folder" method is not at all a bad one. /Very/ easy to maintain, resilent (especially when you consider the self-repairing capabilities of some filesystems) and the polar opposite of a "customer lock-in". :-) Regards, Rudy Wieser -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestions on mechanism or existing code - maintain persistence of file download history
On Thu, Jan 30, 2020 at 7:41 PM R.Wieser wrote: > A database /sounds/ good, but what happens when you ctrl-c outof a > non-atomic operation ? How do you fix that ?IOW: Databases can be > corrupted for pretty-much the same reason as for a simple datafile (but with > much worse consequences). Uhh Proper databases don't HAVE non-atomic operations. That's kinda their job. Unless you mean that there's a non-atomic operation that consists of placing some kind of "claim" on a URL and then going and downloading it, and finally reporting completion, but that's easily enough handled by simply ensuring that there are no downloader processes running, and then clear any incomplete claims. But that's still not corrupting the database. Maybe you've only ever worked with half-baked apologies for database systems? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestions on mechanism or existing code - maintain persistence of file download history
On Thu, Jan 30, 2020 at 7:41 PM R.Wieser wrote: > Also think of the old adagio: BTW, the word you want here is "adage", unless you mean that it's a piece of music being played slowly :) ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestions on mechanism or existing code - maintain persistence of file download history
Chris, > Uhh > > Proper databases don't HAVE non-atomic operations. That's kinda their job. Uhh... yes, /singular/ operations are considered to be atomic. A series of operations /ment/ to be executed as a single one on the other hand aren't. > Unless you mean that there's a non-atomic operation that consists of [snip] >> Also think of the old adagio: "I had a problem, I guess that that went right over your head. :-)/You/ might know exactly what should and shouldn't be done, what makes you think the OP currently does ? > But that's still not corrupting the database. Depending on your definition of corruption. An unreadable file is often described as being corrupt, though the same can be said of a database in an inconsistent state. Regards, Rudy Wieser >> Also think of the old adagio: > > BTW, the word you want here is "adage", unless you mean that > it's a piece of music being played slowly :) Thanks. -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestions on mechanism or existing code - maintain persistence of file download history
On Thu, Jan 30, 2020 at 8:36 PM R.Wieser wrote: > > Chris, > > > Uhh > > > > Proper databases don't HAVE non-atomic operations. That's kinda their job. > > Uhh... yes, /singular/ operations are considered to be atomic. A series of > operations /ment/ to be executed as a single one on the other hand aren't. That's what transactions are for. If you have a series of operations meant to be executed as an atomic operation, you begin a transaction, do the operations, and then commit. Again, that is the *job* of the database. > >> Also think of the old adagio: "I had a problem, > > I guess that that went right over your head. :-)/You/ might know exactly > what should and shouldn't be done, what makes you think the OP currently > does ? > > > But that's still not corrupting the database. > > Depending on your definition of corruption. An unreadable file is often > described as being corrupt, though the same can be said of a database in an > inconsistent state. > If the OP doesn't know how to use a database, that doesn't change my recommendations regarding the use of a database. I don't understand why you're denigrating databases, when basically the only way to mess up transactional integrity is to fail to use them properly, which is something easily learned. The downside of a database is that it might be overkill, but it's safe against corruption. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Help on PyQt5 !!
Hey guys. I might be asking the most childish question. I have a window in pyqt5 (a file in python). Let's call it win1. I have another pyqt5 (another file in python). Let's call it win2. Now win2 is called when add button in win1 is clicked. So far no problem. But win2 has another button called save. When this is called I want a label to appear in win1. How can I do this?? And how can I have a number of labels like this when a certain condition is fulfilled?? Thank you in advance for your help. -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestions on mechanism or existing code - maintain persistence of file download history
Chris, > That's what transactions are for. Again, >> I guess that that went right over your head. :-)/You/ might know >> exactly >> what should and shouldn't be done, what makes you think the OP currently >> does ? > I don't understand why you're denigrating databases, Am I denigrating a nut-and-bolt when glue would be an easier solution ? The problem with most people (programmers included) that they often over-think (and thus over-complicate) stuff.I think that a database is /definitily/ overcomplicating stuff, especially when looking at the OP who's supposed to write and maintain it. Y personal MMV. Regards, Rudy Wieser -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestions on mechanism or existing code - maintain persistence of file download history
On Thu, Jan 30, 2020 at 11:11 PM R.Wieser wrote: > > Chris, > > > That's what transactions are for. > > Again, > > >> I guess that that went right over your head. :-)/You/ might know > >> exactly > >> what should and shouldn't be done, what makes you think the OP currently > >> does ? > > > I don't understand why you're denigrating databases, > > Am I denigrating a nut-and-bolt when glue would be an easier solution ? > > The problem with most people (programmers included) that they often > over-think (and thus over-complicate) stuff.I think that a database is > /definitily/ overcomplicating stuff, especially when looking at the OP who's > supposed to write and maintain it. Y personal MMV. > Okay, sure... but you didn't say that. You said that a database could become corrupted. If you read back in this thread, you'll see that the first recommendations were NOT about databases, and then databases were offered as an alternative. You then said that they could become corrupted just as files can. This is outright false, and is also not helpful to the discussion. So I responded. But I wasn't advocating for the use of a database; my first and strongest recommendation was, and still is, a stateless system wherein the files themselves are the entire indication of which documents have been downloaded. So what IS your line of argument? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Was: Dynamic Data type assignment
On Thu, 30 Jan 2020 20:40:50 +1300 DL Neil via Python-list wrote: > Why do we have [argparse] at the cmdLN and yet not have something > similar for input? Because argparse works on input, too? Many examples on https://docs.python.org/3/library/argparse.html¹ explicitly pass a list of strings to parser.parse_args instead of letting that function default to the list of command line arguments. ¹ e.g., https://docs.python.org/3/library/argparse.html#parents Dan -- “Atoms are not things.” – Werner Heisenberg Dan Sommers, http://www.tombstonezero.net/dan -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestions on mechanism or existing code - maintain persistence of file download history
On Thu, 30 Jan 2020 23:34:59 +1100 Chris Angelico wrote: > ... I wasn't advocating for the use of a database; my first and > strongest recommendation was, and still is, a stateless system wherein > the files themselves are the entire indication of which documents have > been downloaded. Yes, I like stateless systems, too, but that system isn't stateless. As I understand the problem of a "crudely persistem download manager," there's a collection of to-be-downloaded URLs (which may be empty) and some data that's been downloaded (which may also be empty). You can certainly encode a lot of that directly in the file system, but it's still state. Using a database instead solves a lot of the tricky bits that the bare file system doesn't (which is what ChrisA said in what I snipped). It's just Greenspun's Tenth Rule with the words C, Fortran, and Common Lisp crossed out and Python and ACID Database written in crayon. Dan -- “Atoms are not things.” – Werner Heisenberg Dan Sommers, http://www.tombstonezero.net/dan -- https://mail.python.org/mailman/listinfo/python-list
ANN: Wing Python IDE 7.2.1.0 released
Wing 7.2.1 has been released. This update fixes debug process group termination, avoids failures seen when pasting some Python code, prevents crashing in vi browse mode when the first line of the file is blank, and fixes some other usability issues. For details see the change log: https://wingware.com/pub/wingpro/7.2.1.0/CHANGELOG.txt == Auto-Reformatting with Black and YAPF (Wing Pro) == Wing 7.2 adds support for Black and YAPF for code reformatting, in addition to the previously available built-in autopep8 reformatting. To use Black or YAPF, they must first be installed into your Python with pip, conda, or other package manager. Reformatting options are available from the Source > Reformatting menu group, and automatic reformatting may be configured in the Editor > Auto-reformatting preferences group. Details: https://wingware.com/doc/edit/auto-reformatting == Improved Support for Virtualenv == Wing 7.2 improves support for virtualenv by allowing the command that activates the environment to be entered in the Python Executable in Project Properties, Launch Configurations, and when creating new projects. The New Project dialog now also includes the option to create a new virtualenv along with the new project, optionally specifying packages to install. Details: https://wingware.com/doc/howtos/virtualenv == Support for Anaconda Environments == Similarly, Wing 7.2 adds support for Anaconda environments, so the conda activate command can be entered when configuring the Python Executable and the New Project dialog supports using an existing Anaconda environment or creating a new one along with the project. Details: https://wingware.com/doc/howtos/anaconda == And More == Wing 7.2 also makes it easier to debug modules with python -m, simplifies manual configuration of remote debugging, allows using a command line for the configured Python Executable, and fixes a number of usability issues. For a complete list of new features in Wing 7, see What's New in Wing 7: https://wingware.com/wingide/whatsnew == Downloads == Wing Pro: https://wingware.com/downloads/wing-pro/7.2/binaries Wing Personal: https://wingware.com/downloads/wing-personal/7.2/binaries Wing 101: https://wingware.com/downloads/wing-101/7.2/binaries Compare products: https://wingware.com/downloads See https://wingware.com/doc/install/upgrading for details on upgrading from Wing 6 and earlier, and https://wingware.com/doc/install/migrating for a list of compatibility notes. -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestions on mechanism or existing code - maintain persistence of file download history
Chris, >> I think that a database is /definitily/ overcomplicating stuff, > > Okay, sure... but you didn't say that. I'm sorry ? In my first reply I described a file-based approach and mentioned that the folder approach is a rather good one. What do you think I ment there ? > You said that a database could become corrupted. Yes, I did. Here: [Quote] IOW: Databases can be corrupted for pretty-much the same reason as for a simple datafile (but with much worse consequences). [/quote] Please do notice the part between the brackets. > So what IS your line of argument? To be frank, the fact that you think you should still ask baffles me. I hope that the above clears it up though. Regards, Rudy Wieser -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestions on mechanism or existing code - maintain persistence of file download history
On Fri, Jan 31, 2020 at 2:01 AM R.Wieser wrote: > > Chris, > > >> I think that a database is /definitily/ overcomplicating stuff, > > > > Okay, sure... but you didn't say that. > > I'm sorry ? In my first reply I described a file-based approach and > mentioned that the folder approach is a rather good one. What do you think > I ment there ? > > > You said that a database could become corrupted. > > Yes, I did. Here: > > [Quote] > IOW: Databases can be corrupted for pretty-much the same reason as for a > simple datafile (but with much worse consequences). > [/quote] > > Please do notice the part between the brackets. Yes, and then you backpedalled furiously when I showed that proper transactions prevent this. After which... I lost track of what you were actually trying to say. So either justify this position, showing that a database can become corrupted the same way a flat file can, in normal use; or explain to me what you're actually arguing here. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Help on PyQt5 !!
On 1/30/20 4:38 AM, Souvik Dutta wrote: > Hey guys. I might be asking the most childish question. I have a window in > pyqt5 (a file in python). Let's call it win1. > I have another pyqt5 (another file in python). Let's call it win2. Now win2 > is called when add button in win1 is clicked. So far no problem. But win2 > has another button called save. When this is called I want a label to > appear in win1. How can I do this?? And how can I have a number of labels > like this when a certain condition is fulfilled?? Thank you in advance for > your help. There are some additional resources you might consider using, and will likely get faster responses with regards to Qt questions. First is the PyQt5 mailing list. https://www.riverbankcomputing.com/mailman/listinfo/pyqt . Also there are the normal Qt forums: https://forum.qt.io/ As to your question, you can add a label to a window the same way you would add the label in the first place. Likely you'll want a QBoxLayout widget to place the label(s) into, with the addWidget() method. Take a look at the Qt docs for signal, methods, etc. https://doc.qt.io/qt-5/qboxlayout.html . For the most part, PyQt5's method calls, signals, and slots, are the same in Python as they are in C++. -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestions on mechanism or existing code - maintain persistence of file download history
Chris, > Yes, and then you backpedalled furiously when I showed that > proper transactions prevent this. You're a fool, out for a fight. /You/ might know exactly how to handle a database to make sure its /transactions/ will not leave the database in a corrupt state, but as I mentioned a few posts back: [quote] I think that a database is /definitily/ overcomplicating stuff, especially when looking at the OP who's supposed to write and maintain it. [/quote] Mind the part after the comma. Also, you happily ignored the part in my second post where I mentioned that corruption can be of a different kind - one that you have absolutily /no/ control over: [quote] An unreadable file is often described as being corrupt [/quote] > I lost track of what you were actually trying to say. Well, all that I wanted to say - to the OP - is in my first reply. You could always try to re-read it. The other posts where just responses to your "that can't be!" challenges to me. > So either justify this position, showing that a database can > become corrupted the same way a flat file can, :-) In the /same/ way ?Thats rather multi-interpretable, don't you think ? Are you trying to set me up ? My point was that the /effect/ of a same corruption is different for both: A flat file is /much/ easier to rescue. But, do you remember what the OP said ? [quote] want to download these as a 'background task'. ... you can CTRL-C out, [/quote] Why now do I think that, when such a backgroud process is forgotten and the 'puter switched off, the file, database or otherwise, could easily get damaged ? And that a ctrl-c at the wrong moment could cause the same if the cleanup (of the database, which could easily still be, in its own thread, busy housekeeping) isn't done correctly. Also, have you /never/ encountered a corrupted file, database or otherwise ? You must not have done much with 'puters at all ... Though I think I'm going to end our conversation here, as you're way more agressive than the subject calls for. Goodbye chris. Have a good life. Regards, Rudy Wieser -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestions on mechanism or existing code - maintain persistence of file download history
On Fri, Jan 31, 2020 at 4:11 AM R.Wieser wrote: > But, do you remember what the OP said ? > [quote] > want to download these as a 'background task'. ... you can CTRL-C out, > [/quote] > > Why now do I think that, when such a backgroud process is forgotten and the > 'puter switched off, the file, database or otherwise, could easily get > damaged ? And that a ctrl-c at the wrong moment could cause the same if the > cleanup (of the database, which could easily still be, in its own thread, > busy housekeeping) isn't done correctly. > > Also, have you /never/ encountered a corrupted file, database or otherwise ? > You must not have done much with 'puters at all ... > On the contrary, I grew up with IBM DB2 and then PostgreSQL, both of which are actually resilient against power failures. I've also recovered data from crashed hard drives. So, yeah, I think I know what I'm talking about. Sorry. Have a nice day. Enjoy dealing with corruption in places where ordinary best-practices and basic tutorials can prevent it from ever happening. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Using Python and scikitlearn, is there a way to return the "feature dependency" percentage?
Hello, I am relatively new to Python and Machine Learning. I have a basic dataset for insurance fraud and a script that generates the model and runs the predictions. I am able to output the accuracy percentages, but I would like to also output the feature dependencies: For example, what role did each attribute play in the prediction? The policy_number would be 0.0% where as the claim_amount would likely be 56.2%, does this make sense? Is there a scikit function for this? Also, is "feature dependency" even the correct term? Thank you for your help! -Matt -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestions on mechanism or existing code - maintain persistence of file download history
Err, well, thanks for that discussion gents... As it happens I do know how to use a database, but I regard it as overkill for what I am trying to do here. I think a combination of hashing the URL, and using a suffix to indicate the result of previous downloaded attempts, will work adequately for my needs. Thanks again Jon N -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestions on mechanism or existing code - maintain persistence of file download history
On 30/01/20 9:35 PM, R.Wieser wrote: MRAB's scheme does have the disadvantages to me that Chris has pointed out. Nothing that can't be countered by keeping copies of the last X number of to-be-dowloaded-URLs files. That's a good idea, but how would the automated system 'know' to give-up on the current file and utilise generation n-1? Unable to open the file or ??? As for rewriting every time, you will /have/ to write something for every action (and flush the file!), if you think you should be able to ctrl-c (or worse) out of the program. Which is the nub of the problem! Using ctrl+c is a VERY BAD idea. Depending upon the sophistication of the solution/existing code, surely there is another way... Even closing/pulling-out the networking connection to cause an exception within Python, would enable management of a more 'clean' and 'data safe' shutdown! (see also 'sledgehammer to crack a nut') Why do you need to abandon the process mid-way? But, you could opt to write this sessions successfully downloaded URLs to a seperate file, and only merge that with the origional one program start. That together with an integrity check of the seperate file (eventually on a line-by-line (URL) basis) should make the origional files corruption rather unlikely. What is the OP's definition of "unlikely" or "acceptable risk"? If RDBMS == "unnecessary complexity", then (presumably) 'concern' will be commensurately low, and much of the discussion to-date, moot? I've not worked on 'downloads' (which I take to mean data files, eg forms from the tax office - guess what task I'm procrastinating over?) but have automated the downloading of web page content/headers. There are so many reasons why such won't work first-time, when they should every time; that it may be quite difficult to detect 'corruption' (as distinct from so many of these other issues that may arise)... A database /sounds/ good, but what happens when you ctrl-c outof a non-atomic operation ? How do you fix that ?IOW: Databases can be corrupted for pretty-much the same reason as for a simple datafile (but with much worse consequences). [apologies for personal comment] I, (with my skill-set, tool-set, collection of utilities, ... - see earlier mention of "bias") reach for an RDBMS more quickly than many*. Mea culpa or 'more power to [my] right arm'? The DB suggestion (posted earlier) involved only a single table, to which fields would be added/populated during processing as a record of progress/status. Thus, replacing the single file that the OP (originally) outlined as fitting his/her needs, with a single DB-table. Accordingly, there is no non-atomic transaction in the proposal - UPDATE is atomic in most (competent) RDBMS. (again, in my ignorance of that project, please don't (anyone) think I'm including/excluding SQLite) Contrarily, if the 'single table idea' is hardly a "database" by most definitions, why bother? The answer lies in the very mechanisms to combat corruptions and interruptions being discussed! As a fundamentally-lazy person, I'd rather leave the RDBMS-coders to wrestle with such complexities 'for me'. Then, I can 'stand on the shoulders' of such 'giants', by driving their (competently working) 'black box'... (YMMV!) Now, it transpires, the OP possesses DB skills. So, (s)he is in a position to make the go/no decision which suits the actual spec. Yahoo! (not TM) Also think of the old adagio: "I had a problem, and than I thought I could use X. Now I have two problems..." - with X traditionally being "regular expressions". In other words: do KISS (keep it ) Good point! (I'm not a great fan of RegEx-es either) - reduce/avoid complexity, "simple is better than complex"! (Python: import this) Surely though, it is only appropriate to dive into the concerns and complexities of DB accuracy and "consistency", if we do likewise with file systems? The rationale of my 'laziness' argument 'for' using an RDBMS, also applies to plain-vanilla file systems. Do I want to deal with the complexities of managing files and corruptions, in that arena? (you could easily guess the answer to that!) Do you? (the answer may be quite different - but no matter, I'm not going to say you are "wrong", as long as in making such a decision (files?DB) we compare 'like with like' - in fact, before that: as long as the client's spec says that we need to be worrying about such detail! (otherwise YAGNI applies!) By the way: The "just write the URLs in a folder" method is not at all a bad one. /Very/ easy to maintain, resilent (especially when you consider the self-repairing capabilities of some filesystems) and the polar opposite of a "customer lock-in". :-) +1 Be aware that formation rules for URLs are not congruent with OS FS rules! (such concerns don't apply if the URLs are data within a file/table) * was astonished to discover (a show-of-hands poll at some conference or other) that
QTableWidget help!!
Hi, I want to add a QTableWidget to an existing window with other labels and button. How can I add the QTableWidget without having to set it as centralwidget and be able to move it and set geometry. Because I am not being able to move the central widget by using self.centralwidget.setGeometry . The error is cannot find reference. Please advise. -- https://mail.python.org/mailman/listinfo/python-list
