Package directory question

[Robert Latest]

From: Robert Latest 

Hello,

I'm building an application which consists of two largely distinct parts, a
frontend and a backend. The directory layout is like this:

|-- jobwatch
|   |-- backend
|   |   |-- backend.py
|   |   |-- __init__.py
|   |   `-- tables.py
|   |-- frontend
|   |   |-- __init__.py
|   |   |-- main.py
|   `-- __init__.py
`-- setup.py

Because the main.py script needs to import the tables.py module from backend, I
 put this at the top if main.py:

   sys.path.append('../..')
   import jobwatch.backend.tables as tables

My question is: Is this the way it should be done? It looks fishy. The only
alternative I could come up with is to put a symlink to tables.py into the
frontend directory, which also seems fishy. Eventually I want to package all
this up neatly to be able to use only little wrapper scripts for the backend
(running as a service) and the frontend (a wsgi app).

Any thoughts?

Thanks
robert

--- BBBS/Li6 v4.10 Toy-3
 * Origin: Prism bbs (1:261/38)
-- 
https://mail.python.org/mailman/listinfo/python-list

Question about email-handling modules

[Robert Latest]

Hello,

I'm new to Python but have lots of programming experience in C, C++ and 
Perl. Browsing through the docs, the email handling modules caught my eye 
because I'd always wanted to write a script to handle my huge, ancient, and 
partially corrupted email archives.

Of course I know that this kind of project shouldn't be tackled by a 
beginner in a language, but I still thought I'd give it a spin. 

So I wrote the stuff at the bottom. It lists senders, subjects and 
addressees of all messages in an mbox.

Things that I don't understand:

1. Why can I get the 'subject' and 'from' header field unsig the [] 
notation, but not 'to'? When I print Message.keys I get a list of all header 
fields of the message, including 'to'. What's the difference between 
message['to'] and message.get('to')?

2. Why can't I call the get_payload() method on the message? What I get is 
this cryptic error: "AttributeError: Message instance has no attribute 
'get_payload'". I'm trying to call a method here, not an attribute. It makes 
no difference if I put parentheses after get_payload or not. I looked into 
the email/Message module and found get_payload defined there. 

I don't want to waste your time by requesting that you pick apart my silly 
example. But maybe you can give me a pointer in the right direction. This is 
python 2.4 on a Debian box.

---

#!/usr/bin/python
import mailbox
import email  # doesn't make a difference
from email import Message # neither does this

mbox = file("mail.old/friends")

for message in mailbox.UnixMailbox(mbox):
subject = message['subject']
frm = message['from']
#to = message['to'] # this throws a "Key Error"
to = message.get('to'); # ...but this works
print frm, "writes about", subject, "to", to
#print message.get_payload() # this doesn't work

--

robert
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Question about email-handling modules

[Robert Latest]

Steven D'Aprano wrote:

> message['to'] looks up the key 'to', raising an exception if it doesn't 
> exist. message.get('to') looks up the key and returns a default value if 
> it doesn't exist.

Ah, so the [] notation got hung up on some message right at the beginning 
and didn't even let the script continue. Makes sense.

> All methods are attributes (although the opposite is not the case), so if 
> a method doesn't exist, you will get an AttributeError.

I see. I've already gathered that Python likes to use different words for 
common things (attribute instead of member or method).

> Damned if I can work out how to actually *use* the email module to read 
> an mbox mail box. I might have to RTFM :(

Yeah, I think I haven't picked the right module to get started with.

> But note that message.get_payload() will return either a string (for 
> single part emails) or a list of Messages (for multi-part messages).

Yes, I did note that.

Thanks for the tips (also to the others who have answered).

Python looks like fun though. Maybe I should try to tackle some other 
problem first.

robert
-- 
http://mail.python.org/mailman/listinfo/python-list

popen question

[Robert Latest]

Hello,

look at this function:

--
def test():
child = os.popen('./slow')
for line in child:
print line
-

The program "slow" just writes the numbers 0 through 9 on stdout, one line a 
second, and then quits.

I would have expected the python program to spit out a numbers one by one, 
instead I see nothing for 10 seconds and then the whole output all at once.

How can I get and process the pipe's output at the pace it is generated?

Thanks,

robert
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: popen question

[Robert Latest]

Marc 'BlackJack' Rintsch wrote:

> Both processes have to make their communication ends unbuffered or line
> buffered.

Yeah, I figured something like that.

> And do whatever is needed to output the numbers from ``slow``
> unbuffered or line buffered.

Hm, "slow" of course is just a little test program I wrote for this purpose. 
In reality I want to call another program whose behavior I can't influence 
(well, technically I could because it's open-source, but let's assume it to 
be a black box for now).

If 'slow' or some other program does buffered output, how come I can see 
its output line-by-line in the shell?

robert
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: popen question

[Robert Latest]

Hrvoje Niksic wrote:

> stdio uses different buffering strategies depending on the output
> type.  When the output is a TTY, line buffering is used; when the
> output goes to a pipe or file, it is fully buffered.

Makes sense.

> If you see lines one by one, you are in luck, and you can fix things
> on the Python level simply by avoiding buffering in popen.  If not,
> you will need to resort to more advanced hackery (e.g. fixing stdio
> using LD_PRELOAD).

Do I really? After all, the shell itself doesn't hack stdio, does it?
Anyway, I'm taking this over to comp.unix.programmer since it really isn't a 
python problem.

Thanks,
robert
-- 
http://mail.python.org/mailman/listinfo/python-list

(SOLVED) Re: popen question

[Robert Latest]

pexpect is the solution. Seems to wrap quite a bit of dirty pseudo-tty 
hacking.

robert
-- 
http://mail.python.org/mailman/listinfo/python-list

"Canonical" way of deleting elements from lists

[Robert Latest]

Hello,

>From a list of strings I want to delete all empty ones. This works:

while '' in keywords: keywords.remove('')

However, to a long-term C programmer this looks like an awkward way of 
accomplishing a simple goal, because the list will have to be re-evaluated 
in each iteration. Is there a way to just walk the list once and throw out 
unwanted elements as one goes along?

I started programming back when such little things were real performance 
issues, so I have some sort of cringe reflex when something looks 
inefficient.

robert
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: "Canonical" way of deleting elements from lists

[Robert Latest]

Fredrik Lundh wrote:

> keywords = filter(None, keywords) # get "true" items only

Makes seinse. BTW, where can I find all methods of the built-in types? 
Section 3.6 only talks about strings and mentions the list append() method 
only in an example. Am I too stupid to read the manual, or is this an 
omission?

robert
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: "Canonical" way of deleting elements from lists

[Robert Latest]

Hrvoje Niksic wrote:

> keywords[:] = (s for s in keywords if s)

Looks good but is so far beyond my own comprehension that I don't dare 
include it in my code ;-)

robert
-- 
http://mail.python.org/mailman/listinfo/python-list

How does unicode() work?

[Robert Latest]

Here's a test snippet...

import sys
for k in sys.stdin:
print '%s -> %s' % (k, k.decode('iso-8859-1'))

...but it barfs when actually fed with iso8859-1 characters. How is this 
done right?

robert
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How does unicode() work?

[Robert Latest]

Robert Latest wrote:
> ...but it barfs when actually fed with iso8859-1 characters.

Specifically, it says:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xf6 in position 0: 
ordinal not in range(128)

which doesn't make sense to me, because I specifically asked for the 
iso8859-1 decoder, not the 'ascii' one.

robert
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How does unicode() work?

[Robert Latest]

John Machin wrote:

> When mixing unicode strings with byte strings, Python attempts to
> decode the str object to unicode, not encode the unicode object to
> str.

Thanks for the explanation. Of course I didn't want to mix Unicode and Latin 
in one string, my snippet just tried to illustrate the point. I'm new to 
Python -- I came from C, and C gives a rat's ass about encoding. It just 
dumps bytes and that's that.

robert
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: "Canonical" way of deleting elements from lists

[Robert Latest]

Sion Arrowsmith wrote:
> Robert Latest  <[EMAIL PROTECTED]> wrote:
>> BTW, where can I find all methods of the built-in types? 
>>Section 3.6 only talks about strings and mentions the list append() method 
>>only in an example. Am I too stupid to read the manual, or is this an 
>>omission?
>
> 3.6 talks about features common to all "sequence" types. Strings
> are discussed specifically in 3.6.1 ("String Methods"). Lists are
> similarly discussed in 3.6.4 ("Mutable Sequence Types").

OK, the latter then. Too stupid. Thanks ;-)

robert
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: encrypting python modules

[Robert Latest]

Paul Sijben wrote:

> The problem: I have a client-server app written in python. I want to
> make sure that the client is not:
> 1) destabilized by users accidentally or on purpose dropping python
> files in the path (after which calling the helpdesk will not be useful)
> 2) extended with "new features" without me knowing about it (again
> resulting in calls to my helpdesk...)

You could check the MD5 hashes of your files.

robert
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: encrypting python modules

[Robert Latest]

Paul Sijben wrote:
>> 
>> You could check the MD5 hashes of your files.
>
> indeed but I still need to hook into import to do that reliably, right?

Depends. In a job I once had I just supplied a shell script that spat out 
the MD5 sums of my sources. When I got a support request I had the customer 
run the script and email the result to me so I could check if anything had 
changed. (A malicious person could of course have stored the good MD5 sums, 
then done his changes, and then made his request, sending me the old sums.)
If your customer is cooperative and trustworthy, this is a good way to find 
stupid mistakes. If he ain't, drop him ;-)

robert
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: encrypting python modules

[Robert Latest]

Steven D'Aprano wrote:

> No, it's a serious question. You distribute Python code, and you're 
> worried that your users will modify the source code and then neglect to 
> mention it when they report bugs which they introduced.
>
> Before you build an elephant-proof fence around your house, it is quite 
> reasonable to ask whether or not there are actually elephants in your 
> neighbourhood.

Depending on where you are, there are probably plenty. Recently I started 
hacking around in a GPLed Python app (GRAMPS, if anybody cares). This 
program has a built-in bug report feature which makes it easy to email bugs 
with associated stack trace etc. to the developers. Unfortunately the bug 
report window also automatically popped up even when stuff went wrong within 
my own broken code. For such a case it would be good if a software could 
somehow detect modifications of itself and its associated modules.

And, contrary to the advice I gave elsethread, unfortunately it's impossible 
to just drop uncooperative customers when you develop GPL software ;-)

robert
-- 
http://mail.python.org/mailman/listinfo/python-list

Question on sort() key function

[Robert Latest]

Hello,

I have this class:

class File:
def __init__(self):
self.name = ''
self.path = ''
self.date = 0
self.mod_date = 0
self.keywords = []
self.url = ''


...and after creating a list of File objects called flist, I'd like to sort 
it like thus:

flist.sort(key=File.mod_date.toordinal)

However, Python says:
AttributeError: class File has no attribute 'mod_date'

Well if you ask me, there are many things that may be said about my File 
class, but the absence of the attribute 'mod_date' ain't one of them. What 
do you think?

And yes, this loop works fine:

for f in flist:
print f.mod_date.isoformat()

(which IMO proves that all mod_date members are properly initialized as 
datetime objects).

robert
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Question on sort() key function

[Robert Latest]

Paul Rubin wrote:
> The attribute is on instances of File, not on the class itself.  See
> if this works:
>
>flist.sort(key=lambda f: f.mod_date.toordinal)

It doesn't throw an error any more, but neither does it sort the list. This, 
however, works:

--
def by_date(f1, f2):
return f1.mod_date.toordinal() - f2.mod_date.toordinal()

flist.sort(by_date)
--

So I'm sticking with it, although I sort of liked the key approach.

robert
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Question on sort() key function

[Robert Latest]

Peter Otten wrote:
> Robert Latest wrote:
>
>> Paul Rubin wrote:
>>> The attribute is on instances of File, not on the class itself.  See
>>> if this works:
>>>
>>>flist.sort(key=lambda f: f.mod_date.toordinal)
>> 
>> It doesn't throw an error any more, but neither does it sort the list. This, 
>> however, works:
>> 
>> --
>> def by_date(f1, f2):
>>  return f1.mod_date.toordinal() - f2.mod_date.toordinal()
>> 
>> flist.sort(by_date)
>> --
>> 
>> So I'm sticking with it, although I sort of liked the key approach.
>> 
>> robert
>
> This should work then:
>
> def date_key(f):
> return f.mod_date.toordinal()
> flist.sort(key=date_key)
>
> This can also be written as
>
> flist.sort(key=lambda f: f.mod_date.toordinal())

Well, that's almost Paul's (non-working) suggestion above, but it works 
because of the parentheses after toordinal. Beats me how both versions can 
be valid, anyway.

To me it's all greek. I grew up with C function pointers, and they 
always work.

robert
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python tricks

[Robert Latest]

RajNewbie wrote:
>Is there a way - a python trick - to have a check such that if the
> loop goes for more than x number of steps, it will cause an exception?
>
>I do understand that we can use the code like -
>i = 0
>while True:
>  i++
>  if i > 200: raise infinite_Loop_Exception
>  ...
>  if : break
>
>But I am not very happy with this code for 3 reasons
>1. Verbosity (i=0 and i++) which doesnt add to the logic
>2. The loop now has dual focus. - incrementing i, etc.
>3.A person looks into the code and thinks 'i'
> has special significance. His/her mind will be focused on not the
> actual reason for the loop.

Maybe you should call the counter variable something meaningful instead
of -- of all things -- "i", which is idiomatic for soething entirely
else. And add a comment, and be done with it.

>The solution that I had in mind is:
>while True:
>  ...
>  if : break
>  if inifinte_loop(): raise infiinte_loop_exception

This is a lot less understandable because whoever is working with your
code will now have to check an additional function  rather than a
pretty self-explanatory variable increment.

robert
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python tricks

[Robert Latest]

RajNewbie wrote:

> But, I still feel it would be much more aesthetically pleasing if I
> can call a single procedure like
> if infinite_loop() -> to do the same.

You may find it aesthetically pleasing, and it may very well be, but it
will obfuscate your code and make it less maintainable.

robert
--
http://mail.python.org/mailman/listinfo/python-list

Memory debugging tool for Python/C API?

[Robert Latest]

Hello,

extending Python in C ist just the coolest thing since sliced bread (and
I'm particularly happy because I really had started to miss C when I did
more and more things in Python).

I've got one question though. Tha C/API documentation is not quite clear
(to me, anyway) on what happens exactly to ressources between the
C/Python side. One example is this: I wrote a function that assembles a
string into a dynamically-allocated buffer, then creates a PyString from
it which it returns to the Python side:

PyObject *dyn_string(void)
{
char *buffer = malloc(20);
PyObject *pystr;
strcpy(buffer, "Hello, World");
pystr = PyString_FromString(buffer);
free(buffer);
return pystr;
}

This works, but I'm not sure if PyString...() really makes a new copy of
the data (ellowing me to use free()), or if this will segfault one day.
OTOH I wouldn't know how to safely dispose of the string data otherwise.

Another example (all on the C side of things) is a function where I
build a dictionary from a set of keys, putting PyNone into each value
(Py_INCREF()ing PyNone each time).  At another point some of the values
are replaced by other PyObjects. At first I used PyDECREF() on each
value before setting the new value, but that botched up the dictionary
beyond repair. Without the PyDECREF() calls everything worked fine.

Now to my actual question: Is there some sort of debugging tool that I
could use to analyze my code to see if everything gets properly
allocated and deallocated and if the reference counting works right?

Thanks,
robert
--
http://mail.python.org/mailman/listinfo/python-list

Re: Confounded by Python objects

[Robert Latest]

satoru wrote:

> As to "sample", it never get assigned to and when you say "append" the
> class variable is changed in place.
> hope my explaination helps.

Sure does, thanks a lot.

Here's an interesting side note: After fixing my "Channel" thingy the
whole project behaved as expected. But there was an interesting hitch.
The main part revolves around another class, "Sequence", which has a
list of Channels as attribute. I was curious about the performance of my
script, because eventually this construct is supposed to handle
megabytes of data. So I wrote a simple loop that creates a new Sequence,
fills all the Channels with data, and repeats.

Interistingly, the first couple of dozens iterations went satisfactorily
quickly (took about 1 second total), but after a hundred or so times it
got really slow -- like a couple of seconds per iteration.

Playing around with the code, not really knowing what to do, I found
that in the "Sequence" class I had again erroneously declared a class-level
attribute -- rather harmlessly, just a string, that got assigned to once in each
iteration on object creation. 

After I had deleted that, the loop went blindingly fast without slowing
down.

What's the mechanics behind this behavior?

robert
--
http://mail.python.org/mailman/listinfo/python-list

Re: Pointers/References in Python?

[Robert Latest]

Gary Herron wrote:
> No need.  A Python list contains *references* to objects, not copies of 
> objects.  (The same is true of variables, dictionaries, sets, and so 
> on...). 

Good to know. I just wanted to make sure before writing more code which in
the end might not scale well.

Thanks to all for the help!

robert
--
http://mail.python.org/mailman/listinfo/python-list

mktime overflow in March 2008?

[Robert Latest]

Here's what happens on my Windows machine (Win XP / Cygwin) at work.
I've googled a bit about this problem but only found references to
instances where people referred to dates before the Epoch.

Of course at home on my Linux box everything works.

I know that everything has its limits somewhere, but I've never
heard of March 2008 being a problem.

Tomorrow I'm going to write a test loop that shows me the exact last
second that mktime is going to be able to handle. Sort of like the way
Calvin's father explains the weight limit specification of bridges.

I wonder if the datetime module is any better, considering that under
the hood it probably uses the same C library functions.

I like to stick with the "seconds since..." approach because I need to
store millions of dates/times efficiently in a sqlite database.

$ python
Python 2.5.1 (r251:54863, May 18 2007, 16:56:43) 
[GCC 3.4.4 (cygming special, gdc 0.12, using dmd 0.125)] on cygwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> t = time.strptime("Mar 30, 2007 2:43:32 am", "%b %d, %Y %I:%M:%S
>>> %p")
>>> time.mktime(t)
1175215412.0
>>> t = time.strptime("Mar 30, 2008 2:43:32 am", "%b %d, %Y %I:%M:%S
>>> %p")
>>> time.mktime(t)
Traceback (most recent call last):
  File "", line 1, in 
OverflowError: mktime argument out of range
>>> 

robert
--
http://mail.python.org/mailman/listinfo/python-list

Re: mktime overflow in March 2008?

[Robert Latest]

Tim Roberts wrote:
> What time zone are you in?  March 30, 2008, was a Sunday.  If that happened
> to be the date your country transitioned to summer time, then the hour
> between 2 AM and 3 AM would not exist.
>
> Does it work if you use 3:43 AM?

Actually you're right; I'm in Middle Europe, and DST was started on
March 30, 2am. I need to check this.

robert
--
http://mail.python.org/mailman/listinfo/python-list

[SOLVED] How can this script fail?

[Robert Latest]

Dennis Lee Bieber wrote:
>> On 2008-08-27 12:37, [EMAIL PROTECTED] wrote:
>> > Got file  in dir 
>> > Found in path:
>> > 
>> > 
>> 
>> This looks strange... the same module in two dirs that only
>> differ by case ?
>>
>   Even stranger when you take into account that under Windows, path
> names are case-preserving/case-ignored -- so Lib and lib are the SAME
> directory (I suspect some environment variable has the directory listed
> twice with the change in spelling -- PythonPath perhaps?)

Please note that one of the lines is the directory that was found in the
"os.walk" loop and the other is the one in sys.path -- I just had them
printed out on adjacent lines to point out the difference in case.

Anyway, the thing was fixed when I put C:\Python25 into the DOS $PATH
environment variable. Still strange though.

robert
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python as network protocol

[Robert Latest]

Grant Edwards wrote:
> On 2009-11-10, Steven D'Aprano  wrote:
>>> How do you know for sure? Maybe the OP wants to use this thing
>>> with 3 known researchers working on a cluster that is not even
>>> visible to the outside world.
>
> And those three researchers are perfect?  They've never even
> made a typographical error?

Anybody with sufficient access can bring down a system by accident. It
doesn't matter if it's a single computer or an isolated cluster.

The question on how accident-proof the network needs to be depends on
how many people's work is affected by someone bringing it down. Since we
know nothing about the scale of the network we can't say that the
proposed protocol is "too unsafe".

robert
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: New syntax for blocks

[Robert Latest]

r wrote:
> Forgive me if i don't properly explain the problem but i think the
> following syntax would be quite beneficial to replace some redundant
> "if's" in python code.
>
> if something_that_returns_value() as value:
> #do something with value
>
> # Which can replace the following syntactical construct...
>
> value = something_that_returns_value()
> if value:
> #do something with value
>
> i dunno, just seems to make good sense. You save one line of code but
> more importantly one indention level.

Typical case in matching regexes. But where do we save an indentation
level?

Also it's not the "if" that is (if at all) redundant here but the assignment. 

robert
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: New syntax for blocks

[Robert Latest]

r wrote:
> Just thinking out loud here...what if variable assignments could
> return a value... hmmm? Not to them selfs of course but to a caller,
> like an if statement...
>
> if a=openfile:
>   # do something with a

That's like in C. I sometimes miss it in Python.

robert
-- 
http://mail.python.org/mailman/listinfo/python-list

Implementation of an lru_cache() decorator that ignores the first argument

[Robert Latest via Python-list]

Hi all,

in a (Flask) web application I often find that many equal (SQLAlchemy) queries
are executed across subsequent requests. So I tried to cache the results of
those queries on the module level like this:

@lru_cache()
def query_db(db, args):
# do the "expensive" query
return result

This obviously doesn't work because each request uses a new database session,
so the db argument always changes from one call to the next, triggering a new
query against the database. But even if that weren't so, the function would
keep returning the same value forever (unless it's kicked out of the cache) and
not reflect the (infrequent) changes on the database. So what I need is some
decorator that can be used like this:

@lru_ignore_first(timeout=10)
def query_db(db, args):
# do the "expensive" query
return result

This is what I came up with. I'm quite happy with it so far.  Question: Am I
being too clever? is it too complicated? Am I overlooking something that will
come back and bite me later? Thanks for any comments!

from functools import wraps, lru_cache
from time import time, sleep

def lru_ignore_first(timeout=0, **lru_args):

class TimeCloak():
'''All instances compare equal until timeout expires'''
__slots__ = ('x', 't', 'timeout')

def __init__(self, timeout):
self.timeout = timeout
self.t = 0
self.x = None

def __hash__(self):
return self.t

def __eq__(self, other):
return self.t == other.t

def update(self, x):
self.x = x
if self.timeout:
t = int(time())
if t >= self.t + self.timeout:
self.t = t

cloak = TimeCloak(timeout)

def decorator(func):

@lru_cache(**lru_args)
def worker(cloak, *a, **b):
return func(cloak.x, *a, **b)

@wraps(func)
def wrapped(first, *a, **kw):
cloak.update(first)
return worker(cloak, *a, **kw)

return wrapped

return decorator

@lru_ignore_first(3)
def expensive(first, par):
'''This takes a long time'''
print('Expensive:', first, par)
return par * 2

for i in range(10):
r = expensive(i, 100)
sleep(1)
print(r)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Implementation of an lru_cache() decorator that ignores the first argument

[Robert Latest via Python-list]

Hi Chris and dh,

thanks for your --as usually-- thoughtful and interesting answers. Indeed, when
doing these web applications I find that there are several layers of useful,
maybe less useful, and unknown caching. Many of my requests rely on a
notoriously unreliable read-only database outside of my control, so I cache the
required data into a local DB on my server, then I do some in-memory caching of
expensive data plots because I haven't figured out how to reliably exploit the
client-side caching ... then every middleware on that path may or may not
implement its own version of clever or not-so-clever caching. Probably not a
good idea to try and outsmart that by adding yet another thing that may break
or not be up-to-date at the wrong moment.

That said, the only caching that SQLAlchemy does (to my knowledge) is that it
stores retrieved DB items by their primary keys in the session. Not worth much
since the session gets created and dumped on each request by SQA's unit of work
paradigm. But the DB backend itself may be caching repeated queries.

Back to Python-theory: The "Cloak" object is the only way I could think of to
sneak changing data past lru_cache's key lookup mechanism. Is there some other
method? Just curious.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: for -- else: what was the motivation?

[Robert Latest via Python-list]

Chris Angelico wrote:
> Yes, I'm aware that code readability becomes irrelevant for
> short-duration projects. Beside the point. I'm wondering how important
> it really is to have the shortest block first.

I usually put the most expected / frequent / not negated block first if the
whole if/else statement is not "too long". Sometimes whatever you want to do
becomes pointless if a certain conditions is not met, in which case I do an
early break or return and have no else block at all.

> Given that for-else is an excellent, if rarely-used, construct

I knew it existed but coming from C I never thought to exploit it. I know I
wrote loops like this:

found = None
while not found:
found = search(something)
if found:
break
if not found:
complain()

Need to look into using "else" in these cases.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: for -- else: what was the motivation?

[Robert Latest via Python-list]

Grant Edwards wrote:
> I've followed that advice for several decades. I find it much easier
> to read code that's organized that way -- particularly when the
> difference in block sizes is large (e.g. the first block is one line,
> and the second is a a hundred).

If any conditionally executed blocks is a hundred lines, I believe your code
needs refactoring. I know mine does. Either the long block should go into an
extra function, or you do a "fail and bail" (just learned that phrase).
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What to use for finding as many syntax errors as possible.

[Robert Latest via Python-list]

 wrote:
> Cameron,
>
> Your suggestion makes me shudder!

Me, too

> Removing all earlier lines of code is often guaranteed to generate errors as
> variables you are using are not declared or initiated, modules are not
> imported and so on.

all of which aren't syntax errors, so the method should still work. Ugly as
hell though. I can't think of a reason to want to find multiple syntax errors
in a file.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: for -- else: what was the motivation?

[Robert Latest via Python-list]

Axy wrote:
>> Also not really a justification for "shortest block first". Wanting
>> some elaboration on that. What's the value in it?
>
> Well, the value is productivity. No need to save puzzles "what this 
> hanging else belongs to?"

If you find yourself asking that question, the if-block is probably too long to
begin with.

> Code small things first and return early, same 
> as taking a test: do easy and quick things first and boring and 
> difficult ones later.

Yes, but in that case you have a very long indented "else" block, and at the
point where the unindent happens you are scratching your head again like
before. Better to immediately return or break and not to use any "else" block
at all.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What to use for finding as many syntax errors as possible.

[Robert Latest via Python-list]

Michael F. Stemper wrote:
> How does one declare a variable in python? Sometimes it'd be nice to
> be able to have declarations and any undeclared variable be flagged.

To my knowledge, the closest to that is using __slots__ in class definitions.
Many a time have I assigned to misspelled class members until I discovered
__slots__.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What to use for finding as many syntax errors as possible.

[Robert Latest via Python-list]

Antoon Pardon wrote:
> I would like a tool that tries to find as many syntax errors as possible 
> in a python file.

I'm puzzled as to when such a tool would be needed. How many syntax errors can
you realistically put into a single Python file before compiling it for the
first time?

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: for -- else: what was the motivation?

[Robert Latest via Python-list]

 wrote:
> I had another crazy thought that I AM NOT ASKING anyone to do. OK?
>
> I was wondering about a sort of catch method you could use that generates a
> pseudo-signal only when the enclosed preceding  loop exits normally as a
> sort of way to handle the ELSE need without the use of a keyword known by
> the language. All you would need is an object of the right kind that is
> thrown and optionally caught.


(untested)

try:
while condition:
if not do_something():
raise RuntimeError
except RuntimeError:
pass
else:
print('Loop exited normally')

Ironically, this again relies on the much-used "else" and adds the overhead of
exception handling. Also from a natural language perspective I find the "try
...  except ... else" clause just as questionable as "while ... else." Since
we're discussing weird keywords: Maybe we can find another use for "finally."
In fact, one could argue that "while ... finally" could make just as much sense
as "while ... else" (but I won't).

> Of course, even if I fleshed this out and even if anyone thought it made
> sense, there is no such need now as Python has made a choice that meets the
> need even if few may dare use it or even know about it! LOL!
-- 
https://mail.python.org/mailman/listinfo/python-list

xml.etree and namespaces -- why?

[Robert Latest via Python-list]

Hi all,

For the impatient: Below the longish text is a fully self-contained Python
example that illustrates my problem.

I'm struggling to understand xml.etree's handling of namespaces. I'm trying to
parse an Inkscape document which uses several namespaces. From etree's
documentation:

If the XML input has namespaces, tags and attributes with prefixes in the
form prefix:sometag get expanded to {uri}sometag where the prefix is
replaced by the full URI.

Which means that given an Element e, I cannot directly access its attributes
using e.get() because in order to do that I need to know the URI of the
namespace. So rather than doing this (see example below):

label = e.get('inkscape:label')

I need to do this:

label = e.get('{' + uri_inkscape_namespace + '}label')

...which is the method mentioned in etree's docs:

One way to search and explore this XML example is to manually add the URI
to every tag or attribute in the xpath of a find() or findall().
[...]
A better way to search the namespaced XML example is to create a
dictionary with your own prefixes and use those in the search functions.

Good idea! Better yet, that dictionary or rather, its reverse, already exists,
because etree has used it to unnecessarily mangle the namespaces in the first
place. The documentation doesn't mention where it can be found, but we can
just use the 'xmlns:' attributes of the  root element to rebuild it. Or
so I thought, until I found out that etree deletes exactly these attributes
before handing the  element to the user.

I'm really stumped here. Apart from the fact that I think XML is bloated shit
anyway and has no place outside HTML, I just don't get the purpose of etree's
way of working:

1) Evaluate 'xmlns:' attributes of the  element
2) Use that info to replace the existing prefixes by {uri}
3) Realizing that using {uri} prefixes is cumbersome, suggest to
   the user to build their own prefix -> uri dictionary
   to undo the effort of doing 1) and 2)
4) ...but witholding exactly the information that existed in the original
   document by deleting the 'xmlns:' attributes from the  tag

Why didn't they leave the whole damn thing alone? Keep  intact and keep
the attribute 'prefix:key' literally as they are. For anyone wanting to use
the {uri} prefixes (why would they) they could have thrown in a helper
function for the prefix->URI translation.

I'm assuming that etree's designers knew what they were doing in order to make
my life easier when dealing with XML. Maybe I'm missing the forest for the
trees. Can anybody enlighten me? Thanks!


 self-contained example
import xml.etree.ElementTree as ET

def test_svg(xml):
root = ET.fromstring(xml)
for e in root.iter():
print(e.tag) # tags are shown prefixed with {URI}
if e.tag.endswith('svg'):
# Since namespaces are defined inside the  tag, let's use the info
# from the 'xmlns:' attributes to undo etree's URI prefixing
print('Element :')
for k, v in e.items():
print('  %s: %s' % (k, v))
# ...but alas: the 'xmlns:' attributes have been deleted by the parser

xml = '''


http://www.inkscape.org/namespaces/inkscape";
   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd";
   xmlns="http://www.w3.org/2000/svg";
   xmlns:svg="http://www.w3.org/2000/svg";>
  
  
  

  

'''

if __name__ == '__main__':
test_svg(xml)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: xml.etree and namespaces -- why?

[Robert Latest via Python-list]

Jon Ribbens wrote:
> That's because you *always* need to know the URI of the namespace,
> because that's its only meaningful identifier. If you assume that a
> particular namespace always uses the same prefix then your code will be
> completely broken. The following two pieces of XML should be understood
> identically:
>
> http://www.inkscape.org/namespaces/inkscape";>
>   
>
> and:
>
> http://www.inkscape.org/namespaces/inkscape";>
>   
>
> So you can see why e.get('inkscape:label') cannot possibly work, and why
> e.get('{http://www.inkscape.org/namespaces/inkscape}label') makes sense.

I get it. It does.

> The xml.etree author obviously knew that this was cumbersome, and
> hence you can do something like:
>
> namespaces = {'inkspace': 'http://www.inkscape.org/namespaces/inkscape'}
> element = root.find('inkspace:foo', namespaces)
>
> which will work for both of the above pieces of XML.

Makes sense. It forces me to make up my own prefixes which I can then safely
use in my code rather than relying on the xml's generator to not change their
prefixes.

BTW, I only now thought to look at what actually is at Inkscape's namespace
URI, and it turns out to be quite a nice explanation of what a namespace is and
why it looks like a URL.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Need help with custom string formatter

[Robert Latest via Python-list]

Hi Stefan, 

I have now implemented a version of this, works nicely. I have a few minor
questions / remarks:

>   result += ' ' *( length - len( result ))

Nice, I didn't know that one could multiply strings by negative numbers without
error.

> def __init__( self ):
> super().__init__()

Isn't this a no-op? Probably a leftover from my stuff.

> def format_field( self, value, format_string ):
> if re.match( r'\d+', format_string )and type( value )== str:

Why do you prefer re.match(r'\d+', x) over x.isdigit()?

> return super().format_field( value, format_string )

Why do you prefer super().format_field() over plain format()? The doc says:
"format_field() simply calls format()." So I figured I might do the same.

Thanks!
-- 
https://mail.python.org/mailman/listinfo/python-list

Need help with custom string formatter

[Robert Latest via Python-list]

Hi all,

I would like to modify the standard str.format() in a way that when the
input field is of type str, there is some character replacement, and the
string gets padded or truncated to the given field width. Basically like
this:

fmt = MagicString('<{s:6}>')
print(fmt.format(s='Äußerst'))

Output:


I've written a function fix_format() which, given a string and a field width,
does just that. However, I find myself unable to implement a Formatter that
uses this function in the intened way. See the example below, I hope I
sprinkled it with enough comments to make my intents clear.  Thanks for any
enlightenment. The interesting part starts somewhere in the middle.

### Self contained example
import re
from string import Formatter

_replacements = [(re.compile(rx), repl) for rx, repl in (\
('Ä', 'Ae'),
('ä', 'ae'),
('Ö', 'Oe'),
('ö', 'oe'),
('Ü', 'Ue'),
('ü', 'ue'),
('ß', 'ss'))]

def fix_format(text, width):

# Seven regex passes seems awfully inefficient. I can't think of a
# better way. Besides the point though.
for rx, repl in _replacements:
text = re.sub(rx, repl, text)

# return truncated / padded version of string
return text[:width] + ' ' * max(0, width - len(text))

class Obj():
"""I'm just an object with some attributes"""
def __init__(self, **kw):
self.__dict__.update(kw)

o = Obj(x="I am X, and I'm too long",
y="ÄÖÜ Ich bin auch zu lang")
z = 'Pad me!'

format_spec = '<{o.x:6}>\n<{o.y:6}>\n<{z:10}>'

# Standard string formatting
print('Standard string formatting:')
print(format_spec.format(o=o, z=z))

# Demonstrate fix_format()
print('\nWanted output:')
print('<' + fix_format(o.x, 6) + '>')
print('<' + fix_format(o.y, 6) + '>')
print('<' + fix_format(z, 10) + '>')

# This is where my struggle begins. #

class MagicString(Formatter):
def __init__(self, format_spec):
self.spec = format_spec
super().__init__()

def format(self, **kw):
return(self.vformat(self.spec, [], kw))

def get_field(self, name, a, kw):
# Compound fields have a dot:
obj_name, _, key = name.partition('.')
obj = getattr(kw[obj_name], key) if key else kw[obj_name]
if isinstance(obj, str):
# Here I would like to call fix_format(), but I don't know where
# to get the field width.
print('get_field(): <' + obj + '>')
else:
# Here I'd like to use the "native" formatter of whatever type
# the field is.
pass
return obj, key

def get_value(self, key, a, kw):
'''I don't understand what this method is for, it never gets called'''
raise NotImplementedError

fmt = MagicString(format_spec)
print('\nReal output:')
print(fmt.format(o=o, z=z))

# Weirdly, somewhere on the way the standard formatting kicks in, too, as
# the 'Pad me!' string does get padded (which must be some postprocessing,
# as the string is still unpadded when passed into get_field())

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Need help with custom string formatter

[Robert Latest via Python-list]

Stefan Ram wrote:

[the solution]

thanks, right on the spot. I had already figured out that format_field() is the
one method I need, and thanks for the str.translate method. I knew that raking
seven RE's across the same string HAD to be stupid.

Have a nice weekend!

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Need help with custom string formatter

[Robert Latest via Python-list]

Cameron Simpson wrote:
> Stefan's code implements it's own format_field and falls back to the 
> original format_field(). That's standard subclassing practice, and worth 
> doing reflexively more of the time - it avoids _knowing_ that 
> format_field() just calls format().
>
> So I'd take Stefan's statement above to imply that calling format() 
> directly should work.

Yup, makes sense.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why can't the pointer in a PyCapsule be NULL?

[Robert Latest via Python-list]

Stefan Ram wrote:
> Robert Latest  writes:
>>the question is in the subject. I'd like the pointer to be able to be NULL
>>because that would make my code slightly cleaner. No big deal though.
>
>   In Usenet, it is considered good style to have all relevant
>   content in the body.

Makes sense.

>   . On a superficial level, the answer is: "Because
>   PyCapsule_GetPointer uses NULL to indicate failure."

Makes sense, too. Thanks.
-- 
https://mail.python.org/mailman/listinfo/python-list

Why can't the pointer in a PyCapsule be NULL?

[Robert Latest via Python-list]

Hi all,

the question is in the subject. I'd like the pointer to be able to be NULL
because that would make my code slightly cleaner. No big deal though.
-- 
https://mail.python.org/mailman/listinfo/python-list

Line continuation and comments

[Robert Latest via Python-list]

I found myself building a complicated logical condition with many ands and ors
which I made more manageable by putting the various terms on individual lines
and breaking them with the "\" line continuation character. In this context it
would have been nice to be able to add comments to lines terms which of course
isn't possible because the backslash must be the last character on the line.

Question: If the Python syntax were changed to allow comments after line-ending
backslashes, would it break any existing code? I can't think of an example.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Line continuation and comments

[Robert Latest via Python-list]

Edmondo Giovannozzi wrote:
> Il giorno mercoledì 22 febbraio 2023 alle 09:50:14 UTC+1 Robert Latest ha
> scritto:
>> I found myself building a complicated logical condition with many ands and
>> ors which I made more manageable by putting the various terms on individual
>> lines and breaking them with the "\" line continuation character. In this
>> context it would have been nice to be able to add comments to lines terms
>> which of course isn't possible because the backslash must be the last
>> character on the line. 
>> 
>> Question: If the Python syntax were changed to allow comments after
>> line-ending 
>> backslashes, would it break any existing code? I can't think of an example.
>
> Well you can if you use parenthesis like in:
> x = 5
> a = (x > 3 and
> # x < 21 or
>  x > 100
>  )
> You don't need the "\" to continue a line in this case

I like that. Never thought of it.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Line continuation and comments

[Robert Latest via Python-list]

Paul Bryan wrote:
> Adding to this, there should be no reason now in recent versions of
> Python to ever use line continuation. Black goes so far as to state
> "backslashes are bad and should never be used":
>
> https://black.readthedocs.io/en/stable/the_black_code_style/
  future_style.html#using-backslashes-for-with-statements

Then I wonder how Mr. Black would go about these long "dot chaining"
expressions that packages like pandas and sqlalchemy require.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Line continuation and comments

[Robert Latest via Python-list]

Robert Latest wrote:
> Paul Bryan wrote:
>> Adding to this, there should be no reason now in recent versions of
>> Python to ever use line continuation. Black goes so far as to state
>> "backslashes are bad and should never be used":
>>
>> https://black.readthedocs.io/en/stable/the_black_code_style/
>   future_style.html#using-backslashes-for-with-statements
>
> Then I wonder how Mr. Black would go about these long "dot chaining"
> expressions that packages like pandas and sqlalchemy require.

Just found out that parentheses work there, too.
-- 
https://mail.python.org/mailman/listinfo/python-list

How to implement logging for an imported module?

[Robert Latest via Python-list]

Hello,

I'm trying to add logging to a module that gets imported by another module. But
I only get it to work right if the imported module knows the name of the
importing module. The example given in the "Logging Cookbook" also rely on this
fact.

https://docs.python.org/3/howto/logging-cookbook.html#logging-cookbook

I couldn't find any information on how to implement logging in a library that
doesn't know the name of the application that uses it. How is that done?

Here's some example code consisting of the main module foo.py and imported
modules bar.py and baz.py


### foo.py
import logging

import bar, baz

formatter = logging.Formatter(
'%(asctime)s - %(name)s - %(levelname)s - %(message)s')

ch = logging.StreamHandler()
ch.setFormatter(formatter)

logger = logging.getLogger('foo')
logger.addHandler(ch)
logger.setLevel(logging.DEBUG)

logger.debug('debug from <%s>', __name__)
logger.info('info from <%s>', __name__)
logger.warning('warning from <%s>', __name__)
logger.error('error from <%s>', __name__)

bar.func()
baz.func()

### bar.py
'''This "generic" approach doesn't honor loglevel or formats
when imported by another module'''
import logging

l = logging.getLogger(__name__)
def func():
l.debug('debug from <%s>', __name__)
l.info('info from <%s>', __name__)
l.warning('warning from <%s>', __name__)
l.error('error from <%s>', __name__)

### baz.py
'''This only works if the importing module is named 'foo', which
precludes its use as a library module'''
import logging

l = logging.getLogger('foo.baz')
def func():
l.debug('debug from <%s>', __name__)
l.info('info from <%s>', __name__)
l.warning('warning from <%s>', __name__)
l.error('error from <%s>', __name__)

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to implement logging for an imported module?

[Robert Latest via Python-list]

Joseph L. Casale wrote:
>> I couldn't find any information on how to implement logging in a library that
>> doesn't know the name of the application that uses it. How is that done?
>
> That's not how it works, it is the opposite. You need to know the name of its
> logger, and since you imported it, you do.

[much snipping]

> Last word of advice, don't fight it by hacking up or patching (somehow?), it
> will simply not work right for any other case even slightly different than
> the one you somehow beat into submission.

I didn't waht to hack the logging system, it's just that I wasn't sure of its
design principles. I had hoped that if I set up a logger (including levels and
formatter) in my main app, the loggers in the imported modules would somwhow
automagically follow suit. Now I understand that for each imported module I
must import its logger, too, and decide how to deal with its messages.

> I hope that helps,

Much appreciated,
robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why assert is not a function?

[Robert Latest via Python-list]

Chris Angelico (and oters) wrote:

[interesting stuff]

I'm a late contributor here, but I'd just say: If you'd prefer a function for
assert, just define a function that does what you want and be done with it.
Much harder to do if assert() were a built-in function but you'd rather have a
keyword ;-)

robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to implement logging for an imported module?

[Robert Latest via Python-list]

Richard Damon wrote:
> On 3/8/21 4:16 AM, Robert Latest via Python-list wrote:
>> Joseph L. Casale wrote:
>>>> I couldn't find any information on how to implement logging in a library
>>>> that doesn't know the name of the application that uses it. How is that
>>>> done?
>>> That's not how it works, it is the opposite. You need to know the name of
>>> its logger, and since you imported it, you do.
>> [much snipping]
>>
>>> Last word of advice, don't fight it by hacking up or patching (somehow?),
>>> it will simply not work right for any other case even slightly different
>>> than the one you somehow beat into submission.
>> I didn't waht to hack the logging system, it's just that I wasn't sure of
>> its design principles. I had hoped that if I set up a logger (including
>> levels and formatter) in my main app, the loggers in the imported modules
>> would somwhow automagically follow suit. Now I understand that for each
>> imported module I must import its logger, too, and decide how to deal with
>> its messages.
>>
>>
> Each instance of the logger inherents from a 'parent' logger, except for the
> top level logger which has the name None (as in the singleton of NoneType),
> and unless told otherwise will inherit it properties from its parent.
>
> Thus, if you get the root logger with logging.getLogger() you can set
> properties there, and unless a child logger has specifical been told not to
> inherit or has been specifically given a different value.
>
> General convention is that modules will use their name as the name of their
> logger, as that is generally unique.
>

I must admit I'm still struggling with the very basics of logging. I don't
understand the behavior of the code samples below at all, see comments.
It seems that logging.debug() et al have some side effects that are required
to get a logger to notice its level in the first place.


# Example 1:
# Why does the logger "mylog" require
# a call to the root logger in order to work?

import logging

mylog = logging.getLogger('foo')
mylog.setLevel(logging.DEBUG)

mylog.debug('1 mylog.debug()')   # prints nothing
logging.debug('2 logging.debug()') # prints nothing
mylog.debug('3 mylog.debug()')   # works


# Example 2:
# Why do I have to call 'logging.debug' before the root
# logger works?

import logging

mylog = logging.getLogger() # now mylog is the root logger
mylog.setLevel(logging.DEBUG) # setting level of root logger

mylog.debug('1 mylog.debug()')   # prints nothing, why?
logging.debug('2 logging.debug()') # works
mylog.debug('3 mylog.debug()')   # works

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: A 35mm film camera represented in Python object

[Robert Latest via Python-list]

D.M. Procida wrote:
> Hi everyone, I've created  -
> a representation of a Canonet G-III QL17 in Python.

[...]

> The Canonet G-III QL17 is one of my favourites. One of my reasons for
> writing this code is to appreciate the intricate mechanical logic
> embodied in the machine.

I love both photography with mechanical camears (The Nikon FE2 being my
favorite) and programming. So I absolutely love your project. Also I think its
totally nuts. I won't spend a second looking at your code or get otherwise
involved, but I wish you best of luck!

robert
-- 
https://mail.python.org/mailman/listinfo/python-list

[SQLAlchemy] Struggling with association_proxy

[Robert Latest via Python-list]

I'm trying to implement a many-to-many relationship that associates Baskets
with Items via an association object called Link which holds the quantity of
each item. I've done that in SQLAlchemy in a very pedestrian way, such as when
I want to have six eggs in a basket:

1. Find ID of Item with name 'egg'
2. See if there is an association object with the egg ID and the basket ID
3a. if yes, set its quantity to 6
3b if no, create it with quantity 6 and add it to the items colletion in basket

The association_proxy documentation suggests that this could be done elegantly
in such a way that I could simply write

basket.contents['egg'] = 6

and be done with it. I've tried to follow the documentation at 
https://docs.sqlalchemy.org/en/14/orm/extensions/associationproxy.html
but I don't understand it: It keeps creating new keyword instances rather
re-using existing ones, thus defeating the many-to-many idea. Here's what I've
come up so far, but it predictably fails because I don't want it to create new
Items on its own:

from sqlalchemy import create_engine, Column, Integer,\
String, ForeignKey
from sqlalchemy.ext.associationproxy import association_proxy
from sqlalchemy.orm import relationship, sessionmaker, backref
from sqlalchemy.orm.collections import attribute_mapped_collection
from sqlalchemy.ext.declarative import declarative_base

Base = declarative_base()

class Basket(Base):
__tablename__ = 'basket'
id= Column(Integer, primary_key=True)
contents = association_proxy('basket_contents', 'id')

class Link(Base):
__tablename__ = 'link'
item_id   = Column(ForeignKey('item.id'), primary_key=True)
basket_id = Column(ForeignKey('basket.id'), primary_key=True)
quantity  = Column(Integer)
basket = relationship(Basket, backref=backref('basket_contents',
collection_class=attribute_mapped_collection('quantity')))
item = relationship('Item')
name = association_proxy('item', 'name')

def __init__(self, name, quantity):
# this doesn't work b/c it calls Item.__init__() rather than
# looking for an existing Item
self.name = name
self.quantity = quantity

class Item(Base):
__tablename__ = 'item'
id= Column(Integer, primary_key=True)
name  = Column(String(10), unique=True)
weight= Column(Integer)
color = String(10)

engine = create_engine('sqlite://')
Base.metadata.create_all(engine)
Session = sessionmaker(engine)

db = Session()

egg = Item(name='egg', weight=50, color='white')

b = Basket()

# fails because in Link.__init__(), SQLAlchemy wants to create a new Item
# rather than using the existing one.
b.contents['egg'] = 6

db.add(b)

db.commit()

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .title() - annoying mistake

[Robert Latest via Python-list]

Mats Wichmann wrote:
> The problem is that there isn't a standard for title case,

The problem is that we owe the very existence of the .title() method to too
much weed being smoked during Python development. It makes specific assumptions
about a specific use case of one specific language. It doesn't get more
idiotic, frankly.

robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .title() - annoying mistake

[Robert Latest via Python-list]

Chris Angelico wrote:
> On Sun, Mar 21, 2021 at 4:31 AM Robert Latest via Python-list
> wrote:
>>
>> Mats Wichmann wrote:
>> > The problem is that there isn't a standard for title case,
>>
>> The problem is that we owe the very existence of the .title() method to too
>> much weed being smoked during Python development. It makes specific
>> assumptions about a specific use case of one specific language. It doesn't
>> get more idiotic, frankly.
>>
>
> The problem is that you haven't read the documentation :) It very carefully
> does NOT define itself by language, and its behaviour is identical regardless
> of the language used.

The documentation says: "The algorithm uses a simple language-independent
definition of a word as groups of consecutive letters..."

Yes, I get that. But the purpose it (improperly) serves only makes sense in the
English language. Which is also the reason they called it title() and not
capitalize_words(). Frankly, I can't think of any situation where this method
would have any use -- in any language, including English. It is just a
completely arbitrary feature, as would be a function that capitalizes only the
last letter of each word.

robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .title() - annoying mistake

[Robert Latest via Python-list]

Benjamin Schollnick wrote:
>
> I’m sorry Robert, but just because it doesn’t meet your requirements, doesn’t
> mean it’s useless.
>
> I use .title to normalize strings for data comparison, all the time.  It’s a
> perfect alternative to using .UPPER or .lower.  
>
> Right in the documentation, it specifically acknowledges .title working in
> foreign character sets. 
>
> So while I feel for the fact that it doesn’t met your requirements, please
> keep in mind, it does meet other peoples requirements.  
>
> As with everything here, it’s open source.  If you feel that there should be
> a revised version that does met your requirements create it, or gather a
> bunch of people and go the route of SCANDIR and open-source it, and petition
> that it be moved into the standard library.
>
> Since this seems to be bugging you this much, come up with a solution.  

I agree with everything you say. Especially the open source part. But wouldn't
you agree that .title() with all its arbitrary specificity to appear in the
very core of a general purpose language is quite an oddity?

robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .title() - annoying mistake

[Robert Latest via Python-list]

Chris Angelico wrote:
> On Sun, Mar 21, 2021 at 10:31 PM Robert Latest via Python-list
> wrote:
>> Yes, I get that. But the purpose it (improperly) serves only makes sense in
>> the English language.
>
> Why? Do titles not exist in other languages? Does no other language
> capitalize words in book or other titles?

I wonder if .title() properly capitalizes titles in any language. It doesn't in
English (nor does it purport to), so it begs the question why it is there in
the first place. German and Spanish don't have any special capitalization rules
for titles; I don't know about any other languages. 

robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .title() - annoying mistake

[Robert Latest via Python-list]

Benjamin Schollnick wrote:
>
>> I agree with everything you say. Especially the open source part. But
>> wouldn't you agree that .title() with all its arbitrary specificity to
>> appear in the very core of a general purpose language is quite an oddity?
>
> No, because it book ends the issue.
>
> Upper - Converts everything to uppercase Lower - Converts everything to
> lowercase
>
> Title - Covers the cases in-between upper/lower.  

My only issue is that I completely fail to see how this function would be
useful enough to warrant the inclusion into the *core* of a general-purpose
language, including its misleading name. The fact that the function's behavior
is correctly documented doesn't make its very existence less bewildering to me.
Consider this function:

def add_seventeen(n): '''Return n with 16.8 added''' return n + 16.8

It's like .title(): It does almost the thing its name suggests, it is correctly
documented, it is useful to anybody who happens to want 16.8 added to numbers,
and it might erroneously be used by someone who wants exactly 17 added and
didn't bother to read the docs.

> And as I mentioned the sheer amount of work that would be needed would
> probably cover a phd dissertation or more…  It’s a huge problem set to
> respect one language, let alone all languages.  

And that's why I believe that such a function should be delegated to a natural
language-processing library and not the core of Python.

> So the only answer is to not respect the languages, and leave that up to a
> later implementation or for someone else to assist in adding in support.

And that too.

> Heck, how do we prevent it from titlecasing abbreviations?  (This is plain
> text not XML….  If it was XML it would be easy!)

And that too.

BTW I have no beef with the person who invented .title() nor with anybody who
uses it. I know that everybody can join the Python development community and
propose the removal of .title() and the inclusion of add_seventeen(). That
said, I doubt that .title() would make it into Python today if it weren't there
already. I'm having fun with this.

robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .title() - annoying mistake

[Robert Latest via Python-list]

Grant Edwards wrote:
> On 2021-03-20, Robert Latest via Python-list  wrote:
>> Mats Wichmann wrote:
>>> The problem is that there isn't a standard for title case,
>>
>> The problem is that we owe the very existence of the .title() method to too
>> much weed being smoked during Python development. It makes specific
>> assumptions about a specific use case of one specific language. It doesn't
>> get more idiotic, frankly.
>
> Ah, you've never used PHP then?
>
> I haven't checked but it's a fair bit that PHP has 3-4 different built-in
> ways to do it, and they're all broken in interestingly unpredictable ways.

I believe that 100%. PHP is the reason I switched to Python/WSGI, and I'm
loving it. 

robert
-- 
https://mail.python.org/mailman/listinfo/python-list

How to set up a 'listening' Unix domain socket

[Robert Latest via Python-list]

Hello,

I'm trying to set up a server that receives data on a Unix domain socket using
the code below.

import os from socketserver import UnixStreamServer, StreamRequestHandler

SOCKET = '/tmp/test.socket'

class Handler(StreamRequestHandler):

def handle(self): data = selr.rfile.read() print(data)

if os.path.exists(SOCKET): os.unlink(SOCKET) with UnixStreamServer(SOCKET,
Handler) as server: server.serve_forever()


However, when I try to send somthing to that socket, I get this error message:

$ echo "Hello" | socat - UNIX-SENDTO:/tmp/test.socket 2021/03/22 11:03:22
socat[2188] E sendto(5, 0x55a22f414990, 6, 0, AF=1 "/tmp/test.socket", 18):
Protocol wrong type for socket

I don't understand that error. Here's /tmp/test.socket:

$ stat /tmp/test.socket
  File: ‘/tmp/test.socket’
  Size: 0   Blocks: 0  IO Block: 4096   socket
Device: fd00h/64768dInode: 201443577   Links: 1
Access: (0775/srwxrwxr-x)  Uid: ( 1001/  dh)   Gid: ( 1001/  dh)
Context: unconfined_u:object_r:user_tmp_t:s0
Access: 2021-03-22 09:47:49.516298102 +0100
Modify: 2021-03-22 09:47:49.516298102 +0100
Change: 2021-03-22 09:47:49.516298102 +0100
 Birth: -

Sadly all examples I can find on the web are for TCP sockets, not Unix domain.

Any tips?
robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .title() - annoying mistake

[Robert Latest via Python-list]

Chris Angelico wrote:
> If you still, after all these posts, have not yet understood that
> title-casing *a single character* is a significant thing,

I must admit I have no idea what title-casing even is, but I'm eager to learn.
The documentation only talks about "words" and "first characters" and
"remaining characters." So a single character gets converted to uppercase,
whatever that may mean in the context of .title(). The .upper() method is
different in that it only applies to "cased" characters, so .title() may or may
not work differently on single characters. 

robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to set up a 'listening' Unix domain socket

[Robert Latest via Python-list]

Chris Angelico wrote:
>
> Hmm, your formatting's messed up, but the code looks fine to me. (Be aware
> that you seem to have a "selr" where it should be "self".)

Didn't catch that because my program didn't even get to that point ;-)

>
>> However, when I try to send somthing to that socket, I get this error
>> message:
>>
>> $ echo "Hello" | socat - UNIX-SENDTO:/tmp/test.socket 2021/03/22 11:03:22
>> socat[2188] E sendto(5, 0x55a22f414990, 6, 0, AF=1 "/tmp/test.socket", 18):
>> Protocol wrong type for socket
>>
>
> Not familiar with socat, but here's some simple Python code to trigger your
> server:
>
 import socket sock = socket.socket(socket.AF_UNIX)
 sock.connect("/tmp/test.socket") sock.send(b"Hello, world")
> 12
 sock.close()


Works perfectly, thanks! I'm probably not using socat right.

robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .title() - annoying mistake

[Robert Latest via Python-list]

Chris Angelico wrote:
> There are a small number of characters which, when case folded, become
> more than one character. The sharp S from German behaves thusly:
>
 "ß".upper(), "ß".lower(), "ß".casefold(), "ß".title()
> ('SS', 'ß', 'ss', 'Ss')

Now we're getting somewhere. I'm a native German speaker and I can tell you
that this doesn't happen in the real world, simply because 'ß' never appears at
the beginning of a word and thus is never "title cased". The only occurence of
uppercase 'ß' is in all-caps text, which Python handles properly:

'bißchen'.upper()
'BISSCHEN'

...that is, properly until 2008, when the capital 'ß' was officially introduced
into German ortography:

https://en.wikipedia.org/wiki/%C3%9F#Capital_form
"Traditionally, ß did not have a capital form, although some type designers
introduced de facto capitalized variants of ß. In 2017, the Council for German
Orthography ultimately adopted capital ß, ẞ, into German orthography, ending a
long orthographic debate.[3] [...] The capital variant (U+1E9E ẞ LATIN CAPITAL
LETTER SHARP S) was encoded by ISO 10646 in 2008." So Python 3.6.8 is about 12
years behind.

As a German I also appreciate the reduced occurence of the letter combination
'SS'.

That said, the concept of "title casing" doesn't even exist in German. Titles
are spelt just like any regular sentence. I know only two definitions of the
concept "title case":

1) From Wikipedia
"Title case or headline case is a style of capitalization used for rendering
the titles of published works or works of art in English. [...]"

2) From Python (paraphrased):
"Perform an arbitrary (but defined) operation on the characters of a string."

I don't mind .title() being in Python. I would very much mind to be the person
in charge of maintaining it and having to port it into new versions of Python,
always keeping an eye on the evolution of Unicode or other standards (see
above).

It probably just comes down to me not being able to conjure up a single
sensible use case for .title() as well as the whole concept of "title casing"
in the context of a programming language.

> The neat thing about Unicode is 

[many things]

> The documentation sometimes shorthands things with terms like "upper
> case" and "lower case", but that's partly because being pedantically
> correct in a docstring doesn't actually help anything, and the code
> itself IS correct.

...but hard to maintain and useless. I just love to hate .title() ;-)

robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .title() - annoying mistake

[Robert Latest via Python-list]

Karsten Hilbert wrote:
> and life with that wart.

Perfectly willing to as long as everybody agrees it's a wart ;-)

robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to set up a 'listening' Unix domain socket

[Robert Latest via Python-list]

> Chris Angelico wrote:

[Helpful stuff]

I'm actually trying to implement a SocketHandler for a Python logger. However,
I can't get my handler on the client side to send anything. Do I need to
subclass logging.SocketHandler and fill the various methods with meaning? The
documentation doesn't say so.

I've noticed that when the server is not running, logging.SocketHandler creates
the socket. I don't understand why it would; isn't it the server's job to
create the socket?

## CLIENT 

import socket
import pickle
from logging import getLogger
from logging.handlers import SocketHandler

SOCKET = '/tmp/test.socket'

# This doesn't send anything to the socket
_log = getLogger(__name__)
_log.addHandler(SocketHandler(None, SOCKET))
_log.error('Logging something')

# String gets sent to socket
sock = socket.socket(socket.AF_UNIX)
sock.connect(SOCKET)
sock.send(b"Hello, world")
sock.close()

# Pickled object works fine
sock = socket.socket(socket.AF_UNIX)
sock.connect(SOCKET)
sock.send(pickle.dumps(dict(a=1)))
sock.close()


## SERVER 

import os
import pickle
from socketserver import UnixStreamServer, StreamRequestHandler

SOCKET = '/tmp/test.socket'

class Handler(StreamRequestHandler):

def handle(self):
data = self.rfile.read()
try:
obj = pickle.loads(data)
except:
print('Unpickled: ', data)
else:
print('Pickled: ', type(obj))

if os.path.exists(SOCKET):
os.unlink(SOCKET)
with UnixStreamServer(SOCKET, Handler) as server:
server.serve_forever()

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .title() - annoying mistake

[Robert Latest via Python-list]

Chris Angelico wrote:
> Cool thing is, nobody in Python needs to maintain anything here.

That's great because I'm actually having trouble with sending log messages over
the socket conection you helped me with, would you mind having a look?

Regards,
robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .title() - annoying mistake

[Robert Latest via Python-list]

Benjamin Schollnick wrote:

> I’m sorry, but it’s as if he’s arguing for the sake of arguing.  It’s
> starting to feel very unproductive, and unnecessary.

That was never five minutes just now!

robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Ad-hoc SQL query builder for Python3?

[Robert Latest via Python-list]

Rich Shepard wrote:
> For those interested I've found a couple of possibilities: PyPika and
> SQLbuilder. I'll be looking deeply into them to learn their capabilities.

In case nobody mentioned it before, don't forget to take a look at SQLAlchemy.
The object-relational-mapper (ORM) creates a 1:1 mapping of Python objects to
SQL table rows.

-- 
robert
-- 
https://mail.python.org/mailman/listinfo/python-list

How to "cast" an object to a derived class?

[Robert Latest via Python-list]

Hi all, let's assume I'm using a module that defines some class "Opaque" and
also a function that creates objects of that type. In my program I subclass
that type because I want some extra functionality. But how can I "promote" a
given Opaque instance to the derived class? Of course I could just create an
instance of the derived type and copy all attributes from the original object
to the new one, but that either breaks when the opaque.py module changes, or it
requires introspection. It's easily done of course but seems overly clumsy. Is
there a more Pythonic way?

# this is the module opaque.py
class Opaque(): def __init__(self, x): assert isinstance(x, int) self.n = x

def make_opaque(): return Opaque(0)

# this is my program
import opaque class MyClass(opaque.Opaque):
# generic __init__ b/c I don't want to have to know anything about that
# class
def __init__(self, *a, **k): super().__init__(*a, *k) self.__something = 0

def get_something(self): return self.something

op = opaque.make_opaque()

# But I want to have this as type MyClass. This obviously doesn't work:
my_object = MyClass(op)
# But what does?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to "cast" an object to a derived class?

[Robert Latest via Python-list]

Stefan Ram wrote:
> Robert Latest  writes: But how can I "promote" a
>>given Opaque instance to the derived class?
>
>   Sometimes, one can use containment instead of inheritance.

Nah, doesn't work in my case. I'm trying to write a wrapper around
xml.etree.ElemenTree and .Element  to circumvent its idiotic namespace
handling. For that I need inheritance since I want to override the find..()
functions to return my derived MyElement classes. I think it could work but I
somehow need to convert the root Element to a MyElement.

-- 
https://mail.python.org/mailman/listinfo/python-list

Type annotation pitfall

[Robert Latest via Python-list]

Hi all,

this just caused me several hours of my life until I could whittle it down to
this minimal example. Simple question: Why is the x member of object "foo"
modified by initializing "bar"?

Obviously, initializing foo with None doesn't set foo.x at all. So I guess x
stays a class property, not an instance property. And instantiating bar just
added an item to the class property but didn't instantiate a new set. So
basically, neither foo nor bar ever had their "own" x, right?

Oooohh, dangerous! Never use mutable types in type hint, unless it's in
explicit dataclasses (which automatically add an appropriate __init__()?)

Now I must fine-comb all my code for more of these.

class Foo():
x : set = set()

def __init__(self, s):
if s:
self.x.add(s)

foo = Foo(None)
print(foo.x) # prints 'set()'
bar = Foo('abc')
print(foo.x) # prints '{'abc'}

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Advanced ways to get object information from within python

[Robert Latest via Python-list]

Julius Hamilton wrote:
> dir(scrapy) shows this:
>
> ['Field', 'FormRequest', 'Item', 'Request', 'Selector', 'Spider',
> '__all__', '__builtins__', '__cached__', '__doc__', '__file__',
> '__loader__', '__name__', '__package__', '__path__', '__spec__',
> '__version__', '_txv', 'exceptions', 'http', 'item', 'link',
> 'linkextractors', 'selector', 'signals', 'spiders', 'twisted_version',
> 'utils', 'version_info']
>
> I wish there was a convenient way for me to know what all of these are.

['%s: %s' % (x, type(getattr(scrapy, x))) for x in dir(scrapy)]

-- 
https://mail.python.org/mailman/listinfo/python-list

"undefined symbol" in C extension module

[Robert Latest via Python-list]

Hi guys,

I've written some CPython extension modules in the past without problems. Now
after moving to a new Archlinux box with Python3.10 installed, I can't build
them any more. Or rather, I can build them but not use them due to "undefined
symbols" during linking. Here's ldd's output when used on the "spam" example
library from the docs:

linux-vdso.so.1 (0x7ffe2454a000)
libc.so.6 => /usr/lib/libc.so.6 (0x7fb6b6eb9000)
/usr/lib64/ld-linux-x86-64.so.2 (0x7fb6b70a4000)
undefined symbol: PyObject_Init (./build/lib.linux-x86_64-3.10/
spam.cpython-310-x86_64-linux-gnu.so)

[and more of the same]

It seems that I need to install some library to make this work, but which one?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: "undefined symbol" in C extension module

[Robert Latest via Python-list]

Dan Stromberg wrote:
> Perhaps try:
> https://stromberg.dnsalias.org/svn/find-sym/trunk
>
> It tries to find symbols in C libraries.
>
> In this case, I believe you'll find it in -lpythonx.ym

Thanks! Found out that ldd produces many errors also with working python
libraries. Turns out I tried to revive a package from v2.7. Need to adapt
it to 3.x
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Long running process - how to speed up?

[Robert Latest via Python-list]

Shaozhong SHI wrote:
> Can it be divided into several processes?

I'd do it like this:

from time import sleep
from threading import Thread

t = Thread(target=lambda: sleep(1))
t.run()

# do your work here

t.wait()

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Best way to check if there is internet?

[Robert Latest via Python-list]

Abdur-Rahmaan Janhangeer wrote:
> Well, nice perspective.
>
> It's a valid consideration, sound theory
> but poor practicality according to me.

On the contrary: It is absolutely the right and only way to do it.

> It you view it like this then between the moment
> we press run or enter key on the terminal,
> our Python interpreter might get deleted.

Yeah, but by your logic you'd have to check the existence of /usr/bin/python
each time before you run it.

> Though it's nice to go in and catch exceptions,
> if you have a long operation upcoming it might be nice
> to see if your key element is present.

That is true, but you still must make sure that your long operation fails in a
controlled manner if the necessary ressource becomes unavailable in the
meantime.

> Checking by exceptions is what the snippet
> i shared does. But we check for the ability to do it
> before we start.

Nothing wrong with checking before.

> Of course, it can change in the seconds that follow. But it's too much pure
> logic at work.

No. It's the correct way of doing it.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Best way to check if there is internet?

[Robert Latest via Python-list]

Abdur-Rahmaan Janhangeer wrote:
> I've got my answers but the 'wandering' a bit
> on this thread is at least connected to the topic ^^.
>
> Last question: If we check for wifi or ethernet cable connection?
>
> Wifi sometimes tell of connected but no internet connection.
> So it detected that there is or there is no internet ...

Your application obviously needs to make a certain network connection. Why
don't you just check if the connection can be made? Why would you care if it's
via cable or Wifi?
-- 
https://mail.python.org/mailman/listinfo/python-list

Threading question .. am I doing this right?

[Robert Latest via Python-list]

I have a multi-threaded application (a web service) where several threads need
data from an external database. That data is quite a lot, but it is almost
always the same. Between incoming requests, timestamped records get added to
the DB.

So I decided to keep an in-memory cache of the DB records that gets only
"topped up" with the most recent records on each request:


from threading import Lock, Thread


class MyCache():
def __init__(self):
self.cache = None
self.cache_lock = Lock()

def _update(self):
new_records = query_external_database()
if self.cache is None:
self.cache = new_records
else:
self.cache.extend(new_records)

def get_data(self):
with self.cache_lock:
self._update()

return self.cache

my_cache = MyCache() # module level


This works, but even those "small" queries can sometimes hang for a long time,
causing incoming requests to pile up at the "with self.cache_lock" block.

Since it is better to quickly serve the client with slightly outdated data than
not at all, I came up with the "impatient" solution below. The idea is that an
incoming request triggers an update query in another thread, waits for a short
timeout for that thread to finish and then returns either updated or old data.

class MyCache():
def __init__(self):
self.cache = None
self.thread_lock = Lock()
self.update_thread = None

def _update(self):
new_records = query_external_database()
if self.cache is None:
self.cache = new_records
else:
self.cache.extend(new_records)

def get_data(self):
if self.cache is None:
timeout = 10 # allow more time to get initial batch of data
else:
timeout = 0.5
with self.thread_lock:
if self.update_thread is None or not self.update_thread.is_alive():
self.update_thread = Thread(target=self._update)
self.update_thread.start()
self.update_thread.join(timeout)

return self.cache

my_cache = MyCache()

My question is: Is this a solid approach? Am I forgetting something? For
instance, I believe that I don't need another lock to guard self.cache.append()
because _update() can ever only run in one thread at a time. But maybe I'm
overlooking something.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Threading question .. am I doing this right?

[Robert Latest via Python-list]

Chris Angelico wrote:
> Depending on your database, this might be counter-productive. A
> PostgreSQL database running on localhost, for instance, has its own
> caching, and data transfers between two apps running on the same
> computer can be pretty fast. The complexity you add in order to do
> your own caching might be giving you negligible benefit, or even a
> penalty. I would strongly recommend benchmarking the naive "keep going
> back to the database" approach first, as a baseline, and only testing
> these alternatives when you've confirmed that the database really is a
> bottleneck.

"Depending on your database" is the key phrase. This is not "my" database that
is running on localhost. It is an external MSSQL server that I have no control
over and whose requests frequently time out.

> Hmm, it's complicated. There is another approach, and that's to
> completely invert your thinking: instead of "request wants data, so
> let's get data", have a thread that periodically updates your cache
> from the database, and then all requests return from the cache,
> without pinging the requester. Downside: It'll be requesting fairly
> frequently. Upside: Very simple, very easy, no difficulties debugging.

I'm using a similar approach in other places, but there I actually have a
separate process that feeds my local, fast DB with unwieldy data. But that is
not merely replicating, it actually preprocesses and "adds value" to the data,
and the data is worth retaining on my server. I didn't want to take that
approach in this instance because it is a bit too much overhead for essentially
"throwaway" stuff. I like the idea of starting a separated "timed" thread in
the same application. Need to think about that.

Background: The clients are SBCs that display data on screens distributed
throughout a manufacturing facility. They periodically refresh every few
minutes. Occasionally the requests would pile up waiting for the databsase, so
that some screens displayed error messages for a minute or two. Nobody cares
but my pride was piqued and the error logs filled up.

I've had my proposed solution running for a few days now without errors. For me
that's enough but I wanted to ask you guys if I made some logical mistakes.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Threading question .. am I doing this right?

[Robert Latest via Python-list]

Greg Ewing wrote:
> * If more than one thread calls get_data() during the initial
> cache filling, it looks like only one of them will wait for
> the thread -- the others will skip waiting altogether and
> immediately return None.

Right. But that needs to be dealt with somehow. No data is no data.

> * Also if the first call to get_data() times out it will
> return None (although maybe that's acceptable if the caller
> is expecting it).

Right. Needs to be dealt with.

> * The caller of get_data() is getting an object that could
> be changed under it by a future update.

I don't think that's a problem. If it turns out to be one I'll create a copy of
the data while I hold the lock and pass that back.
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Best way to check if there is internet?

[Robert Latest via Python-list]

Chris Angelico wrote:
> Every language learns from every other.

Except Visual Basic, which didn't learn anything from anywhere, and all that
can be learned from it is how not to do it. Ugh.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Threading question .. am I doing this right?

[Robert Latest via Python-list]

Chris Angelico wrote:
> I'm still curious as to the workload (requests per second), as it might still
> be worth going for the feeder model. But if your current system works, then
> it may be simplest to debug that rather than change.

It is by all accounts a low-traffic situation, maybe one request/second. But
the view in question opens four plots on one page, generating four separate
requests. So with only two clients and a blocking DB connection, the whole
application with eight uwsgi worker threads comes down. Now with the "extra
load thread" modification, the app worked fine for several days with only two
threads.

Out of curiosity I tried the "feeder thread" approach with a dummy thread that
just sleeps and logs something every few seconds, ten times total. For some
reason it sometimes hangs after eight or nine loops, and then uwsgi cannot
restart gracefully probably because it is still waiting for that thread to
finish. Also my web app is built around setting up the DB connections in the
request context, so using an extra thread outside that context would require
doubling some DB infrastructure. Probably not worth it at this point.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: SQLAlchemy: JSON vs. PickleType vs. raw string for serialised data

[Robert Latest via Python-list]

Albert-Jan Roskam wrote:
>  The event may have arbitrary, but dict-like data associated with it,
>  which I want to add in the field 'info'.  This data never needs to be
>  modified, once the event has been inserted into the DB.
>
>  What type should the info field have?  JSON, PickleType, String, or
>  something else?
>
>  I couldn't find any really reliable sounding information about the
>  relative
>  pros and cons, apart from a Reddit thread claiming that pickled dicts
>  are larger than dicts converted to JSON or String.

I've done exactly this. Since my data was strictly ASCII I decided to go for
JSON. But in the end you're the only one who can decide this because only you
know the data. That's why you won't find any hard and fast rule for this.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: SQLAlchemy: JSON vs. PickleType vs. raw string for serialised data

[Robert Latest via Python-list]

Loris Bennett wrote:
> Thanks for the various suggestions.  The data I need to store is just a
> dict with maybe 3 or 4 keys and short string values probably of less
> than 32 characters each per event.  The traffic on the DB is going to be
> very low, creating maybe a dozen events a day, mainly triggered via a
> command-line interface, although I will probably set up one or two cron
> jobs, each of which might generate another 0 to maybe 5 records a day.
>
> I could go for JSON (or rather LONGSTRING, as JSON is just an alias for
> LONGSTRING, but JSON is not available on the version of MariaDB I am
> using).  However, that seems like overkill, since I am never going to
> have to store anything near 4 GB in the field.  So I should probably in
> fact just use say VARCHAR(255).
>
> WDYT?

Using TypeDecorator to transparently convert between a dict and its JSON string
representation and MutableDict to track changes, you will get a completely
transparent attribute that works just like a dict. Make sure to check that the
generated JSON fits into your column width. I once got bitten by the fact that
VARCHAR(x) can hold only x/4 characters in utf8mb4 character set.
-- 
https://mail.python.org/mailman/listinfo/python-list

Weird side effect of default parameter

[Robert Latest via Python-list]

Hello,

I don't understand the behavior of the code below. Why does the dict property
"a" of both objects contain the same keys? This is only if "a=dict" is in
the initializer. If I put self.a = dict() into the init function, I get two
separate dicts



class Foo(object):
def __init__(self, x, a=dict()):
self.x = x
self.a = a
self.a[x] = x


c = Foo(1)
d = Foo(2)

print(c.__dict__)
print(d.__dict__)


robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Weird side effect of default parameter

[Robert Latest via Python-list]

Steven D'Aprano wrote:
> Python function default values use *early binding*: the default parameter 
> is evaluated, ONCE, when the function is defined, and that value is used 
> each time it is needed.

Thanks, "early binding" was the clue I was missing.

robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Package directory question

[Robert Latest via Python-list]

Hello,

I'm building an application which consists of two largely distinct
parts, a frontend and a backend. The directory layout is like this:

|-- jobwatch
|   |-- backend
|   |   |-- backend.py
|   |   |-- __init__.py
|   |   `-- tables.py
|   |-- frontend
|   |   |-- __init__.py
|   |   |-- main.py
|   `-- __init__.py
`-- setup.py

Because the main.py script needs to import the tables.py module from
backend, I put this at the top if main.py:

   sys.path.append('../..')
   import jobwatch.backend.tables as tables

My question is: Is this the way it should be done? It looks fishy. The
only alternative I could come up with is to put a symlink to tables.py
into the frontend directory, which also seems fishy. Eventually I want
to package all this up neatly to be able to use only little wrapper
scripts for the backend (running as a service) and the frontend (a wsgi
app).

Any thoughts?

Thanks
robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Package directory question

[Robert Latest via Python-list]

Ben Finney wrote:
> Robert Latest via Python-list  writes:
>
>> Because the main.py script needs to import the tables.py module from
>> backend, I put this at the top if main.py:
>>
>>sys.path.append('../..')
>>import jobwatch.backend.tables as tables
>>
>> My question is: Is this the way it should be done? It looks fishy. The
>> only alternative I could come up with is to put a symlink to tables.py
>> into the frontend directory, which also seems fishy.
>
> Your fish-sense is working correctly. Both of those are hard-coding the
> path, when the Python import mechanism is designed so you don't do that.

[...]

> * To install for use while also developing, add the ‘--editable’ option.

Ah, that's what I needed. Of course the problem I had was only present
during development. I haven't really looked into pip yet, so far I've
been using only "python setup.py install".

Thanks,
robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Python2.7 unicode conundrum

[Robert Latest via Python-list]

Hi folks,
what semmingly started out as a weird database character encoding mix-up
could be boiled down to a few lines of pure Python. The source-code
below is real utf8 (as evidenced by the UTF code point 'c3 a4' in the
third line of the hexdump). When just printed, the string "s" is
displayed correctly as 'ä' (a umlaut), but the string representation
shows that it seems to have been converted to latin-1 'e4' somewhere on
the way.
How can this be avoided?

dh@jenna:~/python$ cat unicode.py
# -*- encoding: utf8 -*-

s = u'ä'

print(s)
print((s, ))

dh@jenna:~/python$ hd unicode.py 
  23 20 2d 2a 2d 20 65 6e  63 6f 64 69 6e 67 3a 20  |# -*- encoding: |
0010  75 74 66 38 20 2d 2a 2d  0a 0a 73 20 3d 20 75 27  |utf8 -*-..s = u'|
0020  c3 a4 27 0a 0a 70 72 69  6e 74 28 73 29 0a 70 72  |..'..print(s).pr|
0030  69 6e 74 28 28 73 2c 20  29 29 0a 0a  |int((s,))..|
003c
dh@jenna:~/python$ python unicode.py
ä
(u'\xe4',)
dh@jenna:~/python$



-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python2.7 unicode conundrum

[Robert Latest via Python-list]

Richard Damon wrote:
> Why do you say it has been convert to 'Latin'. The string prints as
> being Unicode. Internally Python doesn't store strings as UTF-8, but as
> plain Unicode (UCS-2 or UCS-4 as needed), and code-point E4 is the
> character you want.

You're right, this wasn't the minimal example for my problem after all.
Turns out that the actual issue is somewhere between SQLAlchemy and
MySQL. I took a more specific questioon overt to stackoverflow.com

Thanks
robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Why doesn't this method have access to its "self" argument?

[Robert Latest via Python-list]

Hi all,

I'm still trying to find a simple way to implement a method (but not the
full class) in C. Suggestions I have recived so far are good, but seem to be
over the top for such a (seemingly) simple problem. I came up with the idea
of implementing the class in Python, and just setting the method after the
actual class definition. See example below. The "py_method" is added to the
class later on, and it seems to become a proper method of the class (and its
instances) just as if it had been defined inside the class itself. Of course
it also has access to its "self" argument.

If I do the same thing with a method implemented in the external module
"cmethod", and I call that method on an instance, it gets passed a NULL
pointer as its "self" argument and so has no access to the instance's "data"
attribute.

I found a workaround using a wrapper method which calls a C function,
passing the instance as a separate argument. It works, and I cannot see any
disadvantage. It's just not as elegant as I'd like it to be, and I don't
understand WHY the C "method" doesn't receive a pointer to the Python
instance. Maybe somebody can clarify.


Here's what happens when I build, install and run the minimal example below:

rl@dc:~/c/cwsf/python_module$ python test.py
py_method(): <__main__.Test instance at 0xb71f5a2c>
c_method(): self at (nil)
c_function(): self at (nil), instance at 0xb71f5a2c
<__main__.Test instance at 0xb71f5a2c>
'Some data'
c_function(): self at (nil), instance at 0xb71f5a2c
<__main__.Test instance at 0xb71f5a2c>
'New Data'
rl@dc:~/c/cwsf/python_module$

Minimal example files:


== test.py ===

import cmethod

class Test():
def __init__(self):
self.data = "Some data"

def wrapper(self):
return cmethod.c_function(self)

def py_method(self):
print "py_method(): %s" % repr(self)

# add methods to the class "after the fact"
Test.py_method = py_method
Test.c_method = cmethod.c_method

foo = Test()
foo.py_method() # works as expected
foo.c_method()  # is passed NULL in its self argument, why?
foo.wrapper()   # works fine, updates "data" attribute
foo.wrapper()   # works fine, sees updated "data" attribute

 cmethod.c ===

#include 
#include 

static PyObject *c_method(PyObject *self, PyObject *args) {
(void) args; /* not used */
fprintf(stderr, "c_method(): self at %p\n",
(void*) self); /* always prints 0x0 */
Py_INCREF(Py_None);
return Py_None;
}

static PyObject *c_function(PyObject *self, PyObject *args) {
PyObject *instance;
PyObject *o;
/* Retrieve instance from wrapper through argument tuple: WORKS */
PyArg_UnpackTuple(args, "c_function", 1, 1, &instance);
fprintf(stderr, "c_function(): self at %p, instance at %p\n",
(void*) self, (void*) instance);
PyObject_Print(instance, stderr, 0); fputc('\n', stderr);
/* Get and set attributes of instance: WORKS */
o = PyObject_GetAttrString(instance, "data");
PyObject_Print(o, stderr, 0); fputc('\n', stderr);
PyObject_SetAttrString(instance, "data",
PyString_FromString("New Data"));
/* Side question: Do I have to DECREF "o"? */
Py_INCREF(Py_None);
return Py_None;
}

static PyMethodDef methods[] =
{
 {"c_method", c_method, METH_VARARGS, "I want to be a class method"},
 {"c_function", c_function, METH_VARARGS, "I am a function"},
 {NULL, NULL, 0, NULL}
};

PyMODINIT_FUNC initcmethod(void)
{
 (void) Py_InitModule("cmethod", methods);
}


 setup_cmethod.py ===

from distutils.core import setup, Extension

module1 = Extension('cmethod', sources = ['cmethod.c'])

setup (name = 'cmethod',
version = '0.1',
description = 'struggling to implement a class method in C',
ext_modules = [module1])


robert
-- 
https://mail.python.org/mailman/listinfo/python-list