Reportlab / platypus bug?
Hello,
I have found an error, and I created a minimal working example. The minimal
working example starts with the very first example from Platypus user guide:
from reportlab.platypus import SimpleDocTemplate, Paragraph, Spacer
from reportlab.lib.styles import getSampleStyleSheet
from reportlab.lib.pagesizes import A4
from reportlab.lib.units import inch
PAGE_HEIGHT = A4[1]
PAGE_WIDTH = A4[0]
styles = getSampleStyleSheet()
Title = "Hello world"
pageinfo = "platypus example"
def myFirstPage(canvas, doc):
canvas.saveState()
canvas.setFont('Times-Bold', 16)
canvas.drawCentredString(PAGE_WIDTH / 2.0, PAGE_HEIGHT - 108, Title)
canvas.setFont('Times-Roman', 9)
canvas.drawString(inch, 0.75 * inch, "First Page / %s" % pageinfo)
canvas.restoreState()
def myLaterPages(canvas, doc):
canvas.saveState()
canvas.setFont('Times-Roman', 9)
canvas.drawString(inch, 0.75 * inch, "Page %d %s" % (doc.page, pageinfo))
canvas.restoreState()
def go():
Story = [Spacer(1, 2 * inch)]
style = styles["Normal"]
for i in range(100):
bogustext = ("This is Paragraph number %s. " % i) * 20
p = Paragraph(bogustext, style)
Story.append(p)
Story.append(Spacer(1, 0.2 * inch))
doc = SimpleDocTemplate("phello.pdf")
doc.build(Story, onFirstPage=myFirstPage, onLaterPages=myLaterPages)
go()
If I change it to this (e.g. generate two identical files):
doc = SimpleDocTemplate("phello.pdf")
doc.build(Story, onFirstPage=myFirstPage, onLaterPages=myLaterPages)
doc = SimpleDocTemplate("phello2.pdf")
doc.build(Story, onFirstPage=myFirstPage, onLaterPages=myLaterPages)
then it builds phello.pdf correctly, but builds a totally empty phello2.pdf
(960 bytes, a single white empty page).
It is hard to explain as it is, but something even more interesting happens
if you try to make them totally independent, and create a copy of the story
as well:
import copy
doc = SimpleDocTemplate("phello.pdf")
doc.build(copy.copy(Story), onFirstPage=myFirstPage, onLaterPages=myLaterPages)
doc = SimpleDocTemplate("phello2.pdf")
doc.build(copy.copy(Story), onFirstPage=myFirstPage, onLaterPages=myLaterPages)
This will render phello.pdf correctly, and it will throw this error when
rendering phello2.pdf:
Traceback (most recent call last):
File "C:\Projects\test\test2.py", line 48, in
go()
File "C:\Projects\test\test2.py", line 45, in go
doc.build(copy.copy(Story), onFirstPage=myFirstPage,
onLaterPages=myLaterPages)
File
"C:\Users\nagyl\.virtualenvs\test-NC9-O-tN\lib\site-packages\reportlab\platypus\doctemplate.py",
line 1314, in build
BaseDocTemplate.build(self,flowables, canvasmaker=canvasmaker)
File "C:\Users\nagyl\.virtualenvs\
test-NC9-O-tN\lib\site-packages\reportlab\platypus\doctemplate.py", line
1079, in build
self.handle_flowable(flowables)
File "C:\Users\nagyl\.virtualenvs\
test-NC9-O-tN\lib\site-packages\reportlab\platypus\doctemplate.py", line
958, in handle_flowable
raise LayoutError(ident)
reportlab.platypus.doctemplate.LayoutError: Flowable This is Paragraph number 6. This is Paragraph
number 6. This(439.27559055118115 x 72) too large on page 1 in frame
'normal'(439.27559055118115 x 685.8897637795277) of template 'First'
And finally, here is the "solution" that solves all problems:
def go():
def create_story():
Story = [Spacer(1, 2 * inch)]
style = styles["Normal"]
for i in range(100):
bogustext = ("This is Paragraph number %s. " % i) * 20
p = Paragraph(bogustext, style)
Story.append(p)
Story.append(Spacer(1, 0.2 * inch))
return Story
doc = SimpleDocTemplate("phello.pdf")
doc.build(create_story(), onFirstPage=myFirstPage,
onLaterPages=myLaterPages)
doc = SimpleDocTemplate("phello2.pdf")
doc.build(create_story(), onFirstPage=myFirstPage,
onLaterPages=myLaterPages)
This does not throw an error, and it renders both files correctly. But I do
not see why this last version works, and the previous one (that uses
copy.copy) does not.
I was looking for an explanation in the user guide. I was looking for a
note, telling me that a story can be used for document generation only
once. But there is no such thing in the docs. Or maybe I just did not find
it. Can somebody please explain what is happening here?
Environment details: Python 3.10.1 amd64 on Windows, reportlab 3.6.6
Thanks,
Laszlo
--
https://mail.python.org/mailman/listinfo/python-list
Re: Reportlab / platypus bug?
I will, thanks. I just wanted to make sure that this is a real bug, and not a mistake on my side. Barry ezt írta (időpont: 2022. márc. 13., V 23:29): > > > > On 13 Mar 2022, at 21:41, Les wrote: > > > > I have found an error, and I created a minimal working example. The > minimal > > working example starts with the very first example from Platypus user > guide: > > I would suggest that you report to reportlab.com directly, any fix will > come from them. > > Barry > -- https://mail.python.org/mailman/listinfo/python-list
Re: Reportlab / platypus bug?
Good point. I can confirm, that it works with copy.deepcopy. Probably you are right, the story is somehow consumed by doc.build. But it is not documented anywhere. I'm going to submit a bug report, thanks. -- https://mail.python.org/mailman/listinfo/python-list
Re: Reportlab / platypus bug?
Unfortunately, the reportlab-users mailing list is unavailable (I cannot subscribe). There is paid support but since I already have a workaround, I won't pay for this. I think this is a documentation error of the reportlab package. (They do not mention that stories cannot be reused.) I think we can say that my original problem is solved, because I have a workaround that always works. Schachner, Joseph ezt írta (időpont: 2022. márc. 14., H 19:09): > I realize this is Python code, but I doubt that the question is a Python > question. I have used Python +numpy, scipy, matplotlib for years. I have > not used reportlab and have no idea about the reported problem except that > I will be very surprised if it turns out to be a Python language issue. > Is there possibly a better place to ask this question? > > > Teledyne Confidential; Commercially Sensitive Business Data > > -----Original Message- > From: Les > Sent: Sunday, March 13, 2022 4:56 PM > To: [email protected] > Subject: Reportlab / platypus bug? > > Hello, > > I have found an error, and I created a minimal working example. The > minimal working example starts with the very first example from Platypus > user guide: > > from reportlab.platypus import SimpleDocTemplate, Paragraph, Spacer from > reportlab.lib.styles import getSampleStyleSheet from > reportlab.lib.pagesizes import A4 from reportlab.lib.units import inch > > PAGE_HEIGHT = A4[1] > PAGE_WIDTH = A4[0] > styles = getSampleStyleSheet() > > Title = "Hello world" > pageinfo = "platypus example" > > > def myFirstPage(canvas, doc): > canvas.saveState() > canvas.setFont('Times-Bold', 16) > canvas.drawCentredString(PAGE_WIDTH / 2.0, PAGE_HEIGHT - 108, Title) > canvas.setFont('Times-Roman', 9) > canvas.drawString(inch, 0.75 * inch, "First Page / %s" % pageinfo) > canvas.restoreState() > > > def myLaterPages(canvas, doc): > canvas.saveState() > canvas.setFont('Times-Roman', 9) > canvas.drawString(inch, 0.75 * inch, "Page %d %s" % (doc.page, > pageinfo)) > canvas.restoreState() > > > def go(): > Story = [Spacer(1, 2 * inch)] > style = styles["Normal"] > for i in range(100): > bogustext = ("This is Paragraph number %s. " % i) * 20 > p = Paragraph(bogustext, style) > Story.append(p) > Story.append(Spacer(1, 0.2 * inch)) > doc = SimpleDocTemplate("phello.pdf") > doc.build(Story, onFirstPage=myFirstPage, onLaterPages=myLaterPages) > > > go() > > > If I change it to this (e.g. generate two identical files): > > doc = SimpleDocTemplate("phello.pdf") > doc.build(Story, onFirstPage=myFirstPage, onLaterPages=myLaterPages) doc = > SimpleDocTemplate("phello2.pdf") doc.build(Story, onFirstPage=myFirstPage, > onLaterPages=myLaterPages) > > > then it builds phello.pdf correctly, but builds a totally empty phello2.pdf > (960 bytes, a single white empty page). > > It is hard to explain as it is, but something even more interesting > happens if you try to make them totally independent, and create a copy of > the story as well: > > import copy > doc = SimpleDocTemplate("phello.pdf") > doc.build(copy.copy(Story), onFirstPage=myFirstPage, > onLaterPages=myLaterPages) doc = SimpleDocTemplate("phello2.pdf") > doc.build(copy.copy(Story), onFirstPage=myFirstPage, > onLaterPages=myLaterPages) > > > This will render phello.pdf correctly, and it will throw this error when > rendering phello2.pdf: > > Traceback (most recent call last): > File "C:\Projects\test\test2.py", line 48, in > go() > File "C:\Projects\test\test2.py", line 45, in go > doc.build(copy.copy(Story), onFirstPage=myFirstPage, > onLaterPages=myLaterPages) > File > > "C:\Users\nagyl\.virtualenvs\test-NC9-O-tN\lib\site-packages\reportlab\platypus\doctemplate.py", > line 1314, in build > BaseDocTemplate.build(self,flowables, canvasmaker=canvasmaker) > File "C:\Users\nagyl\.virtualenvs\ > test-NC9-O-tN\lib\site-packages\reportlab\platypus\doctemplate.py", line > 1079, in build > self.handle_flowable(flowables) > File "C:\Users\nagyl\.virtualenvs\ > test-NC9-O-tN\lib\site-packages\reportlab\platypus\doctemplate.py", line > 958, in handle_flowable > raise LayoutError(ident) > reportlab.platypus.doctemplate.LayoutError: Flowable 0x148e102cb80 frame=normal>This is Paragraph number 6. This is Paragraph > number 6. This(439.27559055118115 x 72) too large on page 1 in frame > 'normal'(439.27
Re: Reportlab / platypus bug?
Dennis Lee Bieber ezt írta (időpont: 2022. márc.
14., H 20:03):
> On Mon, 14 Mar 2022 19:17:31 +0100, Les declaimed the
> following:
>
> >Unfortunately, the reportlab-users mailing list is unavailable (I cannot
> >subscribe). There is paid support but since I already have a workaround, I
> >won't pay for this. I think this is a documentation error of the reportlab
> >package. (They do not mention that stories cannot be reused.)
> >
>
>
> https://github.com/eduardocereto/reportlab/blob/master/src/reportlab/platypus/doctemplate.py
> """
> A document is built when a DocumentTemplate is fed a sequence of Flowables.
> The action of the build consumes the flowables in order and places them
> onto frames on pages as space allows. When a frame runs out of space the
> next frame of the page is used. If no frame remains a new page is created.
> A new page can also be created if a page break is forced.
> """
>
> Well, the code does use the term "consumes"
>
> And down near the bottom (line 980 or so; this is a section that
> does
> multiple passes for special cases)
> """
> # work with a copy of the story, since it is consumed
> tempStory = story[:]
> self.build(tempStory, **buildKwds)
> #self.notify('debug',None)
> """
>
Well, the official user guide (
https://www.reportlab.com/docs/reportlab-userguide.pdf ) only uses the word
"consume" in section "5.5 Frames", but it does not imply that a story
cannot be reused for generating multiple documents. It could also mean that
each document builder consumes each Flowable only once. So I still think
that this behaviour is undocumented.
It is very interesting that they use story[:] to make a swallow copy for
multiple passes. It must work in that particular case (e.g. when you build
the same document multiple times), but it surely does not work when you
build different documents using the same story (throws a LayoutError).
--
https://mail.python.org/mailman/listinfo/python-list
Re: Reportlab / platypus bug?
Robin Becker ezt írta (időpont: 2022. márc. 15., K, 14:06): > > > Hi Les, so far as I know the reportlab-users list is still running it is > hosted (nad has been for many years) at > > > https://pairlist2.pair.net/mailman/listinfo/reportlab-users > > is that the address you used? I see messages in the archives so some > people can use it.-- > Yes, it is. I tried to subscribe (twice), but never got the confirmation email. Checked in the spam folder too, but it is nowhere to be found. -- https://mail.python.org/mailman/listinfo/python-list
Re: Reportlab / platypus bug?
> > > > > as a test I subscribed under my private email address and the list > responded pretty quickly; the request confirmation > email ended up in spam though. I believe the list is a fairly old version > of mailman, but I don't have any access to the > server. > I tried again, and now I got the confirmation e-mail (in Spam). Thank you! > > -- https://mail.python.org/mailman/listinfo/python-list
Re: Reportlab / platypus bug?
Greg Ewing ezt írta (időpont: 2022. márc. 16., Sze, 1:01): > On 16/03/22 2:20 am, Les wrote: > > I tried to subscribe (twice), but never got the confirmation > > email. Checked in the spam folder too, but it is nowhere to be found. > > Is there any chance your email provider has some kind of quarantine > system separate from your spam folder? > Probably not, I was trying to register with gmail. It is interesting, I did not get the confirmation email (tried twice). Next day, I got it instantly. > > -- https://mail.python.org/mailman/listinfo/python-list
Python/Rexx Interface Now Available
A Python Rexx Interface is now available to allow Python scripting to access REXX libraries and utilities. See http://www.lestec.com.au for details Regards, Les -- http://mail.python.org/mailman/listinfo/python-list
Re: Clueless with cPickle
OK, looks like it is time for a break! Thanks for the replies! -- http://mail.python.org/mailman/listinfo/python-list
Clueless with cPickle
I am working on a homework assignment and trying to use cPickle to store
the answers from questor.py I believe I have the syntax correct but am not
sure if I am placing everything where it needs to be. Any help would be
greatly appreciated. When I attempt to run what I have I end up with the
following:
Traceback (most recent call last):
File "/home/les/workspace/Module 2/questor.py", line 18, in ?
f = file(questorlistfile)
NameError: name 'questorlistfile' is not defined
I thought that I had defined questorlistfile on the 4th line below
# define some constants for future use
import cPickle as p
#import pickle as p
questorfile = 'questor.data' # the name of the file where we will store the
object
questorlist = []
# Write to the file
f = file(questorfile, 'w')
p.dump(questorlist, f) # dump the object to a file
f.close()
del questorlist # remove the shoplist
# Read back from the storage
f = file(questorlistfile)
storedlist = p.load(f)
print storedlist
kQuestion = 'question'
kGuess = 'guess'
# define a function for asking yes/no questions
def yesno(prompt):
ans = raw_input(prompt)
return (ans[0]=='y' or ans[0]=='Y')
# define a node in the question tree (either question or guess)
class Qnode:
# initialization method
def __init__(self,guess):
self.nodetype = kGuess
self.desc = guess
# get the question to ask
def query(self):
if (self.nodetype == kQuestion):
return self.desc + " "
elif (self.nodetype == kGuess):
return "Is it a " + self.desc + "? "
else:
return "Error: invalid node type!"
# return new node, given a boolean response
def nextnode(self,answer):
return self.nodes[answer]
# turn a guess node into a question node and add new item
# give a question, the new item, and the answer for that item
def makeQuest( self, question, newitem, newanswer ):
# create new nodes for the new answer and old answer
newAnsNode = Qnode(newitem)
oldAnsNode = Qnode(self.desc)
# turn this node into a question node
self.nodetype = kQuestion
self.desc = question
# assign the yes and no nodes appropriately
self.nodes = {newanswer:newAnsNode, not newanswer:oldAnsNode}
def traverse(fromNode):
# ask the question
yes = yesno( fromNode.query() )
# if this is a guess node, then did we get it right?
if (fromNode.nodetype == kGuess):
if (yes):
print "I'm a genius!!!"
return
# if we didn't get it right, return the node
return fromNode
# if it's a question node, then ask another question
return traverse( fromNode.nextnode(yes) )
def run():
# start with a single guess node
topNode = Qnode('python')
done = 0
while not done:
# ask questions till we get to the end
result = traverse( topNode )
# if result is a node, we need to add a question
if (result):
item = raw_input("OK, what were you thinking of? ")
print "Enter a question that distinguishes a",
print item, "from a", result.desc + ":"
q = raw_input()
ans = yesno("What is the answer for " + item + "? ")
result.makeQuest( q, item, ans )
print "Got it."
# repeat until done
print
done = not yesno("Do another? ")
print
# immediate-mode commands, for drag-and-drop or execfile() execution
if __name__ == '__main__':
run()
print
raw_input("press Return>")
else:
print "Module questor imported."
print "To run, type: questor.run()"
print "To reload after changes to the source, type: reload(questor)"
# end of questor.py
--
http://mail.python.org/mailman/listinfo/python-list
Re: [OT] Python and Excel
oliver wrote: That would be amazing. I still have nightmares of when I had to create this big options analysis VBA program in Excel 2007. Odd - I haven't found VBA itself to be all that horrible. Yeah, it's got some serious weaknesses but it jut hasn't bit me badly, yet. Then again, it's not something I do a lot of. On Mon, Dec 18, 2017, 14:21 MRAB, wrote: Those who use Excel might find this interesting: Microsoft Considers Adding Python as an Official Scripting Language to Excel https://www.bleepingcomputer.com/news/microsoft/microsoft-considers-adding-python-as-an-official-scripting-language-to-excel/ -- https://mail.python.org/mailman/listinfo/python-list -- Les Cargill -- https://mail.python.org/mailman/listinfo/python-list
async serial port in Python.
What I'd like to do is set up *some* sort of method in Python to asynchronously use callbacks to receive characters from a serial port or 20 serial ports. If I have to hook an event loop or even spawn a thread - fine! but it needs to allow for making things event-driven. For lack of a better term, I'd like this to at least have "select()/epoll() semantics". And then I'd like to extend that to TCP and UDP ports. Is this even possible? I keep running into "not done yet" stuff on this front, but I'm not very up on Python. It may be ... what's the term ... ? ... un-Pythonic, and I would accept that as an explanation. And no, I do not have a pet distro or version of Python. Any is fine with me. The Brent Welch book for Tcl has a totally complete example of this laid out in great detail. That's so 15-20 years ago - try to catch up, guys :) -- Les Cargill -- https://mail.python.org/mailman/listinfo/python-list
Re: async serial port in Python.
Les Cargill wrote: What I'd like to do is set up *some* sort of method in Python to asynchronously use callbacks to receive characters from a serial port or 20 serial ports. If I have to hook an event loop or even spawn a thread - fine! but it needs to allow for making things event-driven. For lack of a better term, I'd like this to at least have "select()/epoll() semantics". And then I'd like to extend that to TCP and UDP ports. Is this even possible? I keep running into "not done yet" stuff on this front, but I'm not very up on Python. It may be ... what's the term ... ? ... un-Pythonic, and I would accept that as an explanation. And no, I do not have a pet distro or version of Python. Any is fine with me. The Brent Welch book for Tcl has a totally complete example of this laid out in great detail. That's so 15-20 years ago - try to catch up, guys :) Update: Found this: http://pyserial-asyncio.readthedocs.io/en/latest/index.html Seems to work. It's still "experimental"? -- Les Cargill -- https://mail.python.org/mailman/listinfo/python-list
What is a mechanism equivalent to "trace variable w ..." in Tcl for Python?
A really interesting design approach in Tcl is to install a callback
when a variable is written to. This affords highly event-driven
programming.
Example ( sorry; it's Tcl ) :
namespace eval events {
set preRPM -1
proc handleRPM { args } {
# do stuff to handle an RPM change here
variable ::motor::RPM
variable preRPM
puts "RPM changed from $preRPM to $RPM
set preRPM $RPM
}
}
...
trace variable ::motor::RPM w ::events::handleRPM
...
set ::motor::RPM 33.33
What is an equivalent mechanism in Python?
Thanks in advance.
--
Les Cargill
--
https://mail.python.org/mailman/listinfo/python-list
Re: Is Microsoft Windows secretly downloading childporn to your computer ?!
Juha Nieminen wrote: In comp.lang.c++ Steve Hayes wrote: You download things FROM a computer, you upload them TO a computer. It's a matter of perspective. If a hacker breaks into your computer and starts a download from somewhere else into your computer, isn't the hacker "downloading" things to your computer? --- news://freenews.netfront.net/ - complaints: [email protected] --- Down is towards an end node; up is towards the backbone. Servers live closer to the backbone. Usually. Or rather did when the nomenclature was forged. -- Les Cargill -- https://mail.python.org/mailman/listinfo/python-list
GPG.py adapted to use subprocess, for Windows?
Hi: Does anyone know if the GPG.py module has been updated to use the subprocess module from python 2.4. My web searches find only the original GPG module that requires a fork() -- according to my quick tests anyway. i was about to adapt this to use subprocess, for Windows, but thought i would ask here first. Thanks Les Schaffer -- http://mail.python.org/mailman/listinfo/python-list
Re: GPG.py adapted to use subprocess, for Windows?
Les Schaffer wrote: > Does anyone know if the GPG.py module errr, THIS GPG module: http://www.historical-id.info/files/python/GPG.py Les -- http://mail.python.org/mailman/listinfo/python-list
Re: GPG.py adapted to use subprocess, for Windows?
Les Schaffer wrote: > errr, THIS GPG module: http://www.historical-id.info/files/python/GPG.py sigh, actually its updated version: http://trac.t7a.org/isconf/browser/trunk/lib/python/isconf/GPG.py looks like i can use the subprocess module straight instead of the PopenHI, except for the complication that PopenHI provides *4* pipes to the subprocess ... les -- http://mail.python.org/mailman/listinfo/python-list
GC and security
i am working on a python application that uses encryption as part of its security features. so then at some point someone has to enter a passphrase into the system and passed into a decryption functions (we are using gpg via subprocess). so i am curious. so long as i drop all reference to the passphrase string(s), eventually it gets garbage collected and the memory recycled. so "before long" the phrase is gone from memory. is there a best practice way to do this? thanks Les Schaffer -- http://mail.python.org/mailman/listinfo/python-list
Re: GC and security
Paul Rubin wrote: > You can't rely on anything like that, either on the Python GC side or > from the OS (which might have long since written the passphrase out to > the swap disk) without special arrangement. we offered to disable swap for this app (its not memory intensive) but this level of precaution was beyond what is currently desired. i recently learned that Windows can be asked to zero the swap file during shutdown, though i know there are ways around this one pass write. > Some OS's have system > calls to lock user pages in memory and prevent swapping, and GPG tries > to use them. "Best practice" if you're doing a high security app > involves using special hardware modules to wrap the keys. understood, i meant best practice in terms of the less rigorous garbage collection. if the collect() function hastens garbage collection for unreferenced strings like a passphrase, it costs us nothing and buys us a wee bit. > The > relevant standard is FIPS 140-2, with FIPS-140-3 in preparation: > > http://csrc.nist.gov/cryptval/140-2.htm > http://csrc.nist.gov/cryptval/140-3.htm thanks for these. we may be called upon to up the security level at some point. > > For most purposes (e.g. some random web service), this stuff is > overkill, though. we're more sensitive than a web service, but not at the level of hardware protection. it is health data related, and for the moment we exceed the OMB's latest on laptop security: http://www.whitehouse.gov/omb/memoranda/fy2006/m06-16.pdf i don't see a mention of swap files on there, but maybe i missed it. and the OMB doc exceeds the security level required by the client app. les schaffer -- http://mail.python.org/mailman/listinfo/python-list
Re: GC and security
Aahz wrote: > Assuming you're talking about CPython, strings don't really participate > in garbage collection. Keep in mind that the primary mechanism for > reaping memory is reference counting, and generally as soon as the > refcount for an object goes to zero, it gets deleted from memory. ok so far ... > Garbage collection only gets used for objects that refer to other > objects, so it would only apply if string refcounts are being held by > GC-able objects. you lost me by here ... is there something different about string objects than other objects in Python? are you saying EVERY string in Python stays in memory for the lifetime of the app? > Also keep in mind, of course, that deleting objects has nothing to do > with whether the memory gets overwritten... no chance of being overwritten until deleted, no? and once deleted over time there is some probability of being overwritten, no? and i am curious how that works. it sounds like you are saying once a string, always the same string, in python. if thats true, i am glad i know that. Les Schaffer -- http://mail.python.org/mailman/listinfo/python-list
Re: GC and security
Paul Rubin wrote: > GC simply releases the memory for other uses in the application. It > doesn't necessarily zero the memory. release is followed by some probability of eventual overwrite; check. > > Just what attack are you trying to protect against, if swap space is > less of a problem than leaving keys around in ram? keys are on a USB drive key ring. gpg accesses the key ring as needed, but in a separate process. and gpg is done with its work early on in our app lifetime. comes back at end to encrypt and then app is done. > Keep in mind that the weakest part of this application is likely to be > the passphrase itself. Is there a way to get rid of it? we got some suggestions from other parts of this thread. or do you mean getting rid of the need for a passphrase? the passhprase protects the private key on the USB drive. > Is this data on a laptop? Why do you want to do encryption in the > application, instead of using an encrypted file system? i looked at EFS and TrueCrypt. There was some questions (from MySQL pro) about how MySQL writes would interact with EFS. also, EFS seems to store key pairs on disk, and the passphrase was limited to the Windows user login password. i remember looking at TrueCrypt and deciding against it. couldn't see how to interact with it via python script. vaguely recall being dissatisfied with keys and passphrase. but the main reason? we were asked to encrypt the MySQL tables carrying sensitive information. > Is there some > obstacle to using a token (like a smart card) to hold the key? USB drive holds the GPG key. the drive must be inserted at start of application, and must be pulled after authentication otherwise the app warns and shuts down. The USB drive carries a digital signature, and also encrypted identifying information for the user. Les Schaffer -- http://mail.python.org/mailman/listinfo/python-list
Re: GC and security
Tim Peters wrote: > Purely accidental -- nothing guaranteed -- details can (& do) change > across releases. Read obmalloc.c for a tour of the accidents du jour. cool. thanks for the pointer! > Not true, so be glad to forget it. forget what??? ;-) > A curious possibility: if you do a debug build of Python, obmalloc.c > arranges to overwrite all of an object's memory as soon as the object > is reclaimed (by any means, refcounting or otherwise). That wasn't > for "security" (faux or otherwise), it's to make it easier to detect > buggy C code referencing freed memory. i liked the other Tim's suggestion of a secure string ;-) Les Schaffer -- http://mail.python.org/mailman/listinfo/python-list
Re: GC and security
Fredrik Lundh wrote:
> Fredrik Lundh wrote:
>
>> a time (if that's possible; if not, you may need to write a custom
>> extension that builds a command string in a C-level buffer, runs the
>> command, and then overwrites the buffer before returning).
myself, i enjoy building C extensions, but would rather skip it for this
app.
>
> on the other hand, subprocess seems to support the buffer interface, so
> the following slightly convoluted approach could be used instead of such
> an extension:
>
>>>> cmd = [101, 99, 104, 111, 32, 39, 104, 101, 108, 108, 111, 39]
>>>> cmd = array.array("b", cmd) # build mutable buffer
>>>> subprocess.call([buffer(cmd)], shell=True)
> 'hello'
>>>> for i in range(len(cmd)): cmd[i] = 0 # nuke it
i'll see if we can fit this into our subprocess scheme. if so, this is
good enough for now. think we'll use this for the mysql password too.
> the secret text will be visible in memory during the subprocess call,
> but it won't linger around once the for-loop has finished.
good enough for current rock and roll.
> (don't forget to put a try/finally clause around the critical part)
okey doky.
Les Schaffer
--
http://mail.python.org/mailman/listinfo/python-list
Re: GC and security
Paul Rubin wrote: > gpg is fairly careful about passphrases. Why are you collecting the > passphrase in the Python app instead of letting gpg handle it? as i recall we did that because we needed the passphrase more than once, and didnt want to have to make the users type in something that long that many times within a minute or so. i forget whether gpg can be given a list of files to decrypt. but cuz of what we are doing, i still believe we would need to call gpg more than once. Fred Lundh's scheme for blanking the passphrase looks good enough for now. > > Yes, I mean get rid of the need for a passphrase, though since the > encrypted key is accessible on the USB drive, there's no way around > it. With smart cards it's generally considered ok to use a short PIN > instead of a passphrase; the card itself enforces a maximum # of > incorrect guesses. by any chance, do you have any experience with these USB/fingerprint things? i think the PI on this project will consider smart cards overkill, but we will suggest these as alternatives before we get out into the field. > Does using an encrypted FS not take care of that? yea but ... we are being asked to backup the MySQL tables onto the USBKey, so they need to be encrypted there as well. which means we need some kind of EFS on there as well. i wouldnt want to use more than one kind of encryption in this app, or better said, i dont want more than one set of keys/passes in this app. so we'd need an EFS on the Windows machines and on the USB keys that can utilize the same encryption keys. > Also, I think there > are some FS's that use the Windows Crypto API (CAPI) either for bulk > encryption or for key management, so you can use secure passphrases, > hardware tokens, or whatever. FS's other than Microsoft's EFS? i'll take a look at their capabilities. and there is still MySQls comment about dealing with encrypted file systems. we were advised by one of their people to test to make sure the writes are not interfered with on an EFS. > This is better than nothing but it's very easy to duplicate a USB key, > either intentionally or by spilling the contents through a routine > backup procedure, etc. but they still need the passphrase, hence keeping our eye on that silly string. not worried about accidents at the moment. if the USB keys had drive serial numbers we could validate on that. but i just checked mine and it has none. WMI reports that physical drives have a "Signature", not sure what that is. > A crypto token (USB dongle or smart card) is > way preferable for this type of thing. GPG has smart card support > that you might be able to use: > > http://www.g10code.com/p-card.html > http://www.gnupg.org/(en)/howtos/card-howto/en/smartcard-howto-single.html would definitely consider this for Gen II. > > You might want to discuss this on sci.crypt, where specialists hang > out. As is fairly typical in these situations, it would help a lot if > you could describe the application in more detail. need to get permission from PI on the project. thanks for the comments, they validate my concerns. if you know, or are, a pro in python and security, we might be able to manage a small consulting gig. but if its not python-relevant, lets talk offlist. in any case, many thanks. Les Schaffer -- http://mail.python.org/mailman/listinfo/python-list
Re: [CentOS] App Question.
bruce wrote: Hi list!! Got a question, and I can't find a good answer for, so I figured i'd post here. I'm working on a project that involves a number of smaller apps to be developed, and run. In order to build this overall application, I'm trying to find a web based app that I can use to manage the entire process. I need to be able to create the basic app, and then to test the app, and then to run the app in a production environment. I need to be able to schedule when the app is tested/run in the production system, as well as be able to track the status of the test/production app. My overall system will have 100's of these smaller child apps. Has anyone heard of, or have any pointers to any kind of webbased app that I can use (open source) to manage this entire kind of process. I would prefer a web based app, that runs on a LAMP platform to manage this overall process It is sort of java-oriented, but cruisecrontrol manages that sort of thing: http://cruisecontrol.sourceforge.net/index.html. -- Les Mikesell [EMAIL PROTECTED] -- http://mail.python.org/mailman/listinfo/python-list
Re: Reading data from a Microsoft Access 2003 database
Jimoid wrote: I use Ubuntu 64 bit and need to develop a programme (ideally in Python) to work on data that is contained in a Microsoft Access 2003 database. I do not need to modify the database, simply read a few columns of data from some tables. this worked like a charm for me: http://code.activestate.com/recipes/528868-extraction-and-manipulation-class-for-microsoft-ac/ Les -- http://mail.python.org/mailman/listinfo/python-list
Re: Reading data from a Microsoft Access 2003 database
Adam Tauno Williams wrote: The OP: "I use Ubuntu 64 bit" woops, my bad ... -- http://mail.python.org/mailman/listinfo/python-list
any issues with long running python apps?
i have been asked to guarantee that a proposed Python application will run continuously under MS Windows for two months time. And i am looking to know what i don't know. The app would read instrument data from a serial port, store the data in file, and display in matplotlib. typical sampling times are about once per hour. our design would be to read the serial port once a second (long story, but it worked for ten days straight so far) and just store and display the values once an hour. so basically we'd have a long running for-loop. maybe running in a separate thread. i have thought through the issues on our end: we have to properly handle the serial port and make sure our GUI doesn't hang easily. we'd have to be sure we don't have any memory leaks in numpy or matplotlib and we're not storing data in memory stupidly. i've never looked into Windows serial port driver reliability over the long term. But we think if the serial port hangs, we can detect that and react accordingly. but none of this has anything to do with Python itself. i am sure python servers have been running reliably for long periods of time, but i've never had to deal with a two-month guarantee before. is there something else i am missing here that i should be concerned about on the pure-Python side of things? something under the hood of the python interpreter that could be problematic when run for a long time? or need we only concern ourselves with the nuts behind the wheel:that is, we the developers? thanks Les -- http://mail.python.org/mailman/listinfo/python-list
Re: any issues with long running python apps?
thanks to all for the replies. the Windows memory fragmentation was one of the "i didn't know that" items. we will use 64-bit Windows OS if the job happens. agree with all the other suggestions: multiple threads for data and GUI, etc. Also, might push for Linux even though the company is Windows-only presently. but thinking about the issue some more as well as everyone's comments, have decided to proceed cautiously and let the client decide first that they really need this app regardless of guaranteed Windows-based uptime. Les -- http://mail.python.org/mailman/listinfo/python-list
Re: any issues with long running python apps?
Stefan Behnel wrote: Les Schaffer, 16.07.2010 15:07: agree with all the other suggestions: multiple threads for data and GUI, The way I read it, the suggestion was to use separate processes, not multiple threads. That's a pretty important difference. check. processes, not threads. Les -- http://mail.python.org/mailman/listinfo/python-list
