Re: Own network protocol

[pfranken85]

Hello Steven!

Thank you for your answer!

RPyC indeed looks great! I need to deep dive into the API reference, but I 
think its capabilities will suffice to do what I want. Do you know whether 
non-python related clients can work with this as well, i.e. are their 
corresponding tools on the android/iphone side? Just to know whether they could 
serve as clients as well.

Cheers!

Am Samstag, 27. Dezember 2014 11:14:13 UTC+1 schrieb Steven D'Aprano:
> pfranke...@gmail.com wrote:
> 
> > What kind of protocol do you recommend for this? UDP or TCP? Do you
> > recommend the use of frameworks such as twisted?
> 
> I don't recommend something as low-level as inventing your own protocol, or
> as heavy-weight as Twisted.
> 
> Have you considered a remote-procedure call library like rpyc or pyro?
> 
> http://rpyc.readthedocs.org/
> https://pypi.python.org/pypi/Pyro4
> 
> 
> -- 
> Steven

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Own network protocol

[pfranken85]

Hello Tim,

Am Samstag, 27. Dezember 2014 14:19:21 UTC+1 schrieb Tim Chase:
> The eventual solution would depend on a variety of factors:
> 
> - how critical is synchronization?
> 
> - do clients need to know if they missed a message? (somebody
>   disconnected from the LAN for a moment)

This would be nice indeed. At least, the user should be notified that the 
connection was lost and the current values may not be uptodate any more.

> - do clients need a way to receive historical messages in the event
>   they were offline during the broadcast? (a power outage knocked out
>   Client #18 at the time of the last update)

no, this is not necessary.


> 
> - are all your clients on the same IP subnet? (you could use a
>   broadcast packet)

yes, this assumption can be made.

> 
> - would you rather push data as it changes, or have clients poll for
>   the current state? (you write "it should send the update to the
>   connected clients" which suggests a push architecture, yet you also
>   want to have clients able to send updates: "should be possible for
>   the client to send a particular request to the server...i.e., switch
>   on LEX X")

Indeed, I would prefer a "push" setting, in particular to avoid additional 
overhead from the constant polling. Besides, this resembles more the scenario 
present at the server side: it gets notifications via callbacks in case 
anything has changed.

> - are you concerned about security/authentication? Can a rogue device
>   send a message pretending to be the server?  What would/should
>   happen if an unintended client snoops the traffic? Does it matter?
> The suggestions would look very different if you were just building a
> hobby notification system as a demo in a contained home/lab/office,
> vs. if you were building an industrial control system for monitoring a
> remote location and conveying security info.

Concerning the latter two points: Introducing a possible security layer is 
something I would like to do in the future, so the selection of the network 
protocol/system should definitely keep this in mind.

What do you think of the RPyC?

Thanks for your valuable input!

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Searching through more than one file.

[Albert-Jan Roskam]

-
On Sun, Dec 28, 2014 8:12 PM CET Dave Angel wrote:

>On 12/28/2014 12:27 PM, Seymore4Head wrote:
>> I need to search through a directory of text files for a string.
>> Here is a short program I made in the past to search through a single
>> text file for a line of text.
>> 
>> How can I modify the code to search through a directory of files that
>> have different filenames, but the same extension?
>> 
>
>You have two other replies to your specific question, glob and os.listdir.  I 
>would also mention the module fileinput:
>
>https://docs.python.org/2/library/fileinput.html


Ah, I was just about to say that. I found out about this gem after reading 
Dough Helmann's book. Here are some usage examples: 
http://pymotw.com/2/fileinput/


>import fileinput
>from glob import glob
>
>fnames = glob('*.txt')
>for line in fileinput.input(fnames):
>pass # do whatever
>
>If you're not on Windows, I'd mention that the shell will expand the wildcards 
>for you, so you could get the filenames from argv even simpler.  See first 
>example on the above web page.
>
>
>I'm more concerned that you think the following code you supplied does a 
>search for a string.  It does something entirely different, involving making a 
>crude dictionary.  But it could be reduced to just a few lines, and probably 
>take much less memory, if this is really the code you're working on.
>
>> fname = raw_input("Enter file name: ")  #"*.txt"
>> fh = open(fname)
>> lst = list()
>> biglst=[]
>> for line in fh:
>>  line=line.rstrip()
>>  line=line.split()
>>  biglst+=line
>> final=[]
>> for out in biglst:
>>  if out not in final:
>>  final.append(out)
>> final.sort()
>> print (final)
>> 
>
>Something like the following:
>
>import fileinput
>from glob import glob
>
>res = set()
>fnames = glob('*.txt')
>for line in fileinput.input(fnames):
>res.update(line.rstrip().split())
>print sorted(res)
>
>
>
>
>-- DaveA
>-- https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Own network protocol

[Tim Chase]

On 2014-12-29 00:34, pfranke...@gmail.com wrote:
> Am Samstag, 27. Dezember 2014 14:19:21 UTC+1 schrieb Tim Chase:
> > - do clients need to know if they missed a message? (somebody
> >   disconnected from the LAN for a moment)
> 
> This would be nice indeed. At least, the user should be notified
> that the connection was lost and the current values may not be
> uptodate any more.

This sounds like exactly the job that a message bus/queue was
designed for.  RabbitMQ or ZeroMQ are often the go-to solutions for
this:

http://www.rabbitmq.com/how.html
http://zeromq.org/intro:read-the-manual

and take care of all the ugly edge cases.  Your server just pushes
notification messages into the queue, and the clients subscribe to
those notifications (known as the pub/sub model).  For example

http://zguide.zeromq.org/page:all#Getting-the-Message-Out

> > - are all your clients on the same IP subnet? (you could use a
> >   broadcast packet)
> 
> yes, this assumption can be made.

Both RabbitMQ and ZeroMQ should support broadcast packets if this is
a case you want to optimize for.

> Introducing a possible security layer is something I would like to
> do in the future, so the selection of the network protocol/system
> should definitely keep this in mind.

It's certainly the sort of thing you'd want to build in from the
start.  Fortunately, it sounds like most of what you would want to do
can be solved by encrypting and/or signing the messages that you put
into the message bus.

> What do you think of the RPyC?

I don't have any experience with it personally, but I have found
Steven to be a fairly reliable source of trustworthy information, so
on that alone I'd be willing to investigate RPyC.

-tkc




-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Own network protocol

[Burak Arslan]

Hi,


On 12/29/14 10:18, pfranke...@gmail.com wrote:
> Hello Steven!
>
> Thank you for your answer!
>
> RPyC indeed looks great! I need to deep dive into the API reference, but I 
> think its capabilities will suffice to do what I want. Do you know whether 
> non-python related clients can work with this as well, i.e. are their 
> corresponding tools on the android/iphone side? Just to know whether they 
> could serve as clients as well.
>

If you need more than inter-process messaging, you can use Spyne, which
uses standard protocols instead of inventing its own.

I suggest using TwistedMessagePack transport to wrap messages and
MessagePack protocol for the actual content. Spyne also supports some
declarative validation so it can be used as a public daemon.

Disclaimer: I'm the author of Spyne.

Best,
Burak
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Searching through more than one file.

[Cem Karan]

On Dec 29, 2014, at 2:47 AM, Rick Johnson  wrote:

> On Sunday, December 28, 2014 11:29:48 AM UTC-6, Seymore4Head wrote:
>> I need to search through a directory of text files for a string.
>> Here is a short program I made in the past to search through a single
>> text file for a line of text.
> 
> Step1: Search through a single file. 
> # Just a few more brush strokes...
> 
> Step2: Search through all files in a directory. 
> # Time to go exploring! 
> 
> Step3: Option to filter by file extension. 
> # Waste not, want not!
> 
> Step4: Option for recursing down sub-directories. 
> # Look out deeply nested structures, here i come!
> # Look out deeply nested structures, here i come!
> # Look out deeply nested structures, here i come!
> # Look out deeply nested structures, here i come!
> # Look out deeply nested structures, here i come!
> # Look out deeply nested structures, here i come!
> # Look out deeply nested structures, here i come!
> [Opps, fell into a recursive black hole!]
> # Look out deeply nested structures, here i come!
> # Look out deeply nested structures, here i come!
> # Look out deeply nested structures, here i come!
> # Look out deeply nested structures, here i come!
> [BREAK]
> # Whew, no worries, MaximumRecursionError is my best friend! 
> 
> ;-)
> 
> In addition to the other advice, you might want to check out os.walk()

DEFINITELY use os.walk() if you're going to recurse through a directory tree.  
Here is an untested program I wrote that should do what you want.  Modify as 
needed:

"""
# This is all Python 3 code, although I believe it will run under Python 2
# as well.  

# os.path is documented at https://docs.python.org/3/library/os.path.html
# os.walk is documented at https://docs.python.org/3/library/os.html#os.walk
# losging is documented at https://docs.python.org/3/library/logging.html

import os
import os.path
import logging

# Logging messages can be filtered by level.  If you set the level really
# low, then low-level messages, and all higher-level messages, will be
# logged.  However, if you set the filtering level higher, then low-level
# messages will not be logged.  Debug messages are lower than info messages,
# so if you comment out the first line, and uncomment the second, you will
# only get info messages (right now you're getting both).  If you look
# through the code, you'll see that I go up in levels as I work my way 
# inward through the filters; this makes debugging really, really easy.
# I'll start out with my level high, and if my code works, I'm done. 
# However, if there is a bug, I'll work my downwards towards lower and
# lower debug levels, which gives me more and more information.  Eventually
# I'll hit a level where I know enough about what is going on that I can 
# fix the problem.  By the way, if you comment out both lines, you shouldn't
# have any logging at all.
logging.basicConfig(level=logging.DEBUG)
##logging.basicConfig(level=logging.INFO)

EXTENSIONS = {".txt"}

def do_something_useful(real_path):
# I deleted the original message, so I have no idea 
# what you were trying to accomplish, so I'm punting 
# the definition of this function back to you.
pass

for root, dirs, files in os.walk('/'):
for f in files:
# This expands symbolic links, cleans up double slashes, etc.
# This can be useful when you're trying to debug why something
# isn't working via logging.
real_path = os.path.realpath(os.path.join(root, f))
logging.debug("operating on path '{0!s}'".format(real_path))
(r, e) = os.path.splitext(real_path)
if e in EXTENSIONS:
# If we've made a mistake in our EXTENSIONS set, we might never
# reach this point.  
logging.info("Selected path '{0!s}'".format(real_path))
do_something_useful(real_path)
"""

As a note, for the sake of speed and your own sanity, you probably want to do 
the easiest/computationally cheapest filtering first here.  That means 
selecting the files that match your extensions first, and then filtering those 
files by their contents second.

Finally, if you are planning on parsing command-line options, DON'T do it by 
hand!  Use argparse (https://docs.python.org/3/library/argparse.html) instead.

Thanks,
Cem Karan

-- 
https://mail.python.org/mailman/listinfo/python-list

Python 2.7 segfaults on 'import site'

[Chris Angelico]

rosuav@sikorsky:~$ python -S
Python 2.7.3 (default, Mar 13 2014, 11:03:55)
[GCC 4.7.2] on linux2
>>>
rosuav@sikorsky:~$ python
Segmentation fault

This is the system Python on Debian Wheezy, and I haven't changed
site.py at all. This broke some time in the last few days, and I'm
trying to figure out why. (I have been fiddling with my python3, but
the system Python is untouched, as you can see from the above.)
Attempting to 'import site' manually causes the same segfault; things
get interesting if I create my own local site.py though:

rosuav@sikorsky:~/aaa$ touch site.py
rosuav@sikorsky:~/aaa$ python
Segmentation fault
rosuav@sikorsky:~/aaa$ python -S
Python 2.7.3 (default, Mar 13 2014, 11:03:55)
[GCC 4.7.2] on linux2
>>> import site
>>>

So clearly an empty 'site' can be imported safely, but running Python
without -S segfaults.

Can anyone advise as to where I should look for the cause of the trouble?

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: suggestions for VIN parsing

[Denis McMahon]

On Sun, 28 Dec 2014 16:27:20 -0700, Vincent Davis wrote:

> On Fri, Dec 26, 2014 at 12:15 PM, Denis McMahon
> 
> wrote:
> 
>> Note, I think the 1981 model year ran KCA - DCA prefixes, not as shown
>> on the website you quoted.

> ​Denis,
> Regarding the KCA - DCA prefixes, do you have a source as to why you
> think this?

Study the website carefully. Calculate the prefixes for the 1981 and 1982 
model years according to the table on the website.

K .. D would be the appropriate month prefixes for the 1981 model year, 
but if both the 1981 and 1982 model years used DA as a year prefix, there 
would be some prefixes that appeared twice, in the 1981 model year and 
the 1982 model year.

Which suggests to me that there's been a slip up somewhere.

-- 
Denis McMahon, denismfmcma...@gmail.com
-- 
https://mail.python.org/mailman/listinfo/python-list

CSV Dictionary

[JC]

Hello,

I have csv file like:

id,name,userid,password
1,ABC,def@ghi,1234
2,DEF,ghi@jkl,asdf
3,GHI,jkl@mno,zxcv
.
.
.

I need to get that csv file into a dictionary. Here is my code:

import csv

with open('x.csv','rb') as f:
rdr = csv.DictReader(f,delimiter=',')
flds = rdr.fieldnames
rows = 0
d = {}
for row in rdr:
rows += 1
i = 0
for i in range(len(flds)):
d[flds[i]] = row[flds[i]]

It shows only the last record, like:
d = {'Password': 'zxcv', 'UserId': 'jkl@mno', 'id': '3', 'Name': 'GHI'}

How could I get the all the records?

Thanks.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: CSV Dictionary

[Skip Montanaro]

On Mon, Dec 29, 2014 at 9:35 AM, JC  wrote:
> How could I get the all the records?

This should work:

with open('x.csv','rb') as f:
rdr = csv.DictReader(f,delimiter=',')
rows = list(rdr)

You will be left with a list of dictionaries, one dict per row, keyed
by the values in the first row:

>>> import csv
>>> with open('x.csv','rb') as f:
... rdr = csv.DictReader(f,delimiter=',')
... rows = list(rdr)
...
>>> import pprint
>>> pprint.pprint(rows)
[{'id': '1', 'name': 'ABC', 'password': '1234', 'userid': 'def@ghi'},
 {'id': '2', 'name': 'DEF', 'password': 'asdf', 'userid': 'ghi@jkl'},
 {'id': '3', 'name': 'GHI', 'password': 'zxcv', 'userid': 'jkl@mno'}]

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 2.7 segfaults on 'import site'

[Dave Angel]

On 12/29/2014 09:33 AM, Chris Angelico wrote:

rosuav@sikorsky:~$ python -S
Python 2.7.3 (default, Mar 13 2014, 11:03:55)
[GCC 4.7.2] on linux2



rosuav@sikorsky:~$ python
Segmentation fault

This is the system Python on Debian Wheezy, and I haven't changed
site.py at all. This broke some time in the last few days, and I'm
trying to figure out why. (I have been fiddling with my python3, but
the system Python is untouched, as you can see from the above.)
Attempting to 'import site' manually causes the same segfault; things
get interesting if I create my own local site.py though:

rosuav@sikorsky:~/aaa$ touch site.py
rosuav@sikorsky:~/aaa$ python
Segmentation fault
rosuav@sikorsky:~/aaa$ python -S
Python 2.7.3 (default, Mar 13 2014, 11:03:55)
[GCC 4.7.2] on linux2

import site



So clearly an empty 'site' can be imported safely, but running Python
without -S segfaults.

Can anyone advise as to where I should look for the cause of the trouble?



My gut feeling is you should check if there is another site.py* anywhere 
on your system.  (Use find rather than just manually checking the 
sys.path locations)



--
DaveA
--
https://mail.python.org/mailman/listinfo/python-list

Re: Python 2.7 segfaults on 'import site'

[Irmen de Jong]

On 29-12-2014 15:33, Chris Angelico wrote:

> So clearly an empty 'site' can be imported safely, but running Python
> without -S segfaults.
> 
> Can anyone advise as to where I should look for the cause of the trouble?
> 
> ChrisA

Perhaps starting python with -v provides some more info on when exactly it is 
crashing
during the startup process. (i.e. is it really site.py that crashes it?)

Irmen



-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 2.7 segfaults on 'import site'

[Chris Angelico]

On Tue, Dec 30, 2014 at 2:57 AM, Dave Angel  wrote:
> My gut feeling is you should check if there is another site.py* anywhere on
> your system.  (Use find rather than just manually checking the sys.path
> locations)

I used imp.find_module and it showed the one in
/usr/lib/python2.7/site.py would be used. However, I did find one
point of interest: site.py is *not* loaded from the current directory,
hence the difference between "python" and "python -S -c 'import
site'". Also, importing any of a bunch of other modules also
segfaults. I haven't yet tracked down the ultimate cause of the crash,
but it could be any module that gets imported by a bunch of other
modules. My current suspicion is on re.py, or something it imports.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: CSV Dictionary

[JC]

On Mon, 29 Dec 2014 09:47:23 -0600, Skip Montanaro wrote:

> On Mon, Dec 29, 2014 at 9:35 AM, JC  wrote:
>> How could I get the all the records?
> 
> This should work:
> 
> with open('x.csv','rb') as f:
> rdr = csv.DictReader(f,delimiter=',')
> rows = list(rdr)
> 
> You will be left with a list of dictionaries, one dict per row, keyed by
> the values in the first row:
> 
 import csv with open('x.csv','rb') as f:
> ... rdr = csv.DictReader(f,delimiter=',')
> ... rows = list(rdr)
> ...
 import pprint pprint.pprint(rows)
> [{'id': '1', 'name': 'ABC', 'password': '1234', 'userid': 'def@ghi'},
>  {'id': '2', 'name': 'DEF', 'password': 'asdf', 'userid': 'ghi@jkl'},
>  {'id': '3', 'name': 'GHI', 'password': 'zxcv', 'userid': 'jkl@mno'}]
> 
> Skip

Thanks. That did the job.
But now I see another problem. I cannot use the "rdr" object anymore. For 
example, I wanted to count the number of records. I used - 
count = sum(1 for r in rdr)
It returned 0 records.
Do I have to open the file again to get 'rdr' work again?

Thanks again.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 2.7 segfaults on 'import site'

[Chris Angelico]

On Tue, Dec 30, 2014 at 3:05 AM, Irmen de Jong  wrote:
> On 29-12-2014 15:33, Chris Angelico wrote:
>
>> So clearly an empty 'site' can be imported safely, but running Python
>> without -S segfaults.
>>
>> Can anyone advise as to where I should look for the cause of the trouble?
>>
>> ChrisA
>
> Perhaps starting python with -v provides some more info on when exactly it is 
> crashing
> during the startup process. (i.e. is it really site.py that crashes it?)

*facepalm* Yeah, that's the one I didn't think of. But now I have an
even bigger question.

...
...
...
# /usr/lib/python2.7/sre_constants.pyc matches
/usr/lib/python2.7/sre_constants.py
import sre_constants # precompiled from /usr/lib/python2.7/sre_constants.pyc
Segmentation fault

Why is sre_constants segfaulting?! It's such a simple file! I assume
the "matches" comment means it thinks there's no problem with the .pyc
file; is it possible that, despite those checks, there's an issue
there? I could delete the .pyc and try again, but I'm scared that'll
fix the problem and leave me unable to figure out the true cause :)

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: CSV Dictionary

[Skip Montanaro]

On Mon, Dec 29, 2014 at 10:11 AM, JC  wrote:
> Do I have to open the file again to get 'rdr' work again?

Yes, but if you want the number of records, just operate on the rows
list, e.g. len(rows).

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Future of python on android

[Anssi Saari]

"Fetchinson ."  writes:

> So what's the future proof way of writing/deploying/installing python
> programs on android?

Kivy is it I believe. I've meant to look into it but haven't gotten
around to it...
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: CSV Dictionary

[JC]

On Mon, 29 Dec 2014 10:32:03 -0600, Skip Montanaro wrote:

> On Mon, Dec 29, 2014 at 10:11 AM, JC  wrote:
>> Do I have to open the file again to get 'rdr' work again?
> 
> Yes, but if you want the number of records, just operate on the rows
> list, e.g. len(rows).
> 
> Skip

Yes, I did that. But if I need to use 'rdr' again, I have to open it 
again, right? Is there any ways to get 'rdr' point to first record again?

Thanks.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: CSV Dictionary

[Tim Chase]

On 2014-12-29 16:11, JC wrote:
> On Mon, 29 Dec 2014 09:47:23 -0600, Skip Montanaro wrote:
> 
> > On Mon, Dec 29, 2014 at 9:35 AM, JC  wrote:
> >> How could I get the all the records?
> > 
> > This should work:
> > 
> > with open('x.csv','rb') as f:
> > rdr = csv.DictReader(f,delimiter=',')
> > rows = list(rdr)
> > 
> > You will be left with a list of dictionaries, one dict per row,
> > keyed by the values in the first row:
> > 
>  import csv with open('x.csv','rb') as f:
> > ... rdr = csv.DictReader(f,delimiter=',')
> > ... rows = list(rdr)
>
> But now I see another problem. I cannot use the "rdr" object
> anymore. For example, I wanted to count the number of records. I
> used - count = sum(1 for r in rdr)
> It returned 0 records.

You exhausted that iterator.  Instead, if you put them all in the
list, just take len(rows)

Alternatively, if you want to have keyed O(1) access instead of O(N)
access to items in your CSV file, load them into a dictionary,
choosing the field you want to use for lookup:

  with open('x.csv', 'rb') as f:
rdr = csv.DictReader(f) # delimiter defaults to ","
d = dict(
  (row["userid"], row)
  for row in rdr
  )
  print("Loaded %i row(s)" % len(d))
  print("def@ghi: %r" % (d["def@ghi"]))

-tkc



-- 
https://mail.python.org/mailman/listinfo/python-list

Re: suggestions for VIN parsing

[Vincent Davis]

On Mon, Dec 29, 2014 at 7:47 AM, Denis McMahon 
wrote:

> K .. D would be the appropriate month prefixes for the 1981 model year,
> but if both the 1981 and 1982 model years used DA as a year prefix, there
> would be some prefixes that appeared twice, in the 1981 model year and
> the 1982 model year.
>

​Ah , I had not looked close at that yet. I found a different more
extensive site.
http://www.britishonly.com/tech/joust/techtiptriumphmf.htm​


Vincent Davis
720-301-3003
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: suggestions for VIN parsing

[Vincent Davis]

On Sun, Dec 28, 2014 at 11:50 PM, Rick Johnson  wrote:

> 3. I see that you are utilizing regexps to aid in the logic,
> and although i agree that regexps are overkill for this
> problem (since it could "technically" be solved with string
> methods) if *I* had to solve this problem, i would use the
> power of regexps -- although i would use them more wisely ;-)
>
> I have not studied the data thoroughly, but just by "grazing
> over" the code you posted i can see a few distinct patterns
> that emerge from the VIN data-set. Here is a description of
> the patterns:
>
> "\d+n"
> "\d+na"
> "d\d+"
> "du\d+"
>
> and the last pattern being all digits:
>
> "\d+"
>
> Even though your "verbose-run-on-conditional" would most
> likely execute faster, i prefer to write code (when
> performance is not mission critical!) in the most readable
> and maintainable fashion. And in order to achieve that goal,
> you always want to keep the main logic as succinct as
> possible whist encapsulating the difficult bits in "suitably
> abstracted structures".
>

​Rick,
Thanks for your suggestions, I was just starting version2 and wanted to do
something like you suggest.
Another question. I what to change the logic so that rather than return THE
match it return all matches. I want to do this for 2 reasons, 1, it would
act as a kinda test, If I only expect one match and I get more than I
likely have a problem, 2, I found a more extensive (maybe better) list of
frame numbers ,
I could see some overlapping although I have not looked real close yet.



Vincent Davis
720-301-3003
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: CSV Dictionary

[Tim Chase]

On 2014-12-29 16:37, JC wrote:
> On Mon, 29 Dec 2014 10:32:03 -0600, Skip Montanaro wrote:
> 
> > On Mon, Dec 29, 2014 at 10:11 AM, JC 
> > wrote:
> >> Do I have to open the file again to get 'rdr' work again?
> > 
> > Yes, but if you want the number of records, just operate on the
> > rows list, e.g. len(rows).
> 
> Yes, I did that. But if I need to use 'rdr' again, I have to open
> it again, right? Is there any ways to get 'rdr' point to first
> record again?

Yeah-but.  What do you need rdr for again?  You've already sucked all
the information out of it into your data-structure.  If you needed
other information, you should pull it out the first time you iterate
through the DictReader.

-tkc


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 2.7 segfaults on 'import site'

[Peter Otten]

Chris Angelico wrote:

> On Tue, Dec 30, 2014 at 3:05 AM, Irmen de Jong 
> wrote:
>> On 29-12-2014 15:33, Chris Angelico wrote:
>>
>>> So clearly an empty 'site' can be imported safely, but running Python
>>> without -S segfaults.
>>>
>>> Can anyone advise as to where I should look for the cause of the
>>> trouble?
>>>
>>> ChrisA
>>
>> Perhaps starting python with -v provides some more info on when exactly
>> it is crashing during the startup process. (i.e. is it really site.py
>> that crashes it?)
> 
> *facepalm* Yeah, that's the one I didn't think of. But now I have an
> even bigger question.
> 
> ...
> ...
> ...
> # /usr/lib/python2.7/sre_constants.pyc matches
> /usr/lib/python2.7/sre_constants.py
> import sre_constants # precompiled from
> /usr/lib/python2.7/sre_constants.pyc Segmentation fault
> 
> Why is sre_constants segfaulting?! It's such a simple file! I assume
> the "matches" comment means it thinks there's no problem with the .pyc
> file; is it possible that, despite those checks, there's an issue
> there? I could delete the .pyc and try again, but I'm scared that'll
> fix the problem and leave me unable to figure out the true cause :)

Maybe the interpreter itself is corrupted? Google suggests that debsums may 
be the tool to find out.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 2.7 segfaults on 'import site'

[Chris Angelico]

On Tue, Dec 30, 2014 at 4:28 AM, Peter Otten <__pete...@web.de> wrote:
> Maybe the interpreter itself is corrupted? Google suggests that debsums may
> be the tool to find out.

That's part of what I'm trying to track down, but why it should have
changed in the past few days is beyond me. There's no way that
rebuilding Python 3.5 should break Python 2.7 on the same system,
right? I haven't installed any Debian updates this week.

Just tried debsums, and it isn't complaining about any corruption or damage.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: suggestions for VIN parsing

[Rick Johnson]

On Monday, December 29, 2014 10:58:59 AM UTC-6, Vincent Davis wrote:

> Rick, Thanks for your suggestions, I was just starting
> version2 and wanted to do something like you suggest.
> Another question. I what to change the logic so that
> rather than return THE match it return all matches. I want
> to do this for 2 reasons, 1, it would act as a kinda test,
> If I only expect one match and I get more than I likely
> have a problem,

The best solution for this problem is to raise an exception.
And for completeness sake i would suggest creating a custom
exception.

class VINPatternDuplicityError(Exception):
pass

try:
vin_to_year(vin): # I like Tim's func-name better!
except VINPatternDuplicityError:
print 'Oops, my logic is not sound *enough*!'
else:
print 'Huh! Crossing my fingers and toes really does work!'

> 2, I found a more extensive (maybe better)
> list of frame numbers, I could see some overlapping
> although I have not looked real close yet.

Sorry, i don't have time to study the data set --  but If
your data set contains overlaps *within* the same "mapped set"
then you're forced to handle those as special cases. 


 Also, i just realized a few minor flaws with my approach:


1. Since my logic is *ONLY* iterating over the keys of the
mapping set, a good old "linear-access data structure" (like
list or tuple) would be a better choice for each of the
"mapping sets", and by using a linear data structure, you can
control the order of range tests.

This could be useful in catching corner cases early!

2. I used "tuples-for-keys-and-strings-for-values" in each
of my unique map sets, and although Python will happily
accept this sort of hash, i can't help but feel that it is
in some way "unnatural". Although we are looking up year
values based on ranges, and the natural "left-to-right"
order dictates *unconsciously* that keys should be ranges
and values should be years, iteration is not bounded by
these ritualized "reading and comprehension laws", NO, and
as such, storing year values as keys and ranges as values is
perfectly acceptable -- and only requires a small change to
the iteration logic.

However, if you ever *did* decide to change the order from a
"left-to-right" to a "right-to-left" please be sure to write
a thoughtful comment explaining how and why this approach
was taken, because, even though we (as programmers) tend to
be a logical bunch, we are still slaves to unconscious
"natural instincts" when reading and comprehending written
text, and as such, our comprehension system can break down
when faced with relationship that contradicts our
internalized idea of what is "natural".

Somewhere a malevolent English teacher is smacking a ruler
on her palm and smiling a most devilish grin. <--SADIST!
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 2.7 segfaults on 'import site'

[Peter Otten]

Chris Angelico wrote:

> On Tue, Dec 30, 2014 at 4:28 AM, Peter Otten <__pete...@web.de> wrote:
>> Maybe the interpreter itself is corrupted? Google suggests that debsums
>> may be the tool to find out.
> 
> That's part of what I'm trying to track down, but why it should have
> changed in the past few days is beyond me. There's no way that
> rebuilding Python 3.5 should break Python 2.7 on the same system,
> right? 

It will be easier to reason about that once we know what is broken ;)

> I haven't installed any Debian updates this week.
> 
> Just tried debsums, and it isn't complaining about any corruption or
> damage.

OK. sre_constants.py looks pretty generic, the only module it imports (_sre) 
is a built-in and the interpreter is known-good. If the modules imported 
before sre_constants.py are known-good, too, and no other debian user sees 
the same problem I'm out of realistic ideas.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 2.7 segfaults on 'import site'

[Ian Kelly]

On Mon, Dec 29, 2014 at 9:28 AM, Chris Angelico  wrote:
> Why is sre_constants segfaulting?! It's such a simple file! I assume
> the "matches" comment means it thinks there's no problem with the .pyc
> file; is it possible that, despite those checks, there's an issue
> there? I could delete the .pyc and try again, but I'm scared that'll
> fix the problem and leave me unable to figure out the true cause :)

You could try renaming the .pyc instead of deleting it.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: suggestions for VIN parsing

[Rick Johnson]

On Monday, December 29, 2014 12:27:33 PM UTC-6, Rick Johnson wrote:
>
> [...]
>

[Addendum]


 WHEN LOGICAL AND SUBJECTIVE CONSTRUCTS INTERSECT,
 "SEMANTIC WARS" ARE WAGED!


I believe this thread has exposed the putrid entrails of the
unconscious "semantic wars" for which we are all
unwitting participants when reading and writing code. It is a
war for which there are no winners, and only losers. It is
a war propagated by us, and then waged against *US*! With
the goal of infecting our code bases with incomprehensible
constructs.

The interpretation of source code is far too subjective, as
evidenced by the ubiquitous spaghetti code in the wild, and
we must produce solutions that will more rigidity restrict
the use of our lexical constructs, in such a precise manner
as to reduce (or even remove) any loopholes for which
subjective constructs may enter.

Merely relying on "faith" that people will practice "safe
coding" is as wishful a fantasy as hoping that crack heads
or viral hosts will practice "safe sex". And don't get in a
hurry to pat yourself on the back just because you provided
a "candy bowl" filled with free prophylactics as some
"gesture" of your "altruistic nature", because:

  *SURPRISE*, THEY WON'T TO USE THEM!

Instead of living in a world of "wishful fantasies" and
"hoping" that people will do the "right thing", you need to
design your language in a manner that will prevent them from
doing the "wrong thing"! That will prevent them from
injecting subjective constructs into what should be a purely
logical process!

I think we can all agree that there is nothing "subjective"
about the *mechanics* of "iteration" (or can we?). Any
attempt to "re-interpret" the *mechanics* of iteration, or
understand it from a subjective perspective is fool-hearty.
Iteration *IS* by definition: "moving along a linear path
from one target to the next target until the targets are
exhausted (or the process is interrupted)" Note: "linear"
does not necessarily mean a "strait path" --neither in a
two-dimensional or three-dimensional Euclidean sense-- the
linear aspect of iteration is not in "relative terms" (as in
the directional vector from one target to another target),
but in "absolute terms" (as in the process of moving from a
predefined beginning to an predefined end).

   I am the alpha and the omega!

Iteration is a *truly* logical process, whereas, the
comprehension of free-form lexical constructs is highly
subjective (like: hash key->value pairs). So what do we do
when faced with such "semantically dichotomies"? Should we
give into our basic ritualized instincts? In some cases the
answer is not so clear. Consider the example "mappings" that
i posted earlier which could be defined in two distinct
manners -- both of which inject subjective constructs!


OPTION 1: Define the keys as ranges and the values as years.

  From the perspective of the mapping, the semantic order is
  flipped. The "range" has no semantical meaning UNLESS it
  is defined by a *year*. From this perspective, the "year"
  is all that gives *meaning* to the "range", not the
  inverse!
  
  However, from the perspective of the loop (which iterates
  over keys), the order is correct. The logic of the loop
  *demands* that "ranges" give meaning to "years", and NOT the
  inverse!


OPTION 2: Define the keys as years and the values as ranges.
 
  total inverse transformation of option 1
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 2.7 segfaults on 'import site'

[Chris Angelico]

On Tue, Dec 30, 2014 at 7:17 AM, Ian Kelly  wrote:
> On Mon, Dec 29, 2014 at 9:28 AM, Chris Angelico  wrote:
>> Why is sre_constants segfaulting?! It's such a simple file! I assume
>> the "matches" comment means it thinks there's no problem with the .pyc
>> file; is it possible that, despite those checks, there's an issue
>> there? I could delete the .pyc and try again, but I'm scared that'll
>> fix the problem and leave me unable to figure out the true cause :)
>
> You could try renaming the .pyc instead of deleting it.

Hmm, and in doing so I just learned that they don't, after all, have
any sort of timestamp in them - I thought they did. Renamed it, reran
"python -v", it created a new .pyc with the same MD5 sum. So, it's not
that. Weird.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 2.7 segfaults on 'import site'

[Chris Angelico]

On Tue, Dec 30, 2014 at 5:58 AM, Peter Otten <__pete...@web.de> wrote:
> OK. sre_constants.py looks pretty generic, the only module it imports (_sre)
> is a built-in and the interpreter is known-good. If the modules imported
> before sre_constants.py are known-good, too, and no other debian user sees
> the same problem I'm out of realistic ideas.

I'm looking at the source for it, and it *mentions* _sre, but doesn't
seem to actually import it. sre_compile.py has already imported it,
apparently successfully, before sre_constants loads up.

Ah! The issue doesn't seem to be sre_constants per se; I can import
that just fine. In fact, "import re" segfaults *after* importing
sre_constants. If I take a copy of a file into the current directory
and run "python -Sc 'import re'", it uses my copy of the file (though
just "python" doesn't, which probably means dot isn't added to
sys.path soon enough?); and by using a tweaked version of one file or
other, I've managed to narrow down the problem. And simply copying
sre_parse.py locally prevents the segfault, even without edits!

By inserting "raise Exception" at the point where I think the problem
is, I get this traceback, which approximates to the call depth I've
dug into so far:

$ python -Sc 'import re'
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib/python2.7/re.py", line 222, in 
_pattern_type = type(sre_compile.compile("", 0))
  File "/usr/lib/python2.7/sre_compile.py", line 500, in compile
p = sre_parse.parse(p, flags)
  File "sre_parse.py", line 665, in parse
raise Exception
Exception

Unlike playing with sre_constants.pyc, this one *does* result in a
different file after renaming away the .pyc. So somehow, SOMEHOW, the
.pyc file became corrupt. Is this something worth reporting? I now
have what appears to be a file whose presence in the current directory
can crash Python.

Thanks for the help, all! At least now I have a working system again.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: suggestions for VIN parsing

[Rick Johnson]

[Addendum #2]

 WHO'S RESPONSIBLE FOR THE "SEMANTIC WARS"?


I've always believed in the philosophy of:

  "Responsibility to the responsible". 
  
After reading that statement, a person might think that the
ultimate responsibility for bad code composition is the
programmer, but if you've come that conclusion, it's obvious
you only understand *superficially* what the statement means.
Later i will enlighten you on who is directly responsible
for bad code, but first, let's go on a slight tangent to gain
some context.

A celebrity by the name of "Philip Seymour Hoffman" was
recently in the news. For those of you who don't know, he
died of a illegal drug overdose. Personally i never knew
much of the guy until his name was all over news. And i
really don't care that he's dead or alive -- i'm quite
indifferent on the subject. But he provides us with a great
example of "responsibility to the responsible".

Before proceeding i must underscore that i personally feel
prohibition of any substances is counterproductive to free
societies. I don't use any substances myself, but if other
people enjoy drinking "Drain-O", who am *I* to stop them?
They are only hurting themselves.

SO WHO IS RESPONSIBLE FOR "Hoffman's" DEATH?

Not long after the "celeb" croaked, the police and media
were pounding the war drums to find the "despicable" drug
dealer who killed Mr. Hoffman. The police even claimed to be
"casting a wide net" and that this drug dealer was going to
be "punished severely". Hmm, i wonder if they would have gone
to all that trouble for "Joe Crackhead"?

FREAKING HYPOCRITES!

What did the dealer do? All he did was to provide a grown
man with a substance that the man wanted. Hoffman knew damn
good and well that cocaine and heroin were deadly, yet, he
made the conscience choice to ingest them -- so i argue he
got what he deserved!

As a "freedom fan-boy", i fully support my fellow citizens
right to harm themselves, but don't expect me to waste one
second of pity on a dead drug addict!

SO YOU"RE SAYING THAT PROGRAMMERS ARE RESPONSIBLE?

No, i'm saying the opposite! While the sole blame for
willfully injecting toxins should be on the drug user and
not the dealer, the sole blame for bad code in the wild is
NOT the programmer, but the the language designer himself.

WHAT? HOW DO YOU RESOLVE SUCH LOGIC?

Because drug-use harms *ONLY* those who choose to ingest the
drugs, whereas bad code harms *EVERYONE* in the coding
community -- and even those outside the community! 

EVER HEARD OF M$?

I can write all the good code i want, but if you choose to
write bad code then your code is going to reduce the
"collective quality" of *all* code. And worse, if i'm forced
to maintain your spaghetti CRAP, i will suffer from your
mistakes.

AND *I* SHOULD NOT HAVE TO SUFFER FOR *YOUR* MISTAKES!

You see, this is yet another example of how the
"degenerates" never suffer. They just keep pumping out
garbage that the rest of us are forced to eat. And why should
they stop?

Language designers should take a lesson from evolution.
Evolution does not suffer the weak nor does it have pity for
the stupid. Evolution is only concerned with the process of
"self perfection". It will cast off *any* weight that is
holding it down. It will cull *any* entity that does not
possess traits of the highest quality currently available.

LANGUAGE DESIGNERS SHOULD NOT DEAL IN "PITY"

Instead of giving programmers tools for which to slow and
even destroy the collective evolution of code, language
designers should use programmers as pawns as part of a
"collective mind-hive" who will evolve the collective code
base along a strait and narrow path, a path that leads to
perfection, but that *never* allows attaining it!

THAT COOKIE JAR SHOULD ALWAYS BE *JST* OUTSIDE ARMS LENGTH!

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 2.7 segfaults on 'import site'

[Gregory Ewing]

Chris Angelico wrote:

On Tue, Dec 30, 2014 at 7:17 AM, Ian Kelly  wrote:


You could try renaming the .pyc instead of deleting it.


Hmm, and in doing so I just learned that they don't, after all, have
any sort of timestamp in them - I thought they did.


I think it contains the timestamp of the .py file that
it was created from, to facilitate detecting when it's
out of date.

% python -c 'import foo'
% md5 foo.pyc
MD5 (foo.pyc) = 4082584f949c04d46300281492ddbd5d
% touch foo.py
% python -c 'import foo'
% md5 foo.pyc
MD5 (foo.pyc) = b3445bc53fafbfdaf73f3c3611acbd75

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: suggestions for VIN parsing

[Rick Johnson]

[ADDENDUM #3]

I got off on a tangent in that last post, now i intend to re-focus the 
discussion.

> OPTION 1: Define the keys as ranges and the values as years.
> 
>   From the perspective of the mapping, the semantic order is
>   inversed. The "range" has no semantical meaning UNLESS it
>   is defined by a *year*. From this perspective, the "year"
>   is all that gives *meaning* to the "range", not the
>   inverse!
>   
>   However, from the perspective of the loop (which iterates
>   over keys), the order is correct. The logic of the loop
>   *demands* that "ranges" give meaning to "years", and NOT the
>   inverse!
> 
> 
> OPTION 2: Define the keys as years and the values as ranges.
>  
>   total inverse transformation of of option 1


  "But Rick, your getting all "semantical" about
  "semantics" -- what does all this mean?"
 
My main point is that: the majority of programmers are
inherently better at "engineering their own solution" than
"comprehending someone else's solution". And that fact is a
detriment to evolution of the "collective code" base.

  "And why is that?"

Well the answer is simple, but the solution is not! 

When you are starting a new project, and before you start
writing code, you assemble a "base architecture" of the code
in your mind. And even though this architecture might be
highly abstracted, you still have a good basis from
which to understand the code for which your are about to
write.

However, when you attempt to comprehend some else's code,
not only are you missing the author's "base architecture",
you will inevitably make assumptions about the code that
reflect your own methods of solving problems, which might be
in total conflict to the current authors design!

What the programming community is missing is a concept known
to graphics folks as: "Level Of Detail" (LOD).

LOD can be transformed into CS as "levels of architecture
abstraction". The idea is that as you move further and
further away from a "model" that you should only "see" the
relevant bits. 

For instance, If you have a model of the earth and moon,
and your viewing the earth from the moon, than drawing the
people on the earth does not make any sense, because you
could not see them from such a distance.

Just as you move further and further away from a real object
(say a building), you start to see less and less detail,
HOWEVER, you begin to see the *extents* of the building. You
begin to understand the global architecture of the model. But
more importantly, you understand the whole without
subjecting yourself to "subjective documentations"

HOW DO WE TRANSFORM LOGIC WITHOUT SUBJECTIVE INTERFERENCE?

Understanding the "concept" is quite simple, however,
transforming the concept into practice is quite difficult.
For instance, opening up a code file and viewing a function,
or viewing an entire class (or other larger structure) is
not going to help you understand the global hierarchy of the
entire program. 

"That's what documentation is for Rick!"

Ha! Well that depends on the writing prowess of the author.
And even *if* the author has honed his skills, he can NEVER
guarantee that every mind which reads his words will
transform the concepts "perfectly". No, i'm guess i'm more
realist on the subject than yourself.

Instead of forcing every programmer to be an award winning
documentation expert, i argue we must move away from the
subjective influence of documentation, and move forcefully
into a world of logical representations, exposed by "levels
of detail" that *concretely* define abstractions in the
purest form.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 2.7 segfaults on 'import site'

[Steven D'Aprano]

Chris Angelico wrote:

> Unlike playing with sre_constants.pyc, this one *does* result in a
> different file after renaming away the .pyc. So somehow, SOMEHOW, the
> .pyc file became corrupt. Is this something worth reporting? I now
> have what appears to be a file whose presence in the current directory
> can crash Python.

Time to run some SMART tests on your hard drive, methinks.


-- 
Steven

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 2.7 segfaults on 'import site'

[Chris Angelico]

On Tue, Dec 30, 2014 at 12:40 PM, Steven D'Aprano
 wrote:
> Chris Angelico wrote:
>
>> Unlike playing with sre_constants.pyc, this one *does* result in a
>> different file after renaming away the .pyc. So somehow, SOMEHOW, the
>> .pyc file became corrupt. Is this something worth reporting? I now
>> have what appears to be a file whose presence in the current directory
>> can crash Python.
>
> Time to run some SMART tests on your hard drive, methinks.

There's no other evidence of failure, SMART data shows no errors, and
I can copy the file to another directory and replicate the issue. It
appears to be a perfectly normal file, and one that can crash Python.
But maybe this falls into the same category as "you can crash Python
with ctypes, it's pretty easy".

Are .pyc files compatible across revisions? Could I carry this file to
a 2.7.9 and see if the crash still happens?

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: suggestions for VIN parsing

[Chris Angelico]

On Tue, Dec 30, 2014 at 10:45 AM, Rick Johnson
 wrote:
> [ADDENDUM #3]

This reminds me of the transcript of "Still Alive" at the end of Portal.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 2.7 segfaults on 'import site'

[Ian Kelly]

On Mon, Dec 29, 2014 at 7:05 PM, Chris Angelico  wrote:
> Are .pyc files compatible across revisions? Could I carry this file to
> a 2.7.9 and see if the crash still happens?

PEP 6 requires that .pyc files for a particular major release must
work with all the bug fix releases for that version.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 2.7 segfaults on 'import site'

[Ian Kelly]

On Mon, Dec 29, 2014 at 11:34 PM, Ian Kelly  wrote:
> On Mon, Dec 29, 2014 at 7:05 PM, Chris Angelico  wrote:
>> Are .pyc files compatible across revisions? Could I carry this file to
>> a 2.7.9 and see if the crash still happens?
>
> PEP 6 requires that .pyc files for a particular major release must
> work with all the bug fix releases for that version.

Although since it sounds like the issue is corrupt byte code I'd be
surprised if the result isn't still a crash in 2.7.9.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 2.7 segfaults on 'import site'

[Chris Angelico]

On Tue, Dec 30, 2014 at 5:34 PM, Ian Kelly  wrote:
> On Mon, Dec 29, 2014 at 7:05 PM, Chris Angelico  wrote:
>> Are .pyc files compatible across revisions? Could I carry this file to
>> a 2.7.9 and see if the crash still happens?
>
> PEP 6 requires that .pyc files for a particular major release must
> work with all the bug fix releases for that version.

Cool. I've tried it on a 2.7.8 (the most recent that I have handy),
and it still crashed, though I haven't yet grabbed a 2.7.9.

Using uncompyle, I compared the two files. They're identical after
decompilation but for one line:

# Original file:
def parse(str, flags=0, pattern=None):
# Working pyc, decompiled:
def parse(str, flags = 0, pattern = None):
# Broken pyc, decompiled:
def parse(str, flags, pattern, source, p = 0, tail = None):

So once again, I have some information... and another "Huh?". The
names source, p, and tail are the other local names used in that
function, so it seems that somehow the number of arguments has become
corrupted... in an otherwise-perfect file. The files are nearly
identical, but it's not a simple corruption; there's a byte inserted
at one point and deleted at another. This doesn't look like a failing
hard drive, but I don't know what it *does* look like.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list