Re: [Tutor] Why isn't my simple if elif script not working?

2012-07-18 Thread Steven D'Aprano

Alexandre Zani wrote:


What you want to write is this:

elif name == "John Cleese" or name == "Michael Palin":


elif name in ("John Cleese", "Michael Palin"):

is better.


--
Steven

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor


Re: [Tutor] Why isn't my simple if elif script not working?

2012-07-18 Thread Alexandre Zani
On Wed, Jul 18, 2012 at 6:09 AM, Steven D'Aprano  wrote:
> Alexandre Zani wrote:
>
>> What you want to write is this:
>>
>> elif name == "John Cleese" or name == "Michael Palin":
>
>
> elif name in ("John Cleese", "Michael Palin"):
>
> is better.
>
>
> --
> Steven
>
>
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/mailman/listinfo/tutor

Better how?
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor


Re: [Tutor] Why isn't my simple if elif script not working?

2012-07-18 Thread Steven D'Aprano

Alexandre Zani wrote:

On Wed, Jul 18, 2012 at 6:09 AM, Steven D'Aprano  wrote:

Alexandre Zani wrote:


What you want to write is this:

elif name == "John Cleese" or name == "Michael Palin":


elif name in ("John Cleese", "Michael Palin"):

is better.



Better how?



It's shorter, you don't have to repeat the reference to `name` twice, it is 
more easily extensible if you add additional names, and it is likely to be a 
*tiny* bit faster -- it moves the equality test out of pure-Python code into a 
tuple method, which will be written in C.




--
Steven
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor


Re: [Tutor] Why isn't my simple if elif script not working?

2012-07-18 Thread Alexandre Zani
On Wed, Jul 18, 2012 at 7:44 AM, Steven D'Aprano  wrote:
> Alexandre Zani wrote:
>>
>> On Wed, Jul 18, 2012 at 6:09 AM, Steven D'Aprano 
>> wrote:
>>>
>>> Alexandre Zani wrote:
>>>
 What you want to write is this:

 elif name == "John Cleese" or name == "Michael Palin":
>>>
>>>
>>> elif name in ("John Cleese", "Michael Palin"):
>>>
>>> is better.
>
>
>> Better how?
>
>
>
> It's shorter, you don't have to repeat the reference to `name` twice, it is
> more easily extensible if you add additional names, and it is likely to be a
> *tiny* bit faster -- it moves the equality test out of pure-Python code into
> a tuple method, which will be written in C.
>

Wadayano... You're right, it is faster!

Thanks.


>
>
>
> --
> Steven
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/mailman/listinfo/tutor
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor


[Tutor] writing function changeColor

2012-07-18 Thread Aditi Pai
Hello,

I am trying to write a function changeColor for an assignment. Two things
that I am unsure about for this assignment are how to assign different
colors to integers so that, red will be 1, blue will be 2, etc. Also, I
learned python so that if I were to put in 0.9, it'd decrease red by 10%.
The way this function needs to be written, -0.1 decreases red by 10%

Where can I find information on these two topics?
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor


Re: [Tutor] writing function changeColor

2012-07-18 Thread Emile van Sebille

On 7/18/2012 10:10 AM Aditi Pai said...

Hello,

I am trying to write a function changeColor for an assignment.


I'd look in the documentation of whatever it is you're trying to change 
the color of.  It should be explained there how to set and change colors.


You're going to end up with something that could look like:

def change_color_to(new_color):
color_of_thing = new_color

def modify_color_of_thing(by_n_pct):
color_of_thing = old_color_of_thing * by_n_pct

where the docs of the thing you're changing provide the names and 
handling of modifying the color.


HTH,

Emile



Two
things that I am unsure about for this assignment are how to assign
different colors to integers so that, red will be 1, blue will be 2,
etc. Also, I learned python so that if I were to put in 0.9, it'd
decrease red by 10%. The way this function needs to be written, -0.1
decreases red by 10%

Where can I find information on these two topics?


___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor





___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor


Re: [Tutor] writing function changeColor

2012-07-18 Thread Aditi Pai
Hey Emile! Thanks for the advice. I think maybe I should have combined the
two parts of my email. The function is just called changeColor. I don't
actually want to change the color as much as alter the color. I'm working
off of this example:

def decreaseRed(picture):
for p in getPixels(picture):
value=getRed(p)
setRed(p,value*0.5)

Except for me, I am asked to also make the argument so that it accepts an
integer and a float. Not sure how to assign different colors and do that.

On Wed, Jul 18, 2012 at 2:05 PM, Emile van Sebille  wrote:

> On 7/18/2012 10:10 AM Aditi Pai said...
>
>  Hello,
>>
>> I am trying to write a function changeColor for an assignment.
>>
>
> I'd look in the documentation of whatever it is you're trying to change
> the color of.  It should be explained there how to set and change colors.
>
> You're going to end up with something that could look like:
>
> def change_color_to(new_color):
> color_of_thing = new_color
>
> def modify_color_of_thing(by_n_**pct):
> color_of_thing = old_color_of_thing * by_n_pct
>
> where the docs of the thing you're changing provide the names and handling
> of modifying the color.
>
> HTH,
>
> Emile
>
>
>  Two
>> things that I am unsure about for this assignment are how to assign
>> different colors to integers so that, red will be 1, blue will be 2,
>> etc. Also, I learned python so that if I were to put in 0.9, it'd
>> decrease red by 10%. The way this function needs to be written, -0.1
>> decreases red by 10%
>>
>> Where can I find information on these two topics?
>>
>>
>> __**_
>> Tutor maillist  -  Tutor@python.org
>> To unsubscribe or change subscription options:
>> http://mail.python.org/**mailman/listinfo/tutor
>>
>>
>
>
> __**_
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/**mailman/listinfo/tutor
>
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor


Re: [Tutor] writing function changeColor

2012-07-18 Thread bob gailer

On 7/18/2012 1:10 PM, Aditi Pai wrote:

Hello,

I am trying to write a function changeColor for an assignment.

Is this a class assignment or job-related?

Point us to the assignment, Learn to ask meaningful questions. Otherwise 
you waste your time and ours.
"assign different colors to integers so that, red will be 1, blue will 
be 2, etc."
We need a LOT more detail here. color of what? What does etc mean 
(implies we should be able to figure out what comes next) Could be anything!
Also, I learned python so that if I were to put in 0.9, it'd decrease 
red by 10%.

Makes no sense.

The way this function needs to be written, -0.1 decreases red by 10%

Where can I find information on these two topics?

Google?

--
Bob Gailer
919-636-4239
Chapel Hill NC

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor


Re: [Tutor] writing function changeColor

2012-07-18 Thread Aditi Pai
Dear Bob,

I am trying my best to learn and your attitude was not welcome. If you
don't want to answer my question, I understand. I'm doing research
elsewhere too and because I am new at this, my familiarity with key terms
and search words is also improving, yet at a slower pace. My goal is to
find people who would help me find the resources I need and not cheat,
which is why I didn't want to post my assignment.

Thanks,
Aditi Pai

On Wed, Jul 18, 2012 at 3:14 PM, bob gailer  wrote:

> On 7/18/2012 1:10 PM, Aditi Pai wrote:
>
>> Hello,
>>
>> I am trying to write a function changeColor for an assignment.
>>
> Is this a class assignment or job-related?
>
> Point us to the assignment, Learn to ask meaningful questions. Otherwise
> you waste your time and ours.
>
>  "assign different colors to integers so that, red will be 1, blue will be
>> 2, etc."
>>
> We need a LOT more detail here. color of what? What does etc mean (implies
> we should be able to figure out what comes next) Could be anything!
>
>  Also, I learned python so that if I were to put in 0.9, it'd decrease red
>> by 10%.
>>
> Makes no sense.
>
>  The way this function needs to be written, -0.1 decreases red by 10%
>>
>> Where can I find information on these two topics?
>>
> Google?
>
> --
> Bob Gailer
> 919-636-4239
> Chapel Hill NC
>
>
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor


Re: [Tutor] writing function changeColor

2012-07-18 Thread Emile van Sebille

On 7/18/2012 12:07 PM Aditi Pai said...

Hey Emile! Thanks for the advice. I think maybe I should have combined
the two parts of my email. The function is just called changeColor. I
don't actually want to change the color as much as alter the color.


I'm not sure there's a difference...


I'm working off of this example:

def decreaseRed(picture):
for p in getPixels(picture):
value=getRed(p)
setRed(p,value*0.5)

Except for me, I am asked to also make the argument so that it accepts
an integer and a float.


So, if by that you mean that the function should accept two passed 
values, one an integer and the other a float, then your def line should 
look like:


def changeColor(myint, myfloat):

If, on the other hand, you want a single passed value that can be either 
an integer or a float you'd just have:


def changeColor(myiint):



Not sure how to assign different colors and do that.



If the problem if adapting the example so that the function should 
accept both the picture thingy and a value to replace the hardcoded .5, 
your function def might look like:


def changeColor(picture, myRedIntOrFloatVal):

and then swap out the .5 for myRedIntOrFloatVal

What have you written?

Emile

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor


Re: [Tutor] writing function changeColor

2012-07-18 Thread Aditi Pai
Emile,

So far I have started with

def changeColor(pict,scale,color):

I was told to make different names for the float and integer (float = scale
and color= integer) and then I kept everything else the same, just to test
it out and see if that would work.

So it looks like this:

def changeColor(pict,scale,color):
  for p in getPixels(pict):
value=getRed(p)
setRed(p,value*.9)

Unfortunately I keep getting this error message:

The error was:changeColor() takes at least 3 arguments (1 given)
Inappropriate argument type.
An attempt was made to call a function with a parameter of an invalid type.
This means that you did something such as trying to pass a string to a
method that is expecting an integer.

On Wed, Jul 18, 2012 at 3:20 PM, Emile van Sebille  wrote:

> On 7/18/2012 12:07 PM Aditi Pai said...
>
>  Hey Emile! Thanks for the advice. I think maybe I should have combined
>> the two parts of my email. The function is just called changeColor. I
>> don't actually want to change the color as much as alter the color.
>>
>
> I'm not sure there's a difference...
>
>
>  I'm working off of this example:
>>
>> def decreaseRed(picture):
>> for p in getPixels(picture):
>> value=getRed(p)
>> setRed(p,value*0.5)
>>
>> Except for me, I am asked to also make the argument so that it accepts
>> an integer and a float.
>>
>
> So, if by that you mean that the function should accept two passed values,
> one an integer and the other a float, then your def line should look like:
>
> def changeColor(myint, myfloat):
>
> If, on the other hand, you want a single passed value that can be either
> an integer or a float you'd just have:
>
> def changeColor(myiint):
>
>
>
>  Not sure how to assign different colors and do that.
>>
>
>
> If the problem if adapting the example so that the function should accept
> both the picture thingy and a value to replace the hardcoded .5, your
> function def might look like:
>
> def changeColor(picture, myRedIntOrFloatVal):
>
> and then swap out the .5 for myRedIntOrFloatVal
>
> What have you written?
>
> Emile
>
>
> __**_
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/**mailman/listinfo/tutor
>
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor


Re: [Tutor] writing function changeColor

2012-07-18 Thread Prasad, Ramit
Please do not top post.

> I am trying my best to learn and your attitude was not welcome. If you don't
> want to answer my question, I understand. I'm doing research elsewhere too and
> because I am new at this, my familiarity with key terms and search words is
> also improving, yet at a slower pace. My goal is to find people who would help
> me find the resources I need and not cheat, which is why I didn't want to post
> my assignment.

> > > I am trying to write a function changeColor for an assignment. Two things 
> > > that I am unsure about for this assignment are how to assign different 
> > > colors to integers so that, red will be 1, blue will be 2, etc. Also, I 
> > > learned python so that if I were to put in 0.9, it'd decrease red by 10%. 
> > > The way this function needs to be written, -0.1 decreases red by 10%

The problem is that you have given us an incomplete description of what
you need. Without a full description, it is difficult to answer without
making a host of assumptions that are probably all invalid. If you can 
provide a full description then we can help better. Even a code sample 
would give us some context. 

>From your original email, I have no idea what UI framework you refer to
(if any), how you get a color or change a color or anything meaningful. 
Although, you might not like Bob's tone he makes a very valid point. 
If you learn to ask better questions, you get better answers.

Thankfully you ended up replying to Emile with some code so I can help
better.

> I was told to make different names for the float and integer (float = scale
> and color= integer) and then I kept everything else the same, just to test it
> out and see if that would work.
> 
> So it looks like this:
> 
> def changeColor(pict,scale,color):
>   for p in getPixels(pict):
> value=getRed(p)
> setRed(p,value*.9)
> 
> Unfortunately I keep getting this error message:
> 
> The error was:changeColor() takes at least 3 arguments (1 given)
> Inappropriate argument type.
> An attempt was made to call a function with a parameter of an invalid type.
> This means that you did something such as trying to pass a string to a method
> that is expecting an integer.

Please always post the exact trace back and not just a portion.
It might not seem like all that is needed is the error message
but it can make a big difference in the amount of help.

Luckily, I can tell what the problem is from this error. You are using 
`changeColor(picture)` which no longer works since you changed changeColor 
to take 3 arguments so you need to change each call to 
`changeColor(picture,scale,color)`.

Also, instead of saying getRed(p), I would do getColor(p, color).
Something like below; although, I leave the implementation of
getColor/setColor up to you. Oh, and I fixed your indentation problem.
(Could have been caused by posting in rich/HTML text and not plain text)

def changeColor( pict, scale color ):
''' Change the intensity of color.
scale can be -1.0 <= x <= 1.0
'''
for p in getPixels(pict):
value=getColor(p,color)
change = 1.0 + scale # -.1 now equals .9 
setColor(p, color, value * change )


Often in Python, you do not worry about whether a number
is a float or an int (unless it happens to be part of the assignment), 
you just do what you need and check that it worked.
You can wrap scale and color in float() or int() respectively
if you want to explicitly convert them.

It might make more sense to take in a % instead, so -10 for scale lessens the 
color by 10%. Lots of UI does things by this using 0.0-1.0 values (as given
in your example), so it makes sense, but it might be easier to think about 
as a percentage. 

Ramit

This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.  
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor


Re: [Tutor] writing function changeColor

2012-07-18 Thread Steven D'Aprano

Emile van Sebille wrote:

On 7/18/2012 12:07 PM Aditi Pai said...

Hey Emile! Thanks for the advice. I think maybe I should have combined
the two parts of my email. The function is just called changeColor. I
don't actually want to change the color as much as alter the color.


I'm not sure there's a difference...



You're being too kind. Of course there isn't a difference. "Change" and 
"alter" are synonyms, so Aditi doesn't want to change the color, but change 
the color instead.


Aditi, that is not very helpful. You're asking us for help, but wasting our 
time with riddles.



--
Steven

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor


Re: [Tutor] writing function changeColor

2012-07-18 Thread Steven D'Aprano

Aditi Pai wrote:

Emile,

So far I have started with

def changeColor(pict,scale,color):

I was told to make different names for the float and integer (float = scale
and color= integer) and then I kept everything else the same, just to test
it out and see if that would work.

So it looks like this:

def changeColor(pict,scale,color):
  for p in getPixels(pict):
value=getRed(p)
setRed(p,value*.9)

Unfortunately I keep getting this error message:

The error was:changeColor() takes at least 3 arguments (1 given)



Is this error message not clear? You have written a function that requires at 
least 3 values, but you have only given 1 value.


I can't tell you what argument you have given, because you have not told us 
what you have done to get this error message.


Aditi, we want to help you, but you're not giving us enough information to go 
on. You're wasting our time, and your own.


Instead of biting at poor Bob, who so far has given you the best and most 
valuable advise, you should pay attention to what we are trying to teach you. 
Would you rather that we just ignored you in silence?


I'm about to move away from the computer for an hour or so. When I come back, 
I will try to give you some more assistance. Or somebody else may try to help. 
In the meantime, I suggest you try reading this page and see if it enlightens you:


http://sscce.org/



--
Steven
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor


[Tutor] string to binary and back... Python 3

2012-07-18 Thread Jordan
OK so I have been trying for a couple days now and I am throwing in the
towel, Python 3 wins this one.
I want to convert a string to binary and back again like in this
question: Stack Overflow: Convert Binary to ASCII and vice versa
(Python)

But in Python 3 I consistently get  some sort of error relating to the
fact that nothing but bytes and bytearrays support the buffer interface
or I get an overflow error because something is too large to be
converted to bytes.
Please help me and then explian what I am not getting that is new in
Python 3. I would like to point out I realize that binary, hex, and
encodings are all a very complex subject and so I do not expect to
master it but I do hope that I can gain a deeper insight. Thank you all.

test_script.py:
import binascii

test_int = 109

test_int = int(str(test_int) + '45670')
data = 'Testing XOR Again!'

while sys.getsizeof(data) > test_int.bit_length():

test_int = int(str(test_int) + str(int.from_bytes(os.urandom(1), 'big')))

print('Bit Length: ' + str(test_int.bit_length()))

key = test_int # Yes I know this is an unnecessary step...

data = bin(int(binascii.hexlify(bytes(data, 'UTF-8')), 16))

print(data)

data = int(data, 2)

print(data)

data = binascii.unhexlify('%x' % data)


wolfrage@lm12-laptop02 ~/Projects $ python3 test_script.py
Bit Length: 134
0b1010100011001010111001101110100011010010110111001100111001001011100010100100010010101100111011101101001011011100011
7351954002991226380810260999848996570230305
Traceback (most recent call last):
File "test_script.py", line 24, in 
data = binascii.unhexlify('%x' % data)
TypeError: 'str' does not support the buffer interface



test_script2.py:
import binascii
test_int = 109
test_int = int(str(test_int) + '45670')
data = 'Testing XOR Again!'
while sys.getsizeof(data) > test_int.bit_length():
test_int = int(str(test_int) + str(int.from_bytes(os.urandom(1), 'big')))
print('Bit Length: ' + str(test_int.bit_length()))
key = test_int # Yes I know this is an unnecessary step...
data = bin(int(binascii.hexlify(bytes(data, 'UTF-8')), 16))
print(data)
data = int(data, 2)
print(data)
data = binascii.unhexlify(bytes(data, 'utf8'))



wolfrage@lm12-laptop02 ~/Projects $ python3 test_script2.py
Bit Length: 140
0b1010100011001010111001101110100011010010110111001100111001001011100010100100010010101100111011101101001011011100011
7351954002991226380810260999848996570230305
Traceback (most recent call last):
File "test_script.py", line 24, in 
data = binascii.unhexlify(bytes(data, 'utf8'))
OverflowError: cannot fit 'int' into an index-sized integer

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor


Re: [Tutor] writing function changeColor

2012-07-18 Thread Devin Jeanpierre
On Wed, Jul 18, 2012 at 3:20 PM, Aditi Pai  wrote:
> I am trying my best to learn and your attitude was not welcome. If you don't
> want to answer my question, I understand. I'm doing research elsewhere too
> and because I am new at this, my familiarity with key terms and search words
> is also improving, yet at a slower pace. My goal is to find people who would
> help me find the resources I need and not cheat, which is why I didn't want
> to post my assignment.

If you don't post the assignment, we don't necessarily know what
advice would constitute cheating.

It's especially important for university students, who can be expelled
for helping other students cheat -- even if they didn't know it was
cheating and the cheater was in another country at another school.

-- Devin
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor


Re: [Tutor] writing function changeColor

2012-07-18 Thread Aditi Pai
Ramit told me not to "top post," and I looked it up and I think I am doing
that same thing again, so if I am let me know how to avoid that. This kind
of discussion board is something I haven't interacted with before.

Steven, thank you for trying to help me. I thought my response to Bob was
completely civil and you seem like a firey character, so I'll try to
respond as evenly as I can to you too. Like I said before, I am on a
learning curve, and the messages Emile and Ramit have sent me both had
constructive criticism as to how to proceed with asking questions as well
as awesome answers to my poorly-worded questions. The little side comments
were not included. I don't mean to "waste your time" or Bob's apparently
but asking that question led me to understand how to ask a better one. I am
trying to improve and I've just started learning. Please don't be mean.
Thank you.

On Wed, Jul 18, 2012 at 4:42 PM, Steven D'Aprano wrote:

> Aditi Pai wrote:
>
>> Emile,
>>
>> So far I have started with
>>
>> def changeColor(pict,scale,color):
>>
>> I was told to make different names for the float and integer (float =
>> scale
>> and color= integer) and then I kept everything else the same, just to test
>> it out and see if that would work.
>>
>> So it looks like this:
>>
>> def changeColor(pict,scale,color):
>>   for p in getPixels(pict):
>> value=getRed(p)
>> setRed(p,value*.9)
>>
>> Unfortunately I keep getting this error message:
>>
>> The error was:changeColor() takes at least 3 arguments (1 given)
>>
>
>
> Is this error message not clear? You have written a function that requires
> at least 3 values, but you have only given 1 value.
>
> I can't tell you what argument you have given, because you have not told
> us what you have done to get this error message.
>
> Aditi, we want to help you, but you're not giving us enough information to
> go on. You're wasting our time, and your own.
>
> Instead of biting at poor Bob, who so far has given you the best and most
> valuable advise, you should pay attention to what we are trying to teach
> you. Would you rather that we just ignored you in silence?
>
> I'm about to move away from the computer for an hour or so. When I come
> back, I will try to give you some more assistance. Or somebody else may try
> to help. In the meantime, I suggest you try reading this page and see if it
> enlightens you:
>
> http://sscce.org/
>
>
>
> --
> Steven
>
> __**_
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/**mailman/listinfo/tutor
>
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor


Re: [Tutor] writing function changeColor

2012-07-18 Thread Aditi Pai
Hey,

Do you know how to take your post off the discussion board? Devin is right!
I was looking for links or online resources but people keep trying to help
me, which is awesome, but I'd like to do my own research. Thanks!!

On Wed, Jul 18, 2012 at 5:31 PM, Aditi Pai  wrote:

> Ramit told me not to "top post," and I looked it up and I think I am doing
> that same thing again, so if I am let me know how to avoid that. This kind
> of discussion board is something I haven't interacted with before.
>
> Steven, thank you for trying to help me. I thought my response to Bob was
> completely civil and you seem like a firey character, so I'll try to
> respond as evenly as I can to you too. Like I said before, I am on a
> learning curve, and the messages Emile and Ramit have sent me both had
> constructive criticism as to how to proceed with asking questions as well
> as awesome answers to my poorly-worded questions. The little side comments
> were not included. I don't mean to "waste your time" or Bob's apparently
> but asking that question led me to understand how to ask a better one. I am
> trying to improve and I've just started learning. Please don't be mean.
> Thank you.
>
>
> On Wed, Jul 18, 2012 at 4:42 PM, Steven D'Aprano wrote:
>
>> Aditi Pai wrote:
>>
>>> Emile,
>>>
>>> So far I have started with
>>>
>>> def changeColor(pict,scale,color):
>>>
>>> I was told to make different names for the float and integer (float =
>>> scale
>>> and color= integer) and then I kept everything else the same, just to
>>> test
>>> it out and see if that would work.
>>>
>>> So it looks like this:
>>>
>>> def changeColor(pict,scale,color):
>>>   for p in getPixels(pict):
>>> value=getRed(p)
>>> setRed(p,value*.9)
>>>
>>> Unfortunately I keep getting this error message:
>>>
>>> The error was:changeColor() takes at least 3 arguments (1 given)
>>>
>>
>>
>> Is this error message not clear? You have written a function that
>> requires at least 3 values, but you have only given 1 value.
>>
>> I can't tell you what argument you have given, because you have not told
>> us what you have done to get this error message.
>>
>> Aditi, we want to help you, but you're not giving us enough information
>> to go on. You're wasting our time, and your own.
>>
>> Instead of biting at poor Bob, who so far has given you the best and most
>> valuable advise, you should pay attention to what we are trying to teach
>> you. Would you rather that we just ignored you in silence?
>>
>> I'm about to move away from the computer for an hour or so. When I come
>> back, I will try to give you some more assistance. Or somebody else may try
>> to help. In the meantime, I suggest you try reading this page and see if it
>> enlightens you:
>>
>> http://sscce.org/
>>
>>
>>
>> --
>> Steven
>>
>> __**_
>> Tutor maillist  -  Tutor@python.org
>> To unsubscribe or change subscription options:
>> http://mail.python.org/**mailman/listinfo/tutor
>>
>
>
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor


Re: [Tutor] string to binary and back... Python 3

2012-07-18 Thread Prasad, Ramit
> OK so I have been trying for a couple days now and I am throwing in the
> towel, Python 3 wins this one.
> I want to convert a string to binary and back again like in this
> question: Stack Overflow: Convert Binary to ASCII and vice versa
> (Python)
>  versa-python>
> But in Python 3 I consistently get  some sort of error relating to the
> fact that nothing but bytes and bytearrays support the buffer interface
> or I get an overflow error because something is too large to be
> converted to bytes.
> Please help me and then explian what I am not getting that is new in
> Python 3. I would like to point out I realize that binary, hex, and
> encodings are all a very complex subject and so I do not expect to
> master it but I do hope that I can gain a deeper insight. Thank you all.

[ snip program ]

> data = bin(int(binascii.hexlify(bytes(data, 'UTF-8')), 16))
> data = binascii.unhexlify('%x' % data)

[ more snipping ]

> wolfrage@lm12-laptop02 ~/Projects $ python3 test_script.py
> Bit Length: 134
> 0b1010100011001010111001101110100011010010110111001100111001001011100010100100010010101100111011101101001011011100011
> 7351954002991226380810260999848996570230305
> Traceback (most recent call last):
> File "test_script.py", line 24, in 
> data = binascii.unhexlify('%x' % data)
> TypeError: 'str' does not support the buffer interface

I think your basic problem is too much conversion because you do not
understand the types. A string is represented by a series of bytes
which are binary numbers. Do you understand the concept behind ASCII?
Each letter has a numeric representation that are sequential.
So the string 'abcd' is equivalent to a series of bytes 65,66,67,68.
It is not equivalent to 65666768 or 65+66+67+68. So your first
task is to convert each character to the numeric equivalent and store
them in a list. Once you have them converted to a list of integers,
you can create another list that is a list of characters.

Look at the functions chr and ord here
( http://docs.python.org/py3k/library/functions.html )

Ramit


Ramit Prasad | JPMorgan Chase Investment Bank | Currencies Technology
712 Main Street | Houston, TX 77002
work phone: 713 - 216 - 5423

--

This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.  
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor


Re: [Tutor] string to binary and back... Python 3

2012-07-18 Thread Dave Angel
On 07/18/2012 05:07 PM, Jordan wrote:
> OK so I have been trying for a couple days now and I am throwing in the
> towel, Python 3 wins this one.
> I want to convert a string to binary and back again like in this
> question: Stack Overflow: Convert Binary to ASCII and vice versa
> (Python)
> 
> But in Python 3 I consistently get  some sort of error relating to the
> fact that nothing but bytes and bytearrays support the buffer interface
> or I get an overflow error because something is too large to be
> converted to bytes.
> Please help me and then explian what I am not getting that is new in
> Python 3. I would like to point out I realize that binary, hex, and
> encodings are all a very complex subject and so I do not expect to
> master it but I do hope that I can gain a deeper insight. Thank you all.
>
> test_script.py:
> import binascii
>
> test_int = 109
>
> test_int = int(str(test_int) + '45670')
> data = 'Testing XOR Again!'
>
> while sys.getsizeof(data) > test_int.bit_length():
>
> test_int = int(str(test_int) + str(int.from_bytes(os.urandom(1), 'big')))
>
> print('Bit Length: ' + str(test_int.bit_length()))
>
> key = test_int # Yes I know this is an unnecessary step...
>
> data = bin(int(binascii.hexlify(bytes(data, 'UTF-8')), 16))
>
> print(data)
>
> data = int(data, 2)
>
> print(data)
>
> data = binascii.unhexlify('%x' % data)
>

I don't get the same error you did.  I get:

 File "jordan.py", line 13
test_int = int(str(test_int) + str(int.from_bytes(os.urandom(1),
'big')))
   ^
IndentationError: expected an indented block


Please post it again, with correct indentation.  if you used tabs, then
expand them to spaces before pasting it into your test-mode mail editor.


I'd also recommend you remove a lot of the irrelevant details there.  if
you have a problem with hexlfy and/or unhexlify, then give a simple byte
string that doesn't work for you, and somebody can probably identify why
not.  And if you want people to run your code, include the imports as well.

As it is, you're apparently looping, comparing the byte memory size of a
string (which is typically 4 bytes per character) with the number of
significant bits in an unrelated number.

I suspect what you want is something resembling (untested):

mybytes = bytes( "%x" % data, "ascii")
newdata = binascii.unexlify(mybytes)


-- 
DaveA
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor


Re: [Tutor] string to binary and back... Python 3

2012-07-18 Thread Prasad, Ramit
> 
> I think your basic problem is too much conversion because you do not
> understand the types. A string is represented by a series of bytes
> which are binary numbers. Do you understand the concept behind ASCII?
> Each letter has a numeric representation that are sequential.
> So the string 'abcd' is equivalent to a series of bytes 65,66,67,68.
> It is not equivalent to 65666768 or 65+66+67+68. So your first
> task is to convert each character to the numeric equivalent and store
> them in a list. Once you have them converted to a list of integers,
> you can create another list that is a list of characters.
> 
> Look at the functions chr and ord here
> ( http://docs.python.org/py3k/library/functions.html )

I forgot to say, that once you have the integer equivalents, 
you can then convert that easily to binary using bin.
I used ast.literal_eval to convert from binary string 
(as returned from bin) to number, but there might be better
ways.

 
Ramit

This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.  
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor


Re: [Tutor] writing function changeColor

2012-07-18 Thread Steven D'Aprano
On Wed, Jul 18, 2012 at 05:31:00PM -0400, Aditi Pai wrote:
> Ramit told me not to "top post," and I looked it up and I think I am doing
> that same thing again, so if I am let me know how to avoid that. This kind
> of discussion board is something I haven't interacted with before.

In your email program, just move the cursor (the linking line where you 
type) after each comment you want to reply to before answering the 
question.

If your email program won't let you do that, either 

1) use a different, better, email program (I can strongly recommend 
Thunderbird); or

2) apologise, but expect to still be (more, or less, gently) told off 
for top-posting.


> Steven, thank you for trying to help me. I thought my response to Bob was
> completely civil and you seem like a firey character, so I'll try to
> respond as evenly as I can to you too. Like I said before, I am on a
> learning curve, and the messages Emile and Ramit have sent me both had
> constructive criticism as to how to proceed with asking questions as well
> as awesome answers to my poorly-worded questions. The little side comments
> were not included. I don't mean to "waste your time" or Bob's apparently
> but asking that question led me to understand how to ask a better one. I am
> trying to improve and I've just started learning. Please don't be mean.

Nobody here is trying to be mean. If you think this was "firey" or 
mean, you should try asking questions on some of the C programming 
language forums.

But you have to understand, this is the first time you have been in the 
situation of having to ask questions you don't know how to put into 
words. But for us, it is about the millionth time somebody has come here 
asking for help but not giving us enough details that we can help.

Imagine that you are working at a burger restaurant, and somebody comes 
in and asks "I want that burger I had last month, you know, the one with 
the sauce that was so good", and either can't or won't tell you any 
more. You've got twenty different burgers on the menu and thirty 
different sauces, how can you possibly know which one the customer is 
talking about? You want to help, right, but you can't. Pretty 
frustrating, yeah, but you smile and do your best, because that's what 
you're paid for. So you ask them to describe the burger, and they say 
"I had it with fries and a Coke" like that helps.

Now imagine how frustrating it is the tenth time somebody has done it 
this week, and it's only Tuesday. AND YOU'RE A VOLUNTEER. You're not 
being paid, you're doing it for free, to help the community, but 
sometimes it seems like the community is made up of demanding, needy 
dumb-arses who couldn't find their own nose without a team of Sherpas 
and a map...

(No offence :)

Just as we should try to remember what it was like to be in your 
situation, so you (and others like you) should try to imagine what it is 
like to be in our shoes. We're not being paid, we're doing this for free 
out of a desire to help others. A little bit of snarkiness is 
just our way of letting off steam.

Forgive us the occasional grumpiness, and we'll forgive you the 
occasional stupid question.

I'll get back to your actual programming question in a moment.



-- 
Steven
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor


Re: [Tutor] string to binary and back... Python 3

2012-07-18 Thread Steven D'Aprano
On Wed, Jul 18, 2012 at 10:22:43PM +, Prasad, Ramit wrote:

> I forgot to say, that once you have the integer equivalents, 
> you can then convert that easily to binary using bin.
> I used ast.literal_eval to convert from binary string 
> (as returned from bin) to number, but there might be better
> ways.

Do you mean something like this?


py> int('111001010100100', 2)
29348


-- 
Steven
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor


Re: [Tutor] writing function changeColor

2012-07-18 Thread Steven D'Aprano
Coming back to your original question...


On Wed, Jul 18, 2012 at 01:10:51PM -0400, Aditi Pai wrote:
> Hello,
> 
> I am trying to write a function changeColor for an assignment. Two things
> that I am unsure about for this assignment are how to assign different
> colors to integers so that, red will be 1, blue will be 2, etc. Also, I

Is your assignment to invent your own colour scheme? Or to use an 
existing colour scheme?

(I'm Australian, we spell colour with a U.)

If you are supposed to use an existing scheme, then you need to find out 
which one you are supposed to use. I can't help with that.

I can tell you that there are generally two main ways to assign colours:

* palettes (also known as indexed colour)

* colour spaces

Palettes aren't terribly helpful in this question. A palette basically 
is a list of usually 256 integers (but sometimes fewer), each one of 
which arbitrarily represents a colour. So you might have:

0 = light blue
1 = pink
2 = dark green
3 = white
4 = very light grey
...
254 = black
255 = tangerine

or something.

Obviously because the order of colours is arbitrary, it is hard, if not 
impossible, to say "start with tangerine and make it 10% more blue".

Colour spaces are more useful. A colour space defines each colour in 
terms of usually three, but sometimes four, numbers, each one of which 
ranges between 0 and 255. Each number represents a different quality of 
the colour.

There are different colour spaces, because none of them describe colour 
perfectly. Some of them are better for modelling additive colours (like 
from your monitor, which generates coloured light); some are better for 
subtractive colours (like your colour printer). The whole field is 
awfully complex and I'm not an expert, but you can start here:

http://en.wikipedia.org/wiki/Color_space_encoding

I *think* that for your purposes, the best colour space to start with is 
the RGB model. In this, each colour is represented by an amount of red, 
green and blue, added together. For example:

"Black" is the complete absence of any colour, so:

Red = 0
Green = 0
Blue = 0

"White" is a complete mix of all colours, so:

Red = 255
Green = 255
Blue = 255


Everything else is somewhere in between, e.g. "Forest Green":

Red: 33 out of a maximum of 255, or 12.9%
Green: 139, or 54.5%
Blue: 33, or 12.9%

Notice that here the percentage represents the amount out of the maximum 
possible for each value, and then do NOT add to 100% in total.

If you increase the amount of blue by 50%, you get:

Red: 33 or 12.9%
Green: 139 or 54.5%
Blue: 33 + 16.5 = 50 (rounded to the nearest integer) or 19.6%

I have no idea what name this colour might have.

Does this help?


-- 
Steven
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor


Re: [Tutor] writing function changeColor

2012-07-18 Thread Steven D'Aprano
On Wed, Jul 18, 2012 at 05:37:17PM -0400, Aditi Pai wrote:
> Hey,
> 
> Do you know how to take your post off the discussion board? 

Do you mean taking a previous post off the discussion list? You can't. 
Once a post has been made, you can't delete it. It is public knowledge, 
archived on a dozen different websites and hundreds of people's email 
inboxes.

If you mean how to take the discussion off-list and make it private, 
that's easy. Instead of replying to the tutor mailing list, email the 
person you want to talk to directly. Don't forget to take tutor out of 
the CC list.

BUT think VERY hard before doing this. Normally you should keep as much 
of the discussion on-list as possible, for three reasons:

1) We're not doing this just to help *you* personally, but to help the 
many other beginners who are reading but not saying anything. If you 
take the discussion off-list, they will be cut out of the discussion and 
can't learn from your questions or our answers.

2) Often one person will make a comment, and you reply with a question, 
but that person is away from the computer for hours or days or even 
permanently. Or they are too busy to answer, or the question bores them 
and they can't be bothered to answer. If the question is asked on-list, 
somebody else can answer in their stead. If you ask privately, you may 
never get a response.

3) None of us are perfect. I've been programming in Python for well over 
a decade, and I still learn things here occasionally, or make mistakes. 
If the questions remain public, we can *all* learn from each other.


Of course, if you want to say something private or personal, you should 
write privately.



-- 
Steven
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor


[Tutor] Problem When Iterating Over Large Test Files

2012-07-18 Thread Ryan Waples
I'm seeing some unexpected output when I use a script (included at
end) to iterate over large text files.  I am unsure of the source of
the unexpected output and any help would be much appreciated.

Background
Python v 2.7.1
Windows 7 32bit
Reading and writing to an external USB hard drive

Data files are ~4GB text (.fastq) file, it has been uncompressed
(gzip).  This file has no errors or formatting problems, it seems to
have uncompressed just fine.  64M lines, each 'entry' is split across
4 consecutive lines, 16M entries.

My python script iterates over data files 4 lines at a time, selects
and writes groups of four lines to the output file.  I will end up
selecting roughly 85% of the entries.

In my output I am seeing lines that don't occur in the original file,
and that don't match any lines in the original file.  The incidences
of badly formatted lines don't seem to match up with any patterns in
the data file, and occur across multiple different data files.

I've included 20 consecutive lines of input and output.  Each of these
5 'records' should have been selected and printed to the output file.
But there is a problem with the 4th and 5th entries in the output, and
it no longer matches the input as expected.  For example the line:
TTCTGTGAGTGATTTCCTGCAAGACAGGAATGTCAGT
never occurs in the original data.

Sorry for the large block of text below.
Other pertinent info, I've tried a related perl script, and ran into
similar issues, but not in the same places.

Any help or insight would be appreciated.

Thanks


__EXAMPLE RAW DATA FILE REGION__

@HWI-ST0747:167:B02DEACXX:8:1101:3182:167088 1:N:0:
CGCGTGTGCAGGTTTATAGAACCAGCTGCAGATTAGTAGCAGCGCACGGAGAGGTGTGTCTGTTTATTGTCCTCAGCAGGCAGACATGTTTGTGGTC
+
@@@DDADDHB9+2A;6(5@CDAC(5(5:5,(8?88?BC@#
@HWI-ST0747:167:B02DEACXX:8:1101:3134:167090 1:N:0:
TTCTAGTGCAGGGCGACAGCGTTGCGGAGCCGGTCCGAGTCTGCTGGGTCAGTCATGGCTAGTTGGTACTATAACGACACAGGGCGAGACCCAGATGCAAA
+
@CCFFFDFHJJIJHHIIIJHGHIJI@GFFDDDFDDCEEEDCCBDCCCCCB>>@C(4@ADCA>>?BBBDDABB055<>-?AA>AD>BA
@HWI-ST0747:167:B02DEACXX:8:1101:3022:167094 1:N:0:
ATTCCGTGCAGGCCAACTCCCGACGGACATCCTTGCTCAGACTGCAGCGATAGTGGTCGATCAGGGCCCTGTTGTTCCATCCCACTCCGGCGACCAGGTTC
+
CCCFHIDHJIIHIIIJIJIIGGIIFHJIIIIEIFHFF>CBAECBDDDC:??B=AAACD?8@:>C@?8CBDDD@D99B@>3884>A
@HWI-ST0747:167:B02DEACXX:8:1101:3095:167100 1:N:0:
CGTGATTGCAGGGACGTTACAGAGACGTTACAGGGATGTTACAGGGACGTTACAGAGACGTTAAAGAGATGTTACAGGGATGTTACAGACAGAGACGTTAC
+


__EXAMPLE PROBLEMATIC OUTPUT FILE REGION__

@HWI-ST0747:167:B02DEACXX:8:1101:3182:167088 1:N:0:
CGCGTGTGCAGGTTTATAGAACCAGCTGCAGATTAGTAGCAGCGCACGGAGAGGTGTGTCTGTTTATTGTCCTCAGCAGGCAGACATGTTTGTGGTC
+
@@@DDADDHB9+2A;6(5@CDAC(5(5:5,(8?88?BC@#
@HWI-ST0747:167:B02DEACXX:8:1101:3134:167090 1:N:0:
TTCTAGTGCAGGGCGACAGCGTTGCGGAGCCGGTCCGAGTCTGCTGGGTCAGTCATGGCTAGTTGGTACTATAACGACACAGGGCGAGACCCAGATGCAAA
+
@CCFFFDFHJJIJHHIIIJHGHIJI@GFFDDDFDDCEEEDCCBDCCCCCB>>@C(4@ADCA>>?BBBDDABB055<>-?AA>AD>BA
TTCTGTGAGTGATTTCCTGCAAGACAGGAATGTCAGT
+
BCCFFDFFFIJIJJHIFGGGGIGGIJIJIGIGIGIGHHIGIIJGJJJIIJIIEHIHHHFFFB@>CCE@BEDCDDAC?CC?ACC??>ADDD
@HWI-ST0747:167:B02DEACXX:8:1304:19473:44548 1:N:0:
CTACAGTGCAGGCACCCGGCCCGCCACAATGAGTCGCTAGAGCGCAATGAGACAAGTAAAGCTGACCAAACCCTTAACCCGGACGATGCTGGG
+
BCCFHIJEHJJIIGIJIGIJIDHDGIGIGGED@CCDDC>C>BBD?BDBAABDDD@BCD@?@BDBDDDBDCCC2




__PYTHON CODE __


import glob

my_in_files = glob.glob ('E:/PINK/Paired_End/raw/gzip/*.fastq')

for each in my_in_files:
#print(each)
out = each.replace('/gzip', '/rem_clusters2' )
#print (out)
INFILE = open (each, 'r')
OUTFILE = open (out , 'w')

# Tracking Variables
Reads = 0
Writes = 0
Check_For_End_Of_File = 0

#Updates
print ("Reading File: " + each)
print ("Writing File: " + out)

# Read FASTQ File by group of four lines
while Check_For_End_Of_File == 0:

# Read the next four lines from the FASTQ file
ID_Line_1   = INFILE.readline()
Seq_Line= INFILE.readline()
ID_Line_2   = INFILE.readline()
Quality_Line= INFILE.readline()

# Strip off leading and trailing whitespace characters
ID_Line_1   = ID_Line_1.strip() 
Seq_Line= Seq_Line.strip()  
ID_Line_2   = ID_Line_2.strip() 
Quality_Line= Quality_Line.strip()

Reads = Reads + 1

#Check that I have not reached the end of file
if Quality_Line == "":
#End of file reached, print update
print ("Saw " + str(Reads) + " reads")
print ("Wrote " + str(Writes) + " reads")
Check_For_End_Of_File = 1
break

 

Re: [Tutor] Problem When Iterating Over Large Test Files

2012-07-18 Thread Steven D'Aprano
On Wed, Jul 18, 2012 at 04:33:20PM -0700, Ryan Waples wrote:
> I'm seeing some unexpected output when I use a script (included at
> end) to iterate over large text files.  I am unsure of the source of
> the unexpected output and any help would be much appreciated.

It may help if you can simplify your script to the smallest amount of 
code which demonstrates the problem. See here for more details:

http://sscce.org/

More suggestions follow below.

> In my output I am seeing lines that don't occur in the original file,
> and that don't match any lines in the original file.

How do you know? What are you doing to test that they don't match the 
original?

I'm not suggesting that you are wrong, I'm just trying to see what steps 
you have already taken.


> The incidences
> of badly formatted lines don't seem to match up with any patterns in
> the data file, and occur across multiple different data files.

Do they occur at random, or is this repeatable?

That is, if you get this mysterious output for files A, B, H and Q 
(say), do you *always* get them for A, B, H and Q?


> I've included 20 consecutive lines of input and output.  Each of these
> 5 'records' should have been selected and printed to the output file.

Earlier, you stated that each record should be four lines. But your 
sample data starts with a record of three lines.

More to follow later (time permitting).


-- 
Steven

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor


Re: [Tutor] Problem When Iterating Over Large Test Files

2012-07-18 Thread Abhishek Pratap
Hi Ryan

One quick comment

I dint get through all your code to figure out the  fine details but
my hunch is you might be having issues related to linux to dos  EOF
char.  Could you check the total  number of lines in your fastq# are
same as read by a simple python file iterator. If not then it is
mostly becoz readline is reading more than one line at a time.


-Abhi

On Wed, Jul 18, 2012 at 4:33 PM, Ryan Waples  wrote:
> I'm seeing some unexpected output when I use a script (included at
> end) to iterate over large text files.  I am unsure of the source of
> the unexpected output and any help would be much appreciated.
>
> Background
> Python v 2.7.1
> Windows 7 32bit
> Reading and writing to an external USB hard drive
>
> Data files are ~4GB text (.fastq) file, it has been uncompressed
> (gzip).  This file has no errors or formatting problems, it seems to
> have uncompressed just fine.  64M lines, each 'entry' is split across
> 4 consecutive lines, 16M entries.
>
> My python script iterates over data files 4 lines at a time, selects
> and writes groups of four lines to the output file.  I will end up
> selecting roughly 85% of the entries.
>
> In my output I am seeing lines that don't occur in the original file,
> and that don't match any lines in the original file.  The incidences
> of badly formatted lines don't seem to match up with any patterns in
> the data file, and occur across multiple different data files.
>
> I've included 20 consecutive lines of input and output.  Each of these
> 5 'records' should have been selected and printed to the output file.
> But there is a problem with the 4th and 5th entries in the output, and
> it no longer matches the input as expected.  For example the line:
> TTCTGTGAGTGATTTCCTGCAAGACAGGAATGTCAGT
> never occurs in the original data.
>
> Sorry for the large block of text below.
> Other pertinent info, I've tried a related perl script, and ran into
> similar issues, but not in the same places.
>
> Any help or insight would be appreciated.
>
> Thanks
>
>
> __EXAMPLE RAW DATA FILE REGION__
>
> @HWI-ST0747:167:B02DEACXX:8:1101:3182:167088 1:N:0:
> CGCGTGTGCAGGTTTATAGAACCAGCTGCAGATTAGTAGCAGCGCACGGAGAGGTGTGTCTGTTTATTGTCCTCAGCAGGCAGACATGTTTGTGGTC
> +
> @@@DDADDHB9+2A;6(5@CDAC(5(5:5,(8?88?BC@#
> @HWI-ST0747:167:B02DEACXX:8:1101:3134:167090 1:N:0:
> TTCTAGTGCAGGGCGACAGCGTTGCGGAGCCGGTCCGAGTCTGCTGGGTCAGTCATGGCTAGTTGGTACTATAACGACACAGGGCGAGACCCAGATGCAAA
> +
> @CCFFFDFHJJIJHHIIIJHGHIJI@GFFDDDFDDCEEEDCCBDCCCCCB>>@C(4@ADCA>>?BBBDDABB055<>-?A @HWI-ST0747:167:B02DEACXX:8:1101:3002:167092 1:N:0:
> CTTTGCTGCAGGCTCATCCTGACATGACCCTCCAGCATGACAATGCCACCAGCCATACTGCTCGTTCTGTGTGTGATTTCCAGCAAGTAAATATGTA
> +
> CCCFHIJIEHIH@AHFAGHIGIIGGEIJGIJIIIGIIIGEHGEHIIJIEHH@FHGH@=ACEHHFBFFCE@AACCA>AD>BA
> @HWI-ST0747:167:B02DEACXX:8:1101:3022:167094 1:N:0:
> ATTCCGTGCAGGCCAACTCCCGACGGACATCCTTGCTCAGACTGCAGCGATAGTGGTCGATCAGGGCCCTGTTGTTCCATCCCACTCCGGCGACCAGGTTC
> +
> CCCFHIDHJIIHIIIJIJIIGGIIFHJIIIIEIFHFF>CBAECBDDDC:??B=AAACD?8@:>C@?8CBDDD@D99B@>3884>A
> @HWI-ST0747:167:B02DEACXX:8:1101:3095:167100 1:N:0:
> CGTGATTGCAGGGACGTTACAGAGACGTTACAGGGATGTTACAGGGACGTTACAGAGACGTTAAAGAGATGTTACAGGGATGTTACAGACAGAGACGTTAC
> +
>
>
> __EXAMPLE PROBLEMATIC OUTPUT FILE REGION__
>
> @HWI-ST0747:167:B02DEACXX:8:1101:3182:167088 1:N:0:
> CGCGTGTGCAGGTTTATAGAACCAGCTGCAGATTAGTAGCAGCGCACGGAGAGGTGTGTCTGTTTATTGTCCTCAGCAGGCAGACATGTTTGTGGTC
> +
> @@@DDADDHB9+2A;6(5@CDAC(5(5:5,(8?88?BC@#
> @HWI-ST0747:167:B02DEACXX:8:1101:3134:167090 1:N:0:
> TTCTAGTGCAGGGCGACAGCGTTGCGGAGCCGGTCCGAGTCTGCTGGGTCAGTCATGGCTAGTTGGTACTATAACGACACAGGGCGAGACCCAGATGCAAA
> +
> @CCFFFDFHJJIJHHIIIJHGHIJI@GFFDDDFDDCEEEDCCBDCCCCCB>>@C(4@ADCA>>?BBBDDABB055<>-?A @HWI-ST0747:167:B02DEACXX:8:1101:3002:167092 1:N:0:
> CTTTGCTGCAGGCTCATCCTGACATGACCCTCCAGCATGACAATGCCACCAGCCATACTGCTCGTTCTGTGTGTGATTTCCAGCAAGTAAATATGTA
> +
> CCCFHIJIEHIH@AHFAGHIGIIGGEIJGIJIIIGIIIGEHGEHIIJIEHH@FHGH@=ACEHHFBFFCE@AACCA>AD>BA
> TTCTGTGAGTGATTTCCTGCAAGACAGGAATGTCAGT
> +
> BCCFFDFFFIJIJJHIFGGGGIGGIJIJIGIGIGIGHHIGIIJGJJJIIJIIEHIHHHFFFB@>CCE@BEDCDDAC?CC?ACC??>ADDD
> @HWI-ST0747:167:B02DEACXX:8:1304:19473:44548 1:N:0:
> CTACAGTGCAGGCACCCGGCCCGCCACAATGAGTCGCTAGAGCGCAATGAGACAAGTAAAGCTGACCAAACCCTTAACCCGGACGATGCTGGG
> +
> BCCFHIJEHJJIIGIJIGIJIDHDGIGIGGED@CCDDC>C>BBD?BDBAABDDD@BCD@?@BDBDDDBDCCC2
>
>
>
>
> __PYTHON CODE __
>
>
> import glob
>
> my_in_files = glob.glob ('E:/PINK/Paired_End/raw/gzip/*.fastq')
>
> for each in my_in_files:
> #print(each)
> out = each.replace('/gzip', '/rem_clusters2' )
> #print (out)
> INFILE = open (each, 'r')
> OUTFILE = open (out , 'w')
>
> # Tracking Variables
> Reads = 0
> Writes = 0
> Check_For_End_Of_File = 0
>
> #Updates
> print ("Reading File: " + each)
> print ("Writing File: " + out)
>
> # Read FASTQ File by group of four lines
> while Check

Re: [Tutor] Problem When Iterating Over Large Test Files

2012-07-18 Thread William R. Wing (Bill Wing)
On Jul 18, 2012, at 7:33 PM, Ryan Waples wrote:

> I'm seeing some unexpected output when I use a script (included at
> end) to iterate over large text files.  I am unsure of the source of
> the unexpected output and any help would be much appreciated.
> 
> Background
> Python v 2.7.1
> Windows 7 32bit
> Reading and writing to an external USB hard drive
> 
> Data files are ~4GB text (.fastq) file, it has been uncompressed
> (gzip).  This file has no errors or formatting problems, it seems to
> have uncompressed just fine.  64M lines, each 'entry' is split across
> 4 consecutive lines, 16M entries.
> 
> My python script iterates over data files 4 lines at a time, selects
> and writes groups of four lines to the output file.  I will end up
> selecting roughly 85% of the entries.
> 
> In my output I am seeing lines that don't occur in the original file,
> and that don't match any lines in the original file.  The incidences
> of badly formatted lines don't seem to match up with any patterns in
> the data file, and occur across multiple different data files.
> 
> I've included 20 consecutive lines of input and output.  Each of these
> 5 'records' should have been selected and printed to the output file.
> But there is a problem with the 4th and 5th entries in the output, and
> it no longer matches the input as expected.  For example the line:
> TTCTGTGAGTGATTTCCTGCAAGACAGGAATGTCAGT
> never occurs in the original data.
> 
> Sorry for the large block of text below.
> Other pertinent info, I've tried a related perl script, and ran into
> similar issues, but not in the same places.
> 
> Any help or insight would be appreciated.
> 
> Thanks

[Data and program snipped]

With apologies - I'm a Mac/UNIX user, not Windows, but those numbers (4GB and 
64M lines) look suspiciously close to the file and record pointer limits to a 
32-bit file system.  Are you sure you aren't bumping into wrap around issues of 
some sort?

Just a thought…

-Bill
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor


Re: [Tutor] writing function changeColor

2012-07-18 Thread Lee Harr

> I learned python so that if I were to put in 0.9, it'd decrease red by 10%.> 
> The way this function needs to be written, -0.1 decreases red by 10%
It sounds like you have something like ...
def f1(color, redchange=0, bluechange=0, greenchange=0):    # some code here to 
make the changes    return modified_color
calling it like ...
>>> c = Color(red=200, blue=100, green=50)>>> c(200, 100, 50)>>> f1(c, 
>>> redchange=0.9)(180, 100, 50)
but what you want is ...
>>> f1a(c, redchange=-0.1)(180, 100, 50)

Maybe, since you already have a function that works, youcould write f1a as a 
function that calls f1 as a helper, usingmodified parameters.
def f1a(color, redchange=0, bluechange=0, greenchange=0):    # some code to 
modify the parameters to fit your function f1    return f1(color, modredchange, 
modbluechange, modgreenchange)

So... we just need some code to convert 0.9 to -0.1 (and weneed to think about 
other possible values that could be sentto f1 and what f1a would want in the 
same situation.)

There's a very simple way to convert 0.9 to -0.1 but I'm notsure that's what 
you want.
Would you also want:
>>> f1(c, redchange=0.5)(100, 100, 50)>>> f1a(c, redchange=-0.5)(100, 100, 50)
How about ...
>>> f1(c, redchange=0)(0, 100, 50)>>> f1a(c, redchange=-1.0)(0, 100, 50)
Or ...
>>> f1(c, redchange=1.2)(240, 100, 50)>>> f1a(c, redchange=0.2)(240, 100, 50)
?
  
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor


Re: [Tutor] Problem When Iterating Over Large Test Files

2012-07-18 Thread Ryan Waples
Thanks for the replies, I'll try to address the questions raised and
spur further conversation.

>"those numbers (4GB and 64M lines) look suspiciously close to the file and 
>record pointer limits to a 32-bit file system.  Are you sure you aren't 
>bumping into wrap around issues of some sort?"

My understanding is that I am taking the files in a stream, one line
at a time and never loading them into memory all at once.  I would
like (and expect) my script to be able to handle files up to at least
50GB.  If this would cause a problem, let me know.

> "my hunch is you might be having issues related to linux to dos  EOF char."

I don't think this is the issue.  99.99% of the lines come out ok,
(see examples).  I do end up with an output file with some 50 some mil
lines.  I can confirm that my python code as written and executed on
Win7 will convert the original file endings from Unix (LF) to windows
(CRLF).  This shouldn't confuse the downstream analysis.

> "What are you doing to test that they don't match the original?"

Those 2 pieces of example data have been grep'd (cygwin) out of the IN
and OUT files, they represent the output of a grep that pull the 20
lines surrounding the line:
@HWI-ST0747:167:B02DEACXX:8:1101:3002:167092 1:N:0:
which is a unique line in each.

I have also grep'd the IN file for a line in the OUT:
  grep ^TTCTGTGAGTGATTTCCTGCAAGACAGGAATGTCAGT$
with no results

The python code posted has a (weak) check, that mostly serves to
confirm that every fourth line of the IN file starts with an "@", this
IS the case for the IN file, but is NOT the case for the OUT file.

I can run my analysis program program on the raw IN file fine, it will
process all entries.  When the OUT file is supplied, it will error at
reads in the pasted text.

> "Earlier, you stated that each record should be four lines. But your sample 
> data starts with a record of three lines."

I've checked again and they look to be four lines, so I'm not sure I understand.
Format:
1) ID line (must start with @) - contains filter criteria
2) Data line 1 - 101 chars
3) just a "+"
4) Data line 2 - 101 chars (may start with @)

> "Do they occur at random, or is this repeatable?"

When I'm back at work I'll confirm again that this is the case, I
should have a better answer here.  I can confirm that it seems to
happen to every (large) file I've tested, no files seem unaffected.

Thanks

__SHORTENED PYTHON CODE__

for each in my_in_files:
out = each.replace('/gzip', '/rem_clusters2' )
INFILE = open (each, 'r')
OUTFILE = open (out , 'w')

# Tracking Variables
Reads = 0
Writes = 0
Check_For_End_Of_File = 0


# Read FASTQ File by group of four lines
while Check_For_End_Of_File == 0:

ID_Line_1   = INFILE.readline()
Seq_Line= INFILE.readline()
ID_Line_2   = INFILE.readline()
Quality_Line= INFILE.readline()

ID_Line_1   = ID_Line_1.strip()
Seq_Line= Seq_Line.strip()
ID_Line_2   = ID_Line_2.strip()
Quality_Line= Quality_Line.strip()

Reads = Reads + 1

#Check that I have not reached the end of file
if Quality_Line == "":
Check_For_End_Of_File = 1
break

#Check that ID_Line_1 starts with @
if not ID_Line_1.startswith('@'):
break

# Select Reads that I want to keep
ID = ID_Line_1.partition(' ')
if (ID[2] == "1:N:0:" or ID[2] == "2:N:0:"):

# Write to file, maintaining 
group of 4
OUTFILE.write(ID_Line_1 + "\n")
OUTFILE.write(Seq_Line + "\n")
OUTFILE.write(ID_Line_2 + "\n")
OUTFILE.write(Quality_Line + "\n")
Writes = Writes +1

INFILE.close()
OUTFILE.close()
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor


Re: [Tutor] Problem When Iterating Over Large Test Files

2012-07-18 Thread William R. Wing (Bill Wing)
On Jul 18, 2012, at 10:33 PM, Ryan Waples wrote:

> Thanks for the replies, I'll try to address the questions raised and
> spur further conversation.
> 
>> "those numbers (4GB and 64M lines) look suspiciously close to the file and 
>> record pointer limits to a 32-bit file system.  Are you sure you aren't 
>> bumping into wrap around issues of some sort?"
> 
> My understanding is that I am taking the files in a stream, one line
> at a time and never loading them into memory all at once.  I would
> like (and expect) my script to be able to handle files up to at least
> 50GB.  If this would cause a problem, let me know.

[Again, stripping out everything else…]

I don't think you understood my concern.  The issue isn't whether or not the 
files are being read as a stream, the issue is that at something like those 
numbers a 32-bit file system can silently fail.  If the pointers that are 
chaining allocation blocks together (or whatever Windows calls them) aren't 
capable of indexing to sufficiently large numbers, then you WILL get garbage 
included in the file stream.

If you copy those files to a different device (one that has just been scrubbed 
and reformatted), then copy them back and get different results with your 
application, you've found your problem.

-Bill
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor


Re: [Tutor] Problem When Iterating Over Large Test Files

2012-07-18 Thread Lee Harr

>   grep ^TTCTGTGAGTGATTTCCTGCAAGACAGGAATGTCAGT$> with no results

How about:
grep TTCTGTGAGTGATTTCCTGCAAGACAGGAATGTCAGT outfile
Just in case there is some non-printing character in there...

Beyond that ... my guess would be that you are either not readingthe file you 
think you are, or not writing the file you think you are  :o)
out = each.replace('/gzip', '/rem_clusters2')
Seems pretty bulletproof, but maybe just print each and out hereto make sure...

Also, I'm curious... Reading your code, I sort of feel like when I 
amlistening to a non-native speaker. I always get the urge to throw out 
thecorrect "Americanisms" for people -- to help them fit in better. So, I hope 
itdoes not make me a jerk, but ...
infile = open(each, 'r') # I'd probably drop the 'r' also...
while not check_for_end_of_file:
reads += 1
head, sep, tail = id_line_1.partition(' ') # or, if I'm only using the one 
thing ..._, _, meaningful_name = id_line_1.partition(' ') # maybe call it 
"selector", then ...
if selector in ('1:N:0:', '2:N:0:'):

Hope this helps.
  
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor


Re: [Tutor] Problem When Iterating Over Large Test Files

2012-07-18 Thread Ryan Waples
On Wed, Jul 18, 2012 at 8:04 PM, William R. Wing (Bill Wing)
 wrote:
> On Jul 18, 2012, at 10:33 PM, Ryan Waples wrote:
>
>> Thanks for the replies, I'll try to address the questions raised and
>> spur further conversation.
>>
>>> "those numbers (4GB and 64M lines) look suspiciously close to the file and 
>>> record pointer limits to a 32-bit file system.  Are you sure you aren't 
>>> bumping into wrap around issues of some sort?"
>>
>> My understanding is that I am taking the files in a stream, one line
>> at a time and never loading them into memory all at once.  I would
>> like (and expect) my script to be able to handle files up to at least
>> 50GB.  If this would cause a problem, let me know.
>
> [Again, stripping out everything else…]
>
> I don't think you understood my concern.  The issue isn't whether or not the 
> files are being read as a stream, the issue is that at something like those 
> numbers a 32-bit file system can silently fail.  If the pointers that are 
> chaining allocation blocks together (or whatever Windows calls them) aren't 
> capable of indexing to sufficiently large numbers, then you WILL get garbage 
> included in the file stream.
>
> If you copy those files to a different device (one that has just been 
> scrubbed and reformatted), then copy them back and get different results with 
> your application, you've found your problem.
>
> -Bill

Thanks for the insistence,  I'll check this out.  If you have any
guidance on how to do so let me know.  I knew my system wasn't
particularly well suited to the task at hand, but I haven't seen how
it would actually cause problems.

-Ryan
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor


Re: [Tutor] Problem When Iterating Over Large Test Files

2012-07-18 Thread Ryan Waples
On Wed, Jul 18, 2012 at 8:23 PM, Lee Harr  wrote:
>
>>   grep ^TTCTGTGAGTGATTTCCTGCAAGACAGGAATGTCAGT$> with no results
>
> How about:
> grep TTCTGTGAGTGATTTCCTGCAAGACAGGAATGTCAGT outfile
> Just in case there is some non-printing character in there...

There are many instances of that sequence of characters in the RAW
input file, but that is what I would expect.




>
> Beyond that ... my guess would be that you are either not readingthe file you 
> think you are, or not writing the file you think you are  :o)
> out = each.replace('/gzip', '/rem_clusters2')
> Seems pretty bulletproof, but maybe just print each and out hereto make 
> sure...

Checked this multiple times

>
> Also, I'm curious... Reading your code, I sort of feel like when I 
> amlistening to a non-native speaker. I always get the urge to throw out 
> thecorrect "Americanisms" for people -- to help them fit in better. So, I 
> hope itdoes not make me a jerk, but ...
> infile = open(each, 'r') # I'd probably drop the 'r' also...

working in science, I try to be as explicit as possible, I've come to
dislike Perl for this reason.

> while not check_for_end_of_file:
> reads += 1
> head, sep, tail = id_line_1.partition(' ') # or, if I'm only using the one 
> thing ..._, _, meaningful_name = id_line_1.partition(' ') # maybe call it 
> "selector", then ...
> if selector in ('1:N:0:', '2:N:0:'):
>

Points taken, thanks.


> Hope this helps.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor


Re: [Tutor] string to binary and back... Python 3

2012-07-18 Thread wolfrage8...@gmail.com
On Thu, Jul 19, 2012 at 12:16 AM, Dave Angel  wrote:

>  On 07/18/2012 05:07 PM, Jordan wrote:
> > OK so I have been trying for a couple days now and I am throwing in the
> > towel, Python 3 wins this one.
> > I want to convert a string to binary and back again like in this
> > question: Stack Overflow: Convert Binary to ASCII and vice versa
> > (Python)
> > <
> http://stackoverflow.com/questions/7396849/convert-binary-to-ascii-and-vice-versa-python
> >
> > But in Python 3 I consistently get  some sort of error relating to the
> > fact that nothing but bytes and bytearrays support the buffer interface
> > or I get an overflow error because something is too large to be
> > converted to bytes.
> > Please help me and then explian what I am not getting that is new in
> > Python 3. I would like to point out I realize that binary, hex, and
> > encodings are all a very complex subject and so I do not expect to
> > master it but I do hope that I can gain a deeper insight. Thank you all.
> >
> > test_script.py:
> > import binascii
> >
> > test_int = 109
> >
> > test_int = int(str(test_int) + '45670')
> > data = 'Testing XOR Again!'
> >
> > while sys.getsizeof(data) > test_int.bit_length():
> >
> > test_int = int(str(test_int) + str(int.from_bytes(os.urandom(1), 'big')))
> >
> > print('Bit Length: ' + str(test_int.bit_length()))
> >
> > key = test_int # Yes I know this is an unnecessary step...
> >
> > data = bin(int(binascii.hexlify(bytes(data, 'UTF-8')), 16))
> >
> > print(data)
> >
> > data = int(data, 2)
> >
> > print(data)
> >
> > data = binascii.unhexlify('%x' % data)
> >
>
> I don't get the same error you did.  I get:
>
>  File "jordan.py", line 13
> test_int = int(str(test_int) + str(int.from_bytes(os.urandom(1),
> 'big')))
>^
>
test_int = int(str(test_int) + str(int.from_bytes(os.urandom(1), \
'big')))
# That was probably just do to the copy and paste.

> IndentationError: expected an indented block
>
>
> Please post it again, with correct indentation.  if you used tabs, then
> expand them to spaces before pasting it into your test-mode mail editor.
>
> I only use spaces and this program did not require any indentation until
it was pasted and the one line above became split across two line. Really
though that was a trivial error to correct.

>
> I'd also recommend you remove a lot of the irrelevant details there.  if
> you have a problem with hexlfy and/or unhexlify, then give a simple byte
> string that doesn't work for you, and somebody can probably identify why
> not.  And if you want people to run your code, include the imports as well.
>
> My problem is not specific to hexlify and unhexlify, my problem is trying
to convert from string to binary and back. That is why all of the details,
to show I have tried on my own.
Sorry that I forgot to include sys and os for imports.


> As it is, you're apparently looping, comparing the byte memory size of a
> string (which is typically 4 bytes per character) with the number of
> significant bits in an unrelated number.
>
> I suspect what you want is something resembling (untested):
>
> mybytes = bytes( "%x" % data, "ascii")
> newdata = binascii.unexlify(mybytes)
>
> I was comparing them but I think I understand how to compare them well,
now I want to convert them both to binary so that I can XOR them together.
Thank you for your time and help Dave, now I need to reply to Ramit.

>
> --
> DaveA
>
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor


Re: [Tutor] Problem When Iterating Over Large Test Files

2012-07-18 Thread Steven D'Aprano
On Wed, Jul 18, 2012 at 04:33:20PM -0700, Ryan Waples wrote:

> I've included 20 consecutive lines of input and output.  Each of these
> 5 'records' should have been selected and printed to the output file.

I count only 19 lines. The first group has only three lines. See below.

There is a blank line, which I take as NOT part of the input but just a 
spacer. Then:

1) Line starting with @
2) Line of bases CGCGT ...
3) Plus sign
4) Line starting with @@@
5) Line starting with @
6) Line of bases TTCTA ...
7) Plus sign

and so on. There are TWO lines before the first +, and three before each 
of the others.



> __EXAMPLE RAW DATA FILE REGION__
> 
> @HWI-ST0747:167:B02DEACXX:8:1101:3182:167088 1:N:0:
> CGCGTGTGCAGGTTTATAGAACCAGCTGCAGATTAGTAGCAGCGCACGGAGAGGTGTGTCTGTTTATTGTCCTCAGCAGGCAGACATGTTTGTGGTC
> +
> @@@DDADDHB9+2A;6(5@CDAC(5(5:5,(8?88?BC@#
> @HWI-ST0747:167:B02DEACXX:8:1101:3134:167090 1:N:0:
> TTCTAGTGCAGGGCGACAGCGTTGCGGAGCCGGTCCGAGTCTGCTGGGTCAGTCATGGCTAGTTGGTACTATAACGACACAGGGCGAGACCCAGATGCAAA
> +
> @CCFFFDFHJJIJHHIIIJHGHIJI@GFFDDDFDDCEEEDCCBDCCCCCB>>@C(4@ADCA>>?BBBDDABB055<>-?A @HWI-ST0747:167:B02DEACXX:8:1101:3002:167092 1:N:0:
> CTTTGCTGCAGGCTCATCCTGACATGACCCTCCAGCATGACAATGCCACCAGCCATACTGCTCGTTCTGTGTGTGATTTCCAGCAAGTAAATATGTA
> +
> CCCFHIJIEHIH@AHFAGHIGIIGGEIJGIJIIIGIIIGEHGEHIIJIEHH@FHGH@=ACEHHFBFFCE@AACCA>AD>BA
> @HWI-ST0747:167:B02DEACXX:8:1101:3022:167094 1:N:0:
> ATTCCGTGCAGGCCAACTCCCGACGGACATCCTTGCTCAGACTGCAGCGATAGTGGTCGATCAGGGCCCTGTTGTTCCATCCCACTCCGGCGACCAGGTTC
> +
> CCCFHIDHJIIHIIIJIJIIGGIIFHJIIIIEIFHFF>CBAECBDDDC:??B=AAACD?8@:>C@?8CBDDD@D99B@>3884>A
> @HWI-ST0747:167:B02DEACXX:8:1101:3095:167100 1:N:0:
> CGTGATTGCAGGGACGTTACAGAGACGTTACAGGGATGTTACAGGGACGTTACAGAGACGTTAAAGAGATGTTACAGGGATGTTACAGACAGAGACGTTAC
> +

Your code says that the first line in each group should start with an @ 
sign. That is clearly not the case for the last two groups.

I suggest that your data files have been corrupted.

> __PYTHON CODE __

I have re-written your code slightly, to be a little closer to "best 
practice", or at least modern practice. If there is anything you don't 
understand, please feel free to ask.

I haven't tested this code, but it should run fine on Python 2.7.

It will be interesting to see if you get different results with this.



import glob

def four_lines(file_object):
"""Yield lines from file_object grouped into batches of four.

If the file has fewer than four lines remaining, pad the batch 
with 1-3 empty strings.

Lines are stripped of leading and trailing whitespace.
"""
while True:
# Get the first line. If there is no first line, we are at EOF
# and we raise StopIteration to indicate we are done.
line1 = next(file_object).strip()
# Get the next three lines, padding if needed.
line2 = next(file_object, '').strip()
line3 = next(file_object, '').strip()
line4 = next(file_object, '').strip()
yield (line1, line2, line3, line4)


my_in_files = glob.glob ('E:/PINK/Paired_End/raw/gzip/*.fastq')
for each in my_in_files:
out = each.replace('/gzip', '/rem_clusters2' )
print ("Reading File: " + each)
print ("Writing File: " + out)
INFILE = open (each, 'r')
OUTFILE = open (out , 'w')
writes = 0

for reads, lines in four_lines( INFILE ):
ID_Line_1, Seq_Line, ID_Line_2, Quality_Line = lines
# Check that ID_Line_1 starts with @
if not ID_Line_1.startswith('@'):
print ("**ERROR**")
print ("expected ID_Line to start with @")
print (lines)
print ("Read Number " + str(Reads))
break
elif Quality_Line != '+':
print ("**ERROR**")
print ("expected Quality_Line = +")
print (lines)
print ("Read Number " + str(Reads))
break
# Select Reads that I want to keep  
ID = ID_Line_1.partition(' ')
if (ID[2] == "1:N:0:" or ID[2] == "2:N:0:"):
# Write to file, maintaining group of 4
OUTFILE.write(ID_Line_1 + "\n")
OUTFILE.write(Seq_Line + "\n")
OUTFILE.write(ID_Line_2 + "\n")
OUTFILE.write(Quality_Line + "\n")
writes += 1
# End of file reached, print update
print ("Saw", reads, "groups of four lines")
print ("Wrote", writes, "groups of four lines")
INFILE.close()
OUTFILE.close()





-- 
Steven
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.pytho

Re: [Tutor] string to binary and back... Python 3

2012-07-18 Thread Mark Lawrence

On 19/07/2012 06:41, wolfrage8...@gmail.com wrote:

On Thu, Jul 19, 2012 at 12:16 AM, Dave Angel  wrote:


  On 07/18/2012 05:07 PM, Jordan wrote:

OK so I have been trying for a couple days now and I am throwing in the
towel, Python 3 wins this one.
I want to convert a string to binary and back again like in this
question: Stack Overflow: Convert Binary to ASCII and vice versa
(Python)
<

http://stackoverflow.com/questions/7396849/convert-binary-to-ascii-and-vice-versa-python


But in Python 3 I consistently get  some sort of error relating to the
fact that nothing but bytes and bytearrays support the buffer interface
or I get an overflow error because something is too large to be
converted to bytes.
Please help me and then explian what I am not getting that is new in
Python 3. I would like to point out I realize that binary, hex, and
encodings are all a very complex subject and so I do not expect to
master it but I do hope that I can gain a deeper insight. Thank you all.

test_script.py:
import binascii

test_int = 109

test_int = int(str(test_int) + '45670')
data = 'Testing XOR Again!'

while sys.getsizeof(data) > test_int.bit_length():

test_int = int(str(test_int) + str(int.from_bytes(os.urandom(1), 'big')))

print('Bit Length: ' + str(test_int.bit_length()))

key = test_int # Yes I know this is an unnecessary step...

data = bin(int(binascii.hexlify(bytes(data, 'UTF-8')), 16))

print(data)

data = int(data, 2)

print(data)

data = binascii.unhexlify('%x' % data)



I don't get the same error you did.  I get:

  File "jordan.py", line 13
 test_int = int(str(test_int) + str(int.from_bytes(os.urandom(1),
'big')))
^


test_int = int(str(test_int) + str(int.from_bytes(os.urandom(1), \
 'big')))
# That was probably just do to the copy and paste.


IndentationError: expected an indented block


Please post it again, with correct indentation.  if you used tabs, then
expand them to spaces before pasting it into your test-mode mail editor.

I only use spaces and this program did not require any indentation until

it was pasted and the one line above became split across two line. Really
though that was a trivial error to correct.


Really?  Are you using a forked version of Python that doesn't need 
indentation after a while loop, or are you speaking with a forked 
tongue? :)  Strangely I believe the latter, so please take note of what 
Dave Angel has told you and post with the correct indentation.






I'd also recommend you remove a lot of the irrelevant details there.  if
you have a problem with hexlfy and/or unhexlify, then give a simple byte
string that doesn't work for you, and somebody can probably identify why
not.  And if you want people to run your code, include the imports as well.

My problem is not specific to hexlify and unhexlify, my problem is trying

to convert from string to binary and back. That is why all of the details,
to show I have tried on my own.
Sorry that I forgot to include sys and os for imports.



As it is, you're apparently looping, comparing the byte memory size of a
string (which is typically 4 bytes per character) with the number of
significant bits in an unrelated number.

I suspect what you want is something resembling (untested):

 mybytes = bytes( "%x" % data, "ascii")
 newdata = binascii.unexlify(mybytes)

I was comparing them but I think I understand how to compare them well,

now I want to convert them both to binary so that I can XOR them together.
Thank you for your time and help Dave, now I need to reply to Ramit.



--
DaveA





___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor




--
Cheers.

Mark Lawrence.



___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor


Re: [Tutor] Problem When Iterating Over Large Test Files

2012-07-18 Thread Ryan Waples
>>>
>>> If you copy those files to a different device (one that has just been 
>>> scrubbed and reformatted), then copy them back and get different results 
>>> with your application, you've found your problem.
>>>
>>> -Bill
>>
>> Thanks for the insistence,  I'll check this out.  If you have any
>> guidance on how to do so let me know.  I knew my system wasn't
>> particularly well suited to the task at hand, but I haven't seen how
>> it would actually cause problems.
>>
>> -Ryan
>> ___
>> The last two lines in my MSG pretty much would be the test. Get another 
>> flash drive, format it as FAT-32 (I assume that's what you are using), then 
>> copy a couple of files to it.  Then copy them back to your current device 
>> and run your program again. If you get DIFFERENT, but still wrong results, 
>> you've found the problem. The largest positive integer a 32-bit binary 
>> number can represent is 2^32, which is 4Gig.  I'm no expert on Window's 
>> files, but I'd be very surprised if when the FAT-32 file system was being 
>> designed, anyone considered the case where a single file could be that large.
>
> -Bill


The hard-drive is formatted as NTFS, because as you say I'm up against
the file size limit of FAT32 , do think this could still be the issue?
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor