Efficient bits manipulation in Python
Hi: I have a bit-code :'1011011', how can I reverse it to '1101101'? Another question is I know how to transform the string '110' into integer 6, does anyone know how to transform integer 6 to a string '110'? Thank you very much:) -- Li -- Time is all we have and you may find one day you have less than you think -- http://mail.python.org/mailman/listinfo/python-list
Re: Efficient bits manipulation in Python
>> I have a bit-code :'1011011', how can I reverse it to '1101101'? >> >> Another question is I know how to transform the string '110' into >> integer 6, does anyone know how to transform integer 6 to a string >> '110'? >> >> Thank you very much:) > > Assuming that you are using 2.6: > > a = 0b1011011 > print bin(a)[:1:-1] > > a = 6 > print bin(a)[2:] Thank you very much, that works:). > > - Max > -- Li -- Time is all we have and you may find one day you have less than you think -- http://mail.python.org/mailman/listinfo/python-list
How to locate the bit in bits string?
Hi: If I use an integer to represent bits: e.g. 99 represents '1100011' How can I locate, say the second bit of 99(i.e. '1')? Although bin(99)[4] could be used to locate it, this transform cost too much memory (99 only needs 2Bytes, while string '1100011' needs 7Bytes). Anyone knows how to locate the second bit without using bin() function? Thank you very much:D -- Li -- Time is all we have and you may find one day you have less than you think -- http://mail.python.org/mailman/listinfo/python-list
Re: How to locate the bit in bits string?
2009/4/29 Tim Chase : > Li Wang wrote: >> >> Hi: >> >> If I use an integer to represent bits: >> e.g. 99 represents '1100011' >> >> How can I locate, say the second bit of 99(i.e. '1')? >> >> Although bin(99)[4] could be used to locate it, this transform cost >> too much memory (99 only needs 2Bytes, while string '1100011' needs >> 7Bytes). >> >> Anyone knows how to locate the second bit without using bin() function? > > You mean > > def get_bit(number, bit): >return (number >> bit) & 1 > > ? > Hummm, I have tried this method too, the problem is its time complexity. If the length of my bits is n, then the time complexity is O(n). When I tried to implement this in practice, it did consume a lot of time. So do you know how could I locate the bit in O(1) time? Transform it into a string is a method, but takes too much space (when I try to process a 2M file, it used more than 100M memory.). Thank you very much. > -tkc > > > > > -- Li -- Time is all we have and you may find one day you have less than you think -- http://mail.python.org/mailman/listinfo/python-list
Re: How to locate the bit in bits string?
2009/4/29 Tim Chase :
>> I want to concatenate two bits string together: say we have '1001' and
>> '111' which are represented in integer. I want to concatenate them to
>> '100' (also in integer form), my method is:
>> ('1001' << 3) | 111
>> which is very time consuming.
>
> You omit some key details -- namely how do you know that "1001" is 4 bits
> and not "1001" (8-bits)? If it's a string (as your current code shows),
> you can determine the length. However, if they are actually ints, your code
> should work fine & be O(1).
Actually, what I have is a list of integer numbers [3,55,99,44], and
by using Huffman coding or fixed length coding, I will know how the
bits-length for each number. When I try to concatenate them (say
10,000 items in the list) all together, the speed is going down
quickly (because of the shifting operations of python long).
>
> This can be abstracted if you need:
>
> def combine_bits(int_a, int_b, bit_len_b):
>return (int_a << bit_len_b) | int_b
>
> a = 0x09
> b = 0x07
> print combine_bits(a, b, 3)
>
> However, if you're using gargantuan ints (as discussed before), it's a lot
> messier. You'd have to clarify the storage structure (a byte string? a
> python long?)
I am using a single python long to store all the items in the list
(say, 10,000 items), so the work does become messier...
>
> -tkc
>
> PS: You may want to CC the mailing list so that others get a crack at
> answering your questions...I've been adding it back in, but you've been
> replying just to me.
Sorry, this is the first time I am using mail-listand always
forgot "reply to all"
Thank you very much:D
>
>
>
>
--
Li
--
Time is all we have
and you may find one day
you have less than you think
--
http://mail.python.org/mailman/listinfo/python-list
Re: How to locate the bit in bits string?
2009/4/29 Tim Chase :
>>> You omit some key details -- namely how do you know that
>>> "1001" is 4 bits and not "1001" (8-bits)? If it's a
>>> string (as your current code shows), you can determine the
>>> length. However, if they are actually ints, your code should work fine &
>>> be O(1).
>>
>> Actually, what I have is a list of integer numbers
>> [3,55,99,44], and by using Huffman coding or fixed length
>> coding, I will know how the bits-length for each number. When
>> I try to concatenate them (say 10,000 items in the list) all
>> together, the speed is going do
>
> Ah, this makes more sense!
>
> I'd try creating a bitwriter object that takes fragments of bytes and writes
> them sequentially to a file-like object. Something like this untested
> class:
>
> class BitWriter:
>def __init__(self, outstream):
> self.out = outstream
> self.datum = 0 # the data accrued
> self.bits = 0 # the number of bits we've accrued
>def write(self, byte_data, bit_count):
> # maybe add some code to ensure
> # that byte_data doesn't have any
> # significant bits beyond bit_count
> datum = (self.datum << bit_count) | byte_data
> self.bits += bit_count
> if self.bits >= 8:
>overflow = self.bits - 8
>self.out.write(chr(datum >> overflow))
>#discard the stuff we wrote
>datum -= (datum >> overflow) << overflow
>self.bits -= 8
>def close(self):
> if self.bits: # we have unflushed data
>data = self.datum << (8 - self.bits)
>self.out.write(chr(data))
> self.out.close()
>
> out = file('compressed.dat', 'wb')
> bw = BitWriter(out)
> for data, bit_count in source():
>out.write(data, bit_count)
> out.close()
>
> As mentioned, it's 100% untested, I don't know what sort of performance it
> has, nor the endian'ness of your data streams, so you might have to twiddle
> bits in the opposite directions and test the performance.
The method looks great, I thought I got your idea: when the
len(bits)>8, we output 8bits and discard them. I believe this will
work efficiently, because I have implemented the algorithm by
manipulating strings ('111001' character string, not bits) in the
similar way, and worked quite fast.:), Thank you very much.
You mentioned "twiddle bits in the opposite directions", that reminds
me of another possible solutions for this problem. Say, I have bits
'1101' and '110', instead of doing (1101<<3) |110, we could try the
opposite way:
firstly, inverse the '1101' to '1011', and '110' to '011'
then, (011 << 4) | 1011,
if there comes another bit '10100'
inverse it again 10100 to 00101
add it to the long bits: (00101 << 7) | 0111011
so on and so forth.
After all the process is done, we inverse the final long bits
Because in this process you do not need to shift the whole long bits,
but only shift the much shorter bits every time, so the time
efficiency should be not bad.
However, there is one issue I cannot not figure it out: How to inverse
the bits efficiently? do you have any idea?
Thank you very much:!:)
Best regards,
Li
>
> -tkc
>
>
>
>
>
>
>
>
--
Li
--
Time is all we have
and you may find one day
you have less than you think
--
http://mail.python.org/mailman/listinfo/python-list
Measure the memory cost in Python
Hi everyone: I want to measure the actual memory cost of a particular step in my program (Python program), does anyone know if there is some function in Python could help me to do this job? Or should I seek other tools to help me? Thank you very much! -- Li -- Time is all we have and you may find one day you have less than you think -- http://mail.python.org/mailman/listinfo/python-list
How to measure the memory cost in Python?
Hi everyone: I want to measure the actual memory cost of a particular step in my program (Python program), does anyone know if there is some function in Python could help me to do this job? Or should I seek other tools to help me? Thank you very much! -- Li -- Time is all we have and you may find one day you have less than you think -- http://mail.python.org/mailman/listinfo/python-list
Python 2.6, File read() problems in Windows Xp
Hi all: I am trying to read a non-text file as a string by using Python read(), however, it seems there is some thing wrong with it. I can use read() on text file correctly, but unable to read .xls file correctly. (The program can read any file correctly in Fedora 10) Any idea how to solve this problem? Thank you very much! -- Li -- Time is all we have and you may find one day you have less than you think -- http://mail.python.org/mailman/listinfo/python-list
Re: Python 2.6, File read() problems in Windows Xp
2009/5/8 Chris Rebert :
> On Thu, May 7, 2009 at 10:04 PM, Li Wang wrote:
>> Hi all:
>>
> the file, e.g. open("the_file.xls", "rb")
> Unlike *nix, Windows differentiates between binary and text files,
> hence the need for the "b" flag to specify which you're dealing with.
Hi
Thank you very much for reply,
The method doesn't work.
Here is the problem: after reading the whole fie as a string, I need
the string to be made of 8bits-symbols.
I do not mind what's the content in the file, what I need to do is
something like reading the file byte by byte and concatenate these
bytes into a single string.
Any suggestions?
Thank you very much
--
Li
--
Time is all we have
and you may find one day
you have less than you think
--
http://mail.python.org/mailman/listinfo/python-list
Re: Python 2.6, File read() problems in Windows Xp
Hi:
Problem solved
Thank you very much, it works, It is my own problem:)
All the best,
Li
2009/5/8 Li Wang :
> 2009/5/8 Chris Rebert :
>> On Thu, May 7, 2009 at 10:04 PM, Li Wang wrote:
>>> Hi all:
>>>
>
>> the file, e.g. open("the_file.xls", "rb")
>> Unlike *nix, Windows differentiates between binary and text files,
>> hence the need for the "b" flag to specify which you're dealing with.
>
> Hi
> Thank you very much for reply,
>
> The method doesn't work.
> Here is the problem: after reading the whole fie as a string, I need
> the string to be made of 8bits-symbols.
>
> I do not mind what's the content in the file, what I need to do is
> something like reading the file byte by byte and concatenate these
> bytes into a single string.
>
> Any suggestions?
>
> Thank you very much
>
>
>
>
>
> --
> Li
> --
> Time is all we have
> and you may find one day
> you have less than you think
>
--
Li
--
Time is all we have
and you may find one day
you have less than you think
--
http://mail.python.org/mailman/listinfo/python-list
Re: Python 2.6, File read() problems in Windows Xp
Hi Dave:
Thank you very much for you explanation:)
> Chances are you forgot the "b" parameter to open(). Unnecessary in Unix, it
> tells the library to *not* translate \r\n to \n upon read, or the inverse
> on write. In other words, with the "b" parameter, the file is read in
> unchanged.
So, if I am using python in Linux, do open('file', 'r') and
open('file', 'rb') work the same way?
Thanks,
Best regards,
Li
--
http://mail.python.org/mailman/listinfo/python-list
Re: Python 2.6, File read() problems in Windows Xp
2009/5/8 Scott David Daniels :
> Li Wang wrote:
>>
>> So, if I am using python in Linux, do open('file', 'r') and
>> open('file', 'rb') work the same way?
>
> You get identical results, but you ar lying to the reader of your code.
> you should include the 'b' if what you want is bytes (or octets if you
> prefer), and not use it if what you expect is "text." Your code
> becomes less confusing by using 'b' properly, even if you see no
> particular difference.
>
Thanks a lot, very helpful:D
>
--
Li
--
Time is all we have
and you may find one day
you have less than you think
--
http://mail.python.org/mailman/listinfo/python-list
Re: Python 2.6, File read() problems in Windows Xp
2009/5/8 :
> On May 8, 5:08 am, Li Wang wrote:
>> Hi Dave:
>> Thank you very much for you explanation:)
>>
>> > Chances are you forgot the "b" parameter to open(). Unnecessary in Unix,
>> > it
>> > tells the library to *not* translate \r\n to \n upon read, or the inverse
>> > on write. In other words, with the "b" parameter, the file is read in
>> > unchanged.
>>
>> So, if I am using python in Linux, do open('file', 'r') and
>> open('file', 'rb') work the same way?
>>
>> Thanks,
>>
>> Best regards,
>> Li
>
> In old Python up to 2.6, YES.
Thank you:D
--
Li
--
Time is all we have
and you may find one day
you have less than you think
--
http://mail.python.org/mailman/listinfo/python-list
Re: Python 2.6, File read() problems in Windows Xp
> (my smtp mail server has been down for a few hours, so you have some other > responses.) > > yes, as far as I can tell from the docs, the 'b' flag doesn't matter in > Unix/Linux. Note that I haven't actually been on a Unix machine since > starting with Python, but it matches what else I know. And if it's > different in Python 3.0, I wouldn't know that either., Thank you very much! -- Li -- Time is all we have and you may find one day you have less than you think -- http://mail.python.org/mailman/listinfo/python-list
mod_python load cx_Oracle error
It's quite weird when I import cx_Oracle in python interactive shell, it works perfectly. but when I import cx_Oracle in a *,py script, handled by mod_python.publisher, it keep reportint : ImportError: libclntsh.so.10.1: cannot open shared object file: No such file or directory Can I anyone have a clue what's the matter, any help would be appreciated! -- http://mail.python.org/mailman/listinfo/python-list
