[Tutor] Help with building bytearray arrays
Admin, please remove my earlier messages. This message is a properly 'self contained' message. Hello, I've been struggling with this for the last day or so and I can't seem to figure out how to make it work. I'll start out by saying that if there's a better approach, then I'm all ears. I'm using the Adafruit Breakout board for the FTDI FT232H part. Along with this comes the Python libraries and Adafruit libraries which I'm using. I don't think that the libraries are the real issue here, but I thought I should mention it just for the sake of information. Basically I'm trying to write a block of unsigned bytes to the device and read back an equal sized block of unsigned bytes. There's a function that is provided called transfer(data_to_send, num_of_bytes) that handles the heavy lifting. Unfortunately there seems to be a bug in the part and if I attempt to send the entire block of bytes (64), the device will lock up. I've been able to determine that if I send 16 bytes at a time, I'm okay. So, I take my bytearray(64) and step through it 16 bytes at a time like this: my function's main pieces are: def transfer_byte_array(): MAX_LOOP_COUNT = 64 slice_size = 16 read_ary = bytearray(MAX_LOOP_COUNT) scratch_ary = bytearray() for step in range (0, MAX_LOOP_COUNT, slice_size): scratch_ary = transfer(data_to_send, slice_size) for bytes in range (0, slice_size): read_ary = scratch_ary[bytes] return(read_ary) Ideally, I'd like to take the slice_size chunks that have been read and concatenate them back togetjer into a long MAX_LOOP_COUNT size array to pass back to the rest of my code. Eg: read_ary = ary_slice[0] + ary_slice[1] + ary_slice[2] + ary_slice[3] I know that the + version doesn't work (or didn't for me) but it is just my attempt at illustrating the overall goal. The problem that I repeatedly run into is with the line: read_ary = scratch_ary[bytes] (or variants thereof) The traceback is this: Traceback (most recent call last): File "SW8T_5.py", line 101, in loop_size = RSI_size_the_loop(Print) File "/home/temp/Python_Scratch/examples/RSI.py", line 350, in RSI_size_the_loop read_ary.append(scratch_ary[singles]) TypeError: an integer or string of size 1 is required or, one of the other common ones that I've seen is TypeError: can't concat bytearray to list This one is confusing because both of the operands are bytearry types.. or at least I thought they should be... when I try to replace the same line with : read_ary += scratch_ary[bytes] or read_ary.append(scratch[bytes]) or read_ary = read_ary + scratch_ary[bytes] I'm obviously missing something fundamental here. Problem is I can't seem to find any examples of people asking this question before on the inter-webs.. the transfer function expects an input of a bytearray and returns the same: def transfer(self, data): """Full-duplex SPI read and write. The specified array of bytes will be clocked out the MOSI line, while simultaneously bytes will be read from the MISO line. Read bytes will be returned as a bytearray object. """ # Build command to read and write SPI data. command = 0x30 | (self.lsbfirst << 3) | (self.read_clock_ve << 2) | self.write_clock_ve logger.debug('SPI transfer with command {0:2X}.'.format(command)) # Compute length low and high bytes. # NOTE: Must actually send length minus one because the MPSSE engine # considers 0 a length of 1 and a length of 65536 length = len(data) len_low = (length-1) & 0xFF len_high = ((length-1) >> 8) & 0xFF # Send command and length. self._assert_cs() self._ft232h._write(str(bytearray((command, len_low, len_high self._ft232h._write(str(bytearray(data))) self._ft232h._write('\x87') self._deassert_cs() # Read response bytes. return bytearray(self._ft232h._poll_read(length)) Thank you in advance to taking time to read. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Help with building bytearray arrays
Point taken on 'bytes'.. thanks. the scratch_ary = bytearray() was my attempt at 'setting' the type of the variable. I had hoped that it would help resolve the error messages telling me that the types didn't go together. Coming from a 'C' background, I find the lack of typing in Python to be confusing. I'm used to working with bytes / words signed and unsigned for a reason. So, if I'm understanding the transfer() function correctly, the function takes and returns a bytearray type. You mentioned about constructing a bytearray if I need one. Can you expand on how I approach that? I'm going to try the experiment you mentioned in hopes of it giving me a better understanding of the 'types' and what happens with the variables. Thank you, On Fri, Sep 7, 2018 at 6:23 PM, Cameron Simpson wrote: > On 07Sep2018 15:45, Chip Wachob wrote: >> >> Basically I'm trying to write a block of unsigned bytes to the device >> and read back an equal sized block of unsigned bytes. There's a >> function that is provided called transfer(data_to_send, num_of_bytes) >> that handles the heavy lifting. Unfortunately there seems to be a bug >> in the part and if I attempt to send the entire block of bytes (64), >> the device will lock up. I've been able to determine that if I send >> 16 bytes at a time, I'm okay. >> >> So, I take my bytearray(64) and step through it 16 bytes at a time like >> this: >> >> my function's main pieces are: >> >> def transfer_byte_array(): >> MAX_LOOP_COUNT = 64 >> slice_size = 16 >> read_ary = bytearray(MAX_LOOP_COUNT) >> scratch_ary = bytearray() >> >> for step in range (0, MAX_LOOP_COUNT, slice_size): >> scratch_ary = transfer(data_to_send, slice_size) >> >> for bytes in range (0, slice_size): >> read_ary = scratch_ary[bytes] >> >> return(read_ary) >> >> >> Ideally, I'd like to take the slice_size chunks that have been read >> and concatenate them back togetjer into a long MAX_LOOP_COUNT size >> array to pass back to the rest of my code. Eg: >> >> read_ary = ary_slice[0] + ary_slice[1] + ary_slice[2] + ary_slice[3] > > > Minor remark: don't use the name "bytes" for a variable, it is a builtin > type name and you're shadowing it. > > It looks to me like "transfer" hands you back a buffer with the read data, > so this: > > scratch_ary = bytearray() > > don't do anything (it gets discarded). > > If you're getting back a bytes or bytearray object from transfer, just > gather them all up in an list: > > returned_buffers = [] > for .. > response = transfer(data_to_send, slice_size) > returned_buffers.append(response) > ... > read_ary = b''.join(returned_buffers) > > Note that that makes a new bytes object for read_ary to refer to. You don't > need the earlier initialisation of read_ary. > > Also note that the bytes object is read only; if that is a problem you'll > need to construct a bytearray instead. > > [...] >> >> The problem that I repeatedly run into is with the line: >> >> read_ary = scratch_ary[bytes] (or variants thereof) >> >> The traceback is this: >> >> Traceback (most recent call last): >> File "SW8T_5.py", line 101, in >>loop_size = RSI_size_the_loop(Print) >> File "/home/temp/Python_Scratch/examples/RSI.py", line 350, in >> RSI_size_the_loop >>read_ary.append(scratch_ary[singles]) >> TypeError: an integer or string of size 1 is required > > > Yeah I thought that looked weird to me too. >> >> or, one of the other common ones that I've seen is >> >> TypeError: can't concat bytearray to list >> >> This one is confusing because both of the operands are bytearry >> types.. or at least I thought they should be... > > > No, one will be a list :-) putting a bunch of: > > print(repr(foo)) > > replacing "foo" with relevant variables will be illuminating to you; you can > see immediately where this are not what you expected. > >> I'm obviously missing something fundamental here. Problem is I can't >> seem to find any examples of people asking this question before on the >> inter-webs.. > > > You have the opposite of my problem. I can often find people asking the same > question, but less often an answer. Or a decent answer, anyway. > > Cheers, > Cameron Simpson ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Additionally
On Fri, Sep 07, 2018 at 10:16:38PM -0400, Chip Wachob wrote: > Thank you. > > I sent a message with the contents of both previous messages. > > Hopefully the admin can just delete the two previous messages > > Thanks for bearing with me. I'm used to posting on forum boards, not via > email. No problem. And if I come across as a bit short tempered, its because I'm suffering a lot of pain and discomfort at the moment due to illness. I'm trying not to let it show, but sometimes it might. -- Steve ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Help with building bytearray arrays
Please try to adopt the inline reply style; we prefer it here. It lets us reply point by point and makes messages read like conversations. Anyway... On 07Sep2018 23:57, Chip Wachob wrote: Point taken on 'bytes'.. thanks. the scratch_ary = bytearray() was my attempt at 'setting' the type of the variable. I had hoped that it would help resolve the error messages telling me that the types didn't go together. Coming from a 'C' background, I find the lack of typing in Python to be confusing. I'm used to working with bytes / words signed and unsigned for a reason. Ok. Variables aren't typed. Variables are references to objects, which _are_ typed. So pointing a variable at a bytearray doesn't set its type in any persistent sense. Thus: x = 1 x = "a string" x = bytearray() All just point "x" at different objects. If you'd like a C metaphor, variables are _like_ (void*) pointers: they can reference any kind of object. Aside: they're not pointers, avoid the term - people will complain. As far as the language spec goes they're references. Which may in a particular implementation be _implemented internally_ using pointers. So, if I'm understanding the transfer() function correctly, the function takes and returns a bytearray type. It would be good to see the specification for the transfer function. They we can adhere to its requirements. Can you supply a URL? You mentioned about constructing a bytearray if I need one. Can you expand on how I approach that? Well, you were getting several bytes or bytearray objects back from transfer and wanted to join them together. The efficient way is to accumulate them in a list and then join them all together later using the bytearry (or bytes) .join method: https://docs.python.org/3/library/stdtypes.html#bytearray.join So: data_list = [] for # send some data, get some data back data = transfer(...) # add that buffer to the data_list data_list.append(data) # make a single bytes object being the concatenation of all the smaller data # chunks all_data = bytes.join(data_list) It looks to me like your transfer() function is handed a bytes or bytearray and returns one. Normally that would be a separate bytes or bytearray. Aside: bytes and bytearray objects are generally the way Python 3 deals with chunks of bytes. A "bytes" is readonly and a bytearray may have its contents modified. From a C background, they're like an array of unsigned chars. I'm going to try the experiment you mentioned in hopes of it giving me a better understanding of the 'types' and what happens with the variables. Sounds good. Cheers, Cameron Simpson ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Help with building bytearray arrays
On 08/09/18 04:57, Chip Wachob wrote: > was my attempt at 'setting' the type of the variable. A variable in Python is just a name. It has no type. Only values have types. So you can set (or change) the type of a value but not of a variable. > Coming from a 'C' background, I find the lack of typing in Python to > be confusing. There is no shortage of typing in Python its just applied to the values(objects) not to their labels. > I'm used to working with bytes / words signed and > unsigned for a reason. The same applies in Python. You just have to separate the concepts of variable name and variable value. In C those two concepts get kind of squished together but in Python the name/value duality is strong. You can always check the type of an object by using the type() function or if you think a type error is likely catch it when it happens with: try:... except TypeError: -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Help with building bytearray arrays
On 08/09/18 03:15, Chip Wachob wrote: > Admin, please remove my earlier messages. No can do, once the emails are sent by the server they are out there on the net, stored in people's mailboxes and in various internet archives. When using a mailing list always check before sending, there's no going back... -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Help with building bytearray arrays
On 08/09/18 03:15, Chip Wachob wrote: > my function's main pieces are: It would probably be better to post the entire function, a partial code sample like this just gives us an inkling of what you are trying to do but not enough to be sure of where the errors lie. > def transfer_byte_array(): >MAX_LOOP_COUNT = 64 >slice_size = 16 >read_ary = bytearray(MAX_LOOP_COUNT) >scratch_ary = bytearray() > >for step in range (0, MAX_LOOP_COUNT, slice_size): > scratch_ary = transfer(data_to_send, slice_size) You pass in two arguments here but in your description of transfer below it is a method of an object that only takes one (self being the object). Something is broken here. I would expect, given the code below, to see something like scratch_ary = someObject.transfer(data_to_send) > for bytes in range (0, slice_size): > read_ary = scratch_ary[bytes] This is replacing rad_ary each time with a single value from scratch_ary. At the end of theloop read_ary wil;l cpontain the last valuye in scratch_ary. You can achieve the same result without a loop: raed_ary = scratch_ary[-1] But I suspect that's not what you want. I think what you really want is the first slice_size elements of scratch_ary: read_ary = scratch_ary[:slice_size] # use slicing > Ideally, I'd like to take the slice_size chunks that have been read > and concatenate them back togetjer into a long MAX_LOOP_COUNT size > array to pass back to the rest of my code. Eg: You need to create a list of read_ary results = [] then after creating each read_ary value append it to results. results.append(read_ary) Then, at the very end, return the summation of all the lists in results. return sum(results,[]) > def transfer(self, data): The self parameter indicates that this is a method definition from a class. That implies that to use it you must create an instance of the class and call the method on that instance. HTH -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Help with building bytearray arrays
On 08Sep2018 20:01, Cameron Simpson wrote: So, if I'm understanding the transfer() function correctly, the function takes and returns a bytearray type. It would be good to see the specification for the transfer function. They we can adhere to its requirements. Can you supply a URL? I see you supplied the whole transfer function in your first message :-( I'll recite it here and walk through its contents to explain: def transfer(self, data): Ok, this is a method, a particular type of function associated with a class instance. (All objects are class instances, and the class is their type.) So to call this you would normally have a control object of some kind. [...] Ah, it looks like you should have an SpiDev instance, inferring from this code: https://github.com/adafruit/Adafruit_Python_GPIO/blob/master/Adafruit_GPIO/SPI.py So suppose you've got such an object and a variable referring to it, let's say it is named "spi". You'd normally call the transfer function like this: spi.transfer(some_data) instead of calling transfer() "bare", as it were. When you call a Python method, the instance itself is implicitly passed as the parameter "self" (well, the first parameter - in Python we always call this "self" like C++ uses "this"). """Full-duplex SPI read and write. The specified array of bytes will be clocked out the MOSI line, while simultaneously bytes will be read from the MISO line. Read bytes will be returned as a bytearray object. """ This is a docstring. It is like a comment but it gets attached to the function and can be inspected. Not your problem. # Build command to read and write SPI data. command = 0x30 | (self.lsbfirst << 3) | (self.read_clock_ve << 2) | self.write_clock_ve This constructs a value just as you would in C. logger.debug('SPI transfer with command {0:2X}.'.format(command)) Write a debugging message. # Compute length low and high bytes. # NOTE: Must actually send length minus one because the MPSSE engine # considers 0 a length of 1 and a length of 65536 length = len(data) len_low = (length-1) & 0xFF len_high = ((length-1) >> 8) & 0xFF All just like C. # Send command and length. self._assert_cs() I would guess that this raises a control signal. Ah, RS-232? So clear-to-send then. self._ft232h._write(str(bytearray((command, len_low, len_high Ok, it looks like this is Python 2, not Python 3. Let's unpack it. "(command, len_low, len_high)" is a tuple of 3 values. A tuple is like a read only list. We're passing that to the bytearray() constructor, which will accept an iterable of values and make a bytes buffer. And we want to write that to "self._ft232h". Which expects a str, which is why I think this is Python 2. In Python 2 there's no "bytes" type and str is an immutable array of 8-bit character values. So writing bytes uses str. So this tabkes a tuple of values, to be bytes, makes those into a bytearray and makes that into a str, and writes that str to the serial line. self._ft232h._write(str(bytearray(data))) We write that data itself. self._ft232h._write('\x87') self._deassert_cs() And here we lower the control signal. # Read response bytes. return bytearray(self._ft232h._poll_read(length)) This calls the _poll_read method, presumably requesting "length" bytes, the same length as the supplied data. I presume we get a str back: we make a new bytearray with those bytes in it and return it. So you do get a new bytearray back from each call, so we can accumulate them into a list and then join them together later to make a single bytearray. Why this faffing about with str and bytearray? Probably for Python 2/3 compatibility, and because you want to deal with bytes (small ints) instead of characters. Ignore it: we're dealing with bytes. Cheers, Cameron Simpson ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] No module named uno in virtual environment
On 09/07/2018 03:10 PM, Jim wrote: > Mint 18.1 > System python3 3.5.2 > > Python3-uno is available to the system python3. How can I make it > available to python 3.6.5 in a virtual environment I installed using venv? with your virtualenv activated, just install it. python -m pip install uno ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] No module named uno in virtual environment
On 09/08/2018 05:26 PM, Mats Wichmann wrote: On 09/07/2018 03:10 PM, Jim wrote: Mint 18.1 System python3 3.5.2 Python3-uno is available to the system python3. How can I make it available to python 3.6.5 in a virtual environment I installed using venv? with your virtualenv activated, just install it. python -m pip install uno My bad, I did not explain what python3-uno does. It is a system package that lets python interact with libreoffice. The uno that pip installs is a html generator. I used apt to install python3-uno on my system. System python3 can use it and the python 3.5 in one of my virtual environments can use it. Python 3.6 in my other virtual environment cannot access it. So to re-phrase my question. Python3-uno is installed on my system what do I need to do to get python 3.6 in a virtual environment to use it? Maybe I am missing something obvious. With all the advice given to use virtual environments I would think that this problem has come up, but googling did not produce a solution. Regards, Jim ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Help with building bytearray arrays
On 08Sep2018 11:40, Alan Gauld wrote: On 08/09/18 03:15, Chip Wachob wrote: Ideally, I'd like to take the slice_size chunks that have been read and concatenate them back togetjer into a long MAX_LOOP_COUNT size array to pass back to the rest of my code. Eg: You need to create a list of read_ary results = [] then after creating each read_ary value append it to results. results.append(read_ary) Then, at the very end, return the summation of all the lists in results. return sum(results,[]) Actually he's getting back bytearray instances from transfer and wants to join them up (his function does a few small transfers to work around an issue with one big transfer). His earlier code is just confused. So he wants: bytearray().join(results) Hacked example: >>> bytearray().join( (bytearray("foo"),bytearray("bah")) ) bytearray(b'foobah') And he's working with bytearrays because the target library is Python 2, where there's no bytes type. Cheers, Cameron Simpson ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] No module named uno in virtual environment
On September 8, 2018 5:46:46 PM MDT, Jim wrote: >On 09/08/2018 05:26 PM, Mats Wichmann wrote: >> On 09/07/2018 03:10 PM, Jim wrote: >>> Mint 18.1 >>> System python3 3.5.2 >>> >>> Python3-uno is available to the system python3. How can I make it >>> available to python 3.6.5 in a virtual environment I installed using >venv? >> >> with your virtualenv activated, just install it. >> >> python -m pip install uno >> > >My bad, I did not explain what python3-uno does. It is a system package > >that lets python interact with libreoffice. The uno that pip installs >is >a html generator. > >I used apt to install python3-uno on my system. System python3 can use >it and the python 3.5 in one of my virtual environments can use it. >Python 3.6 in my other virtual environment cannot access it. > >So to re-phrase my question. Python3-uno is installed on my system what > >do I need to do to get python 3.6 in a virtual environment to use it? > >Maybe I am missing something obvious. With all the advice given to use >virtual environments I would think that this problem has come up, but >googling did not produce a solution. > >Regards, Jim > >___ >Tutor maillist - Tutor@python.org >To unsubscribe or change subscription options: >https://mail.python.org/mailman/listinfo/tutor ok, awkward naming issue. try openoffice-python instead? -- Sent from a mobile device with K-9 Mail. Please excuse my brevity. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor