Re: http.client and dns lookups
> On 31 Jan 2022, at 19:22, Dieter Maurer wrote: > > Michael Welle wrote at 2022-1-30 09:18 +0100: >> ... > The machine this is running on regularly switches >> its network configuration without restarting the Python application. Now >> it turns out that the application is still using an old, outdated dns >> server after such a network configuration switch. > > It is unlikely that Python performs the host -> IP address translation > itself. Almost surely, the translation is delegated to functions > of the underlying C runtime library, e.g. "gethostbyname". And glibc does not have the ability to reload the /etc/resolv.conf I recall. If you rely on that you will need to change to using something like dnsmasq or systemd-resolved so that resolv.conf uses localhost. Barry > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Data unchanged when passing data to Python in multiprocessing shared memory
I am using multiprocesssing.shared_memory to pass data between NASM and Python. The shared memory is created in NASM before Python is called. Python connects to the shm: shm_00 = shared_memory.SharedMemory(name='shm_object_00',create=False). I have used shared memory at other points in this project to pass text data from Python back to NASM with no problems. But now this time I need to pass a 32-bit integer (specifically 32,894) from NASM to Python. First I convert the integer to bytes in a C program linked into NASM: unsigned char bytes[4] unsigned long int_to_convert = 32894; bytes[0] = (int_to_convert >> 24) & 0xFF; bytes[1] = (int_to_convert >> 16) & 0xFF; bytes[2] = (int_to_convert >> 8) & 0xFF; bytes[3] = int_to_convert & 0xFF; memcpy(outbuf, bytes, 4); where outbuf is a pointer to the shared memory. On return from C to NASM, I verify that the first four bytes of the shared memory contain what I want, and they are 0, 0, -128, 126 which is binary 1000 0110, and that's correct (32,894). Next I send a message to Python through a FIFO to read the data from shared memory. Python uses the following code to read the first four bytes of the shared memory: byte_val = shm_00.buf[:4] print(shm_00.buf[0]) print(shm_00.buf[1]) print(shm_00.buf[2]) print(shm_00.buf[3]) But the bytes show as 40 39 96 96, which is exactly what the first four bytes of this shared memory contained before I called C to overwrite them with the bytes 0, 0, -128, 126. So Python does not see the updated bytes, and naturally int.from_bytes(byte_val, "little") does not return the result I want. I know that Python refers to shm00.buf, using the buffer protocol. Is that the reason that Python can't see the data that has been updated by another language? So my question is, how can I alter the data in shared memory in a non-Python language to pass back to Python? Thanks, Jen -- https://mail.python.org/mailman/listinfo/python-list
Re: http.client and dns lookups
> On 1 Feb 2022, at 19:21, Michael Welle wrote: > > Hello, > > Barry writes: > On 31 Jan 2022, at 19:22, Dieter Maurer wrote: >>> >>> Michael Welle wrote at 2022-1-30 09:18 +0100: ... >>> The machine this is running on regularly switches its network configuration without restarting the Python application. Now it turns out that the application is still using an old, outdated dns server after such a network configuration switch. >>> >>> It is unlikely that Python performs the host -> IP address translation >>> itself. Almost surely, the translation is delegated to functions >>> of the underlying C runtime library, e.g. "gethostbyname". >> >> And glibc does not have the ability to reload the /etc/resolv.conf I recall. >> If you rely on that you will need to change to using something like >> dnsmasq or systemd-resolved so that resolv.conf uses localhost. > hmmm, I don't know if that's all. For a test I just changed > /etc/resolv.conf manually while the application was sleeping. After > returning from sleep the application knew the change I had made in > resolv.conf and, interestingly, an exception (name resolution failure, > which is correct with the change I made) surfaces in the application. > That doesn't happen when the 'real' issue occurs. Attaching strace to > the Python process I can see that resolv.conf is stat'ed and open'ed. I > guess now I'm more confused than before ;). There must be an additional > condition that I'm missing. That is good to know, last time I hit this was on centos 6 a little while ago. Barry > > Thanks for helping > hmw > -- > https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
Re: Data unchanged when passing data to Python in multiprocessing shared memory
> On 1 Feb 2022, at 20:26, Jen Kris via Python-list > wrote: > > I am using multiprocesssing.shared_memory to pass data between NASM and > Python. The shared memory is created in NASM before Python is called. > Python connects to the shm: shm_00 = > shared_memory.SharedMemory(name='shm_object_00',create=False). > > I have used shared memory at other points in this project to pass text data > from Python back to NASM with no problems. But now this time I need to pass > a 32-bit integer (specifically 32,894) from NASM to Python. > > First I convert the integer to bytes in a C program linked into NASM: > > unsigned char bytes[4] > unsigned long int_to_convert = 32894; > > bytes[0] = (int_to_convert >> 24) & 0xFF; > bytes[1] = (int_to_convert >> 16) & 0xFF; > bytes[2] = (int_to_convert >> 8) & 0xFF; > bytes[3] = int_to_convert & 0xFF; > memcpy(outbuf, bytes, 4); > > where outbuf is a pointer to the shared memory. On return from C to NASM, I > verify that the first four bytes of the shared memory contain what I want, > and they are 0, 0, -128, 126 which is binary 1000 > 0110, and that's correct (32,894). > > Next I send a message to Python through a FIFO to read the data from shared > memory. Python uses the following code to read the first four bytes of the > shared memory: > > byte_val = shm_00.buf[:4] > print(shm_00.buf[0]) > print(shm_00.buf[1]) > print(shm_00.buf[2]) > print(shm_00.buf[3]) > > But the bytes show as 40 39 96 96, which is exactly what the first four bytes > of this shared memory contained before I called C to overwrite them with the > bytes 0, 0, -128, 126. So Python does not see the updated bytes, and > naturally int.from_bytes(byte_val, "little") does not return the result I > want. > > I know that Python refers to shm00.buf, using the buffer protocol. Is that > the reason that Python can't see the data that has been updated by another > language? > > So my question is, how can I alter the data in shared memory in a non-Python > language to pass back to Python? Maybe you need to use a memory barrier to force the data to be seen by another cpu? Maybe use shm lock operation to sync both sides? Googling I see people talking about using stdatomic.h for this. But I am far from clear what you would need to do. Barry > > Thanks, > > Jen > > -- > https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
Re: Data unchanged when passing data to Python in multiprocessing shared memory
Barry, thanks for your reply.
On the theory that it is not yet possible to pass data from a non-Python
language to Python with multiprocessing.shared_memory, I bypassed the problem
by attaching 4 bytes to my FIFO pipe message from NASM to Python:
byte_val = v[10:14]
where v is the message read from the FIFO. Then:
breakup = int.from_bytes(byte_val, "big")
print("this is breakup " + str(breakup))
Python prints: this is breakup 32894
Note that I had to switch from little endian to big endian. Python is little
endian by default, but in this case it's big endian.
However, if anyone on this list knows how to pass data from a non-Python
language to Python in multiprocessing.shared_memory please let me (and the
list) know.
Thanks.
Feb 1, 2022, 14:20 by [email protected]:
>
>
>> On 1 Feb 2022, at 20:26, Jen Kris via Python-list
>> wrote:
>>
>> I am using multiprocesssing.shared_memory to pass data between NASM and
>> Python. The shared memory is created in NASM before Python is called.
>> Python connects to the shm: shm_00 =
>> shared_memory.SharedMemory(name='shm_object_00',create=False).
>>
>> I have used shared memory at other points in this project to pass text data
>> from Python back to NASM with no problems. But now this time I need to pass
>> a 32-bit integer (specifically 32,894) from NASM to Python.
>>
>> First I convert the integer to bytes in a C program linked into NASM:
>>
>> unsigned char bytes[4]
>> unsigned long int_to_convert = 32894;
>>
>> bytes[0] = (int_to_convert >> 24) & 0xFF;
>> bytes[1] = (int_to_convert >> 16) & 0xFF;
>> bytes[2] = (int_to_convert >> 8) & 0xFF;
>> bytes[3] = int_to_convert & 0xFF;
>> memcpy(outbuf, bytes, 4);
>>
>> where outbuf is a pointer to the shared memory. On return from C to NASM, I
>> verify that the first four bytes of the shared memory contain what I want,
>> and they are 0, 0, -128, 126 which is binary 1000
>> 0110, and that's correct (32,894).
>>
>> Next I send a message to Python through a FIFO to read the data from shared
>> memory. Python uses the following code to read the first four bytes of the
>> shared memory:
>>
>> byte_val = shm_00.buf[:4]
>> print(shm_00.buf[0])
>> print(shm_00.buf[1])
>> print(shm_00.buf[2])
>> print(shm_00.buf[3])
>>
>> But the bytes show as 40 39 96 96, which is exactly what the first four
>> bytes of this shared memory contained before I called C to overwrite them
>> with the bytes 0, 0, -128, 126. So Python does not see the updated bytes,
>> and naturally int.from_bytes(byte_val, "little") does not return the result
>> I want.
>>
>> I know that Python refers to shm00.buf, using the buffer protocol. Is that
>> the reason that Python can't see the data that has been updated by another
>> language?
>>
>> So my question is, how can I alter the data in shared memory in a non-Python
>> language to pass back to Python?
>>
>
> Maybe you need to use a memory barrier to force the data to be seen by
> another cpu?
> Maybe use shm lock operation to sync both sides?
> Googling I see people talking about using stdatomic.h for this.
>
> But I am far from clear what you would need to do.
>
> Barry
>
>>
>> Thanks,
>>
>> Jen
>>
>> --
>> https://mail.python.org/mailman/listinfo/python-list
>>
--
https://mail.python.org/mailman/listinfo/python-list
