Hi, Sending messages with out-of-line data length equal or greater than 512 MB fails silently. The message arrives to the receiver, dataCnt indicates the amount pointed by the sender, but the buffer is not mapped into the receiver's space. This usually results in a unsolved page fault in the receiver when trying to access to addresses that should be covered by that buffer.
The problem is located in ipc_kmsg.c:1422 and ipc_kmsg.c:2416, which both read as this: length = ((number * size) + 7) >> 3; length is a local variable, number is data length, and size is the size for each element (?), 8 in our case. All those are 32 bits vars, so an integer overflow arises for values equal or greater than 512*1024*1024. This can be fixed by changing length to be an unsigned long long, and casting number to that type for this operation. But an interesting related question is if users should be allowed to send an arbitrary amount of OOL data with each message. Perhaps an static limit should be imposed, returning some error for mach_msg, and glibc changed to cope with this situations. What do you think?
