> Why have you chosen to hide information that may be useful in debugging your
> problem?
I’m truly sorry for the inconvenience but I do have some concerns of security
and privacy. I confirm it is not a broadcast address because it is the public
IP of the server and this issue has a probability of 1% to happen. The address
cannot just be a broadcast address at 1% of the time while not at the rest of
99%. I also double checked it by SSHing to the address I copied from the kdump,
if it makes sense.
> So, since the manpage mentions blocking pf, I suggest the hypothesis "it
> returns EACCES because pf is blocking your packets". I can think of several
> ways to test that; what testing have you performed to confirm or rule out
> that possibility? "doas pfctl -d; run test; doas pfctl -e”?
This issue is really hard to reproduce because the application works at most of
the time, but I think you are right. I’ll be watching the pf log in next weeks.
> Alternatively: what's different about *that* call? Does every sento() call
> on that socket fail? What is special about that socket? If other sendto()
> calls succeed, what is different about that call? Earlier setsockopt() calls?
The call failed once after a few days of running. I have no idea of how to
present this but it seems the second and third arguments of sendto call are
always changing?
3058 myapp CALL
sendto(5,0x1689f5f6100,0x5d,0x400<MSG_NOSIGNAL>,0x7f7fffff1144,0x10)
3058 myapp STRU struct sockaddr { AF_INET, xxx.xx.xxx.xx:yyyyy }
3058 myapp GIO fd 5 wrote 93 bytes
"\M^Ai9\M^N\M^U\M-@9
\M-7\M-B\M-m\M^C.\M-ut\M^E~)\M^H'w\M-A\M-p|\M-+DK\M-V\M-8v\M-~c\240\M-=y\M-E\M-*\M-+.\M-I\M-m\M^[\M-s\M-&\\\M^GwP\M-I\r\M^[\M-C\^]B\M-%\M-<\M-q\
\M-rk\M-_\M-g\M-9Y\^B\M-+Y\M-:K\M^X\^E/*\M-4_\M-x\at\M-{\M-$\M-[\^NU\\\M^HPk\M-i\\\M-M\M-H8t
"
3058 myapp RET sendto 93/0x5d
3058 myapp CALL kevent(3,0,0,0x168a7f0c000,128,0)
3058 myapp STRU struct kevent { ident=4, filter=EVFILT_READ,
flags=0x61<EV_ADD|EV_CLEAR|EV_RECEIPT>, fflags=0<>, data=73, udata=0x2 }
3058 myapp RET kevent 1
3058 myapp CALL read(4,0x7f7fffff0af0,0x7d0)
3058 myapp GIO fd 4 read 73 bytes
"\0\0\0\^BE\0\0E\M^_?\0\0@\^Q\M-9\M-:\M-@\M-(_\^D\^A\^A\^A\^A\M-x\M-U\0005\0001\M^U\M-Q\M-B\M-U\^A\0\0\^A\0\0\0\0\0\^A\^Bbt\^Ehenbt\^Ccom\0\0\^A\0\^A\0\0)\^D\M-P\0\
\0\M^@\0\0\0"
3058 myapp RET read 73/0x49
3058 myapp CALL read(4,0x7f7fffff0af0,0x7d0)
3058 myapp GIO fd 4 read 79 bytes
"\0\0\0\^BE\0\0K\M-i\M^E\0\0@\^Qpo\M-@\M-(_\^D\^A\0\0\^A\M-Sy\0005\0007\M-8
\M-dB\^A\0\0\^A\0\0\0\0\0\^A\atracker\^Fcity9x\^Ccom\0\0\^A\0\^A\0\0)\^D\M-P\0\0\M^@\0\0\
\0"
3058 myapp RET read 79/0x4f
3058 myapp CALL read(4,0x7f7fffff0af0,0x7d0)
3058 myapp RET read -1 errno 35 Resource temporarily unavailable
3058 myapp CALL read(7,0x7f7fffff07e0,0xc)
3058 myapp GIO fd 7 read 12 bytes
"4z.\M-"\M-)-\M-o\^E\M-v\^Q\M-5u"
3058 myapp RET read 12/0xc
3058 myapp CALL
sendto(5,0x1690776aa00,0x61,0x400<MSG_NOSIGNAL>,0x7f7fffff1144,0x10)
3058 myapp STRU struct sockaddr { AF_INET, xxx.xx.xxx.xx:yyyyy }
3058 myapp GIO fd 5 wrote 97 bytes
"\M-)\M-,\M-r\M-b\^R\M-Jb\M-@\M-b\M-{\M^G"\M-Z\v\M-v\M-L\^^\M-BI\M-X\M-S\M-L]\M-&\^R%\M-?\M-\\M-kN\M^G:\M-NVI\M-@s\0#\M-V\M-}\M-h\M-T\M^C\^R\M-W\M-"\^?\M^Q[:u\M-=\
\M-Q%\M-c\M-+\M-xZ\^B\M-$\M-N\M-1\^Zy\M^D 9#\^Q\M--\M^Jb\M^T(
\M^@\M^L;~\M-<\M-}0\M-G\M-`4z.\M-"\M-)-\M-o\^E\M-v\^Q\M-5u"
3058 myapp RET sendto 97/0x61
3058 myapp CALL read(7,0x7f7fffff07e0,0xc)
3058 myapp GIO fd 7 read 12 bytes
"\M-j\M->\M-9\^V\^U\^^\M-c/h',\M^D"
3058 myapp RET read 12/0xc
3058 myapp CALL
sendto(5,0x1690776a600,0x67,0x400<MSG_NOSIGNAL>,0x7f7fffff1144,0x10)
3058 myapp STRU struct sockaddr { AF_INET, xxx.xx.xxx.xx:yyyyy }
3058 myapp GIO fd 5 wrote 103 bytes
"\^W\^\\M-B\M^E\M-~\M-c\M-3\M-f\M-X\M^Y\M-Z\^V\M-G\M-p\M^L\M^G\M-b\M-T\M^I\M-SWu\M-]\M^Jq\M^Q\M-Z\^Z\M^[\bW>`/fW\b\M^K\M-*\^]:E\M-T\M-[\M^X\M-5M\M^J\M-j\M-v\^^Z@:\
\M-QfQ\a-\M-\)\M-O$[\M-x]\b\M-Wh5\M^F\M^B|I>
\M^CY\M-%\M-6'\M^?g9F?y\^T\M-}\^E\M-g\M-j\M->\M-9\^V\^U\^^\M-c/h',\M^D"
3058 myapp RET sendto 103/0x67
3058 myapp CALL kevent(3,0,0,0x168a7f0c000,128,0)
3058 myapp STRU struct kevent { ident=4, filter=EVFILT_READ,
flags=0x61<EV_ADD|EV_CLEAR|EV_RECEIPT>, fflags=0<>, data=75, udata=0x2 }
3058 myapp RET kevent 1
3058 myapp CALL read(4,0x7f7fffff0af0,0x7d0)
3058 myapp GIO fd 4 read 75 bytes
"\0\0\0\^BE\0\0GS\M^L\0\0@\^Q\^Fm\M-@\M-(_\^D\^A\0\0\^A\M-n\M-)\0005\0003m\M-_N\M-#\^A\0\0\^A\0\0\0\0\0\^A\^Cbtx\aanifilm\^Btv\0\0\^A\0\^A\0\0)\^D\M-P\0\0\M^@\0\0\0"
3058 myapp RET read 75/0x4b
3058 myapp CALL read(4,0x7f7fffff0af0,0x7d0)
3058 myapp RET read -1 errno 35 Resource temporarily unavailable
3058 myapp CALL read(7,0x7f7fffff07e0,0xc)
3058 myapp GIO fd 7 read 12 bytes
"\M^T\M-|\M-e\M-Fk-\M^P\M-u}\M-j\M^U="
3058 myapp RET read 12/0xc
3058 myapp CALL
sendto(5,0x1690776a500,0x63,0x400<MSG_NOSIGNAL>,0x7f7fffff1144,0x10)
3058 myapp STRU struct sockaddr { AF_INET, xxx.xx.xxx.xx:yyyyy }
3058 myapp GIO fd 5 wrote 99 bytes
"\M-3\M-l7X{PM[\M^[\M^S\M-Rv:\M^C2\^_=-?\M^[\M-p\M-Kq;/0\M^_\M-`\M-Yx\M-B_\^CyX\M-B\M^?\M^N\M^H\M-[ykO\M-d\M-qg\M-]\M^L\M-BulvAF\M-"\M-d%)\M-Y\M-mK*`\M^L\M^B\M-I\
\M-5\^B\M-/<\M-?\0005\^Y3\M^?q\M-n\M^Ui\M-^\M^?\M^X!\M-w\^RA\M^T\M-|\M-e\M-Fk-\M^P\M-u}\M-j\M^U="
3058 myapp RET sendto 99/0x63
3058 myapp CALL kevent(3,0,0,0x168a7f0c000,128,0)
3058 myapp STRU struct kevent { ident=4, filter=EVFILT_READ,
flags=0x61<EV_ADD|EV_CLEAR|EV_RECEIPT>, fflags=0<>, data=69, udata=0x2 }
3058 myapp RET kevent 1
3058 myapp CALL read(4,0x7f7fffff0af0,0x7d0)
3058 myapp GIO fd 4 read 69 bytes
"\0\0\0\^BE\0\0AHN\0\0@\^Q\^Q\M-1\M-@\M-(_\^D\^A\0\0\^A\M-u\M-D\0005\0-\M-w\r{\M-{\^A\0\0\^A\0\0\0\0\0\^A\^Dpow7\^Ccom\0\0\^A\0\^A\0\0)\^D\M-P\0\0\M^@\0\0\0"
3058 myapp RET read 69/0x45
3058 myapp CALL read(4,0x7f7fffff0af0,0x7d0)
3058 myapp GIO fd 4 read 79 bytes
"\0\0\0\^BE\0\0K\M-{\M-3\0\0@\^Q^A\M-@\M-(_\^D\^A\0\0\^A\M-U\M^F\0005\0007\^V\M-J\M^C\M^L\^A\0\0\^A\0\0\0\0\0\^A\atracker\^Fcity9x\^Ccom\0\0\^A\0\^A\0\0)\^D\M-P\0\0\
\M^@\0\0\0"
3058 myapp RET read 79/0x4f
3058 myapp CALL read(4,0x7f7fffff0af0,0x7d0)
3058 myapp RET read -1 errno 35 Resource temporarily unavailable
3058 myapp CALL read(7,0x7f7fffff07e0,0xc)
3058 myapp GIO fd 7 read 12 bytes
"\M^Xr\M-r\M-[%\M-n\\\M^L\M-e&\M^_\M-i"
3058 myapp RET read 12/0xc
3058 myapp CALL
sendto(5,0x1689f5f6500,0x5d,0x400<MSG_NOSIGNAL>,0x7f7fffff1144,0x10)
3058 myapp STRU struct sockaddr { AF_INET, xxx.xx.xxx.xx:yyyyy }
3058 myapp RET sendto -1 errno 13 Permission denied
3058 myapp CALL close(5)
3058 myapp RET close 0
3058 myapp CALL munmap(0x1688d3c8000,0x1a1000)
3058 myapp RET munmap 0
3058 myapp CALL munmap(0x169192ad000,0x1a1000)
3058 myapp RET munmap 0
3058 myapp CALL munmap(0x1691cb22000,0x1a1000)
3058 myapp RET munmap 0
3058 myapp CALL munmap(0x168220f8000,0x1a1000)
3058 myapp RET munmap 0
3058 myapp CALL close(4)
3058 myapp RET close 0
3058 myapp CALL close(3)
3058 myapp RET close 0
Siegfried
[email protected]
> On Jun 22, 2021, at 3:20 PM, Philip Guenther <[email protected]> wrote:
>
> On Mon, Jun 21, 2021 at 9:07 PM Siegfried Levin <[email protected]
> <mailto:[email protected]>> wrote:
> Thanks a lot for the hint. Unfortunately I’m still not able to see why sendto
> failed with 13 Permission denied. The AF_INET address masked is the correct
> one of my server, not a broadcast address. A sendto before this one to the
> same address just worked.
>
> 3058 myapp CALL
> sendto(5,0x1689f5f6500,0x5d,0x400<MSG_NOSIGNAL>,0x7f7fffff1144,0x10)
> 3058 myapp STRU struct sockaddr { AF_INET, xxx.xxx.xxx.xxx:yyyy }
>
> Why have you chosen to hide information that may be useful in debugging your
> problem?
>
> "Hi, I'm asking for help but I have to hide addresses because...this
> application is insecure if anyone else has its IP+port? Because I've never
> heard of shodan and don't believe that people are constantly scanning the
> Internet? And while I don't know why it's failing I'm 1000% sure that
> there's no information to be gained from seeing the IP, so if it later turns
> out my understanding of 'broadcast address' is incorrect, the time I've
> wasted for myself and others will be...a total loss?"
>
>
> 3058 myapp RET sendto -1 errno 13 Permission denied
> 3058 myapp CALL close(5)
> 3058 myapp RET close 0
> The dump file is like 600MB. I can provide more trace log if it is necessary
> for locating the root cause.
>
> Use the scientific method:
> * make a testable hypothesis
> * devise a test for that
> * perform the test
> * determine whether the hypothesis has been ruled out or confirmed
>
> So, since the manpage mentions blocking pf, I suggest the hypothesis "it
> returns EACCES because pf is blocking your packets". I can think of several
> ways to test that; what testing have you performed to confirm or rule out
> that possibility? "doas pfctl -d; run test; doas pfctl -e"?
>
>
> Alternatively: what's different about *that* call? Does every sento() call
> on that socket fail? What is special about that socket? If other sendto()
> calls succeed, what is different about that call? Earlier setsockopt() calls?
>
>
> You say "I can confirm the packet was not sent to a broadcast address": *how*
> have you confirmed that your understanding of 'broadcast address' matches the
> kernel's understanding? It ain't just 255.255.255.255....
>
>
> Philip Guenther
>
>
>
>
> Siegfried
> [email protected] <mailto:[email protected]>
>
>
>
>
> > On Jun 15, 2021, at 8:50 PM, Theo de Raadt <[email protected]
> > <mailto:[email protected]>> wrote:
> >
> > use ktrace
> >
> > Siegfried Levin <[email protected]
> > <mailto:[email protected]>> wrote:
> >
> >> Hi,
> >>
> >> I have a application run by a normal user communicating with the server
> >> with UDP. It crashes very occasionally, like once per week, due to EACCES
> >> when sending a UDP packet. According to the manpage
> >> (https://man.openbsd.org/OpenBSD-6.9/sendmsg.2#EACCES
> >> <https://man.openbsd.org/OpenBSD-6.9/sendmsg.2#EACCES>), the reason might
> >> be either being blocked by PF or sending to a broadcast address. I can
> >> confirm the packet was not sent to a broadcast address. However, I cannot
> >> figure out what rule could block the connection occasionally either. The
> >> application can be brought back online without changing any configuration.
> >> Does anyone know what might fix this? I can also rewrite the code to make
> >> it ignore the error and keep trying but that might not be a proper
> >> solution. Running it as root might not be a good idea, too.
> >>
> >> It happens since OpenBSD 6.8. Now I’m running it on 6.9. The application
> >> is written in Rust.
> >>
> >> Siegfried
> >> [email protected] <mailto:[email protected]>
> >>
> >>
> >>
> >>
>
