Re: increase netcat's buffer...

2014-10-30 Thread John-Mark Gurney
Ted Unangst wrote this message on Thu, Oct 30, 2014 at 12:09 -0400: > On Mon, Oct 13, 2014 at 15:02, Arne Becker wrote: > > > OK, no more fiddling with O_NONBLOCK. > > New diff below, tested with tcpbench and file transfers. > > I think this is good. Thanks, committed. We'll let it sit for a whil

Re: increase netcat's buffer...

2014-10-30 Thread Ted Unangst
On Mon, Oct 13, 2014 at 15:02, Arne Becker wrote: > OK, no more fiddling with O_NONBLOCK. > New diff below, tested with tcpbench and file transfers. I think this is good. Thanks, committed. We'll let it sit for a while and then see what if any changes need to take place on the buffer side. Maybe

Re: increase netcat's buffer...

2014-10-29 Thread Arne Becker
Ping?

Re: increase netcat's buffer...

2014-10-13 Thread Arne Becker
Hi again. >> +/* make all fds non-blocking */ >> +for (n = 0; n < 4; n++) { >> +if (pfd[n].fd == -1) >> +continue; >> +flags = fcntl(pfd[n].fd, F_GETFL, 0); >> +/* >> + * For sockets and pipes, we want non-block, but setti

Re: increase netcat's buffer...

2014-07-09 Thread Ted Unangst
On Wed, Jul 09, 2014 at 18:33, Arne Becker wrote: > atelnet uses atomicio, which depends on blocking sockets. Since we call > atelnet from readwrite, the sockets are likely non-blocking. > If we enter the for()-loop in atelnet, we set the sockets to blocking > and remember that, so we do it only on

Re: increase netcat's buffer...

2014-07-09 Thread Arne Becker
Hi. >> -err(1, "bind failed"); >> +errx(1, "bind failed: %s", strerror(errno)); >> freeaddrinfo(ares); > > This doesn't seem necessary, or correct. Indeed. New patch below. >> @@ -640,7 +648,7 @@ timeout_connect(int s, const struct sock >>

Re: increase netcat's buffer...

2014-07-08 Thread Ted Unangst
On Thu, Jun 26, 2014 at 13:43, Arne Becker wrote: > Hi. > >> Now soliciting diffs to change readwrite to a loop with two buffers >> that poll()s in all four directions. :) > > Good thing you made me remember I wrote just this a while ago. > This is my first OpenBSD diff, so tell me if I missed a

Re: increase netcat's buffer...

2014-06-27 Thread Remco
Ted Unangst wrote: > On Mon, Jun 09, 2014 at 21:54, Theo de Raadt wrote: >>> > A better patch is probably the following which also increases the size >>> > of the buffer to at least 64k: >>> >>> Agreed. >> >> One thing to be aware of. That function is syncronous. It will read >> as much as it c

Re: increase netcat's buffer...

2014-06-26 Thread Ted Unangst
On Thu, Jun 26, 2014 at 11:12, sven falempin wrote: > What is a 'good maximum' size for a buffer like this in amd64 world and > arm ? > From which amount should we consider to not use the stack. (I know it > depends the type of function, > but lets focus on this case ) Stack rlimit is 4096K, so

Re: increase netcat's buffer...

2014-06-26 Thread sven falempin
On Thu, Jun 26, 2014 at 10:09 AM, Ted Unangst wrote: > On Thu, Jun 26, 2014 at 08:33, sven falempin wrote: > >> For a review i dislike >> + unsigned char stdinbuf[BUFSIZE]; >> and the memmove on it: >> >> Dear tech savvy, isn it better to malloc a buffer like this instead of >> alloca it ? >

Re: increase netcat's buffer...

2014-06-26 Thread Ted Unangst
On Thu, Jun 26, 2014 at 08:33, sven falempin wrote: > For a review i dislike > + unsigned char stdinbuf[BUFSIZE]; > and the memmove on it: > > Dear tech savvy, isn it better to malloc a buffer like this instead of > alloca it ? > or just a static buffer would be better so it is in the progr

Re: increase netcat's buffer...

2014-06-26 Thread sven falempin
On Thu, Jun 26, 2014 at 9:37 AM, Arne Becker wrote: > Hi. > >> If the buffer is fixed, dont bother memmove it, just remember the >> begining and the end: >> http://en.wikipedia.org/wiki/Circular_buffer > > There's a tradeoff - lots of memmove vs. lots of very small reads/writes > if you get near t

Re: increase netcat's buffer...

2014-06-26 Thread Arne Becker
Hi. > If the buffer is fixed, dont bother memmove it, just remember the > begining and the end: > http://en.wikipedia.org/wiki/Circular_buffer There's a tradeoff - lots of memmove vs. lots of very small reads/writes if you get near the end of the buffer. My gut feeling told me that local things,

Re: increase netcat's buffer...

2014-06-26 Thread Stuart Henderson
On 2014/06/26 08:33, sven falempin wrote: > i have Zero idea if it is right or wrong, just warn because the symbol > was lonely. A diff only tells part of the story, it is also necessary to look at the surrounding code.

Re: increase netcat's buffer...

2014-06-26 Thread sven falempin
On Thu, Jun 26, 2014 at 8:21 AM, Arne Becker wrote: > Hi. > >>> + /* listen and net in gone, queues empty, done */ >>> + if (lflag && pfd[POLL_NETIN].fd == -1 >> >> lflag ??? >> warning only one ref in the diff >> > > lflag is a global, the "listen" flag, as in: > nc -l

Re: increase netcat's buffer...

2014-06-26 Thread Arne Becker
Hi. >> + /* listen and net in gone, queues empty, done */ >> + if (lflag && pfd[POLL_NETIN].fd == -1 > > lflag ??? > warning only one ref in the diff > lflag is a global, the "listen" flag, as in: nc -l 127.0.0.1 80 I believe this is correct. Only when we listen do

Re: increase netcat's buffer...

2014-06-26 Thread Stuart Henderson
On 2014/06/26 08:13, sven falempin wrote: > > + close(net_fd); > > + return; > > + } > > + /* listen and net in gone, queues empty, done */ > > + if (lflag && pfd[POLL_NETIN].fd == -1 > > lflag ??? > warning o

Re: increase netcat's buffer...

2014-06-26 Thread sven falempin
On Thu, Jun 26, 2014 at 7:43 AM, Arne Becker wrote: > Hi. > >> Now soliciting diffs to change readwrite to a loop with two buffers >> that poll()s in all four directions. :) > > Good thing you made me remember I wrote just this a while ago. > This is my first OpenBSD diff, so tell me if I missed a

Re: increase netcat's buffer...

2014-06-26 Thread Arne Becker
Hi. > Now soliciting diffs to change readwrite to a loop with two buffers > that poll()s in all four directions. :) Good thing you made me remember I wrote just this a while ago. This is my first OpenBSD diff, so tell me if I missed anything obvious. Tested quite extensively originally; for this

Re: increase netcat's buffer...

2014-06-22 Thread Ted Unangst
On Fri, Jun 13, 2014 at 07:50, sven falempin wrote: >> Now soliciting diffs to change readwrite to a loop with two buffers >> that poll()s in all four directions. :) >> >> > Is using kqueue ok ? Sorry. Returning to this point in the thread, I agree with Theo the answer is no. People are running t

Re: increase netcat's buffer...

2014-06-22 Thread sven falempin
On Sun, Jun 22, 2014 at 9:20 AM, John-Mark Gurney wrote: > sven falempin wrote this message on Tue, Jun 17, 2014 at 07:42 -0400: > > On Mon, Jun 16, 2014 at 10:57 PM, Ted Unangst > wrote: > > > > > On Sat, Jun 14, 2014 at 10:55, Ted Unangst wrote: > > > > On Fri, Jun 13, 2014 at 10:40, sven fale

Re: increase netcat's buffer...

2014-06-22 Thread John-Mark Gurney
sven falempin wrote this message on Tue, Jun 17, 2014 at 07:42 -0400: > On Mon, Jun 16, 2014 at 10:57 PM, Ted Unangst wrote: > > > On Sat, Jun 14, 2014 at 10:55, Ted Unangst wrote: > > > On Fri, Jun 13, 2014 at 10:40, sven falempin wrote: > > > > Now soliciting diffs to change readwrite

Re: increase netcat's buffer...

2014-06-17 Thread Theo de Raadt
It should not use kevent. That makes the code non-portable. Use poll().

Re: increase netcat's buffer...

2014-06-17 Thread sven falempin
On Mon, Jun 16, 2014 at 10:57 PM, Ted Unangst wrote: > On Sat, Jun 14, 2014 at 10:55, Ted Unangst wrote: > > On Fri, Jun 13, 2014 at 10:40, sven falempin wrote: > > Now soliciting diffs to change readwrite to a loop with two buffers > that poll()s in all four directions. :) >

Re: increase netcat's buffer...

2014-06-17 Thread sven falempin
On Mon, Jun 16, 2014 at 10:57 PM, Ted Unangst wrote: > On Sat, Jun 14, 2014 at 10:55, Ted Unangst wrote: > > On Fri, Jun 13, 2014 at 10:40, sven falempin wrote: > > Now soliciting diffs to change readwrite to a loop with two buffers > that poll()s in all four directions. :) >

Re: increase netcat's buffer...

2014-06-16 Thread Ted Unangst
On Sat, Jun 14, 2014 at 10:55, Ted Unangst wrote: > On Fri, Jun 13, 2014 at 10:40, sven falempin wrote: Now soliciting diffs to change readwrite to a loop with two buffers that poll()s in all four directions. :) >>> Is using kqueue ok ? >>> >>> like : http://pastebin.com/F1

Re: increase netcat's buffer...

2014-06-14 Thread Ted Unangst
On Fri, Jun 13, 2014 at 10:40, sven falempin wrote: >>> >>> Now soliciting diffs to change readwrite to a loop with two buffers >>> that poll()s in all four directions. :) >>> >>> >> Is using kqueue ok ? >> >> like : http://pastebin.com/F1c5Hswi uh, maybe. the kqueue changes make it harder to revi

Re: increase netcat's buffer...

2014-06-13 Thread sven falempin
On Fri, Jun 13, 2014 at 7:50 AM, sven falempin wrote: > > > > On Tue, Jun 10, 2014 at 12:45 PM, Ted Unangst wrote: > >> On Mon, Jun 09, 2014 at 21:54, Theo de Raadt wrote: >> >> > A better patch is probably the following which also increases the >> size >> >> > of the buffer to at least 64k: >>

Re: increase netcat's buffer...

2014-06-13 Thread sven falempin
On Tue, Jun 10, 2014 at 12:45 PM, Ted Unangst wrote: > On Mon, Jun 09, 2014 at 21:54, Theo de Raadt wrote: > >> > A better patch is probably the following which also increases the size > >> > of the buffer to at least 64k: > >> > >> Agreed. > > > > One thing to be aware of. That function is sync

Re: increase netcat's buffer...

2014-06-10 Thread Bernte
On 10/06/14 19:57, Bernte wrote: > On 10/06/14 17:45, Ted Unangst wrote: >> Now soliciting diffs to change readwrite to a loop with two buffers >> that poll()s in all four directions. :) >> > > Would libevent also be an option? > > bernd Ignore that, it is already using poll, so need to change t

Re: increase netcat's buffer...

2014-06-10 Thread Bernte
On 10/06/14 17:45, Ted Unangst wrote: > Now soliciting diffs to change readwrite to a loop with two buffers > that poll()s in all four directions. :) > Would libevent also be an option? bernd

Re: increase netcat's buffer...

2014-06-10 Thread Ted Unangst
On Mon, Jun 09, 2014 at 21:54, Theo de Raadt wrote: >> > A better patch is probably the following which also increases the size >> > of the buffer to at least 64k: >> >> Agreed. > > One thing to be aware of. That function is syncronous. It will read > as much as it can get, then it will do an "a

Re: increase netcat's buffer...

2014-06-09 Thread Theo de Raadt
> > A better patch is probably the following which also increases the size > > of the buffer to at least 64k: > > Agreed. One thing to be aware of. That function is syncronous. It will read as much as it can get, then it will do an "atomic" write operation to flush the buffer out the other way.

Re: increase netcat's buffer...

2014-06-09 Thread Ted Unangst
On Mon, Jun 09, 2014 at 20:12, John-Mark Gurney wrote: > A better patch is probably the following which also increases the size > of the buffer to at least 64k: Agreed.

increase netcat's buffer...

2014-06-09 Thread John-Mark Gurney
So, I was using nc (on FreeBSD) to image an HD over the network and it was consuming much cpu. It turns out that the buffer used by netcat is only 2k in size, though the buffer on the stack is 16k. This patch increased plan to use the entire buffer: --- netcat.obsd.c.orig 2014-05-19 18:25:23.000