Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-19 Thread David Higgs
On Feb 19, 2008 9:55 AM, chris rapier <[EMAIL PROTECTED]> wrote: > dhiggs wrote: > > If someone can split SSH into multiple threads, it should be just as > > possible to split it into multiple processes. However, I expect that > > most high-speed SSH traffic is SCP-/SFTP-based and therefore largel

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-19 Thread chris rapier
dhiggs wrote: If someone can split SSH into multiple threads, it should be just as possible to split it into multiple processes. However, I expect that most high-speed SSH traffic is SCP-/SFTP-based and therefore largely I/O bound, so it hasn't been high on anyone's requirements list. Based on

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-16 Thread Ted Unangst
On 2/16/08, Marco S Hyman <[EMAIL PROTECTED]> wrote: > librthread seems to be stagnating, waiting upon kernel work that > no one is either interested in or, if interested, has the time > to do. I'm not even sure there is consensus on what kernel > changes are needed. i worked on rthreads for as l

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-16 Thread Marco S Hyman
> Actually, this is the part of the discussion that interests me. Is > threading > a doomed hope on OpenBSD, a model of utilizing multiple cores which the > developers have zero interest supporting? Has the work on libc_r and the > like > been abandoned completely? libc_r is gone, rep

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-16 Thread David Higgs
On Feb 16, 2008 2:45 PM, Matthew Weigel <[EMAIL PROTECTED]> wrote: > Marco Peereboom wrote: > > The discussion on kernel threads is irrelevant. It is not about having > > some lower level support that will magically make threads not suck. > > Actually, this is the part of the discussion that inter

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-16 Thread chefren
On 2/16/08 8:48 PM, Matthew Weigel wrote: Marco Peereboom wrote: I can take this only so long before replying... I tried to walk away too. However... The discussion on kernel threads is irrelevant. It is not about having some lower level support that will magically make threads not suck.

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-14 Thread Ted Unangst
On Wed, Feb 13, 2008 at 5:23 PM, Matthew Weigel <[EMAIL PROTECTED]> wrote: > bofh wrote: > > > 2) lack of tuits to take on a major change for not really any major > > perceived gains: this is not an itch any of the developers currently have. > > ...except the developers (outside OpenBSD) who deve

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-14 Thread Marco Peereboom
I can take this only so long before replying... POSIX threads are a complete a total disaster. They are essentially non portable. Anyone who has written threaded code that has to be portable has at one point considered suicide. We are not talking about running on OpenBSD and Linux here; we are

Re: Removing One Giant Lock, was, Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-14 Thread scott
or o/s dev. brewhaahaa from a while back.) -Original Message- From: Geoff Steckel <[EMAIL PROTECTED]> To: scott <[EMAIL PROTECTED]> Cc: OpenBSD Subject: Removing One Giant Lock, was, Re: Multi-Threaded SSH/SCP made by university of Puttsburgh Date: Thu, 14 Feb 2008 03:18:33 -05

Removing One Giant Lock, was, Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-14 Thread Geoff Steckel
Maybe 20 years ago Alliant Computer successfully converted 4.4BSD to MP (8 core processors and 20? peripheral processors). They developed all the locking strategies for all of the I/O and scheduling. The locking was quite fine-grained and seemed to work quite well. It would be well worth while se

Engineering was Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-14 Thread Geoff Steckel
Matthew Weigel wrote: Geoff Steckel wrote: I'm sure you're extremely bright and can do it. It's not about me. If the OpenBSD developers *can't*, they should just drop any efforts to refine the big SMP lock, any effort to provide kernel threads, any effort to make libc reentrant. And if t

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-14 Thread scott
f obsd and dfbsd would make a whole lot of sense. -Original Message- From: Matthew Weigel <[EMAIL PROTECTED]> To: OpenBSD Subject: Re: Multi-Threaded SSH/SCP made by university of Puttsburgh Date: Thu, 14 Feb 2008 01:25:15 -0600 Mailer: Thunderbird 2.0.0.9 (Windows/20071031) Delivered-T

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-13 Thread Matthew Weigel
Matthew Weigel wrote: but b) splitting work across threads will have minimal performance value Hrrrm. "across multiple processes". Sorry for the mistake. -- Matthew Weigel hacker unique & idempot.ent

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-13 Thread Matthew Weigel
Geoff Steckel wrote: I'm sure you're extremely bright and can do it. It's not about me. If the OpenBSD developers *can't*, they should just drop any efforts to refine the big SMP lock, any effort to provide kernel threads, any effort to make libc reentrant. And if they do that, then a) th

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-13 Thread Geoff Steckel
Matthew Weigel wrote: This is the same model, albeit with better performance characteristics (assuming kernel threads), that Apache uses, and one of the great things about this model is that it's very easy to tightly control what memory threads share I'm sure you're extremely bright and can do

Program complexity control, was: Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-13 Thread Geoff Steckel
Tony Abernethy wrote: Geoff Steckel wrote:>> And yes, error recovery is a very significant part of any non-trivial useful program which does (for instance) network I/O, because the universe of possible errors is large. Error recover? Does anyone ever debug error recover? Is there any way anyone

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-13 Thread Matthew Weigel
Geoff Steckel wrote: This discussion of threads is isomorphic to: Why do CPUs have MMUs? Why don't we write "good guys' timeshared programs" which run cooperatively and noninterfering in a shared unprotected environment? Answer: it's too hard to write perfect programs.

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-13 Thread bofh
On Feb 13, 2008 10:05 PM, Matthew Weigel <[EMAIL PROTECTED]> wrote: > > Yet, somehow, these disinterested developers with no time managed to > support and assist the newcomer to create something useful and > acceptable. How is this any different? > No idea. Since I'm not a developer, I can be v

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-13 Thread Tony Abernethy
Geoff Steckel wrote: > This discussion of threads is isomorphic to: > > Why do CPUs have MMUs? > Why don't we write "good guys' timeshared programs" which run >cooperatively and noninterfering in a shared unprotected >environment? > > Answer: it's too hard to write per

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-13 Thread Darren Spiteri
Not that I've been constrained by CPU in the past few years, but one could easily script a parallel solution with: split -> for loop scp constrain by cores -> cat. A lot less trouble.

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-13 Thread Matthew Weigel
Geoff Steckel wrote: IMnsHO, threads should never be used unless absolutely necessary. They are very bad software practice: they share data and resources in uncontrolled ways It's controlled by the code you write. they encode state implicitly in the program counters of the threads Th

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-13 Thread Geoff Steckel
This discussion of threads is isomorphic to: Why do CPUs have MMUs? Why don't we write "good guys' timeshared programs" which run cooperatively and noninterfering in a shared unprotected environment? Answer: it's too hard to write perfect programs. Compartmentalized se

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-13 Thread Geoff Steckel
bofh wrote: On Feb 13, 2008 6:59 PM, <[EMAIL PROTECTED]> wrote: I'm no coding guru but I know basics about race conditions. what I don't know is why other OSs don't have these problems or how these things can I think the basic issue boils down to the following: 1) lack of complete thread s

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-13 Thread Matthew Weigel
bofh wrote: Which comes back to - it's not an itch any one of the developers have, and they don't have the tuits to do it. Funny, seems like a decently large number of drivers (yep, running in the kernel, which means they need to be taken seriously) have been developed by people who, at the

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-13 Thread bofh
On Feb 13, 2008 8:23 PM, Matthew Weigel <[EMAIL PROTECTED]> wrote: > bofh wrote: > > > 2) lack of tuits to take on a major change for not really any major > > perceived gains: this is not an itch any of the developers currently > have. > > ...except the developers (outside OpenBSD) who developed

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-13 Thread Matthew Weigel
bofh wrote: 2) lack of tuits to take on a major change for not really any major perceived gains: this is not an itch any of the developers currently have. ...except the developers (outside OpenBSD) who developed the patch. -- Matthew Weigel hacker [EMAIL PROTECTED]

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-13 Thread bofh
On Feb 13, 2008 6:59 PM, <[EMAIL PROTECTED]> wrote: > I'm no coding guru but I know basics about race conditions. what I don't > know is why other OSs don't have these problems or how these things can > I think the basic issue boils down to the following: 1) lack of complete thread support righ

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-13 Thread sebastian . rother
> On 2/13/08 11:17 PM, Benjamin Bennett wrote: > >> I wasn't saying "we can work on security" afterwards. This is something >> that [to our knowledge] has not been worked on previously, and what >> we're providing is code that we consider experimental (due to lack of >> review) to get the ball rol

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-13 Thread Matthew Weigel
chefren wrote: The security problems OpenBSD people see are not in robustness of this particular program, how well it works, (without seeing I believe that's OK and the code is interesting). The security problems are, for example, that code running in one core can access data from the other co

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-13 Thread chefren
On 2/13/08 11:17 PM, Benjamin Bennett wrote: I wasn't saying "we can work on security" afterwards. This is something that [to our knowledge] has not been worked on previously, and what we're providing is code that we consider experimental (due to lack of review) to get the ball rolling and ge

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-13 Thread Benjamin Bennett
chefren wrote: > > > On 2/13/08 10:27 PM, Benjamin Bennett wrote: > >>> It is very unlikely that this patch will be integrated - it adds threads >>> to OpenSSH, which introduce many new security considerations. >> >> Using threads is really just a means to an end, and happened to be the >> most con

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-13 Thread chefren
On 2/13/08 10:27 PM, Benjamin Bennett wrote: It is very unlikely that this patch will be integrated - it adds threads to OpenSSH, which introduce many new security considerations. Using threads is really just a means to an end, and happened to be the most convenient means. If that's a show s

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-13 Thread Benjamin Bennett
Dries Schellekens wrote: [snip] A "none" cipher also should like a bad idea. Just use another file transfer protocol in that case. SSH is more than just encryption. The none switch gives the benefit of SSH authentication & MAC for data streams that do not require secrecy. --ben

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-13 Thread Benjamin Bennett
Damien Miller wrote: On Wed, 13 Feb 2008, [EMAIL PROTECTED] wrote: I just wanted to bring it to your attention that the university of Pittsburgh provides a HPC-Patch for OpenSSH 4.7 wich may is worth looking at (include it into the base if possible? who knows..). :) Is crypto really a bottlen

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-13 Thread Benjamin Bennett
[EMAIL PROTECTED] wrote: > I just wanted to bring it to your attention that the university of > Pittsburgh provides a HPC-Patch for OpenSSH 4.7 wich may is worth looking > at (include it into the base if possible? who knows..). :) fyi, this work is being done at the Pittsburgh Supercomputing Cente

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-13 Thread Geoff Steckel
Jussi Peltola wrote: On Wed, Feb 13, 2008 at 08:52:42AM -0500, Geoff Steckel wrote: A properly engineered solution would use separate processes and a good interprocess communication system. This is not a suggestion but a question to reduce my stupidity, but wouldn't standard unix pipes and for

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-13 Thread Jussi Peltola
On Wed, Feb 13, 2008 at 08:52:42AM -0500, Geoff Steckel wrote: > A properly engineered solution would use separate processes and a good > interprocess communication system. This is not a suggestion but a question to reduce my stupidity, but wouldn't standard unix pipes and forks/other child proce

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-13 Thread Geoff Steckel
[EMAIL PROTECTED] wrote: On Feb 13, 2008 1:11 PM, <[EMAIL PROTECTED]> wrote: It seams to outperform the normal SSH a lot and this speedup looks kinda impressiv btw. The speedup looks impressive for 1 connection (because different cores are computing AES-CTR), but what happens when multiple SC

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-13 Thread Henning Brauer
* [EMAIL PROTECTED] <[EMAIL PROTECTED]> [2008-02-13 14:09]: 1: > Threads could get secured too. 2: > A threaded OpenSSL would be an advantage for the userland crypto too 3: > I'm not skilled enough pick one. may I suggest option 3? -- Henning Brauer, [EMAIL PROTECTED], [EMAIL PROTECTED] BS Web

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-13 Thread sebastian . rother
> On Feb 13, 2008 1:11 PM, <[EMAIL PROTECTED]> wrote: > >> It seams to outperform the normal SSH a lot and this speedup looks kinda >> impressiv btw. > > The speedup looks impressive for 1 connection (because different cores > are computing AES-CTR), but what happens when multiple SCP/SFTP > conne

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-13 Thread sebastian . rother
> On Wed, 13 Feb 2008, [EMAIL PROTECTED] wrote: > >> I just wanted to bring it to your attention that the university of >> Pittsburgh provides a HPC-Patch for OpenSSH 4.7 wich may is worth >> looking >> at (include it into the base if possible? who knows..). :) > > Is crypto really a bottleneck for

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-13 Thread Dries Schellekens
On Feb 13, 2008 1:11 PM, <[EMAIL PROTECTED]> wrote: > It seams to outperform the normal SSH a lot and this speedup looks kinda > impressiv btw. The speedup looks impressive for 1 connection (because different cores are computing AES-CTR), but what happens when multiple SCP/SFTP connections are b

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-13 Thread Marco Peereboom
Yay race conditions!! I'll venture to say that this will not go in... On Wed, Feb 13, 2008 at 01:11:50PM +0100, [EMAIL PROTECTED] wrote: > I just wanted to bring it to your attention that the university of > Pittsburgh provides a HPC-Patch for OpenSSH 4.7 wich may is worth looking > at (include i

Re: Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-13 Thread Damien Miller
On Wed, 13 Feb 2008, [EMAIL PROTECTED] wrote: > I just wanted to bring it to your attention that the university of > Pittsburgh provides a HPC-Patch for OpenSSH 4.7 wich may is worth looking > at (include it into the base if possible? who knows..). :) Is crypto really a bottleneck for non-HPC use

Multi-Threaded SSH/SCP made by university of Puttsburgh

2008-02-13 Thread sebastian . rother
I just wanted to bring it to your attention that the university of Pittsburgh provides a HPC-Patch for OpenSSH 4.7 wich may is worth looking at (include it into the base if possible? who knows..). :) It seams to outperform the normal SSH a lot and this speedup looks kinda impressiv btw. http://ww