tcp/ip rewrite for summer of code

[Joshua Stratton]

Hello,

I recently entered a proposal for the TCP/IP rewrite for Google's Summer of
Code.  I would like to invite anyone who has access as a mentor to check it
out.  For those of you who don't, I put the details of the proposal, my
planned schedule, my experience, and my interests on an external website,
that anyone can see.  I would love to hear some feedback on how I could make
my proposal a better project.

http://www.cs.utah.edu/~jstratto/soc_hurd/

Basically, it follows the proposed idea of the TCP/IP stack broken into two
separate entities.  I think it would be useful to also implement the UDP
protocol.

Thanks,
Josh

Re: tcp/ip rewrite for summer of code

[Joshua Stratton]

>
>
> As far as security and efficiency is concerned, I highly recommend
> reading 'Network Subsystems Reloaded: A High-Performance, Defensible
> Network Subsystem'[1].
>
>  1. 
> http://www.cs.jhu.edu/~sarat/usenix-net-2004.ps
>
> Having a capability-oriented network stack would be great, BTW.


Thanks, Pierre.  That was an interesting paper.  I would love to implement
this kind of implementation for Google SoC, even though it's a bit more
complicated than I imagined.  They seemed to get some great benefits for
only minor drops in peak bandwidth.  Their actual implementation doesn't
seem unfeasible.  The idea of having clients handle resources seems very
different to me, but provide some incredible advantages.  I think it would
need some additional helper library could encapsulate a bit of this on the
client side if the developer chose to.  Would this type of implementation be
something you guys would want to see as a Google Summer of Code?

Thanks,
Josh


>
> Quickly,
> Pierre
> --
> [EMAIL PROTECTED]
> OpenPGP 0xD9D50D8A
>
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.6 (GNU/Linux)
>
> iD8DBQFH6q7Pxe13INnVDYoRAjAXAKDnHiuXTlAIguilg7fs/b6Os0BokQCfTcTQ
> xOQ4ujb355LholWcOvBBw4Q=
> =5MAz
> -END PGP SIGNATURE-
>
>

Re: tcp/ip rewrite for summer of code

[Joshua Stratton]

I've rewritten my proposal based on some feedback from the mailing list.
Once again, more about the proposal and myself can be found off Google's
website at:

http://www.cs.utah.edu/~jstratto/soc_hurd/

I really liked the paper Pierre referenced and think a modular approach with
client-based memory management would provide a great network stack for Hurd
being fault-tolerant, secure, and modular with negligible performance
penalty.  If those interested in the network stack could read my new
proposal, I'd like to get some additional feedback as to how I could refine
this idea.

Josh

On Wed, Mar 26, 2008 at 9:05 PM, Joshua Stratton <[EMAIL PROTECTED]>
wrote:

>
> > As far as security and efficiency is concerned, I highly recommend
> > reading 'Network Subsystems Reloaded: A High-Performance, Defensible
> > Network Subsystem'[1].
> >
> >  1. 
> > http://www.cs.jhu.edu/~sarat/usenix-net-2004.ps<http://www.cs.jhu.edu/%7Esarat/usenix-net-2004.ps>
> >
> > Having a capability-oriented network stack would be great, BTW.
>
>
> Thanks, Pierre.  That was an interesting paper.  I would love to implement
> this kind of implementation for Google SoC, even though it's a bit more
> complicated than I imagined.  They seemed to get some great benefits for
> only minor drops in peak bandwidth.  Their actual implementation doesn't
> seem unfeasible.  The idea of having clients handle resources seems very
> different to me, but provide some incredible advantages.  I think it would
> need some additional helper library could encapsulate a bit of this on the
> client side if the developer chose to.  Would this type of implementation be
> something you guys would want to see as a Google Summer of Code?
>
> Thanks,
> Josh
>
>
> >
> > Quickly,
> > Pierre
> > --
> > [EMAIL PROTECTED]
> > OpenPGP 0xD9D50D8A
> >
> > -BEGIN PGP SIGNATURE-
> > Version: GnuPG v1.4.6 (GNU/Linux)
> >
> > iD8DBQFH6q7Pxe13INnVDYoRAjAXAKDnHiuXTlAIguilg7fs/b6Os0BokQCfTcTQ
> > xOQ4ujb355LholWcOvBBw4Q=
> > =5MAz
> > -END PGP SIGNATURE-
> >
> >
>

Re: tcp/ip rewrite for summer of code

[Joshua Stratton]

I'd still like some feedback from the Hurd developers about what they would
like to see in the TCP/IP rewrite.  From what I envision, it would be
modular design of two or more translators (perhaps one for each protocol).
Although it's quite a different approach, I really like the idea of
client-based responsibility for the memory buffers and other data
structures.  I would personally like to know who works most with the
networking stack now and what they would like to see in a TCP/IP rewrite.
If you haven't read the paper Pierre posted, it's a very interesting model
and I would like to know what those developers involved in the network stack
like and don't like about it (
http://www.cs.jhu.edu/~sarat/usenix-net-2004.ps<http://www.cs.jhu.edu/%7Esarat/usenix-net-2004.ps>).
It provides an architecture that is very secure and stable.  I also think
the modularity of the individual protocols would be a great fit for Hurd.

Josh
http://www.cs.utah.edu/~jstratto/soc_hurd/<http://www.cs.utah.edu/%7Ejstratto/soc_hurd/>

On Thu, Mar 27, 2008 at 5:23 AM, Joshua Stratton <[EMAIL PROTECTED]>
wrote:

> I've rewritten my proposal based on some feedback from the mailing list.
> Once again, more about the proposal and myself can be found off Google's
> website at:
>
> http://www.cs.utah.edu/~jstratto/soc_hurd/<http://www.cs.utah.edu/%7Ejstratto/soc_hurd/>
>
> I really liked the paper Pierre referenced and think a modular approach
> with client-based memory management would provide a great network stack for
> Hurd being fault-tolerant, secure, and modular with negligible performance
> penalty.  If those interested in the network stack could read my new
> proposal, I'd like to get some additional feedback as to how I could refine
> this idea.
>
> Josh
>
>
> On Wed, Mar 26, 2008 at 9:05 PM, Joshua Stratton <[EMAIL PROTECTED]>
> wrote:
>
> >
> > > As far as security and efficiency is concerned, I highly recommend
> > > reading 'Network Subsystems Reloaded: A High-Performance, Defensible
> > > Network Subsystem'[1].
> > >
> > >  1. 
> > > http://www.cs.jhu.edu/~sarat/usenix-net-2004.ps<http://www.cs.jhu.edu/%7Esarat/usenix-net-2004.ps>
> > >
> > > Having a capability-oriented network stack would be great, BTW.
> >
> >
> > Thanks, Pierre.  That was an interesting paper.  I would love to
> > implement this kind of implementation for Google SoC, even though it's a bit
> > more complicated than I imagined.  They seemed to get some great benefits
> > for only minor drops in peak bandwidth.  Their actual implementation doesn't
> > seem unfeasible.  The idea of having clients handle resources seems very
> > different to me, but provide some incredible advantages.  I think it would
> > need some additional helper library could encapsulate a bit of this on the
> > client side if the developer chose to.  Would this type of implementation be
> > something you guys would want to see as a Google Summer of Code?
> >
> > Thanks,
> > Josh
> >
> >
> > >
> > > Quickly,
> > > Pierre
> > > --
> > > [EMAIL PROTECTED]
> > > OpenPGP 0xD9D50D8A
> > >
> > > -BEGIN PGP SIGNATURE-
> > > Version: GnuPG v1.4.6 (GNU/Linux)
> > >
> > > iD8DBQFH6q7Pxe13INnVDYoRAjAXAKDnHiuXTlAIguilg7fs/b6Os0BokQCfTcTQ
> > > xOQ4ujb355LholWcOvBBw4Q=
> > > =5MAz
> > > -END PGP SIGNATURE-
> > >
> > >
> >
>

updated proposal

[Joshua Stratton]

Olaf made some comments on my proposal and wanted to know a bit more about
my actual implementation in the Hurd itself.  I've done added a bit more to
the proposal to explain what I feel is a good implementation.  Basically, I
was thinking the network stack could be divided into different translators
per protocol and give the client access to different layers based on his
needs.

A network interface that registers an IP address would be listed with the
others interfaces with each having a respective hierarchy of transport
protocols underneath.

For example,

/ip/eth0/tcp/
/ip/eth0/udp/
/ip/eth1/tcp/
/ip/eth1/udp/
/ip/lo/tcp/
/ip/lo/udp/
/ip/tcp/
/ip/udp/

In this example, the client could choose from the first six options to get
the interface of its choice.  The last two could would let the network stack
decide which network interface provided the connection.  In this way the
client could request a link for a TCP connection, for example, for eth0
using /ip/eth0/tcp/ or might not care and use /ip/tcp/ and let the server
decide using any heuristic it wants (round-robin, etc.)

In this way, it would be very versatile for providing different routing
schemes and filtering options to the client.

Thanks,
Josh

Re: updated proposal

[Joshua Stratton]

I thought a directory structure might be a more intuitive interface.  It
doesn't matter too much to me, as long as it stays intuitive down the road.
I guess since it's really only going to implement two layers of the OSI
model, it doesn't matter.  A list might be more accessible.

Thanks for the feedback.

Josh

On Sat, Mar 29, 2008 at 10:04 AM, Carl Fredrik Hammar <[EMAIL PROTECTED]>
wrote:

> Hi,
>
> > Olaf made some comments on my proposal and wanted to know a bit more
> about
> > my actual implementation in the Hurd itself.  I've done added a bit more
> > to the proposal to explain what I feel is a good implementation.
> > Basically, I was thinking the network stack could be divided into
> > different translators per protocol and give the client access to
> different
> > layers based on his needs.
>
> Yes, this is roughly how a hurdish network stack has been envisioned
> in the past.
>
> > A network interface that registers an IP address would be listed with
> the
> > others interfaces with each having a respective hierarchy of transport
> > protocols underneath.
> >
> > For example,
> >
> > /ip/eth0/tcp/
> > /ip/eth0/udp/
> > /ip/eth1/tcp/
> > /ip/eth1/udp/
> > /ip/lo/tcp/
> > /ip/lo/udp/
> > /ip/tcp/
> > /ip/udp/
> >
> > In this example, the client could choose from the first six options to
> get
> > the interface of its choice.  The last two could would let the network
> > stack decide which network interface provided the connection.  In this
> way
> > the client could request a link for a TCP connection, for example, for
> > eth0 using /ip/eth0/tcp/ or might not care and use /ip/tcp/ and let the
> > server decide using any heuristic it wants (round-robin, etc.)
>
> Shouldn't it be /eth0/ip/tcp/?  I.e. with internet protocol is layered
> over ethernet.  Though it might be that I have misunderstood your
> example or the protocol stack in general (this is not my area of
> expertise).
>
> In any case, I'm not sure why you have chosen directories.  Why not
> just: eth0, eth1, ip0, ip1, tcp0, tcp1, tcp0+1 etc. where tcp0+1
> works like your /ip/tcp/?
>
> Regards,
>   Fredrik
>

Re: Hurdish TCP stack (was: updated proposal)

[Joshua Stratton]

Hey,

I did some reading up on the Plan9 design for their network hierarchy.  I
think it's interesting.  I wouldn't mind using it just so the layout would
be more commonplace (for those who may have used Plan9).  I also like the
access to the interface statistics.  Plan9, from what I've read, tries to
abstract the interfaces to look the same to the client, which seems a little
abstract to me if a client is going to control the data structures.  It
might be the opposite of what the client wants.  IMHO, I think the client
should be smart enough to choose the interface it wants as I don't think
that's a real hurdle for developers.

I heard the deadline was extended by a week by someone on the GIMP developer
list and that it would be updated on Google's timeline for SoC.  However, I
have not seen this change so I am assuming today is in fact the last day.  I
am going to try to organize my last few ideas for the project layout today
before the closing period, but they'll be just ideas since it seems several
of the Hurd developers have provided a lot of different ideas for the actual
file structure they would like to see.  I do like the Plan9 approach, but
I'm not sure if a "transparent" interface is really what the Hurd group
would want.

Thanks,
Josh

On Sun, Mar 30, 2008 at 4:51 PM, <[EMAIL PROTECTED]> wrote:

> Hi,
>
> On Sat, Mar 29, 2008 at 05:04:48PM +0100, Carl Fredrik Hammar wrote:
>
> > > /ip/eth0/tcp/ /ip/eth0/udp/ /ip/eth1/tcp/ /ip/eth1/udp/ /ip/lo/tcp/
> > > /ip/lo/udp/ /ip/tcp/ /ip/udp/
> [...]
> > Shouldn't it be /eth0/ip/tcp/?  I.e. with internet protocol is layered
> > over ethernet.
>
> Indeed, the more I think about it, the more I get convinced that you are
> right :-)
>
> There are some things to keep in mind though. When /ip is the top level,
> it can naturally serve as a master translator/multiplexer, managing the
> other nodes. With the device directories at the top, the question arises
> how to handle them...
>
> The simplest (but least elegant) approach would be simply to create
> static directories for each available device; and whenever an IP
> translator is attached, tell it the device explicetily. (settrans
> /net/eth0/ip /hurd/ip -d eth0)
>
> Another approach would be to have a rather dumb device translator, which
> tells the attached nodes what device to use, so it needn't be passed
> explicitely.
>
> Ideally, the device translator would actually sit between the kernel
> device and the clients, managing access... But I wouldn't consider that
> part of the IP stack task. (Actually someone was working on a BPF
> translator in the past, but it wasn't finished :-( )
>
> A second question is how to handle listening on all available
> interfaces. (I think the socket interface allows for that?) A top-level
> ip translator could manage that in a natural way; with the devices at
> the top level, we need some additional facility to keep track of all
> available interfaces. It shouldn't be hard, but needs to be taken into
> account.
>
> > In any case, I'm not sure why you have chosen directories.  Why not
> > just: eth0, eth1, ip0, ip1, tcp0, tcp1, tcp0+1 etc. where tcp0+1 works
> > like your /ip/tcp/?
>
> First of all, I hope you didn't mean to imply that the numbers in ip0
> and tcp0 correspond to that in eth0... Because that would be plain wrong
> :-) lo for example has no number at all; while we could have both eth0
> and ppp0 for example. So, if we want to enumerate the ip interfaces like
> that, the enumeration will be completely independant of the device
> names.
>
> But is it really useful to have such an enumeration at all? I think the
> socket interface doesn't use anything like that; interfaces are selected
> either by device name or by IP... (Correct me if I'm wrong.) And I don't
> see any reason to do different with the filesystem-based interface.
>
> The '+' syntax for listening on multiple devices looks interesting. I
> wonder though how useful it is. Does the socket interface provide a
> possibility for listening on multiple but not all devices? If not, is is
> useful to add such an ability?...
>
> In any case, we still need a way to specify the '*' case without listing
> all individual devices, I think.
>
> As for flattening the hierarchy, I'm very sceptical about that. True, it
> would simplify things: You could just have all the nodes in a normal
> directory for example, statically set up with passive translators;
> everything could be done with simple single-node translators.
>
> This is pretty ugly, though. It means that all the relations always have
> to be specified explicitely, e.g. "settrans /net/tcp0 /hurd/tcp
> /net/ip0". Also, it means a totally static setup.
>
> Instead of a normal directory, some kind of translator (multiplexer)
> could be used to manage all the nodes in /net. It could be more dynamic,
> and manage the associations between the nodes automatically. That would
> be very intransparent though; and it would destroy much of the
> decentralization and f

hurd internet through qemu

[Joshua Stratton]

I have followed few pages of setting up the Hurd's network through qemu,
however it seems that eth0 is never configured on boot so the following
command breaks because no eth0 device has been configured.

# settrans -afgp /servers/socket/2 /hurd/pfinet -i eth0 -a 10.0.2.15
-g 10.0.2.2 -m 255.255.255.0

Re: Hurdish TCP stack (was: updated proposal)

[Joshua Stratton]

If anyone hasn't read up on how Plan9 runs their network stack, they have a
separate directory of each connection.  An example in the paper is shown as
the following,

# cd /net/tcp/2   <--- this is like the second TCP connection
# ls -l
ctl
data
listen
local
remote
status

They use an interesting system to control their connections using ASCII
strings.  For example changing the packet size would be as simple as "2400 >
ctl" would change the packet size to 2400 (some syntax to that effect).
They say this approach makes it compatible with remote applications that can
manipulate servers through a common interface.  Supposedly all network
connections use this interface (TCP, UDP, LP).

I think this approach would fit nicely into the Hurd's translator
architecture.  However, I'm not sure if I like the directory structure they
use.  I would think the network interface should be shown like

/net/eth0/tcp/2

It might be worthwhile--but possible bad style?--to duplicate both
hierachies so on may browse the connections by device or generally.

Any preferences/comments on this?

Josh

On Mon, Mar 31, 2008 at 10:42 AM, Joshua Stratton <[EMAIL PROTECTED]>
wrote:

> Hey,
>
> I did some reading up on the Plan9 design for their network hierarchy.  I
> think it's interesting.  I wouldn't mind using it just so the layout would
> be more commonplace (for those who may have used Plan9).  I also like the
> access to the interface statistics.  Plan9, from what I've read, tries to
> abstract the interfaces to look the same to the client, which seems a little
> abstract to me if a client is going to control the data structures.  It
> might be the opposite of what the client wants.  IMHO, I think the client
> should be smart enough to choose the interface it wants as I don't think
> that's a real hurdle for developers.
>
> I heard the deadline was extended by a week by someone on the GIMP
> developer list and that it would be updated on Google's timeline for SoC.
> However, I have not seen this change so I am assuming today is in fact the
> last day.  I am going to try to organize my last few ideas for the project
> layout today before the closing period, but they'll be just ideas since it
> seems several of the Hurd developers have provided a lot of different ideas
> for the actual file structure they would like to see.  I do like the Plan9
> approach, but I'm not sure if a "transparent" interface is really what the
> Hurd group would want.
>
> Thanks,
> Josh
>
>
> On Sun, Mar 30, 2008 at 4:51 PM, <[EMAIL PROTECTED]> wrote:
>
> > Hi,
> >
> > On Sat, Mar 29, 2008 at 05:04:48PM +0100, Carl Fredrik Hammar wrote:
> >
> > > > /ip/eth0/tcp/ /ip/eth0/udp/ /ip/eth1/tcp/ /ip/eth1/udp/ /ip/lo/tcp/
> > > > /ip/lo/udp/ /ip/tcp/ /ip/udp/
> > [...]
> > > Shouldn't it be /eth0/ip/tcp/?  I.e. with internet protocol is layered
> > > over ethernet.
> >
> > Indeed, the more I think about it, the more I get convinced that you are
> > right :-)
> >
> > There are some things to keep in mind though. When /ip is the top level,
> > it can naturally serve as a master translator/multiplexer, managing the
> > other nodes. With the device directories at the top, the question arises
> > how to handle them...
> >
> > The simplest (but least elegant) approach would be simply to create
> > static directories for each available device; and whenever an IP
> > translator is attached, tell it the device explicetily. (settrans
> > /net/eth0/ip /hurd/ip -d eth0)
> >
> > Another approach would be to have a rather dumb device translator, which
> > tells the attached nodes what device to use, so it needn't be passed
> > explicitely.
> >
> > Ideally, the device translator would actually sit between the kernel
> > device and the clients, managing access... But I wouldn't consider that
> > part of the IP stack task. (Actually someone was working on a BPF
> > translator in the past, but it wasn't finished :-( )
> >
> > A second question is how to handle listening on all available
> > interfaces. (I think the socket interface allows for that?) A top-level
> > ip translator could manage that in a natural way; with the devices at
> > the top level, we need some additional facility to keep track of all
> > available interfaces. It shouldn't be hard, but needs to be taken into
> > account.
> >
> > > In any case, I'm not sure why you have chosen directories.  Why not
> > > just: eth0, eth1, ip0, ip1, tcp0, tcp1, tcp0+1 etc. where tcp0+1 works
> > > like your /ip/tcp/?
> >
> > First of all, 

client-side memory buffers

[Joshua Stratton]

I was on the irc channel talking about the feasibility using client-side
memory buffers for a new network stack.  Based on some feedback about
difficulties of implementing this in the Hurd, I thought I would ask anyone
if they thought this would be especially difficult--particularly Marcus and
Neal.

Thanks,
Josh

Re: Hurdish TCP stack

[Joshua Stratton]

> > Supposedly all network connections use this interface (TCP, UDP, LP).
>
> What is LP?...


LP is a special protocol written by the Plan9 team that is something in
between TCP and UPD.  They wanted a reliable stream but as fast as UDP and a
few other benefits.  They said  TCP was too slow.  I'm not sure if it's
justified, and I wasn't planning on implementing it.  I could since
according to them it's about half the code length as the TCP protocol and a
lot simpler.  They developed it specifically for remote commands from what
I've read.  Like I said, I'm not sure if it's really justified and not sure
if anyone else uses it.


> > However, I'm not sure if I like the directory structure they use.  I
> > would think the network interface should be shown like
> >
> > /net/eth0/tcp/2
> >
> > It might be worthwhile--but possible bad style?--to duplicate both
> > hierachies so on may browse the connections by device or generally.
>
> This sounds perfectly reasonable. Don't see any bad style in there :-)


Great.  I didn't think it was a bad idea.  I would let whoever browse
through either hierarchy.


>
> -antrik-
>
> PS. http://learn.to/quote :-)
>
>
>

Re: client-side memory buffers

[Joshua Stratton]

On Mon, Mar 31, 2008 at 9:23 PM, Joshua Stratton <[EMAIL PROTECTED]>
wrote:

> I was on the irc channel talking about the feasibility using client-side
> memory buffers for a new network stack.  Based on some feedback about
> difficulties of implementing this in the Hurd, I thought I would ask anyone
> if they thought this would be especially difficult--particularly Marcus and
> Neal.


Although I'm not sure, I've read Marcus implemented POSIX shared memory now,
so I figured that would be the connected memory buffer the server would
write to.  If I understand it correctly (and I would love to get a better
understanding of this), the network stack could make this memory block and
assign this to the client (or the client would have ownership).

I was considering a domain socket, but supposedly this works much faster.
Also, using shmctl the server could create resources assigned to the client
such that the network stack itself wouldn't be responsible for the memory
and it would be freed when the client died.

So is POSIX shared memory now functioning perfectly in the HURD?  And would
this mechanism function well for this client-side memory management for the
new TCP/IP stack?

Josh

Re: client-side memory buffers

[Joshua Stratton]

On Tue, Apr 1, 2008 at 2:28 AM, Neal H. Walfield <[EMAIL PROTECTED]> wrote:

> At Mon, 31 Mar 2008 21:23:41 -0600,
> Joshua Stratton wrote:
> >
> > I was on the irc channel talking about the feasibility using client-side
> > memory buffers for a new network stack.  Based on some feedback about
> > difficulties of implementing this in the Hurd, I thought I would ask
> anyone
> > if they thought this would be especially difficult--particularly Marcus
> and
> > Neal.
>
> The way to do client side allocations is to have the client pass a
> memory object to the server.  There is a problem with this approach,
> however, as it allows the client to interfere with the server's
> operations.


>
> The problem is exactly the same as that with L4's data spaces.  When
> the server maps and accesses the memory object, the client can revoke
> the mapping at any time (via memory_object_lock_request), causing the
> server to fault.  If you manage to unmap the memory while the server
> is blocked on it (waiting for it to be paged in) and has a lock,
> you've successfully created a denial of service.


Okay, so it's a bad idea, for example, to juggle ownership of the memory
object so the client cannot unmap while the server is operating on it?


>
> Without changing the Mach interfaces, the best approach is to have the
> server allocate the memory and to transfer it to the client using the
> normal Mach buffer passing semantics.


Great.  I'll look into that.


>
> Neal
>
>

Re: Hurdish TCP stack (was: updated proposal)

[Joshua Stratton]

>
>
> > I think this approach would fit nicely into the Hurd's translator
> > architecture.  However, I'm not sure if I like the directory structure
> they
> > use.  I would think the network interface should be shown like
> >
> > /net/eth0/tcp/2
> >
> > It might be worthwhile--but possible bad style?--to duplicate both
> > hierachies so on may browse the connections by device or generally.
> >
> > Any preferences/comments on this?
>
> It's clearly a mistake to map the directory tree to the protocols stack.
> The TCP implementation is a global layer, it handles network interfaces
> internally and must not be bound to any interface (ask yourself how to
> implement INADDR_ANY, or IPv4 capable IPv6 sockets). In addition,
> separating the network and transport layers implies several problems.
> The most obvious one concerns performance. The Mach IPC subsystem
> provides nice virtual copy support, but this facility creates an
> important overhead that severely impacts high speed connections.
> Carefully shared memory between the servers may help though.


I'm not sure what you mean  that it's a mistake to map the directory tree to
the protocol stack.  Do you think the Plan9 implementation, for example, is
also clearly a mistake then?  I'm not sure if I see the problem.  Wouldn't
something like INADDR_ANY be a simple hook into the IP layer?  From what I
understand, a bind on INADDR_ANY locks that port for all interfaces anyway
so I don't see why this would be a problem.  Just have a special IP binding
for getting every interface and that connection is shown in every interface
directory.  Is the problem here with my implementation or a fault you
generally see in the Plan9 implementation?

Thanks,
Josh

Re: client-side memory buffers

[Joshua Stratton]

The problem you described was the client owning the memory object, sending
it to the server, and the server having the ability to unmap the memory
because it has ownership, if I understand correctly.  I assumed that a lock
was built into the system to prevent this, but I was wondering if this
weren't the case, the client could give the ownership to the server before
the server does any operations so the client could not unmap the memory
object.  The server would then give the ownership back to the client after
the operation is complete such that the client couldn't unmap the memory
while the server is using it, and in the default state the client would have
the responsibility of the memory block (which would help the denial of
service inside the network stack).

Josh

On Tue, Apr 1, 2008 at 9:51 AM, Neal H. Walfield <[EMAIL PROTECTED]> wrote:

> At Tue, 1 Apr 2008 08:11:30 -0600,
> Joshua Stratton wrote:
> > On Tue, Apr 1, 2008 at 2:28 AM, Neal H. Walfield <[EMAIL PROTECTED]>
> wrote:
> > > The problem is exactly the same as that with L4's data spaces.  When
> > > the server maps and accesses the memory object, the client can revoke
> > > the mapping at any time (via memory_object_lock_request), causing the
> > > server to fault.  If you manage to unmap the memory while the server
> > > is blocked on it (waiting for it to be paged in) and has a lock,
> > > you've successfully created a denial of service.
> >
> >
> > Okay, so it's a bad idea, for example, to juggle ownership of the memory
> > object so the client cannot unmap while the server is operating on it?
>
> I don't understand your example.
>
> Neal
>

Re: Hurdish TCP stack (was: updated proposal)

[Joshua Stratton]

Okay, I understand.  I think part of this goes back to a question on the
reason for enumeration on these sockets (tcp0, tcp1, etc.) if they aren't
used directly in the socket interface for the developer.  I assumed it was
as a convenience for other programs that were monitoring the network.  I
agree with you that the Plan9 scheme (ex. /dev/net/tcp/123) is the proper
way to interface with the network.  With two separate hierarchies, the other
one (ex. /dev/eth0/tcp/2) would be a convenience (possibly not a useful one,
but it seemed some liked the idea).

Thanks,
Josh

On Tue, Apr 1, 2008 at 9:20 AM, Richard Braun <[EMAIL PROTECTED]> wrote:

> On Tue, Apr 01, 2008 at 08:07:23AM -0600, Joshua Stratton wrote:
> > > It's clearly a mistake to map the directory tree to the protocols
> stack.
> > > The TCP implementation is a global layer, it handles network
> interfaces
> > > internally and must not be bound to any interface (ask yourself how to
> > > implement INADDR_ANY, or IPv4 capable IPv6 sockets). In addition,
> > > separating the network and transport layers implies several problems.
> > > The most obvious one concerns performance. The Mach IPC subsystem
> > > provides nice virtual copy support, but this facility creates an
> > > important overhead that severely impacts high speed connections.
> > > Carefully shared memory between the servers may help though.
> >
> >
> > I'm not sure what you mean  that it's a mistake to map the directory
> tree to
> > the protocol stack.  Do you think the Plan9 implementation, for example,
> is
> > also clearly a mistake then?  I'm not sure if I see the problem.
>  Wouldn't
> > something like INADDR_ANY be a simple hook into the IP layer?  From what
> I
> > understand, a bind on INADDR_ANY locks that port for all interfaces
> anyway
> > so I don't see why this would be a problem.  Just have a special IP
> binding
> > for getting every interface and that connection is shown in every
> interface
> > directory.  Is the problem here with my implementation or a fault you
> > generally see in the Plan9 implementation?
>
> /dev/net/tcp/123 is OK. /dev/eth0/tcp/321 isn't. As I said, you must
> consider TCP as a global layer, otherwise you will run into name space
> conflict problems. For example, a socket in listen mode on INADDR_ANY
> is bound to no interface, or all interfaces (at your choice). In the
> second case, you should have a unique name for this socket. Having
> multiple names like /dev/net/eth0/tcp/1 and /dev/net/eth1/tcp/2 makes
> no sense and there are more steps to identify those objects and be sure
> they are the same. Making sure the name is unique, i.e. using 123 in all
> interface directories is rather complicated for almost no benefit.
> Using Plan9 naming scheme, you avoid those problems.
>
> You could have /dev/net/tcp/123/if to obtain the underlying interface of
> a socket, using a special string like "any" for unbound sockets.
>
> --
> Richard Braun
>
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.1 (GNU/Linux)
>
> iD8DBQFH8lLWBlWsEPLYRi8RAgcuAJ4s1iOh3s3FHSXq5HxpPn8i3JKI2ACfTfhS
> x9zyMT27Q2lEuTD2QPg85+U=
> =X7/K
> -END PGP SIGNATURE-
>
>

Re: client-side memory buffers

[Joshua Stratton]

On Tue, Apr 1, 2008 at 12:32 PM, Pierre THIERRY <
[EMAIL PROTECTED]> wrote:

> Scribit Joshua Stratton dies 01/04/2008 hora 10:48:
> > The problem you described was the client owning the memory object,
> > sending it to the server, and the server having the ability to unmap
> > the memory because it has ownership, if I understand correctly.
>
> The problem is that the *client* can unmap.


Right.  That was a typo.  Sorry.


>
>
> Quickly,
> Pierre
> --
> [EMAIL PROTECTED]
> OpenPGP 0xD9D50D8A
>
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.6 (GNU/Linux)
>
> iD8DBQFH8n/Wxe13INnVDYoRAkomAJ0TMlyu/rvSjD4bT/sA8Bn498aJkwCgvQ58
> yJREB9jk9haLb56l/rZx/ek=
> =rLwT
> -END PGP SIGNATURE-
>
>

Re: client-side memory buffers

[Joshua Stratton]

On Tue, Apr 1, 2008 at 3:50 PM, Neal H. Walfield <[EMAIL PROTECTED]> wrote:

> Please don't top post.
>
> At Tue, 1 Apr 2008 10:48:02 -0600,
> Joshua Stratton wrote:
> >
> > The problem you described was the client owning the memory object,
> sending
> > it to the server, and the server having the ability to unmap the memory
> > because it has ownership, if I understand correctly.
>
> No.  The client has the ability to DoS the server because it manages
> the memory object.


What exactly is the difference between manages and owns?


>
>
> >  I assumed that a lock
> > was built into the system to prevent this, but I was wondering if this
> > weren't the case, the client could give the ownership to the server
> before
> > the server does any operations so the client could not unmap the memory
> > object.  The server would then give the ownership back to the client
> after
> > the operation is complete such that the client couldn't unmap the memory
> > while the server is using it, and in the default state the client would
> have
> > the responsibility of the memory block (which would help the denial of
> > service inside the network stack).
>
> If the server owns the memory, that means it is account to the
> server.  In which case, why not just let the server allocate it?


I realize to some extent this would just cause the same problems as the
server managing the memory, but my thought was it would reduce the amount of
time the server is responsible for the data.  Do you think the client-side
memory model is worthwhile?  And would the server allocating the memory
passing it to the client using the Mach semantics allow this client-side
memory model while avoiding the ability for clients to unmap the data?

Josh


>
>
> Neal
>

Re: client-side memory buffers

[Joshua Stratton]

On Wed, Apr 2, 2008 at 12:25 AM, Neal H. Walfield <[EMAIL PROTECTED]> wrote:

> At Tue, 1 Apr 2008 18:01:25 -0600,
> Joshua Stratton wrote:
> > On Tue, Apr 1, 2008 at 3:50 PM, Neal H. Walfield <[EMAIL PROTECTED]>
> wrote:
> >
> > > Please don't top post.
> > >
> > > At Tue, 1 Apr 2008 10:48:02 -0600,
> > > Joshua Stratton wrote:
> > > >
> > > > The problem you described was the client owning the memory object,
> > > sending
> > > > it to the server, and the server having the ability to unmap the
> memory
> > > > because it has ownership, if I understand correctly.
> > >
> > > No.  The client has the ability to DoS the server because it manages
> > > the memory object.
> >
> >
> > What exactly is the difference between manages and owns?
>
> I was using ownership as a synonym for accounted to, and manage as a
> synonym for being able to control (e.g., schedule).  So if the server
> is accounted the memory but the client can control the memory, then
> the server is susceptible to destructive interference from the client.
>
> > Do you think the client-side
> > memory model is worthwhile?  And would the server allocating the memory
> > passing it to the client using the Mach semantics allow this client-side
> > memory model while avoiding the ability for clients to unmap the
> > data?
>
> Yes, I think such accounting is worthwhile, it is what I am doing with
> Viengoos, however, I question the ability to realize it using Mach's
> interfaces.


What's Viengoos?  Is that a new microkernel?


>
>
> Neal
>

Re: client-side memory buffers

[Joshua Stratton]

>
> > Do you think the client-side
> > memory model is worthwhile?  And would the server allocating the memory
> > passing it to the client using the Mach semantics allow this client-side
> > memory model while avoiding the ability for clients to unmap the
> > data?
>
> Yes, I think such accounting is worthwhile, it is what I am doing with
> Viengoos, however, I question the ability to realize it using Mach's
> interfaces.
>
> Neal
>


Hey, I've rewritten my SoC proposal again.  It eliminates the EROS model
completely.  From the feedback I've received from several people, many
things such as the client-side memory management is too difficult (if not
impossible) in Mach.

The proposal now focuses on a Plan9-style interface for the TCP/IP network
stack.  This would provide a file-system hierarchy for browsing.  The
implementation would also provide an ASCII interface similar to Plan9's
where the information is easily accessible to the console and the user can
specify commands piped directly to the translator with no need for external
libraries.

Josh

Re: hurd internet through qemu

[Joshua Stratton]

Does anyone using the Hurd through qemu find the networking a little flaky?
I had it working yesterday using scripts and using the same scripts today
don't provide network access.

I start qemu with:

qemu debian-hurd-k16-qemu.img -net nic,model=ne2k_isa

and start the network with:

settrans -afgp /servers/socket/2 /hurd/pfinet -i eth0 -a 10.0.2.15 -g
10.0.2.2 -m 255.255.255.0


I have 10.0.2.3 in my /etc/resolve.conf file.

Thanks,
Josh

On Tue, Apr 1, 2008 at 3:20 AM, Thomas Schwinge <[EMAIL PROTECTED]> wrote:

> Hello!
>
> Please don't top-post, see 
> <http://bddebian.com/~wiki/mailing_lists/<http://bddebian.com/%7Ewiki/mailing_lists/>
> >.
> Please keep discussions on public mailing lists, see
> <http://bddebian.com/~wiki/community/communication/<http://bddebian.com/%7Ewiki/community/communication/>
> >.
>
>
> On Mon, Mar 31, 2008 at 03:27:28PM -0600, Joshua Stratton wrote:
> > On Mon, Mar 31, 2008 at 3:18 PM, Thomas Schwinge <[EMAIL PROTECTED]>
> wrote:
> > > On Mon, Mar 31, 2008 at 12:19:47PM -0600, Joshua Stratton wrote:
> > > > I have followed few pages of setting up the Hurd's network through
> qemu,
> > > > however it seems that eth0 is never configured on boot so the
> following
> > > > command breaks because no eth0 device has been configured.
> > >
> > > Strange.  It used to work out of the box without doing anything
> special,
> > > given that...
> > >
> > >$ qemu --help
> > >[...]
> > >-net none   use it alone to have zero network devices; if no
> -net
> > > option
> > >is provided, the default is '-net nic -net user'
> >
> > Am I using a newer or older version of qemu that uses an unsupported
> NIC?
>
> Indeed I can confirm now that with the new version of Ubuntu I installed
> some weeks (read: months) ago, the standard QEMU-emulated `ne2k_pci'
> device doesn't work any longer with GNU Mach.
>
> Resorting to ``-net nic,model=ne2k_isa -net user'' does make it work
> again.  This is also the only device from the list at
> <http://fabrice.bellard.free.fr/qemu/qemu-doc.html#SEC10>, section
> ``Network options'' that actually does work.
>
> What have QEMU folks been doing to the NE2KPCI device that it doesn't
> work with GNU Mach any longer?
>
> Someone should put some hints onto the wiki page
> <http://www.bddebian.com/~wiki/hurd/running/qemu/<http://www.bddebian.com/%7Ewiki/hurd/running/qemu/>
> >.
>
>
> Regards,
>  Thomas
>
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.2.2 (GNU/Linux)
>
> iD8DBQFH8f5Hgfzh735dTTURAtOKAJ4n4rb85UcGfanhTT8lK03ffs26wACgjjNd
> QftgmsBIJJqi5Xw2T/JC8/Y=
> =jj4Z
> -END PGP SIGNATURE-
>
>

Re: hurd internet through qemu

[Joshua Stratton]

On Thu, May 1, 2008 at 12:07 AM, Shakthi Kannan <[EMAIL PROTECTED]> wrote:

> Hi,
>
> --- On Thu, May 1, 2008 at 2:59 AM, Joshua Stratton
> <[EMAIL PROTECTED]> wrote:
> | Does anyone using the Hurd through qemu find the networking a little
> flaky?
> \--
>
> No. I have tested the K16 with networking.


The networking was working, so I have also tested it on K16.  It just
doesn't work anymore and I don't understand why, since I don't think I've
changed anything recently.


>
>
> ---
> | qemu debian-hurd-k16-qemu.img -net nic,model=ne2k_isa
> \--
>
> Why do you need to specify -net nic,model? I have never passed any
> -net parameters when used with qemu.


Neither did tschwinge, but apparently Ubuntu uses a default NIC that the
Hurd doesn't detect.  tschwinge recommended that I specify the NIC, and it
was detected.


>
>
> ---
> | I have 10.0.2.3 in my /etc/resolve.conf file.
> \--
>
> /etc/resolv.conf.
>
> Regards,
>
> SK
>
> --
> Shakthi Kannan
> http://www.shakthimaan.com
>
>
>

Re: hurd internet through qemu

[Joshua Stratton]

On Thu, May 1, 2008 at 5:36 AM, Joshua Stratton <[EMAIL PROTECTED]>
wrote:

> On Thu, May 1, 2008 at 12:07 AM, Shakthi Kannan <[EMAIL PROTECTED]>
> wrote:
>
> > Hi,
> >
> > --- On Thu, May 1, 2008 at 2:59 AM, Joshua Stratton
> > <[EMAIL PROTECTED]> wrote:
> > | Does anyone using the Hurd through qemu find the networking a little
> > flaky?
> > \--
> >
> > No. I have tested the K16 with networking.
>
>
> The networking was working, so I have also tested it on K16.  It just
> doesn't work anymore and I don't understand why, since I don't think I've
> changed anything recently.
>
>
> >
> >
> > ---
> > | qemu debian-hurd-k16-qemu.img -net nic,model=ne2k_isa
> > \--
> >
> > Why do you need to specify -net nic,model? I have never passed any
> > -net parameters when used with qemu.
>
>
> Neither did tschwinge, but apparently Ubuntu uses a default NIC that the
> Hurd doesn't detect.  tschwinge recommended that I specify the NIC, and it
> was detected.
>

I spoke with youpi and andrei on IRC for a while and they gave me a few
suggestions to see what the problem was.  They concluded that it was
something to do with my qemu configuration on my system.  I run Ubuntu 7.10
and home and Ubuntu 7.10 AMD64 at work and had identical problems.
Switching to kvm, which I assume has different configuration settings fixed
the problem.

Thanks,
Josh