Re: [PATCH 2/7] libpager: fix assertion unsigned against 0

2022-01-19 Thread Samuel Thibault
Etienne Brateau, le mer. 19 janv. 2022 20:29:40 +0100, a ecrit: > Comparing an unsigned to be superior or equals to 0 is always true, > instead move the assert before the substraction and compare it to the > value to substract. Applied, thanks! > --- > libpager/pager-memcpy.c

Re: [PATCH 1/7] libpager: fix a comparision between signed and unsigned

2022-01-19 Thread Samuel Thibault
I have rather changed it to vm_size_t, which makes much more sense here. Etienne Brateau, le mer. 19 janv. 2022 20:29:39 +0100, a ecrit: > --- > libpager/lock-object.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/libpager/lock-object.c b/libpage

[PATCH 1/7] libpager: fix a comparision between signed and unsigned

2022-01-19 Thread Etienne Brateau
--- libpager/lock-object.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libpager/lock-object.c b/libpager/lock-object.c index c022d0c4..bb222ea6 100644 --- a/libpager/lock-object.c +++ b/libpager/lock-object.c @@ -30,7 +30,7 @@ _pager_lock_object (struct pager *p

[PATCH 2/7] libpager: fix assertion unsigned against 0

2022-01-19 Thread Etienne Brateau
Comparing an unsigned to be superior or equals to 0 is always true, instead move the assert before the substraction and compare it to the value to substract. --- libpager/pager-memcpy.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/libpager/pager-memcpy.c b/libpager

Re: [PATCH 1/4] libpager: pager_write_page () should not unmap page

2021-05-08 Thread Samuel Thibault
> doc/hurd.texi | 6 +++--- > libpager/pager.h| 6 +++--- > storeio/pager.c | 8 +++- > tmpfs/pager-stubs.c | 6 +++--- > 4 files changed, 12 insertions(+), 14 deletions(-) > > diff --git a/doc/hurd.texi b/doc/hurd.texi > index c8db86b5..239d82da 100644 >

[PATCH 3/4] libpager: Track which pages are precious

2021-05-08 Thread Sergey Bugaev
This introduces the PM_PRECIOUS bit, which gets set when a precious page is offered to the kernel using pager_offer_page () with the precious argument set, and cleared when the page is returned back. --- libpager/data-return.c | 3 ++- libpager/offer-page.c | 2 ++ libpager/priv.h| 1

[PATCH 2/4] libpager: Store omit_data in an array

2021-05-08 Thread Sergey Bugaev
tracking other information about the pages; so do the same for whether the data should be omited. --- libpager/data-return.c | 12 +++- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/libpager/data-return.c b/libpager/data-return.c index c0f5aaf7..db2f84e6 100644 --- a/libpager

[PATCH 1/4] libpager: pager_write_page () should not unmap page

2021-05-08 Thread Sergey Bugaev
Clarify this in the documentation, and fix the storeio implementation. --- doc/hurd.texi | 6 +++--- libpager/pager.h| 6 +++--- storeio/pager.c | 8 +++- tmpfs/pager-stubs.c | 6 +++--- 4 files changed, 12 insertions(+), 14 deletions(-) diff --git a/doc/hurd.texi b/doc

[PATCH 4/4] libpager: Do not throw away precious pages

2021-05-08 Thread Sergey Bugaev
The kernel invokes memory_object_data_return () with dirty = 0 in two cases: if notification on eviction was requested, or when returning precious pages. Properly distinguish between the two cases, and only throw the clean page away in the former case. --- libpager/data-return.c | 54

Re: [PATCH 5/6] libpager: Fix overallocating pagemap

2021-05-08 Thread Samuel Thibault
ger than > they needed to be. Indeed, applied, thanks! > --- > libpager/pagemap.c | 9 + > 1 file changed, 5 insertions(+), 4 deletions(-) > > diff --git a/libpager/pagemap.c b/libpager/pagemap.c > index 63c5f44d..1570c75b 100644 > --- a/libpager/pagemap.c > +

Re: [PATCH 6/6] libpager: Use libc heap for pagemap

2021-05-08 Thread Samuel Thibault
ts new pagemap entries to be set > to zero. This is the cause of those tarfs hangs I've been seeing > lately. > > I'm sorry, and here's a hotfix. Applied, thanks! > Sergey > > -- >8 -- > Subject: [PATCH] libpager: Properly zero-initialize pagemap >

Re: [PATCH 6/6] libpager: Use libc heap for pagemap

2021-05-08 Thread Samuel Thibault
we will not allocate a whole page of memory if we need to > store just a few integers. Applied, thanks! > --- > libpager/object-terminate.c | 5 +++-- > libpager/pagemap.c | 26 +- > 2 files changed, 12 insertions(+), 19 deletions(-) >

Re: [PATCH 4/6] libpager: Store pagemapsize as vm_size_t

2021-05-08 Thread Samuel Thibault
Sergey Bugaev, le jeu. 06 mai 2021 15:56:29 +0300, a ecrit: > On a 64-bit system, there can be a lot more pages > than a 32-bit int can fit. Indeed, applied, thanks! > --- > libpager/pagemap.c | 2 +- > libpager/priv.h| 2 +- > 2 files changed, 2 insertions(+), 2 dele

Re: [PATCH 2/6] libpager: Do not flush in-core pages on offer

2021-05-08 Thread Samuel Thibault
l > to flush (i.e. drop) its in-core copy, and replace it with the offered > one. Fix this by not doing that, and calling memory_object_data_supply () > directly. Indeed, applied, thanks! > --- > libpager/offer-page.c | 7 --- > 1 file changed, 7 deletions(-) > > diff

Re: [PATCH 1/6] libpager: Fix mixing up success and error

2021-05-08 Thread Samuel Thibault
Sergey Bugaev, le jeu. 06 mai 2021 15:56:26 +0300, a ecrit: > _pager_pagemap_resize () returns an error or 0 on success, > not a boolean. Quite trivially bogus indeed :) Applied, thanks! > --- > libpager/offer-page.c | 27 ++- > 1 file changed, 14 in

Re: [PATCH 6/6] libpager: Use libc heap for pagemap

2021-05-07 Thread Sergey Bugaev
ng lately. I'm sorry, and here's a hotfix. Sergey -- >8 -- Subject: [PATCH] libpager: Properly zero-initialize pagemap Unlike mmap () and calloc (), reallocarray () does not automatically zero-fill the newly allocated memory. Do so explicitly. --- libpager/pagemap.c | 2 ++ 1 file

[PATCH 6/6] libpager: Use libc heap for pagemap

2021-05-06 Thread Sergey Bugaev
integers. --- libpager/object-terminate.c | 5 +++-- libpager/pagemap.c | 26 +- 2 files changed, 12 insertions(+), 19 deletions(-) diff --git a/libpager/object-terminate.c b/libpager/object-terminate.c index 95298217..3e7df167 100644 --- a/libpager/object-terminate.c

[PATCH 5/6] libpager: Fix overallocating pagemap

2021-05-06 Thread Sergey Bugaev
The code tried to round up the allocation size to a multiple of page size. But we actually allocate newsize * sizeof (*p->pagemap) bytes, not newsize bytes, which meant allocations were sizeof (*p->pagemap) times larger than they needed to be. --- libpager/pagemap.c | 9 + 1 file c

[PATCH 4/6] libpager: Store pagemapsize as vm_size_t

2021-05-06 Thread Sergey Bugaev
On a 64-bit system, there can be a lot more pages than a 32-bit int can fit. --- libpager/pagemap.c | 2 +- libpager/priv.h| 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/libpager/pagemap.c b/libpager/pagemap.c index 963f6567..63c5f44d 100644 --- a/libpager/pagemap.c

[PATCH 3/6] libpager: Add error handling to various functions

2021-05-06 Thread Sergey Bugaev
Instead of ignoring errors, we might as well return them to the caller. This technically changes the function signatures, but should not break any users since they can continue to ignore the return value. --- doc/hurd.texi | 16 ++-- libpager/data-unlock.c | 13

[PATCH 2/6] libpager: Do not flush in-core pages on offer

2021-05-06 Thread Sergey Bugaev
this by not doing that, and calling memory_object_data_supply () directly. --- libpager/offer-page.c | 7 --- 1 file changed, 7 deletions(-) diff --git a/libpager/offer-page.c b/libpager/offer-page.c index ddea236a..0b0ca2eb 100644 --- a/libpager/offer-page.c +++ b/libpager/offer-page.c @@ -

[PATCH 0/6] Various libpager fixes

2021-05-06 Thread Sergey Bugaev
While hacking further on tarfs mmap support, I've come across some issues in libpager and I thought I'd fix them. The issues range from plain bugs (such as pager_offer_page () always silently failing) to minor things that could be improved. The patches are fairly independent, but they t

[PATCH 1/6] libpager: Fix mixing up success and error

2021-05-06 Thread Sergey Bugaev
_pager_pagemap_resize () returns an error or 0 on success, not a boolean. --- libpager/offer-page.c | 27 ++- 1 file changed, 14 insertions(+), 13 deletions(-) diff --git a/libpager/offer-page.c b/libpager/offer-page.c index 9f090bcb..ddea236a 100644 --- a/libpager/offer

[RFC PATCH tarfs 2/6] Link to libpager and initialize it on startup

2021-04-29 Thread Sergey Bugaev
uests; error_t err; /* Defaults to tarfs. */ @@ -68,6 +72,14 @@ main (int argc, char **argv) error (EXIT_FAILURE, err, "cannot create root node"); ul_node = netfs_startup (bootstrap_port, 0); + /* Init libpager. */ + tarfs_pager_bucket = ports_create_bucket (); + if (!tar

Re: doc libpager

2020-08-26 Thread Joan Lledó
Hi, thanks for your help, I'll take a look :) El 26/8/20 a les 16:23, gfle...@disroot.org ha escrit: > > Hi Joan, > > I can not reply in your thread "Implement paging on the pci arbiter" > because i suscribed right now. hope you this links! > > This

doc libpager

2020-08-26 Thread gfleury
Hi Joan, I can not reply in your thread "Implement paging on the pci arbiter" because i suscribed right now. hope you this links! This doc talk about libpager. https://www.freesoft.org/software/hurd/building.pdf (https://www.freesoft.org/software/hurd/building.pdf) https://

libpager, proc server, rpctrace, and other meandering

2018-02-02 Thread Brent W. Baccala
Hi - I'm writing to update the list on my current work and possibly get some useful suggestions. I'm trying to test the multi-client libpager in a subhurd, but I'm having problems even getting the old libpager code to work cleanly in a subhurd, so I know I've got problems th

libpager pagemap

2017-08-31 Thread Brent W. Baccala
Hi - My work on a multi-client libpager has progressed to the point where I have pseudo code for the logic and am considering how to implement the pagemap. I'm trying to achieve three goals with the pagemap: 1. keep pagemap entries as small as possible 2. support arbitrary numbers of cl

Re: libpager

2017-08-10 Thread Brent W. Baccala
, but I thought that it was a gnumach modification. > > Glancing at this git repository, I see that it's all translator stuff. > > Not merely, see e.g.: > > https://git.sceen.net/hurd/hurd.git/commit/?h=mplaneta/gsoc1 > 2/review_v1&id=2ca8719606eeca99c425b3282017a6412

Re: libpager

2017-08-06 Thread Justus Winter
"Brent W. Baccala" writes: >> Are you aware of the pager rework in [0]? I'm considering to review and >> merge these changes. It may make sense to try to come up with a common >> API. What is the state of your library, is it a drop-in replacement? >> >> 0: https://git.sceen.net/hurd/hurd.git/l

Re: libpager

2017-08-06 Thread Brent W. Baccala
#x27;ll study it. Is there corresponding gnumach work? The state of my library is that it's in pseudo code. It's intended to be a drop-in replacement for libpager. agape brent

Re: libpager

2017-08-06 Thread Justus Winter
"Brent W. Baccala" writes: > Hi - > > I've learned some interesting things about libpager that I'd like to share > with the list. Cool :) > I've found two bugs in the existing code that were "fixed" by the new > demuxer that sequentially se

libpager

2017-08-06 Thread Brent W. Baccala
Hi - I've learned some interesting things about libpager that I'd like to share with the list. I've found two bugs in the existing code that were "fixed" by the new demuxer that sequentially services requests on each pager object. For example, the data return code set

multi-client libpager

2017-07-17 Thread Brent W. Baccala
Hi - Just a quick FYI to let you all know that I'm working on a multi-client libpager. This will allow execution of binaries on remote nodes in a Hurd cluster. I'm well into pseudo coding the logic; you can follow my work in the netmsg github repository: g...@github.com:Br

Re: rpctrace / libpager / signal preemptor

2016-12-01 Thread Brent W. Baccala
On Wed, Nov 30, 2016 at 11:12 PM, Samuel Thibault wrote: > > Err, I'm sorry, did you perhaps miss the fix I made after: > > commit 406b031c996ec4cd8c76d251de8b7bf462d8b975 > Author: Samuel Thibault > Date: Sun Nov 20 16:16:24 2016 +0100 > > Fix SIGBUS code > Err, yes, I missed that. I do

Re: rpctrace / libpager / signal preemptor

2016-12-01 Thread Samuel Thibault
Hello, Brent W. Baccala, on Wed 30 Nov 2016 22:47:09 -1000, wrote: > The patch works, but is incomplete.  Samuel's test programs attempt to access > unmapped memory addresses, which generate KERN_MEMORY_FAILURE, but ext2fs > attempts to access mapped addresses back by a memory manager returning fa

Re: rpctrace / libpager / signal preemptor

2016-12-01 Thread Brent W. Baccala
On Wed, Nov 16, 2016 at 9:05 AM, Samuel Thibault wrote: > Samuel Thibault, on Wed 16 Nov 2016 19:50:07 +0100, wrote: > > Samuel Thibault, on Wed 16 Nov 2016 19:46:52 +0100, wrote: > > > The attached testcase does get the faulting address. > > > > And the attached testcase doesn't. > > And is fixe

Re: rpctrace / libpager / signal preemptor

2016-11-21 Thread Samuel Thibault
Samuel Thibault, on Mon 21 Nov 2016 09:16:19 +0100, wrote: > Brent W. Baccala, on Sun 20 Nov 2016 21:50:04 -1000, wrote: > > I suppose I could test your patch just by building the > > library itself, but the Debian package build calls "make check" and it's > > causing me all kinds of grief. > > Y

Re: rpctrace / libpager / signal preemptor

2016-11-21 Thread Samuel Thibault
Hello, Brent W. Baccala, on Sun 20 Nov 2016 21:50:04 -1000, wrote: > I suppose I could test your patch just by building the > library itself, but the Debian package build calls "make check" and it's > causing me all kinds of grief. You can use export DEB_BUILD_OPTIONS=nocheck to avoid that, se

Re: rpctrace / libpager / signal preemptor

2016-11-21 Thread Svante Signell
On Sun, 2016-11-20 at 21:50 -1000, Brent W. Baccala wrote: > On Sun, Nov 20, 2016 at 5:37 AM, Samuel Thibault > wrote: > > Samuel Thibault, on Sun 20 Nov 2016 14:50:50 +0100, wrote: > >  > Sorry I haven't answered for a few days.  I've been trying to test your patch > by building a new glibc packa

Re: rpctrace / libpager / signal preemptor

2016-11-20 Thread Brent W. Baccala
d it's causing me all kinds of grief. So much so that I wonder if something haven't changed recently; I've built glibc before without these problems. At least I'm getting a good look at how a Hurd system behaves when it runs out of swap. Do you have any idea what other fixes

Re: rpctrace / libpager / signal preemptor

2016-11-20 Thread Samuel Thibault
Samuel Thibault, on Sun 20 Nov 2016 14:50:50 +0100, wrote: > Samuel Thibault, on Wed 16 Nov 2016 20:05:49 +0100, wrote: > > Samuel Thibault, on Wed 16 Nov 2016 19:50:07 +0100, wrote: > > > Samuel Thibault, on Wed 16 Nov 2016 19:46:52 +0100, wrote: > > > > The attached testcase does get the faulting

Re: rpctrace / libpager / signal preemptor

2016-11-20 Thread Samuel Thibault
Samuel Thibault, on Wed 16 Nov 2016 20:05:49 +0100, wrote: > Samuel Thibault, on Wed 16 Nov 2016 19:50:07 +0100, wrote: > > Samuel Thibault, on Wed 16 Nov 2016 19:46:52 +0100, wrote: > > > The attached testcase does get the faulting address. > > > > And the attached testcase doesn't. > > And is f

Re: rpctrace / libpager / signal preemptor

2016-11-16 Thread Samuel Thibault
Samuel Thibault, on Wed 16 Nov 2016 19:50:07 +0100, wrote: > Samuel Thibault, on Wed 16 Nov 2016 19:46:52 +0100, wrote: > > The attached testcase does get the faulting address. > > And the attached testcase doesn't. And is fixed by the attached patch, could you try it? Samuel Index: glibc-2.24/s

Re: rpctrace / libpager / signal preemptor

2016-11-16 Thread Samuel Thibault
Samuel Thibault, on Wed 16 Nov 2016 19:46:52 +0100, wrote: > The attached testcase does get the faulting address. And the attached testcase doesn't. > I really believe the issue is related to this: Confirmed :) > > Note that there is a > > > > /* XXX what if handler != action->handler (for

Re: rpctrace / libpager / signal preemptor

2016-11-16 Thread Samuel Thibault
Samuel Thibault, on Mon 14 Nov 2016 01:07:40 +0100, wrote: > > Once that's been resolved, then we're back to the problem with signal > > preemptors!  libpager/pager-memcpy.c includes the following code: > > > >   void fault (int signo, long i

Re: rpctrace / libpager / signal preemptor

2016-11-13 Thread Samuel Thibault
hen we're back to the problem with signal > preemptors!  libpager/pager-memcpy.c includes the following code: > >   void fault (int signo, long int sigcode, struct sigcontext *scp) >     { >   assert (scp->sc_error == EKERN_MEMORY_ERROR); >   err = pager_get_error (p

rpctrace / libpager / signal preemptor

2016-11-08 Thread Brent W. Baccala
Hi - Just a status update about what I'm working on. My primary goal right now is to get ext2fs working right when a ramdisk fills up. dd hangs and/or crashes ext2fs instead of cleanly erroring out. First, there's a problem in libpager --- a/libpager/data-unlock.c +++ b/libpager/dat

Holy war (was: libpager multi-client support)

2016-10-22 Thread Olaf Buddenhagen
Hi, On Mon, Oct 17, 2016 at 09:51:41PM -1000, Brent W. Baccala wrote: > I've been thinking more about the data structures involved with the new > libpager code, and they're complex enough that I'd like to write them in > C++11. Yay, flamebait! ;-) Two years ago, I w

Re: libpager multi-client support

2016-10-18 Thread Richard Braun
On Tue, Oct 18, 2016 at 01:56:48PM -1000, Brent W. Baccala wrote: > Can it be done in C? Sure. But following my principle of asking how to > write the code in the simplest manner, this one is obvious: STL containers > win. I'm not saying it can't be done. I'm saying it's not that hard. -- Rich

Re: libpager multi-client support

2016-10-18 Thread Brent W. Baccala
On Tue, Oct 18, 2016 at 2:37 AM, Richard Braun wrote: > > From my fifteen years or so experience working in various projects with > C++ and other languages, my opinion is that C++ is never the best > choice. It makes you trade lines of code for increased complexity in > understanding and followin

Re: libpager multi-client support

2016-10-18 Thread Richard Braun
On Mon, Oct 17, 2016 at 09:51:41PM -1000, Brent W. Baccala wrote: > What do you guys think? >From my fifteen years or so experience working in various projects with C++ and other languages, my opinion is that C++ is never the best choice. It makes you trade lines of code for increased complexity i

Re: libpager multi-client support

2016-10-18 Thread Brent W. Baccala
Aloha - I've been thinking more about the data structures involved with the new libpager code, and they're complex enough that I'd like to write them in C++11. The pagemap has to contain a list of all clients with copies of a page, and a queue of clients waiting to access the pag

Re: workflow with Debian patches and Git repositories (was: libpager multi-client support)

2016-10-16 Thread Brent W. Baccala
On Sat, Oct 15, 2016 at 11:17 PM, Kalle Olavi Niemitalo wrote: > > I also have a "hurd-debian" working tree. I extracted that from > the Debian source package, so that I got a .pc directory with the > correct state information. I then added a .git directory cloned > from "git://anonscm.debian.o

workflow with Debian patches and Git repositories (was: libpager multi-client support)

2016-10-16 Thread Kalle Olavi Niemitalo
"Brent W. Baccala" writes: > My first question is for advice on managing my workflow. I'm using the > Debian hurd package, which adds patches on top of a snapshot taken from > savannah's git tree. I think I want to work on the Debian-ized code, since > that's what's installed on my system, but

Re: libpager multi-client support

2016-10-14 Thread Kalle Olavi Niemitalo
"Brent W. Baccala" writes: > Second, I can't find anywhere in the hurd source tree where this function > is actually used. Commit 84cf9c0f312637b670cc87224ff7e7c4da659e36 on 2013-09-17 removed the ufs directory, in which the offer_data function used to call pager_offer_page. The last argument o

libpager multi-client support

2016-10-13 Thread Brent W. Baccala
aloha - I've started looking at adding multi-client support to libpager. My first question is for advice on managing my workflow. I'm using the Debian hurd package, which adds patches on top of a snapshot taken from savannah's git tree. I think I want to work on the Debian-i

Proxy memory objects (was: Denial of service attack via libpager)

2016-09-21 Thread Olaf Buddenhagen
Hi, On Mon, Aug 29, 2016 at 11:15:48AM +0200, Richard Braun wrote: > OK, this comes from the fact that io_map directly provides memory > objects indeed... Do we actually want to pass them around ? How > come calls like memory_object_init (specifically meant to be used > between the kernel and the

Re: Denial of service attack via libpager

2016-08-30 Thread Richard Braun
On Mon, Aug 29, 2016 at 03:58:29PM -1000, Brent W. Baccala wrote: > On Sun, Aug 28, 2016 at 11:15 PM, Richard Braun wrote: > > OK, this comes from the fact that io_map directly provides memory > > objects indeed... Do we actually want to pass them around ? How > > come calls like memory_object_ini

Re: Denial of service attack via libpager

2016-08-29 Thread Brent W. Baccala
On Sun, Aug 28, 2016 at 11:15 PM, Richard Braun wrote: > On Sun, Aug 28, 2016 at 05:12:35PM -1000, Brent W. Baccala wrote: > > > The obvious additional client would be a remote kernel, but as the > exploit > > program that I posted shows, it could just as easily be an unprivileged > > process. Y

Re: Denial of service attack via libpager

2016-08-29 Thread Richard Braun
On Mon, Aug 29, 2016 at 11:20:47AM +0200, Richard Braun wrote: > In addition, I've just thought about something else : if we handle > multiple clients, how do we make sure that two kernels, caching the > same file, don't just completely corrupt its content ? We'd need > some kind of cooperation to

Re: Denial of service attack via libpager

2016-08-29 Thread Richard Braun
On Sun, Aug 28, 2016 at 05:12:35PM -1000, Brent W. Baccala wrote: > I should elaborate on what I found with exec. After I fixed the problem > with the exec server mmap'ing the library's ELF headers, it just got on a > little bit further in the process, and then croaked when it tried to mmap > the

Re: Denial of service attack via libpager

2016-08-29 Thread Richard Braun
On Sun, Aug 28, 2016 at 05:12:35PM -1000, Brent W. Baccala wrote: > So we still have to mmap across the network. We certainly don't want to > avoid mmap's entirely for program text and (especially) for shared > libraries. Although I admit that it would be best to detect when the mmap > fails and

Re: Denial of service attack via libpager

2016-08-28 Thread Brent W. Baccala
On Sun, Aug 28, 2016 at 12:49 PM, Richard Braun wrote: > > I'm really not seeing the relation between "multiple clients" and > "multiple threads". Libpager must be able to handle multiple clients > with a single thread, otherwise we don't control sca

Re: Denial of service attack via libpager

2016-08-28 Thread Richard Braun
On Sun, Aug 28, 2016 at 12:20:14PM -1000, Brent W. Baccala wrote: > So, there's several problems here: > > 1. libpager can't handle multiple clients > 2. the kernel can't recover from a failed attempt to get a file's memory > object > 3. ext2fs can&#x

Denial of service attack via libpager

2016-08-28 Thread Brent W. Baccala
f halt...(ipc/rcv) timed out startup: halting Mach (flags 0x8)... It's a 60 second timeout that must terminate the ext2fs translator abnormally, because the file system is left dirty. So, there's several problems here: 1. libpager can't handle multiple clients 2. the kernel can't

Re: [PATCH hurd 2/5] libpager: use libports notification functions

2014-12-10 Thread Samuel Thibault
Justus Winter, le Wed 10 Dec 2014 17:21:56 +0100, a écrit : > Now that we do not use sequence numbers anymore, we can simply use > libports notification server functions. > > * libpager/Makefile (SRCS): Drop `no-senders.c', and `notify-stubs.c'. > (OBJS): Drop `noti

Re: [PATCH hurd 1/5] libpager: remove the `seqno' parameters

2014-12-10 Thread Samuel Thibault
Justus Winter, le Wed 10 Dec 2014 17:21:55 +0100, a écrit : > * libpager/Makefile (MIGSFLAGS): Drop `-DSEQNOS'. > * libpager/priv.h (struct pager): Drop fields `seqno' and `waitingforseqno'. > * libpager/chg-compl.c: Adopt accordingly. > * libpager/data-request.c:

[PATCH hurd 1/5] libpager: remove the `seqno' parameters

2014-12-10 Thread Justus Winter
* libpager/Makefile (MIGSFLAGS): Drop `-DSEQNOS'. * libpager/priv.h (struct pager): Drop fields `seqno' and `waitingforseqno'. * libpager/chg-compl.c: Adopt accordingly. * libpager/data-request.c: Likewise. * libpager/data-return.c: Likewise. * libpager/data-unlock.c: Likew

[PATCH hurd 2/5] libpager: use libports notification functions

2014-12-10 Thread Justus Winter
Now that we do not use sequence numbers anymore, we can simply use libports notification server functions. * libpager/Makefile (SRCS): Drop `no-senders.c', and `notify-stubs.c'. (OBJS): Drop `notifyServer.o'. * libpager/demuxer.c (pager_demuxer): Use libports server functions.

Re: [PATCH hurd 09/30] libpager: add a payload-aware intrans function

2014-12-01 Thread Samuel Thibault
Justus Winter, le Thu 27 Nov 2014 14:18:49 +0100, a écrit : > * libpager/mig-decls.h (begin_using_pager_payload): New function. > * libpager/mig-mutate.h: Add mutators. Ack. > --- > libpager/mig-decls.h | 6 ++ > libpager/mig-mutate.h | 3 +++ > 2 files changed, 9 inser

[PATCH hurd 09/30] libpager: add a payload-aware intrans function

2014-11-27 Thread Justus Winter
* libpager/mig-decls.h (begin_using_pager_payload): New function. * libpager/mig-mutate.h: Add mutators. --- libpager/mig-decls.h | 6 ++ libpager/mig-mutate.h | 3 +++ 2 files changed, 9 insertions(+) diff --git a/libpager/mig-decls.h b/libpager/mig-decls.h index 0c7b402..7e6b64f 100644

Re: [PATCH hurd 1/2] libpager: make the request queue more memory-efficient

2014-11-23 Thread Samuel Thibault
ocating memory is dangerous. > > Fix this by not allocating space for the response message, rather, use > a chunk of the workers stack space. Also, we only handle the `notify' > and `memory_object' protocol, which both only contain simple routines, > we only need a `mig_respo

[PATCH hurd 1/2] libpager: make the request queue more memory-efficient

2014-11-23 Thread Justus Winter
ssage, rather, use a chunk of the workers stack space. Also, we only handle the `notify' and `memory_object' protocol, which both only contain simple routines, we only need a `mig_response_header_t'. * libpager/demuxer.c (struct request): Remove `inp' and `outp'

Re: [PATCH hurd] libpager: use a fixed number of threads

2014-11-03 Thread Ludovic Courtès
Justus Winter <4win...@informatik.uni-hamburg.de> skribis: > Quoting Samuel Thibault (2014-11-02 13:52:16) >> AIUI, this does work because you had previously separated disk and file >> pagers? > > Yes. > >> > + /* What follows is basically the second part of >> > + mach_msg_server_timeou

Re: [PATCH hurd] libpager: use a fixed number of threads

2014-11-03 Thread Samuel Thibault
Justus Winter, le Mon 03 Nov 2014 12:23:27 +0100, a écrit : > > > + /* What follows is basically the second part of > > > + mach_msg_server_timeout. */ > > > > So you need to copy the copyright years :) > > That's not straight forward as the glibc uses year ranges (1993-2014 > in this

Re: [PATCH hurd] libpager: use a fixed number of threads

2014-11-03 Thread Justus Winter
Hi :) Quoting Samuel Thibault (2014-11-02 13:52:16) > AIUI, this does work because you had previously separated disk and file > pagers? Yes. > > + /* What follows is basically the second part of > > + mach_msg_server_timeout. */ > > So you need to copy the copyright years :) That's

Re: let's use a fixed number of threads for libpager

2014-11-02 Thread Richard Braun
On Thu, Oct 30, 2014 at 04:43:53PM +0100, Richard Braun wrote: > Actually, after applying the libpager patch only, as opposed to using > the continuous integration packages which merge all of Justus' work, > things look much better. So far I haven't been able to reproduce th

Re: [PATCH hurd] libpager: use a fixed number of threads

2014-11-02 Thread Samuel Thibault
Hello, I'm thrilled by the potential of this change :D AIUI, this does work because you had previously separated disk and file pagers? Justus Winter, le Tue 28 Oct 2014 01:37:44 +0100, a écrit : > +/* Start the worker threads libpager uses to service requests. */ &g

Re: let's use a fixed number of threads for libpager

2014-10-31 Thread Richard Braun
On Tue, Oct 28, 2014 at 05:59:16PM +0100, Richard Braun wrote: > Stressing the file system (by building a large package such as > iceweasel) triggers a reference counting assertion [1]. We'll have to > squash this bug hard before the patch can be merged. Actually, after applyin

Re: let's use a fixed number of threads for libpager

2014-10-28 Thread Justus Winter
Quoting Richard Braun (2014-10-28 17:59:16) > On Tue, Oct 28, 2014 at 01:37:43AM +0100, Justus Winter wrote: > > Preliminary testing on my box as well as on darnassus looks promising > > (modulo some pfinet weirdness that Richard is investigating [and imho > > if there is an issue, it should be unr

Re: let's use a fixed number of threads for libpager

2014-10-28 Thread Richard Braun
On Tue, Oct 28, 2014 at 01:37:43AM +0100, Justus Winter wrote: > Preliminary testing on my box as well as on darnassus looks promising > (modulo some pfinet weirdness that Richard is investigating [and imho > if there is an issue, it should be unrelated to this change]). We > will keep you posted.

Re: let's use a fixed number of threads for libpager

2014-10-28 Thread Richard Braun
p you posted. It looks like a netdde-specific problem. Perhaps a small mistake with regard to the static library it includes. Other than that, the patch looks good, and I'm very, very happy to finally see something like this land in libpager. -- Richard Braun

let's use a fixed number of threads for libpager

2014-10-27 Thread Justus Winter
Hello :) this is a new patch that makes libpager use a fixed number of threads, currently one (plus one receiving messages and enqueuing them into a userspace queue). If our store interface improves, we can increase the number of worker threads (the work delegation uses a linear scan over the

[PATCH hurd] libpager: use a fixed number of threads

2014-10-27 Thread Justus Winter
Previously, libpager used an unbounded number of threads to receive messages from the pager bucket. It used sequence barriers to execute the requests to order requests to each object. The sequence barriers are implemented in seqnos.c. A server function uses _pager_wait_for_seqno to wait for its

[PATCH hurd 6/8] libpager: annotate objects managed by libports

2014-10-23 Thread Justus Winter
Label _pager_class and provide a function which prints a human-readable description of a pager object. * libpager/pager-create.c (format_debug_info): New function. (create_class): Label _pager_class. --- libpager/pager-create.c | 20 1 file changed, 20 insertions(+) diff

Re: let's make libpager single-threaded

2014-05-30 Thread Richard Braun
On Fri, May 30, 2014 at 08:15:43PM +0200, Samuel Thibault wrote: > Justus Winter, le Fri 30 May 2014 09:35:37 +0200, a écrit : > > Quoting Richard Braun (2014-05-29 19:12:13) > > > On Thu, May 29, 2014 at 07:04:48PM +0200, Samuel Thibault wrote: > > > > But precisely: once the only thread gets a da

Re: let's make libpager single-threaded

2014-05-30 Thread Samuel Thibault
Justus Winter, le Fri 30 May 2014 09:35:37 +0200, a écrit : > Quoting Richard Braun (2014-05-29 19:12:13) > > On Thu, May 29, 2014 at 07:04:48PM +0200, Samuel Thibault wrote: > > > But precisely: once the only thread gets a data_request, it'll call > > > pager_read_page, which will typically call s

Re: let's make libpager single-threaded

2014-05-30 Thread Richard Braun
On Fri, May 30, 2014 at 09:35:37AM +0200, Justus Winter wrote: > Quoting Richard Braun (2014-05-29 19:12:13) > > On Thu, May 29, 2014 at 07:04:48PM +0200, Samuel Thibault wrote: > > > But precisely: once the only thread gets a data_request, it'll call > > > pager_read_page, which will typically cal

Re: let's make libpager single-threaded

2014-05-30 Thread Justus Winter
Quoting Richard Braun (2014-05-29 19:12:13) > On Thu, May 29, 2014 at 07:04:48PM +0200, Samuel Thibault wrote: > > But precisely: once the only thread gets a data_request, it'll call > > pager_read_page, which will typically call store_read, which will wait > > on the eventual I/O, and no other req

Re: let's make libpager single-threaded

2014-05-29 Thread Richard Braun
On Thu, May 29, 2014 at 07:04:48PM +0200, Samuel Thibault wrote: > But precisely: once the only thread gets a data_request, it'll call > pager_read_page, which will typically call store_read, which will wait > on the eventual I/O, and no other request will be processed during that > time, and thus

Re: let's make libpager single-threaded

2014-05-29 Thread Samuel Thibault
Richard Braun, le Tue 27 May 2014 14:12:07 +0200, a écrit : > > But the paging part also comes from client events, doesn't it? E.g. > > on-demand loads of parts of mapped libraries, etc. > > If on-demand here means on page fault, then no. In Mach, client threads > don't process their page faults,

Re: let's make libpager single-threaded

2014-05-27 Thread Richard Braun
On Fri, May 23, 2014 at 01:44:09AM +0200, Samuel Thibault wrote: > Richard Braun, le Mon 05 May 2014 18:32:26 +0200, a écrit : > > On Mon, May 05, 2014 at 06:01:17PM +0200, Samuel Thibault wrote: > > > ? The patch makes both ext2fs's service_paging_requests and libdiskfs' > > > service_paging_reque

Re: [PATCH 2/7] libpager: drop unused fields from struct pager

2014-05-26 Thread Samuel Thibault
Justus Winter, le Mon 26 May 2014 13:19:14 +0200, a écrit : > * libpager/priv.h (struct pager): Drop fields next, pprev. Ack. > --- > libpager/priv.h | 2 -- > 1 file changed, 2 deletions(-) > > diff --git a/libpager/priv.h b/libpager/priv.h > index d49cbb9..1f8405a 10

[PATCH 2/7] libpager: drop unused fields from struct pager

2014-05-26 Thread Justus Winter
* libpager/priv.h (struct pager): Drop fields next, pprev. --- libpager/priv.h | 2 -- 1 file changed, 2 deletions(-) diff --git a/libpager/priv.h b/libpager/priv.h index d49cbb9..1f8405a 100644 --- a/libpager/priv.h +++ b/libpager/priv.h @@ -59,8 +59,6 @@ struct pager int noterm

Re: let's make libpager single-threaded

2014-05-22 Thread Samuel Thibault
Richard Braun, le Mon 05 May 2014 18:32:26 +0200, a écrit : > On Mon, May 05, 2014 at 06:01:17PM +0200, Samuel Thibault wrote: > > ? The patch makes both ext2fs's service_paging_requests and libdiskfs' > > service_paging_requests become singlethreaded. > > That's what I call the paging part. The f

Re: let's make libpager single-threaded

2014-05-05 Thread Richard Braun
On Mon, May 05, 2014 at 06:01:17PM +0200, Samuel Thibault wrote: > ? The patch makes both ext2fs's service_paging_requests and libdiskfs' > service_paging_requests become singlethreaded. That's what I call the paging part. The front side, where client calls are processed, is still multithreaded, w

Re: let's make libpager single-threaded

2014-05-05 Thread Samuel Thibault
Richard Braun, le Mon 05 May 2014 17:56:37 +0200, a écrit : > On Mon, May 05, 2014 at 04:55:19PM +0200, Samuel Thibault wrote: > > It's an interesting alternative indeed. This however means our ext2fs > > is not multithreaded any longer, which is a bit sad considered that > > we'll want to go para

Re: let's make libpager single-threaded

2014-05-05 Thread Richard Braun
On Mon, May 05, 2014 at 04:55:19PM +0200, Samuel Thibault wrote: > It's an interesting alternative indeed. This however means our ext2fs > is not multithreaded any longer, which is a bit sad considered that > we'll want to go parallel in the end. Well, no, only the paging part becomes single thre

Re: let's make libpager single-threaded

2014-05-05 Thread Samuel Thibault
Hello, Justus Winter, le Mon 05 May 2014 17:23:10 +0200, a écrit : > Just to make sure I got my idea across. With my change, a single > thread would service all requests to disk pagers, another one would > manage the requests to file pagers. Sure. > And those requests are mostly > IO-bound, and

  1   2   >