I think this is really interesting work. I haven't looked at the
implementation but have considered this basic idea several times; I
think it's definitely worth the work.
A random comment on your message:
The mailbox.ext file is always "small", so writing to it should be
fast, and you could even so sync()-s on it. (But rather, use a
journalling filesystem.)
All filesystems _require_ you to use fsync(). Journalling filesystems
aren't magical: if you don't fsync() you have no guarantee that the
data gets to disk. So please call fsync; otherwise, you're making
potentially serious problems for people who use your code.
Larry
Date: Sun, 20 May 2001 18:17:48 +0200 (CEST)
From: Noll Janos <[EMAIL PROTECTED]>
However, mailbox.base is "fixed", it doesn't change at (daemon) runtime,
however it could be updated daily or weeky. The mailbox.ext is something like a
log file, it also contains actions, keys, values, on separate lines. The daemon
writes changes to (the end) of this file at runtime, whenever a folder is
created, deleted, or changed. The daemon currently handles only "user.*"
folders, this was to save space, and this suits my needs.
Any way, if the daemon crashes, no problem, as long as the mailbox.base and
mailbox.ext files are ok. The mailbox.base is even safer, as it's read-only for
the daemon.
About the "write-caching" question: if you use cyrus IMAPd with a text
mailboxes.db, inserting a line into it is always slow, because a new file is
created (mailboxes.db.NEW), the new line is inserted into it, and then it's
moved the "old" mailboxes.db. If you use DB3, that could be better, but even
then, every imapd process must read and process some of the data.
But back to my daemon, I must emphasise, that this is only an actual
implementation of the "socket" mail-folder db protocol. The architecture is
there: one could write a daemon, that uses SQL, a daemon, that does more
caching.
The picture:
+-------------------+ +----------------+
| IMAPD + socket_db |------socket------| mailbox-daemon |
+-------------------+ +----------------+
I defined a small "protocol" for the communication.
An example:
IMAPD wants to get the data for the folder "user.test12"
The steps
1. IMAPd sends a message through the socket: "GET user.test12"
2. mailbox-daemon replies: "OK user.test12 0 default user.test12 lps...."
There are a few more functions (iterated GET, SET, REMOVE) that a daemon must
implement, but that's all. Caching is easy to do, as every GET, SET, etc. gets
to this daemon AND ONLY this daemon.
I'll do some real testing over the weekend, and maybe I'll whip up some more
documentation.
On 17-May-2001 Michael Fair wrote:
> This sounds really cool!
>
> Can you explain more of the details on how you implemented it?
> What happens on system crash?
> How often does it "sync" to the hard drive?
> What's the difference between this and using a write-caching
> hard drive scheme?
>
> -- Michael --
>
> ----- Original Message -----
> From: "Noll Janos" <[EMAIL PROTECTED]>
> To: <[EMAIL PROTECTED]>
> Sent: Monday, May 14, 2001 5:10 PM
> Subject: mailbox-daemon
>
>
>> Hy!
>>
>> Until now, cyrus could handle either a text file folderlist (mailboxes)
> or a
>> DB3 folderlist (mailboxes.db). But now it can user "socket"
> folder-handler,
>> to connect to a daemon. The mailbox-server "daemon" keeps all the
>> mailbox data ("formerly" mailboxes.db) in memory, so it should be faster.
[...]
