Hy!
You might be right, but fsync() might not be needed.
The program deals with the files as follows:
- At the start, mailbox.base and mailbox.ext are opened for reading,
then the program reads all the data, and closes the files.
- 99.9% of the time, the program serves everything from memory.
When a new folder is created, an existing is modified, etc,
the program opens mailbox.ext for writing (append), appends a line to it,
and closes the file.
I think the file close should be equal to a sync. Am I right?
And so, the "worst" that could happen (with a journalling fs), is that you
lose the (new) last line of the file, that's one folder.
By the way, since yesterday, the daemon is in live testing now on my system
with 300'000 folders, and 4-5000 active users (daily). Cross your fingers ;)
Version 0.2 is out, on the page http://opensource.prim.hu/mbdaemon/
(It currently handles only user. folders.)
On 20-May-2001 Lawrence Greenfield wrote:
> I think this is really interesting work. I haven't looked at the
> implementation but have considered this basic idea several times; I
> think it's definitely worth the work.
>
> A random comment on your message:
>
> The mailbox.ext file is always "small", so writing to it should be
> fast, and you could even so sync()-s on it. (But rather, use a
> journalling filesystem.)
>
> All filesystems _require_ you to use fsync(). Journalling filesystems
> aren't magical: if you don't fsync() you have no guarantee that the
> data gets to disk. So please call fsync; otherwise, you're making
> potentially serious problems for people who use your code.
>
> Larry
>
> Date: Sun, 20 May 2001 18:17:48 +0200 (CEST)
> From: Noll Janos <[EMAIL PROTECTED]>
>
> However, mailbox.base is "fixed", it doesn't change at (daemon) runtime,
> however it could be updated daily or weeky. The mailbox.ext is something
> like a
> log file, it also contains actions, keys, values, on separate lines. The
> daemon
> writes changes to (the end) of this file at runtime, whenever a folder is
> created, deleted, or changed. The daemon currently handles only "user.*"
> folders, this was to save space, and this suits my needs.
>
>
> Any way, if the daemon crashes, no problem, as long as the mailbox.base
> and
> mailbox.ext files are ok. The mailbox.base is even safer, as it's
> read-only for
> the daemon.
>
> About the "write-caching" question: if you use cyrus IMAPd with a text
> mailboxes.db, inserting a line into it is always slow, because a new file
> is
> created (mailboxes.db.NEW), the new line is inserted into it, and then
> it's
> moved the "old" mailboxes.db. If you use DB3, that could be better, but
> even
> then, every imapd process must read and process some of the data.
>
>
> But back to my daemon, I must emphasise, that this is only an actual
> implementation of the "socket" mail-folder db protocol. The architecture
> is
> there: one could write a daemon, that uses SQL, a daemon, that does more
> caching.
>
> The picture:
>
> +-------------------+ +----------------+
> | IMAPD + socket_db |------socket------| mailbox-daemon |
> +-------------------+ +----------------+
>
> I defined a small "protocol" for the communication.
> An example:
> IMAPD wants to get the data for the folder "user.test12"
> The steps
> 1. IMAPd sends a message through the socket: "GET user.test12"
> 2. mailbox-daemon replies: "OK user.test12 0 default user.test12
> lps...."
>
> There are a few more functions (iterated GET, SET, REMOVE) that a daemon
> must
> implement, but that's all. Caching is easy to do, as every GET, SET, etc.
> gets
> to this daemon AND ONLY this daemon.
>
> I'll do some real testing over the weekend, and maybe I'll whip up some
> more
> documentation.
>
