On Thu, Jan 6, 2011 at 7:59 PM, Enrico Weigelt wrote:
> * Olaf van der Spek schrieb:
>
>> A transaction to update multiple files in one atomic go?
>
> Yes. The application first starts an transaction, creates/writes/
> removes a bunch of files and then sends a commit. The changes
> should become
* Olaf van der Spek schrieb:
> A transaction to update multiple files in one atomic go?
Yes. The application first starts an transaction, creates/writes/
removes a bunch of files and then sends a commit. The changes
should become visible atomically and the call returns when the
commit() is compl
On Thu, Jan 6, 2011 at 7:33 PM, Enrico Weigelt wrote:
> To come back to the original question, I'd like to know which concrete
> realworld problems should be solved by that. One thing an database-like
> transactional filesystem (w/ MVCC) would be nice is package managers:
> we still have the probl
> Getting people to believe that you can't square a circle[1]
> is very hard,
Just allow an infinite number of steps and it's almost trivial ;-)
> It's like trying teaching a pig to sing.
Well, that works, just sounds a bit like vogon poetry ;-o
> If you give me a specific approach, I can t
On Thu, Jan 6, 2011 at 12:39 PM, Olaf van der Spek wrote:
> On Thu, Jan 6, 2011 at 5:01 AM, Ted Ts'o wrote:
>> On Thu, Jan 06, 2011 at 12:57:07AM +, Ian Jackson wrote:
>>> Ted Ts'o writes ("Re: Safe File Update (atomic)"):
>>> > Then I invite
On Thu, Jan 6, 2011 at 5:01 AM, Ted Ts'o wrote:
> On Thu, Jan 06, 2011 at 12:57:07AM +, Ian Jackson wrote:
>> Ted Ts'o writes ("Re: Safe File Update (atomic)"):
>> > Then I invite you to implement it, and start discovering all of the
>> >
* Ted Ts'o [110105 19:26]:
> So one of the questions is how much should be penalizing programs that
> are doing things right (i.e., using fsync), versus programs which are
> doing things wrong (i.e., using rename and trusting to luck).
Please do not call it "wrong". All those programs doing is no
On Thu, Jan 6, 2011 at 1:54 AM, Ted Ts'o wrote:
>> I was thinking, doesn't ext have this kind of dependency tracking already?
>> It has to write the inode after writing the data, otherwise the inode
>> might point to garbage.
>
> No, it doesn't. We use journaling, and forced data writeouts, to
>
On Thu, Jan 06, 2011 at 12:57:07AM +, Ian Jackson wrote:
> Ted Ts'o writes ("Re: Safe File Update (atomic)"):
> > Then I invite you to implement it, and start discovering all of the
> > corner cases for yourself. :-) As I predicted, you're not going to
>
Ted Ts'o writes ("Re: Safe File Update (atomic)"):
> Then I invite you to implement it, and start discovering all of the
> corner cases for yourself. :-) As I predicted, you're not going to
> believe me when I tell you it's too hard.
How about you reimpleme
On Wed, Jan 05, 2011 at 10:47:03PM +0100, Olaf van der Spek wrote:
>
> That was about soft updates. I'm not sure this is just as complex.
Then I invite you to implement it, and start discovering all of the
corner cases for yourself. :-) As I predicted, you're not going to
believe me when I tell
On Wed, Jan 5, 2011 at 10:37 PM, Ted Ts'o wrote:
>> Ah. So performance isn't the problem, it's just hard too implement.
>> Would've been a lot faster if you said that earlier.
>
> "Too hard to implement" doesn't go far enough. It's also a matter of
> near impossibility to add new features later.
On Wed, Jan 05, 2011 at 09:38:30PM +0100, Olaf van der Spek wrote:
>
> Performance is important, I agree.
> But you're trading performance for safety here.
... but if the safety is not needed, then you're paying for no good
reason. And if performance is needed, then use fsync().
> > OK, what ab
On Wed, Jan 5, 2011 at 7:26 PM, Ted Ts'o wrote:
> On Wed, Jan 05, 2011 at 12:55:22PM +0100, Olaf van der Spek wrote:
>> > If you give me a specific approach, I can tell you why it won't work,
>> > or why it won't be accepted by the kernel maintainers (for example,
>> > because it involves pouring
On Wed, Jan 05, 2011 at 12:55:22PM +0100, Olaf van der Spek wrote:
> > If you give me a specific approach, I can tell you why it won't work,
> > or why it won't be accepted by the kernel maintainers (for example,
> > because it involves pouring far too much complexity into the kernel).
>
> Let's c
On Wed, Jan 5, 2011 at 1:25 AM, Ted Ts'o wrote:
> On Wed, Jan 05, 2011 at 01:05:03AM +0100, Olaf van der Spek wrote:
>>
>> Why is it that you ignore all my responses to technical questions you asked?
>>
>
> In general, because they are either (a) not well-formed, or (b) you
> are asking me to prov
On Wed, Jan 05, 2011 at 01:05:03AM +0100, Olaf van der Spek wrote:
>
> Why is it that you ignore all my responses to technical questions you asked?
>
In general, because they are either (a) not well-formed, or (b) you
are asking me to prove a negative. Getting people to believe that you
can't s
On Mon, Jan 3, 2011 at 3:43 PM, Ted Ts'o wrote:
> On Mon, Jan 03, 2011 at 12:26:29PM +0100, Olaf van der Spek wrote:
>>
>> Given that the issue has come up before so often, I expected there to
>> be a FAQ about it.
>
> Your asking the question over (and over... and over...) doesn't make
> it an F
Ted Ts'o mit.edu> writes:
> actually. The right answer has been known for decades, and it's is
> very simple; write a temp file, copy over the xattr's and ACL's if you
> care (in many cases, such as an application's private state files, it
> won't care, so it can skip this step --- it's only the
On Mon, Jan 3, 2011 at 3:43 PM, Ted Ts'o wrote:
> On Mon, Jan 03, 2011 at 12:26:29PM +0100, Olaf van der Spek wrote:
>>
>> Given that the issue has come up before so often, I expected there to
>> be a FAQ about it.
>
> Your asking the question over (and over... and over...) doesn't make
> it an F
On Mon, Jan 03, 2011 at 12:26:29PM +0100, Olaf van der Spek wrote:
>
> Given that the issue has come up before so often, I expected there to
> be a FAQ about it.
Your asking the question over (and over... and over...) doesn't make
it an FAQ. :-)
Aside from your asking over and over, it hasn't
On Mon, Jan 3, 2011 at 11:35 AM, Shachar Shemesh wrote:
> On 02/01/11 17:37, Olaf van der Spek wrote:
>>
>> A userspace lib is fine with me. In fact, I've been asking for it
>> multiple times. Result: no response.
>>
>>
>
> Excuse me?
>
> You (well, Henrique, but you were CCed) said "how about a u
On Mon, Jan 3, 2011 at 6:28 AM, Enrico Weigelt wrote:
> * Ted Ts'o schrieb:
>
>> This is possible. It would be specific only to file systems that
>> support inodes (i.e., ix-nay for NFS, FAT, etc.).
>
> FAT supports inodes ?
ix-nay: no/except
Olaf
--
To UNSUBSCRIBE, email to debian-devel-req
On Mon, Jan 03, 2011 at 09:49:40AM -0200, Henrique de Moraes Holschuh wrote:
>
> > 1) You care about data loss in the case of power failure, but not in
> > the case of hard drive or storage failure, *AND* you are writing tons
> > and tons of tiny 3-4 byte files and so you are worried about
> > per
Ted,
Thanks for the reply and detailed analysis.
> Which gets me back to the question of use cases. When are we going to
> be using this monster? For many use cases, where the original reason
Where "implicit rollbacks" are desireable, I suppose. It is incompatible
with edit-in-place, anyway.
On Mon, Jan 3, 2011 at 4:25 AM, Ted Ts'o wrote:
> On Sun, Jan 02, 2011 at 04:14:15PM +0100, Olaf van der Spek wrote:
>>
>> Last time you ignored my response, but let's try again.
>> The implementation would be comparable to using a temp file, so
>> there's no need to keep 2 g in memory.
>> Write t
On Sun, Jan 2, 2011 at 7:55 PM, Adam Borowski wrote:
> Note that on the other side of the fence there's something called TxF
Not GA AFAIK.
> And what if you're changing one byte inside a 50 GB file?
> I see an easy implementation on btrfs/ocfs2 (assuming no other writers),
> but on ext3/4, that'
On Sun, Jan 2, 2011 at 6:14 PM, Henrique de Moraes Holschuh >> Maybe I
wasn't clear, in that case I'm sorry. To me, O_ATOMIC is
> Whether this should map to O_ATOMIC in glibc or be something new, I don't
> care. But if it is a flag, I'd highly suggest naming it O_CREATEUNLINKED or
> something else
On 02/01/11 17:37, Olaf van der Spek wrote:
A userspace lib is fine with me. In fact, I've been asking for it
multiple times. Result: no response.
Excuse me?
You (well, Henrique, but you were CCed) said "how about a user space
lib"? I said "I'm working on one, will be ready about this we
* Ted Ts'o schrieb:
> This is possible. It would be specific only to file systems that
> support inodes (i.e., ix-nay for NFS, FAT, etc.).
FAT supports inodes ?
IIRC it puts all file information (including attributes and first
data block) directly into the dirent ...
> Some file systems woul
On Sun, Jan 02, 2011 at 03:14:41PM -0200, Henrique de Moraes Holschuh wrote:
>
> 1. Create unlinked file fd (benefits from kernel support, but doesn't
> require it). If a filesystem cannot support this or the boundary conditions
> are unaceptable, fail. Needs to know the destination name to do t
On Sun, Jan 02, 2011 at 04:14:15PM +0100, Olaf van der Spek wrote:
>
> Last time you ignored my response, but let's try again.
> The implementation would be comparable to using a temp file, so
> there's no need to keep 2 g in memory.
> Write the 2 g to disk, wait one day, append the 1 k, fsync, up
On Sun, Jan 02, 2011 at 04:14:15PM +0100, Olaf van der Spek wrote:
> On Sun, Jan 2, 2011 at 8:09 AM, Ted Ts'o wrote:
> > The O_ATOMIC open flag is highly problematic, and it's not fully
> > specified.
Note that on the other side of the fence there's something called TxF
(Transactional NTFS). I d
On Sun, 02 Jan 2011, Olaf van der Spek wrote:
> On Sun, Jan 2, 2011 at 1:52 PM, Henrique de Moraes Holschuh
> wrote:
> > Olaf, O_ATOMIC is "difficult" in the kernel sense and in the long run. It
> > is an API that is too hard to implement in a sane way, with too many
> > boundary conditions.
> >
On Sun, Jan 2, 2011 at 1:52 PM, Henrique de Moraes Holschuh
wrote:
> Olaf, O_ATOMIC is "difficult" in the kernel sense and in the long run. It
> is an API that is too hard to implement in a sane way, with too many
> boundary conditions.
>
> OTOH, you don't need O_ATOMIC. You need a way for easy
On Sun, Jan 2, 2011 at 8:09 AM, Ted Ts'o wrote:
>> You could ask for a new (non-POSIX?) API that does not ask of a
>> POSIX-like filesystem something it cannot provide (i.e. don't ask for
>> something that requires inode->path reverse mappings). You could ask
>> for syscalls to copy inodes, etc.
On Sun, 02 Jan 2011, Ted Ts'o wrote:
> And of course, Olaf isn't actually offerring to implement this
> hypothetical O_ATOMIC. Oh, no! He's just petulently demanding it,
> even though he can't give us any concrete use cases where this would
> actually be a huge win over a userspace "safe-write" l
On Fri, Dec 31, 2010 at 09:51:50AM -0200, Henrique de Moraes Holschuh wrote:
> On Fri, 31 Dec 2010, Olaf van der Spek wrote:
> > Ah, hehe. BTW, care to respond to the mail I send to you?
>
> There is nothing more I can add to this thread. You want O_ATOMIC. It
> cannot be implemented for all use
On Sun, Jan 2, 2011 at 12:03 AM, Enrico Weigelt wrote:
> I doubt that the only proper solution. As said, an (userland)
> filesystem could also do fine.
Do you think distros like Debian would install such a setup by default?
Olaf
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.or
* Olaf van der Spek schrieb:
> > Doing it in the kernel would be fine (maybe DLM could be used here),
>
> What's DLM?
Distributed lock manager.
> > but would be a nonportable solution for quite a long time ;-o
>
> Since it's the only proper solution I don't think that's a problem.
I doubt th
On Sat, Jan 1, 2011 at 7:13 PM, Cyril Brulebois wrote:
> Olaf van der Spek (01/01/2011):
>> > Doing it in the kernel would be fine (maybe DLM could be used here),
>>
>> What's DLM?
>
> CONFIG_DLM.
DLM seems independent of atomic updates.
Olaf
--
To UNSUBSCRIBE, email to debian-devel-requ...@
Olaf van der Spek (01/01/2011):
> > Doing it in the kernel would be fine (maybe DLM could be used here),
>
> What's DLM?
CONFIG_DLM.
KiBi.
signature.asc
Description: Digital signature
On Fri, Dec 31, 2010 at 5:08 PM, Enrico Weigelt wrote:
>> Not true. Renaming a running executable works just fine, for example.
>
> Well, has been quite a while since I last used Windows, but IIRC
> renaming an running executable was denied.
Maybe on FAT. However, that's OT.
>> >> > Why not desi
* Olaf van der Spek schrieb:
> > renames on open files, as they're also identified by the filename.
>
> Not true. Renaming a running executable works just fine, for example.
Well, has been quite a while since I last used Windows, but IIRC
renaming an running executable was denied.
> >> > Why n
On Fri, Dec 31, 2010 at 3:58 PM, brian m. carlson
wrote:
>> These cases aren't that rare. Windows, for example, tends to deny
>> renames on open files, as they're also identified by the filename.
>> (yes, there're other solutions for this problem, eg. having some
>> internal-only inode numbering,
On Fri, Dec 31, 2010 at 3:44 PM, Enrico Weigelt wrote:
> * Olaf van der Spek schrieb:
>
>> > Well, they're an fundamental concept which sometimes *IS* significant
>> > to the applications. It's very different from systems where each
>> > file has exactly one name (eg. DOS/Windows) or where there'
On Fri, Dec 31, 2010 at 03:44:56PM +0100, Enrico Weigelt wrote:
> * Olaf van der Spek schrieb:
>
> > > Well, they're an fundamental concept which sometimes *IS* significant
> > > to the applications. It's very different from systems where each
> > > file has exactly one name (eg. DOS/Windows) or
* Olaf van der Spek schrieb:
> > Well, they're an fundamental concept which sometimes *IS* significant
> > to the applications. It's very different from systems where each
> > file has exactly one name (eg. DOS/Windows) or where there're just
> > filesnames that point to opaque stream objects tha
On Fri, Dec 31, 2010 at 2:57 PM, Enrico Weigelt wrote:
>> To me, inodes are an implementation detail that shouldn't be exposed.
>
> Well, they're an fundamental concept which sometimes *IS* significant
> to the applications. It's very different from systems where each
> file has exactly one name (
* Olaf van der Spek schrieb:
> > something that requires inode->path reverse mappings). You could ask
> > for syscalls to copy inodes, etc. You could ask for whatever is needed
>
> To me, inodes are an implementation detail that shouldn't be exposed.
Well, they're an fundamental concept which
On Fri, Dec 31, 2010 at 12:51 PM, Henrique de Moraes Holschuh
wrote:
> On Fri, 31 Dec 2010, Olaf van der Spek wrote:
>> Ah, hehe. BTW, care to respond to the mail I send to you?
>
> There is nothing more I can add to this thread. You want O_ATOMIC. It
That's a shame. I thought I provided pretty
On Fri, 31 Dec 2010, Olaf van der Spek wrote:
> Ah, hehe. BTW, care to respond to the mail I send to you?
There is nothing more I can add to this thread. You want O_ATOMIC. It
cannot be implemented for all use cases of the POSIX API, so it will not
be implemented by the kernel. That's all there
On Fri, Dec 31, 2010 at 3:17 AM, Henrique de Moraes Holschuh
wrote:
> On Thu, 30 Dec 2010, Henrique de Moraes Holschuh wrote:
>> BTW: safely removing a file is also tricky. AFAIK, one must open it RW,
>> in exclusive mode. stat it by fd and check whether it is what one
>> expects (regular file, o
On 30/12/10 17:02, Olaf van der Spek wrote:
Got a project page already?
Watch this space. Actual code coming soon(tm).
https://github.com/Shachar/safewrite
Shachar
--
Shachar Shemesh
Lingnu Open Source Consulting Ltd.
http://www.lingnu.com
--
To UNSUBSCRIBE, email to debian-devel-requ.
On Thu, 30 Dec 2010, Henrique de Moraes Holschuh wrote:
> BTW: safely removing a file is also tricky. AFAIK, one must open it RW,
> in exclusive mode. stat it by fd and check whether it is what one
> expects (regular file, ownership). unlink it by fd. close the fd.
Eh, as it was pointed to me b
On Thu, Dec 30, 2010 at 7:46 PM, Ben Hutchings wrote:
>> You're kidding me. Got any source to back this up?
>
> http://support.microsoft.com/?kbid=172190
Interesting. Although no longer available on Vista / 7.
Olaf
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subje
On Thu, 2010-12-30 at 19:29 +0100, Olaf van der Spek wrote:
> On Thu, Dec 30, 2010 at 7:15 PM, Shachar Shemesh wrote:
> > If my (extremely leaky) memory serves me right, Windows has it. It's called
> > "delete and then rename". It is not atomic (since when do Windows care about
> > not breaking st
On Thu, Dec 30, 2010 at 7:20 PM, Shachar Shemesh wrote:
>> Depending on /proc is probably not reasonable.
>> Are you sure it will be atomic? ;)
>>
>>
>
> open old file, get fd (we'll assume it's 5). Do readlink on /proc/self/fd/5,
> and get file's real path. Do everything in said path. It's atomic
On Thu, Dec 30, 2010 at 7:15 PM, Shachar Shemesh wrote:
> If my (extremely leaky) memory serves me right, Windows has it. It's called
> "delete and then rename". It is not atomic (since when do Windows care about
> not breaking stuff), but it does exactly that.
>
> If you delete a file and quickly
On 30/12/10 17:17, Olaf van der Spek wrote:
On Thu, Dec 30, 2010 at 4:12 PM, Shachar Shemesh wrote:
No. I was doing it as code to accompany an article on my company's site
about how it should be done. I was originally out to write the article, and
then decided to add code. A good thing, too
On 30/12/10 19:48, Henrique de Moraes Holschuh wrote:
It doesn't. You need a "copy inode without the file data" filesystem
interface to be able to do that in the first place. It might exist, but I
never heard of it.
If my (extremely leaky) memory serves me right, Windows has it. It's
c
On Thu, Dec 30, 2010 at 6:48 PM, Henrique de Moraes Holschuh
wrote:
>> Why not?
>
> You touched it, it is not the same file/inode anymore.
That's again a regression from the non-atomic case.
>> How does it handle meta-data you don't know about yet?
>
> It doesn't. You need a "copy inode without
On Thu, 30 Dec 2010, Olaf van der Spek wrote:
> On Thu, Dec 30, 2010 at 4:24 PM, Henrique de Moraes Holschuh
> wrote:
> > On Thu, 30 Dec 2010, Olaf van der Spek wrote:
> >> The reason I asked for a kernelland solution is because it's hard if
> >> not impossible to do properly in userland. But some
On 30/12/10 13:46, Henrique de Moraes Holschuh wrote:
Is there a code snippet or lib function that handles this properly?
I don't know. I'd be interested in the answer, though :-)
I'm working on one under the MIT license. Will probably release it by
the end of this week. Will a
On Thu, Dec 30, 2010 at 4:24 PM, Henrique de Moraes Holschuh
wrote:
> On Thu, 30 Dec 2010, Olaf van der Spek wrote:
>> The reason I asked for a kernelland solution is because it's hard if
>> not impossible to do properly in userland. But some kernel devs (Ted
>> and others) don't agree. They reaso
On Thu, Dec 30, 2010 at 4:20 PM, Henrique de Moraes Holschuh
wrote:
>> What if the target name is actually a symlink? To a different volume?
>
> Indeed. You have to check that first, of course :-( This is about safe
> handling of such functions, symlinks always have to be derreferenced and
> thei
On Thu, 30 Dec 2010, Olaf van der Spek wrote:
> The reason I asked for a kernelland solution is because it's hard if
> not impossible to do properly in userland. But some kernel devs (Ted
> and others) don't agree. They reason that the desire to preserve all
> meta-data isn't reasonable by itself.
On Thu, 30 Dec 2010, Olaf van der Spek wrote:
> On Thu, Dec 30, 2010 at 12:46 PM, Henrique de Moraes Holschuh
> wrote:
> > write temp file (in same directory as file to be replaced), fsync temp
>
> What if the target name is actually a symlink? To a different volume?
Indeed. You have to check t
On Thu, Dec 30, 2010 at 4:12 PM, Shachar Shemesh wrote:
> No. I was doing it as code to accompany an article on my company's site
> about how it should be done. I was originally out to write the article, and
> then decided to add code. A good thing, too, as recursively resolving
> symbolic links i
On 30/12/10 17:02, Olaf van der Spek wrote:
On Thu, Dec 30, 2010 at 3:51 PM, Shachar Shemesh wrote:
I'm working on one under the MIT license. Will probably release it by the
end of this week. Will also handle copying the permissions over and
following symlinks.
Sounds great!
Got a pr
On Thu, Dec 30, 2010 at 03:30:29PM +0100, Olaf van der Spek wrote:
> > name the temp file properly, and teach your program to clean old ones up
> > *safely* (see vim swap file handling for an example) when it starts.
>
> What about restoring meta-data? File-owner?
owner, permissions, acl, xattrs,
On Thu, Dec 30, 2010 at 3:51 PM, Shachar Shemesh wrote:
> I'm working on one under the MIT license. Will probably release it by the
> end of this week. Will also handle copying the permissions over and
> following symlinks.
Sounds great!
Got a project page already?
What aboue file owner? Meta-dat
On 30/12/10 13:46, Henrique de Moraes Holschuh wrote:
Is there a code snippet or lib function that handles this properly?
I don't know. I'd be interested in the answer, though :-)
I'm working on one under the MIT license. Will probably release it by
the end of this week. Will also hand
On Thu, Dec 30, 2010 at 12:46 PM, Henrique de Moraes Holschuh
wrote:
> write temp file (in same directory as file to be replaced), fsync temp
What if the target name is actually a symlink? To a different volume?
What if you're not allowed to create a file in that dir.
> If we could use some sys
On Wed, 29 Dec 2010, Olaf van der Spek wrote:
> Writing a temp file, fsync, rename is often proposed. However, the
It is:
write temp file (in same directory as file to be replaced), fsync temp
file[1], rename (atomic), fsync directory[2].
[1] Makes sure file data has been commited to backend
Since the introduction of ext4, some apps/users have had issues with
file corruption after a system crash. It's not a bug in the FS AFAIK
and it's not exclusive to ext4.
Writing a temp file, fsync, rename is often proposed. However, the
durable aspect of fsync isn't always required and this way has
76 matches
Mail list logo
