Hello, On Fri, Sep 12, 2008 at 7:37 PM, <[EMAIL PROTECTED]> wrote:
> A lot of these design considerations hinge on the question how to handle > recursion, i.e. what behaviour is desired if we have a dynamic > translator, and do further magic lookups. > > When doing magic lookups on a dynamic translator exporting a directory > tree, it's rather obvious that we want further magic lookups to work: > Say we use Flavio's unzip translator to look into an OpenOffice > document, we might want to use an XML translator on the individual files > in there. Insert any number of other examples. > > The trickier question is what to do about lookups a dynamic translator > does on the underlying filesystem. (This was actually my main motivation > for suggesting that shadow translators would do lookups on the nsmux > filesystem -- this way, it should work without any special handling.) > > Logically I think we do want that as well: If we use nsmux, it should > work just as if we used a filesystem that introduces namespace-based > translator selection direcly in the standard lookup mechanism, i.e. we > really want *all* lookups to have magic handling. > Do you mean by this that all lookups should pass through nsmux (or the corresponding shadow translator) and be performed on the virtual nodes provided by nsmux? (I'm asking to see whether I understand things correctly) > OTOH, I can't think of any specific case of a translator doing lookups > on the underlying filesystem, using directly or indirectly user-provided > (and thus potentially magic) file names. I'm not saying use cases do not > exist; but they are probably rare, so I'm not sure how much we should > care... Though definitely I feel some uneasiness about just ignoring the > matter. > It seems to me that we shouldn't ignore this matter. Frankly speaking, I think it is a good idea for a translator to use its underlying node for lookups, too (although I don't know any examples either). BTW, I'm not quite sure now that it's really useful to have dynamic > translators continue working on shadow nodes when nsmux goes away. nsmux > is effectively serving as the main filesystem on which the dynamic > translators are invoked; and in other situations, translators do not > have any protection against the underlying filesystem going away > either... If they don't need the underlying filesystem, they will be > fine with being orphaned; otherwise, they produce failures. I was going to ask you several days ago how did you estimate the probability of the situation that nsmux was asked to go away and there were still sane and healthy clients wanting to use their dynamic translator stacks :-) My opinion is that such functionality is not critical, although we should probably try to implement it some time, it being a very interesting feature. Considering that external shadow translators do not turn out as simple > as I had first hoped either, I'm not sure anymore whether it is a useful > idea at all. I haven't abandoned the idea alltogether yet, but probably > we shouldn't spend to much thought on it right now... > I don't expect external shadow translators to be very sophisticated (although I pretty well may have an erroneous opinion :-) ). I think that a shadow translator should be implemented as a one-node libnetfs-based translator; in this case many details of its functionality can just be migrated from the existing nsmux code, with but minor changes. There is another point that popped up during these considerations: What > if a client for some reason does multiple lookups on the same magic > filename? With the current approach, a new dynamic translator is created > every time. While this should pose no semantic problems, if all > translators are well designed (don't have any temporary state not > reflected in shared storage), it could be quite inefficient in extreme > cases. > > I wonder thus whether it wouldn't be advisable to share dynamic > translators resulting from equivalent lookups. But this would again pose > problems with badly designed translators, that behave differently with a > single instance shared between clients than individual instances for > each client... > > OTOH, I already found other situations in the past where we might have a > great many instances of the same translator invoked in parallel or > successively -- so I guess this is really an issue that needs to be > addressed on a more generic level sooner or later... > I've been thinking about this problem for about a month already, and my opinion is that we should share dynamic translators. I wonder whether it is nsmux's responsibility to care about badly designed translators. If dynamic translators would not be shared, it could indeed result in considerable overheads, and the worst thing is that these expenses would appear in the cases of well-designed translators, too. > In any case, I think this is an issue worth putting on a ToDo list [...] > Already done :-) > > * The nodes with translator stacks on them should not go away until > > there are clients using these translator stacks. > > > > * No specific node caching should be done in nsmux, but rather the > > nodes should be created on demand and destroyed, when they are not > > used already (the destruction should happen precisely when the port to > > the topmost translator in the stack will receive a no_senders > > notification) > > > > * In the current implementation of nsmux, the shadow nodes are > > maintained by nsmux, therefore all the translator stacks should be > > shut down *before* nsmux goes away when requested, lest the > > translators should become orphaned. > > > > * A netfs node cannot go away when all references to it are gone, but > > there are senders using the port to the topmost translator in the > > translator stack sitting on this node. To make this possible, there > > should be a way to keep this node alive, probably in a more explicit > > fashion, than a node cache might have done it. > [...] > > * A shadow node may go away in case the real filesystem node it is > > mirroring goes away. > > I don't really understand all of these points: some of them seem to > contradict what I actually said, so I'm not sure there hasn't been some > serious misunderstanding in the original discussion... Let me restate > the requirements in my own words: > > The dynamic translators are created upon client request, and should thus > never go away, unless they are not needed by the client anymore. The > only exception is the underlying node going away for a translator that > actually needs the underlying node, in which case it's best for the > translator also to go away immediately, so the clients immediately get > the information, just as if they had been using the underlying node > directly. Sincerely, I cannot really see how the items I listed contradict what you say here... :-) Anyway, thanks a lot for dwelling upon the details -- it did contribute to my better understanding of the matter. While we can make sure these policies are met by any translators > developed as part of the project, we have no control of external > translators we launch of course -- they might stay around longer than > necessary... > > However, all normal translators will just go away after a couple of > minutes, if there are no outstanding client connections. The delay of > several minutes is actually not necessary for dynamic translators, as > presently they are never reused, so once the original client drops the > reference, we *know* the translator won't be used anymore. It might be > useful in this case to tell the translator explicitely to go away, so it > won't waste resources sitting around -- but I don't think this is very > important, so the ability or non-ability to do so probably should not > affect any other design decisions... > I've put a corresponding item on my TODO list. I think this problem should be addressed quite soon, because having useless processes running makes me feel uncomfortable. All in all, as a first approximation, I think we should *never* > explicitely destroy dynamic translators -- anything else is just > optimization. > OK, I understand that, but still, shouldn't the dynamic translators go away when nsmux goes away? (IIRC, this is what we decided to be the best choice in case of an nsmux without external shadow translators) > > * In the current implementation of nsmux the filter will ask for the > > underlying port to be opened with O_NOTRANS. However, the port the > > filter will get will point to a netfs node, not to the real filesystem > > node. It means, that when the filter will try to trace the translator > > stack, it will find itself only (in the case of 'file,,-u', for > > instance). It will not see the static translator stack on the real > > filesystem. (This problem is, probably, only specific to the current > > implementation of nsmux). > > Not sure what you mean saying that this problem is specific to the > current implementation... It is obviously wrong -- it must (and can) be > fixed, no matter what the implementation. > I used to think that we are going to abandon the existing implementation in the favour of the architecture with external shadow translators, and it seemed to me that this problem is specific of the current implementation. No doubt, I will fix it. > * Why does a shadow translator have to be invisible on the translator > > stack? I don't think a filter will ever want to ignore a shadow > > translator, since it is pretty useless. And nobody, save for a filter, > > will possibly ever require the underlying node opened in O_NOTRANS > > fashion. > > I'm not sure about the possibility of cases where it is actually > necessary for the filter (or any other translator using this mechanism) > not to be aware of the fact that it sits on a shadow node... > > But that is not really the point. It's not as much about hiding the > shadow translator, as about showing the original translator stack > *instead* -- which obviously requires special handling in the parent > filesystem (nsmux). > Ah, I see. This is obviously connected with the filter translator problem I mentioned before. All right, I have several ideas, which could be useful in achieving the required goal. > * Suppose nsmux is asked for 'file,,x'. It will set the shadow > > translator on a copy of its root node and will tell the shadow > > translator to mirror the file 'file'. Then it will set translator 'x' > > on top of the shadow translator. Do I understand everything right? > > Yes, that *was* the idea... > OK :-) May I deduce from the fact that you stress the Past Tense in your words that this mechanism is likely to suffer modifications in the near future? Sorry for the delay in answering... I'm afraid, Thursday and Friday (and a bit of Sunday) will be the only days on which I will be able to regularly afford a decent period of time spent thinking and/or working on nsmux... (I will try to find extra time on other days, but it will happen on a rather accidental basis). scolobb
