oh and knowing what type of fileystem you're on would help with recommendations.
On Fri, Jun 28, 2019 at 1:51 PM Michael Di Domenico <mdidomeni...@gmail.com> wrote: > > i'm not familiar with the imagenet set, but i'm suprised you'd see a > bottleneck. my understanding of the ML image sets is that they're > mostly read. do you have things like noatime set on the filesystem? > do you know specifically which ops are pounding the metadata? > > On Fri, Jun 28, 2019 at 1:47 PM Mark Hahn <h...@mcmaster.ca> wrote: > > > > Hi all, > > I wonder if anyone has comments on ways to avoid metadata bottlenecks > > for certain kinds of small-io-intensive jobs. For instance, ML on imagenet, > > which seems to be a massive collection of trivial-sized files. > > > > A good answer is "beef up your MD server, since it helps everyone". > > That's a bit naive, though (no money-trees here.) > > > > How about things like putting the dataset into squashfs or some other > > image that can be loop-mounted on demand? sqlite? perhaps even a format > > that can simply be mmaped as a whole? > > > > personally, I tend to dislike the approach of having a job stage tons of > > stuff onto node storage (when it exists) simply because that guarantees a > > waste of cpu/gpu/memory resources for however long the stagein takes... > > > > thanks, mark hahn. > > -- > > operator may differ from spokesperson. h...@mcmaster.ca > > _______________________________________________ > > Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing > > To change your subscription (digest mode or unsubscribe) visit > > https://beowulf.org/cgi-bin/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
