On Wed, 3 Jun 2009, John Hearns wrote:
Quite often the architecture of storage is a secondary consideration, in the rush to get a Shiny New Fast machine on site and working.
Well, I've seen it ignored even outside of that rush - in the design phase. And I confess of being guilty of doing this as well, but I learn from mistakes :-)
In HPC, there are a lot of advantages in a central clustered filesystem, where you can prepare your input data, run the simulation, then at the end visualize the data.
In theory, this sounds nice, but in practice it can prove to be a bit more difficult, most times the human factor being the main culprit (just like with the scenarios I presented earlier). Administrative issues (who owns what) can seriously affect the possibility of coupling the HPC and visualisation resources, leading often to duplication of data. Stupid sysadmins or policies can leave the HPC resource with very basic text editors or terminal settings, leading the users to create the input set on their own workstation and constantly copying it over. Only the actual running of the simulation can be tightly linked to the clustered file system...
The systems administrator must put in place strong policies on this - leave your data on the fast storage, it gets deleted after N weeks.
I see duplication of data in almost all cases as a human behaviour problem, not a technical one, which needs human behaviour solutions and not technical ones, so policies are a good solution. But I would argue that users' education is an even better solution - teach them why copying of data is bad and give them easy ways of safely sharing data with their collaborators and not only they will keep the file systems emptier but they will also thank you for the decrease in effort needed to manage the always increasing amounts of data. Such a solution is however only feasible for smaller groups - f.e. an HPC center offering services to several (many) universities won't be able to convince all its users to take time and think about data management and a virtual sucker rod is not as efficient as a real one, so policies would still be required...
-- Bogdan Costescu IWR, University of Heidelberg, INF 368, D-69120 Heidelberg, Germany Phone: +49 6221 54 8240, Fax: +49 6221 54 8850 E-mail: bogdan.coste...@iwr.uni-heidelberg.de _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
