Re: [Beowulf] Station wagon full of tapes

2009-05-26 Thread Greg Lindahl
On Tue, May 26, 2009 at 09:54:15AM -0400, Chris Dagdigian wrote: > - Once we process the data to get the derived results, the primary data > just needs to go somewhere cheap If you only rarely re-read the primary data, I'd think a stack of SATA drives in a cabinet would probably do the trick. Tw

Re: [Beowulf] Station wagon full of tapes

2009-05-26 Thread Nifty Tom Mitchell
On Tue, May 26, 2009 at 03:24:26PM -0400, Robert G. Brown wrote: > >> The comparison of "fixings" vs. the deli goes a long way. . > And even on the smaller scale, the savings add up. That's why we > parents nag our kids to go into the damn kitchen and fix yourself a > sandwich instead of expec

Re: [Beowulf] MPI - time for packing, unpacking, creating a message...

2009-05-26 Thread Nifty Tom Mitchell
On Sat, May 23, 2009 at 12:43:57PM -0300, Bruno Coutinho wrote: > >If you are using Gigabit Ethernet with jumbo frames (9000 bytes for >example): >A will send 3 packets with 4000 bytes and >B will send one of 9000 bytes and one of 7000 bytes. >For the cpu B is better, because w

Re: [Beowulf] Station wagon full of tapes

2009-05-26 Thread Robert G. Brown
On Tue, 26 May 2009, Nifty Tom Mitchell wrote: The comparison of "fixings" vs. the deli goes a long way. How close in time and space is the deli/ market. Do you eat the same thing each day. If there are multiple people do they eat the same, do they eat left overs. Cleanup sanitation -- do you ha

Re: [Beowulf] Station wagon full of tapes

2009-05-26 Thread Andrew M.A. Cater
On Tue, May 26, 2009 at 12:26:19PM -0400, Robert G. Brown wrote: > On Tue, 26 May 2009, Chris Dagdigian wrote: > > Yeah, well, stupidity is a universal problem, even in the > government...;-) But this is why CBAs and smart people (working > together) are so important. > _Especially_ in the governm

Re: [Beowulf] Station wagon full of tapes

2009-05-26 Thread Greg Keller
On May 26, 2009, at 10:20 AM, "Robert G. Brown" Subject: Re: [Beowulf] Station wagon full of tapes To: Jeff Layton Cc: Beowulf Mailing List Message-ID: Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed On Tue, 26 May 2009, Jeff Layton wrote: I haven't seen the cloud ready yet for

Re: [Beowulf] Station wagon full of tapes

2009-05-26 Thread Nifty Tom Mitchell
On Tue, May 26, 2009 at 12:19:11PM -0400, Joe Landman wrote: > Robert G. Brown wrote: > >> Sure, but why wouldn't it be cheaper for e.g. NSF or NIH to fund an >> exact clone of the service Amazon plans to offer and provide it for free >> to its supported research groups (or rather, do bookkeeping b

Re: [Beowulf] Should I go for diskless or not?

2009-05-26 Thread Gus Correa
Hi Dr. Cool Santa Dr Cool Santa wrote: @Gus: According to the documentation, Schrodinger software should only run good on MPICH1, otherwise I'd have preferred MPICH2. First off, I know nothing about computational chemistry. Hence, consider my suggestions at your own risk. Did you check with

RE: [Beowulf] Station wagon full of tapes

2009-05-26 Thread Lux, James P
Rgb wrote: > > Sure, but why wouldn't it be cheaper for e.g. NSF or NIH to > fund an exact clone of the service Amazon plans to offer and > provide it for free to its supported research groups (or > rather, do bookkeeping but it is all internal bookkeeping, > moving money from one pocket to an

Re: [Beowulf] Station wagon full of tapes

2009-05-26 Thread Robert G. Brown
On Tue, 26 May 2009, Chris Dagdigian wrote: I deal quite often with the "next-gen" DNA sequencing instruments that produce 1TB/day in TIFF images that are then distilled down to the DNA basecalls before the short reads are subjected to alignment. Then the resulting longer sequences are usual

Re: [Beowulf] Station wagon full of tapes

2009-05-26 Thread Robert G. Brown
On Tue, 26 May 2009, Chris Dagdigian wrote: The flip side to your arguments is that I may not want my tax dollars spent on allowing the NIH to operate peta-scale data repositories. I can't be more specific than this -- my most recent exposure to a large government life science directorate re

Re: [Beowulf] Station wagon full of tapes

2009-05-26 Thread Charlie Peck
On May 26, 2009, at 11:16 AM, Robert G. Brown wrote: Sure, but why wouldn't it be cheaper for e.g. NSF or NIH to fund an exact clone of the service Amazon plans to offer and provide it for free to its supported research groups (or rather, do bookkeeping but it is all internal bookkeeping, mov

[Beowulf] Re: Station wagon full of tapes

2009-05-26 Thread David Mathog
Chris Dagdigian wrote: > We are only a few > technology revolutions away from these boxes showing up in your point > of care primary physician's office (well not really, probably a > backend service lab that your physician outsources to ...) Could be, but this (raw) medical data will not b

Re: [Beowulf] Station wagon full of tapes

2009-05-26 Thread Joe Landman
John Hearns wrote: There was a discussion recently on cloud computing, and how effective it would be for HPC. I made the comment that the time for getting large data sets to/from the cloud provider would be a limiting factor. I think Amazon was listening to me: http://www.theregister.co.uk/2009/

Re: [Beowulf] Station wagon full of tapes

2009-05-26 Thread Joe Landman
Robert G. Brown wrote: Sure, but why wouldn't it be cheaper for e.g. NSF or NIH to fund an exact clone of the service Amazon plans to offer and provide it for free to its supported research groups (or rather, do bookkeeping but it is all internal bookkeeping, moving money from one pocket to anot

Re: [Beowulf] Station wagon full of tapes

2009-05-26 Thread Joe Landman
Chris Dagdigian wrote: The flip side to your arguments is that I may not want my tax dollars spent on allowing the NIH to operate peta-scale data repositories. I can't be more specific than this -- my most recent exposure to a large government life science directorate revealed that they were

Re: [Beowulf] Station wagon full of tapes

2009-05-26 Thread Bogdan Costescu
On Tue, 26 May 2009, Gerry Creager wrote: How can I get sufficient cloud resources for computing... ... that will handle reasonable weather models with their small message MPI chatter, and lots of file I/O? By asking Amazon to equip their cloud nodes with a HPC interconnect or to have fat no

Re: [Beowulf] Station wagon full of tapes

2009-05-26 Thread Robert G. Brown
On Tue, 26 May 2009, Jeff Layton wrote: I haven't seen the cloud ready yet for anything other than embarrassingly parallel codes (i.e. since node, small IO requirements). Has anyone seen differently? (as an example of what might work, CloudBurst seems to be gaining some traction - doing sequenci

Re: [Beowulf] Station wagon full of tapes

2009-05-26 Thread Chris Dagdigian
Hate to promote commercial efforts here but this thread has a timely link to an Amazon AWS event happening in NYC this week. For people in the area who are interested in grilling the AWS product people and techies about their offerings this is probably a good forum to do so. I know from p

Re: [Beowulf] Station wagon full of tapes

2009-05-26 Thread Mike Davis
Jeff Layton wrote: I haven't seen the cloud ready yet for anything other than embarrassingly parallel codes (i.e. since node, small IO requirements). Has anyone seen differently? (as an example of what might work, CloudBurst seems to be gaining some traction - doing sequencing in the cloud. The

Re: [Beowulf] Station wagon full of tapes

2009-05-26 Thread Chris Dagdigian
The flip side to your arguments is that I may not want my tax dollars spent on allowing the NIH to operate peta-scale data repositories. I can't be more specific than this -- my most recent exposure to a large government life science directorate revealed that they were spending $500K/year

Re: [Beowulf] Station wagon full of tapes

2009-05-26 Thread Chris Dagdigian
I deal quite often with the "next-gen" DNA sequencing instruments that produce 1TB/day in TIFF images that are then distilled down to the DNA basecalls before the short reads are subjected to alignment. Then the resulting longer sequences are usually aligned again against a reference geno

Re: [Beowulf] Station wagon full of tapes

2009-05-26 Thread Jeff Layton
Gerry Creager wrote: There was an interesting brainstorming session at Rocks-A-Palooza a couple of weeks ago. Someone wants to offer Amazon resources. Problem remains for me: How can I get sufficient cloud resources for computing (I'll hammer on dataset transport in a moment) that will handl

Re: [Beowulf] Station wagon full of tapes

2009-05-26 Thread Gerry Creager
There was an interesting brainstorming session at Rocks-A-Palooza a couple of weeks ago. Someone wants to offer Amazon resources. Problem remains for me: How can I get sufficient cloud resources for computing (I'll hammer on dataset transport in a moment) that will handle reasonable weather m

RE: [Beowulf] MPI - time for packing, unpacking, creating a message...

2009-05-26 Thread Dan.Kidger
The original question was about relatively small messages - only 500 bytes each You can often get better throughput if you send say two smaller messages rather than one large one. This is since the interconnect can generate multiple RDMA requests that can proceed concurrently. This old paper fr

[Beowulf] Station wagon full of tapes

2009-05-26 Thread John Hearns
There was a discussion recently on cloud computing, and how effective it would be for HPC. I made the comment that the time for getting large data sets to/from the cloud provider would be a limiting factor. I think Amazon was listening to me: http://www.theregister.co.uk/2009/05/22/amazon_cloud_dr