On 11/25/2013 07:11 PM, Adam DeConinck wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
4. I went to a BoF on ROI on HPC investment. All the presentations in
the BoF frustrated me. Not because they were poorly done, but because
they tried to measure the value of a cluster by number of papers
published that used that HPC resource. I think that's a crappy, crappy
metric, but haven't been able to come up with a better one myself yet. I
was very vocal with my comments and criticisms of the presentations, so
if any of the presenters are reading this now, I apologize for
hi-jacking your BoF. Getting good ROI on a cluster is close to my heart,
but is also difficult to quantify and measure. I hope I can be part of
the discussion next year.
Do you have any thoughts you can share on what alternative metrics
might look like, even if you can't think of one that's clearly better?
Unfortunately, I have no better suggestions at this time. The best
metric will vary depending on what your priorities and business goals
are. I've had some discussions on different metrics for different goals,
but nothing fruitful/concrete has come out of them. Too nebulous.
Many people just want to know if their cluster is being fully utilized
and how full the queue is, under the assumption that if the existing
hardware is at capacity, that's enough justification for spending more
money on more hardware, or proves that the cluster was needed. However,
I think even utilization of an HPC resource can be misleading, since
someone could be doing something stupid in their codes that might be
keeping the processors busy, but isn't really helping them get a
solution any quicker. Also, this doesn't measure the opportunity cost of
researcher swho want to use the resource but can't because they don't
know how, or don't even know it's available to them.
Right now, the only answer I can think of regarding system utilization
is "ask your cluster system admin - he sees everything that goes on, and
talks to the users needing support, so he probably has a good feel for
what's going on on your systems." Unfortunately, that doesn't make for
pretty Powerpoint slides.
I have no horse in this race as I've been doing industry HPC for the
past few years, but I'm curious what good metrics for ROI on an academic
or lab cluster might be. Total number of papers? Number of
citations after an N-year time window? [shrug]
There are two metrics that are often discussed in academia: The number
of papers published based on that research, or the amount of grant money
brought in by researchers using that resource. There is a paper Amy Apon
and other on this topic, titled "High Performance Computing
Instrumentation and Research Productivity in U.S.
Universities":
http://www.jiti.net/v10/jiti.v10n2.087-098.pdf
ROI measurement can sometimes be difficult even in an industrial or
commercial setting, especially if the HPC resource is used for R&D or
"engineering support" as opposed to something that feeds directly into
the product.
It can be difficult, but I think it's still much easier than in an
academic setting, where many consider the labor of grad students to be
free, and it's difficult to put a price on the value of research results.
Cheers,
Adam
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (Darwin)
iQEcBAEBCgAGBQJSk+cxAAoJEGqxms5cZfz0vEcH+gLqL/LmLZgDUcASJGn1U+M0
SbHbwFTbA6DOeMym4gAfvfXrKueabFYA4qHPmAaCoC2XVhCr6vORMLIn27hoA0S7
0dI+hN0H4OUC9u9Xgp9HZtPa2YD1nlYSby6A3D751VDW5m1r4Sn3aafoODyZGkcp
7FyAPhCnK+EgUinICgnfaXVMvCsUJ6cMMi2OicKjLPw85KIkRMfyDbAgh3myE7p2
S6SHKBb3PHzWqIAUZ2iEU+C1XJbwH7bw83ciOJN49T0PPX5+I1Lced76RWcIZC40
YqYdF1kUhBBD7VMonByUBAcCRq7fl9cCLDnEh78z3pZ7+/4dqJcmmMjqyyavrmA=
=xgE9
-----END PGP SIGNATURE-----
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
