Re: [Beowulf] Large amounts of data to store and process

2019-03-14 Thread Jonathan Aquilina
I think what I was getting at is why not include the current HPC practices to every day desktops in the sense since we are reaching certain limits and have to write code to take advantage of more and more cores. Why not use MPI and the like to help distribute the software side of things to the c

[Beowulf] lsf job packing on lsf8? Any old-school lsf8 guru's here?

2019-03-14 Thread Robert Taylor
Hi All. We are on a old version of LSF where I am, lsf8. We are trying to implement job packing on a particular queue, and I find some references online to using RES_REQ = order[-slots] in the lsb.queues file, and SCHED_PER_JOB_SORT=Y in lsb.params However, this doesn't work, as they appear to o

Re: [Beowulf] Large amounts of data to store and process

2019-03-14 Thread Jeffrey Layton
Damn. I knew I forgot something. Now where are my glasses. On Thu, Mar 14, 2019, 17:17 Douglas Eadline wrote: > > > I don't want to interrupt the flow but I'M feeling cheeky. One word can > > solve everything "Fortran". There I said it. > > Of course, but you forgot "now get off my lawn" > > --

Re: [Beowulf] Large amounts of data to store and process

2019-03-14 Thread Douglas Eadline
> I don't want to interrupt the flow but I'M feeling cheeky. One word can > solve everything "Fortran". There I said it. Of course, but you forgot "now get off my lawn" -- Doug > > Jeff > > > On Thu, Mar 14, 2019, 17:03 Douglas Eadline wrote: > >> >> > Then given we are reaching these limitati

Re: [Beowulf] Large amounts of data to store and process

2019-03-14 Thread Jeffrey Layton
I don't want to interrupt the flow but I'M feeling cheeky. One word can solve everything "Fortran". There I said it. Jeff On Thu, Mar 14, 2019, 17:03 Douglas Eadline wrote: > > > Then given we are reaching these limitations how come we don’t > integrate > > certain things from the HPC world

Re: [Beowulf] Large amounts of data to store and process

2019-03-14 Thread Douglas Eadline
> Then given we are reaching these limitations how come we don’t integrate > certain things from the HPC world into every day computing so to speak. Scalable/parallel computing is hard and hard costs time and money. In HPC the performance often justifies the means, in other sectors the cost mus

Re: [Beowulf] Large amounts of data to store and process

2019-03-14 Thread Prentice Bisbal via Beowulf
Then given we are reaching these limitations how come we don’t integrate certain things from the HPC world into every day computing so to speak. We have. How many cores does your smartphone have? But in most cases over the past 25 years, HPC has been about incorporating every day computing in

Re: [Beowulf] Large amounts of data to store and process

2019-03-14 Thread Jonathan Aquilina
Then given we are reaching these limitations how come we don’t integrate certain things from the HPC world into every day computing so to speak. On 14/03/2019, 19:14, "Douglas Eadline" wrote: > Hi Douglas, > > Isnt there quantum computing being developed in terms of CPUs at th

Re: [Beowulf] Large amounts of data to store and process

2019-03-14 Thread Douglas Eadline
> Hi Douglas, > > Isnt there quantum computing being developed in terms of CPUs at this > point? QC is (theoretically) unreasonably good at some things at other there may me classic algorithms that work better. As far as I know, there has been no demonstration of "quantum supremacy" where a quant

[Beowulf] Quantum computing

2019-03-14 Thread John Hearns via Beowulf
I think this should have a new thread. I have taken a bit of an interest in quantum computing recently. There are no real qubit based quantum computers which are ready for work at the moment. There ARE demonstrators available from IBM etc. The most advanced machine which is available for work is

Re: [Beowulf] Large amounts of data to store and process

2019-03-14 Thread Jonathan Aquilina
Hi Douglas, Isnt there quantum computing being developed in terms of CPUs at this point? Also is it really about the speed any more rather then how optimized the code is to take advantage of the multiple cores that a system has? On 13/03/2019, 22:22, "Douglas Eadline" wrote: I reali

Re: [Beowulf] Large amounts of data to store and process

2019-03-14 Thread Jonathan Aquilina
Then that is false advertising on the website if it has JIT. You mention growing old while you wait for it to plot how many processes did you start it with? From: John Hearns Date: Thursday, 14 March 2019 at 17:44 To: Jonathan Aquilina Cc: Beowulf Mailing List Subject: Re: [Beowulf] Large amo

Re: [Beowulf] Large amounts of data to store and process

2019-03-14 Thread John Hearns via Beowulf
Jonathan, there is absolutely no need for an apology. Please let me explain. Julia is often referred to as a JIT compilation language. This has bad connotations with most people - it means 'slow'. Now lets be honest - if you have ever tried to do plots in Julia you will end up growing old waiting f

Re: [Beowulf] Large amounts of data to store and process

2019-03-14 Thread Jonathan Aquilina
I do apologize there but I think what is JIT is JuliaDB side of things. Julia has a lot of potential for sure will be interesting to see how it develops as the little I have already played with it im really liking it. From: Beowulf on behalf of John Hearns via Beowulf Reply-To: John Hearns D

Re: [Beowulf] Large amounts of data to store and process

2019-03-14 Thread Douglas Eadline
> I agree with your take about slower progress on the hardware front and > that > software has to improve. DOE funds several vendors to do research to > improve technologies that will hopefully benefit HPC, in particular, as > well as the general market. I am reviewing a vendor's latest report on

Re: [Beowulf] Large amounts of data to store and process

2019-03-14 Thread John Hearns via Beowulf
Jonathan, a small correction if I may. Julia is not JIT - I asked on the Julia discourse. A much better description is Ahead of Time compilation. Not really important, but JIT triggers a certain response with most people. On Thu, 14 Mar 2019 at 07:31, Jonathan Aquilina wrote: > Hi All, > > > >

Re: [Beowulf] Large amounts of data to store and process

2019-03-14 Thread Jonathan Aquilina
Hi All, What sets Julia apart is it is not a compiled language but a Just In Time (JIT) language. I am still getting into it but it seems to be geared to complex and large data sets. As mentioned previously I am still working with a colleague on this prototype. With Julia at least there is an I