On Wed, Oct 14, 2020 at 9:04 AM Oddo Da <oddodao...@gmail.com> wrote: > > Perhaps but I am seeing these at the same level of abstraction - the low > level where you have to spell everything out.
i'm not sure I understand what you expect. Message passing is a low level function of passing bits of memory around the network. i'm not sure how you can abstract that away from the user. if you're writing something to solve an equation you have to know where things are and where they're going. there are certainly plenty of math libraries that use MPI to solve complex equations where the user only has to understand the equation and not the paradigm of how it gets solved in hardware/software. > I see what you wrote above as contradicting yourself. First you say that > Spark/hadoop did not evolve from economic motivations but then you say that > it did. You say that Spark and MPI can be used interchangeably. OK if that's > the case, why does the data science/ML industry just not use MPI everywhere. > Why did they not start with MPI as the underlying paradigm and just build all > the tooling on top of it? I wish they did... ;-) cost was one factor that accelerated spark/hadoop, it's not the only or even the biggest factor. the ML folks didn't start with MPi because the AI frameworks were bred on workstations and then ported to non HPC hardware (aka cloud platforms) where MPI isn't the dominant paradigm. now that ML/AI is taking hold in the HPC community for different aspects and the models are starting to expand beyond the 4-8 gpus you can stick in a single box they are adding MPI underneath (look at horovad) to spread the models over multiple machines (scale out vs scale up). > I see MPI as a low level solution for a problem, at the abstraction level > where you need to spell everything out. It is like the comparison between C > and languages like Scala or Haskell or Julia. I am asking why there is no > progress on the latter in this scenario - we have the message passing > interface level of abstraction, why are we not interested in using this to > build tooling that is at the higher level, where we can hide the how and > focus on the what. In my opinion comparing MPI/C to Scala/Haskell isn't a fair comparison. Haskell was designed to be easy for mathematicians, just like R is for statistians. but both hide pretty much all of the code under the covers. That's great if all you want to do is solve equations and haskell already has the code to implement them. someone could certainly add MPI solvers to haskell and do what i think you're asking for. but someone has to be motivated to do that > 20 years ago, when I was an undergrad, I took a 200-level course in data > structures and algorithms and it was taught in a programming language called > Eiffel. The professor started with saying - "Eiffel is gaining in popularity, > there are many new books being written about Eiffel but none about C. Do you > know why?". I raised my hand and said "because C is an old language and > nothing has changed and everything that was to be said about it, was already > said about it in many previous books". At least in that domain we could > discuss these things - new languages and paradigms and tooling abound. In the > world of HPC, not so much. C has morphed into other things, julia/C++/C#/go/etc, but C isn't going to change unless the underlying hardware changes (ie quantum). it's also a matter of intellectual build, there's so much intellectual property in C around the world and no real reason to just rip it out. i don't have a horse in this race either. we can go back and forth all day, but i'm not sure i understand what you're ultimately try to get at (or i missed it). to state that HPC hasn't evolved in 20yrs i believe is a gross misrepresentation and to say that it's not evolved into something great like Spark or Scala, isn't a fair comparison. _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
