These would be numerical data such as integers or floating point numbers.
-Original Message-
From: Tony Brian Albers
Sent: 04 March 2019 08:04
To: beowulf@beowulf.org; Jonathan Aquilina
Subject: Re: [Beowulf] Large amounts of data to store and process
Hi Jonathan,
From my limited know
Hi Jonathan,
From my limited knowledge of the technologies, I would say that HBase
with file pointers to the files placed on HDFS would suit you well.
But if the files are log files, consider some tools that are suited for
analyzing those like Kibana.
/tony
On Mon, 2019-03-04 at 06:55 +, J
Hi Tony,
Sadly I cant go into much detail due to me being under an NDA. At this point
with the prototype we have around 250gb of sample data but again this data is
dependent on the type of air craft. Larger aircraft and longer flights will
generate a lot more data as they have more sensors and
On Mon, 2019-03-04 at 06:38 +, Jonathan Aquilina wrote:
> Good Morning all,
>
> I am working on a project that I sadly cant go into much detail but
> there will be quite large amounts of data that will be ingested by
> this system and would need to be efficiently returned as output to
> the en
Good Morning all,
I am working on a project that I sadly cant go into much detail but there will
be quite large amounts of data that will be ingested by this system and would
need to be efficiently returned as output to the end user in around 10 min or
so. I am in discussions with another partn
Thank you very much Greg,Douglas, John and Michael.
You very kindly "overwhelmed" me, and I thank you for that, with hints of
things I didn't know about. So my very next step will be to understand each
of your hints, and I guess I will come back with some more practical
questions about them.
In the
I Third OpenHPC, or at least the Warewulf underpinnings in it.
http://warewulf.lbl.gov/
For "learning" the software stack you may consider beefing up your current
node and running virtualized environment inside it? I use the community
version of Proxmox (https://www.proxmox.com/en/downloads). On
One thing to remember, a cluster is often defined by its environment
and goals.
1. HPC type clusters, as discussed on this list, operate in
a particular way (cluster nodes are provisioned in a reproducible
way, job scheduler that provides dynamic resources
to users, clusters are optimized for pro
On Sun, 3 Mar 2019 09:28:03 +, you wrote:
>ps. If you are interested in parallelism... there is Julia.
>https://docs.julialang.org/en/v1/manual/parallel-computing/index.html
Also Chapel https://chapel-lang.org/
___
Beowulf mailing list, Beowulf@beo
ps. If you are interested in parallelism... there is Julia.
https://docs.julialang.org/en/v1/manual/parallel-computing/index.html
I would also advise setting up just one server and install the latest
CentOS.
You can start with some tutorials on MPI - which is the current standard
for parallelism.
I second OpenHPC. It is actively maintained and easy to set up.
Regarding the hardware, have a look at Doug Eadlines Limulus clusters. I
think they would be a good fit.
Dougs site is excellent in general https://www.clustermonkey.net/
Also some people build Raspberry Pi clusters for learning.
O
11 matches
Mail list logo
