On 3/4/19 8:00 PM, Lux, Jim (337K) via Beowulf wrote:
I'm munging through not very much satellite telemetry (a few GByte), using sqlite3.. Here's some general observations: 1) if the data is recorded by multiple sensor systems, the clocks will *not* align - sure they may run NTP, but.... 2) Typically there's some sort of raw clock being recorded with the data (in ticks of some oscillator, typically) - that's what you can use to put data from a particular batch of sources into a time order. And then you have the problem of reconciling the different clocks. 3) Watch out for leap seconds in time stamps - some systems have them (UTC), some do not (GPS, TAI) - a time of 23:59:60 may be legal. 4) you need to have a way to deal with "missing" data, whether it's time tags, or actual measurements - as well as "gaps in the record" 5) Be aware of the need to de-dupe data - same telemetry records from multiple sources.
Being satellite data, I am assuming you have relativistic corrections to the time, depending upon orbit, accuracy of the clock, data analysis needs, etc. . [1][2]
Missing data, of various types may be handled in data frame packages. R, Julia, and I think Python can all handle this without too much pain.
[1] https://gssc.esa.int/navipedia/index.php/Relativistic_Clock_Correction [2] http://www.astronomy.ohio-state.edu/~pogge/Ast162/Unit5/gps.html -- Joe Landman e: joe.land...@gmail.com t: @hpcjoe w: https://scalability.org g: https://github.com/joelandman l: https://www.linkedin.com/in/joelandman _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
