Re: telemetry data retention strategy

2012-10-18 Thread deinspanjer
On Wednesday, August 15, 2012 4:06:38 PM UTC-7, Dave Mandelin wrote: > On Wednesday, August 15, 2012 2:03:38 PM UTC-7, Taras Glek wrote: > > > Hi, > > > > > > According to metrics we have about 1TB of telemetry data in hadoop. This > > > is almost a year worth of telemetry data. Our telemetry

Re: telemetry data retention strategy

2012-08-15 Thread Dave Mandelin
On Wednesday, August 15, 2012 2:03:38 PM UTC-7, Taras Glek wrote: > Hi, > > According to metrics we have about 1TB of telemetry data in hadoop. This > is almost a year worth of telemetry data. Our telemetry ping packets > keep growing as we add more probes. As the hadoop database gets bigger, >

Re: telemetry data retention strategy

2012-08-15 Thread Taras Glek
On 8/15/2012 2:42 PM, Benjamin Smedberg wrote: On 8/15/2012 5:03 PM, Taras Glek wrote: Hi, According to metrics we have about 1TB of telemetry data in hadoop. This is almost a year worth of telemetry data. Our telemetry ping packets keep growing as we add more probes. As the hadoop database gets

Re: telemetry data retention strategy

2012-08-15 Thread Benjamin Smedberg
On 8/15/2012 5:03 PM, Taras Glek wrote: Hi, According to metrics we have about 1TB of telemetry data in hadoop. This is almost a year worth of telemetry data. Our telemetry ping packets keep growing as we add more probes. As the hadoop database gets bigger, query times get worse, etc. We need

telemetry data retention strategy

2012-08-15 Thread Taras Glek
Hi, According to metrics we have about 1TB of telemetry data in hadoop. This is almost a year worth of telemetry data. Our telemetry ping packets keep growing as we add more probes. As the hadoop database gets bigger, query times get worse, etc. We need to decide on what data we can throw away