Re: Data lost in Cassandra 3.5 single instance via Erlang driver

2016-06-15 Thread Eric Stevens
As a side note, if you're inserting records quickly enough that you're potentially doing multiple in the same millisecond, it seems likely to me that your partition size is going to be too large at a day level unless your writes are super bursty: ((appkey, pub_date), pub_timestamp). You might need

Re: Data lost in Cassandra 3.5 single instance via Erlang driver

2016-06-15 Thread linbo liao
Thanks Ben, Paul, Alain. I debug at client side find the reason is pub_timestamp duplicated. I will use timeuuid instead. Thanks, Linbo 2016-06-15 13:09 GMT+08:00 Alain Rastoul : > On 15/06/2016 06:40, linbo liao wrote: > >> I am not sure, but looks it will cause the update other than insert.

Re: Data lost in Cassandra 3.5 single instance via Erlang driver

2016-06-14 Thread Alain Rastoul
On 15/06/2016 06:40, linbo liao wrote: I am not sure, but looks it will cause the update other than insert. If it is true, the only way is request includes IF NOT EXISTS, inform the client it failed? Thanks, Linbo Hi Linbo, +1 with what Ben said, timestamp has a millisecond precision and is a

Re: Data lost in Cassandra 3.5 single instance via Erlang driver

2016-06-14 Thread Paul Fife
If pub_timestamp could possibly match I'd suggest making it a timeuuid type instead. With the above schema it's not a failure or data loss if the timestamp is duplicated - your writes all probably made it - the duplicates just got overwritten. On Tue, Jun 14, 2016 at 9:40 PM, linbo liao wrote:

Re: Data lost in Cassandra 3.5 single instance via Erlang driver

2016-06-14 Thread linbo liao
I am not sure, but looks it will cause the update other than insert. If it is true, the only way is request includes IF NOT EXISTS, inform the client it failed? Thanks, Linbo 2016-06-15 10:59 GMT+08:00 Ben Slater : > Is it possible that your pub_timestamp values are colliding (which would > resu

Re: Data lost in Cassandra 3.5 single instance via Erlang driver

2016-06-14 Thread Ben Slater
Is it possible that your pub_timestamp values are colliding (which would result in an update rather than an insert)? On Wed, 15 Jun 2016 at 12:55 linbo liao wrote: > Hi, > > I use Erlang driver to send data to Cassandra, do testing at local > environment meet data lost issue. I have no idea what

Data lost in Cassandra 3.5 single instance via Erlang driver

2016-06-14 Thread linbo liao
Hi, I use Erlang driver to send data to Cassandra, do testing at local environment meet data lost issue. I have no idea what step is wrong. *Environment:* 1. Ubuntu 12.04 LTS x64bit 2. Cassandra 3.5 single instance, not a cluster, installed via the offical installation document, and didn't cha