Are you really sure of that ? cause Tracing says the opposite when I do it on
one node :
cqlsh:system> TRUNCATE hints ;
Tracing session: fe0dbc80-49dd-11e3-a39b-e367b713537d
activity
| timestamp | source | source_elapsed
----------------------------------------------------------------------------------------+--------------+------------+----------------
execute_cql3_query | 07:59:15,019 | 10.0.0.37 | 0
Parsing TRUNCATE
hints ; | 07:59:15,020 | 10.0.0.37 | 991
Preparing
statement | 07:59:15,021 | 10.0.0.37 | 2009
Enqueuing truncate messages to hosts [/10.0.0.185, /10.0.0.33, /10.0.0.17,
/10.0.0.37] | 07:59:15,023 | 10.0.0.37 | 4459
Message received from
/10.0.0.37 | 07:59:15,186 | 10.0.0.33 | 121
Applying truncation of
system.hints | 07:59:15,200 | 10.0.0.33 | 11064
Sending message to
/10.0.0.37 | 07:59:15,214 | 10.0.0.37 | 195660
Sending message to
/10.0.0.33 | 07:59:15,217 | 10.0.0.37 | 198664
Message received from
/10.0.0.37 | 07:59:15,230 | 10.0.0.185 | 149
Applying truncation of
system.hints | 07:59:15,240 | 10.0.0.185 | 6248
Message received from
/10.0.0.37 | 07:59:15,244 | 10.0.0.17 | 117
Sending message to
/10.0.0.185 | 07:59:15,247 | 10.0.0.37 | 134087
Sending message to
/10.0.0.17 | 07:59:15,248 | 10.0.0.37 | 134760
Message received from
/10.0.0.37 | 07:59:15,251 | 10.0.0.37 | 232058
Applying truncation of
system.hints | 07:59:15,254 | 10.0.0.37 | 234904
Applying truncation of
system.hints | 07:59:15,259 | 10.0.0.17 | 8574
Enqueuing response to truncate operation to
/10.0.0.37 | 07:59:15,388 | 10.0.0.17 | 144491
Sending message to
/10.0.0.37 | 07:59:15,389 | 10.0.0.17 | 145341
Message received from
/10.0.0.17 | 07:59:15,461 | 10.0.0.37 | 441863
Processing response from
/10.0.0.17 | 07:59:15,502 | 10.0.0.37 | 482942
Enqueuing response to truncate operation to
/10.0.0.37 | 07:59:15,578 | 10.0.0.33 | 391940
Sending message to
/10.0.0.37 | 07:59:15,578 | 10.0.0.33 | 392364
Message received from
/10.0.0.33 | 07:59:15,632 | 10.0.0.37 | 613172
Processing response from
/10.0.0.33 | 07:59:15,632 | 10.0.0.37 | 613303
Enqueuing response to truncate operation to
/10.0.0.37 | 07:59:15,736 | 10.0.0.37 | 717066
Sending message to
/10.0.0.37 | 07:59:15,738 | 10.0.0.37 | 719165
Message received from
/10.0.0.37 | 07:59:15,739 | 10.0.0.37 | null
Processing response from
/10.0.0.37 | 07:59:15,739 | 10.0.0.37 | null
Enqueuing response to truncate operation to
/10.0.0.37 | 07:59:15,915 | 10.0.0.185 | 685116
Sending message to
/10.0.0.37 | 07:59:15,915 | 10.0.0.185 | 685519
Message received from
/10.0.0.185 | 07:59:15,936 | 10.0.0.37 | null
Processing response from
/10.0.0.185 | 07:59:15,936 | 10.0.0.37 | null
Request
complete | 07:59:15,936 | 10.0.0.37 | 917846
I also tried to generate different hints on different nodes, and truncating the
table on one node removed data everywhere :
node 1
at t1
cqlsh> select * from system.hints ;
target_id | hint_id |
message_version | mutation
--------------------------------------+--------------------------------------+-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
748cb454-da16-4218-b6ac-728f0fe0cf41 | 9f152ea0-49e9-11e3-af88-ed7ea188ea44 |
6 |
0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf278eb3a0000000000006000376616c00000004eacf278eb3a0000000040000270f
78078974-1f02-4da8-9902-cca56291a9dc | 474e4c30-49ec-11e3-af88-ed7ea188ea44 |
6 |
0x0006706e735f6672000868706172736f6e7300000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf6b945e10000000000006000376616c00000004eacf6b945e1000000004000010f7
78078974-1f02-4da8-9902-cca56291a9dc | 4e64df70-49ec-11e3-af88-ed7ea188ea44 |
6 |
0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf6c49f270000000000006000376616c00000004eacf6c49f2700000000400001618
at t2
cqlsh:pns_fr> TRUNCATE system.hints ;
cqlsh:pns_fr> select * from system.hints ;
node 2
at t1
cqlsh:pns_fr> select * from system.hints ;
target_id | hint_id |
message_version | mutation
--------------------------------------+--------------------------------------+-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
78078974-1f02-4da8-9902-cca56291a9dc | aa8f4da0-49ea-11e3-a8dc-e367b713537d |
6 |
0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf424d88b0000000000006000376616c00000004eacf424d88b000000004000004bc
at t2
cqlsh:pns_fr> select * from system.hints ;
Regards
--
Cyril SCETBON
On 09 Nov 2013, at 22:48, Jonathan Ellis <[email protected]> wrote:
> Hints are not replicated (technically: replicated with LocalStrategy),
> so truncating hints will always only affect the local node.
>
> On Sat, Nov 9, 2013 at 3:44 PM, Cyril Scetbon <[email protected]> wrote:
>> Hi,
>>
>> I would like to add a feature request for local truncates of hints but I
>> want to know your opinion before.
>>
>> We faced an issue last week concerning 3 new nodes that where not seen by
>> some other nodes in the ring. a week later we resolved the issues but it was
>> taking days with a high CPU load (85%) to apply GBytes of Hints. I noticed
>> that CF Hints is defined with LocalStategy and so thought the truncate was
>> only local but that wasn't. We had to remove sstables, commitlog and cached
>> to remove all that old hints stored but not removed after
>> max_hint_window_in_ms as target nodes where not dead (they were only not
>> accepting hints).
>>
>> So, I suggest that the truncate of hints stay local (maybe you need it to
>> stay global per default ?) or that we could do it locally (a truncate
>> argument or a new command like the one used for CONSISTENCY)
>>
>> thanks
>> --
>> Cyril SCETBON
>>
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder, http://www.datastax.com
> @spyced
