Thanks I've created JIRA CASSANDRA-6367 for that functionality. Regards.
On 17 Nov 2013, at 20:33, Jonathan Ellis <jbel...@gmail.com> wrote: > Right. > On Nov 17, 2013 11:00 AM, "Cyril Scetbon" <cyril.scet...@free.fr> wrote: > >> and/or a nodetool purgelocalhints command ? >> >> On 15 Nov 2013, at 05:56, Jonathan Ellis <jbel...@gmail.com> wrote: >> >>> You're right. You'd want to expose the node-local truncate over JMX. >>> >>> On Sun, Nov 10, 2013 at 3:48 AM, Cyril Scetbon <cyril.scet...@free.fr> >> wrote: >>>> Are you really sure of that ? cause Tracing says the opposite when I do >> it on one node : >>>> >>>> cqlsh:system> TRUNCATE hints ; >>>> >>>> Tracing session: fe0dbc80-49dd-11e3-a39b-e367b713537d >>>> >>>> activity >> | timestamp | source | source_elapsed >>>> >> ----------------------------------------------------------------------------------------+--------------+------------+---------------- >>>> >> execute_cql3_query | 07:59:15,019 | 10.0.0.37 | 0 >>>> Parsing >> TRUNCATE hints ; | 07:59:15,020 | 10.0.0.37 | 991 >>>> >> Preparing statement | 07:59:15,021 | 10.0.0.37 | 2009 >>>> Enqueuing truncate messages to hosts [/10.0.0.185, /10.0.0.33, / >> 10.0.0.17, /10.0.0.37] | 07:59:15,023 | 10.0.0.37 | 4459 >>>> Message received >> from /10.0.0.37 | 07:59:15,186 | 10.0.0.33 | 121 >>>> Applying truncation >> of system.hints | 07:59:15,200 | 10.0.0.33 | 11064 >>>> Sending >> message to /10.0.0.37 | 07:59:15,214 | 10.0.0.37 | 195660 >>>> Sending >> message to /10.0.0.33 | 07:59:15,217 | 10.0.0.37 | 198664 >>>> Message received >> from /10.0.0.37 | 07:59:15,230 | 10.0.0.185 | 149 >>>> Applying truncation >> of system.hints | 07:59:15,240 | 10.0.0.185 | 6248 >>>> Message received >> from /10.0.0.37 | 07:59:15,244 | 10.0.0.17 | 117 >>>> Sending message >> to /10.0.0.185 | 07:59:15,247 | 10.0.0.37 | 134087 >>>> Sending >> message to /10.0.0.17 | 07:59:15,248 | 10.0.0.37 | 134760 >>>> Message received >> from /10.0.0.37 | 07:59:15,251 | 10.0.0.37 | 232058 >>>> Applying truncation >> of system.hints | 07:59:15,254 | 10.0.0.37 | 234904 >>>> Applying truncation >> of system.hints | 07:59:15,259 | 10.0.0.17 | 8574 >>>> Enqueuing response to truncate >> operation to /10.0.0.37 | 07:59:15,388 | 10.0.0.17 | 144491 >>>> Sending >> message to /10.0.0.37 | 07:59:15,389 | 10.0.0.17 | 145341 >>>> Message received >> from /10.0.0.17 | 07:59:15,461 | 10.0.0.37 | 441863 >>>> Processing response >> from /10.0.0.17 | 07:59:15,502 | 10.0.0.37 | 482942 >>>> Enqueuing response to truncate >> operation to /10.0.0.37 | 07:59:15,578 | 10.0.0.33 | 391940 >>>> Sending >> message to /10.0.0.37 | 07:59:15,578 | 10.0.0.33 | 392364 >>>> Message received >> from /10.0.0.33 | 07:59:15,632 | 10.0.0.37 | 613172 >>>> Processing response >> from /10.0.0.33 | 07:59:15,632 | 10.0.0.37 | 613303 >>>> Enqueuing response to truncate >> operation to /10.0.0.37 | 07:59:15,736 | 10.0.0.37 | 717066 >>>> Sending >> message to /10.0.0.37 | 07:59:15,738 | 10.0.0.37 | 719165 >>>> Message received >> from /10.0.0.37 | 07:59:15,739 | 10.0.0.37 | null >>>> Processing response >> from /10.0.0.37 | 07:59:15,739 | 10.0.0.37 | null >>>> Enqueuing response to truncate >> operation to /10.0.0.37 | 07:59:15,915 | 10.0.0.185 | 685116 >>>> Sending >> message to /10.0.0.37 | 07:59:15,915 | 10.0.0.185 | 685519 >>>> Message received >> from /10.0.0.185 | 07:59:15,936 | 10.0.0.37 | null >>>> Processing response >> from /10.0.0.185 | 07:59:15,936 | 10.0.0.37 | null >>>> >> Request complete | 07:59:15,936 | 10.0.0.37 | 917846 >>>> >>>> I also tried to generate different hints on different nodes, and >> truncating the table on one node removed data everywhere : >>>> >>>> >>>> node 1 >>>> >>>> at t1 >>>> >>>> cqlsh> select * from system.hints ; >>>> >>>> target_id | hint_id >> | message_version | mutation >>>> >> --------------------------------------+--------------------------------------+-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>>> 748cb454-da16-4218-b6ac-728f0fe0cf41 | >> 9f152ea0-49e9-11e3-af88-ed7ea188ea44 | 6 | >> 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf278eb3a0000000000006000376616c00000004eacf278eb3a0000000040000270f >>>> 78078974-1f02-4da8-9902-cca56291a9dc | >> 474e4c30-49ec-11e3-af88-ed7ea188ea44 | 6 | >> 0x0006706e735f6672000868706172736f6e7300000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf6b945e10000000000006000376616c00000004eacf6b945e1000000004000010f7 >>>> 78078974-1f02-4da8-9902-cca56291a9dc | >> 4e64df70-49ec-11e3-af88-ed7ea188ea44 | 6 | >> 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf6c49f270000000000006000376616c00000004eacf6c49f2700000000400001618 >>>> >>>> at t2 >>>> >>>> cqlsh:pns_fr> TRUNCATE system.hints ; >>>> cqlsh:pns_fr> select * from system.hints ; >>>> >>>> node 2 >>>> >>>> at t1 >>>> >>>> cqlsh:pns_fr> select * from system.hints ; >>>> >>>> target_id | hint_id >> | message_version | mutation >>>> >> --------------------------------------+--------------------------------------+-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>>> 78078974-1f02-4da8-9902-cca56291a9dc | >> aa8f4da0-49ea-11e3-a8dc-e367b713537d | 6 | >> 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf424d88b0000000000006000376616c00000004eacf424d88b000000004000004bc >>>> >>>> at t2 >>>> >>>> cqlsh:pns_fr> select * from system.hints ; >>>> >>>> Regards >>>> -- >>>> Cyril SCETBON >>>> >>>> On 09 Nov 2013, at 22:48, Jonathan Ellis <jbel...@gmail.com> wrote: >>>> >>>>> Hints are not replicated (technically: replicated with LocalStrategy), >>>>> so truncating hints will always only affect the local node. >>>>> >>>>> On Sat, Nov 9, 2013 at 3:44 PM, Cyril Scetbon <cyril.scet...@free.fr> >> wrote: >>>>>> Hi, >>>>>> >>>>>> I would like to add a feature request for local truncates of hints >> but I want to know your opinion before. >>>>>> >>>>>> We faced an issue last week concerning 3 new nodes that where not >> seen by some other nodes in the ring. a week later we resolved the issues >> but it was taking days with a high CPU load (85%) to apply GBytes of Hints. >> I noticed that CF Hints is defined with LocalStategy and so thought the >> truncate was only local but that wasn't. We had to remove sstables, >> commitlog and cached to remove all that old hints stored but not removed >> after max_hint_window_in_ms as target nodes where not dead (they were only >> not accepting hints). >>>>>> >>>>>> So, I suggest that the truncate of hints stay local (maybe you need >> it to stay global per default ?) or that we could do it locally (a truncate >> argument or a new command like the one used for CONSISTENCY) >>>>>> >>>>>> thanks >>>>>> -- >>>>>> Cyril SCETBON >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Jonathan Ellis >>>>> Project Chair, Apache Cassandra >>>>> co-founder, http://www.datastax.com >>>>> @spyced >>>> >>> >>> >>> >>> -- >>> Jonathan Ellis >>> Project Chair, Apache Cassandra >>> co-founder, http://www.datastax.com >>> @spyced >> >>
