and/or a nodetool purgelocalhints command ? On 15 Nov 2013, at 05:56, Jonathan Ellis <jbel...@gmail.com> wrote:
> You're right. You'd want to expose the node-local truncate over JMX. > > On Sun, Nov 10, 2013 at 3:48 AM, Cyril Scetbon <cyril.scet...@free.fr> wrote: >> Are you really sure of that ? cause Tracing says the opposite when I do it >> on one node : >> >> cqlsh:system> TRUNCATE hints ; >> >> Tracing session: fe0dbc80-49dd-11e3-a39b-e367b713537d >> >> activity >> | timestamp | source | source_elapsed >> ----------------------------------------------------------------------------------------+--------------+------------+---------------- >> >> execute_cql3_query | 07:59:15,019 | 10.0.0.37 | 0 >> Parsing >> TRUNCATE hints ; | 07:59:15,020 | 10.0.0.37 | 991 >> Preparing >> statement | 07:59:15,021 | 10.0.0.37 | 2009 >> Enqueuing truncate messages to hosts [/10.0.0.185, /10.0.0.33, /10.0.0.17, >> /10.0.0.37] | 07:59:15,023 | 10.0.0.37 | 4459 >> Message received from >> /10.0.0.37 | 07:59:15,186 | 10.0.0.33 | 121 >> Applying truncation of >> system.hints | 07:59:15,200 | 10.0.0.33 | 11064 >> Sending message to >> /10.0.0.37 | 07:59:15,214 | 10.0.0.37 | 195660 >> Sending message to >> /10.0.0.33 | 07:59:15,217 | 10.0.0.37 | 198664 >> Message received from >> /10.0.0.37 | 07:59:15,230 | 10.0.0.185 | 149 >> Applying truncation of >> system.hints | 07:59:15,240 | 10.0.0.185 | 6248 >> Message received from >> /10.0.0.37 | 07:59:15,244 | 10.0.0.17 | 117 >> Sending message to >> /10.0.0.185 | 07:59:15,247 | 10.0.0.37 | 134087 >> Sending message to >> /10.0.0.17 | 07:59:15,248 | 10.0.0.37 | 134760 >> Message received from >> /10.0.0.37 | 07:59:15,251 | 10.0.0.37 | 232058 >> Applying truncation of >> system.hints | 07:59:15,254 | 10.0.0.37 | 234904 >> Applying truncation of >> system.hints | 07:59:15,259 | 10.0.0.17 | 8574 >> Enqueuing response to truncate operation to >> /10.0.0.37 | 07:59:15,388 | 10.0.0.17 | 144491 >> Sending message to >> /10.0.0.37 | 07:59:15,389 | 10.0.0.17 | 145341 >> Message received from >> /10.0.0.17 | 07:59:15,461 | 10.0.0.37 | 441863 >> Processing response from >> /10.0.0.17 | 07:59:15,502 | 10.0.0.37 | 482942 >> Enqueuing response to truncate operation to >> /10.0.0.37 | 07:59:15,578 | 10.0.0.33 | 391940 >> Sending message to >> /10.0.0.37 | 07:59:15,578 | 10.0.0.33 | 392364 >> Message received from >> /10.0.0.33 | 07:59:15,632 | 10.0.0.37 | 613172 >> Processing response from >> /10.0.0.33 | 07:59:15,632 | 10.0.0.37 | 613303 >> Enqueuing response to truncate operation to >> /10.0.0.37 | 07:59:15,736 | 10.0.0.37 | 717066 >> Sending message to >> /10.0.0.37 | 07:59:15,738 | 10.0.0.37 | 719165 >> Message received from >> /10.0.0.37 | 07:59:15,739 | 10.0.0.37 | null >> Processing response from >> /10.0.0.37 | 07:59:15,739 | 10.0.0.37 | null >> Enqueuing response to truncate operation to >> /10.0.0.37 | 07:59:15,915 | 10.0.0.185 | 685116 >> Sending message to >> /10.0.0.37 | 07:59:15,915 | 10.0.0.185 | 685519 >> Message received from >> /10.0.0.185 | 07:59:15,936 | 10.0.0.37 | null >> Processing response from >> /10.0.0.185 | 07:59:15,936 | 10.0.0.37 | null >> >> Request complete | 07:59:15,936 | 10.0.0.37 | 917846 >> >> I also tried to generate different hints on different nodes, and truncating >> the table on one node removed data everywhere : >> >> >> node 1 >> >> at t1 >> >> cqlsh> select * from system.hints ; >> >> target_id | hint_id >> | message_version | mutation >> --------------------------------------+--------------------------------------+-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >> 748cb454-da16-4218-b6ac-728f0fe0cf41 | 9f152ea0-49e9-11e3-af88-ed7ea188ea44 >> | 6 | >> 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf278eb3a0000000000006000376616c00000004eacf278eb3a0000000040000270f >> 78078974-1f02-4da8-9902-cca56291a9dc | 474e4c30-49ec-11e3-af88-ed7ea188ea44 >> | 6 | >> 0x0006706e735f6672000868706172736f6e7300000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf6b945e10000000000006000376616c00000004eacf6b945e1000000004000010f7 >> 78078974-1f02-4da8-9902-cca56291a9dc | 4e64df70-49ec-11e3-af88-ed7ea188ea44 >> | 6 | >> 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf6c49f270000000000006000376616c00000004eacf6c49f2700000000400001618 >> >> at t2 >> >> cqlsh:pns_fr> TRUNCATE system.hints ; >> cqlsh:pns_fr> select * from system.hints ; >> >> node 2 >> >> at t1 >> >> cqlsh:pns_fr> select * from system.hints ; >> >> target_id | hint_id >> | message_version | mutation >> --------------------------------------+--------------------------------------+-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >> 78078974-1f02-4da8-9902-cca56291a9dc | aa8f4da0-49ea-11e3-a8dc-e367b713537d >> | 6 | >> 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf424d88b0000000000006000376616c00000004eacf424d88b000000004000004bc >> >> at t2 >> >> cqlsh:pns_fr> select * from system.hints ; >> >> Regards >> -- >> Cyril SCETBON >> >> On 09 Nov 2013, at 22:48, Jonathan Ellis <jbel...@gmail.com> wrote: >> >>> Hints are not replicated (technically: replicated with LocalStrategy), >>> so truncating hints will always only affect the local node. >>> >>> On Sat, Nov 9, 2013 at 3:44 PM, Cyril Scetbon <cyril.scet...@free.fr> wrote: >>>> Hi, >>>> >>>> I would like to add a feature request for local truncates of hints but I >>>> want to know your opinion before. >>>> >>>> We faced an issue last week concerning 3 new nodes that where not seen by >>>> some other nodes in the ring. a week later we resolved the issues but it >>>> was taking days with a high CPU load (85%) to apply GBytes of Hints. I >>>> noticed that CF Hints is defined with LocalStategy and so thought the >>>> truncate was only local but that wasn't. We had to remove sstables, >>>> commitlog and cached to remove all that old hints stored but not removed >>>> after max_hint_window_in_ms as target nodes where not dead (they were >>>> only not accepting hints). >>>> >>>> So, I suggest that the truncate of hints stay local (maybe you need it to >>>> stay global per default ?) or that we could do it locally (a truncate >>>> argument or a new command like the one used for CONSISTENCY) >>>> >>>> thanks >>>> -- >>>> Cyril SCETBON >>>> >>> >>> >>> >>> -- >>> Jonathan Ellis >>> Project Chair, Apache Cassandra >>> co-founder, http://www.datastax.com >>> @spyced >> > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder, http://www.datastax.com > @spyced
