anyone ? -- Cyril SCETBON
On 10 Nov 2013, at 10:48, Cyril Scetbon <cyril.scet...@free.fr> wrote: > Are you really sure of that ? cause Tracing says the opposite when I do it on > one node : > > cqlsh:system> TRUNCATE hints ; > > Tracing session: fe0dbc80-49dd-11e3-a39b-e367b713537d > > activity > | timestamp | source | source_elapsed > ----------------------------------------------------------------------------------------+--------------+------------+---------------- > > execute_cql3_query | 07:59:15,019 | 10.0.0.37 | 0 > Parsing > TRUNCATE hints ; | 07:59:15,020 | 10.0.0.37 | 991 > Preparing > statement | 07:59:15,021 | 10.0.0.37 | 2009 > Enqueuing truncate messages to hosts [/10.0.0.185, /10.0.0.33, /10.0.0.17, > /10.0.0.37] | 07:59:15,023 | 10.0.0.37 | 4459 > Message received from > /10.0.0.37 | 07:59:15,186 | 10.0.0.33 | 121 > Applying truncation of > system.hints | 07:59:15,200 | 10.0.0.33 | 11064 > Sending message to > /10.0.0.37 | 07:59:15,214 | 10.0.0.37 | 195660 > Sending message to > /10.0.0.33 | 07:59:15,217 | 10.0.0.37 | 198664 > Message received from > /10.0.0.37 | 07:59:15,230 | 10.0.0.185 | 149 > Applying truncation of > system.hints | 07:59:15,240 | 10.0.0.185 | 6248 > Message received from > /10.0.0.37 | 07:59:15,244 | 10.0.0.17 | 117 > Sending message to > /10.0.0.185 | 07:59:15,247 | 10.0.0.37 | 134087 > Sending message to > /10.0.0.17 | 07:59:15,248 | 10.0.0.37 | 134760 > Message received from > /10.0.0.37 | 07:59:15,251 | 10.0.0.37 | 232058 > Applying truncation of > system.hints | 07:59:15,254 | 10.0.0.37 | 234904 > Applying truncation of > system.hints | 07:59:15,259 | 10.0.0.17 | 8574 > Enqueuing response to truncate operation to > /10.0.0.37 | 07:59:15,388 | 10.0.0.17 | 144491 > Sending message to > /10.0.0.37 | 07:59:15,389 | 10.0.0.17 | 145341 > Message received from > /10.0.0.17 | 07:59:15,461 | 10.0.0.37 | 441863 > Processing response from > /10.0.0.17 | 07:59:15,502 | 10.0.0.37 | 482942 > Enqueuing response to truncate operation to > /10.0.0.37 | 07:59:15,578 | 10.0.0.33 | 391940 > Sending message to > /10.0.0.37 | 07:59:15,578 | 10.0.0.33 | 392364 > Message received from > /10.0.0.33 | 07:59:15,632 | 10.0.0.37 | 613172 > Processing response from > /10.0.0.33 | 07:59:15,632 | 10.0.0.37 | 613303 > Enqueuing response to truncate operation to > /10.0.0.37 | 07:59:15,736 | 10.0.0.37 | 717066 > Sending message to > /10.0.0.37 | 07:59:15,738 | 10.0.0.37 | 719165 > Message received from > /10.0.0.37 | 07:59:15,739 | 10.0.0.37 | null > Processing response from > /10.0.0.37 | 07:59:15,739 | 10.0.0.37 | null > Enqueuing response to truncate operation to > /10.0.0.37 | 07:59:15,915 | 10.0.0.185 | 685116 > Sending message to > /10.0.0.37 | 07:59:15,915 | 10.0.0.185 | 685519 > Message received from > /10.0.0.185 | 07:59:15,936 | 10.0.0.37 | null > Processing response from > /10.0.0.185 | 07:59:15,936 | 10.0.0.37 | null > Request > complete | 07:59:15,936 | 10.0.0.37 | 917846 > > I also tried to generate different hints on different nodes, and truncating > the table on one node removed data everywhere : > > > node 1 > > at t1 > > cqlsh> select * from system.hints ; > > target_id | hint_id | > message_version | mutation > --------------------------------------+--------------------------------------+-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > 748cb454-da16-4218-b6ac-728f0fe0cf41 | 9f152ea0-49e9-11e3-af88-ed7ea188ea44 | > 6 | > 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf278eb3a0000000000006000376616c00000004eacf278eb3a0000000040000270f > 78078974-1f02-4da8-9902-cca56291a9dc | 474e4c30-49ec-11e3-af88-ed7ea188ea44 | > 6 | > 0x0006706e735f6672000868706172736f6e7300000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf6b945e10000000000006000376616c00000004eacf6b945e1000000004000010f7 > 78078974-1f02-4da8-9902-cca56291a9dc | 4e64df70-49ec-11e3-af88-ed7ea188ea44 | > 6 | > 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf6c49f270000000000006000376616c00000004eacf6c49f2700000000400001618 > > at t2 > > cqlsh:pns_fr> TRUNCATE system.hints ; > cqlsh:pns_fr> select * from system.hints ; > > node 2 > > at t1 > > cqlsh:pns_fr> select * from system.hints ; > > target_id | hint_id | > message_version | mutation > --------------------------------------+--------------------------------------+-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > 78078974-1f02-4da8-9902-cca56291a9dc | aa8f4da0-49ea-11e3-a8dc-e367b713537d | > 6 | > 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf424d88b0000000000006000376616c00000004eacf424d88b000000004000004bc > > at t2 > > cqlsh:pns_fr> select * from system.hints ; > > Regards > -- > Cyril SCETBON > > On 09 Nov 2013, at 22:48, Jonathan Ellis <jbel...@gmail.com> wrote: > >> Hints are not replicated (technically: replicated with LocalStrategy), >> so truncating hints will always only affect the local node. >> >> On Sat, Nov 9, 2013 at 3:44 PM, Cyril Scetbon <cyril.scet...@free.fr> wrote: >>> Hi, >>> >>> I would like to add a feature request for local truncates of hints but I >>> want to know your opinion before. >>> >>> We faced an issue last week concerning 3 new nodes that where not seen by >>> some other nodes in the ring. a week later we resolved the issues but it >>> was taking days with a high CPU load (85%) to apply GBytes of Hints. I >>> noticed that CF Hints is defined with LocalStategy and so thought the >>> truncate was only local but that wasn't. We had to remove sstables, >>> commitlog and cached to remove all that old hints stored but not removed >>> after max_hint_window_in_ms as target nodes where not dead (they were only >>> not accepting hints). >>> >>> So, I suggest that the truncate of hints stay local (maybe you need it to >>> stay global per default ?) or that we could do it locally (a truncate >>> argument or a new command like the one used for CONSISTENCY) >>> >>> thanks >>> -- >>> Cyril SCETBON >>> >> >> >> >> -- >> Jonathan Ellis >> Project Chair, Apache Cassandra >> co-founder, http://www.datastax.com >> @spyced >
