I agree. I read about the Choria and very interested to use it. But am running little old version of Ruby, Puppet where I need to plan it for upgrade. But still Choria is in active dev am little concern to go ahead it on Production.
Between I dig into furthermore and found its issue with direct addressing mode only. When I run mco running with -T mcollective or whatever subcollectives it goes very smooth, but when I run mco query using flatfile or -I(direct addressing) against high number of servers only then the issue occurs. So I suspect direct addressing takes time to execute and would curious to understand why it gives such difference. I believe direct addressing should be fast compare with broadcast, correct me if am wrong. On Sunday, January 28, 2018 at 3:11:34 PM UTC+5:30, R.I.Pienaar wrote: > > > On 28 Jan 2018, at 10:35, kk21987 <[email protected] <javascript:>> > wrote: > > Some improvement with your debug help! Looks like the reply-to is working > perfect as expected. The actual problem is triggering the command to all > the servers. > > I have total of 9824 servers connected with mcollective.nodes and when I > run mco query against all the servers to execute some command then > dispatching that mco query message all the consumers takes time which is > why the pending messages are there and whenever the each consumers receives > the message its slowly decreasing. > > > Great > > > > Now I need to look for some tuning to get the message dispatched to all > the consumer very quickly, so it will be good. Any suggestion or best > tuning parameters available for this. I will try to look for ActiveMQ > documentation to get the something. > > > No I gave up making activemq scale this far. I have some large rabbitmq > clusters but that requires a ton of hardware too > > Choria with NATS or the Choria broker would do this comfortably on a > single small 4GB VM with a tremendous speed improvement. > > > > > > On Sunday, January 28, 2018 at 2:03:34 PM UTC+5:30, kk21987 wrote: > >> Thanks. >> >> Yes I captured the client debug log as well and could see that it publish >> message with reply-to properly. >> >> Direct Addressing: >> >> D, [2018-01-28T02:24:56.851292 #17287] DEBUG -- : activemq.rb:402:in >> `publish' Sending a direct message to ActiveMQ target >> '/queue/mcollective.nodes' with headers '{"mc_identity"=>"mco-server-name", >> "timestamp"=>"1517127896000", "expires"=>"1517127966000", >> "reply-to"=>"/queue/data.consumer.all"}' >> >> Broadcast: >> >> D, [2018-01-28T02:25:19.345299 #17340] DEBUG -- : activemq.rb:409:in >> `publish' Sending a broadcast message to ActiveMQ target >> '/topic/mcollective.shell.agent' with headers '{"expires"=>"1517127989000", >> "reply-to"=>"/queue/data.consumer.all", "timestamp"=>"1517127919000"}' >> >> Here is the way I used to exclude the queue consumer destination on >> Broker-A. >> >> <queue physicalName="data.consumer.>"/> >> >> I couldn't take the debug logs on ActiveMQ due to the servers count is >> little high, so it generate huge logs and couldn't get it exactly. But I >> used ActiveMQ-Cli to export the JMS queue and here is a example of one >> pending message >> >> >> <jms-message> >> <header> >> >> <message-id>ID:broker-A-16618-1517126209254-79:5:-1:1:559</message-id> >> <delivery-mode>1</delivery-mode> >> <destination>queue://mcollective.nodes</destination> >> <expiration>1517127457000</expiration> >> <priority>4</priority> >> <redelivered>false</redelivered> >> <reply-to>queue://data.consumer.all</reply-to> >> <timestamp>2018-01-28T02:16:27</timestamp> >> </header> >> <properties> >> <property> >> <name>content-type</name> >> <value>text/plain; charset=UTF-8</value> >> </property> >> <property> >> <name>mc_identity</name> >> <value>mcollective-server-name</value> >> </property> >> </properties> >> </jms-message> >> >> >> >> >> On Sunday, January 28, 2018 at 12:40:14 PM UTC+5:30, R.I.Pienaar wrote: >> >>> >>> >>> On 28 Jan 2018, at 06:06, kk21987 <[email protected]> wrote: >>> >>> My apologize. I misunderstood few things. >>> >>> Basically when I run mco query in direct addressing mode then it publish >>> that message as queue where as broadcast mode publish the message as topic. >>> So I suspect its not an issue with the modes whatever the mco query use. >>> The only thing I want to understand is, lets say I run mco query against >>> 1000servers with --reply-to (fire and forget, no acknowledgement and >>> nothing. Correct me if am wrong), so all the response message will go to >>> the consumer where the active subscriber is present for the given queue. >>> The response message is not coming to Broker-A which I have configured by >>> starting receiver here as well and perfectly its going to the consumer >>> where its active. >>> >>> >>> That sounds about right. Yes it will use a topic and all nodes will get >>> it >>> >>> >>> My concern is, at the same time I see count is showing in pending >>> message for the queue "mcollective.nodes" and slowly its decreasing(at the >>> same enqueued and dequeued messages count also increasing slowly) which I >>> have verified on ActiveMQ UI Jolokia monitoring (8161). So I would like to >>> understand why such messages count in pending messages, since all I want to >>> just fire the command and forget and don't want to worry about response or >>> whether its executed or not. >>> >>> Name ↑Number Of Pending Messages Number Of Consumers Messages Enqueued >>> Messages Dequeued Views Operations >>> mcollective.nodes 9 84 884 875 Browse Active Consumers >>> Active Producers >>> Send To PurgeDelete >>> >>> >>> >>> You will need to double check in the logs. Your client also logs and at >>> debug >>> level you can verify where the publish to >>> >>> >>> On Sunday, January 28, 2018 at 7:48:15 AM UTC+5:30, kk21987 wrote: >>> >>>> Based on my understanding when I run mco query with reply-to I should >>>> see the reply-to queue name on the response header and its there on >>>> mcollective server logs, which I have verified by enabling debug on the >>>> mcollective servers. >>>> >>>> My worry is, I shouldn't get any response messages back to the Broker-A >>>> and all the response messages should go to the consumer server which is >>>> running elsewhere. This is working as expected for broadcast based mco >>>> query but when I run direct addressing based mco query then the response >>>> message is flowing everywhere instead of going to consumer server only. I >>>> have configured the exclude destination on Broker-A to make sure none of >>>> the brokers are connected with the reply-to queue name except the consumer >>>> server broker. >>>> >>>> On Sunday, January 28, 2018 at 2:08:06 AM UTC+5:30, R.I.Pienaar wrote: >>>> >>>>> >>>>> >>>>> On Sat, 27 Jan 2018, at 15:03, kk21987 wrote: >>>>> > Hi, >>>>> > >>>>> > Am having some issue with --reply-to behavior and not sure if its an >>>>> > issue >>>>> > or the behavior itself like that. My setup is drawn in >>>>> > ...... >>>>> > >>>>> > >>>>> > My environment is enabled with direct_addressing. Basically I always >>>>> fire >>>>> > the mco and forget it and don't want to consume any messages on mco >>>>> client. >>>>> > I have configured separate consumer server where it will consume the >>>>> > messages on the queue whatever declared during the mco query and >>>>> will get >>>>> > result from thie consumer server. The reason for configured separate >>>>> > consumer server is some commands whatever I run will take more time >>>>> to >>>>> > execute and also doing some logic on the response messages based on >>>>> my >>>>> > requirement. >>>>> > >>>>> > I have configured exclude destination to make sure the queue >>>>> whatever am >>>>> > using in reply-to not connected with Broker-A from other brokers. >>>>> > >>>>> > When I run mco query using regex filter it pubish message in >>>>> broadcast mode >>>>> > and consumer server consume the response without any issue. At the >>>>> same >>>>> > time when I check on the Broker-A ActiveMQ UI monitoring I noticed >>>>> Messages >>>>> > Enqueued Dequeued count is NOT increased which means nothing coming >>>>> back to >>>>> > Broker-A. >>>>> > >>>>> > But when I run mco query using direct addressing mode (-I) along >>>>> with >>>>> > reply-to the message is published as direct addressing and consumer >>>>> server >>>>> > consume the response fine here as well. At the same time when I >>>>> check on >>>>> > the Broker-A ActiveMQ UI monitoring I noticed Messages Enqueued >>>>> Dequeued >>>>> > count is increased which means message is coming to Broker-A as well >>>>> at the >>>>> > same time message is available on consumer server. >>>>> > >>>>> > Am not sure such difference between direct addressing and broadcast >>>>> even >>>>> > though both use the same reply-to mechanism. I don't want any >>>>> response >>>>> > messages come back to Broker-A and all the response messages should >>>>> go to >>>>> > Consumer Server only. Can I have some lights on this please? >>>>> >>>>> what you describe should work, I guess the best hints lie in the >>>>> server debug log. Does it send a reply and does it do so to the right >>>>> target? >>>>> >>>>> >>>>> -- >>>>> R.I.Pienaar / www.devco.net / @ripienaar >>>>> >>>> -- >>> >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "mcollective-users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >>> -- > > --- > You received this message because you are subscribed to the Google Groups > "mcollective-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > For more options, visit https://groups.google.com/d/optout. > > -- --- You received this message because you are subscribed to the Google Groups "mcollective-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
