Taking a look at the logs, i can see the below log line in
https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/260/faseela-l2-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-oxygen/18/odl_1/odl1_karaf.log.gz
10 mins before the test suite starts :
2018-07-25T23:00:41,688 | INFO | pool-241-thread-1 |
WeightedCentralizedSwitchScheduler | 361 -
org.opendaylight.netvirt.natservice-impl - 0.6.3.SNAPSHOT | addSwitch: Adding
167740445697660 dpnId to switchWeightsMap
2018-07-25T23:00:42,274 | DEBUG | epollEventLoopGroup-9-1 | SalFlowServiceImpl
| 385 - org.opendaylight.openflowplugin.impl - 0.6.3.SNAPSHOT | Flow add with
id=DHCPTableMissFlowForExternalTunnel finished without error
But the flow is not present on the switch when the suite starts.
We have 3 OVS in the test, but i see only this flow-id log for 2 switches.
Anyways the compute in question, is 167740445697660, so let us assume that the
flow was getting pushed for the same, but doesn't show up on switch. But there
are different failures showing up in the logs including the below one. Let me
know if this could impact:
2018-07-25T23:01:09,328 | ERROR |
opendaylight-cluster-data-notification-dispatcher-97 | DefaultConfigPusher |
378 - org.opendaylight.openflowplugin.applications.of-switch-config-pusher -
0.6.3.SNAPSHOT | Future (eventually) failed: addFlow
org.opendaylight.controller.md.sal.dom.api.DOMRpcImplementationNotAvailableException:
No implementation of RPC
AbsoluteSchemaPath{path=[(urn:opendaylight:module:config?revision=2014-10-15)set-config]}
available at
org.opendaylight.controller.md.sal.dom.broker.impl.RoutedDOMRpcRoutingTableEntry.invokeRpc(RoutedDOMRpcRoutingTableEntry.java:85)
[229:org.opendaylight.controller.sal-broker-impl:1.7.3.SNAPSHOT] at
org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRoutingTable.invokeRpc(DOMRpcRoutingTable.java:178)
[229:org.opendaylight.controller.sal-broker-impl:1.7.3.SNAPSHOT] at
org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRouter.invokeRpc(DOMRpcRouter.java:102)
[229:org.opendaylight.controller.sal-broker-impl:1.7.3.SNAPSHOT] at
Proxybb8ecbf1_eb9d_4480_a18e_4a2007f00a3c.invokeRpc(Unknown Source) [?:?] at
Proxy40f45215_a726_4be9_ac6b_649862163662.invokeRpc(Unknown Source) [?:?] at
org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.invoke0(RpcServiceAdapter.java:68)
[226:org.opendaylight.controller.sal-binding-broker-impl:1.7.3.SNAPSHOT] at
org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.access$000(RpcServiceAdapter.java:46)
[226:org.opendaylight.controller.sal-binding-broker-impl:1.7.3.SNAPSHOT] at
org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter$RpcInvocationStrategy.invoke(RpcServiceAdapter.java:165)
[226:org.opendaylight.controller.sal-binding-broker-impl:1.7.3.SNAPSHOT] at
org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.invoke(RpcServiceAdapter.java:99)
[226:org.opendaylight.controller.sal-binding-broker-impl:1.7.3.SNAPSHOT] at
com.sun.proxy.$Proxy113.setConfig(Unknown Source)
[388:org.opendaylight.openflowplugin.model.flow-service:0.6.3.SNAPSHOT] at
org.opendaylight.openflowplugin.openflow.ofswitch.config.DefaultConfigPusher.onDataTreeChanged(DefaultConfigPusher.java:84)
[378:org.opendaylight.openflowplugin.applications.of-switch-config-pusher:0.6.3.SNAPSHOT]
at
org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataTreeChangeListenerAdapter.onDataTreeChanged(BindingDOMDataTreeChangeListenerAdapter.java:41)
[226:org.opendaylight.controller.sal-binding-broker-impl:1.7.3.SNAPSHOT] at
org.opendaylight.controller.cluster.datastore.DataTreeChangeListenerActor.dataChanged(DataTreeChangeListenerActor.java:67)
[239:org.opendaylight.controller.sal-distributed-datastore:1.7.3.SNAPSHOT] at
org.opendaylight.controller.cluster.datastore.DataTreeChangeListenerActor.handleReceive(DataTreeChangeListenerActor.java:41)
[239:org.opendaylight.controller.sal-distributed-datastore:1.7.3.SNAPSHOT] at
org.opendaylight.controller.cluster.common.actor.AbstractUntypedActor.onReceive(AbstractUntypedActor.java:38)
[232:org.opendaylight.controller.sal-clustering-commons:1.7.3.SNAPSHOT] at
akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167)
[42:com.typesafe.akka.actor:2.5.11] at
akka.actor.Actor.aroundReceive(Actor.scala:517)
[42:com.typesafe.akka.actor:2.5.11] at
akka.actor.Actor.aroundReceive$(Actor.scala:515)
[42:com.typesafe.akka.actor:2.5.11] at
akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97)
[42:com.typesafe.akka.actor:2.5.11] at
akka.actor.ActorCell.receiveMessage(ActorCell.scala:590)
[42:com.typesafe.akka.actor:2.5.11] at
akka.actor.ActorCell.invoke(ActorCell.scala:559)
[42:com.typesafe.akka.actor:2.5.11] at
akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
[42:com.typesafe.akka.actor:2.5.11] at
akka.dispatch.Mailbox.run(Mailbox.scala:224)
[42:com.typesafe.akka.actor:2.5.11] at
akka.dispatch.Mailbox.exec(Mailbox.scala:234)
[42:com.typesafe.akka.actor:2.5.11] at
akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
[42:com.typesafe.akka.actor:2.5.11] at
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
[42:com.typesafe.akka.actor:2.5.11] at
akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
[42:com.typesafe.akka.actor:2.5.11] at
akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[42:com.typesafe.akka.actor:2.5.11]
From: Faseela K
Sent: Thursday, July 26, 2018 7:50 AM
To: D Arunprakash <[email protected]>; 'Anil Vishnoi'
<[email protected]>
Cc: Gobinath . <[email protected]>; 'Sam Hague' <[email protected]>;
Dayavanti Gopal Kamath <[email protected]>; 'Jamo Luhrsen'
<[email protected]>; 'Vishal Thapar' <[email protected]>;
'[email protected]'
<[email protected]>
Subject: RE: Netvirt Cluster CSIT : Table Miss Entry programming failing
randomly
Arun,
Here are the logs with DEBUG enabled.
https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/260/faseela-l2-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-oxygen/18/
This time, table 18 entry is missing.
Thanks,
Faseela
From: Faseela K
Sent: Wednesday, July 25, 2018 4:34 PM
To: D Arunprakash
<[email protected]<mailto:[email protected]>>; Anil Vishnoi
<[email protected]<mailto:[email protected]>>
Cc: Gobinath . <[email protected]<mailto:[email protected]>>; Sam Hague
<[email protected]<mailto:[email protected]>>; Dayavanti Gopal Kamath
<[email protected]<mailto:[email protected]>>;
Jamo Luhrsen <[email protected]<mailto:[email protected]>>; Vishal Thapar
<[email protected]<mailto:[email protected]>>;
[email protected]<mailto:[email protected]>
Subject: RE: Netvirt Cluster CSIT : Table Miss Entry programming failing
randomly
Arun,
I have enabled the requested TRACE logs and running the CSIT now.
Ever since I enabled the TRACEs it has not failed :).
Let us see if the issue gets reproduced in another 24 hours.
Thanks,
Faseela
From: D Arunprakash
Sent: Wednesday, July 25, 2018 7:53 AM
To: Faseela K <[email protected]<mailto:[email protected]>>; Anil
Vishnoi <[email protected]<mailto:[email protected]>>
Cc: Gobinath . <[email protected]<mailto:[email protected]>>; Sam Hague
<[email protected]<mailto:[email protected]>>; Dayavanti Gopal Kamath
<[email protected]<mailto:[email protected]>>;
Jamo Luhrsen <[email protected]<mailto:[email protected]>>; Vishal Thapar
<[email protected]<mailto:[email protected]>>;
[email protected]<mailto:[email protected]>
Subject: RE: Netvirt Cluster CSIT : Table Miss Entry programming failing
randomly
Faseela,
Could you please enable the following package logging and provide the logs?
log:set DEBUG org.opendaylight.openflowplugin.impl
Regards,
Arun
From: Faseela K
Sent: Tuesday, July 24, 2018 11:44 PM
To: Anil Vishnoi <[email protected]<mailto:[email protected]>>
Cc: D Arunprakash
<[email protected]<mailto:[email protected]>>; Gobinath .
<[email protected]<mailto:[email protected]>>; Sam Hague
<[email protected]<mailto:[email protected]>>; Dayavanti Gopal Kamath
<[email protected]<mailto:[email protected]>>;
Jamo Luhrsen <[email protected]<mailto:[email protected]>>; Vishal Thapar
<[email protected]<mailto:[email protected]>>;
[email protected]<mailto:[email protected]>
Subject: RE: Netvirt Cluster CSIT : Table Miss Entry programming failing
randomly
Hi Arun,
We are still hitting this issue, and this can be seen on stable/oxygen as
well.
In some cases the flow will be programmed initially, but after some time
it disappears.
I do see a lot of "Switch Idle state occurred,
node=/10.30.170.51:60858|auxId=0" messages in the logs. Is that anything to
worry about?
If there is any specific TRACE logs to be enabled while running the CSIT,
could you please let me know? I can run the CSIT with the same and supply the
logs.
Thanks,
Faseela
From: Faseela K
Sent: Tuesday, July 17, 2018 1:24 PM
To: Anil Vishnoi <[email protected]<mailto:[email protected]>>
Cc: D Arunprakash
<[email protected]<mailto:[email protected]>>; Gobinath .
<[email protected]<mailto:[email protected]>>; Sam Hague
<[email protected]<mailto:[email protected]>>; Dayavanti Gopal Kamath
<[email protected]<mailto:[email protected]>>;
Jamo Luhrsen <[email protected]<mailto:[email protected]>>; Vishal Thapar
<[email protected]<mailto:[email protected]>>;
[email protected]<mailto:[email protected]>
Subject: Netvirt Cluster CSIT : Table Miss Entry programming failing randomly
Anil,
I have raised the below JIRA, where I see Table Miss flow for Table 43 is
sometimes not getting programmed on one of the computes in netvirt 3 node CSIT.
https://jira.opendaylight.org/browse/OPNFLWPLUG-1028
I do see flow being pushed to config/inventory by netvirt, but it is missing on
switch.
This makes the basic DHCP itself fail for netvirt as this is one of the
important flows needed for all broadcasts to work.
Any pointers?
Thanks,
Faseela
_______________________________________________
openflowplugin-dev mailing list
[email protected]
https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev
