artnaseef opened a new issue, #1731:
URL: https://github.com/apache/activemq/issues/1731
The deadlock below was observed in ActiveMQ 5.16.7. Reviewing changes to
the code involved between 5.16.7 and the latest on main as of today
(2026-02-28), it appears this deadlock still exists in the latest sources on
`main`.
```
Found one Java-level deadlock:
=============================
"ActiveMQ Transport: tcp:///xxx.xxx.xxx.1:47818@61616":
waiting to lock monitor 0x00007fe0a004a168 (object 0x00007fe1d44164c8, a
java.util.LinkedList),
which is held by "ActiveMQ Transport: tcp:///xxx.xxx.xxx.2:59618@61616"
"ActiveMQ Transport: tcp:///xxx.xxx.xxx.2:59618@61616":
waiting to lock monitor 0x00007fe1908c7d68 (object 0x00007fe1d5107548, a
java.util.LinkedList),
which is held by "ActiveMQ Transport: tcp:///xxx.xxx.xxx.3:40762@61616"
"ActiveMQ Transport: tcp:///xxx.xxx.xxx.3:40762@61616":
waiting to lock monitor 0x00007fe0a004a168 (object 0x00007fe1d44164c8, a
java.util.LinkedList),
which is held by "ActiveMQ Transport: tcp:///xxx.xxx.xxx.2:59618@61616"
Java stack information for the threads listed above:
===================================================
"ActiveMQ Transport: tcp:///xxx.xxx.xxx.1:47818@61616":
at
org.apache.activemq.store.jdbc.JDBCMessageStore.addMessage(JDBCMessageStore.java:131)
- waiting to lock <0x00007fe1d44164c8> (a java.util.LinkedList)
at
org.apache.activemq.store.memory.MemoryTransactionStore.addMessage(MemoryTransactionStore.java:362)
at
org.apache.activemq.store.memory.MemoryTransactionStore$1.asyncAddQueueMessage(MemoryTransactionStore.java:161)
at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:885)
at org.apache.activemq.broker.region.Queue.send(Queue.java:756)
at
org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:511)
at
org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:477)
at
org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:343)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154)
at
org.apache.activemq.broker.scheduler.SchedulerBroker.send(SchedulerBroker.java:320)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154)
at
org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
at
org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:295)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154)
at
org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:580)
at
org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:769)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:335)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:200)
at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:125)
at
org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:301)
at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)
at java.lang.Thread.run(Thread.java:750)
"ActiveMQ Transport: tcp:///xxx.xxx.xxx.2:59618@61616":
at org.apache.activemq.broker.region.Queue.onAdd(Queue.java:776)
- waiting to lock <0x00007fe1d5107548> (a java.util.LinkedList)
at
org.apache.activemq.store.jdbc.JDBCMessageStore.addMessage(JDBCMessageStore.java:149)
- locked <0x00007fe1d44164c8> (a java.util.LinkedList)
at
org.apache.activemq.store.memory.MemoryTransactionStore.addMessage(MemoryTransactionStore.java:362)
at
org.apache.activemq.store.memory.MemoryTransactionStore$1.asyncAddQueueMessage(MemoryTransactionStore.java:161)
at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:885)
at org.apache.activemq.broker.region.Queue.send(Queue.java:756)
at
org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:511)
at
org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:477)
at
org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:343)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154)
at
org.apache.activemq.broker.scheduler.SchedulerBroker.send(SchedulerBroker.java:320)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154)
at
org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
at
org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:295)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154)
at
org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:580)
at
org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:769)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:335)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:200)
at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:125)
at
org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:301)
at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)
at java.lang.Thread.run(Thread.java:750)
"ActiveMQ Transport: tcp:///xxx.xxx.xxx.3:40762@61616":
at
org.apache.activemq.store.jdbc.JDBCMessageStore$2.run(JDBCMessageStore.java:153)
- waiting to lock <0x00007fe1d44164c8> (a java.util.LinkedList)
at
org.apache.activemq.broker.region.Queue.rollbackPendingCursorAdditions(Queue.java:788)
- locked <0x00007fe1d5107548> (a java.util.LinkedList)
at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:895)
at org.apache.activemq.broker.region.Queue.send(Queue.java:756)
at
org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:511)
at
org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:477)
at
org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:343)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154)
at
org.apache.activemq.broker.scheduler.SchedulerBroker.send(SchedulerBroker.java:320)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154)
at
org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
at
org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:295)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154)
at
org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:580)
at
org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:769)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:335)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:200)
at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:125)
at
org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:301)
at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)
at java.lang.Thread.run(Thread.java:750)
Found 1 deadlock.
```
These are the key code snippets invovled:
```
org.apache.activemq.store.jdbc.JDBCMessageStore#addMessage
synchronized (pendingAdditions) { pendingAdditions.remove(sequence);
}
org.apache.activemq.broker.region.Queue#rollbackPendingCursorAdditions
synchronized (indexOrderedCursorUpdates) {
```
There are 2 locks being acquired in different orders by 2 threads.
The lead-up is an exception thrown by `store.addMessage`, and the `catch`
clause then calls `rollbackPendingCursorAdditions` on the queue.
Here is the full log of the offending exception:
```
WARN | Error while closing connection: IO Error: Connection timed out, due
to: IO Error: Connection timed out
java.sql.SQLRecoverableException: IO Error: Connection timed out
at
oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:993)
at
oracle.jdbc.driver.OraclePreparedStatement.executeForRowsWithTimeout(OraclePreparedStatement.java:9482)
at
oracle.jdbc.driver.OraclePreparedStatement.executeLargeBatch(OraclePreparedStatement.java:9586)
at
oracle.jdbc.driver.T4CPreparedStatement.executeLargeBatch(T4CPreparedStatement.java:1447)
at
oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:9517)
at
oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:237)
at
org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:241)
at
org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:241)
at
org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:241)
at
org.apache.activemq.store.jdbc.TransactionContext.executeBatch(TransactionContext.java:138)
at
org.apache.activemq.store.jdbc.TransactionContext.executeBatch(TransactionContext.java:116)
at
org.apache.activemq.store.jdbc.TransactionContext.close(TransactionContext.java:169)
at
org.apache.activemq.store.jdbc.JDBCMessageStore.addMessage(JDBCMessageStore.java:168)
at
org.apache.activemq.store.memory.MemoryTransactionStore.addMessage(MemoryTransactionStore.java:362)
at
org.apache.activemq.store.memory.MemoryTransactionStore$1.asyncAddQueueMessage(MemoryTransactionStore.java:161)
at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:885)
at org.apache.activemq.broker.region.Queue.send(Queue.java:756)
at
org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:511)
at
org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:477)
at
org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:343)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154)
at
org.apache.activemq.broker.scheduler.SchedulerBroker.send(SchedulerBroker.java:320)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154)
at
org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
at
org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:295)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154)
at
org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:580)
at
org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:769)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:335)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:200)
at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:125)
at
org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:301)
at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)
at java.lang.Thread.run(Thread.java:750)
```
Note that I created a modified version of the code to reliably reproduce the
problem in local testing. That test code can be seen on branch
`asn-debug-jdbc-slow-exception` on my github fork of the project:
[https://github.com/artnaseef/activemq/commit/e83f0603fa20f99a40a31b5d86ac3c73e4f724ba](https://github.com/artnaseef/activemq/commit/e83f0603fa20f99a40a31b5d86ac3c73e4f724ba)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
For further information, visit: https://activemq.apache.org/contact
