I'll paste an output from jmap -hist below, but I don't see anything really
obvious.
After taking a look at the hibernate connection code/config, I see that its
creating its own org.apache.commons.dbcp.BasicDataSource. I'm thinking of
adding a check for a new property "jndi" and if exists, then grab the
connection from jndi and return it. This would be in
https://github.com/geoserver/geoserver/blob/master/src/extension/monitor/hibernate/src/main/java/org/geoserver/monitor/hib/MonitoringDataSource.java#L43.
I feel like this would be a productive change as it gives users more
flexibility on the datastore. Not sure if it will help me debug the
original issue or not.
Debugger attached successfully.
Server compiler detected.
JVM version is 25.91-b14
Iterating over heap. This may take a while...
Object Histogram:
num #instances #bytes Class description
--------------------------------------------------------------------------
1: 1503194 5388248056 byte[]
2: 15058 1079421176 short[]
3: 1612662 5302276344 int[]
4: 6408289 421293344 char[]
5: 4352097 243717432
java.util.concurrent.locks.AbstractQueuedSynchronizer$Node
6: 5664192 181254144 java.lang.String
7: 1045195 144382128 java.lang.Object[]
8: 1941330 93183840 java.util.HashMap$Node
9: 365526 90970312 double[]
10: 507738 75089024 java.util.HashMap$Node[]
11: 752820 48180480 java.util.HashMap
12: 942625 45246000 com.vividsolutions.jts.geom.Envelope
13: 248974 26250792 java.lang.String[]
14: 722921 21745360 java.lang.Class[]
15: 533780 21351200 java.util.ArrayList
16: 102355 21258360 byte[][]
17: 416125 19974000 java.math.BigInteger
18: 16790 17595920 java.util.TimerTask[]
19: 253273 16209472 com.vividsolutions.jts.geom.Polygon
20: 231165 14794560 java.nio.StringCharBuffer
21: 253561 14199416
com.vividsolutions.jts.geom.LinearRing
22: 80535 14174160
org.postgresql.jdbc3.Jdbc3PreparedStatement
23: 326772 13070880
com.vividsolutions.jts.geom.Coordinate
24: 231628 12971168 java.nio.HeapByteBuffer
25: 38783 12417256 float[]
26: 140498 12363824 org.hsqldb.index.NodeAVLDisk
27: 181305 11603520 java.util.TreeMap$Entry
28: 286954 11478160 java.util.ArrayList$Itr
29: 227118 10901664
org.geotools.geometry.jts.LiteCoordinateSequence
30: 280868 8987776 java.awt.geom.Point2D$Double
31: 130362 8343168 java.lang.ref.Finalizer
32: 200430 8017200
java.util.concurrent.ConcurrentHashMap$MapEntry
33: 57079 7854136 java.util.Hashtable$Entry[]
34: 161206 7737888 java.util.concurrent.ConcurrentHashMap$Node
35: 56001 7616136 java.lang.reflect.Constructor
36: 135912 7611072 java.lang.ThreadLocal$ThreadLocalMap$Entry
37: 16790 7521920 java.util.TimerThread
38: 77932 6858016 java.util.LinkedHashMap
39: 100934 6459776 java.util.LinkedHashMap$Entry
40: 80550 6444000 org.postgresql.core.v3.SimpleQuery
41: 133923 6428304 org.geoserver.catalog.impl.ModificationProxy
42: 39274 6283840 java.lang.reflect.Method
43: 253273 6080856 com.vividsolutions.jts.geom.LinearRing[]
44: 57481 5978024
org.geotools.parameter.DefaultParameterDescriptor
45: 120938 5805024 java.io.File
46: 64827 5704776 java.util.regex.Matcher
47: 64230 5652240 org.springframework.core.ResolvableType
48: 109630 5262240 java.util.Hashtable$Entry
49: 164408 5261056 java.lang.StringBuilder
50: 63337 4352752 long[]
51: 42858 4114368 org.geotools.geometry.jts.PolygonIterator
52: 102614 4104560 org.geotools.feature.NameImpl
53: 50859 4068720 java.util.TreeMap
54: 16945 4066800
org.geoserver.wms.GetMapRequest$OptionalParameters
55: 50734 4058720
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync
56: 23376 4051432 java.lang.Class
57: 13425 4030632
java.util.concurrent.ConcurrentHashMap$Node[]
58: 60506 3872384 org.geotools.util.LocalName
59: 58169 3722816 org.geotools.referencing.NamedIdentifier
60: 20279 3569104 javax.media.jai.WritableRenderedImageAdapter
61: 62457 3497592 org.postgresql.core.v3.SimpleParameterList
62: 108209 3462688 org.geoserver.catalog.MetadataMap
63: 60893 3410008 java.io.ObjectStreamClass$WeakClassKey
64: 52408 3354112 java.util.Hashtable
65: 20953 3352480 sun.awt.image.ByteInterleavedRaster
66: 136574 3277776 java.util.HashSet
67: 48957 3133248 java.util.StringTokenizer
68: 17573 3092848 java.io.ObjectStreamClass
69: 93077 2978464 java.awt.Rectangle
70: 37109 2968720 org.geotools.map.FeatureLayer
71: 16866 2833488 org.geoserver.wms.WMSMapContent
72: 57715 2770320 java.lang.ref.PhantomReference
73: 38042 2739024 org.postgresql.core.Field
74: 46600 2609600 java.util.HashMap$KeyIterator
75: 16933 2573816 java.awt.image.ComponentColorModel
76: 105860 2540640 java.lang.Boolean
77: 45153 2528568 java.util.LinkedHashMap$LinkedKeyIterator
78: 52280 2509440
java.util.concurrent.locks.ReentrantLock$NonfairSync
79: 34763 2502936
org.geotools.feature.simple.SimpleFeatureImpl
80: 38537 2466368 java.util.zip.Inflater
81: 17040 2453760 org.springframework.beans.BeanWrapperImpl
82: 43597 2441432 org.geotools.geometry.jts.ReferencedEnvelope
83: 58603 2344120 org.geotools.parameter.Parameter
84: 72081 2306592
java.util.Collections$UnmodifiableRandomAccessList
85: 69123 2211936 java.util.Arrays$ArrayList
86: 91022 2184528 java.util.concurrent.atomic.AtomicBoolean
87: 45455 2181840 java.lang.ref.WeakReference
88: 50773 2030920
java.util.concurrent.locks.ReentrantReadWriteLock
89: 50650 2026000 java.util.AbstractList$Itr
90: 24072 1925760 org.geoserver.wms.MapLayerInfo
91: 18300 1903200 java.util.concurrent.ConcurrentHashMap
92: 16945 1897840 org.geoserver.wms.GetMapRequest
93: 118489 1895824 java.lang.Object
94: 47274 1890960 java.util.Stack
95: 33653 1884568 java.io.ObjectStreamField
96: 46723 1868920 java.lang.StringBuffer
97: 2105 1850064 java.nio.ByteBuffer[]
98: 22719 1817520 org.geotools.feature.type.AttributeTypeImpl
99: 13250 1802000 org.geotools.jdbc.JDBCFeatureReader
100: 56248 1799936
java.util.concurrent.locks.ReentrantReadWriteLock$Sync$HoldCounter
101: 31970 1790320 java.util.HashMap$EntryIterator
102: 22270 1781600
java.util.concurrent.ConcurrentHashMap$EntryIterator
103: 22270 1781600
java.util.concurrent.ConcurrentHashMap$KeyIterator
104: 16766 1743664 sun.awt.image.BufImgSurfaceData
105: 30947 1733032
org.springframework.core.SerializableTypeWrapper$MethodParameterTypeProvider
106: 13405 1715840
org.geotools.feature.simple.SimpleFeatureTypeImpl
107: 26506 1694944 com.vividsolutions.jts.geom.Coordinate[]
108: 17288 1659648 org.postgresql.core.v3.SimpleQuery
109: 41316 1652640 java.util.LinkedList$Node
110: 25652 1641728
org.geotools.feature.type.AttributeDescriptorImpl
111: 51168 1637376
org.springframework.core.annotation.AnnotationUtils$AnnotationCacheKey
112: 14392 1611904 java.net.URL
113: 33389 1602672 org.geotools.geometry.jts.LiteShape2
114: 28157 1576792 java.lang.reflect.WeakCache$CacheKey
115: 16327 1567392 org.springframework.core.MethodParameter
116: 20545 1479240 java.awt.image.BufferedImage
117: 16743 1473384 org.geotools.map.MapContent
118: 16695 1469160
javax.imageio.stream.MemoryCacheImageOutputStream
119: 25393 1422008
com.vividsolutions.jts.geom.prep.PreparedPolygon
120: 42864 1372672 com.vividsolutions.jts.geom.LineString[]
121: 56413 1353912 java.lang.Integer
122: 16870 1349600 org.geotools.map.MapViewport
123: 33609 1344360 org.geotools.map.MapContent$LayerList
124: 20918 1338752 java.awt.image.PixelInterleavedSampleModel
125: 55489 1331736 java.lang.Long
126: 20507 1312448 java.awt.image.DataBufferByte
127: 23278 1303568 java.lang.ref.SoftReference
128: 17600 1282712 com.vividsolutions.jts.geom.Polygon[]
129: 10365 1243800 java.lang.reflect.Field
130: 51791 1242984 java.util.concurrent.locks.ReentrantLock
131: 17213 1239336 org.hsqldb.RowAVLDisk
132: 15381 1230480
org.geotools.referencing.operation.transform.AffineTransform2D
133: 50787 1218888
java.util.concurrent.locks.ReentrantReadWriteLock$Sync$ThreadLocalHoldCounter
134: 50779 1218696
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock
135: 50773 1218552
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock
136: 18603 1190592 org.geoserver.ows.util.KvpMap
137: 6456 1187008
org.apache.commons.collections.map.AbstractHashedMap$HashEntry[]
138: 10497 1175664 org.geowebcache.storage.TileObject
139: 47807 1147368 java.util.HashMap$EntrySet
140: 2625 1134000 java.lang.Thread
141: 4252 1130520 com.vividsolutions.jts.geom.Geometry[]
142: 14128 1130240 org.geoserver.catalog.impl.DimensionInfoImpl
143: 17485 1119040 org.geoserver.wfs.kvp.SRSEnvelope
144: 23247 1115856 java.util.LinkedList
145: 34763 1112416 org.geotools.filter.identity.FeatureIdImpl
146: 5022 1111536
java.io.ObjectInputStream$HandleTable$HandleList[]
147: 19800 1108800 java.lang.reflect.Proxy$Key1
148: 10649 1107496 org.geotools.coverage.CategoryList
149: 27289 1091560 java.util.regex.Pattern$5
150: 17040 1090560 org.springframework.core.io.ResourceEditor
151: 17040 1090560
org.springframework.core.io.support.ResourceArrayPropertyEditor
152: 16945 1084480
org.geoserver.wms.GetMapRequest$MandatoryParameters
153: 21117 1013616 javax.media.jai.WritablePropertySourceImpl
154: 31249 999968 com.vividsolutions.jts.geom.PrecisionModel
155: 11299 994312 org.geotools.feature.type.GeometryTypeImpl
156: 30255 968160 java.util.Collections$SynchronizedSet
157: 17040 954240
org.springframework.beans.propertyeditors.URIEditor
158: 19799 950352
org.apache.commons.collections.map.AbstractHashedMap$HashEntry
159: 39190 940560 com.sun.proxy.$Proxy21
160: 38666 927984 java.util.zip.ZStreamRef
161: 11855 910240 boolean[]
162: 7522 902640
com.sun.org.apache.xerces.internal.dom.ElementNSImpl
163: 15919 891464 com.vividsolutions.jts.geom.MultiPolygon
164: 36489 875736
java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$UnmodifiableEntry
165: 36417 874008 java.util.LinkedHashMap$LinkedKeySet
166: 36290 870960 org.geotools.filter.LiteralExpressionImpl
167: 12067 868824 org.geotools.data.postgis.WKBReader
168: 225 866256
it.geosolutions.imageio.plugins.tiff.TIFFField[]
169: 17873 857904 java.util.Collections$UnmodifiableMap
170: 26618 851776
org.springframework.beans.TypeConverterDelegate
171: 26405 844960
com.vividsolutions.jts.geom.impl.CoordinateArraySequence
172: 21117 844680 javax.media.jai.PropertyChangeSupportJAI
173: 11629 837288
com.google.common.cache.LocalCache$StrongAccessEntry
174: 17339 832272 org.postgresql.jdbc.FieldMetadata
Jason Newmoyer
Newmoyer Geospatial Solutions
843.606.0424
[email protected]
On Sat, Dec 10, 2016 at 10:43 AM, Andrea Aime <[email protected]>
wrote:
> Hi Jason,
> did you find out what was going on in the end?
>
> Cheers
> Andrea
>
> On Wed, Dec 7, 2016 at 9:01 PM, Jason Newmoyer <
> [email protected]> wrote:
>
>> Thank you for the thorough response. I will poke around with JMap and see
>> what I can figure out.
>>
>> The GWC service is indeed the primary endpoint being requested.
>>
>> I'm also suspect of our memory settings in PostgreSQL. We have tons of
>> memory available and use pgtune to make use of it.
>>
>> I'll let you know what I find out.
>>
>>
>> Jason Newmoyer
>> Newmoyer Geospatial Solutions
>> 843.606.0424 <(843)%20606-0424>
>> [email protected]
>>
>>
>>
>> On Sat, Dec 3, 2016 at 4:42 AM, Andrea Aime <[email protected]
>> > wrote:
>>
>>> On Fri, Dec 2, 2016 at 7:11 PM, Jason Newmoyer <
>>> [email protected]> wrote:
>>>
>>>> Setup is: RHEL 6, GeoServer (war) 2.10.0, Jetty 9.3, PostgreSQL 9.4,
>>>> monitoring plugin with hibernate enabled, configured for JNDI datasource
>>>> provided by Jetty c3p0
>>>>
>>>
>>> I know that saving with hibernate tends to become slower and slower and
>>> ... slower, but that is related to the inserts themselves
>>> on the table, checking for the primary key uniqueness becomes
>>> progressively slower until it becomes completely
>>> unusable. That's why, at least in my company, we never use the hibernate
>>> storage, but enable audit mode to dump
>>> on file instead, and then transfer the info into elasticsearch using
>>> logstash.
>>>
>>> Database storage progressive slowdown should be fixable by removing the
>>> primary key, however that is probably going to require
>>> other changes, I have vague memories that the code might saving a record
>>> and then updating it later, which might not work
>>> without a primary key to refer to for the update statement.
>>>
>>>
>>>>
>>>> System is under pretty heavy use (1000's of requests per hour) and
>>>> while the extension does what its intended to do, we end up with an
>>>> unstable GeoServer after several hours and must be restarted. System
>>>> reports resident memory slowly increasing unbounded for the java process.
>>>> At about 2x the max heap size setting it starts to fall over. Services
>>>> become unreachable and log starts printing huge lists of threads. Even the
>>>> shutdown signal to Jetty seems to be ignored and the process must be
>>>> SIGKILL'd.
>>>>
>>>
>>> You should collect a memory dump using jmap when things start getting
>>> bad and see what's filling the memory.
>>> Or just get one using Yourkit, they do have a 15 days trials license
>>> which is normally good enough for a one-off analysis.
>>>
>>>
>>>> I'm curious if this has been reported before if anyone can provide
>>>> leads as to the cause. I checked JIRA and didn't see anything. I'm willing
>>>> to peak around in the monitor code.
>>>>
>>>> Also, a lot of our users and using KML superoverlays and this causes
>>>> about 90% of our requests to be operation='dispatch' which seem completely
>>>> useless to me. Anyone know what the purpose of that is? I'm interested in
>>>> making a configuration parameter to ignore these.
>>>>
>>>
>>> I'm not sure about this one... You probably want to check somewhere
>>> around here:
>>> https://github.com/geoserver/geoserver/blob/master/src/exten
>>> sion/monitor/core/src/main/java/org/geoserver/monitor/ows/
>>> MonitorCallback.java#L77
>>> although you'll probably also end up looking into the main dispatcher
>>> here:
>>> https://github.com/geoserver/geoserver/blob/master/src/ows/s
>>> rc/main/java/org/geoserver/ows/Dispatcher.java
>>>
>>> I'm making a guess here, maybe you are hitting some GWC service? Those
>>> are not playing by the same rules as the GeoServer inner ones
>>> and I believe that could be a reason why monitoring has troubles
>>> figuring out the details
>>>
>>> Cheers
>>> Andrea
>>>
>>> --
>>> ==
>>> GeoServer Professional Services from the experts! Visit
>>> http://goo.gl/it488V for more information.
>>> ==
>>>
>>> Ing. Andrea Aime
>>> @geowolf
>>> Technical Lead
>>>
>>> GeoSolutions S.A.S.
>>> Via di Montramito 3/A
>>> 55054 Massarosa (LU)
>>> phone: +39 0584 962313 <+39%200584%20962313>
>>> fax: +39 0584 1660272 <+39%200584%20166%200272>
>>> mob: +39 339 8844549 <+39%20339%20884%204549>
>>>
>>> http://www.geo-solutions.it
>>> http://twitter.com/geosolutions_it
>>>
>>> *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*
>>>
>>> Le informazioni contenute in questo messaggio di posta elettronica e/o
>>> nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
>>> loro utilizzo è consentito esclusivamente al destinatario del messaggio,
>>> per le finalità indicate nel messaggio stesso. Qualora riceviate questo
>>> messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
>>> darcene notizia via e-mail e di procedere alla distruzione del messaggio
>>> stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
>>> divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
>>> utilizzarlo per finalità diverse, costituisce comportamento contrario ai
>>> principi dettati dal D.Lgs. 196/2003.
>>>
>>>
>>>
>>> The information in this message and/or attachments, is intended solely
>>> for the attention and use of the named addressee(s) and may be confidential
>>> or proprietary in nature or covered by the provisions of privacy act
>>> (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
>>> Code).Any use not in accord with its purpose, any disclosure, reproduction,
>>> copying, distribution, or either dissemination, either whole or partial, is
>>> strictly forbidden except previous formal approval of the named
>>> addressee(s). If you are not the intended recipient, please contact
>>> immediately the sender by telephone, fax or e-mail and delete the
>>> information in this message that has been received in error. The sender
>>> does not give any warranty or accept liability as the content, accuracy or
>>> completeness of sent messages and accepts no responsibility for changes
>>> made after they were sent or for other risks which arise as a result of
>>> e-mail transmission, viruses, etc.
>>>
>>> -------------------------------------------------------
>>>
>>
>>
>
>
> --
> ==
> GeoServer Professional Services from the experts! Visit
> http://goo.gl/it488V for more information.
> ==
>
> Ing. Andrea Aime
> @geowolf
> Technical Lead
>
> GeoSolutions S.A.S.
> Via di Montramito 3/A
> 55054 Massarosa (LU)
> phone: +39 0584 962313 <+39%200584%20962313>
> fax: +39 0584 1660272 <+39%200584%20166%200272>
> mob: +39 339 8844549 <+39%20339%20884%204549>
>
> http://www.geo-solutions.it
> http://twitter.com/geosolutions_it
>
> *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*
>
> Le informazioni contenute in questo messaggio di posta elettronica e/o
> nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
> loro utilizzo è consentito esclusivamente al destinatario del messaggio,
> per le finalità indicate nel messaggio stesso. Qualora riceviate questo
> messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
> darcene notizia via e-mail e di procedere alla distruzione del messaggio
> stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
> divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
> utilizzarlo per finalità diverse, costituisce comportamento contrario ai
> principi dettati dal D.Lgs. 196/2003.
>
>
>
> The information in this message and/or attachments, is intended solely for
> the attention and use of the named addressee(s) and may be confidential or
> proprietary in nature or covered by the provisions of privacy act
> (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
> Code).Any use not in accord with its purpose, any disclosure, reproduction,
> copying, distribution, or either dissemination, either whole or partial, is
> strictly forbidden except previous formal approval of the named
> addressee(s). If you are not the intended recipient, please contact
> immediately the sender by telephone, fax or e-mail and delete the
> information in this message that has been received in error. The sender
> does not give any warranty or accept liability as the content, accuracy or
> completeness of sent messages and accepts no responsibility for changes
> made after they were sent or for other risks which arise as a result of
> e-mail transmission, viruses, etc.
>
> -------------------------------------------------------
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
