Hello Jochen! Apologies for the delayed response...
This issue is being discussed in more detail @ https://github.com/Graylog2/graylog2-server/issues/3116 Please do let me know if you have any views on the same. Thank you! Regards, -Yogesh On Tuesday, December 6, 2016 at 10:05:35 PM UTC+5:30, Jochen Schalanda wrote: > > Hi Yogesh, > > please post your complete Graylog configuration file and describe your > setup in more detail. > > Cheers, > Jochen > > On Friday, 25 November 2016 02:34:53 UTC+1, Yogesh Rao wrote: >> >> Hi,Github Issue # 3116 >> <https://github.com/Graylog2/graylog2-server/issues/3116> >> >> We have just started using graylog2 (2.1.2) on centos 7. I keep getting >> below exception after few hours >> >> 2016-11-24T17:25:13.737+05:30 WARN [ProxiedResource] Unable to call >> http://10.101.160.95:9000/api/system on node >> >> java.net.SocketTimeoutException: timeout >> >> at okio.Okio$3.newTimeoutException(Okio.java:210) ~[graylog.jar:?] >> >> at okio.AsyncTimeout.exit(AsyncTimeout.java:288) ~[graylog.jar:?] >> >> at okio.AsyncTimeout$2.read(AsyncTimeout.java:242) ~[graylog.jar:?] >> >> at okio.RealBufferedSource.indexOf(RealBufferedSource.java:325) >> ~[graylog.jar:?] >> >> at okio.RealBufferedSource.indexOf(RealBufferedSource.java:314) >> ~[graylog.jar:?] >> >> at >> okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:210) >> ~[graylog.jar:?] >> >> at okhttp3.internal.http.Http1xStream.readResponse(Http1xStream.java:186) >> ~[graylog.jar:?] >> >> at >> okhttp3.internal.http.Http1xStream.readResponseHeaders(Http1xStream.java:127) >> >> ~[graylog.jar:?] >> >> at >> okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:53) >> >> ~[graylog.jar:?] >> >> at >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) >> >> ~[graylog.jar:?] >> >> at >> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) >> >> ~[graylog.jar:?] >> >> at >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) >> >> ~[graylog.jar:?] >> >> at >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) >> >> ~[graylog.jar:?] >> >> at >> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:109) >> ~[graylog.jar:?] >> >> at >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) >> >> ~[graylog.jar:?] >> >> at >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) >> >> ~[graylog.jar:?] >> >> at >> okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) >> ~[graylog.jar:?] >> >> at >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) >> >> ~[graylog.jar:?] >> >> at >> okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:124) >> >> ~[graylog.jar:?] >> >> at >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) >> >> ~[graylog.jar:?] >> >> at >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) >> >> ~[graylog.jar:?] >> >> at >> org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:59) >> >> ~[graylog.jar:?] >> >> at >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) >> >> ~[graylog.jar:?] >> >> at >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) >> >> ~[graylog.jar:?] >> >> at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:170) >> ~[graylog.jar:?] >> >> at okhttp3.RealCall.execute(RealCall.java:60) ~[graylog.jar:?] >> >> at retrofit2.OkHttpCall.execute(OkHttpCall.java:174) ~[graylog.jar:?] >> >> at >> org.graylog2.shared.rest.resources.ProxiedResource.lambda$null$0(ProxiedResource.java:76) >> >> ~[graylog.jar:?] >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_102] >> >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> >> [?:1.8.0_102] >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> >> [?:1.8.0_102] >> >> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102] >> >> Caused by: java.net.SocketTimeoutException: Read timed out >> >> at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_102] >> >> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) >> ~[?:1.8.0_102] >> >> at java.net.SocketInputStream.read(SocketInputStream.java:170) >> ~[?:1.8.0_102] >> >> at java.net.SocketInputStream.read(SocketInputStream.java:141) >> ~[?:1.8.0_102] >> >> at okio.Okio$2.read(Okio.java:138) ~[graylog.jar:?] >> >> at okio.AsyncTimeout$2.read(AsyncTimeout.java:238) ~[graylog.jar:?] >> >> ... 29 more >> >> 2016-11-24T17:25:13.751+05:30 WARN [jvm] >> [graylog-b8d88e71-dbbb-44ee-9951-2bcdb45b047a] [gc][old][19743][529] >> duration [12.4s], collections [1]/[12.6s], total [12.4s]/[10.4m], memory >> [3.7gb]->[3.7gb]/[3.8gb], all_pools {[young] >> [1.6gb]->[1.6gb]/[1.6gb]}{[survivor] [188.7mb]->[191.2mb]/[204.7mb]}{[old] >> [1.9gb]->[1.9gb]/[2gb]} >> 2016-11-24T17:25:13.821+05:30 INFO [BeatsCodec] inside >> decodeMessagesRawMessage{id=52182873-b20e-11e6-9b98-14feb5ea6076, >> journalOffset=960249, codec=beats, payloadSize=4169, >> timestamp=2016-11-24T06:22:05.798Z, remoteAddress=/10.101.160.6:49324} >> >> >> Heap Size set for Graylog2 is 4G, I noticed the heap size jumps up from >> 1.5G to 4G as soon as the exceptions are encountered, eventually leading to >> slower GC's and finally ends up unresponsive. >> >> All the buffer processors are set to 1 >> process_buffer and output buffer are set to 32768 >> >> Data pumped into graylog is around 4 GB >> >> Graylog is configured on a Core 2 Duo machine with 8GB Ram and is pumping >> data into 3 elasticsearch (clustered) instances. >> >> >> How can i troubleshoot this issue to figure out what the actual problem >> is ? >> > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/16650eae-26bc-46c6-9f4c-47a63d350d94%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
