ohhh... AIX.. oprofile & latencytop is for Linux only :( so. If i understand correctly -- You know that slowdown you are interesting lay in protobuf -- all other [network / request data size / request complexity ] is the same at the start and after the week
from my exp [oh, also very limited] AIX networking is the last thing to check definitely, i cant to propose smth usefull... :( may be, -- You alloc tooo memory -- You need to use profiling tools -- reduce problem to smth small & publish code Dmitry пятница, 11 января 2013 г., 12:45:40 UTC+4 пользователь [email protected] написал: > > Thanks for your suggestions. Please find my response below: > > > On Friday, January 11, 2013 1:16:19 PM UTC+5:30, Дмитрий Дьяченко wrote: >> >> if i undestand correctly, You have a looong-run application. >> > Yes, you are right, my application is a really looooong running one. It is > expected to run 24 x 7 x 365. > > >> - at start You have 2 ms cycle request-response >> > The actual request response cycle is a bit more than 2 ms, because after > the request is recieved by the application (#9 in my list), it does a few > things, which may take longer. But, the time taken to decode the protobuf > request is 2 ms at start. > > >> - after a week You have 200 ms cycle >> > Right, the time taken to decode gradually increases and becomes 200 ms. If > left running for longer, it even reached 1 sec in 15 days time. > > >> - Your app is correct and only place where problem may lay is protobuf? >> > From the logs which are there in my application, I have seen that only the > time taken to decode is gradually increasing. I have also done some memory > and thread profiling of my application, but did not find any irregularities > (like increase in memory/threads) > > >> >> can You do the following? >> -- run 'latencytop' at start and after a week? >> -- run 'oprofile' at start and after a week? >> > My application is running on IBM AIX server (using IBM JVM). Are the above > tools available for AIX? > > >> -- may be its reasonable to writeout times from crititical points of Ypur >> app? smth like: before req? after req? before response - after response >> > I have logs specific for this purpose which are there at the start and end > of these points. Like, before decode and after decode, etc. From these logs > I found out the point which was gradually slowing down. > > >> from my [very limited] exp a such problems traditionally becomes from >> unintendend sources -- memory allocator, log sub-system, database, swap, >> etc... >> > The only other component involved over here (other than the application > and JMV) is the network. But, I don't think network is the issue because > the decoding is done after the request object is fully received from the > client. Apache MINA is responsible for getting the request from the client. > > >> >> Thanks, >> Dmitry >> >> пятница, 11 января 2013 г., 10:50:54 UTC+4 пользователь >> [email protected] написал: >>> >>> My application uses Google Protocol Buffer with Apache MINA. There are >>> clients who connect to the MINA Connector, send request to the server and >>> then wait for the response from the server. The flow of control is as >>> follows: >>> 1. Client sends request >>> 2. Request received at MINA filter (<Class> >>> extends CumulativeProtocolDecoder) >>> 3. doDecode() method is called >>> 4. A request object (generated from *.proto file) is created >>> using the <RequestObject>.parseFrom(bytes) >>> 5. The request is passed on to the IOHandler (<Class> >>> extends IoHandlerAdapter) >>> 6. messageReceived() method is called >>> 7. In this method, the request object (from #7) >>> is processed to create the list of requests which has been sent by the >>> client. >>> * 8. At this point, we have noticed that the >>> time taken to process the request object (#7) is gradually degrading with >>> time. From a initial period of around 2 ms, the time period is going up to >>> 200 ms in just 8 days of continuous usage. And this value gradually >>> increases with time.* >>> 9. The request list is processed in the >>> application >>> 10. The response object is created >>> 11. The response is sent to the client. >>> >>> Any suggestion would be highly appreciated. >>> >> -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To view this discussion on the web visit https://groups.google.com/d/msg/protobuf/-/aJPz4InH5sQJ. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
