Do you have AIE and/or Recon jobs running at all? Are those engines even installed anywhere? I have also seen this with special characters being passed in Diary fields. That was a few versions back but something to look at.
On Tue, 28 Jun 2011 08:30:24 -0400, Reiser, John J wrote > Mark, > I've been working with BMC Support since last week. They have had > me send many log files trying to capture an event but nothing has > shown up yet. I'll check into Spotlight. > > Theo, > Already disabled escalations, alerts etc. I even disabled every > filter with a notify action because that was the first bit of > workflow that would send arserver.exe running away. The system still > overloads. > > Joe, > I'll ask the VM admins to check the NIC settings. Since the SQL > Server is remote and on a huge SAN that side should be ok. The DBA > said he saw no unusual traffic to the ARSystem db. > > Rick, > There ARE FOUR Lights. Sorry can you tell I'm losing it. > The VM has 2 CPUs configured. > VMware Tools says version 8.3.7 build 341836. > > LJ, > I've been capturing Filter, thread, API, SQL logs and sent them to > BMC Support. They see no long queries or transactions that could be > causing the high cpu usage. I did combine sql and api. I'll add the > filter logs and see what kind of timing I get between the three. > > Thanks for the feedback. > If BMC can't solve it today I think I'm going to use Misi's rrrChive > and export and reload the user data after we rollback the meta(?) > data from before the problem started. > > --- > John J. Reiser > Remedy Developer/Administrator > Senior Software Development Analyst > Lockheed Martin - MS2 > The star that burns twice as bright burns half as long. > Pay close attention and be illuminated by its brilliance. - > paraphrased by me > > -----Original Message----- > From: Action Request System discussion list(ARSList) > [mailto:[email protected]] On Behalf Of Walters, Mark Sent: > Tuesday, June 28, 2011 4:59 AM To: [email protected] Subject: > EXTERNAL: Re: arserver.exe is consuming 100% cpu - possible DB > corruption? (Long Post) > > Grab a copy of Spotlight on Windows from www.quest.com and you can > use it to view the various threads within the arserverd.exe and work > out which one is causing the high CPU load. Once you have this you > can reference the thread/sql/api/filter logs to see what activity it > is. > > Mark > > I work for BMC, I don't speak for them. > > -----Original Message----- > From: Action Request System discussion list(ARSList) > [mailto:[email protected]] On Behalf Of Reiser, John J Sent: 27 > June 2011 22:27 To: [email protected] Subject: arserver.exe is > consuming 100% cpu - possible DB corruption? (Long Post) > > Hello Listers, > ARS 7.6.03 > MS 2003 Enterprise > MS SQL 2005 (remote) > Total home grown system. No OOTB modules. > > I have a real stumper here. It even has BMC scratching their heads. > I have a production system that is experiencing cpu overload that > runs up to 99 in the processes and sits there. The ARSystem server > is virtual machine. We thought maybe it was a MS "Patch Tuesday" > issue and we removed the 10 recent MS patches one at a time and > restarted the machine each time. The problem still exists after the > arserver service starts. Sometime immediately and sometimes it will > sit for 1- 20 minutes before it starts to hog the CPUs. To eliminate > any other OS and file system issues we grabbed a two week old backup > image of the server and restored it. The system came back ok for a > short while and then started to lock up the CPU again. Working with > BMC I set the logs on and restarted. We saw the system jump to 100% > within a minute and captured a 10MB arsql.log file. It can force the > overload at anytime by firing filter workflow with a notification > action in it. I disabled this one filter but the system still loaded > up. I added a Filter that ran a 0 and the only action was Goto 1000 > to jump all Filter actions that fired on the change of the Status > field in question. Still no joy. I've disabled every piece of Notify > workflow. That worked the best and kept the system alive for longer > stretches but we can't run a system that way. > > I've come to the realization that there may be corrupted information > in the DB object tables and I wanted to get some feedback. Using > rrrChive I can pull a copy of every form's data since, say, two > weeks ago. Then have the DBA restore the entire system from that > date. After the restore I would use rrrChive to reload the two > weeks' data (Modified date' > "06/11/2011") and hope for the best. > > Any workflow that was changed in the last two weeks is negligible > and could be recreated/updated as needed. > > Do you think this is a viable solution? > When I asked the BMC tech if I could dump the T,H & B tables ; > restore the db and reload the T, H & B tables he reminded me that > the arschema and other meta tables would probably be out of synch. > That's when I thought of using rrrChive. > > Sorry to be so long winded but I need to get this back online, BMC > can't find anything in the logs and I don't want to lose the tickets > we've taken in the last week. > > --- > John J. Reiser > Remedy Developer/Administrator > Senior Software Development Analyst > Lockheed Martin - MS2 > The star that burns twice as bright burns half as long. > Pay close attention and be illuminated by its brilliance. - > paraphrased by me > > _____________________________________________________________________________ __ > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend > wwrug11 www.wwrug.com ARSList: "Where the Answers Are" > > _____________________________________________________________________________ __ > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org > attend wwrug11 www.wwrug.com ARSList: "Where the Answers Are" > > _____________________________________________________________________________ __ > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org > attend wwrug11 www.wwrug.com ARSList: "Where the Answers Are" -- Open WebMail Project (http://openwebmail.org) _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug11 www.wwrug.com ARSList: "Where the Answers Are"
