FYI, there appear to be some fundamental problems with upgrading a 32-bit ARS
7.6 installation to 64-bit on a 64-bit Windows server. The 7.6.03 installer
for ARS _appeared_ to successfully upgrade the ARS 7.6.00.006 instance to
64-bit, in place, using the existing java 1.6.0_20 32-bit JDK/JRE and the newly
added java 1.6.0_20 64-bit JDK/JRE. The problems showed up when I attempted to
upgrade Atrium Core from 7.6.00.002 to 7.6.03, and ITSM from 7.6.00.001 to
7.6.03.
This test was on a Windows 2008 R2 Standard x64 vm with 8 gb RAM and two Xeon
E5440 processors, with a clean install last May (for reference, with sample
data only) of ARS 7.6.00.004, Atrium 7.6.00.001, ITSM 7.6.00.001, SLM 7.6.00
and Patch 001, RKM 7.5.00.001, and SRM 7.6.00.001. It had subsequently been
patched to ARS Patches 5 and 6, Atrium Core Patch 2, and SRM Patch 2.
Both of the application installers hang and fail (Atrium), or complete with
warnings (ITSM), after failing to restart the AR Server services properly.
Neither application installer is able to successfully shut down the AR Service
– the service shows as stopped in the Services MMC, but the arserver.exe
remains resident in the Task Manager (along with all of its bastard children –
the 32-bit java processes that it spawns). Then the application installer is
unable to restart the service, and it hangs. The errors look like this:
FROM THE ATRIUM CORE UPGRADE
Fri Sep 10 10:19:20 2010 : Action Request System(R) Server x64 Version 7.6.03
Build 001 201008170035
(c) Copyright 1991-2010 BMC Software, Inc.
Fri Sep 10 10:19:20 2010 ProcessMain : Another copy of the server is already
running on the same RPC socket (ARERR 35)
Fri Sep 10 10:19:20 2010 d:*program files (x86)*bmc
software*arsystem*confRemedyMutex390600
Fri Sep 10 10:19:20 2010: AR System server terminated -- fatal error
encountered (ARNOTE 21)
FROM THE ITSM UPGRADE
Tue Sep 14 21:41:30 2010 : Action Request System(R) Server x64 Version 7.6.03
Build 001 201008170035
(c) Copyright 1991-2010 BMC Software, Inc.
Tue Sep 14 21:41:30 2010 ProcessMain : Another copy of the server is already
running on the same RPC socket (ARERR 35)
Tue Sep 14 21:41:30 2010 d:*program files (x86)*bmc
software*arsystem*confRemedyMutex390600
Tue Sep 14 21:41:30 2010: AR System server terminated -- fatal error
encountered (ARNOTE 21)
This happens near the beginning of the Atrium Core installer; it sits there
dead in the water until you cancel it. The only way I got it to complete was
to wait until it hung, then go in manually in Task Manager and End Process on
arserver.exe (I should have also killed all of the *32 processes, as they were
left resident), and then manually starting the BMC Remedy Action Request System
Server ITSMT01 service in the Services console. This restarted the AR server
and allowed the Atrium Core upgrade to continue.
The ITSM upgrade is of course another multi-hour process (3:40 PM to 9:40 PM),
and the restart came near the end, not the beginning, so I was no longer in the
office to deal with it manually. The installer failed to stop anything, so
after issuing errors it completed with warnings. The Remedy ARS Server was
left stopped in the mmc (all other BMC services were running), but the
arserver.exe remained resident and this morning I could still log in from a
User Tool and open the applications.
I have had a ticket open with support on the Atrium Core upgrade since last
week, and they don’t know why the installer could not control the services,
but they have concluded that my intervention allowed the upgrade to complete
properly. I was able to go on and perform the post install changes to
Normalization, and create a CI in Asset Management and see it reconciled into
the CMDB class overnight, so in theory the Atrium CMDB is working. The ITSM
upgrade _may_ have succeeded as well, but there remains a fundamental problem
with the AR Server that is baffling the application installers. I suspect that
the service was not re-registered properly in the registry as a 64-bit
processes, because the installer is having problems addressing it; the errors
in the application install logs read:
(Sep 14 2010 09:51:41.691 PM
-0500),SEVERE,com.bmc.smbu.install.common.rule.engine.ar.state.change.WindowsArStateChangeStrategy,
THROWABLE EVENT {Description=[Error received when getting pid of
Service],Detail=[BMC Remedy Action Request System Server ITSMT01]},
Throwable=[java.io.IOException: The service is not running.
Again, the Service is NOT running in the Services mmc, but the arserver.exe is
still alive in the Task Manager, resulting in:
(Sep 14 2010 09:51:41.692 PM
-0500),SEVERE,com.bmc.smbu.install.common.rule.engine.ar.state.change.WindowsArStateChangeStrategy,
LOG EVENT {Description=[Failed to start service BMC Remedy Action Request
System Server ITSMT01],Detail=[Could not start AR Server in timely manner]}
(Sep 14 2010 09:51:41.693 PM
-0500),CONFIG,com.bmc.smbu.install.common.rule.engine.ar.state.change.WindowsArStateChangeStrategy,
LOG EVENT {Description=[BMC Remedy Action Request System Server ITSMT01:
Error state],Detail=[Stopped]}
(Sep 14 2010 09:51:41.693 PM
-0500),SEVERE,com.bmc.install.product.bmcremedyitsmsuite.core.tasks.RLEConfigurationTask,
THROWABLE EVENT {Description=[ITSMPostInstallTask],Detail=[Failed to restart
AR System Server]},
Throwable=[com.bmc.smbu.install.common.rule.engine.CommandExecutionException:
java.lang.Exception: Could not start AR Server in timely manner
I am going to try this on a physical server that I installed three weeks ago as
a clean 7.6.00.006 reference server (all of the ITSM SLM RKM SRM apps at the
highest patch levels) on Windows Server 2003 Enterprise x64 to see if the same
problems occur. I also want to see if something else odd happens that occurred
on the 2008 R2 VM – the Atrium Core 7.6.03 upgrade installer completely deleted
all List servers:
After the installation of Atrium Core 7.6.03, there is no longer an RPC Program
Number of 390635 defined for List threads, at all!!! This occurred after
selecting the checkbox to allow the Atrium installer to configure the threads
itself. There is a new Private Thread 390612 5 16, but the List Type is no
longer present and RPC Prog Number 390635 is no longer defined.
Needless to say, that was an unexpected behavior as well, but support is fine
with it!
In any case, this is certainly slowing me down in addressing the second
fundamental change to the HPD:Help Desk and CHG:Infrastructure Change modules
in a row – in each release since 7.0, which is of course where most of our
customizations are.
Christopher Strauss, Ph.D.
Call Tracking Administration Manager
University of North Texas Computing & IT Center
http://itsm.unt.edu/
