I just worked through a longstanding issue that we have had in our environment related to Dev Cache Mode. Due to extremely slow performance when making admin changes on a 7.0 server, I had gotten into the habit of keeping our admin server set with Dev Cache Mode turned on, which significantly sped up migrations to production and such. However, in this environment (7.1 p5 server), I was seeing exactly the opposite - admin changes were very frequently timing out and migrations were painfully slow (for example, a patch that took a few minutes to apply in dev and stage took an hour and twenty minutes to apply in production). Thinking that it should go faster since Dev Cache Mode was turned on, I looked everywhere I could think of to try and isolate why it was so slow. I got BMC involved in my search twice. This last time, the support rep forwarded this same KB article to me and recommended that I leave Dev Cache Mode turned OFF in order to improve performance with admin actions. I tried it, and it worked. My migrations now go very quickly, and I haven't seen any timeouts since. The only slowness is in the initial period where the AR server creates a copy of the cache in memory, but that's generally fairly quick, since it's just doing an in memory copy rather than trying to pull it all from the database again (like I suspect it was doing on the 7.0 server).
In short, in our environment, we see significantly better performance making admin changes in production with Dev Cache Mode turned off. The support rep told me that since AR Server 7.1, their recommendation is to keep it turned off. That said, if I understand it correctly, your mileage may vary depending on what kind of activity you have going on on the server you're trying to make changes on. As I understand it, when Dev Cache Mode is turned on, since it doesn't make a copy of the cache for existing users, it has to wait until user operations have ceased before it can make the change, then it makes the change in between user operations. In our case, since this server also runs all our back-end processes (e-mail, escalations, etc.), and we apparently have some things going on that take a long time to complete, the admin thread ended up having to wait a long time (more than 10 minutes in some cases) before it could get a lock on the cache and make the changes. This was what was causing the timeouts and slow performance that we saw. Now, since it simply makes a relatively quick copy of the cache for existing user operations (just system operations in our case, but still, they're ongoing), the overall time to make the changes is quick, because once the cache is copied, it can proceed with the changes without having to wait for a break between user operations. So, if you don't have much going on on the server you're trying to make admin changes on, it could be quicker to leave Dev Cache Mode turned on, because then you don't have to wait for the cache to get copied (I think it takes less than 30 seconds in our case to copy about 400MB of cache). However, if you have lots of stuff going on (e-mail, user access, escalations, etc.), you may see better performance with it turned off. I hope this helps a bit. Lyle From: Action Request System discussion list(ARSList) [mailto:[email protected]] On Behalf Of Tony Worthington Sent: Friday, November 20, 2009 9:36 AM To: [email protected] Subject: Re: Development Cache Mode revisited ** I missed your question... sorry. - No admin-changes were done - And No changes to Groups of the type View/Change were done This is from BMC KB KM-000000024487. It doesn't answer your ? directly, but is full of good information. What may cause poor performance in the Admin Tool? When poor performance is seen in the Admin Tool when saving objects, it may be a result of how AR Server handles information it has read into memory. At server startup, AR Server reads information from the database into memory, for example all field properties are read into memory, all users and groups and active links, etc. When this information that has been read into memory changes, there are two different ways for AR Server to handle the event. It can update the existing memory footprint so that users all see the change that was made. It can create a separate memory footprint so existing users still utilize the existing memory footprint, while new users logging in would utilize the new memory footprint. Either method will cause a performance hit, but the big difference between the two ways of handling the event is where the performance impact is seen. Note: How noticeable the performance hit is depends on several factors, but mostly the amount of data that needs to be read into memory. This is usually a direct result of the number of objects (forms and workflow) that exist. The more objects, the greater the performance impact. Is there a way to improve performance or control which option AR Server uses? Yes. There is a server configuration, called Development Cache Mode, that was introduced to help with performance when dealing with systems that have large amounts of forms and workflow objects in a development environment. The Cache-Mode setting is available in ARServer 6.0.1 patch 1339 and above, and in ARServer 5.1.2 patch 1338 and above as well as in 6.3 and above. Note: only the 6.3 and above Admin Tool shows the GUI checkbox for Development Cache Mode in the Server Information dialog on the Configuration tab. If you're using an Admin Tool that does not have the checkbox for Development Cache Mode, then the Cache-Mode parameter must be manually added to the ar.cfg file. It can be found in the Admin Tool under File->Server Information on the Configuration tab. It is a check box called "Development Cache Mode" and by default, it is not selected. When selected, it puts a parameter in the ar.cfg/ar.conf file called Cache-Mode: 1. There are two cache modes, production and development. In production mode (Cache-Mode: 0), which is the default behavior if the line is not set in ar.cfg/ar.conf file (i.e. the Development Cache Mode is not checked in the Admin Tool), AR Server operates as it always has -- whenever an admin operation will change the memory cache, the admin thread makes a new copy of the memory cache and makes its changes there. While this slows performance for the admin making the change, it allows the change to be made without affecting other users of the system so it's appropriate for a production environment with many users. In development mode (Cache-Mode: 1 in the ar.cfg/ar.conf file or where the Development Cache Mode is selected in the Admin Tool), the admin thread will instead lock other threads out of the current copy of the cache while it makes its change in that copy. On each API call that will cause a change to the cache, the admin thread locks the cache, waits for all other threads to stop using the cache, makes its modifications, and unlocks the cache. The thread locks the entire cache, which means other threads cannot process API calls or escalations while the admin thread processes its API call. In a development environment this should be an acceptable cost. This effectively gives priority to the administrator at the expense of other users. It is therefore not really suitable for a production environment (blocking all users for the entire time that that the change is taking place is not good in production), but is great where development is the primary aim of the system. This mode (development mode) has a couple of notable effects -- The first is that the server will no longer copy the cache, though it will wait for user operations to complete before it proceeds with making a change. This will almost always be faster than copying the cache and the difference in time will be greater with the number of objects on the server. In other words, the larger the application and number of objects (i.e. forms, active links, etc.) on the server, the greater the effect of the cache mode will be. The second is that the server's memory footprint will be smaller so that it is less likely to cause paging; this will again improve the server performance. Changes to the cache mode only take effect after AR Server has been restarted. The expectation is that in a development environment, admin changes have priority over other users. In a production environment the opposite is true. So, the key thing to understand when using this parameter is that we lock the object when a change is applied (when we issue a Set, Delete, or Create API call), not when it is opened. This means that any number of admin tools may have an object open; if one of them changes the object and a second one makes another change, the first is overwritten. That behavior is no different than without the change, since there is only one admin thread and all admin operations are serialized. This should improve performance immensely when saving a form because you are only dealing with the current form and not all objects on the server being cached. Is there any risk to having Development Cache Mode enabled in production? Yes. An issue was discovered when development cache mode is enabled that can cause AR Server to appear hung. This is thoroughly described in KM-000000025743. This is NOT a common issue. Normally, there would be no risk other than the described behavior of AR Server when Admin changes occur. ------ Here's 25743 There is an important side-effect that can cause AR Server to appear hung and users receive timeout errors, like ARERR 93. This occurs under specific circumstances, but specifically when the Development Cache Mode is selected in the Admin Tool under File->Server Information on the Configuration tab (see KB 24487 for information on Development Cache Mode). Here is the specific course of events that leads to the problem: An escalation starts running and puts a Read Lock on the Server Cache tables (in memory). An Admin Operation (like a form or workflow save) starts but cannot put an Exclusive lock on the cache until the Escalation is done with its lock. The escalation continues to run. Users and utilities try to perform ARServer work (APIs). But they need a read lock on the cache and are forced to wait. The escalation keeps running. Users start getting timeouts (ARERR 93, etc.) and the Admin Tool times out. The escalation keeps running. Finally the escalation completes and the Admin Operation can start. Admin Operation completes and users work can start. Since the users already timed out, they've likely given up and moved into trouble shooting mode, unaware of whats happening with AR Server. Normally, escalations run pretty quick, but in some cases, this isn't true. For instance, lets say an escalation fires on a form with 200,000 records in it, updating each one. Each record touched causes Filters to fire which do Push Fields actions to several other forms for example, resulting in each of the 200,000 records to take roughly 18 seconds to complete. This escalation would take a very long time to complete and all the while, AR Server would appear as though its hung because of the Development Cache mode being enabled. When Development Cache mode is enabled, it will attempt to put an exclusive lock on the server cache tables in memory. When disabled, a separate memory footprint is created so there is no locking conflict with the cache and therefore this problem would not occur. This behavior is not considered a bug, but rather a circumstantial issue caused by the design of the product. Currently there is no workaround to this issue, but being aware of it means that there are options. For instance, normally Development Cache Mode is only enabled on a development server. If this is the case, then usually you can disable escalations or temporarily disable development cache mode in development (if the escalation must run for testing, etc.). You could also alter the escalation(s) so it doesn't touch so many records or doesn't cause filters to fire, etc. In other words, speed up the escalation processing time so any locks are very quick, or disable the escalation (or all escalations), or disable development cache mode. NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:[email protected] ARSlist: "Where the Answers Are"
