Cisco Subscriber Edge Services Manager Installation Guide 3.3
Tuning SESM for Optimal Performance
Downloads: This chapterpdf (PDF - 208.0KB) | Feedback

Tuning SESM for Optimal Performance

Table Of Contents

Tuning SESM for Optimal Performance

Platform Patches and Platform Specific Java/J2SE Patches

Java Virtual Machine

JVM Memory Allocation

Garbage Collection

Java Version

Jetty Web Server

SESM Memory Management

SESM Cache Settings

SSG MBEAN Settings

SSG MBEAN Statistics

SESM Logging


Tuning SESM for Optimal Performance


This chapter describes how to ensure optimal performance for SESM. The information in this chapter is relevant to both RADIUS and SPE deployments, but for SPE deployments, there are additional considerations, which this chapter does not cover (for example, configuring the SPE component).

This chapter contains the following topics:

Platform Patches and Platform Specific Java/J2SE Patches

Java Virtual Machine

Jetty Web Server

SESM Memory Management

SESM Cache Settings

SSG MBEAN Settings

SSG MBEAN Statistics

SESM Logging

Platform Patches and Platform Specific Java/J2SE Patches

Ensure that the platform running SESM applications conforms to the system requirements described in "Before You Begin" and is kept up to date with the latest available patches from the supplier. This must include general operating system patches and Java (J2SE)-specific patches.

For Sun/Solaris platforms, patch updates can be controlled and automated using the Sun Patch Manager. For more details, see the Sun website.

Without the Patch Manager, comparing installed patches with the latest available patches can be difficult. One approach is to keep a copy of the latest installed patch clusters. On a regular basis, visit the Sun website and compare the size of the latest available patch cluster files or read the associated readme for details.

The failsafe approach is to always install the latest available patch clusters, which can be downloaded from the Sun website.

Java Virtual Machine

JVM Memory Allocation

Garbage Collection

Java Version

JVM Memory Allocation

SESM is a suit of Java applications, all of which, individually, run within a Java Virtual Machine (JVM). The amount of memory available to the JVM, and hence the application, is controlled by the Java options, -Xms and -Xmx, which are passed to the JVM at startup.

By default, the amount of memory allocated to the JVM for each of the SESM web applications is set to 64 MB. This is set in the start script, <install-directory/jetty/bin/start.sh.


Note The start script, start.sh, is common to all SESM web applications. If you are running multiple SESM applications on the same server, it might be necessary to modify the SESM start scripts to allocate different amounts of memory to different applications


The amount of memory allocated to the JVM depends on many factors, such as the level of load or throughput to be supported, functions and features supported by the application, the type of resources requested by the client, and content of JSPs. For guidelines on estimating required memory and CPU for the SESM web portal see "Before You Begin."

The use of swap space degrades performance of the application, so you must ensure that only physical memory is used by the JVM. To do so, take into account the memory requirements of the OS and any other applications running on the same platform, and deduct this from the amount of physical memory on the machine before you allocate memory to the JVM.

The Java -Xms option sets the initial size of the Java heap. The -Xmx option sets the maximum size of the Java heap. Both should be set to the same value for optimal performance.

It is important to note that if the JVM is allowed to run out of memory, the results are unpredictable and can be catastrophic. It is therefore essential that you configure the system to prevent this from occurring. The Jetty logs should be monitored for OutOfMemory exceptions. If any instance of this exception is found, you must reconfigure.

Garbage Collection

The JVM has its own methods of memory management including a process called Garbage Collection. When a Java application releases an object, the memory that object occupied does not become immediately available for re-use.The Garbage Collector is responsible for reclaiming the memory from unused objects

The behavior of the Garbage Collection process can be modified using Java Garbage Collection options at start-up. It is recommended to use the default behavior. However, The Garbage Collection mechanism within Java is continually being improved and new releases of JVM might contain improvements or new Garbage Collection options that will improve performance. Further information on the garbage collector specific to Java 1.4.2 is available on the Sun website for Java developers.

Java Version

With each release of SESM, the latest mature version of Java—Java Runtime Environment (JRE)—is bundled with the SESM install image. It is recommended that you use this version, even if you must replace the JRE with a Java Development Kit (JDK).

Newer versions of Java often provide improvements in performance and Garbage Collection. Upgrading to later versions of SESM will ensure that you benefit from these improvements. However, if you do now want to upgrade SESM is not wanted, you can upgrade just the JVM.

Different versions of JVM can be downloaded with installation instructions from the Sun website for Java developers.

Jetty Web Server

Jetty is the default web server bundled with SESM, developed by Mort Bay Consulting, partners of Core Developers Network.

Some of the more important aspects of Jetty attributes and settings, are detailed here, but further and more detailed information can be found at the Mort Bay Consulting and Core Developers Network websites. For further details on optimizing Jetty, refer to the Jetty Optimization Guide on the Core Developers Network website.

Jetty MBeans are exposed via the SESM Application Manager (AM), which enables both monitoring and modifying the behavior of Jetty during runtime. Monitoring of the behavior is enhanced by Jetty's support of statistics collection, again available via AM.

Collecting statistics can, in some circumstances, affect the performance. Refer to the Statistics section of the Jetty Optimization Guide for details.

maxThreads

This limits the number of threads that can be allocated to connections for HTTP listeners, and effectively limits the number of simultaneous users of the server and consequently memory usage.

The number of simultaneous users depends on the client browsers used and the version of HTTP they are based on. The latest and most commonly used browsers are based on HTTP-1.1 and use persistent connections.

It is important to note that this is the main control limiting the load on the web server and web application, and must be set according to the memory and CPU available.

If OutOfMemory exceptions are occurring, you must either reduce maxThreads or increase the amount of memory allocated to the JVM.

minThreads

This is the minimum number of unused threads to keep within the thread pool, which enables Jetty to respond to a sudden increase in the load with little latency. A HTTP listener is considered to be low on resources when less than minThreads threads are available for allocation to the thread pool.

For optimal performance, a good rule of thumb is to set minThreads to 20% of maxThreads.

maxIdleTimeMs

This is effectively the maximum time a connection will be maintained without receiving requests. When this timer expires, the thread/connection will be released. Consequently, associated resources/memory will also be released, which will then be available for garbage collection.

Reducing the value of maxIdleTimeMs, reduces the time during which resources are needlessly in use. But this must be weighed up against the overhead of reestablishing a connection.

The value of this attribute is determined to a large extent by what the average subscriber is expected to do at the web portal, for example, go to the web portal, log in, and activate a service. If you assume that most subscribers would achieve these tasks within one minute, then that is a good value for this attribute.

lowResourcePersistTimeMS

This is effectively the maximum time a connection will be maintained without receiving requests, when Jetty is low on resources. It is recommended to leave this attribute at its default value.

lowOnResources

This is a condition or state that Jetty enters when less than minThreads are available for allocation to the thread pool on which, maxIdleTimeMs is replaced by lowResourcePersistTimeMS.

SESM Memory Management

When a new session is created, it is stored in a cache. To prevent memory filling up with sessions, the cache is managed as follows:

The cache contains a configurable number of sessions stored in a hard cache. The hard cache will not hold any more sessions than its configured limit. The default size of the hard cache is 2000 sessions.

When the hard cache is full, older sessions are pushed into a soft cache. Depending on how the garbage collector is implemented, Java memory management can remove sessions from the soft cache, if memory is low.

Older sessions can be removed from either cache once they have reached the configurable session timeout value. This prevents old sessions remaining in low usage systems, and the data going stale. You can change the session timeout to remove unused sessions quicker or slower, using the sessionCachePeriod attribute in the SESM MBean.

To manage session memory, change the hard cache size using the sessionCacheSize attribute in the SESM MBean. 2000 sessions generally fit easily into the default 64 MB of memory.

SESM Cache Settings

The following attributes are in the SESM MBean, which also provides statistics such as the number of active sessions and active authenticated sessions.

sessionCachePeriod

This is the maximum period a SESM session will remain idle. Decreasing the value of this attribute means that SESM sessions will be aged out sooner, freeing resources and memory. This must be weighed against the overhead of (re)creating SESM sessions.

As with the value of the Jetty attribute maxIdleTimeMs, you should consider the expected average client behavior when determining the value of this attribute.

profileCachePeriod

This is the maximum period a service profile, obtained from the RADIUS/AAA server, will remain idle. In addition, this is the frequency with which SESM sessions are checked for sessionCachePeriod.

SSG MBEAN Settings

THROTTLE

This determines the maximum number of outstanding RADIUS access-requests from the SESM web portal to SSG. RADIUS packets are identified by an identification field in the RADIUS packet. When the SESM web portal receives a response from SSG, it uses the identification, in the received RADIUS packet to determine to which request it corresponds.

The identification field is an octet; therefore the maximum value that can be applied to THROTTLE is 255.

The value of this attribute is applied to each SSG socket created by the SESM Web Portal. In the case where client subnets are mapped to SSGs, the maximum number of SSG sockets will be equal to the number of SSGs in the network

When a port-bundle host key is used (hence no client subnet mappings to SSGs), the maximum number of SSG Sockets will be equal to the sum of all source IP addresses in the port map configurations of all the SSGs in the network.

It is important to note that when THROTTLE number of requests are outstanding, if further access-requests are made to the SSG, these requests will be buffered by virtue of the Java thread being placed in the Wait state, until the requests outstanding drops below THROTTLE.

Under heavy load, this buffering can grow significantly, resulting in large delays in communication with SSG. Even when load has reduced or been removed, the buffer might take some time to clear.

The only protection against excessive buffering is again the maxThreads parameter in Jetty.

RETRIES

This is the number of attempts the SESM web portal will make in sending an access-request to the SSG, if it does not receive a response.


Note The name of this attribute is misleading. If RETRIES is set to 1, and no response to an access-request is received, it will not resend the request.


This value is set to 3 by default but it is recommended that you set it to 1. This is because the RADIUS configuration on the gateway device (SSG) would typically be set to resend access-requests to the RADIUS server when it does not receive a response. Therefore, if both SESM and the gateway are configured to resend, a cascade will result.

It is important to note, however, that not every access-request from the SESM web portal to the gateway will result in an access-request from the gateway to the RADIUS server, for example, the SESM web portal sends access-requests to the gateway requesting the status of a particular host. Also, service activation requests from the SESM web portal to the gateway will not result in an access-request from the gateway to the RADIUS server if the gateway already has the service profile cached.

TIMEOUTSECS

This is the time, in seconds, that the SESM web portal will wait for a response to an access-request sent to the gateway. The value of this attribute must be consistent with the RADIUS configuration on the gateway device. The following example is a section of a RADIUS configuration on a gateway device.

radius-server retransmit 3 
radius-server timeout 10 

TIMEOUTSECS must be greater than retransmit * timeout, which, in this example is 30 seconds.

SSG MBEAN Statistics

Any bottleneck in communication between the SESM web portal and the gateway will significantly affect overall performance. Such a bottleneck could exist for a number of reasons. Statistics provided in the SSG MBean can help determine the cause of the bottleneck.

numSent

This represents the number of access-request RADIUS packets sent by the SESM web application to the SSG.

numReceived

This represents the number of access-accept and access-reject RADIUS packets received by the SESM web application. Under perfect conditions, this number should follow numSent.

numExceptions

This represents input and output errors when RADIUS packets are sent and received.

numRetries

This represents the number of times a response to an access-request was not received within the period defined by TIMEOUTSECS. If this number is high compared to numSent, it might be an indication that either the SSG or the RADIUS server is not able to handle the load, or is slow in responding.

Try increasing the value of TIMEOUTSECS. If this does not solve the problem, you must investigate further and determine whether the SSG routers or RADIUS server are under too much load.

numDiscarded

This represents the number of RADIUS packets discarded because they arrived after the period defined by TIMEOUTSECS. If this number is high, try increasing the value of TIMEOUTSECS.

numInvalid

This represents the number of invalid RADIUS packets received because of an invalid authentication field or invalid packet length.

numCollisions

This represents the number of times the SESM web application tried to send an access-request to the SSG, but was not able to because the number of requests pending was equal to the value defined by THROTTLE.

If the value of numCollisions is high, try increasing the value of THROTTLE, to a maximum of 255. However, increasing the value of THROTTLE will increase the load on the SSG routers and on the RADIUS server, which could result in the number of numRetries and/or numDiscarded going up.

SESM Logging

SESM web applications and the Jetty web server produce logging information that is sent to log files on disk. This can have a significant affect on performance.

Write access to disk is slow compared to CPU speed. If logging information is being generated faster than it can be written to disk, the logging information itself uses up memory. This can also result in Java threads entering a Wait state because of the Thread Safe implementation of logging.

It is recommended that in production, you do not set the logging levels higher than the default settings. Reducing the level of logging for both Jetty and the web application, can improve performance.