For most features, when you begin to configure settings for a new mode, those settings will initially be empty by default. There is a distinction between empty settings and having no settings in a mode. As an example, consider a very simple cluster composed of one group and one machine. Imagine that you have an LDAP query configured at the cluster level. There are no settings configured at the group or machine levels:

Now, imagine that you create new LDAP query settings for the group. The result will be something like this:

The group-level settings now override the cluster-level setting; however, the new group settings are initially empty. The group mode does not actually have any LDAP queries of its own configured. Note that a machine within this group will inherit this “empty” set of LDAP queries from the group.

Next, you can add an LDAP query to the group, for example:

Now the cluster level has one set of queries configured while the group has another set of queries. The machine will inherit its queries from the group.

A machine can create or join a cluster only via the clusterconfig command.

• When a new cluster is created , all of that cluster’s initial settings will be inherited from the machine that creates the cluster. If a machine was previously configured in “standalone” mode, its standalone settings are used when creating the cluster.
• When a machine joins a cluster, all of that machine’s clusterable settings will be inherited from the cluster level. In other words, everything except certain machine-specific settings (IP addresses, etc) will be lost and will be replaced with the settings from the cluster and/or the group selected for that machine to join. If a machine was previously configured in “standalone” mode, its standalone settings are used when creating the cluster, and no settings at the machine level are maintained.

If the current machine is not already part of a cluster, issuing the clusterconfig command presents the option to join an existing cluster or create a new one.

At this point you can add machines to the new cluster. Those machines can communicate via SSH or CCS.

newyork.example.com> clusterconfig

Do you want to join or create a cluster?

1. No, configure as standalone.

2. Create a new cluster.

3. Join an existing cluster over SSH.

4. Join an existing cluster over CCS.

[1]> 2

Enter the name of the new cluster.

[]> americas

New cluster committed: Wed Jun 22 10:02:04 2005 PDT

Creating a cluster takes effect immediately, there is no need to commit.

Cluster americas

Choose the operation you want to perform:

- ADDGROUP - Add a cluster group.

- SETGROUP - Set the group that machines are a member of.

- RENAMEGROUP - Rename a cluster group.

- DELETEGROUP - Remove a cluster group.

- REMOVEMACHINE - Remove a machine from the cluster.

- SETNAME - Set the cluster name.

- LIST - List the machines in the cluster.

- LISTDETAIL - List the machines in the cluster with detail.

- DISCONNECT - Temporarily detach machines from the cluster.

- RECONNECT - Restore connections with machines that were previously detached.

- PREPJOIN - Prepare the addition of a new machine over CCS.

[]>

For machines that are part of a cluster, the CLI can be switched into different modes. Recall that a mode refers to a specific, named, member of a level.

The CLI mode determines precisely where a configuration setting will be modified. The default is “machine” mode for the machine the user logged into, the “login host.”

Use the clustermode command to switch between different modes.

clustermode

clustermode group northamerica

clustermode machine losangeles.example.com

The prompt in the CLI changes to indicate your current mode.

(Cluster Americas)> 

or

(Machine losangeles.example.com)>

In machine mode, the prompt will include the fully qualified domain name of the machine.

All non-restricted (see Restricted Commands) commands have new operations: CLUSTERSHOW and CLUSTERSET . CLUSTERSHOW is used to show in which modes a command is configured (see New Operation Added). The CLUSTERSET operation allows you to move or copy the current settings (configurable with the current command) from one mode to another or between levels (e.g. from a machine to a group).

A copy retains the settings for the current mode. A move resets (clears) the configuration of the current mode; i.e., following a move, no settings will be configured for the current mode.

For example, if you have configured Good Neighbor Table settings (the destconfig command) for group northamerica , and you decide that you want the entire cluster to have these settings, you can use the clusterset operation from within the destconfig command to copy (or move) the current settings to the cluster mode. (See Experimenting with New Configurations.)

Caution

Exercise caution when moving or copying configuration settings to avoid inconsistent dependencies. For example, if you move or copy listeners with disclaimer stamping configured to another machine, and that new machine does not have the same disclaimers configured, disclaimer stamping will not be enabled on the new machine.

You use the REMOVEMACHINE operation of clusterconfig to remove a machine permanently from a cluster. When a machine is permanently removed from a cluster, its configuration is “flattened” such that it will work the same as it did when it was part of the cluster. For example, if there is only a cluster-mode Global Unsubscribe table, the Global Unsubscribe table data will be copied to the machine’s local configuration when the machine is removed from the cluster.

All CLI commands in AsyncOS are now cluster-aware. The behavior of some commands will change slightly when issued in a cluster mode. For example, the behavior of the following commands changes when issued on a machine that is part of a cluster:

Most CLI commands and their corresponding GUI pages can be run in any mode (cluster, group, or machine). However, some commands and pages are restricted to one mode only.

The system interface (either the GUI and the CLI) will always will make it clear that a command is restricted and how it is restricted. It is easy to switch to the appropriate mode for configuring the command.

• In the GUI, use the “Change Mode” menu or the “Settings for this features are currently defined at:” links to switch modes.
• In the CLI, use the clustermode command to switch modes.

If a you try to run one of these commands in group or machine mode, you will be given a warning message and the opportunity to switch to the appropriate mode.

Note

The passwd command is a special case because it needs to be usable by guest users. If a guest user issues the passwd command on a machine in a cluster, it will not print the warning message but will instead just silently operate on the cluster level data without changing the user’s mode. All other users will get the above written behavior (consistent with the other restricted configuration commands).

The following commands are restricted to machine mode :

antispamstatus

etherconfig

resume

suspenddel

antispamupdate

featurekey

resumedel

suspendlistener

antivirusstatus

hostrate

resumelistener

techsupport

antivirusupdate

hoststatus

rollovernow

tophosts

bouncerecipients

interfaceconfig

routeconfig

topin

deleterecipients

ldapflush

sbstatus

trace

delivernow

ldaptest

setgateway

version

diagnostic

nslookup

sethostname

vofflush

dnsflush

quarantineconfig

settime

vofstatus

dnslistflush

rate

shutdown

workqueue

dnslisttest

reboot

status

dnsstatus

resetcounters

suspend

If a you try to run one of the commands above in cluster or group mode, you will be given a warning message and the opportunity to switch to an appropriate mode.

The following commands are further restricted to the login host (i.e., the specific machine you are logged into). These commands require access to the local file system.

DNS is required to connect a machine to the cluster. Cluster communication is normally initiated using the DNS hostnames of the machines (not the hostname of an interface on the machine). A machine with an unresolvable hostname would be unable to actually communicate with any other machines in the cluster, even though it is technically part of the cluster.

Your DNS must be configured to have the hostname point to the correct IP interface on the email gateway that has SSH or CCS enabled. This is very important. If DNS points to another IP address that does not have SSH or CCS enabled it will not find the host. Note that centralized management uses the “main hostname,” as set with the sethostname command, not the per-interface hostname.

If you use an IP address to connect to another machine in the cluster, the machine you connect to must be able to make a reverse look up of the connecting IP address. If the reverse look up times out because the IP address isn’t in the DNS, the machine cannot connect to the cluster.

Cluster Communication Security (CCS) is a secure shell service similar to a regular SSH service. Cisco implemented CCS in response to concerns regarding using regular SSH for cluster communication. SSH communication between two machines opens regular logins (admin, etc.) on the same port. Many administrators prefer not to open regular logins on their clustered machines.

Tip: never enable Cluster Communication Services, even though it is the default, unless you have firewalls blocking port 22 between some of your clustered machines. Clustering uses a full mesh of SSH tunnels (on port 22) between all machines. If you have already answered Yes to enabling CCS on any machine, remove all machines from the cluster and start again. Removing the last machine in the cluster removes the cluster.

CCS provides an enhancement where the administrator can open up cluster communication, but not CLI logins. By default, the service is disabled. You will be prompted to enable CCS from the interfaceconfig command when you are prompted to enable other services. For example:

Do you want to enable SSH on this interface? [Y]>

Which port do you want to use for SSH?

[22]>

Do you want to enable Cluster Communication Service on this interface?

[N]> y

Which port do you want to use for Cluster Communication Service?

[2222]>

The default port number for CCS is 2222. You may change this to another open, unused, port number if you prefer. After the join is complete and the joining machine has all the configuration data from the cluster, the following question is presented:

Do you want to enable Cluster Communication Service on this interface? [N]> y


Which port do you want to use for Cluster Communication Service?

[2222]>

The machines that are “cluster aware” will continually verify network connections to other machines within the cluster. This verification is done by periodic “pings” sent to other machines in the cluster.

If all attempts to communicate with a particular machine fail, then the machine that has been trying to communicate will log a message saying that the remote host has disconnected. The system will send an alert to the administrator that the remote host went down.

Even if a machine is down, the verification pings will continue to be sent. When a machine rejoins the cluster network, a synchronization command will be issued so that any previously offline machines can download any updates. The synchronization command will also determine if there have been any changes on one side but not the other. If so, then the previously down machine will silently download the updates.

A machine may be disconnected from a cluster. Occasionally, you may intend to deliberately disconnect the machine, for example, because you are upgrading the machine. A disconnect could also occur by accident, for example, due to a power failure or other software or hardware error. A disconnect can also occur if one email gateway attempts to open more than the maximum number of SSH connections allowed in a session. A machine that is disconnected from a cluster can still be accessed directly and configured; however, any changes made will not be propagated to other machines within the cluster until the disconnected machine becomes reconnected.

When a machine reconnects to the cluster, it tries to reconnect to all machines at once.

In theory, two machines in a cluster that are disconnected could commit a similar change to their local databases at the same time. When the machines are reconnected to the cluster, an attempt will be made to synchronize these changes. If there is a conflict, the most recent change is recorded (supersedes any other changes).

During a commit, the email gateway checks every variable that is being changed. The commit data includes version information, sequence identification numbers, and other information that can be compared. If the data you are about to change is found to be in conflict with previous changes, you will be given the option to discard your changes. For example, you might see something like this:

(Machine mail3.example.com)> clustercheck

This command is restricted to "cluster" mode. Would you like to switch to "cluster"

mode? [Y]> y

Checking Listeners (including HAT, RAT, bounce profiles)...

Inconsistency found!

Listeners (including HAT, RAT, bounce profiles) at Cluster enterprise:

mail3.example.com was updated Mon Sep 12 10:59:17 2005 PDT by 'admin' on

mail3.example.com

test.example.com was updated Mon Sep 12 10:59:17 2005 PDT by 'admin' on

mail3.example.com

How do you want to resolve this inconsistency?

1. Force entire cluster to use test.example.com version.


2. Force entire cluster to use mail3.example.com version.

3. Ignore.

[1]>

If you choose not to discard your changes, they are still intact (but uncommitted). You can review your changes against the current settings and decide how to proceed.

You can also use the clustercheck command at any time to verify that the cluster is operating correctly.

losangeles> clustercheck

Do you want to check the config consistency across all machines in the cluster? [Y]> y

Checking losangeles...

Checking newyork...

No inconsistencies found.

It is recommended that you avoid configuring the following settings on the email gateway.

In a centrally managed environment, some interdependent settings are configured in different modes. The flexibility of the configuration model allows you to configure settings at multiple modes, and the laws of inheritance govern which settings will be used on a per-machine basis. However, some settings have dependencies on other settings, and the availability of the dependent settings’ configuration is not limited to settings at the same mode. Thus, it is possible to configure a setting for one level that references a setting that is configured for a specific machine at a different level.

The most common example of an interdependent setting involves a select field on a page that pulls data from a different cluster section. For example, the following features can be configured in different modes:

• using LDAP queries
• using dictionaries or text resources
• using bounce or SMTP authentication profiles.

Within centralized management, there are restricted and non-restricted commands. (See Restricted Commands.) Non-restricted commands are generally configuration commands that can be shared across the cluster.

The listenerconfig command is an example of a command that can be configured for all machines in a cluster. Non-restricted commands represent commands that can be mirrored on all machines in a cluster, and do not require machine-specific data to be modified.

Restricted commands, on the other hand, are commands that only apply to a specific mode. For example, users cannot be configured for specific machines — there must be only one user set across the whole cluster. (Otherwise, it would be impossible to login to remote machines with the same login.) Likewise, since the Mail Flow Monitor data, System Overview counters, and log files are only maintained on a per-machine basis, these commands and pages must be restricted to a machine.

You will notice that while Scheduled Reports may be configured identically across the whole cluster, the viewing of reports is machine-specific. Therefore, within a single Scheduled Reports page in the GUI, configuration must be performed at the cluster mode, but viewing of reports must be done at the machine mode.

The System Time pages encompass the settz , ntpconfig , and settime commands, and thus represents a mixture of restricted and non-restricted commands. In this case, settime must be restricted to machine-only modes (since time settings are specific for machine), while settz and ntpconfig may be configured at cluster or group modes.

In this representation, the listener “IncomingMail” is referencing a footer named “disclaimer” that has been configured at the machine level only. The drop-down list of available footer resources shows that the footer is not available on the machine “buttercup.run” which is also available in the cluster. There are two solutions to this dilemma:

• promote the footer “disclaimer” from the machine level to the cluster level
• demote the listener to the machine level to remove the interdependency

In order to fully maximize the features of a centrally managed system, the former solution is preferred. Be aware of interdependencies among settings as you tailor the configuration of your clustered machines.

When you create the cluster, the machine you happen to be logged into is automatically added to the cluster as the first machine, and also added to the Main_Group. Its machine level settings effectively get moved to the cluster level as much as possible. There are no settings at the group level, and the only settings left at the machine level are those which do not make sense at the cluster level, and cannot be clustered. Examples are IP addresses, featurekeys, etc.

Leave as many settings at the cluster level as possible. If only one machine in the cluster needs a different setting, copy that cluster setting to the machine level for that machine. Do not move that setting. If you move a setting which has no factory default (e.g. HAT table, SMTPROUTES table, LDAP server profile, etc.), the systems inheriting the cluster settings will have blank tables and will probably not process email.

To have that machine re-inherit the cluster setting, manage the CM settings and delete the machine setting. You will only know if a machine is overriding the cluster setting when you see this display:

Settings are defined:

To inherit settings from a higher level: Delete Settings for this feature at this mode.

You can also Manage Settings.

Settings for this feature are also defined at:

Cluster: xxx

Or this display:

Delete settings from:

Cluster: xxx

Machine: yyyy.domain.com

Q. I have a previously configured standalone machine and I join an existing cluster. What happens to my settings?

A. When a machine joins a cluster, all of that machine's clusterable settings will be inherited from the cluster level. Upon joining a cluster, all locally configured non-network settings will be lost, overwritten with the settings of the cluster and any associated groups. (This includes the user/passphrase table; passphrases and users are shared within a cluster).

Q. I have a clustered machine and I remove it (permanently) from the cluster. What happens to my settings?

A. When a machine is permanently removed from a cluster, its configuration hierarchy is “flattened” such that the machine will continue to work the same as it did when it was part of the cluster. All settings that the machine has been inheriting will be applied to the machine in the standalone setting.

For example, if there is only a cluster-mode Global Unsubscribe table, that Global Unsubscribe table data will be copied to the machine's local configuration when the machine is removed from the cluster.

Q. Are log files aggregated within centrally managed machines?

A. No. Log files are still retained for each individual machines. Cisco Secure Manager Email and Web Gateway can be used to aggregate mail logs from multiple machines for the purposes of tracking and reporting.

Q. How does User Access work?

A. The email gateway share one database for the entire cluster. In particular, there is only admin account (and passphrase) for the entire cluster.

Q. How should I cluster a data center?

A. Ideally, a data center would be a “group” within a cluster, not its own cluster. However, if the data centers do not share much between themselves, you may have better results with separate clusters for each data center.

Q. What happens if systems are offline and they reconnect?

A. Systems attempt to synchronize upon reconnecting to the cluster.

Q. Is the centralized management feature a “peer-to-peer” architecture or a “primary/secondary” architecture?

A. Because every machine has all of the data for all of the machines (including all machine-specific settings that it will never use), the centralized management feature can be considered a peer-to-peer architecture.

Q. How do I set up a box so it is not a peer? I want a “secondary” system.

A. Creating a true “secondary” machine is not possible with this architecture. However, you can disable the HTTP (GUI) and SSH (CLI) access at the machine level. In this manner, a machine without GUI or CLI access only be configured by clusterconfig commands (that is, it can never be a login host). This is similar to having a secondary machine, but the configuration can be defeated by turning on login access again.

Q. Can I create multiple, segmented clusters?

A. Isolated “islands” of clusters are possible; in fact, there may be situations where creating them may be beneficial, for example, for performance reasons.

Q. I would like to reconfigure the IP address and hostname on one of my clustered email gateways. If I do this, will I lose my GUI/CLI session before being able to run the reboot command?

A. Follow these steps:

1. Add the new IP address
2. Move the listener onto the new address
3. Leave the cluster
4. Change the hostname
5. Make sure that oldmachinename does not appear in the clusterconfig connections list when viewed from any machine
6. Make sure that all GUI sessions are logged out
7. Make sure that CCS is not enabled on any interface (check via interfaceconfig or Network > Listeners)
8. Add the machine back into the cluster

Q. Can the Destination Controls function be applied at the cluster level, or is it local machine level only?

A. It may be set at a cluster level; however, the limits are on a per-machine basis. So if you limit to 50 connections, that is the limit set for each machine in the cluster.

Q. What can I do to maximize efficiency and minimize problems when setting up a cluster?

1. Initial Planning
   • Try to configure as many things as possible at the cluster level.
   • Manage by machines only for the exceptions.
   • If you have multiple data centers, for example, use groups to share traits that are neither cluster-wide nor necessarily machine-specific.
   • Use the same name for Interfaces and Listeners on each of the email gateways.
2. Be aware of restricted commands.

3. Pay attention to interdependencies among settings.

   For example, the listenerconfig command (even at the cluster level) depends on interfaces that only exist at a machine level. If the interface does not exist at the machine level on all machines in the cluster, that listener will be disabled.

   Note that deleting an interface would also affect listenerconfig .

4. Pay attention to your settings!

   Remember that previously-configured machines will lose their independent settings upon joining a cluster. If you want to re-apply some of these previously configured settings at the machine level, be sure to take note of all settings before joining the cluster.

   Remember that a “disconnected” machine is still part of the cluster. When it is reconnected, any changes you made while it was offline will be synchronized with the rest of the cluster.

   Remember that if you permanently remove a machine from a cluster, it will retain all of the settings it had as part of that cluster. However, if you change your mind and re-join the cluster, the machine will lose all standalone settings.

   Use the saveconfig command to keep records of settings.