To restart all Policy Server (QNS) services on all VMs, execute the following from the Cluster Manager:

/var/qps/bin/control/restartall.sh

Note	This script only restarts the Policy Server (QNS) services. It does not restart any other services.

Use summaryall.sh or statusall.sh to see details about these services.

To restart all Policy Server (QNS) services on a single CPS VM, execute the following from the Cluster Manager:

/var/qps/bin/control/restartqns.sh <hostname>

where <hostname> is the CPS node name of the VM (qns01, qns02, lb01, pcrfclient01, and so on).

CPS can monitor the state of the services and components of the cluster from the OAM (pcrfclient) VMs. By default, this functionality is disabled.

This functionality can be enabled by setting the cluster_state_monitor option to true in the CPS Deployment Template (Excel spreadsheet).

To update an existing deployment to support this functionality, modify this setting in your CPS Deployment Template and redeploy the csv files as described in the CPS Installation Guide for VMware.

This monitoring system reports the state of the system as an integer value as described in the following table:

The current cluster state is reported in the following file on the OAM (pcrfclient): /var/run/broadhop.cluster_state

The determine_cluster_state command logs output of the cluster state monitoring process into /var/log/broadhop/determine_cluster_state.log.

In addition to the monitoring functionality, CPS can also use the cluster state to regulate the startup of some of the CPS services pending the appropriate state of the cluster.

By default this functionality is disabled. It can be enabled for the entire CPS cluster, or for troubleshooting purposes can be enabled or disabled on a per-VM basis.

Note	Cluster State Monitoring must be enabled for Controlled Startup to function.

By default, the Audit History uses a 1 GB capped collection in MongoDB. The capped collection automatically removes documents when the size restriction threshold is hit. The oldest document is removed as each new document is added. For customers who want more than 1 GB of audit data, contact the assigned Cisco Advanced Services Engineer to get more information.

Configuration in Policy Builder is done in GB increments. It is possible to enter decimals, for example, 9.5 will set the capped collection to 9.5 GB.

When using a capped collection, MongoDB places a restriction on the database and does not allow the deletion of data from the collection. Therefore, the entire collection must be dropped and re-created. This means that the PurgeAuditHistory queries have no impact on capped collections.

As a consequence of the XSS defense changes to the API standard operation, any XML data sent in an AuditRequest must be properly escaped even if inside CDATA tags.

For example, &lt;ExampleRequest&gt;...&lt;/ExampleRequest&gt;

For more information on AuditType, refer to Cisco Policy Suite Unified API 2.3.0 Guide.

By default, Audit History is ON but it can be turned OFF.

• ua.client.submit.audit=true — property used by Policy Builder and set in /etc/broadhop/pb/pb.conf

• Submit Requests to Audit Log — Unified API plug-in configuration in Policy Builder.

The Audit History does not log read requests by default.

• GetRefDataBalance

• GetRefDataServices

• GetSubscriber

• GetSubscriberCount

• QueryAuditHistory

• QueryBalance

• QuerySession

• QueryVoucher

• SearchSubscribers

The Unified API also has a Policy Builder configuration option to log read requests which is set to false by default.

All APIs are automatically logged into the Audit Logging History database, except for QueryAuditHistory and KeepAlive. All Unified API requests have an added Audit element that should be populated to provide proper audit history.

The query is very flexible - it uses regex automatically for the id and dataid, and only one of the following are required: id, dataid, or request. The dataid element typically will be the networkId (Credential) value of a subscriber.

Note	Disable Regex. The use of regular expressions for queries can be turned off in the Policy Builder configuration.

The id element is the person or application who made the API request. For example, if a CSR logs into Control Center and queries a subscriber balance, the id will be that CSR's username.

The dataid element is typically the subscriber's username. For example, if a CSR logs into Control Center and queries a subscriber, the id will be that CSR's username, and the dataid will be the subscriber's credential (networkId value). For queries, the dataid value is checked for spaces and then tokenized and each word is used as a search parameter. For example, “networkId1 networkId2” is interpreted as two values to check.

The fromDate represents the date in the past from which to start the purge or query. If the date is null, the api starts at the oldest entry in the history.

The toDate represents the date in the past to which the purge or query of data includes. If the date is null, the api includes the most recent entry in the purge or query.

By default, the Audit History database is capped at 1 GB. Mongo provides a mechanism to do this and then the oldest data is purged as new data is added to the repository. There is also a PurgeAuditHistory request which can purge data from the repository. It uses the same search parameters as the QueryAuditHistory and therefore is very flexible in how much or how little data is matched for the purge.

Note	Regex Queries! Be very careful when purging records from the Audit History database. If a value is given for dataid, the server uses regex to match on the dataid value and therefore will match many more records than expected. Use the QueryAuditHistory API to test the query.

Each purge request is logged after the purge operation completes. This ensures that if the entire repo is destroyed, the purge action that destroyed the repo will be logged.

The Control Center version 2.0 automatically logs all requests.

This API purges the Audit History.

The query is very flexible - it uses regex automatically for the id and dataid, and only one of the following are required: id, dataid, or request. The dataid element typically will be the networkId (Credential) value of a subscriber.

The id element is the person or application who made the API request. For example, if a CSR logs into Control Center and queries a subscriber balance, the id will be that CSR's username.

The dataid element is typically the subscriber's username. For example, if a CSR logs into Control Center and queries a subscriber, the id will be that CSR's username, and the dataid will be the subscriber's credential (networkId value). For queries, the dataid value is checked for spaces and then tokenized and each word is used as a search parameter. For example, “networkId1 networkId2” is interpreted as two values to check.

The fromDate represents the date in the past from which to start the purge or query. If the date is null, the api starts at the oldest entry in the history.

The toDate represents the date in the past to which the purge or query of data includes. If the date is null, the api includes the most recent entry in the purge or query.

Note	Size-Capped Database If the database is capped by size, then the purge request ignores the request key values and drops the entire database due to restrictions of the database software.

Schema

<PurgeAuditHistoryRequest>
<key> AuditKeyType </key> [1]
</PurgeAuditHistoryRequest>

Example

<se:Envelope xmlns:se="http://schemas.xmlsoap.org/soap/envelope/">
  <se:Body>
    <PurgeAuditHistoryRequest xmlns="http://broadhop.com/unifiedapi/soap/types">
      <key>
        <id>username</id>
        <dataid>subscriber</dataid>
        <request>API Name</request>
        <fromDate>2011-01-01T00:00:00Z</fromDate>
        <toDate>2011-01-01T00:00:00Z</toDate>
      </key>
    </PurgeAuditHistoryRequest>
  </se:Body>
</se:Envelope>

To purge all CreateSubscriberRequest:

<se:Envelope xmlns:se="http://schemas.xmlsoap.org/soap/envelope/">
  <se:Body>
    <PurgeAuditHistoryRequest xmlns="http://broadhop.com/unifiedapi/soap/types">
      <key>
        <request>CreateSubscriberRequest</request>
      </key>
    </PurgeAuditHistoryRequest>
  </se:Body>
</se:Envelope>

To purge all CreateSubscriberRequest by CSR:

<se:Envelope xmlns:se="http://schemas.xmlsoap.org/soap/envelope/">
  <se:Body>
    <PurgeAuditHistoryRequest xmlns="http://broadhop.com/unifiedapi/soap/types">
      <key>
        <id>csrusername</id>
        <request>CreateSubscriberRequest</request>
      </key>
    </PurgeAuditHistoryRequest>
  </se:Body>
</se:Envelope>

To purge all actions by CSR for a given subscriber for a date range:

<se:Envelope xmlns:se="http://schemas.xmlsoap.org/soap/envelope/">
  <se:Body>
    <PurgeAuditHistoryRequest xmlns="http://broadhop.com/unifiedapi/soap/types">
      <key>
        <id>csrusername</id>
        <dataid>subscriber@gmail.com</dataid>
        <fromDate>2010-01-01T00:00:00Z</fromDate>
        <toDate>2012-11-01T00:00:00Z</toDate>
      </key>
    </PurgeAuditHistoryRequest>
  </se:Body>
</se:Envelope>

This API queries the Audit History.

The query is very flexible - it uses regex automatically for the id and dataid, and only one of the following are required: id, dataid, or request. The dataid element typically will be the networkId (Credential) value of a subscriber.

The id element is the person or application who made the API request. For example, if a CSR logs into Control Center and queries a subscriber balance, the id will be that CSR's username.

The dataid element is typically the subscriber's username. For example, if a CSR logs into Control Center and queries a subscriber, the id will be that CSR's username, and the dataid will be the subscriber's credential (networkId value). For queries, the dataid value is checked for spaces and then tokenized and each word is used as a search parameter. For example, "networkId1 networkId2" is interpreted as two values to check.

The fromDate represents the date in the past from which to start the purge or query. If the date is null, the api starts at the oldest entry in the history.

The toDate represents the date in the past to which the purge or query of data includes. If the date is null, the api includes the most recent entry in the purge or query.

Schema:

<QueryAuditHistoryRequest>
<key> AuditKeyType </key> [1]
</QueryAuditHistoryRequest>

Example:

<se:Envelope xmlns:se="http://schemas.xmlsoap.org/soap/envelope/">
  <se:Body>
    <QueryAuditHistoryRequest xmlns="http://broadhop.com/unifiedapi/soap/types">
      <key>
        <id>username</id>
        <dataid>subscriber</dataid>
        <request>API Name</request>
        <fromDate>2011-01-01T00:00:00Z</fromDate>
        <toDate>2011-01-01T00:00:00Z</toDate>
      </key>
    </QueryAuditHistoryRequest>
  </se:Body>
</se:Envelope>

To find all CreateSubscriberRequest:

<se:Envelope xmlns:se="http://schemas.xmlsoap.org/soap/envelope/">
  <se:Body>
    <QueryAuditHistoryRequest xmlns="http://broadhop.com/unifiedapi/soap/types">
      <key>
        <request>CreateSubscriberRequest</request>
      </key>
    </QueryAuditHistoryRequest>
  </se:Body>
</se:Envelope>

To find all CreateSubscriberRequest by CSR:

<se:Envelope xmlns:se="http://schemas.xmlsoap.org/soap/envelope/">
  <se:Body>
    <QueryAuditHistoryRequest xmlns="http://broadhop.com/unifiedapi/soap/types">
      <key>
        <id>csrusername</id>
        <request>CreateSubscriberRequest</request>
      </key>
    </QueryAuditHistoryRequest>
  </se:Body>
</se:Envelope>

To find all actions by CSR for a given subscriber for a date range:

<se:Envelope xmlns:se="http://schemas.xmlsoap.org/soap/envelope/">
  <se:Body>
    <QueryAuditHistoryRequest xmlns="http://broadhop.com/unifiedapi/soap/types">
      <key>
        <id>csrusername</id>
        <dataid>subscriber@gmail.com</dataid>
        <fromDate>2010-01-01T00:00:00Z</fromDate>
        <toDate>2012-11-01T00:00:00Z</toDate>
      </key>
    </QueryAuditHistoryRequest>
  </se:Body>
</se:Envelope>

The Policy Builder automatically logs all save operations (Publish and Save to Client) to the Audit History database and also to a log file.

• Policy Builder Publish submits an entry to the Audit Logging Server (goes to database).

• Policy Builder Save to Client Repository submits an entry to the Audit Logging Server (goes to database).

• Whenever a screen is saved locally (Save button) XML is generated and logged for that user in /var/log/broadhop/qns-pb.log.

Example log in qns-pb.log from Local Save in Policy Builder:

2013-02-06 11:57:01,214 [UIThread [vt75cjqhk7v4noguyc9c7shp]] DEBUG c.b.c.r.BroadhopResourceSetAudit -
Audit: Local file change made by: broadhop. Updated File:
file:/var/broadhop/pb/workspace/tmp-ITC2/checkout/ConfiguredExtensionPoint-43730cd7-b238-4b29-a828-d9b4
47e5a64f-33851.xmi

XML Representation of changed screen:

<?xml version="1.0" encoding="UTF-8"?>
<policy:ConfiguredExtensionPoint xmlns:policy="http://broadhop.com/policy"
id="43730cd7-b238-4b29-a828-d9b447e5a64f-33851">
  <extensionPoint
    href="virtual:URI#_vxG4swK1Ed-M48DL9vicxQ"/>
  <policies 
    href="Policy-default-_sY__4L_REeGCdakzuzzlAg.xmi#_sY__4L_REeGCdakzuzzlAg"/>
</policy:ConfiguredExtensionPoint>

Controlling Local Save output:

In the logback.xml file that controls Policy Builder logging, add com.broadhop.client.resourceset.BroadhopResourceSetAudit as a category and set it to the desired level.

For reporting purposes the following is the database structure in Mongo:

{

"_id" : ObjectId("5097d75be4b0d5f7ab0d90fe"),

"_id_key" : "username",

"comment_key" : "comment",

"data_id_key" : [ "networkId11921" ],

"timestamp_key" : ISODate("2012-11-05T15:12:27.673Z"),

"request_key" : "DeleteQuotaRequest",

"data_key" :

"<DeleteQuotaRequest><audit><id>username</id></audit><networkId><![CDATA
[networkId11921]]></networkId><balanceCode>DATA</balanceCode><code>Recurring</code>
<hardDelete>false</hardDelete></DeleteQuotaRequest>

"}

In the /usr/share/doc/audit-version/ directory, the audit package provides a set of pre-configured rules files.

The Linux Audit system provides a way to track security-relevant information on your system. Based on pre-configured rules, Audit generates log entries to record as much information about the events that are happening on your system as possible.

In the /usr/share/doc/audit-version/ directory, the audit package provides a set of pre-configured rules files.

To use these pre-configured rule files, create a backup of your original /etc/audit/audit.rules file and copy the configuration file of your choice over the /etc/audit/audit.rules file:

cp /etc/audit/audit.rules /etc/audit/audit.rules_backup
cp /usr/share/doc/audit-version/stig.rules /etc/audit/audit.rules

For more information on auditd process, refer to the link.

The policy tracing and execution analyzer is 3-tier architecture:

• Tier 1 — command line utilities to manage the policy trace generation and extract policy traces.

• Tier 2 — policy server creation of policy traces using triggers defined in Tier 1.

• Tier 3 — storage of the policy traces in a MongoDB.

All commands are located on the Control Center virtual machine within /var/qps/bin/control directory. There are two main scripts which can be used for tracing: trace_ids.sh and trace.sh.

• The trace_ids.sh script maintains all rules for activating and deactivating traces within the system.

• The trace.sh script allows for the real time or historical retrieval of traces.

Before running trace_ids.sh and trace.sh, confirm which database you are using for traces. For more information, refer to Policy Trace Database. If no database has been configured, then by default the scripts connects to primary database member of SPR-SET1.

The default location of the policy trace database is the administrative database and can be optionally specified in the trace database fields. These fields are defined at the cluster level in the system configurations.

Note	Make sure to run all trace utility scripts from /var/qps/bin/control directory only.

Cisco Policy Suite (CPS) is built around a distributed system that runs on a large number of virtualized nodes. Previous versions of the CPS software allowed operators to add custom accounts to each of these virtual machines (VM), but management of these disparate systems introduced a large amount of administrative overhead.

CPS has been designed to leverage the Terminal Access Controller Access Control System Plus (TACACS+) to facilitate centralized management of users. Leveraging TACACS+, the system is able to provide system-wide authentication, authorization, and accounting (AAA) for the CPS system.

Further the system allows users to gain different entitlements based on user role. These can be centrally managed based on the attribute-value pairs (AVP) returned on TACACS+ authorization queries.

To provide sufficient information for the Linux-based operating system running on the VM nodes, there are several attribute-value pairs (AVP) that must be associated with the user on the ACS server used by the deployment. User records on Unix-like systems need to have a valid “passwd” record for the system to operate correctly. Several of these fields can be inferred during the time of user authentication, but the remaining fields must be provided by the ACS server.

A standard “passwd” entry on a Unix-like system takes the following form:

<username>:<password>:<uid>:<gid>:<gecos>:<home>:<shell>

When authenticating the user via TACACS+, the software can assume values for the username, password, and gecos fields, but the others must be provided by the ACS server. To facilitate this need, the system depends on the ACS server provided these AVP when responding to a TACACS+ Authorization query for a given username:

• uid

  A unique integer value greater than or equal to 501 that serves as the numeric user identifier for the TACACS+ authenticated user on the VM nodes. It is outside the scope of the CPS software to ensure uniqueness of these values.

• gid

  The group identifier of the TACACS+ authenticated user on the VM nodes. This value should reflect the role assigned to a given user, based on the following values:

  • gid=501 (qns-su)

    This group identifier should be used for users that are entitled to attain super-user (or 'root') access on the CPS VM nodes.

  • gid=504 (qns-admin)

    This group identifier should be used for users that are entitled to perform administrative maintenance on the CPS VM nodes.

    Note	For stopping/starting the Policy Servrer (QNS) process on node, the qns-admin user should use monit:

    For example,

    sudo monit stop qns-1
    sudo monit start qns-1

  • gid=505 (qns-ro)

    This group identifier should be used for users that are entitled to read-only access to the CPS VM nodes.

• home

  The user's home directory on the CPS VM nodes. To enable simpler management of these systems, the users should be configured with a pre-deployed shared home directory based on the role they are assigned with the gid.

  • home=/home/qns-su should be used for users in the qns-su group (gid=501)

  • home=/home/qns-admin should be used for users in the qnsadmin group (gid=504)

  • home=/home/qns-ro should be used for users in the qns-ro group (gid=505)

• shell

  The system-level login shell of the user. This can be any of the installed shells on the CPS VM nodes, which can be determined by reviewing the contents of /etc/shells on one of the CPS VM nodes. Typically, this set of shells is available in a CPS deployment:

  • /bin/sh

  • /bin/bash

  • /sbin/nologin

  • /bin/dash

  • /usr/bin/sudosh

  The /usr/bin/sudosh shell can be used to audit user's activity on the system.

The user environment of the Linux-based VMs needs to be able to lookup a user's passwd entry via different columns in that record at different times. The TACACS+ NSS module provided as part of the CPS solution however is only able to query the Access Control Server (ACS) for this data using the username. For this reason the system relies upon the Name Service Cache Daemon (NSCD) to provide this facility locally after a user has been authorized to use a service of the ACS server.

More details on the operations of NSCD can be found by referring to online help for the software (nscd --help) or in its man page (nscd(8)). Within the CPS solution it provides a capability for the system to lookup a user's passwd entry via their uid as well as by their username.

To avoid cache coherence issues with the data provided by the ACS server the NSCD package has a mechanism for expiring cached information.

The default NSCD package configuration on the CPS VM nodes has the following characteristics:

• Valid responses from the ACS server are cached for 600 seconds (10 minutes)

• Invalid responses from the ACS server (user unknown) are cached for 20 seconds

• Cached valid responses are reloaded from the ACS server 5 times before the entry is completely removed from the running set -- approximately 3000 seconds (50 minutes)

• The cache are persisted locally so it survives restart of the NSCD process or the server

It is possible for an operator to explicitly expire the cache from the command line. To do so the administrator need to get the shell access to the target VM and execute the following command as a root user:

# nscd -i passwd

The above command will invalidate all entries in the passwd cache and force the VM to consult with the ACS server for future queries.

There may be some unexpected behaviors of the user environment for TACACS+ authenticated users connected to the system when their cache entries are removed from NSCD. This can be corrected by the user by logging out of the system and logging back into it or by issuing the following command, which forces the system to query the ACS server:

# id -a “$USER”

• All the VMs were created using the deployment process.

• This procedure assumes the datastore that will be used to have the virtual disk has sufficient space to add the virtual disk.

• This procedure assumes the datastore has been mounted to the VMware ESX server, regardless of the backend NAS device (SAN or iSCSI, etc).