Introduction
The Identity Services Engine (ISE) Version 1.3 supports a new API called pxGrid. This modern and flexible protocol that supports authentication, encryption, and privileges (groups) allows for easy integration with other security solutions. This document describes the usage of pxLog application which has been written as a proof of concept. pxLog is able to receive syslog messages from Intrusion Prevention System (IPS) and send pxGrid messages to the ISE in order to quarantine the attacker. As a result, ISE uses RADIUS Change of Authorization (CoA) in order to change the authorization status of the endpoint that limits the network access. All of this happens transparently to the end user.
For this example, Snort has been used as the IPS, but any other solution could be used. Actually it does not have to be an IPS. All that is required is to send the syslog message to pxLog with the IP address of the attacker. This creates a possibility for the integration of a large number of solutions.
This document also presents how to troubleshoot and test pxGrid solutions, with the typical problems and limitations.
Disclaimer: The pxLog application is not supported by Cisco. This article has been written as a proof of concept. The primary purpose was to use it during the betatesting of pxGrid implementation on the ISE.
Prerequisites
Requirements
Cisco recommends that you have experience with Cisco ISE configuration and basic knowledge of these topics:
- ISE deployments and authorization configuration
- CLI configuration of Cisco Catalyst Switches
Components Used
The information in this document is based on these software and hardware versions:
- Microsoft Windows 7
- Cisco Catalyst 3750X Series Switch Software, Versions 15.0 and Later
- Cisco ISE Software, Versions 1.3 and Later
- Cisco AnyConnect Mobile Security with Network Access Manager (NAM), Version 3.1 and Later
- Snort Version 2.9.6 with Data Acquisition (DAQ)
- pxLog Application Installed on Tomcat 7 with MySQL Version 5
Network Diagram and Traffic Flow

Here is the traffic flow, as illustrated in the network diagram:
- A Microsoft Windows 7 user connects to the switch and performs 802.1x authentication.
- The switch uses the ISE as the Authentication, Authorization, and Accounting (AAA) server. The Dot1x Full Access authorization rule is matched and full network access is granted (DACL: PERMIT_ALL).
- The user tries to connect with the trusted network and violates the Snort rule.
- As a result, Snort sends an alert to the pxLog application (via syslog).
- The pxLog application performs verification against its local database. It is configured in order to catch syslog messages sent by Snort and extract the IP address of the attacker. Then it uses pxGrid to send a request towards the ISE in order to quarantine the attacker IP address (the ISE is a pxGrid controller).
- The ISE re-evaluates its authorization policy. Because the endpoint is quarantined, the Session:EPSStatus EQUALS Quarantine condition is met and a different authorization profile is matched (Dot1x Quarantine). The ISE sends a CoA Terminate to the switch in order to terminate the session. This triggers the re-authentication and a new Downloadable ACL (DACL) (PERMIT_ICMP) is applied, which provides the limited network access to the end user.
- At this stage, the administrator might decide to unquarantine the endpoint. This can be achieved via the GUI of pxLog. Again, the pxGrid message towards the ISE is sent.
- The ISE performs a similar operation as in Step 6. This time, the endpoint is no longer quarantined and full access is provided.
pxLog
Architecture

The solution is to install a set of applications on a Linux machine:
- The pxLog application written in Java and deployed on the Tomcat server. That application consists of:
- Servlet that processes web requests - This is used in order to access the administrative panel via the web browser.
- Enforcer module - Thread that is started together with servlet. The Enforcer reads syslog messages from the file (optimized), processes those messages as per the configured rules, and executes actions (like quarantine via pxGrid).
- The MySQL database that contains the configuration for pxLog (rules and logs).
- The syslog server that receives syslog messages from external systems and writes them to a file.
Installation
The pxLog application uses these libraries:
- jQuery (for AJAX support)
- JavaServer Pages Standard Tag Library (JSTL) (Model View Controller (MVC) model, data is separated from logic: JavaServer Page (JSP) code is used to render only, no HTML code in Java classes)
- Log4j as a logging subsystem
- MySQL connector
- displaytag for rendering/sorting tables
- pxGrid API by Cisco (currently Version alpha 147)
All of those libraries are already in the lib directory of the project so there is no need to download any more Java ARchive (JAR) files.
In order to install the application:
- Unpack the whole directory to the Tomcat Webapp directory.
- Edit the WEB-INF/web.xml file. The only required change is the serverip variable, which should point to the ISE. Also the Java Certificate KeyStores (one for trusted and one for identity) might be generated (instead of the default). This is used by the pxGrid API that uses the Secure Sockets Layer (SSL) session with both the client and server certificates. Both sides of the communication need to present with the certificate and need to trust each other. Refer to the pxGrid Protocol Requirements section for more information.
- Make sure the ISE hostname is resolved correctly on pxLog (refer to the record in the Domain Name Server (DNS) or /etc/hosts entry). Refer to the pxGrid Protocol Requirements section for more information.
- Configure the MySQL database with the mysql/init.sql script. Credentials can be changed but should be reflected in the WEB-INF/web.xml file.
Snort
This article does not focus on any specific IPS, which is why only a brief explanation is provided.
Snort is configured as inline with DAQ support. Traffic is redirected with iptables:
iptables -I FORWARD -j ACCEPT
iptables -I FORWARD -j NFQUEUE --queue-num 1
Then, after inspection, it is injected and forwarded as per default iptable rules.
A few custom Snort rules have been configured (the /etc/snort/rules/test.rules file is included in the global configuration).
alert icmp any any -> any any (itype:8; dsize:666<>686; sid:100122)
alert icmp any any -> any any (itype:8; ttl: 6; sid:100124)
Snort sends a syslog message when the Time To Live (TTL) of the packet is equal to 6 or the size of the payload is between 666 and 686. Traffic is not blocked by Snort.
Also thresholds should be set up to make sure the alerts are not triggered too often (/etc/snort/threshold.conf):
event_filter gen_id 1, sig_id 100122, type limit, track by_src, count 1, seconds 60
event_filter gen_id 1, sig_id 100124, type limit, track by_src, count 1, seconds 60
Then the syslog server points to the pxLog machine (/etc/snort/snort.conf):
output alert_syslog: host=10.222.0.61:514, LOG_AUTH LOG_ALER
For some versions of Snort, there are bugs related to the syslog configuration, and then the default settings could be used that point to the localhost and syslog-ng could be configured in order to forward specific messages to the pxLog host.
ISE
Configuration
Persona and Certificate
- Enable the pxGrid role, which is disabled on the ISE by default, under Administration > Deployment:

- Verify if the certificates are used for pxGrid under Administration > Certificates > System Certificates:

Endpoint Protection Service (EPS)
EPS should be enabled (disabled by default) from Administration > Settings:

This allows you to use the quarantine/unquarantine functionality.
Authorization Rules

The first rule is encountered only when the endpoint is quarantined. Then limited access is dynamically enforced by the RADIUS CoA. The switch also must be added to Network Devices with the correct shared secret.
Troubleshoot
The pxGrid status can be verified with the CLI:
lise/admin# show application status ise
ISE PROCESS NAME STATE PROCESS ID
--------------------------------------------------------------------
Database Listener running 6717
Database Server running 51 PROCESSES
Application Server running 9486
Profiler Database running 7804
AD Connector running 10058
M&T Session Database running 7718
M&T Log Collector running 9752
M&T Log Processor running 9712
Certificate Authority Service running 9663
pxGrid Infrastructure Service running 14979
pxGrid Publisher Subscriber Service running 15281
pxGrid Connection Manager running 15248
pxGrid Controller running 15089
Identity Mapping Service running 9962
There are also separate debugs for pxGrid (Administration > Logging > Debug Log Configuration > pxGrid). Debug files are stored in the pxGrid directory. The most important data is in the pxgrid/pxgrid-jabberd.log and the pxgrid/pxgrid-controller.log.
Test
Step1. Registration for pxGrid
The pxLog application is automatically deployed when Tomcat starts.
- In order to use pxGrid, register two users in the ISE (one with session access, and one with quarantine). This can be completed from Pxgrid Operations > Register users:

Registration starts automatically:

- At this stage, it is necessary to approve registered users on the ISE (auto approval is disabled by default):

After the approval, pxLog automatically notifies the administrator (via an AJAX call):

ISE shows the status for those two users as Online or Offline (not Pending anymore).
Step2. pxLog Rules Configuration
pxLog must process syslog messages and execute actions based on it. In order to add a new rule, select Manage Rules:

Now the enforcer module looks for this Regular Expression (RegExp) in the syslog message: "snort[". If found, it searches all IP addresses and selects the one before the last one. This matches most security solutions. Refer to the Syslog section for more information. That IP address (attacker) is quarantined via pxGrid. Also a more granular rule might be used (for example, it might include the signature number).
Step3. First Dot1x Session
The Microsoft Windows 7 station initiates a wired dot1x session. Cisco Anyconnect NAM has been used as a supplicant. The Extensible Authentication Protocol-Protected EAP (EAP-PEAP) method is configured.

The ISE Dot1x Full Access authorization profile is selected. The switch downloads the access list in order to grant full access:
3750#show authentication sessions interface g0/17
Interface: GigabitEthernet0/17
MAC Address: 0050.b611.ed31
IP Address: 10.221.0.240
User-Name: cisco
Status: Authz Success
Domain: DATA
Security Policy: Should Secure
Security Status: Unsecure
Oper host mode: single-host
Oper control dir: both
Authorized By: Authentication Server
Vlan Policy: N/A
ACS ACL: xACSACLx-IP-PERMIT_ALL-53fc9dbe
Session timeout: N/A
Idle timeout: N/A
Common Session ID: 0A01000C000037E6BAB267CF
Acct Session ID: 0x00003A70
Handle: 0xA100080E
Runnable methods list:
Method State
dot1x Authc Success
3750#show ip access-lists interface g0/17
permit ip any any
Step4. Microsoft Windows PC Sends the Packet that Triggers the Alarm
This shows what happens if you do send from a Microsoft Windows packet with TTL = 7:
c:\> ping 10.222.0.61 -i 7 -n 1
That value is decremented on Snort in the Forwarding chain and an alarm is raised. As a result, a syslog message towards pxLog is sent:
Sep 6 22:10:31 snort snort[6310]: [1:100124:0] ALERT {ICMP} 10.221.0.240 ->
10.222.0.61
Step5. pxLog
The pxLog receives the syslog message, processes it, and requests to quarantine that IP address. This can be confirmed if you check the logs:

Step6. ISE Quarantine
The ISE reports that the IP address has been quarantined:

As a result, it reviews the authorization policy, chooses quarantine, and sends RADIUS CoA in order to update the authorization status on the switch for that specific endpoint.

That is the CoA terminate message that forces the supplicant to initiate a new session and get limited access (Permit_ICMP):

The result can be confirmed on the switch (limited access for the endpoint):
3750#show authentication sessions interface g0/17
Interface: GigabitEthernet0/17
MAC Address: 0050.b611.ed31
IP Address: 10.221.0.240
User-Name: cisco
Status: Authz Success
Domain: DATA
Security Policy: Should Secure
Security Status: Unsecure
Oper host mode: single-host
Oper control dir: both
Authorized By: Authentication Server
Vlan Policy: N/A
ACS ACL: xACSACLx-IP-PERMIT_ICMP-53fc9dc5
Session timeout: N/A
Idle timeout: N/A
Common Session ID: 0A01000C000037E7BAB7D68C
Acct Session ID: 0x00003A71
Handle: 0xE000080F
Runnable methods list:
Method State
dot1x Authc Success
3750#show ip access-lists interface g0/17
permit icmp any any
Step7. pxLog Unquarantine
At this stage, the administrator decides to unquarantine that endpoint:

The same operation can be executed directly from the ISE:

Step8. ISE Unquarantine
The ISE again reviews the rules and updates the authorization status on the switch (full network access is granted):

The report confirms:

pxLog Functionality
The pxLog application has been written in order to demonstrate the functionality of the pxGrid API. It allows you to:
- Register session and EPS users on the ISE
- Download information about all sessions active on the ISE
- Download information about a specific active session on the ISE (by IP address)
- Download information about a specific active user on the ISE (by username)
- Display the information about all profiles (profiler)
- Display the information about the TrustSec Security Group Tags (SGTs) defined on the ISE
- Check version (capabilities of pxGrid)
- Quarantine based on the IP or MAC address
- Unquarantine based on IP or MAC address
More functionality is planned in the future.
Here are some example screenshots from pxLog:





pxGrid Protocol Requirements
Groups
The client (user) can be a member of one group at a time. The two most commonly used groups are:
- Session - Used in order to browse/download information about sessions/profiles/SGTs
- EPS - Used in order to execute quarantine
Certificates and Java KeyStore
As mentioned previously, both client applications, pxLog and pxGrid controller (ISE), must have certificates configured in order to communicate. The pxLog application keeps those in the Java KeyStore files:
- store/client.jks - Includes the client and the Certificate Authority (CA) certificates
- store/root.jks - Includes the ISE chain: Monitoring and Troubleshooting Node (MnT) identity and the CA certificate
Files are protected by password (default: cisco123). File location and passwords can be changed in WEB-INF/web.xml.
Here are the steps to generate a new Java KeyStore:
- In order to create a root (trusted) keystore, import the CA certificate (cert-ca.der should be in DER format):
pxgrid store # keytool -import -alias ca -keystore root.jks -file cert-ca.der
- When you create a new keystore, choose a password, which is used later in order to access the keystore.
- Import the MnT identity certificate to the root keystore (cert-mnt.der is the identity certificate taken from ISE and should be in DER format):
pxgrid store # keytool -import -alias mnt -keystore root.jks -file cert-mnt.der
- In order to create the client keystore, import the CA certificate:
pxgrid store # keytool -import -alias ca -keystore client.jks -file cert-ca.der
- Create a private key in the client keystore:
pxgrid store # keytool -genkey -alias clientcert -keyalg RSA -keystore client.jks -
keysize 2048
- Generate a Certificate Signing Request (CSR) in the client keystore:
pxgrid store # keytool -certreq -alias clientcert -keystore client.jks -
file cert-client.csr
- Sign the cert-client.csr and import the signed client certificate:
pxgrid store # keytool -import -alias clientcert -keystore client.jks -file cert-
client.der
- Verify that both keystores contain the correct certificates:
pxgrid store # keytool -list -v -keystore client.jks
pxgrid store # keytool -list -v -keystore root.jks
Caution: When the ISE 1.3 node is upgraded, there is an option to keep the identity certificate, but CA signing is removed. As a result, the upgraded ISE uses a new certificate but never attaches the CA certificate in the SSL/ServerHello message. This triggers the failure on the client that expects (as per RFC) to see a full chain.
Hostname
The pxGrid API for several functions (like session download) performs additional validation. The client contacts the ISE and receives the ISE hostname, which is defined by the hostname command in the CLI. Then, the client tries to perform DNS resolution for that hostname and tries to contact and fetch data from that IP address. If the DNS resolution for the ISE hostname fails, the client does not try to get any data.
Caution: Notice that only the hostname is used for this resolution, which is lise in this scenario, not the Fully Qualified Domain Name (FQDN), which is lise.example.com in this scenario.
Note for Developers
Cisco publishes and supports the pxGrid API. There is one package named like this:
pxgrid-sdk-1.0.0-167
Inside there are:
- pxGrid JAR files with classes, which can be easily decoded to Java files to check the code
- Sample Java KeyStores with certificates
- Sample scripts that use sample Java classess that use pxGrid
Syslog
Here is the list of security solutions that send syslog messages with the attacker IP address. These can be easily integrated with pxLog as long as you use the correct RegExp rule in the configuration.
Snort
Snort sends syslog alerts in this format:
host[id] [sig_gen, sig_id, sig_sub] [action] [msg] [proto] [src] [dst]
Here is an example:
snort[6310]: [1:100124:0] ALERT {ICMP} 10.221.0.240 -> 10.222.0.61
The attacker IP address is always the second before the last one (destination). It is simple to build a granular RegExp for a specific signature and extract the attacker IP address. Here is an example RegExp for the signature 100124 and message Internet Control Message Protocol (ICMP):
snort[\.*:100124:.*ICMP.*
Cisco Adaptive Security Appliance (ASA) Inspection
When the ASA is configured for HTTP (example) inspection, the corresponding syslog message looks like this:
Mar 12 2014 14:36:20: %ASA-5-415006: HTTP - matched Class 23:
MS13-025_class in policy-map MS_Mar_2013_policy, URI matched -
Dropping connection from inside:192.168.60.88/2135 to
outside:192.0.2.63/80
Again a granular RegExp could be used in order to filter those messages and extract the attacker IP address, the second before the last one.
Cisco Sourcefire Next Generation Intrusion Prevention Systems (NGIPS)
Here is an example message sent by the Sourcefire sensor:
Jan 28 19:46:19 IDS01 SFIMS: [CA IDS][Policy1][119:15:1] http_inspect: OVERSIZE
REQUEST-URI DIRECTORY [Classification: Potentially Bad Traffic] [Priority: 2]
{TCP} 10.12.253.47:55504 -> 10.15.224.60:80
So again, it is simple to extract the attacker IP address because the same logic applies. Also the policy name and the signature is provided, so the pxLog rule can be granular.
Juniper NetScreen
Here is an example message sent by the older Juniper Intrusion Detection & Prevention (IDP):
dayId="20061012" recordId="0" timeRecv="2006/10/12
21:52:21" timeGen="2006/10/12 21:52:21" domain="" devDomVer2="0"
device_ip="10.209.83.4" cat="Predefined" attack="TROJAN:SUBSEVEN:SCAN"
srcZn="NULL" srcIntf="NULL" srcAddr="192.168.170.20" srcPort="63396"
natSrcAddr="NULL" natSrcPort="0" dstZn="NULL" dstIntf="NULL"
dstAddr="192.168.170.10" dstPort="27374" natDstAddr="NULL" natDstPort="0"
protocol="TCP" ruleDomain="" ruleVer="5" policy="Policy2" rulebase="IDS"
ruleNo="4" action="NONE" severity="LOW" alert="no" elaspedTime="0" inbytes="0"
outbytes="0" totBytes="0" inPak="0" outPak="0" totPak="0" repCount="0"
packetData="no" varEnum="31" misc="<017>'interface=eth2" user="NULL"
app="NULL" uri="NULL"
The IP address of the attacker can be extracted in the same way.
Juniper JunOS
JunOS is similar:
Jul 16 10:09:39 JuniperJunOS: asp[8265]:
ASP_IDS_TCP_SYN_ATTACK: asp 3: proto 6 (TCP),
ge-0/0/1.0 10.60.0.123:2280 -> 192.168.1.12:80, TCP
SYN flood attack
Linux iptables
Here are some example Linux iptables.
Jun 15 23:37:33 netfilter kernel: Inbound IN=lo OUT=
MAC=00:13:d3:38:b6:e4:00:01:5c:22:9b:c2:08:00 src=10.0.0.1 DST=10.0.0.100 LEN=60
TOS=0x10 PREC=0x00 TTL=64 ID=47312 DF PROTO=TCP SPT=40945 DPT=3003 WINDOW=32767
RES=0x00 SYN URGP=0
You can send syslog information for any type of packet with the advanced functionality provided by the iptable modules like connection tracking, xtables, rpfilters, pattern matching, and so on.
FreeBSD IPFirewall (IPFW)
Here is an example message for IPFW blocking fragments:
Sep 7 15:03:14 delta ipfw: 11400 Deny UDP 10.61.216.50 10.81.199.2 in via fxp0
(frag 52639:519@1480)
VPN Readiness and CoA Handling
The ISE is able to recognize the type of sessions in terms of the CoA handling.
- For a wired 802.1x/MAC Authentication Bypass (MAB), the ISE sends the CoA reauthenticate, which triggers a second authentication.
- For a wireless 802.1x/MAB, the ISE sends the CoA terminate, which triggers a second authentication.
- For an ASA VPN, the ISE sends a CoA with a new DACL attached (no second authentication).
The EPS module is simple. When it executes a quarantine, it always sends a CoA terminate packet. For wired/wireless sessions, it is not a problem (all 802.1x supplicants are able to transparently initiate a second EAP session). But when the ASA receives the CoA terminate, it drops the VPN session and the end user is presented with this:

There are two possible solutions to force the AnyConnect VPN to automatically reconnect (configured in the XML profile):
- Autoreconnect, which works only when you lose the connection with the VPN gateway, not for administrative termination
- Always-on, which works and forces AnyConnect to automatically re-establish the session
Even when the new session is established, the ASA chooses the new audit-session-id. From the ISE point-of-view, this is a new session and there is no chance to encounter the quarantine rule. Also for the VPNs, it is not possible to use the MAC address of the endpoint as the identity, as opposed to wired/wireless dot1x.
The solution is to force the EPS to behave like the ISE and send the correct type of CoA based on the session. This functionality will be introduced in ISE Version 1.3.1.
pxGrid Partners and Solutions
Here is a list of pxGrid partners and solutions:
- LogRhythm (Security Information and Event Management (SIEM)) - Supports the Representational State Transfer (REST) API
- Splunk (SIEM) - Supports the REST API
- HP Arcsight (SIEM) - Supports the REST API
- Sentinel NetIQ (SIEM) - Plans to support pxGrid
- Lancope StealthWatch (SIEM) - Plans to support pxGrid
- Cisco Sourcefire - Plans to support pxGrid 1HCY15
- Cisco Web Security Appliance (WSA) - Plans to support pxGrid in April 2014
Here are other partners and solutions:
- Tenable (vulnerability assessment)
- Emulex (packet capture and forensics)
- Bayshore Networks (Data Loss Prevention (DLP) and Internet of Things (IoT) policy)
- Ping Identity (Identity and Access Management (IAM)/Single Sign On (SSO))
- Qradar (SIEM)
- LogLogic (SIEM)
- Symantec (SIEM amd Mobile Device Management (MDM))
Refer to the Marketplace Solutions Catalog for the full list of security solutions.
ISE APIs: REST vs EREST vs pxGrid
There are three types of API available on ISE Version 1.3.
Here is a comparison:
|
REST |
External RESTful |
pxGrid |
Client authentication |
username + password (basic HTTP auth) |
username + password (basic HTTP auth) |
certificate |
Privilege separation |
no |
limited (ERS Admin) |
yes (Groups) |
Accessing |
MnT |
MnT |
MnT |
Transport |
tcp/443 (HTTPS) |
tcp/9060 (HTTPS) |
tcp/5222 (XMPP) |
HTTP method |
GET |
GET/POST/PUT |
GET/POST |
Enabled by default |
yes |
no |
no |
Number of operations |
few |
many |
few |
CoA Terminate |
supported |
no |
supported |
CoA Reauthenticate |
supported |
no |
supported * |
User operations |
no |
yes |
no |
Endpoint operations |
no |
yes |
no |
Endpoint Identity group operations |
no |
yes |
no |
Quarantine (IP, MAC) |
no |
no |
yes |
UnQuarantine (IP, MAC) |
no |
no |
yes |
PortBounce/Shutdown |
no |
no |
yes |
Guest user operations |
no |
yes |
no |
Guest portal operations |
no |
yes |
no |
Network device operations |
no |
yes |
no |
Network device group operations |
no |
yes |
no |
* Quarantine uses unified CoA support from ISE Version 1.3.1.
Downloads
pxLog can be downloaded from Sourceforge .
The Software Development Kit (SDK) is already included. For the latest SDK and API documentation for pxGrid, contact your Partner or the Cisco Account team.
Related Information