Cisco Application Centric Infrastructure with Splunk Enterprise Solution CVD

Contents

Audience. 4

Introduction. 4

Features and Benefits of Cisco ACI for Splunk Enterprise. 4

Business Value. 5

About This Cisco Validated Design. 6

Architecture Overview.. 6

Cisco Application Centric Infrastructure. 6

Cisco Application Policy Infrastructure Controller 7

Cisco ACI Features. 8

Cisco APIC Appliance Features. 11

Cisco Leaf Switch Connection Features. 11

Cisco Spine Switch Connection Features. 12

Splunk Enterprise. 12

Solution Overview.. 13

Solution Details. 13

Installing Splunk Enterprise 6.4.4. 14

Starting Splunk Web Server Setup. 15

Installing Your Splunk License. 16

Installing Cisco ACI App for Splunk Enterprise. 17

Installing Cisco ACI Add-on for Splunk Enterprise. 20

Cisco ACI App for Splunk Enterprise Operation. 22

General Use. 22

Navigation. 22

Within a Dashboard. 23

Visualization Behaviour 23

Time Picker 23

APIC Host 23

Additional Filters. 23

Home Dashboard. 23

APICs Table. 24

Fabric Health: History Chart 24

Home Dashboard Single-Value Visualizations. 24

Help Desk Dashboards. 24

Help Desk: System Faults. 25

Help Desk: System Faults Dashboard Single-Value Visualizations. 25

Faults by Node. 26

Faults by Tenant 26

Faults by Severity. 27

Faults by Domain. 27

Faults by Severity over Time. 27

Faults by Type. 27

Top Faults by Rule. 28

Top Faults by Cause. 28

Latest Affected Objects. 28

Help Desk: Atomic Counters. 29

Help Desk: Path Degradation. 29

Help Desk: System Threshold. 30

Fabric Dashboards. 31

Fabric: Fabric Details. 31

Top Affected Leafs. 32

Top Affected Spines. 32

Health/Fault Details: Leafs. 32

Health/Fault Details: Spines. 33

TCAM Percentage Threshold Statistics. 33

Top TCAM Usage by Node. 33

Leafs – Port Utilization and Thresholds. 33

Spines – Port Utilization and Thresholds. 33

Change Threshold (for Leaf and Spine Utilization) 33

Fabric: Authentication. 34

Authentication Dashboard Single-Value Visualizations. 34

Authentication by Admin. 35

Authentication Failed by User 35

Authentication Success by User 35

Fabric: Multi Pod. 36

APICs. 36

Fabric Health – History. 36

Leafs. 36

Spines. 37

Critical Faults. 38

EPGs. 38

Tenants Dashboards. 39

Tenants: Tenant Details. 40

Top 10 Affected Tenants’ Health. 40

Top 10 Affected Tenants’ Faults. 41

Tenants: Tenant Utilization. 41

<tenant>-Ingress and <tenant>-Egress Utilization Statistics in Bytes. 42

Tenants: Microsegmentation. 42

No. of EPGs Microsegmented per Tenant 42

Network-Based Attributes. 43

VM-Based Attributes. 43

VM Manager Dashboards. 43

VM Manager: VMware. 44

Search. 44

Setup Guide. 44

Creating Custom Dashboards. 45

Data Indexed by Cisco ACI App for Splunk Enterprise. 45

cisco:apic:stats. 45

cisco:apic:class. 45

cisco:apic:health. 46

cisco:apic:authentication. 46

apicsyslog. 46

Building a Custom Dashboard. 46

Custom Dashboard: Single-Value Visualization. 46

Custom Dashboard: Column Chart Visualization. 47

Custom Dashboard: Table Visualization. 49

Accessing Your Custom Dashboard. 51

Tuning the Cisco ACI App. 52

Conclusion. 52

Appendix. 53

Solution Design and Specifications. 53

Cabinet Configuration. 55

Detailed Connection Diagram... 56

Bill of Materials. 56

Audience

The intended audience for this document includes sales engineers, field consultants, professional services developers, IT managers, partner engineers, and customers who want to combine the benefits of Splunk Enterprise with the Cisco^® Application Centric Infrastructure (Cisco ACI^™) solution.

Introduction

Managing and monitoring IT infrastructure is more complex and difficult than ever before. The rapid rate of change and nearly endless streams of data create new challenges. Today, when problems arise, gaining visibility across your entire infrastructure and finding the root cause quickly is almost impossible. Virtualized and cloud-based infrastructures also add to the support and management challenges.

Cisco Application Centric Infrastructure and Splunk provide the solution. Splunk Enterprise is the market leader in the collection and indexing of machine data from physical, virtual, and cloud-based environments. Splunk in combination with the Cisco ACI solution gives you exceptional access to network and application insights. With built-in dashboards that you can customize to see meaningful data at-a-glance and the capability to see a myriad commonly used metrics and application details, the Cisco ACI App for Splunk Enterprise offers you a robust tool for administering your entire Cisco ACI environment.

The Cisco Validated Design for Cisco ACI with Splunk Enterprise describes the deployment of Cisco ACI in a single-pod environment and how to set upSplunk. It demonstrates how to install the Cisco ACI Add-on for Splunk Enterprise and describes the main features and customization capabilities when running Cisco ACI.

Features and Benefits of Cisco ACI for Splunk Enterprise

Cisco ACI for Splunk Enterprise offers these main features and benefits:

●   Reduced resolution time with accelerated root-cause analysis

◦    Centrally view the operational health of your entire Cisco ACI environment and underlying entities, including Cisco Application Policy Infrastructure Controller (APIC) devices, fabric, tenants, and applications.

◦    In multitenant environments, accelerate root-cause investigation and quickly navigate to the source of application problems using flexible per-role visibility into Cisco ACI performance.

●   Central proactive monitoring of Cisco ACI

◦    Get real-time proactive notification of any Cisco ACI faults including the location and affected objects, physical components, logical and virtual components, fabrics, tenants, applications, virtual machines, leaf nodes, and ports.

●   Operation analytics

◦    Optimize your network capacity and prevent service deterioration with detailed visibility into fabric-path degradation.

◦    Meet compliance and security requirements with user analytics, including authentication tracking reports.

◦    Correlate data from Cisco ACI with data from storage resources, operating systems, applications, and virtual and physical infrastructure for visibility across your entire enterprise.

●   Cisco ACI health and user reports

◦    Gain visibility into Cisco ACI health and key performance indicators (KPIs) with dashboards that include:

◦    At-a-glance view of all APIC devices with their uptime, history of overall fabric health scores over five days, summary of physical inventory including spine and leaf elements, and summary of logical and virtual inventory including tenants, applications, and virtual machines.

◦    Help desk dashboard with context-specific faults grouped by acknowledgment status, time, severity, type, rule, cause, and affected objects.

◦    Tenant dashboard with reports highlighting tenant health scores, affected tenants, and application and endpoint group (EPG) health score details with visibility into the endpoint with which degradation occurred.

◦    Innovative Cisco ACI fabric architecture that offers flexible multipath capabilities including network telemetry with atomic counters to avoid network outages; view fabric path degradation with insight into actual packet loss across any path, without the need to deploy network sniffers to understand the optimal fabric trajectory.

◦    Authentication tracking with eight prebuilt reports, including reports of successful and failed logins, active and inactive users, and user audit and event logs.

For more information, see http://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/solution-overview-c22-731967.html.

Business Value

Cisco ACI with Splunk Enterprise offers exceptional business value:

●   Unified and centralized visibility across your IT infrastructure

◦    Cisco ACI with Splunk Enterprise allows far-reaching visibility across your IT infrastructure. With the capability to unify machine data from physical and virtual servers, storage, and application environments as well as throughout the underlying Cisco ACI fabric and extended network, customers can see their entire system with a “big picture” view previously unavailable.

◦    Related dashboards: All.

●   Holistic health

◦    Environmental health information is central to Cisco ACI functions. Cisco ACI tracks, monitors, and trends the operational health of all components that run through and comprise the fabric. The health of tenants, applications, fabric hardware, and endpoints (both virtual and physical) is interwoven throughout the Cisco ACI for Splunk Enterprise solution.

◦    Related dashboards: Home, Fabric Details, Multi Pod, and Tenant Details.

●   Expedited resolution and root-cause analysis

◦    Quickly identifying faults and determining the root cause is always a challenge. With information from Cisco ACI and from storage resources, operating systems, applications, security devices, and endpoints correlated and then visualized through Splunk dashboards, you gain new insight. Problems previously difficult to identify can now be understood instantly, down to the fault-level component, application, policy, interface, etc. Deployment of this solution can reduce the mean time needed to investigate and resolve problems by up to 70percent^[1].

◦    Related dashboards: System Faults, Atomic Counters, Path Degradation, Tenant Utilization, and System Threshold.

●   Compliance

◦    Establishing an effective compliance and ethics program is now a necessity in nearly all organizations. The Cisco ACI App for Splunk Enterprise provides readily available compliance and security information with user analytics, including authentication and Cisco ACI environmental audit reporting capabilities.

◦    Related dashboard: Authentication.

●   Real virtual insight

◦    With the deep integration between Cisco ACI and VMware and visualization with Splunk, understanding your virtualized environment has never been easier. Every element, from the originating VMware vCenter application, host, virtual machine name, connected interface, associated EPG, etc, contributes to a meaningful view of your virtualized environment.

◦    Related dashboard: VMware.

●   Actionable security information

◦    Today you must be ready to respond when—not if—a security breach occurs. Natively, Cisco ACI supports microsegmentation, which allows organizations to reduce the potential for lateral movement in the event of a security breach. Now, with literally two mouse clicks, all your microsegmented details can be viewed in one place. Problems can be identified and acted on in minutes, not hours.

◦    Related dashboards: Microsegmentation and System Faults.

About This Cisco Validated Design

The Cisco ACI App for Splunk Enterprise solution has been validated using single-pod and multipod Cisco ACI deployments. The remainder of this document details the deployment of Cisco ACI in a single-pod environment with Splunk Enterprise.

Architecture Overview

This section provides an overview of the Cisco ACI and Splunk Enterprise architectures.

Cisco Application Centric Infrastructure

Cisco ACI is an innovative architecture that radically simplifies, optimizes, and accelerates the entire application deployment lifecycle. It uses a holistic systems-based approach, with tight integration between physical and virtual elements, an open ecosystem model, and innovation-spanning application-specific integrated circuits (ASICs), hardware, and software. This unique approach uses a common policy-based operating model across a network that supports Cisco ACI along with security elements (and computing and storage in the future), eliminating IT silos and drastically reducing cost and complexity.

The main benefits of Cisco ACI include:

●   Simplified automation with an application-based policy model

●   Common platform for managing physical, virtual, and cloud-based environments

●   Centralized visibility with real-time application health monitoring

●   Operation simplicity, with common policy, management, and operation models across application, network, and security resources (and computing and storage resources in the future)

●   Open software flexibility for DevOps teams and for ecosystem partner integration

●   Scalable performance and secure multitenancy

Cisco ACI consists of (Figure 1):

●   Cisco Application Policy Infrastructure Controller, or APIC

●   Cisco Nexus^® 9000 Series Switches (Cisco ACI spine and leaf switches)

●   Cisco ACI ecosystem

Cisco Application Policy Infrastructure Controller

The infrastructure controller is the main architectural component of the Cisco ACI solution. It is the unified point of automation and management for the Cisco ACI fabric, policy enforcement, and health monitoring. The APIC appliance is a centralized, clustered controller that optimizes performance and unifies the operation of the physical and virtual environments. The controller manages and operates a scalable multitenant Cisco ACI fabric.

The main features of the controller include:

●   Application-centric network policies

●   Data-model-based declarative provisioning

●   Application and topology monitoring and troubleshooting

●   Third-party integration (Layer 4 through Layer 7 [L4-L7]) services and VMware vCenter and vShield)

●   Image management (spine and leaf)

●   Cisco ACI inventory and configuration

●   Implementation on a distributed framework across a cluster of appliances

●   Health scores for critical managed objects (tenants, application profiles, switches, etc.)

●   Fault, event, and performance management

●   Cisco Application Virtual Switch (AVS), which can be used as a virtual leaf switch

The controller framework enables broad ecosystem and industry interoperability with Cisco ACI. It enables interoperability between a Cisco ACI environment and management, orchestration, virtualization, and L4-L7 services from a broad range of vendors.

Cisco ACI Features

The Cisco ACI mode fabric software is an optimized version of the Cisco NX-OS Software operating system that provides a foundation for building a programmable network infrastructure. NX-OS has been rewritten as a fully object-based switch operating system for Cisco ACI. The object model enables fluid programmability and full access to the underlying components of the infrastructure using representational state transfer (REST) APIs. This approach provides a framework for network control and programmability with a degree of openness that is not found in other systems.

The infrastructure controller provides centralized access to Cisco ACI through an object-oriented REST API framework with XML and JavaScript Object Notation (JSON) binding. It also supports a modernized, user-extensible command-line interface (CLI) and GUI. APIs have full read and write access to Cisco ACI, providing tenant- and application-aware programmability, automation, and system access.

Table 1 summarizes some of the Cisco ACI main features. For more information about additional features or the availability of these features by release, please refer to:

●   Cisco ACI data sheet: http://www.cisco.com/c/en/us/products/collateral/cloud-systems-management/application-policy-infrastructure-controller-apic/datasheet-c78-732414.html

●   Release notes for Cisco ACI and APIC: http://www.cisco.com/c/en/us/support/cloud-systems-management/application-policy-infrastructure-controller-apic/tsd-products-support-general-information.html

●   Release notes for Cisco Nexus 9000 SeriesSwitches: http://www.cisco.com/c/en/us/support/switches/nexus-9000-series-switches/products-release-notes-list.html

Table 1.       Cisco ACI Main Features

Figure 2 shows the Cisco ACI hardware components.

Cisco APIC Appliance Features

The APIC appliance has two form factors: for medium and for large configurations. Medium configurations have a medium-size CPU and hard drive and memory for up to 1000 edge ports. Large configurations have a large-size CPU and hard drive and memory for more than 1000 edge ports. The reference architecture discussed in this document deploys a medium-size appliance.

The APIC appliance uses a purpose-built Cisco UCS^®C220 M4 Rack Server manufactured with an image secured with a Trusted Platform Module (TPM), certificates, and an APIC product ID. To order the appliance clusters and additional Cisco ACI components, refer to the bill of materials (BOM) at the end of this document.

Figure 3 shows the APIC connection features.

Cisco Leaf Switch Connection Features

This section identifies the connection features that you use when connecting the Cisco Nexus 9396PX Switch to the Cisco ACI fabric as a leaf switch (Figure 4).

Cisco Spine Switch Connection Features

Figure 5 identifies the connection features that you use when connecting the Cisco Nexus 9336PQSwitch to the Cisco ACI fabric as a spine switch.

Splunk Enterprise

Splunk Enterprise provides a holistic way of organizing and extracting real-time insights from massive amounts of machine data, making it an excellent tool to pair with Cisco ACI. Because Cisco ACI has a single store of information (the APIC) and that data is indexed through Splunk, you can visualize the entire fabric as well as other parts of the IT infrastructure. Figure 6 shows the Splunk architecture.

The Splunk server software is written in C/C++ and Python and is provided in an all-in-one distribution. Although Splunk has several roles that can be configured (search head, indexer, forward, etc.), the design discussed here deploys all these roles in a single virtual machine. After Splunk is installed, two service processes will be running on your Linux system: splunkd and splunkweb.

●   splunkd is a distributed C/C++ server that accesses, processes, and indexes streaming IT data and also handles search requests. The splunkd processes and indexes your data by streaming it through a series of pipelines, each made up of a series of processors. Pipelines are single threads inside the splunkd process, each configured with a single snippet of XML. Processors are individual, reusable C/C++ or Python functions that act on the stream of IT data passing through a pipeline. Pipelines can pass data to one another through queues. splunkd supports a CLI for searching and viewing results.

●   splunkweb is a Python-based application server that provides the Splunk web user interface. It allows users to search and navigate IT data stored by Splunk servers and to manage the Splunk deployment through the browser interface. splunkweb communicates with your web browser through REST and communicates with splunkd through Simple Object Access Protocol (SOAP).

Solution Overview

The integrated solution of Splunk and Cisco ACI with the APIC at its core provides exceptional visibility and reduced time to troubleshoot through the use of comprehensive dashboards and unified views across all your IT infrastructure (Figure 7). Key health, performance, user, policy, tenant, and configuration data are all available in a centralized and easy-to-consume way using Splunk visualization features. For additional information, refer to the Cisco ACI and Splunk solutions brief at http://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/solution-overview-c22-731967.html.

Solution Details

The Cisco ACI environment and Splunk Enterprise should be deployed in accordance with the reference architecture information included at the end of this document. For detailed information about implementation of your Cisco ACI environment and for configuration and programming guides, consult the following link: http://www.cisco.com/c/en/us/support/cloud-systems-management/application-policy-infrastructure-controller-apic/tsd-products-support-series-home.html?

Installing Splunk Enterprise 6.4.4

Splunk Enterprise software runs on several supported platforms, including Microsoft Windows and several varieties of Unix and Linux. This document describes the installation steps for a deployment using 64-bit Ubuntu Linux 4.4.0-31-generic.

1.     Navigate to the preferred download location on your Linux server. Enter the following command to download the Splunk installation file (Figure 8):

wget -O splunk-6.4.4-b53a5c14bb5e-Linux-x86_64.tgz https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=6.4.4&product=splunk&filename=splunk-6.4.4-b53a5c14bb5e-Linux-x86_64.tgz&wget=true

2.     Enter the following command to unpack and install Splunk:

tar xvzf splunk-6.4.4-b53a5c14bb5e-Linux-x86_64.tgz -C /opt

3.     Export the variable for the splunk directory:

export SPLUNKHOME=/opt/splunk

4.     Navigate to the /$SPLUNKHOME/bin directory:

cd /$SPLUNKHOME/bin

5.     Start Splunk and accept the user license (Figure 9):

sudo ./splunk start - accept-license

Starting Splunk Web Server Setup

When you start Splunk, a web service will run. To access this service, navigate in a web browser to http://your_server_name:8000 (Figure 10).

Installing Your Splunk License

Install your Splunk license as shown in Figures 11a, 11b, and 11c.

Installing Cisco ACI App for Splunk Enterprise

Follow these steps to install the Cisco ACI App for Splunk Enterprise:

1.     Download the Cisco ACI App for Splunk Enterprise from https://splunkbase.splunk.com/app/1896/ (Figure 12).

2.     Download the Cisco ACI Add-on for Splunk Enterprise from https://splunkbase.splunk.com/app/1897/ (Figure 13).

3.     Accept the license agreements and agree to download (Figure 14).

4.     Copy the files to the Splunk server (Figures 15 and 16).

5.     Install the Cisco ACI App for Splunk Enterprise with the following command:

sudo tar xvzf cisco-aci-app-for-splunk-enterprise_22.tgz –C /$SPLUNKHOME/etc/apps/

6.     Restart Splunk:

cd /$SPLUNKHOME/bin

sudo ./splunk restart

7.     Verify the installation by navigating to http://your_server_name:8000 (Figure 17).

8.     Update the application by navigating to http://your_server_name:8000/en-us/_bump and clicking “Bump version” (Figure 18).

9.     Restart Splunk:

cd /$SPLUNKHOME/bin

sudo ./splunk restart

Installing Cisco ACI Add-on for Splunk Enterprise

Follow these steps to install the Cisco ACI Add-on for Spunk Enterprise:

1.     Install the Cisco ACI Add-on for Splunk Enterprise:

sudo tar xvzf cisco-aci-add-on-for-splunk-enterprise_22.tgz-C /$SPLUNKHOME/etc/apps/

2.     Restart Splunk:

cd /$SPLUNKHOME/bin

sudo ./splunk restart

3.     From the Splunk home screen, click the gear icon next to Apps (Figure 19).

4.     On the line for Cisco ACI Add-on for Splunk Enterprise, click “Set up” (Figure 20).

5.     Provide the credentials for your APIC (Figure 21).

6.     Go to Settings (Figure 22) and under Data click “Data inputs” (Figure 23).

7.     In the App column, enable all scripts associated with TA_cisco-ACI (Figure 24).

8.     Restart Splunk:

cd /$SPLUNKHOME/splunk/bin

sudo ./splunk restart

9.     Allow up to 15 minutes to populate the data.

Cisco ACI App for Splunk Enterprise Operation

To launch the application, from the main Splunk screen after login click Cisco ACI App for Splunk Enterprise (Figure 25).

General Use

This section describes features for the general operation of the Cisco ACI App for Splunk Enterprise.

Navigation

Application dashboards are accessible by navigating across the green ribbon. The dashboard categories are Home, Help Desk, Fabric, Tenants, VM Manager, Search, and Setup Guide.

Within a Dashboard

There are several dashboards with readings, metrics, and other useful visualizations related to your Cisco ACI environment. Typically, you can interact with these items to drill down into details, or to further expand information you want to see.

Visualization Behaviour

Visualization options include the following:

●   Bar graph, column graph, and pie chart visualizations: When you interact with bar graphs, column graphs, or pie charts, an in-page drill-down feature will appear below the bar graph, column graph, or pie chart.

●   Table visualizations: Table visualizations are a final level of drill-down feature. If you want to see additional information, click the magnifying glass icon while hovering over the visualization to bring up the Splunk search that was used to produce the table.

●   Single-value visualizations: When you click a single-value visualization, a new tab with an expanded dashboard or table related to the single-value visualization is displayed.

●   Timeline graph visualizations: No further drill-down interactions are available when you interact with timeline graphs.

●   All visualization behavior: Each visualization has a hover bar below it that contains links as described in Figure 26.

Time Picker

Just as in a standard search in Splunk, many of the dashboards contain a time picker to help narrow the range related to information in the dashboard.

APIC Host

The APIC host picker appears on each screen. If you have connected more than one APIC fabric, you can use this drop-down menu to filter by the specific fabric for which you want to view details.

Additional Filters

Certain dashboards have additional filters such as health score, severity, user, source node, destination node, pod name, tenants, applications, EPGs, VMware ESXi hosts, and virtual machines (VMs).

Home Dashboard

The Home dashboard is your starting reference with a high-level overall view of your Cisco ACI fabric (Figure 27).

APICs Table

The APICs table provides information related to the hardware components and base-level configuration (such as IP address) that make up your APIC cluster.

Fabric Health: History Chart

Fabric health over time is depicted as a line graph. Because the data is indexed in Splunk, users can access a longer history than is available in the APIC advanced GUI.

Home Dashboard Single-Value Visualizations

Table 2 lists each single-value visualization and the corresponding dashboard to which it relates. Each dashboard defined in this table is discussed in more detail later in this document.

Table 2.       Visualization-to-Dashboard Mapping

Help Desk Dashboards

The Help Desk dashboards consist of System Faults, Atomic Counters, Path Degradation, and System Threshold (Figure 28).

Help Desk: System Faults

The Help Desk: System Faults dashboard details APIC system faults visualized in several ways (Figure 29).

Help Desk: System Faults Dashboard Single-Value Visualizations

New-tab tables are associated with each single-value visualization in the Help Desk dashboard single-value visualizations.

Faults

Faults is a total count of faults, both Acknowledged and Unacknowledged (Figure 30).

Acknowledged Faults

Acknowledged Faults is a subset of faults that contains only faults that have been acknowledged (Figure 31).

Unacknowledged Faults

Similar to Acknowledged Faults, Unacknowledged Faults is a subset of faults that contains only faults that have not been acknowledged (Figure 32).

Faults by Node

Faults by Node is a pie chart depicting system faults by fabric node. Interacting with a slice will open a detail table below the pie chart containing all instances of faults for that particular fabric node (Figure 33).

Faults by Tenant

Faults by Tenant is a pie chart depicting system faults by tenant. Interacting with a slice will open a detail table below the pie chart containing all instances of faults for that particular tenant (Figure 34).

Faults by Severity

Faults by Severity is a pie chart depicting system faults by level of severity. Interacting with a slice will open a detail table below the pie chart containing all instances of faults with that particular severity level (Figure 35).

Faults by Domain

Faults by Domain is a pie chart depicting system faults by ACI domain. Interacting with a slice will open a detail table below the pie chart containing all instances of faults with that particular domain (Figure 36).

Faults by Severity over Time

Faults by Severity over Time is a timeline graph depicting system faults by severity over time.

Faults by Type

Faults by Type is a bar graph depicting system faults by the type of fault. Interacting with a bar in the graph will open a detail table below the bar graph containing all instances of faults of that particular type (Figure 37).

Top Faults by Rule

Top Faults by Rule is a pie chart depicting system faults sliced by a rule. Interacting with a slice will open a detail table below the pie chart containing all instances of faults with that particular rule (Figure 38).

Top Faults by Cause

Top Faults by Cause is a pie chart depicting system faults sliced by cause. Interacting with a slice will open a detail table below the pie chart containing all instances of faults with that particular cause (Figure 39).

Latest Affected Objects

Latest Affected Objects is a table displaying the fabric objects most recently affected (Figure 40).

Help Desk: Atomic Counters

The Atomic Counters dashboard (Figure 41) contains two table elements that display information when you use Cisco ACI to troubleshoot with atomic counters: Endpoint to Endpoint (EP to EP) and Endpoint Group to Endpoint Group (EPG to EPG). If you have not used atomic counters to troubleshoot EP to EP or EPG to EPG, no results will be displayed.

Help Desk: Path Degradation

The Path Degradation dashboard (Figure 42) contains a table that displays information when you use Cisco ACI to troubleshoot intrafabric traffic using atomic counters. If you have not used atomic counters to troubleshoot intrafabric traffic, no results will be displayed.

Help Desk: System Threshold

The System Threshold dashboard provides easy-to-view user-definable fabric thresholds. Among them are Tenant, EPG, Contracts, Filters, Bridge Domains, and L3OUT Networks, all depicted as easy-to-read gauges (Figure 43). All these visualizations have an in-window Change Threshold link that opens a new tab and allows you to make changes to the thresholds set.

Fabric Dashboards

The Fabric menu on the green navigation bar consists of three dashboards accessible from the drop-down menu. These dashboards are Fabric Details, Authentication, and Multi Pod (Figure 44).

Fabric: Fabric Details

The Fabric Details dashboard displays health statistics for various nodes in your Cisco ACI fabric (Figure 45).

Top Affected Leafs

Top Affected Leafs visualizes health scores in a colored column graph for each leaf node in your Cisco ACI fabric. Interacting with a column in the graph will open seven tables below the graph containing hardware, health, utilization, and fault details related to that particular leaf node (Figure 46).

Top Affected Spines

In the same way as Top Affected Leafs, Top Affected Spines visualizes node health as a colored column graph for each spine in your Cisco ACI fabric. The same seven tables will appear below the column graph when you interact with a specific column in the Top Affected Spines visualization (Figure 47).

Health/Fault Details: Leafs

Health/Fault Details: Leafs is a table listing health and fault information for leaf switches over a period of time specified in the time picker.

Health/Fault Details: Spines

Health/Fault Details: Spines, just like the table for leaf switches, visualizes health and fault information over a specified period of time.

TCAM Percentage Threshold Statistics

TCAM Percentage Threshold Statistics is a simple table showing current settings for Warning Threshold, Critical Threshold, and Max Threshold percentages.

Top TCAM Usage by Node

Top TCAM Usage by Node is a statistics table showing colored bars in a graph for each fabric node (Figure 48). The Change Threshold link in the Top TCAM Usage by Node window will open a new tab and allow you to adjust the TCAM percentage threshold values. Interacting with a bar on the chart will open two additional tables beneath the TCAM Percentage Threshold Statistics bar chart.

Leafs – Port Utilization and Thresholds

The Leafs – Port Utilization and Thresholds table presents summarized egress and ingress information along with threshold levels for each leaf switch (Figure 49).

Spines – Port Utilization and Thresholds

The Spines – Port Utilization and Thresholds table presents summarized egress and ingress information along with threshold levels for each spine switch (Figure 50).

Change Threshold (for Leaf and Spine Utilization)

The Change Threshold link opens a new tab on which you can change values for Warning and Critical thresholds related to port utilization on Cisco ACI fabric leaf and spine switches (Figure 51).

Fabric: Authentication

The Authentication dashboard displays information about users, authentication attempts, and audit information (Figure 52).

Authentication Dashboard Single-Value Visualizations

New-tab tables are associated with each single-value visualization on the Authentication dashboard:

●   All Users (Figure 53)

●   Local Users (Figure 54)

●   Remote Users (Figure 55)

Authentication by Admin

Authentication by Admin is a pie chart depicting successful authentications by the admin user by IP address. Clicking the chart will open a table below the main visualizations window with historical data related to the pie slice selected (Figure 56).

Authentication Failed by User

Authentication Failed by User is a column chart depicting failed authentications by user. Clicking an individual column will open a table below the main visualizations window with historical data related to that specific user (Figure 57).

Authentication Success by User

Authentication Success by User is a column chart depicting successful authentications by user. Clicking an individual column will open a table below the main visualizations window with historical data related to that specific user (Figure 58).

Fabric: Multi Pod

Multi Pod setup and configuration are outside the scope of this document. However, a customer who deploys the Cisco ACI App for Splunk Enterprise will have access to the Multi Pod dashboard (Figure 59). The Multi Pod dashboard provides an overall view of each pod in a multipod environment. In addition to the time picker filter, users can filter by health score and pod name.

APICs

The APICs table has important details related to your APIC cluster, such as name, management IP address, and pod membership.

Fabric Health – History

Fabric Health – History depicts the history of the fabric health for each pod of your multipod deployment as a health trend over time.

Leafs

Leafs provides a count of total leaf switches categorized by pod and represented by a column graph (Figure 60). When you interact with a column on the graph, an additional visualization will open below the column chart with specific health information for each individual leaf switch.

Affected Leafs of pod-#

You can drill down further by interacting with a specific leaf switch in the column chart. Doing so will open six tables with hardware-specific information for that leaf as shown in Figure 61.

Spines

Spines displays a count of total spine switches categorized by pod and represented by a column graph (Figure 62). When you interact with the column on the graph, an additional visualization will open below the column chart with specific health information for each individual spine switch.

Affected Spines of pod-#

You can drill down further by interacting with a specific spine switch in the column chart. Doing so will open six tables with hardware-specific information for that spine switch as shown in Figure 63.

Critical Faults

Critical Faults is a pie chart depicting pods in your multipod environment. When you select a slice, a new visualization appears below the Critical Faults pie chart.

Time Chart: Critical Fault (30-day period) for pod-x

The Critical Fault chart depicts critical faults over a 30-day period for the selected pod.

EPGs

EPGs are represented as a pie chart of the pods of your multipod environment. Interacting with a slice will open two new visualizations below the EPGs pie chart.

EPGs with Static Ports for pod-x

EPGs with Static Ports for pod-x displays, by tenant, a count of EPGs with port assignments (Figure 64). Interacting with a particular column will open two additional tables below the column graph with static port information and EPG health for the selected tenant.

EPG Static Port Details for Tenant: tenant

EPG Static Port Details for Tenant: tenant displays information about the port and EPG assignments for the selected tenant (Figure 65).

EPG Health Details for Tenant: tenant

EPG Health Details for Tenant: tenant displays information about EPG health for the selected Tenant. (Figure 66).

EPGs Unassigned to Any Pod

If EPGs are created but are not assigned to ports in your Cisco ACI fabric, they will be depicted in this column graph (Figure 67). Interacting with columns among the tenants listed in the column graph will open a table below it.

EPG Health Details for Tenant: tenant

EPG Health Details for Tenant: tenant displays information about EPG health for the selected Tenant. This information is displayed when selecting a tenant from among the columns of tenants in the EPGs Unassigned to any Pod column graph (Figure 68).

Tenants Dashboards

The Tenants menu on the green navigation bar consists of three dashboards accessible from the drop-down menu. These dashboards are Tenant Details, Tenant Utilization, and Micro segmentation (Figure 69).

Tenants: Tenant Details

The Tenant Details dashboard displays basic health details by tenant (Figure 70).

Top 10 Affected Tenants’ Health

Top 10 Affected Tenants’ Health is a bar chart that shows colored health scores by tenant. Interacting with a bar in the visualization will open additional on-screen panels beneath the bar chart with details related to the selected tenant.

Application Health for Tenant: tenant

The Application Health for Tenant: tenant table shows health scores by application for the selected tenant (Figure 71).

End Point Group Health for Tenant: tenant

The End Point Group for Tenant: tenant table shows health scores by EPG and related applications for the selected tenant (Figure 72).

Application Statistics

The Application Statistics table shows utilization statistics for each application of the selected tenant (Figure 73).

Client End Point Details

The Client End Point Details table lists endpoint information for the selected tenant (Figure 74).

Top 10 Affected Tenants’ Faults

Top 10 Affected Tenants’ Faults is a pie chart depicting fault count by tenant. Interacting with a particular slice will open a table below the pie chart with additional information.

<tenant> Tenant Fault Details

The Tenant Fault Details table shows related faults for the tenant selected (Figure 75).

Tenants: Tenant Utilization

The Tenant Utilization dashboard displays packet information categorized by tenant (Figure 76). Interacting with either the Ingress or Egress Utilization column charts will open two tables beneath the column charts with additional information.

<tenant>-Ingress and <tenant>-Egress Utilization Statistics in Bytes

The <tenant>-Ingress Utilization Statistics in Bytes and <tenant>-Egress Utilization Statistics in Bytes tables display port and ingress and egress statistics for the selected tenant (Figure 77).

Tenants: Microsegmentation

The Microsegmentation dashboard displays information about microsegmented endpoints by tenant (Figure 78). Microsegmentation uses two primary filtering mechanisms: network-based and virtual machine–based attribute filtering.

No. of EPGs Microsegmented per Tenant

No. of EPGs Microsegmented per Tenant is a column chart listing each tenant that contains one or more microsegmented EPG and a count of them. Interacting with a column in the chart opens three additional tables to the right and below the column chart.

Health Details of Microsegmented EPGs for Tenant: tenant

The Health Details table shows health details for microsegmented EPGs of the selected tenant (Figure 79).

Microsegmented Domains (VMs and Bare-Metal)

The Microsegmented Domains table shows Cisco ACI domain and associated details for microsegmented EPGs of the selected tenant (Figure 80).

Client Endpoints

The Client Endpoints table shows endpoint details associated with the microsegmented EPGs of the selected tenant (Figure 81).

Network-Based Attributes

Network-Based Attributes is a table with specific information related to the value of a particular network attribute and the specific filter used to microsegment an endpoint based on the particular network attribute (Figure 82).

VM-Based Attributes

VM-Based Attributes is a table with specific information related to the value of a particular virtual machine attribute and the specific filter used to microsegment an endpoint based on the particular virtual machine attribute (Figure 83).

VM Manager Dashboards

The VM Manager dashboards contain information related to virtualized endpoints (Figure 84). At this time, only VMware is supported, but future versions of the application will support other virtualized tools.

VM Manager: VMware

The VMware dashboard contains important endpoint details related to your VMware virtualized environment (Figure 85). Comprehensive filtering of this information is possible using the time picker drop-down menu or filtering by tenant, application, EPG, ESX host, or virtual machine. This table contains no additional drill-down capabilities.

Search

The Search window is similar to the main Splunk Search application, but it applies specifically to your Cisco ACI fabric and machine data gathered from the Cisco ACI App for Splunk Enterprise (Figure 86).

Setup Guide

Setup Guide is a guide to the setup and configuration contained in this document and is provided for easy future reference (Figure 87).

Creating Custom Dashboards

Splunk provides a native capability to create custom dashboards with visualizations based on searches of indexed data. This section discusses the source types containing information about your Cisco ACI environment indexed through the Cisco ACI App for Splunk Enterprise and describes the process for creating a custom dashboard.

Data Indexed by Cisco ACI App for Splunk Enterprise

One primary index is created when you use the Cisco ACI App for Splunk Enterprise. This index is referred to as the apic index. This index contains five source types, which are discussed in detail here (Figure 88).

cisco:apic:stats

The cisco:apic:stats source type contains information related to historical total and average aggregated statistics for ingress and egress packets in a specified fabric.

cisco:apic:class

The cisco:apic:class source type contains the majority of configuration data (excluding health information) about managed objects in the specified fabric.

cisco:apic:health

The cisco:apic:health source type contains historical health information for the managed objects of the specified fabric.

cisco:apic:authentication

The cisco:apic:authentication source type contains user-authentication data.

apicsyslog

The apicsyslog source type contains syslog data.

Building a Custom Dashboard

Splunk offers many ways to visualize data searched from an index. This document discusses the setup for three primary visualizations, explains the search used to build the visualizations, and describes how to create or add the visualizations to your custom dashboard.

Custom Dashboard: Single-Value Visualization

You will get a distinct count of the number of microsegmented EPGs to use for this visualization.

1.     Click Search on the main navigation bar.

2.     Search the apic index (index=apic) to find EPGs (component=fvEPG) that are attribute based (isAttrBasedEPg=yes), which indicates that the EPG is microsegmented. Then pipe ( | ) the results to the statistics command (stats) requesting a distinct count based on the name (dc(name)) of the EPG with the following search string:

index=apic component=fvEPG isAttrBasedEPg=yes | stats dc(name)

3.     Click the Visualization tab in the Search window and verify that the visualization type is set to Single Value (Figure 89).

4.     In the upper-right portion of the Search window, click the Save As drop-down menu and select Dashboard Panel.

5.     Configure the Save As Dashboard Panel as shown in Figure 90. Then click Save.

6.     In the Your Dashboard Panel Has Been Created dialog box, click View Dashboard. Your custom dashboard should look similar to Figure 91.

Custom Dashboard: Column Chart Visualization

For this visualization, you will display errors by severity level categorized by tenant.

1.     Click Search on the main navigation bar.

2.     Perform the search as follows:

a. Search the apic index (index=apic).

b. Filter the source type by apic health (sourcetype=cisco:apic:health).

c. Filter by the specific apic cluster, referencing a node of that cluster by IP address (apic_host=10.23.248.116).

d. Include all tenants (component=fvTenant) and all events that contain “warning,” “minor,” or “major” ((warning OR minor OR major)).

e. Pipe ( | ) the data to the chart command showing a count of each type of error for each tenant and categorized by severity (chart count over name by severity).

Here is the complete search:

index=apic sourcetype=cisco:apic:health apic_host=10.23.248.116 component=fvTenant (warning OR minor OR major) | chart count over name by severity

3.     Click the Visualization tab in the Search window and verify that the visualization type is set to Column Chart (Figure 92).

4.     In the upper-right portion of the Search window, click the Save As drop-down menu and select Dashboard Panel.

5.     Configure the Save As Dashboard Panel as shown in Figure 93. Then click Save.

6.     In the Your Dashboard Panel Has Been Created dialog box, click View Dashboard. Your custom dashboard should now look similar to Figure 94.

Custom Dashboard: Table Visualization

For the final visualization, you will represent the virtualization information for your VMware environment in a table.

1.     Click Search on the main navigation bar.

2.     This search is a little more complex:

a.   Enter a pipe ( | ) character to indicate that what follows is a macro.

b.    Enter the name of the macro enclosed in a single quotation mark (`) character: for example, `end_point_detail`.

c.    Pass the results of the macro to a pipe ( | ) followed by the search command and each of the limiters to search (search apic_host=10.23.248.116 Tenant=* Application=* EPG=* VirtualMachine=* ESX-Host=*).

d.   Pass these search results further down the pipeline to the table command to list the table headers related to the data you want displayed (| table Tenant, Application, EPG, EPG-Health, VirtualMachine, state, Network-Adapter, ESX-Host, vCenter, Interface).

e.   For the final pipeline connection, use the rename command to change some of the header names to make them more user friendly (| rename VirtualMachine AS "VirtualMachine" ESX-Host AS "ESX host" Network-Adapter AS "Network Adapter" EPG-Health AS "EPG Health" state AS "State").

Here is the complete search:

| `end_point_detail` | search apic_host=10.23.248.116 Tenant=* Application=* EPG=* VirtualMachine=* ESX-Host=* | table Tenant, Application, EPG, EPG-Health, Virtual Machine, state, Network-Adapter, ESX-Host, vCenter, Interface | rename VirtualMachine AS "Virtual Machine" ESX-Host AS "ESX host" Network-Adapter AS "Network Adapter" EPG-Health AS "EPG Health" state AS "State"

3.     On the Statistics tab, view the table resulting from the search (Figure 95).

4.     In the upper-right portion of the Search window, click the Save As drop-down menu and select Dashboard Panel.

5.     Configure the Save As Dashboard Panel as shown in Figure 96. Then click Save.

6.     In the Your Dashboard Panel Has Been Created dialog box, click View Dashboard. Your completed custom dashboard should now look similar to Figure 97.

Accessing Your Custom Dashboard

You can access your newly created custom dashboard by searching for it in the Find field or by assigning it as a home dashboard.

Find Field Method

The Find field is accessible on the far right of the black ribbon in the Splunk web interface (Figure 98). Typing the name of your custom dashboard and selecting it will display it.

Home Dashboard Assignment Method

To assign your newly created dashboard as a home dashboard, follow these steps:

1.     Click the Splunk > link in the upper-left corner of the webpage.

2.     On the Splunk start page (Figure 99), click anywhere in the box that says “Choose a home dashboard.”

3.     In the Choose Default Dashboard dialog box, select your dashboard from the drop-down list and click Save (Figure 100).

Your custom dashboard is now accessible from the Splunk start page.

Tuning the Cisco ACI App

As installed, the Cisco ACI App for Splunk Enterprise requires no additional modifications. However, depending on your Splunk license consumption, you may want to make modifications to better align your use with your Splunk license.

The Splunk scripts used to enable the application specify data polling based at a predefined interval (represented in seconds). Increasing this interval (to a higher number) will result in a longer polling cycle, less frequent indexing, slightly less-current data, and lower Splunk license consumption. Decreasing the interval (to a lower number) will do the opposite, resulting a shorter polling cycle, more frequent indexing, more-current data, and greater consumption of your Splunk license.

You should adjust these timers only if you need to reconcile your Splunk license or to acquire a view of your data that is closer to a real-time view.

Conclusion

Cisco ACI allows you to automate provisioning of network and application services, provide a multitenant environment with whitelist networking, and deploy a highly secure and policy-based microsegmented endpoint environment, while integrating physical and virtual endpoints and achieving outstanding scalability.

Splunk, the world leader in making sense of your machine data, enhances Cisco ACI further by providing organized dashboards on which you can easily view your entire system, troubleshoot, rapidly assess root causes, and monitor system health, in real time or historically, for all your Cisco ACI physical, software, application, virtualized, and connected components.

Appendix

Solution Design and Specifications

Table 3 summarizes the specifications for the Cisco ACI and Splunk Enterprise reference design.

Table 3.       Cisco ACI, Splunk Enterprise, and Cisco ACI App for Splunk Enterprise Reference Architecture

Cabinet Configuration

Figure 101 shows the Cisco ACI physical infrastructure and connections.

Note:    Splunk can be installed either within a Cisco ACI fabric network or on a fabric network other than Cisco ACI. Likewise, Splunk can run on a bare-metal server or in a host-based virtualized environment. The three servers listed in Figure 101 are shown strictly to illustrate a sample physical environment and connection layout.

Detailed Connection Diagram

Figure 102 shows Cisco ACI fabric connectivity.

Bill of Materials

Tables 4 through 7 provide the ordering information for the single-pod Cisco ACI environment with Splunk Enterprise.

Table 4.       Cisco ACI APIC Appliance Bill of Materials

Table 5.       Cisco ACI Spine Switch Bill of Materials

Table 6.       Cisco ACI Leaf Switch Bill of Materials

Table 7.       Splunk Enterprise Software and Support