Table Of Contents

Performance Monitoring

Real-Time Performance Monitoring

Device Manager Real-Time Performance Monitoring

Fabric Manager Real-Time ISL Statistics

Historical Performance Monitoring

Creating a Flow with Performance Manager

Creating a Collection with Performance Manager

Using Performance Thresholds

Using the Performance Manager Configuration Wizard

Viewing Performance Manager Reports

Performance Summary

Performance Tables and Details Graphs

Viewing Performance of Host-Optimized Port Groups

Viewing Performance Manager Events

Generating Top10 Reports in Performance Manager

Generating Top10 Reports Using Scripts

Exporting Data Collections to XML Files

Exporting Data Collections in Readable Format

Configuring Performance Manager for Use with Cisco Traffic Analyzer

Performance Monitoring

---

Cisco Fabric Manager and Device Manager provide multiple tools for monitoring the performance of the overall fabric, SAN elements, and SAN links. These tools provide real-time statistics as well as historical performance monitoring.

This chapter contains the following sections:

•Real-Time Performance Monitoring

•Historical Performance Monitoring

Real-Time Performance Monitoring

Real-time performance statistics are a useful tool in dynamic troubleshooting and fault isolation within the fabric. Real-time statistics gather data on parts of the fabric in user-configurable intervals and display these results in Fabric Manager and Device Manager.

Device Manager Real-Time Performance Monitoring

Device Manager provides an easy tool for monitoring ports on the Cisco MDS 9000 Family switches. This tool gathers statistics at a configurable interval and displays the results in tables or charts. These statistics show the performance of the selected port in real-time and can be used for performance monitoring and troubleshooting. For a selected port, you can monitor any of a number of statistics including traffic in and out, errors, class 2 traffic, and FICON data. You can set the polling interval from ten seconds to one hour, and display the results based on a number of selectable options including absolute value, value per second, and minimum or maximum value per second.

Device Manager checking for oversubscription on the host-optimized four-port groups on relevant modules. Right-click the port group on a module and choose Check Oversubscription from the pop-up menu.

Device manager provides two performance views, the Summary View tab, and the configurable monitor option per port.

To configure the summary view in Device Manager, follow these steps:

---

Step 1 Click the Summary tab on the main display.

You see all of the active ports on the switch, as well as the configuration options available from the Summary view shown in Figure 60-1.

Figure 60-1 Device Manager Summary Tab

Step 2 Select a value from the Poll Interval drop-down list.

Step 3 Decide how you want your data to be interpreted by looking at the Show Rx/Tx drop-down menu. The table updates each polling interval to show an overview of the receive and transmit data for each active port on the switch.

Step 4 Select a value from the Show Rx/Tx drop-down list. If you select Util%, you need to also select values from the two Show Rx/Tx > %Util/sec drop-down lists. The first value is the warning level and the second value is the critical threshold level for event reporting.

Note that you can also display percent utilization for a single port by selecting the port and clicking the Monitor Selected Interface Traffic Util % icon.

Step 5 Click the Save Configuration icon.

---

The configurable monitor per port option gives statistics for in and out traffic on that port, errors, class 2 traffic and other data that can be graphed over a period of time to give a real-time view into the performance of the port.

To configure per port monitoring using Device Manager, follow these steps:

---

Step 1 Click the Device tab.

Step 2 Right-click the port you are interested in and choose Monitor from the drop-down menu.

You see the port real-time monitor dialog box shown in Figure 60-2.

Figure 60-2 Device Manager Monitor Dialog Box

Step 3 Select a value from the Interval drop-down list to determine how often data is updated in the table shown here.

Step 4 Click a statistical value in the table then click one of the graphing icons to display a running graph of that statistic over time. You see a graph window that contains options to change the graph type.

---

Tip You can open multiple graphs for statistics on any of the active ports on the switch.

---

---

Fabric Manager Real-Time ISL Statistics

You can configure Fabric Manager to gather ISL statistics in real time. These ISL statistics include receive and transmit utilization, bytes per second, as well as errors and discards per ISL.

To configure ISL statistics using Fabric Manager, follow these steps:

---

Step 1 Choose Performance > ISLs in Real-Time.

You see any ISL statistics in the Information pane (see Figure 60-3).

Figure 60-3 ISL Performance in Real Time

Step 2 Select a value from the Poll Interval drop-down list.

Step 3 Select two values from the Bandwidth utilization thresholds drop-down lists, one value for the minor threshold and one value for the major threshold.

The table shown updates each polling interval to show the statistics for all configured ISLs in the fabric.

Step 4 Select a row in the table to highlight that ISL in blue in the Topology map.

---

Historical Performance Monitoring

Performance Manager gathers network device statistics historically and provides this information graphically using a web browser. It presents recent statistics in detail and older statistics in summary. Performance Manager also integrates with external tools such as Cisco Traffic Analyzer.

See the "Performance Manager Architecture" section on page 8-1 for an overview of Performance Manager.

Creating a Flow with Performance Manager

Performance Manager has a Flow Configuration Wizard that steps you through the process of creating host-to-storage, storage-to-host, or bidirectional flows. See the "Flow Statistics Configuration" section on page 8-3 for information on creating flows.

Creating a Collection with Performance Manager

The Performance Manager Configuration Wizard steps you through the process of creating collections using configuration files. Collections are defined for one or all VSANs in the fabric. Collections can include statistics from the SAN element types described in Table 60-1.

Table 60-1 Performance Manager Collection Types

Collection Type	Description
ISLs	Collects link statistics for ISLs.
Host	Collects link statistics for SAN hosts.
Storage	Collects link statistics for a storage elements.
Flows	Collects flow statistics defined by the Flow Configuration Wizard.

Using Performance Thresholds

The Performance Manager Configuration Wizard allows you to set up two thresholds that trigger events when the monitored traffic exceeds the percent utilization configured. These event triggers can be set as either Critical or Warning events that are reported on the Fabric Manager web client Events browser page.

You must choose either absolute value thresholds or baseline thresholds that apply to all transmit or receive traffic defined in the collection. Click the Use absolute values radio button on the last screen of the Performance Manager Configuration Wizard to configure thresholds that apply directly to the statistics gathered. These statistics, as a percent of the total link capacity, are compared to the percent utilization configured for the threshold type. If the statistics exceed either configured threshold, an event is shown on the Fabric Manager web client Events tab.

As an example, the collection has absolute value thresholds set for 60% utilization (for warning) and 80% utilization (for critical). If Performance Manager detects that the traffic on a 1-Gigabit link in its collection exceeds 600 Mbps, a warning event is triggered. If the traffic exceeds 800 Mbps, a critical event is triggered.

Baseline thresholds are defined for a configured time of day or week (1 day, 1 week, or 2 weeks). The baseline is created by calculating the average of the statistical results for the configured time each day, week, or every 2 weeks. Table 60-2 shows an example of the statistics used to create the baseline value for a collection defined at 4 pm on a Wednesday.

Table 60-2 Baseline Time Periods for a Collection Started on Wednesday at 4pm

Baseline Time Window	Statistics Used in Average Calculation
1 day	Every prior day at 4 pm
1 week	Every prior Wednesday at 4 pm
2 weeks	Every other prior Wednesday at 4 pm

Baseline thresholds create a threshold that adapts to the typical traffic pattern for each link for the same time window each day, week, or every 2 weeks. Baseline thresholds are set as a percent of the average (110% to 500%), where 100% equals the calculated average.

As an example, a collection is created at 4 pm on Wednesday, with baseline thresholds set for 1 week, at 150% of the average (warning) and 200% of the average (critical). Performance Manager recalculates the average for each link at 4 pm every Wednesday by taking the statistics gathered at that time each Wednesday since the collection started. Using this as the new average, Performance Manager compares each received traffic statistic against this value and sends a warning or critical event if the traffic on a link exceeds this average by 150% or 200% respectively.

Table 60-3 shows two examples of 1-Gigabit links with different averages in our example collection and at what traffic measurements the Warning and Critical events are sent.

Table 60-3 Example of Events Generated for 1-Gigabit Links

Average	Warning Event Sent at 150%	Critical Event Sent at 200%
400 Mbps	600 Mbps	800 Mbps
200 Mbps	300 Mbps	400 Mbps

Set these thresholds on the last screen of the Collections Configuration Wizard by checking the Send events if traffic exceeds threshold check box.

Using the Performance Manager Configuration Wizard

See the "Creating Performance Collections" section on page 7-50.

Viewing Performance Manager Reports

You can view Performance Manager statistical data using preconfigured reports that are built on demand and displayed in a web browser. These reports provide summary information as well as detailed statistics that can be viewed for daily, weekly, monthly, or yearly results.

Choose Performance > Reports to access Performance Manager reports from Fabric Manager. This opens a web browser window showing the default Fabric Manager web client event summary report. Click the Performance tab to view the Performance Manager reports. Performance Manager begins reporting data ten minutes after the collection is started

---

Note Fabric Manager Web Server must be running for this to work. See the "Launching Fabric Manager Web Server" section on page 7-7.

---

Performance Summary

The Performance Summary page presents a dashboard display of the throughput and link utilization for hosts, ISLs, storage, and flows for the last 24-hour period. The summary provides a quick overview of the fabric's bandwidth consumption and highlights any hotspots.

The report includes network throughput pie charts and link utilization pie charts. Use the navigation tree on the left to show summary reports for monitored fabrics or VSANs. The summary displays charts for all hosts, storage elements, ISLs, and flows. Each pie chart shows the percent of entities (links, hosts, storage, ISLs, or flows) that measure throughput or link utilization on each of six predefined ranges. Move the mouse over a pie chart section to see how many entities exhibit that range of statistics. Double-click any pie chart to bring up a table of statistics for those hosts, storage elements, ISLs, or flows.

Performance Tables and Details Graphs

Click Host, Storage, ISL, or Flow to view traffic over the past day for all hosts, storage, ISLs, or flows respectively. A table lists all of the selected entities, showing transmit and receive traffic and errors and discards, if appropriate. The table can be sorted by any column heading. The table can also be filtered by day, week, month, or year. Tables for each category of statistics display average and peak throughput values and provide hot-links to more detailed information.

Clicking a link in any of the tables opens a details page that shows graphs for traffic by day, week, month, and year. If flows exist for that port, you can see which storage ports sent data. The details page also displays graphs for errors and discards if they are part of the statistics gathered and are not zero.

If you double-click a graph on a Detail report, it will launch the Cisco Traffic Analyzer for Fibre Channel, if configured. The aliases associated with hosts, storage devices, and VSANs in the fabric are passed to the Cisco Traffic Analyzer to provide consistent, easy identification.

Viewing Performance of Host-Optimized Port Groups

You can monitor the performance of host-optimized port groups by clicking Performance > End Devices and selecting Port Groups from the Type drop-down list.

Viewing Performance Manager Events

Performance Manager events are viewed through Fabric Manager Web Server. To view problems and events in Fabric Manager Web Server, choose a fabric and then click the Events tab to see a summary or detailed report of the problems and events that have occurred in the selected fabric.

Generating Top10 Reports in Performance Manager

You can generate historical Top10 reports that can be saved for later review. These reports list the entities from the data collection, with the most active entities appearing first. This is a static, one-time only report that generates averages and graphs of the data collection as a snapshot at the time the report is generated.

---

Tip Name the reports with a timestamp so that you can easily find the report for a given day or week.

---

These Top10 reports differ from the other monitoring tables and graphs in Performance Manager in that the other data is continuously monitored and is sortable on any table column. The Top10 reports are a snapshot view at the time the report was generated.

---

Note Top10 reports require analyzing the existing data over an extended period of time and can take hours or more to generate on large fabrics.

---

See the "Creating a Custom Report Template" section on page 7-33 for information on creating a Top10 report.

Generating Top10 Reports Using Scripts

You can generate Top10 reports manually by issuing the following commands:

•On UNIX, run the script:

"/<user_directory>/.cisco_mds9000/bin/pm.sh display pm/pm.xml <output_directory>"

•On Windows, run the script:

"c:\Program Files\Cisco Systems\MDS 9000\bin\pm.bat display pm\pm.xml 
<output_directory>"

On UNIX, you can automate the generation of the Top10 reports on your Fabric Manager Server host by adding the following cron entry to generate the reports once an hour:

0 * * * * /<user_directory>/.cisco_mds9000/bin/pm.sh display pm/pm.xml <output_directory>

If your crontab does not run automatically or Java complains about an exception similar to Example 60-1, you need to add "-Djava.awt.headless=true" to the JVMARGS command in /<user_directory>/.cisco_mds9000/bin/pm.sh.

Example 60-1 Example Java Exception

in thread "main" java.lang.InternalError Can't connect to X11 window server using '0.0' as 
the value of the DISPLAY variable.

Exporting Data Collections to XML Files

The RRD files used by Performance Manager can be exported to a freeware tool called rrdtool. The rrd files are located in pm/db on the Fabric Manager Server. To export the collection to an XML file, enter the following command at the operating system command-line prompt:

/bin/pm.bat xport xxx yyy

In this command, xxx is the RRD file and yyy is the XML file that is generated. This XML file is in a format that rrdtool is capable of reading with the command:

rrdtool restore filename.xml filename.rrd

You can import an XML file with the command:

bin/pm.bat pm restore <xmlFile> <rrdFile>

This reads the XML export format that rrdtool is capable of writing with the command:

rrdtool xport filename.xml filename.rrd.

The pm xport and pm restore commands can be found on your Fabric Manager Server at bin\PM.bat for Windows platforms or bin/PM.sh on UNIX platforms. For more information on the rrdtool, refer to the following website: http://www.rrdtool.org.

Exporting Data Collections in Readable Format

Cisco MDS SAN-OS Release 2.1(1a) introduces the ability to export data collections in comma-separated format (CSV). This format can be imported to various tools, including Microsoft Excel. You can export these readable data collections either from the Fabric Manager Web Server menus or in batch mode from the command line on Windows or UNIX. Using Fabric Manager Web Server, you can export one file. Using batch mode, you can export all collections in the pm.xml file.

---

Note Fabric Manager Web Server must be running for this to work. See the "Launching Fabric Manager Web Server" section on page 7-7.

---

To export data collections to Microsoft Excel using Fabric Manager Web Server, follow these steps:

---

Step 1 Click the Performance tab on the main page.

You see the overview table.

Step 2 Click the Flows sub-tab.

Step 3 Right-click the name of the entity you want to export and select Export to Microsoft Excel.

You see the Excel chart for that entity in a pop-up window.

---

To export data collections using command-line batch mode, follow these steps:

---

Step 1 Go to the installation directory on your workstation and then go to the bin directory.

Step 2 On Windows, enter .\pm.bat export C:\Program Files\Cisco Systems\MDS 9000\pm\pm.xml <export directory>. This creates the csv file (export.csv) in the export directory on your workstation.

Step 3 On UNIX, enter ./pm.sh export /usr/local/cisco_mds9000/pm/pm.xml <export directory>. This creates the csv file (export.csv) in the export directory on your workstation.

---

When you open this exported file in Microsoft Excel, the following information displays:

•Title of the entity you exported and the address of the switch the information came from.

•The maximum speed seen on the link to or from this entity.

•The VSAN ID and maximum speed.

•The timestamp, followed by the receive and transmit data rates in bytes per second.

Configuring Performance Manager for Use with Cisco Traffic Analyzer

Performance Manager works in conjunction with the Cisco Traffic Analyzer to allow you to monitor and manage the traffic on your fabric. Using Cisco Traffic Analyzer with Performance Manager requires the following components:

•A configured Fibre Channel Switched Port Analyzer (SPAN) destination (SD) port to forward Fibre Channel traffic.

•A Port Analyzer Adapter 2 (PAA-2) to convert the Fibre Channel traffic to Ethernet traffic.

•Cisco Traffic Analyzer software to analyze the traffic from the PAA-2.

To configure Performance Manager to work with the Cisco Traffic Analyzer, follow these steps:

---

Step 1 Set up the Cisco Traffic Analyzer according to the instructions in the Cisco MDS 9000 Family Port Analyzer Adapter 2 Installation and Configuration Note.

Step 2 Get the following three items of information:

•The IP address of the management workstation on which you are running Performance Manager and Cisco Traffic Analyzer.

•The path to the directory where Cisco Traffic Analyzer is installed.

•The port that is used by Cisco Traffic Analyzer (the default is 3000).

Step 3 Start the Cisco Traffic Analyzer.

a. Choose Performance > Traffic Analyzer > Open.

b. Enter the URL for the Cisco Traffic Analyzer, in the format

http://<ip address>:<port number>

where:

ip address is the address of the management workstation on which you have installed the Cisco Traffic Analyzer, and

:port number is the port that is used by Cisco Traffic Analyzer (the default is :3000).

c. Click OK.

d. Choose Performance > Traffic Analyzer > Start.

e. Enter the location of the Cisco Traffic Analyzer, in the format

D:\<directory>\ntop.bat

where:

D: is the drive letter for the disk drive where the Cisco Traffic Analyzer is installed, and

directory is the directory containing the ntop.bat file.

f. Click OK.

Step 4 Create the flows you want Performance Manager to monitor, using the Flow Configuration Wizard. See the "Creating a Flow with Performance Manager" section

Step 5 Define the data collection you want Performance Manager to gather, using the Performance Manager Configuration Wizard. See the "Creating a Collection with Performance Manager" section.

a. Choose the VSAN you want to collect information for or choose All VSANs.

b. Check the types of items you want to collect information for (Hosts, ISLs, Storage Devices, and Flows).

c. Enter the URL for the Cisco Traffic Analyzer in the format

http://<ip address>/<directory>

where:

ip address is the address of the management workstation on which you have installed the Cisco Traffic Analyzer, and directory is the path to the directory where the Cisco Traffic Analyzer is installed.

d. Click Next.

e. Review the data collection on this and the next section to make sure this is the data you want to collect.

f. Click Finish to begin collecting data.

---

Note Data is not collected for JBOD or for virtual ports. If you change the data collection configuration parameters during a data collection, you must stop and restart the collection process for your changes to take effect.

---

Step 6 Choose Performance > Reports to generate a report. Performance Manager Web Server must be running. See the "Launching Fabric Manager Web Server" section on page 7-7. You see Web Services; click Custom then select a report template.

---

Note It takes at least five minutes to start collecting data for a report. Do not attempt to generate a report in Performance Manager during the first five minutes of collection.

---

Step 7 Click the Cisco Traffic Analyzer at the top of the Host or Storage detail pages to view the Cisco Traffic Analyzer information, or choose Performance > Traffic Analyzer > Open. The Cisco Traffic Analyzer page will not open unless ntop has been started already.

---

Note For information on capturing a SPAN session and starting a Cisco Traffic Analyzer session to view it, refer to the Cisco MDS 9000 Family Port Analyzer Adapter 2 Installation and Configuration Note.

---

---

Note For information on viewing and interpreting your Performance Manager data, see the "Historical Performance Monitoring" section.

For information on viewing and interpreting your Cisco Traffic Analyzer data, refer to the Cisco MDS 9000 Family Port Analyzer Adapter 2 Installation and Configuration Note.

---

---

For performance drill-down, Fabric Manager Server can launch the Cisco Traffic Analyzer in-context from the Performance Manager graphs. The aliases associated with hosts, storage devices, and VSANs are passed to the Cisco Traffic Analyzer to provide consistent, easy identification.