- Preface
- Product Overview
- Network Design
- Getting Started
- Configuring the VDS
- System Monitoring
- System Reporting
- System Maintenance
- Troubleshooting
- Creating Bulk Configuration Files
- BMS Communication
- SNMP MIB and Trap Information
- Using the TV VDS Streamer Application Monitoring Tool
- Engineering Access Level Pages
- CDDM Management Utility
- Software Licensing Information
- VDS-TV Tunables
- Installation, Upgrade, and Downgrade via YUM
- Configuring for Port Resiliency
Cisco VDS-VR Software 4.1 Configuration Guide
- Updated:
- December 14, 2015
Chapter: System Monitoring
System Monitoring
The CDSM provides tools that can be used for system monitoring and system diagnostics. The topics covered in this chapter include:
- System Level Monitoring
- Monitoring Content Objects
- Monitoring Stream Objects
- Array Level Monitoring
- Server Level Monitoring
- Recommended Monitoring Schedule
Note
If Virtual Video Infrastructure (VVI) with split-domain management is enabled, the CDSM pages associated with the Vaults and Caching Nodes display only on the VVI Manager (VVIM), and the CDSM pages associated with the Streamers display only on the Stream Manager. For more information, see Virtual Video Infrastructure.
System Level Monitoring
The System Level Monitoring pages provide an overall view of the health and activity of the VDS. The System Level links are:
To view the System Level Monitoring pages, click Monitor from any page in the CDSM, and then click System Health or System Snapshot, as appropriate.
Alarms Table
Any time there is an alarmed event, an alarm is displayed in the CDSM banner. The Alarms table is displayed when you roll your mouse over the alarm icon. See Figure 5-1. Clicking the alarmed event in the Alarm table takes you to the CDSM page that has more information. For example, in Figure 5-1, clicking System health problems reported takes you to the System Health page.
Figure 5-1 CDSM Banner—System Health Alarm
The following errors and situations are monitored and registered in the Alarms table if found, and linked to the System Cleanup page:
- Orphaned server IDs
- Multiple or duplicate Cache Locate IP addresses
- Out of range Group IDs
- ServerMap and StatMap inconsistencies
- Extra or incorrect SERVERMAP15 entries
See System Cleanup for more information.
In addition to the System Cleanup page links, the following situations are monitored and registered in the Alarms table:
- System clock is out of synchronization
- MSA events exist for the current CDSM day
- Incorrect IDs on the Stream Manager (for ISA environments only)
- Missing or incorrect initial IDs (Group, Server, and Setup)
- Primary VDS server (Vault, Setup, Control) fails over to another VDS server.
- Primary CDSM fails and secondary CDSM becomes the primary
- Primary and secondary CDSM port failures
- Primary and secondary CDSM Linux file system threshold exceeded
If a VDS server system clock is off from that of the CDSM (VVIM or Stream Manager) by more than two minutes, an alert is added to the Alarms table. Clicking the alert takes you to the Configure > System Level > System NTP Server page.
If MSA events exist (System Failures) for the current CDSM day, an alert is added to the Alarms table. Clicking the alert takes you to the Monitor > System Level > System Failures page.
Incorrect IDs on Stream Manager in ISA environment
If VVI is enabled in an ISA environment and you logged in to the Stream Manager, and if there are errors associated with the IDs; an alert is added to the Alarms table. Clicking the alert takes you to the Maintain > Software > ID Manager page.
Missing or Incorrect Initial IDs
If the CDSM is initially configured incorrectly as a legacy VDS or VVI with central management, then reconfigured or reinstalled for a VVIM or Stream Manager, the starting IDs for group IDs, server IDs, and setup IDs need to be changed from the old system to the new system. An alert is added to the Alarms table to inform you of the situation. Clicking the alert takes you to the Maintain > Software ID Management page to correct the situation.
System Health
The System Health page provides a top-level view of the overall health of each group in the VDS and each server in each group.
To view the System Health page, choose Monitor > System Health. See Figure 5-2.
The colored boxes for each group on the System Health Monitor page have the following meaning:
- Green—All servers in the group are operating.
- Yellow—One or more servers are not operational, but have not reached any thresholds.
- Red—One or more servers are not operational and have reached a threshold.
To view the VDS servers for each group, click the plus sign (+) for the group. The VDS servers for the group are displayed, along with the IP address or host name, software version, and server role of each VDS server. If a server is currently acting as one of the following server roles, an icon is displayed next to that server indicating the specific role:
The Server Role legend is displayed in the left-panel.
The colored boxes for each server on the System Health Monitor page have the following meaning:
- Green—All components are operating.
- Yellow—Some components are not operational, but have not reached a threshold.
- Red—Some components are not operational and have reached a threshold.
The servers can have the following states:
- Online —Server is operational.
- Down—Server is down or database is down.
- No Ingest—Vault is offline for ingest (still accepting cache-fill traffic)
- Offline—Vault is offline for all traffic (ingest and cache-fill), Streamer or Caching Node is offline.
You can view the details of a monitored area of a server by clicking the box in the appropriate column.
- When you click the Network check box, you are taken to the NIC Monitor page. See NIC Monitor for more information.
- When you click the Disk check box, you are taken to the Disk Monitor page. See Disk Monitor for more information.
- When you click the Services check box, you are taken to the Services Monitor page. See Services Monitor for more information.
- When you click the Vitals check box, you are taken to the Server Vitals Monitor page. See Server Vitals for more information.
Note The Vitals column is displayed only if the CDSM Health Monitor feature is enabled. For more information, see CDSM or VVIM Health Monitoring.
The time shown at the bottom of the left-panel menu is not the current time, but rather the CDSM time that is used for the health status and monitoring the system.
System Snapshot
The System Snapshot page provides an overview of the current activity on the VDS. A summary of the state of all streams, content ingests, and disk usage is displayed. See Figure 5-3.
Figure 5-3 System Snapshot Page
In a VVI, the Stream Manager only displays the stream-related date and the VVIM only displays the bandwidth and content-related data.
Table 5-1 describes the information displayed on the System Snapshot page.
|
|
|
|---|---|
How often the information is refreshed. The default is 30 seconds. The range is 10 to 300. All field values that are updated, based on the refresh rate, are initially shown in a green colored font. |
|
Total number of stream objects the VDS is currently streaming. |
|
Total number of high-definition stream objects the VDS is currently streaming. |
|
Total number of standard-definition stream objects the VDS is currently streaming. |
|
Active Stream Bandwidth1 |
|
Active Fill Bandwidth 1 |
Total bandwidth, in megabits, used for caching content among Vaults and Streamers. |
Total number of content objects currently stored, ingested, provisioned for ingest, and failed ingest on the VDS. |
|
Total number of content objects currently stored on the VDS. |
|
Total number of content objects currently being ingested on the VDS. |
|
Total number of content objects that have been requested for ingestion, but have not yet begun active ingestion. |
|
Total number of content objects that have been created but do not yet contain any information (in other words, they are blank shells). |
|
Total number of content objects that failed to complete the ingest process. |
|
Total number of content objects that are in a Temp Out of Service state. The backoffice may put a content object into this state for a certain amount of time. |
|
|
1.Active Stream Bandwidth and Active Fill Bandwidth values are only accurate if the clocks on the VDS servers are synchronized with the CDSM. |
System Failures
To view the details of system failures, do the following:
Step 1 Choose Monitor > System Level > System Failures. The System Failures page is displayed.
Each system failure is listed by date and time, followed by the session ID.
Note Stream Failure monitoring displays only the system failures for the current day. To view past system failures, see System Failures.
Step 2 From the System Failures drop-down list, choose the time stamp and session ID of the stream object and click Display. The system failure details are displayed. See Figure 5-4.
To delete a system failure, display the object and click Delete.
Figure 5-4 System Failures Page
Table 5-2 describes the stream failure details.
|
|
|
|
|---|---|---|
| Server responsible for streaming this stream object. To view the IP address associated with the Server ID, see Configuring the Servers. |
||
| All servers that are part of the same VDS system (managed by one CDSM) have the same Group ID. This Group ID corresponds to the CDSM GUI array ID and should be unique across an enterprise. Table 5-3 describes the ID mapping between the CDSM GUI and the CServer. |
||
| Operation that was taking place when the stream failed; for example, createStream, LSCP Command(), or createServant, destroy. These are the measurement points or transactional states of the system at the time of the failure. |
||
| Failed task is the event category that provides the type of execution sequence that the call stack was currently in at the time of the failure. The following list provides examples of the high-level categories: |
||
| Error code provides a description of the event that caused an error. See Table 5-4 , Table 5-5 , and Table 5-7 for descriptions of the error codes. |
||
Table 5-3 lists the CDSM GUI ID names and maps them to the CServer names in the setupfile and.arroyorc files.
|
|
|
|---|---|
Table 5-4 lists the Managed Services Architecture (MSA) error codes. Some MSA monitored events are monitored for the VDS as well, and are prefaced by “AVS_” instead of “MSA_.” They are denoted with a footnote in the table. Some MSA monitored events are not errors, but rather information about an event. Not all MSA events trigger an SNMP trap.
|
|
|
|
|---|---|---|
MSA_INT_ERR2 |
There is an internal error. INT_ERR has a subset of error codes that specifically describe where the error occurred. See Table 5-5 . |
|
MSA_FLOW 1 |
Entry or exit of a measured or tracked flow, or some other important check point, and is recorded as non-realtime. |
|
MSA_CMPT_NOT_EXIST 1 |
||
MSA_REQ_TIMEOUT 1 |
||
MSA_CMPT_OUT_OF_SVC 1 |
||
MSA_REQ_NOT_IMPL 1 |
||
MSA_RES_INVALID 1 |
||
MSA_RES_DUP 1 |
||
It was determined that a component that was being added has the same name as a previously created component. |
||
Component is destroyed outside of the normal expected flows. |
||
Video server timed out waiting for the client to issue a resume or play command after the initial creation of the stream. |
||
Resource is currently out of capacity and cannot satisfy the request. |
||
Entry or exit of a measured or tracked flow, and is recorded in real-time. |
||
LSC response; client does not have permission for the request. |
||
|
2.This event is monitored by the VDS as well as MSA, and is displayed with the prefix “AVS_” instead of “MSA_.” |
Table 5-5 lists the error codes for internal errors and external errors. Internal errors are errors that occurred in the VDS and specifically describe where the error occurred. External errors are errors that occurred in the network or network components, which includes the ContentStore, StreamService, and so on. The error codes listed in Table 5-5 provide more detail to the MSA_INT_ERR or AVS_INT_ERR error code.
Table 5-6 lists the error codes for errors that could occur during ingest or during trick-mode file creation, which cause system failures.
Table 5-7 lists the Managed Services Architecture (MSA) error codes for the optional Ingest Manager feature.
Table 5-8 lists the internal cserver error codes sent by Recorder to Recorder Manager (RM) through the reasonCode and reasonText fields in the RecordingStatus message.
Monitoring Content Objects
The content objects links on the Monitor System Level page provides information on the status of content ingests. The following different ingest states are monitored:
In a VVI with split-domain management, the Stream Manager displays the following completed ingest details: Content Name, File Size, Rate, Create Time, and Last Modified time of the ingested content. For the other completed ingests fields, see the same content asset on the VVIM.
Ingests
Viewing Completed Ingests is a different procedure than viewing the other types of monitored ingests. This section contains the following topics:
Viewing and Deleting Completed Ingests
To view the details of completed ingests, do the following:
Step 1 Choose Monitor > System Level > Completed Ingests.
Step 2 The following methods can be used to display a list of content objects:
- Enter the first character of the content object name in the text box. A drop-down list of content objects is displayed. If there are more than 25 content objects that start with that first character you entered, you are prompted to continue entering the next character of the content object name or click Display. You can continue to enter characters to reduce the list (you can also delete characters to increase the list) and at any point click Display. After you click Display, a list of content objects is displayed that has the same beginning characters that you entered in the text box.
- In the Browse Content box, click one of the characters. A list of content objects that begin with that character is displayed.
- In the Quick Lists box, the following options are offered:
– Most Recent Ingests (max 100) —Lists the 100 most recent completed ingests sorted by ingest date.
– List All Contents —Lists all completed ingests sorted by content name. This option is available only if the number of completed ingests is less than 100.
– Content Status (Damaged Only) —Lists status information only for damaged completed ingests.
After you perform one of these methods, a list is displayed. The list of content objects can span several pages. To view the next page, click the page number.
Figure 5-5 shows an example of the Completed Ingests list generated with any of the methods, except the Content Status (Damaged Only). The content name, file size, duration, and date the object was ingested are displayed.
Figure 5-5 Completed Ingests List
Figure 5-6 shows an example of the Content Status information that displays when you choose Content Status (Damaged Only).
Figure 5-6 Completed Ingests List—Content Status (Damaged Only)
Step 3 To view the details of a content object, click the content name. The Ingest Details are displayed (Figure 5-7).
Figure 5-7 Completed Ingests—Ingest Details
Table 5-10 describes the content object details that are displayed for each type of ingest.
To delete the completed ingest, click Delete.
Viewing Other Ingests
To view the details of active, provisioned, unprovisioned, and failed ingests, do the following:
Step 1 Choose Monitor > System Level from any page in the CDSM, and then click the link for the type of content object you want to view:
Step 2 Choose a content object from the drop-down list and click Display. The details of the content object are displayed. Figure 5-7 shows an example of the ingest details.
By typing the first character of the content object name, you can jump to that section of the list.
In addition, you can perform a text string search by typing the text string you want to search for in the Search Ingests field and clicking Search. A list of content objects that contain the text string are listed. To see the content object details, click the content object name listed. To return to the previous page without selecting a content object, click Back.
Table 5-10 describes the content object details that are displayed for each type of ingest.
Note The Unprovisioned Ingests page displays only the content name and the date the content object was considered unprovisioned.
Package Expirations
Note Package Expirations are part of the optional Ingest Manager feature. This option is listed only on the Monitoring System Level left-panel menu if the Ingest Manager is enabled in your deployment.
To view the details or adjust the license expiration of a package expiration, do the following:
Step 1 Choose Monitor > System Level > Package Expiration. The Package Expiration page is displayed.
Step 2 In the Available Packages field, begin typing the package name, then select the package name from the list provided.
Figure 5-8 shows an example of the Package Expiration details.
Figure 5-8 Package Expiration Page
Table 5-11 describes the package expiration details that are displayed.
|
|
|
|---|---|
Actual Package Expiration is the License Expiration plus the Additional Package Window in the format MM-DD-YYYY |
Step 4 To adjust the license expiration date, enter the number of days (positive or negative) in the Adjust License Expiration field and click Update.
The license expiration is adjusted by the number of days you entered. The Additional Package Window is not affected and is still applied to create the Actual Package Expiration.
Step 5 In the Asset Details section, to view the metadata associated with the package expiration, click the details hyperlink next to the metadata you want to view.
To delete a package, choose the package from the Available Packages list by typing the characters of the package name, click Display, and then click Delete in the Package Expiration Details section.
Publish Failures
Note Publish Failures are part of the optional Ingest Manager feature. This option is listed only on the Monitoring System Level left-panel menu if the Ingest Manager is included in your deployment.
The Publish Failures page lists the packages that were not able to be published to the backoffice and provides a mechanism to republish the package.
To publish an unpublished package, or delete an unpublished package, do the following:
Step 1 Choose Monitor > System Level > Publish Failures. The Publish Failures page is displayed.
Step 2 From the Unpublished Packages drop-down list, choose a package and click Display. The Unpublished Package details are displayed.
By typing the first character of the package name, you can jump to that section of the list.
Figure 5-9 shows an example of the Publish Failures details.
Figure 5-9 Publish Failures Page
Table 5-12 describes the publish failures details that are displayed.
|
|
|
|---|---|
Internal identifier for what failed. In the example in Figure 5-9, the publishing of the package failed. |
|
Step 3 In the Assets section, to view the metadata associated with the unpublished package, click the plus sign (+) next to the metadata you want to view.
Step 4 To restart the publishing process and set the creation date to today, click Publish.
Package Publish Status
The Package Publish Status page lists the publishing status of a particular package in the corresponding VOD Market Site(s).
To view the publishing status of a package in the VOD Market Site(s), do the following:
Step 1 Choose Monitor > System Level > Package Publish Status. The Package Publish Status page is displayed.
Step 2 From the Available Packages drop-down list, choose a package and from View All drop-down list choose a corresponding publishing status and click Display. The Package Publish Status details are displayed along with the correponding VOD Market Site(s).
Table 5-13 describes the publish failures details that are displayed.
|
|
|
|---|---|
The package publish status is displayed with a colored icon. The publishing status corresponding to the colored icon is specified below: |
Unpublished Packages
The Unpublished Packages page provides a list of all unpublished packages and their corresponding VOD Market Site(s) and the retry counts.
To view a list of all unpublished packages, do the following:
Step 1 Choose Monitor > System Level > Unpublished Packages. The Unpublished Packages page is displayed.
Step 2 The Unpublished Packages page lists all unpublished packages and their corresponding VOD Market Site(s) and the retry counts. To refresh the list, click Refresh.
Monitoring Stream Objects
The monitored stream objects consist of the Stream Monitor page.
Stream Monitor
To view the details of stream objects, do the following:
Step 1 Choose Monitor > System Level > Stream Monitor. The Stream Monitor page is displayed.
Step 2 The following methods can be used to display a stream object or a list of stream objects:
- Enter the first character of the session ID in the text box. A drop-down list of stream objects is displayed. If there are more than 25 objects that start with that first character you entered, you are prompted to continue entering the next character of the object name or choose one that is listed. You can continue to enter characters to reduce the list (you can also delete characters to increase the list) and at any point choose one. After you choose one, the stream details are displayed (Figure 5-11).
- In the Quick Lists box, the following options are offered:
– Most Recent 100 —Lists the 100 most recent stream objects.
– All Streams —Lists all streams. This option is available only if the number of streams is less than 100.
– Search by Specific Field —You can perform a text string search by selecting the field you want to search on, entering the complete text string (for example, the full IP address) in the Search field, and clicking Search. A list of stream objects that match the text string in the field you selected are listed. Click the linked field (session ID as seen in Figure 5-10) to see the stream monitor details for the selected object, or click Back to return to the previous page.
After you perform one of the Quick List methods, a list is displayed. The list of stream objects can span several pages. To view the next page, click the page number.
The stream object list is generated by entering a specific value for the selected field (for example, Destination IP) in the Quick List box. The stream object list displays the session ID, stream start time, Destination IP and port (or subnet address if Stream Destination is enabled), service group, and the TSID in and out if applicable.
Figure 5-10 shows an example of the stream object list generated by clicking Most Recent 100 or All Streams. The Session handles and the stream start time are displayed.
Figure 5-10 Stream Monitor—Stream List
Step 3 To view the details of a stream object, click the session ID. The Stream Details are displayed.
Figure 5-11 shows an example of the stream object details.
Figure 5-11 Stream Monitor—Stream Details
Table 5-14 describes the stream details.
The Graph Stream button displays the trick-mode activity of the stream (Figure 5-12).
Note If Trick Mode Capture is disabled, the Graph Stream is not displayed. For information on enabling the Trick Mode Capture, see Trick Mode Capture.
Figure 5-12 Stream Activity Report
To delete a stream object, display the object and click Delete.
Array Level Monitoring
The Array Level Monitoring pages provide an overall view of the health and activity of an specified array, monitoring and deletion of barker streams, and a display of the Playout Schedule if the Playout Scheduler is enabled. The Array Level links are:
Array Snapshot
The Array Snapshot page provides an overview of the current activity for the specified array of servers on the VDS. A summary of the state of all streams, content ingests, and disk usage is displayed.
The fields displayed on the Array Snapshot page are the same fields that are displayed on the System Snapshot page, with the active stream bandwidth and active fill bandwidth shown for each Stream Group. For descriptions of the fields, see Table 5-1.
Barker Stream Monitor
Note The Barker Stream feature is optional and is not listed on the Array Level left-panel menu if it is not included in your deployment. The Barker Stream feature is also not available if the Stream Destination is set to IPTV. For more information, see Stream Destination.
The Barker Stream Monitor page lists the barker streams currently configured. Figure 5-13 shows an example of barker streams.
To delete a barker stream configuration, click Delete.
Figure 5-13 Barker Stream Monitor Page
Playout Monitor
The Playout Monitor page displays the Playout Schedule that is currently configured for the selected day and channels.
Note The Playout Monitor page is part of the TV Playout feature and is displayed only if TV Playout feature is enabled. For more information, see Playout Scheduler.
To view the Playout Schedule, do the following:
Step 1 Choose Monitor > Array Level > Playout Monitor. The Playout Monitor page displays the calendar. (Figure 5-14).
Figure 5-14 Playout Monitor Page—Calendar
Step 2 To view the days that have scheduled content for a channel, from the Channel drop-down list, select a channel. The days that have been scheduled for the selected channel are highlighted in the calendar.
For example, in Figure 5-14, CHAN-31 has been selected and October 10, 11, and 12 are highlighted, indicating those days have been scheduled content for CHAN-31.
Step 3 From the calendar, click the day you want to schedule. If the month you are scheduling is not shown, use the left and right arrows on either side of the calendar to change the month.
Note Today’s date is displayed with a box around it.
If you selected a channel from the Channel drop-down list, then only that channel is displayed in the Playout Monitor.
The schedule for the day you selected is displayed.
Note The Playout Monitor page displays the delivery service mode for the Playout Scheduler application. To change the delivery service mode (active-active or active-standby), see Configuring the TV Playout Application.
The timeslots have different colors depending on the status of the scheduled content and the type of content. The Playout Scheduler page displays a legend describing the different colors for the timeslots in the schedule.
Small timeslots are marked blue. To view the program information on small timeslots, click the timeslot. The page refreshes and the schedule for the small timeslot is displayed at the bottom of the page.
Server Level Monitoring
The Server Level Monitoring pages provide detail information on the health and activity of a VDS server or the CDSM (or VVIM).
To view the Server Level Monitoring pages, do the following:
Step 1 Choose Monitor > Server Level, and then click one of the following as applicable:
Step 2 Choose the IP address of the server from the drop-down list and click Display.
Disk Monitor
The Disk Monitor page provides real-time information on the status of a disk.
To view the current status of a disk, choose the IP address of the server from the drop-down list on the Disk page, click Display, and roll your mouse over one of the disks displayed in the graphic. If the server is a Lindenhurst CDE, click one of the disks displayed in the graphic. Figure 5-15 shows an example of a Streamer server.
To change how often the information is refreshed, enter the number of seconds in the Data Refresh Rate field. The default is 30 seconds. The range is 10 to 300. All field values that are updated, based on the refresh rate, are initially shown in a green font.
Figure 5-15 Disk Monitor Page—Streamer
For Vault servers, the Disk Availability line graph shows the percentage of disk space available. The gigabytes displayed for “Total Space,” “Available Space,” and “% Used” are the sum of all the disks installed on the Vault server.
The Linux File System Stats table shows the combined total storage space for the partitions of the disk drives, the combined available storage space for the partitions of the disk drives, and the percentage of used storage for each combined partition. In Figure 5-15, the hda2 partition has an alarm indicator because the usage has exceeded the user-defined threshold of 40 percent. There is also an alarm icon for partitions that have changed to read-only. For information on setting thresholds, see Setting System Thresholds.
Table 5-15 describes the information displayed when a disk is selected.
|
|
|
|---|---|
The Self-Monitoring, Analysis and Reporting Technology (S.M.A.R.T.) will display "Suspended" for a suspended drive. |
|
Number of bytes written to the disk since it was powered on. |
S.M.A.R.T
The VDS incorporates S.M.A.R.T. to monitor the reliability of a hard drive, predict drive failures, and to carry out different types of drive self-tests. S.M.A.R.T is firmware, native to most disk drives, that monitors disk attributes over time, making it possible to perform predictive failure analysis. Advanced warning of predictive failures allows the operator to perform preventative maintenance.
To view the current read/write activity that has occurred in the last five seconds on the selected disk, click Graph Disks. Figure 5-16 shows an example of the Disk Activity graph.
Figure 5-16 Disk Monitor—Streamer Disk Activity Graph
The Disk Activity graph displays an average calculation of the amount of data read (in megabytes per second) and data writes over a five-second period.
NIC Monitor
The NIC Monitor page displays the status of each network interface card (NIC) on the server.
To view the current status of a NIC on a server, choose the IP address of the server from the drop-down list on the NIC Monitor page, click Display, and click one of the NIC ports displayed in the graphic. Figure 5-17 shows an example of the eth4 interface on a Streamer server.
To change how often the information is refreshed, enter the number of seconds in the Data Refresh Rate field. The default is 10 seconds for a VDS server and 30 seconds for a CDSM (or VVIM). The range is 10 to 300. All field values that are updated, based on the refresh rate, are initially shown in a green font.
Figure 5-17 NIC Monitor Page—Streamer
Table 5-16 describes the information displayed for a NIC port.
To view the average transmit and receive activity that has occurred in the last two seconds for each port on this server, click Graph Ports. Figure 5-18 shows an example of the Port Activity graph.
Figure 5-18 NIC Monitor—Streamer Port Activity Graph
Server Vitals
The Server Vitals page provides current values for monitored system components. Server components are monitored and if a threshold is exceeded, the System Health Monitor page reports the event and an SNMP trap is sent to the Network Management System (NMS).
Note The Server Vitals page is displayed only if the CDSM Health Monitor feature is enabled. For more information, see CDSM or VVIM Health Monitoring.
To view the current values of the monitored components, as well as the threshold settings, choose the IP address of the server from the drop-down list on the Server Vitals page and click Display.
To change how often the information is refreshed, enter the number of seconds in the Data Refresh Rate field. The default is 10 seconds for a VDS server and 30 seconds for a CDSM (or VVIM). The range is 10 to 300. All field values that are updated, based on the refresh rate, are initially shown in a green colored font.
To change the temperature format to Fahrenheit, choose ºF for the Temperature Format. The default is Celsius (ºC).
The monitored components are different for each Content Delivery Engine (CDE) model. Figure 5-19 shows an example of the Server Vitals page for a Vault (CDE420).
Figure 5-19 Server Vitals Page
Cache/Fill Bandwidth
The Cache/Fill Bandwidth page displays details on the content caching activity on a Streamer.
To view the caching activity on a server, choose the IP address of the server from the drop-down list on the Cache/Fill Bandwidth page and click Display. Figure 5-20 shows an example.
To change how often the information is refreshed, enter the number of seconds in the Data Refresh Rate field. The default is 10 seconds for a VDS server and 30 seconds for a CDSM (or VVIM). The range is 10 to 300. All field values that are updated, based on the refresh rate, are initially shown in a green font.
Figure 5-20 Cache/Fill Bandwidth Page
Table 5-17 describes the services listed in the Cache/Fill Bandwidth page.
Note The values in the Cache/Fill Bandwidth page are only accurate if the clocks on the VDS servers are synchronized with the CDSM. If the clocks on the VDS servers are out of sync with the CDSM by more than two minutes, no values are displayed.
Services Monitor
The Services Monitor page displays whether specific processes are running on a server.
To view the current status of the services running on a server, choose the IP address of the server from the drop-down list on the Services Monitor page and click Display. Figure 5-21 shows an example of a Streamer server.
Figure 5-21 Services Monitor Page—Streamer
Table 5-18 describes the services listed on the Services Monitor page. All services described in Table 5-18 may not be listed on the Services Monitor page. The services listed are determined by the type of VDS deployment.
Recommended Monitoring Schedule
This monitoring schedule is recommended to ensure that the VDS is functioning as expected and identify potential issues that may cause down time.
Do not attempt to access the Linux command line unless you are familiar with the VDS, the Linux operating system, and the Linux command line.
Note Some error warnings in the logs are only informational and no action is necessary.
Daily Tasks
The following tasks should be performed daily:
- Choose Monitor > System Level > System Health and check the System Health Monitor page for red or yellow states on any of the servers. Click any red or yellow boxes to see detail information on disk, NIC, or services. See System Health for more information.
- Choose Monitor > System Level > Failed Ingests to check for any failed ingests. See Ingests for more information.
- Choose Monitor > System Level > System Failures to check for any system failures. See System Failures for more information. It is also possible to run a report for the previous day. See System Failures for more information.
Weekly Tasks
The following tasks should be performed weekly:
- Monitoring Tasks for Streamers and Vaults
- Monitoring Tasks for Vaults
- Monitoring Tasks for Streamers
Note All commands require that you log into each Linux operating system as root. Some tasks have a CDSM option.
Monitoring Tasks for Streamers and Vaults
To monitor the Streamer and Vaults weekly, do the following:
Step 1 Recover used disk space. Log in to each server as the root user and run the following command:
If the disk usage is greater than 75 percent, recover the disk space using the following methods:
a. Search and remove any core files.
b. Copy any archived logs to an external device and delete them from the /arroyo/archive directory.
c. Check for the presence of old install or upgrade ISO files in the /root directory and delete them.
Step 2 Verify the services are running. Choose Monitor > Server Level > Services Monitor to check the services for each server, or log in to each server and run the following commands:
Step 3 Check the CServer interfaces to verify the status of the Ethernet adapters. Choose Monitor > Server Level NIC Monitor, or log in to each server and use the following commands
a. Use the grep -i Link command to verify that all adapters should have a status of “link up,” except those adapters that are not being used.
b. Use the grep -i Speed command to verify that each adapter that has a “link up” status should have a speed of 1000.
c. Use the grep -i State command to verify that all adapters should have an “up” state, except those adapters that are not being used.
Step 4 Check the CServer streaming and cache-fill interfaces using the following command:
Step 5 Check the database thread count using the following command:
Two connections for each Vault and Streamer should be listed with a status of “ESTABLISHED.”
Step 6 Check the protocol timing logs for errors or problems. Also, look at the protocol timing logs for packet retransmissions.
Step 7 Look for warning messages.
Note The “WARNING” messages can sometimes be misleading; for example, “datawait” and “slow disk” messages occur normally and do not indicate a problem.
Step 8 The number of GOIDs for a particular content object must be the same on all servers (Vaults and Streamers) that are supposed to have the content. The number of Vaults that must have the same number of GOIDs for a particular content object is determined by the mirrored copy configuration (see Configuring the Servers). The number of GOIDs is also dependent on the trick speeds you configured (see Configuring Ingest Tuning). If the GOID is different between a Vault and a Streamer, session setup is not created properly because of an issue of “no content available.” This is because there is no content on the Vault that matches the GOID the Streamer has.
Monitoring Tasks for Vaults
In addition to the weekly monitoring tasks for both the Vaults and Streamers, the Vaults can also be monitored in the following ways:
1. Check the available space on the Vault hard drives. Choose Monitor > Server Level > Disk Monitor. The disk availability is shown as a percentage and as a number of gigabytes. Alternatively, view the protocol timing logs by running the following command:
The number returned indicates the percentage of the disk space available on this server. If the number is 5 or lower, then steps need to be taken to increase storage space by adding more Vaults, replacing drives with higher capacity drives, or removing unused content.
2. Check the /home/isa/ContentStore/server/ContentStore.log for ingest errors on each Vault. The master Vault has an additional log.
Monitoring Tasks for Streamers
In addition to the weekly monitoring tasks for both the Vaults and Streamers, the Streamers can also be monitored in the following ways:
1. In an ISA environment, look for any errors in the /Streaming/lscp_server/LSCPService.log on the primary Control server, and /Streaming/master/StreamService.log on the primary Setup server.
Monthly Tasks
The monthly monitoring tasks consist of the following:
1. Choose Monitor > System Level > System Snapshot and check that the “Disk Available” amount meets the requirements for the expected movie storage in the next three to six months.
2. Run the reports for the last month that are suitable for your requirements and save them as comma-separated value (CSV) files.
3. Using the bandwidth and streaming reports, check that the VDS is not exceeding required usage per service area.
4. Run a quick security check.
a. Ensure that the CDSM changes can be attributed to individual users and not to a generic admin account.
b. Reset CDSM passwords if necessary.
c. Reset Linux passwords if necessary.
d. Check that access policies and firewalls are still enforced.
Other Tasks
If you have access to an anything on demand (XOD) application, do the following:
1. Check the inspect-live log for excessive errors.
2. Check the inspect-live log for excessive communication times with the BMS or VDS.
Feedback