Cisco TV CDS 2.4 RTSP Software Configuration Guide for the Cisco ISM (Integrated Service Module) Line Card
System Monitoring
Downloads: This chapterpdf (PDF - 0.99MB) The complete bookPDF (PDF - 9.24MB) | Feedback

System Monitoring

Table Of Contents

System Monitoring

System Level Monitoring

System Health

System Snapshot

Monitoring Content Objects

Ingests

Viewing Completed Ingests

Viewing Active Ingests

Package Expirations

Publish Failures

Monitoring Stream Objects

Stream Monitor

Stream Failures

Array Level Monitoring

Array Snapshot

Server Level Monitoring

Disk Monitor

S.M.A.R.T

NIC Monitor


Server Vitals

Cache/Fill Bandwidth

Services Monitor

Recommended Monitoring Schedule

Daily Tasks

Weekly Tasks

Monitoring Tasks for Streamers and Vaults

Monitoring Tasks for Vaults

Monitoring Tasks for Streamers

Monthly Tasks

Other Tasks


System Monitoring


The CDSM provides tools that can be used for system monitoring and system diagnostics. The topics covered in this chapter include:

System Level Monitoring

Monitoring Content Objects

Monitoring Stream Objects

Array Level Monitoring

Server Level Monitoring

Recommended Monitoring Schedule


Note If Virtual Video Infrastructure (VVI) with split-domain management is enabled, the CDSM pages associated with the Vaults and Caching Nodes display only on the VVI Manager (VVIM), and the CDSM pages associated with the Streamers display only on the Stream Manager. For more information, see the "Virtual Video Infrastructure" section on page D-6.


System Level Monitoring

The System Level Monitoring pages provide an overall view of the health and activity of the CDS. The System Level links are:

System Health

System Snapshot

To view the System Level Monitoring pages, click Monitor from any page in the CDSM, and then click System Health or System Snapshot, as appropriate.

System Health

The System Health page provides a top-level view of the overall health of each group in the CDS and each server in each group. Any time there is an alarmed event, an alarm is displayed in the CDSM banner. The Alarms menu is displayed when you roll your mouse over the alarm icon. See Figure 4-1. Clicking the alarmed event in the Alarm menu takes you to the CDSM page that has more information. For example, in Figure 4-1, clicking System health problems reported takes you to the System Health page.

Figure 4-1 CDSM Banner—System Health Alarm

To view the System Health page, choose Monitor > System Health. See Figure 4-2.

Figure 4-2 System Health Page

The colored boxes on the System Health Monitor page have the following meaning:

Green—All components are operating.

Yellow—Some components are not operational.

Red—All components have failed.

You can view the details of a monitored area of a server by clicking the box in the appropriate column.

When you click the Network check box you are taken to the NIC Monitor page. See the "NIC Monitor" section for more information.

When you click the Disk check box you are taken to the Disk Monitor page. See the "Disk Monitor" section for more information.

When you click the Services check box you are taken to the Services Monitor page. See the "Services Monitor" section for more information.

When you click the Vitals check box you are taken to the Server Vitals Monitor page. See the "Server Vitals" section for more information.


Note The Vitals column is displayed only if the CDSM Health Monitor feature is enabled. For more information, see the "CDSM or VVIM Health Monitoring" section on page D-9.


The time shown at the bottom of the left-panel menu is not the current time, but rather the CDSM time that is used for the health status and monitoring the system.

System Snapshot

The System Snapshot page provides an overview of the current activity on the CDS. A summary of the state of all streams, content ingests, and disk usage is displayed. See Figure 4-3.

Figure 4-3 System Snapshot Page

In a VVI, the Stream Manager only displays the stream-related date and the VVIM only displays the bandwidth and content-related data.

Table 4-1 describes the information displayed on the System Snapshot page.

Table 4-1 System Snapshot Page 

Field
Description

Data Refresh Rate

How often the information is refreshed. The default is 10 seconds. The range is 10 to 300. All field values that are updated, based on the refresh rate, are initially shown in a green colored font.

Total Streams

Total number of stream objects the CDS is currently streaming.

HD Streams

Total number of high-definition stream objects the CDS is currently streaming.

SD Streams

Total number of standard-definition stream objects the CDS is currently streaming.

Active Stream Bandwidth1

Total bandwidth, in megabits, used for active streams.

Active Fill Bandwidth1

Total bandwidth, in megabits, used for caching content among Vaults and Streamers.

Total Content

Total number of content objects currently stored, ingested, provisioned for ingest, and failed ingest on the CDS.

Active Ingests

Total number of content objects currently being ingested on the CDS.

Completed Ingests

Total number of content objects currently stored on the CDS.

Prov. (push) Ingests

Total number of content objects that have been requested for ingestion, but have not yet begun active ingestion.

Failed Ingests

Total number of content objects that failed to complete the ingest process.

Temp Out of Service

Total number of content objects that are in a Temp Out of Service state. The backoffice may put a content object into this state for a certain amount of time.

Total Disk

Total disk space, in bytes, on the CDS.

Disk Used

Total used disk space, in bytes, on the CDS.

Disk Available

Total available disk space, in bytes, on the CDS.

1 Active Stream Bandwidth and Active Fill Bandwidth values are only accurate if the clocks on the CDS servers are synchronized with the CDSM.


Monitoring Content Objects

The content objects links on the Monitor System Level page provides information on the status of content ingests. Table 4-2 describes the different ingest states that are monitored.

Table 4-2 Monitored Ingest States 

Completed Ingests

Lists content objects that have been fully ingested.

Active Ingests

Lists content objects that are in the process of being ingested.

Package Expiration

Lists Package Expiration information, including expiration dates and all associated metadata. Allows for package expiration adjustments.

Publish Failures

Lists the packages that were not able to be published to the backoffice and provides a mechanism to republish the package.


In a VVI with split-domain management, the Stream Manager displays the following completed ingest details: Content Name, File Size, Rate, Create Time, and Last Modified. For the other completed ingests fields, see the same content asset on the VVIM.

Ingests

Viewing Completed Ingests is a different procedure than viewing the other types of monitored ingests. This section contains the following topics:

Viewing Completed Ingests

Viewing Active Ingests

Viewing Completed Ingests

To view the details of completed ingests, do the following:


Step 1 Choose Monitor > System Level > Completed Ingests.

Step 2 The following methods can be used to display a list of content objects:

Enter the first character of the content object name in the text box. A drop-down list of content objects is displayed. If there are more than 25 content objects that start with that first character you entered, you are prompted to continue entering the next character of the content object name or click Display. You can continue to enter characters to reduce the list (you can also delete characters to increase the list) and at any point click Display. After you click Display, a list of content objects is displayed that has the same beginning characters that you entered in the text box.

In the Browse Content box, click one of the characters. A list of content objects that begin with that character is displayed.

In the Quick Lists box, the following options are offered:

Most Recent Ingests (max 100)—Lists the 100 most recent completed ingests sorted by ingest date.

List All Contents—Lists all completed ingests sorted by content name. This option is available only if the number of completed ingests is less than 100.

Content Status—Lists status information for each completed ingest.

Content Status (Damaged Only)—Lists status information only for damaged completed ingests.

After you perform one of these methods, a list is displayed. The list of content objects can span several pages. To view the next page, click the page number.

Figure 4-4 shows an example of the Completed Ingests list generated with any of the methods, except the Content Status options. The content names and the date they were ingested are displayed.


Note In a VVI, the Stream Manager displays the Streamed Content page, which has a subset of the information displayed on the Completed Ingest page.


Figure 4-4 Completed Ingests List

Figure 4-5 shows an example of the Content Status information that displays when you choose Content Status or Content Status (Damaged Only).

Figure 4-5 Completed Ingests List—Content Status

Table 4-3 Content Status Fields

Field
Description

Content Name

Name of the content.

Duration

Duration of the content.

GOID

Global Object ID for the content object associated with the content.

Version

Trick-play speed of the content object. The value, IGate, refers to an index file, which allows for the jumps between trick speeds, and so on. The value, redo, is an undo file. When the content is ingested, if there are any changes made during the ingest, the changes are recorded in the redo file. If the content is sent by using FTP Out, the changes are undone and the original file is sent.

Server ID

Server ID of the Vault that is storing the content object.

Status

Status of the storing process of the content object, either complete or partial.


Step 3 To view the details of a content object, click the content name. The Ingest Details are displayed (Figure 4-6).

Figure 4-6 Completed Ingests—Ingest Details

Table 4-4 describes the content object details that are displayed for each type of ingest.

Table 4-4 Content Object Details 

Field
Description

Content Name

Name of the content object.

Factory ID

Factory responsible for this content object.

Ingest IP

IP address of the ingest interface on the Vault used to download the content.

File Size

File size, in bytes, of this content object.

Rate

Rate of ingest in bits per second (3750000 = 3.75 Mbps).

Create Time

Time and date this content object was created.

Last Modified

Time and date this content object was last modified.

Op State

Operational state of this content object. The possible operational states are:

Created—Content is loading.

In Service—Content is available for streaming.

Out of Service—Content is not available for streaming.

Admin State

Administrative state of this content object. The possible administrative states are:

Unprovisioned—Content is loading.

In Service—Content is available for streaming.

Out of Service—Content is not available for streaming.

Push Provision

Type of FTP provisioned. The provision types are:

FTP pull

FTP push

Content Copies

These fields display the following information about the copies of the content:

Speed/Direction—The trick-mode speed and direction (fast forward or rewind).

Server ID—The server ID where the copy is stored.

Status—The status of the stored content.

Trick Speeds [1-8]

Trick speeds 1 through 8 show the trick-mode speeds for this content object.

Full Content ID

Full identification of this content object.

Ior1

Interoperable Object Reference (IOR) for this content object.

URL1

Uniform Resource Locator (URL) address of the content has the following:

Protocol used (for example, FTP)

Username and password (for example, videolan:mpeg4ftp)

IP address of the content provider server (for example, 192.168.100.184)

Directory where the content is stored on the provider server (for example, videolan)

Name of the file (for example, long_encore_3.75.mpg)

Asset Ior1

Asset IOR associated with this content object.

1 Not displayed in Figure 4-6.



Viewing Active Ingests

To view the details of active ingests, do the following:


Step 1 Choose Monitor > System Level > Active.

Step 2 Choose a content object from the drop-down list and click Display. The details of the content object are displayed. Figure 4-6 shows an example of the ingest details.

By typing the first character of the content object name, you can jump to that section of the list.

In addition, you can perform a text string search by typing the text string you want to search for in the Search Ingests field and clicking Search. A list of content objects that contain the text string are listed. To see the content object details, click the content object name listed. To return to the previous page without selecting a content object, click Back.

Table 4-4 describes the content object details that are displayed for active ingests.


Package Expirations


Note Package Expirations are part of the optional Ingest Manager feature. This option is listed only on the Monitoring System Level left-panel menu if the Ingest Manager is included in your deployment.


To view the details or adjust the license expiration of a package expiration, do the following:


Step 1 Choose Monitor > Package Expiration. The Package Expiration page is displayed.

Step 2 From the Available Packages drop-down list, choose a package and click Display. The Package Expiration details are displayed.

By typing the first character of the package name, you can jump to that section of the list.

Figure 4-8 shows an example of the Package Expiration details.

Figure 4-7 Package Expiration Page

Table 4-5 describes the package expiration details that are displayed.

Table 4-5 Package Expiration Details 

Field
Description

Package Name

Name of the package.

License Expiration

Date the package expires.

Additional Package Window

Additional time added to the package.

Actual Package Expiration

Actual Package Expiration is the License Expiration plus the Additional Package Window.


Step 3 To adjust the license expiration, enter the number of days (positive or negative) in the Adjust License Expiration field and click Update.

The license expiration is adjusted by the number of days you entered. The Additional Package Window is not affected and is still applied to create the Actual Package Expiration.

Step 4 In the Asset Details section, to view the metadata associated with the package expiration, click the plus sign (+) next to the metadata you want to view.


To delete a package, choose the package from the Available Packages drop-down list, click Display, and then click Delete in the Package Expiration Details section.

Publish Failures


Note Publish Failures are part of the optional Ingest Manager feature. This option is listed only on the Monitoring System Level left-panel menu if the Ingest Manager is included in your deployment.


The Publish Failures page lists the packages that were not able to be published to the backoffice and provides a mechanism to republish the package.

To publish an unpublished package, or delete an unpublished package, do the following:


Step 1 Choose Monitor > Publish Failures. The Publish Failures page is displayed.

Step 2 From the Unpublished Packages drop-down list, choose a package and click Display. The Unpublished Package details are displayed.

By typing the first character of the package name, you can jump to that section of the list.

Figure 4-8 shows an example of the Publish Failures details.

Figure 4-8 Publish Failures Page

Table 4-6 describes the publish failures details that are displayed.

Table 4-6 Publish Failures Details 

Field
Description

Package Name

Name of the package.

Source URL

Location of the original package information.

Target URL

Location where to place the package information.

Module Type

Internal identifier for what failed. In the example in Figure 4-8, the publishing of the package failed.

Creation Date

Date the database record was created for this failure.


Step 3 In the Assets section, to view the metadata associated with the unpublished package, click the plus sign (+) next to the metadata you want to view.

Step 4 To restart the publishing process and set the creation date to today, click Publish.

To delete an unpublished package, click Delete.


Monitoring Stream Objects

The monitored stream objects consist of:

Stream Monitor

Stream Failures

Stream Monitor


Step 1 Choose Monitor > Stream Monitor. The Stream Monitor page is displayed.

Step 2 The following methods can be used to display a a stream object or a list of stream objects:

Enter the first character of the session handle in the text box. A drop-down list of stream objects is displayed. If there are more than 25 objects that start with that first character you entered, you are prompted to continue entering the next character of the object name or choose one that is listed. You can continue to enter characters to reduce the list (you can also delete characters to increase the list) and at any point choose one. After you choose one, the stream details are displayed (Figure 4-10).

In the Quick Lists box, the following options are offered:

Most Recent 100—Lists the 100 most recent stream objects.

All Streams—Lists all streams. This option is available only if the number of streams is less than 100.

Destination IP, Destination Port, Client IP, Client Port—If you do not know the session handle, you can perform a text string search by selecting the field you want to search on, entering the text string in the Search String field, and clicking Search. A list of stream objects that contain the text string in the field you selected are listed. Click the linked field (session handle as seen in Figure 4-9) to see the stream monitor details for the selected object, or click Back to return to the previous page.

After you perform one of these methods, a list is displayed. The list of stream objects can span several pages. To view the next page, click the page number.

Figure 4-9 shows an example of the stream object list generated by clicking Most Recent 100 or All Streams. The Session IDs and the stream start time are displayed.

The stream object list generated by entering the destination IP, destination port, client IP, or client port in the Quick List box, displays the session ID, stream start time, destination IP, destination port, client IP, and client port.

Figure 4-9 Stream Monitor—Stream List

Step 3 To view the details of a stream object, click the session handle. The Stream Details are displayed.

Figure 4-10 shows an example of the stream object details.

Figure 4-10 Stream Monitor—Stream Details

Table 4-7 describes the stream details.

Table 4-7 Stream Details 

Field
Description

Session Handle

Internal unique identifier assigned to the stream session by the Streamer.

RTSP Session ID

RTSP identifier of the stream session that is used in communications with the backoffice and set-top box.

External Session ID

Session identifier generated by the backoffice.

Destination IP

IP address used to receive the stream (for example, the QAM or set-top box).

Destination Port

Port number used to receive the stream (for example, the QAM or set-top box).

Client IP

IP address of the set-top box that requested the stream.

Client Port

Set-top box port number used to send the stream request.

Session Group ID

Identifier defined by the backoffice for a logical grouping of sessions.

Client ID

Identifier of the client that is receiving the stream.

Service Group

Service group used for this session.

Service Area

Logical name that describes the service group, which is a logical grouping of subscribers.

Entitlement

This is an identifier that specifies if this a valid SETUP request. This is used to pass back to the authentication server during the setup process. This is only used in a EventIS environment.

Allocated Bandwidth

Bandwidth allocated for this stream session.

Current Operation

Current set-top box operation. The possible operations are:

Normal Play

Paused

Fast Forward

Rewind

Operation Start

Start time of the currently active operation.

Stream State

Stream state has the following possible values: playing, paused, stopped.

LSCP Mode

Valid LSCP modes are:

Open—The server is not transporting a media stream.

Pause—The server is not transporting a media stream.

Search transport—The server is searching for start NPT (normal play time). When at start NPT, it enters the Transport mode.

Transport—The server is transporting the media stream and pauses at the end of the stream. If scale is positive, indicating a forward direction, end of stream is the end of media. If scale is negative, indicating a reverse direction, end of stream is the beginning of media.

Transport pause—The server is transporting the media stream and pauses at stop NPT.

Search transport pause—The server is transporting the media stream and pauses at stop NPT.

Pause search transport—The server is transporting the media stream. It does so until stop NPT, and then transitions to search transport mode.

End of stream—The server is not transporting a media stream.

LSCP is not used in a Motorola environment. In an nABLE Motorola environment, the value is always 0.

Stream Looping

This field indicates whether stream looping is turned on or off for this stream.

From STB

If From STB has the value yes, then the set-top box created this session.

If From STB has the value no, then the backoffice created this session.

Stream Node

Logical node on the transport network used to stream the session to the STB. The value is a three-digit number, with 000 being the default. In an nABLE Motorola environment, the value is typically -1.

Position

Current stream position. The position is the number of milliseconds from the beginning of the video in NPT format.

History

Current transaction (trick mode, play, pause) history of this stream session.


The Graph Stream button displays the trick-mode activity of the stream (Figure 4-11).


Note If Trick Mode Capture is disabled, the Graph Stream is not displayed. For information on enabling the Trick Mode Capture, see the "Trick Mode Capture" section on page D-5.


Figure 4-11 Stream Activity Report

To delete a stream object, display the object and click Delete.


Stream Failures

To view the details of Failed Streams, do the following:


Step 1 Choose Monitor > Failed Streams. The Stream Failures page is displayed.

Each failed stream is listed by the date and time the stream failed, followed by the session ID of the failed stream.


Note Stream Failure monitoring displays only the failed streams for the current day. To view past stream failures, see the "Stream Failures" section on page 5-15.


Step 2 From the Stream Failures drop-down list, choose the timestamp and session ID of the stream object and click Display. The stream failure details are displayed. See Figure 4-12.

To delete a failed stream, display the object and click Delete.

Figure 4-12 Stream Failures Page

Table 4-8 describes the stream failure details.

Table 4-8 Stream Failure Details 

Field
Description

Session ID

Session ID of the failed stream.

Failure Date

Date and time the failure occurred.

QAM IP

IP address of the QAM device associated with the failure.

Service Group

Service group associated with the failure.

Server ID

Server responsible for streaming this stream object. To view the IP address associated with the Server ID, see the "Configuring the Servers" section on page 3-58.

Group ID

All servers that are part of the same CDS system (managed by one CDSM) have the same Group ID. This Group ID corresponds to the CDSM GUI array ID and should be unique across an enterprise. Table 4-9 describes the ID mapping between the CDSM GUI and the CServer.

Failed Operation

Operation that was taking place when the stream failed, for example, createStream, LSCP Command(), or createServant, destroy. These are the measurement points or transactional states of the system at the time of the failure.

Failed Task

Failed task is the event category that provides the type of execution sequence that the call stack was currently in at the time of the failure. The list of the high-level categories are:

 

Tune In

Load Application

Load Catalog

Eligibility Check

Select Subscription

Purchase Subscription

Select Movie

Purchase Check

Play Movie

Movie Setup

Movie Control

Movie Confirm

Purchase Confirm

Purchase Log

Stop Movie

Movie Release

Error Code

Error code provides a description of the event that caused an error. See Table 4-10 for descriptions of the error codes.


Table 4-9 lists the CDSM GUI ID names and maps them to the CServer names in the setupfile and .arroyorc files.

Table 4-9 ID Names in the CDSM GUI and CServer Files 

CDSM GUI ID Name
CServer Files ID Name

Array ID on the Array Name page

groupid

Group ID on the Server-Level pages

groupid

Stream Group ID on the Server Setup page

arrayid

Cache Group ID on the Server Setup page

arrayid

Vault Group ID on the Server Setup page

arrayid

Stream Group ID on the Configuration Generator page

arrayid


Table 4-10 lists the error codes for internal errors that specifically describe where the error occurred, and also lists errors in the network or network components.

Table 4-10 Error Codes 

Numeric Error Code
Error Code
Description

2024

FTP_CONNECTION_FAILED

Connection to FTP server failed.

2025

FTP_SERVER_BIND_FAILED

FTP server can not bind to the port.

2026

FTP_PUSH_TIMEOUT

FTP push timeout (PASV is not served fast enough).

2027

FTP_QUIT_RECEIVED_DURING_INGEST

FTP server received QUIT request.

2036

NOT_ENOUGH_QAM_BANDWIDTH

Not enough QAM bandwidth.

2041

CONTENT_CAN_NOT_BE_LOCATED

Content is not found in the related content stores.

3006

SET_DESTINATION_FAILED

Failed while setting destination of the stream.

6001

RTSP_REQUEST_INVALID

RTSP request was not understood by the RTSP server. Either the syntax is incorrect of the method is unknown.

6002

SESSION_NOT_FOUND

RTSP session ID could not be found in the RTSP server.

6003

RTSP_METHOD_NOT_VALID_IN_STATE

RTSP request is not valid for the current RTSP session state.

6004

RTSP_RANGE_NOT_VALID

RTSP range header does not have valid values.

6005

TRANSPORT_UNSUPPORTED

Transport header could not be parsed.

6006

NO_DESTINATION_DEFINED

Transport header does not have a stream destination value.

6007

INTERNAL_SERVER_ERROR

There was an internal server error during an RTSP request.

6008

RTSP_SERVICE_UNAVAILABLE

RTSP service is unavailable.

6009

UNSUPPORTED_RTSP_VERSION

RTSP version is not supported.

6010

UNSUPPORTED_OPTION

RTSP require header is not supported.

6011

STREAMING_ERROR_READING_CONTENT

There was an error reading the content in a stream.


Table 4-11 lists the error codes for errors that could occur during ingest or during trick-mode file creation, which cause stream failures.

Table 4-11 MPEG Error Codes 

Numeric Error Code
Error Code
Description

8001

TRICK_INGEST_NO_INGEST_OBJECTS

Ingest fails. Check available system memory.

8002

TRICK_INGEST_TOO_MANY_SPEEDS

Too many trick speeds. Change trick speed configuration.

8003

TRICK_INGEST_NULL_INGEST_OBJECTS

Ingest fails. Check available system memory.

8004

TRICK_INGEST_INVALID_SPEED_DENOMINATOR

Ingest fails. Change trick speed configuration.

8005

TRICK_INGEST_INVALID_SPEED_LT_2X

Ingest fails. Change trick speed configuration.

8006

TRICK_INGEST_NULL_DERIVED_INGEST_OBJECTS

Ingest fails. Check available system memory.

8007

TRICK_RSDVR_DYNAMIC_TRICK_CREATION
_FAILS

RS-DVR trick-mode file creation fails. Change trick speed configuration.

8008

TRICK_INGEST_CDN_AVC_UNSUPPORTED

Do not attempt to ingest an Advanced Video Coding (AVC) stream on a VVI system.

8009

TRICK_RSDVR_BAD_STREAM_TYPE

RS-DVR trick-mode file creation fails. Stream should already have failed ingest.

8010

TRICK_INGEST_ABORTED

General ingest failure. Check ingest feed.

8011

TRICK_INGEST_INSUFFICIENT_DATA

Ingest fails. Check ingest feed.

8012

TRICK_INGEST_STREAM_TOO_BIG

Ingest fails. The limit is 162 GB or about 12 hours at a known bitrate.

8013

TRICK_RSDVR_WRITE_OVERFLOW

RS-DVR trick-mode file creation fails. Check ingest feed.

8014

TRICK_INGEST_VBR_UNSUPPORTED

Ingest fails. Check ingest feed.

8015

TRICK_INGEST_RATE_FORCED

Streaming rate may be incorrect. Check ingest feed.

8016

TRICK_INGEST_PAT_NOT_FOUND

Program association table (PAT) not found. Check ingest feed.

8017

TRICK_INGEST_DEFAULTING_PMT_PID

Program map table (PMT) process ID (PID) not determined. Check ingest feed.

8018

TRICK_INGEST_DEFAULTING_PROGRAM_NUMBER

Program number not determined. Check ingest feed.

8019

TRICK_INGEST_DEFAULTING_VIDEO_PID_AND_
TYPE

Video PID or type not determined. Check ingest feed.

8020

TRICK_INGEST_BITRATE_INDETERMINATE

Bitrate cannot be determined. Check ingest feed or adjust ingest configuration parameters.

8021

TRICK_INGEST_FIRST_PTS_NOT_FOUND

First presentation time stamp (PTS) not determined. Check ingest feed.

8022

TRICK_INGEST_CANNOT DETERMINE_FRAMERATE

Frame rate not determined. Check ingest feed.

8023

TRICK_INGEST_PMT_NOT_FOUND

PMT not found. Check ingest feed.

8024

TRICK_INGEST_MULTIPLE_VIDEO_PIDS

Multiple video PIDs found. Check ingest feed.

8025

TRICK_INGEST_PID_REPLACEMENT_CANCELLED

PIDs could not be standardized. Check ingest feed.

8026

TRICK_INGEST_OVERFLOW

Ingest fails because of ring buffer overflow. Check ingest feed.

8027

TRICK_INGEST_WRITE_ERROR

Ingest fails because of a 1x write problem. Check ingest feed.

8028

TRICK_INGEST_OVERFLOW_ON_RETRY

Ingest fails even after a retry (ring buffer overflow). Check ingest feed.

8029

TRICK_INGEST_KNOBS_FAILURE

Ingest fails. Check ingest feed or adjust ingest configuration parameters.

8030

TRICK_INGEST_KNOBS_FAILURE_PAT_PMT

Ingest fails. No PAT or PMT found. Check ingest feed or adjust ingest configuration parameters.

8031

TRICK_INGEST_KNOBS_FAILURE_BITRATE

Ingest fails. Bitrate cannot be computed. Check ingest feed or adjust ingest configuration parameters.

8032

TRICK_INGEST_KNOBS_FAILURE
_DISCONTINUITIES

Ingest fails. Too many discontinuities. Check ingest feed or adjust ingest configuration parameters.

8033

TRICK_INGEST_KNOBS_FAILURE_CONTINUITY
_COUNTERS

Ingest fails. Too many continuity counter errors. Check ingest feed or adjust ingest configuration parameters.

8034

TRICK_INGEST_KNOBS_FAILURE_SYNC

Ingest fails. Too many sync errors. Check ingest feed or adjust ingest configuration parameters.

8035

TRICK_INGEST_KNOBS_FAILURE_SYNC_TIME

Ingest fails. Sync loss too long. Check ingest feed or adjust ingest configuration parameters.

8036

TRICK_INGEST_KNOBS_FAILURE_PIC_GAPS

Ingest fails. Too many picture gaps. Check ingest feed or adjust ingest configuration parameters.

8037

TRICK_INGEST_KNOBS_FAILURE_PIC_GAP_TIME

Ingest fails. Picture gap too long. Check ingest feed or adjust ingest configuration parameters.

8038

TRICK_INGEST_SEQUENCE_HEADER_NOT_FOUND

Ingest fails. Could not find a Sequence Header. Check ingest feed.

8039

TRICK_INGEST_SPS_NOT_FOUND

Ingest fails. Could not find an SPS. Check ingest feed.

8040

TRICK_INGEST_CDN_SEQ_WRITE_FAILED

Ingest fails because of a Sequence Header write error. Check ingest feed.

8041

TRICK_INGEST_CDN_NONCONFORMAL
_FRAME_START

VVI: Ingest fails. Invalid frame start. Check ingest feed.

8042

TRICK_INGEST_SPLIT_SEQEND_SEQ_PAIR

Ingest fails. SequenceEnd/SequenceHeader pair not consecutive. Check ingest feed.

8043

TRICK_INGEST_PIC_SIZE_CHANGED

Ingest fails. Picture size changed. Check ingest feed.

8044

TRICK_INGEST_PIC_SIZE_H_OR_V_ZERO

Ingest fails. Picture size H or V zero. Check ingest feed.

8045

TRICK_INGEST_HORIZONTAL_PIC_SIZE_EXCEEDS_MAX

Ingest fails. Horizontal size exceeds max (1920). Check ingest feed.

8046

TRICK_INGEST_VERTICAL_PIC_SIZE_EXCEEDS
_MAX

Ingest fails. Vertical size exceeds max (1088). Check ingest feed.

8047

TRICK_INGEST_SEQUENCE_HEADER_CHANGED

VVI: Ingest fails. Sequence Header changed. Check ingest feed.

8048

TRICK_INGEST_SEQUENCE_HEADER_CHANGE_NO_SEQEND

Ingest fails. Sequence Header changed with no preceding Sequence End.

8049

TRICK_INGEST_SEQUENCE_HEADER_CHANGE
_BAD_PRIOR_STARTCODE

Ingest fails. Sequence Header changed with no immediately preceding Sequence End.

8050

TRICK_INGEST_SEQUENCE_HEADER_CHANGE
_NO_PRIOR_STARTCODE

Ingest fails. Sequence Header changed with no preceding start code

8051

TRICK_INGEST_ILLEGAL_FRAMERATE

Illegal frame rate code. Check ingest feed.

8052

TRICK_INGEST_CDN_ILLEGAL PES_PACKETISATION

VVI: Ingest fails. Illegal PES packetization. Check ingest feed.

8053

TRICK_INGEST_CDN_STREAM_STARTS_WITH
_P_FRAME

VVI: Ingest fails: Stream begins with a P-frame. Check ingest feed.

8054

TRICK_INGEST_CDN_STREAM_STARTS_WITH
_B_FRAME

VVI: Ingest fails: Stream begins with a B-frame. Check ingest feed.

8055

TRICK_INGEST_ZERO_BITRATE

Check ingest feed. Bitrate indeterminate.

8056

TRICK_INGEST_CDN_STREAM_STARTS_WITH
_BAD_I_FRAME

VVI: Ingest fails: Stream begins with a malformed I-frame. Check ingest feed.


Table 4-12 lists the Managed Services Architecture (MSA) error codes for the optional Ingest Manager feature.

Table 4-12 MSA Error Codes for the Optional Ingest Manager Feature

Numeric Error Code
Error Code
Description

7000

MSA_BAD_XML

There was an XML parsing error. Check the ADI XML for errors.

7001

MSA_BAD_REQUEST

The request for content was bad. Check the target backoffice URL.

7002

MSA_UNKNOWN_HOST

The host is unknown. Check the target backoffice URL.

7003

MSA_CONNECTION_DROP

The connection was dropped. Check the URL. The Ingest Manager possibly misformatted the ADI XML.

7004

MSA_BACKOFFICE_TIMEOUT

The backoffice did not respond within the allowed time interval.

7005

MSA_UNKNOWN

An unknown error occurred. Check the /home/isa/bss/log/aim.log.

7006

MSA_FAILED_POST

The Ingest Manager failed to post the ADI to the backoffice.

7007

MSA_PKG_EXPIRED

The package has expired and the retry record is removed.

7008

MSA_ENCRYPT_FAILED

Ingest failed because Ingest Manager was unable to encrypt the content.

7009

MSA_STORE_FAILED

Ingest failed because Ingest Manage had a problem with the storage server.

7010

MSA_BACKOFFICE_FAILED

Ingest failed because Ingest Manage was unable to contact the backoffice.

7011

MSA_INVALID_URL

URL provided for the ingest is invalid.


Table 4-13 lists the MSA error codes for the File Service Interface (FSI).

Table 4-13 MSA Error Codes for FSI

Numeric Error Code
Error Code
Description

9001

MSA_INGEST_INIT

Unable to initialize the ingest from the FTP location.

9002

MSA_CONTENT_ALREADY

Content is already ingesting.

9003

MSA_DEL_RECORD_FAILED

Database record deletion failed.

9004

MSA_INGEST_CANCELED

Ingest has been canceled.

9005

MSA_UNKNOWN_ERROR

Unknown ingesting error has occurred.

9006

MSA_ALLOC_ERROR

Unable to continue because of a memory allocation error.

9007

MSA_FILLCB_FAILED

CServer FillCB call failed.

9008

MSA_ICONTENT_FAILED

CServer call to iContentPlay failed.

9009

MSA_INGEST_FAILED

Unable to ingest the specified content.



Array Level Monitoring

The Array Level Monitoring pages provide an overall view of the health and activity of a specified array.

Array Snapshot

The Array Snapshot page provides an overview of the current activity for the specified array of servers on the CDS. A summary of the state of all streams, content ingests, and disk usage is displayed.

The fields displayed on the Array Snapshot page are the same fields that are displayed on the System Snapshot page, with the active stream bandwidth and active fill bandwidth shown for each Stream Group. For descriptions of the fields, see Table 4-1.

Server Level Monitoring

The Server Level Monitoring pages provide detail information on the health and activity of a Vault or Streamer server.

To view the Server Level Monitoring pages, do the following:


Step 1 Choose Monitor > Server Level, and then click one of the following as applicable:

Disk Monitor

NIC Monitor

Server Vitals 

Cache/Fill Bandwidth

Services Monitor

Step 2 Choose the IP address of the server from the drop-down list and click Display.


Disk Monitor

The Disk Monitor page provides real-time information on the status of a disk.

To view the current status of a disk, choose the IP address of the server from the drop-down list on the Disk page, click Display, and roll your mouse over one of the disks displayed in the graphic. If the server is a Lindenhurst CDE, click one of the disks displayed in the graphic. Figure 4-13 shows an example of a Streamer server.

To change how often the information is refreshed, enter the number of seconds in the Data Refresh Rate field. The default is 10 seconds. The range is 10 to 300. All field values that are updated, based on the refresh rate, are initially shown in a green font.

Figure 4-13 Disk Monitor—Streamer Page

For Vault servers, the Disk Availability line graph shows the percentage of disk space available. The gigabytes displayed for "Total Space," "Available Space," and "% Used" " are the sum of all the disks installed on the Vault server.

The Linux File System Stats table shows the combined total storage space for the partitions of the flash modules , the combined available storage space for the partitions of the flash modules, and the percentage of used storage for each combined partition. In Figure 4-13, the hda2 partition has an alarm indicator because the usage has exceeded the user-defined threshold of 40 percent. For information on setting thresholds, see the "Setting System Thresholds" section on page 7-7.

Table 4-14 describes the information displayed when a flash module is selected.

Table 4-14 Flash Module Status Fields 

Field
Description

Current Temp

The current temperature of the flash module.

Smart Status

The Self-Monitoring, Analysis and Reporting Technology (S.M.A.R.T.) status of a disk as determined by the manufacturer in accordance with the relevant ATA/SCSI standards. S.M.A.R.T. is logic embedded in the firmware that determines when a disk is going bad.

Number of Reads

Number of bytes read from the flash module since it was powered on.

Number of Writes

Number of bytes written to the flash module since it was powered on.


S.M.A.R.T

The CDS incorporates S.M.A.R.T. to monitor the reliability of a hard drive, predict drive failures, and to carry out different types of drive self-tests. S.M.A.R.T is firmware, native to most disk drives, that monitors disk attributes over time, making it possible to perform predictive failure analysis. Advanced warning of predictive failures allows the operator to perform preventative maintenance.

To view the current read/write activity that has occurred in the last five seconds on the selected disk, click Graph Disks. Figure 4-14 shows an example of the Disk Activity graph.

Figure 4-14 Disk Monitor—Vault Disk Activity Graph

The Disk Activity graph displays an average calculation of the amount of data read (in megabytes per second) and data writes over a five-second period.

NIC Monitor

The NIC Monitor page displays the status of each network interface card (NIC) on the server.

To view the current status of a NIC on a server, choose the IP address of the server from the drop-down list on the NIC Monitor page, click Display, and click one of the NIC ports displayed in the graphic. Figure 4-15 shows an example of the eth4 interface on a Vault server.

To change how often the information is refreshed, enter the number of seconds in the Data Refresh Rate field. The default is 10 seconds. The range is 10 to 300. All field values that are updated, based on the refresh rate, are initially shown in a green font.

Figure 4-15 NIC Monitor—Vault Page

Table 4-15 describes the information displayed for a NIC port.

Table 4-15 NIC Port Status Fields 

Field
Description

Port Speed

Speed of the interface in megabits per second (Mbps).

Admin State

Administrative state of the port interface. The administrative state is determined at the time the server is booted. The possible administrative states are up or down.

Op State

Operational state of the port interface. The operational state is either up or down. If the port is not connected to the network or is malfunctioning, the operational state displayed is down.

Media Type

Physical conduit of the interface. The physical type is either copper or fiber optic.

Poll Interval

Number of seconds between each disk polling.

Transmit

Total number of bytes transmitted since this port has been operational and configured as administratively up.

Received

Total number of bytes received since this port has been operational and configured as administratively up.


To view the average transmit and receive activity that has occurred in the last two seconds for each port on this server, click Graph Ports. Figure 4-16 shows an example of the Port Activity graph.

Figure 4-16 NIC Monitor—Vault Port Activity Graph


Server Vitals

The Server Vitals page provides current values for monitored system components. Server components are monitored and if a threshold is exceeded, the System Health Monitor page reports the event and an SNMP trap is sent to the Network Management System (NMS).


Note The Server Vitals page is displayed only if the CDSM Health Monitor feature is enabled. For more information, see the "CDSM or VVIM Health Monitoring" section on page D-9.


To view the current values of the monitored components, as well as the threshold settings, choose the IP address of the server from the drop-down list on the Server Vitals page and click Display.

To change how often the information is refreshed, enter the number of seconds in the Data Refresh Rate field. The default is 10 seconds. The range is 10 to 300. All field values that are updated, based on the refresh rate, are initially shown in a green colored font.

To change the temperature format to Fahrenheit, choose ºF for the Temperature Format. The default is Celsius (ºC).

The monitored components are different for each Content Delivery Engine (CDE) model. Figure 4-17 shows an example of the Server Vitals page for a Vault (CDE420).

Figure 4-17 Server Vitals Page

Cache/Fill Bandwidth

The Cache/Fill Bandwidth page displays details on the content caching activity on a Streamer.

To view the caching activity on a server, choose the IP address of the server from the drop-down list on the Cache/Fill Bandwidth page and click Display. Figure 4-18 shows an example.

To change how often the information is refreshed, enter the number of seconds in the Data Refresh Rate field. The default is 10 seconds. The range is 10 to 300. All field values that are updated, based on the refresh rate, are initially shown in a green font.

Figure 4-18 Cache/Fill Bandwidth Page

Table 4-16 describes the services listed in the Cache/Fill Bandwidth page.


Note The values in the Cache/Fill Bandwidth page are only accurate if the clocks on the CDS servers are synchronized with the CDSM.


Table 4-16 Cache/Fill Bandwidth Fields 

Service
Description

Active Stream Count

The number of active streams on this Streamer.

Active Stream Bandwidth

The bandwidth (in Mbps) used for streaming on this Streamer.

Unique Stream Count

The number of unique streams on this Streamer.

Unique Stream Bandwidth

The bandwidth (in Mbps) used for serving unique streams on this Streamer.

Fill Receive Stream Count

The number of streams on this Streamer that are retrieving content from the Vault to fulfill requests for content.

Actual Fill Stream Bandwidth

The bandwidth (in Mbps) used on this Streamer for retrieving content from the Vault.

Disk Read Stream Count

The number of streams on this Streamer sending content that was retrieved from the disks on the Streamer.

Disk Read Bandwidth

The bandwidth (in Mbps) used on this Streamer for retrieving locally stored content (content on the disks on the Streamer).


Services Monitor

The Services Monitor page displays whether specific processes are running on a server.

To view the current status of the services running on a server, choose the IP address of the server from the drop-down list on the Services Monitor page and click Display. Figure 4-18 shows an example of a Streamer server.

Figure 4-19 Services Monitor Page—Streamer

Table 4-17 describes the services listed on the Services Monitor page.

Table 4-17 CDS Services 

Service
Server
Description

Cisco Cache Server

Both

The Cache server runs on all servers. The Cache server is responsible for the core functions of the CDS.

Cisco FSI Master

Vault

The File Service Interface (FSI) Master process is running if you are looking at a master Vault server. The FSI Master serves as the master Vault process for content file-related requests from the backoffice.

Cisco FSI Server

Vault

The FSI Server runs on every Vault and processes file-related requests from the FSI Master.

Cisco RTSP Server

Streamer

The RTSP server runs on the Streamer as either a client or server depending on the headend environment. The RTSP server handles messaging between the CDS and the backoffice.

Cisco Resource Manager

Streamer

The Resource Manager runs on a Streamer server. The Resource Manager handles orphaned streams.

Cisco DB Server

Both

The DB (database) server runs on all servers and is responsible for keeping track of all data objects in the CDS.

DB Synchronization Status

Both

Displays the status of the database synchronization among all servers. The states are "OK" and "not OK."

Cisco SNMP Server

Both

The SNMP server shows as running when the SNMP agent is running.

Cisco System Manager

Both

The System Manager runs on each server and facilitates communication with the CDSM.

Cisco BWMgr Server

Streamer

The Bandwidth Manager process runs on the Streamer provided the optional Bandwidth Manager feature is part of your deployment. It keeps track of allocated streams and VOD network resources.

Cisco Ingest Manager

Vault

The Ingest Manager process is running if you are looking at a master Vault server and the optional Ingest Manager feature is part of your deployment.


Recommended Monitoring Schedule

This monitoring schedule is recommended to ensure that the CDS is functioning as expected and identify potential issues that may cause down time.


Caution Do not attempt to access the Linux command line unless you are familiar with the CDS, the Linux operating system, and have a basic understanding of the Linux command line.


Note Some error warnings in the logs are only informational and no action is necessary.


Daily Tasks

The following tasks should be performed daily:

Choose Monitor > System Level > System Health and check the System Health Monitor page for red or yellow states on any of the servers. Click any red or yellow boxes to see detail information on disk, NIC, or services. See the "System Health" section for more information.

Choose Monitor > System Level > Failed Ingests to check for any failed ingests. See the "Ingests" section for more information.

Weekly Tasks

The following tasks should be performed weekly:

Monitoring Tasks for Streamers and Vaults

Monitoring Tasks for Vaults

Monitoring Tasks for Streamers


Note All commands require that you log into the Linux operating system as root. Some tasks have a CDSM option.


Monitoring Tasks for Streamers and Vaults

To monitor the Streamer and Vaults weekly, do the following:


Step 1 Recover used disk space. Log in to each server using the root logon and run the following command:

dh -h

Filesystem            Size  Used Avail Use% Mounted on
/dev/hda1              13G  5.2G  7.0G  43% /
/dev/hda6              20G   16G  4.3G  78% /arroyo/log

If the disk usage is greater than 75 percent, recover the disk space using the following methods:

a. Search and remove any core files.

find /arroyo -name core*
find /home/isa -name core*

b. Copy any archived logs to an external device and delete them from the /arroyo/archive directory.

c. Check for the presence of old install or upgrade ISO files in the /root directory and delete them.

find /root -name *.iso
find /arroyo -name *.iso

Step 2 Verify the services are running. Choose Monitor > Server Level > Services Monitor to check the services for each server, or log in to each server and run the following commands:

su - isa
show_calypso_services 

Step 3 Check the CServer interfaces to verify the status of the Ethernet adapters. Choose Monitor > Server Level NIC Monitor, or log in to each server and use the following commands

a. Use the grep -i Link command to verify that all adapters should have a status of "link up," except those adapters that are not being used.

grep -i Link /proc/net/PRO_LAN_Adapters/*.info

b. Use the grep -i Speed command to verify that each adapter that has a "link up" status should have a speed of 1000.

grep -i Speed /proc/net/PRO_LAN_Adapters/*.info

c. Use the grep -i State command to verify that all adapters should have an "up" state, except those adapters that are not being used.

grep -i State /proc/net/PRO_LAN_Adapters/*.info

Step 4 Check the CServer streaming and cache-fill interfaces using the following command:

/home/stats/ifstats

Step 5 Check the database thread count using the following command:

netstat -an | grep 9999

Two connections for each Vault and Streamer should be listed with a status of "ESTABLISHED."

Step 6 Check the protocol timing logs for errors or problems. Also, look at the protocol timing logs for packet retransmissions.

tail -f /arroyo/log/protocoltiming.log.{date} | grep retransmissions 

Step 7 Look for warning messages.

grep -i warning /arroyo/log/protocoltiming.log.<date> | more 


Note The "WARNING" messages can sometimes be misleading; for example, "datawait" and "slow disk" messages occur normally and do not indicate a problem.


Step 8 The number of GOIDs for a particular content object must be the same on all servers (Vaults and Streamers) that are supposed to have the content. The number of Vaults that must have the same number of GOIDs for a particular content object is determined by the mirrored copy configuration (see the "Configuring the Servers" section on page 3-58). The number of GOIDs is also dependent on the trick speeds configured you configured (see the "Configuring Ingest Tuning" section on page 3-17). If the GOID is different between a Vault and a Streamer, session setup is not created properly because of an issue of "no content available." This is because there is no content on the Vault that matches the GOID of the Streamer has.


Monitoring Tasks for Vaults

In addition to the weekly monitoring tasks for both the Vaults and Streamers, the Vaults can also be monitored in the following ways:

1. Check the available space on the Vault hard drives. Choose Monitor > Server Level > Disk Monitor. The disk availability is shown as a percentage and as a number of gigabytes. Alternatively, view the protocol timing logs by running the following command:

tail -f /arroyo/log/protocoltiming.{date} | grep "Capacity Disk:"

The number returned indicates the percentage of the disk space available on this server. If the number is 5 or lower, then steps need to be taken to increase storage space by adding more Vaults, replacing drives with higher capacity drives, or removing unused content.

2. Check the /home/isa/fsi logs for ingest errors on each Vault. The master Vault has an additional log.

3. Look for errors in the following log files in the /home/isa/bss/logs/ directory:

fsi.log

fsi.err

The fsi.log and fsi.err files are related to the ingest activity on a Vault.

Monitoring Tasks for Streamers

In addition to the weekly monitoring tasks for both the Vaults and Streamers, the Streamers can also be monitored in the following ways:

1. Look at the streaming log.

tail -f /arroyo/log/streamevent.log.<date>

2. Look for errors in the following log files in the /home/isa/bss/logs/ directory:

rtsp.log

rtsp.err

The rtsp.log and rtsp.err files are used to check the stream setup and control on a Streamer.

Monthly Tasks

The monthly monitoring tasks consist of the following:

1. Choose Monitor > System Level > System Snapshot and check that the "Disk Available" amount meets the requirements for the expected movie storage in the next three to six months.

2. Run the reports for the last month that are suitable for your requirements and save them as comma-separated value (CSV) files.

3. Using the bandwidth and streaming reports, check that the CDS is not exceeding required usage per service area.

4. Run a quick security check.

a. Ensure that the CDSM changes can be attributed to individual users and not to a generic admin account.

b. Reset CDSM passwords if necessary.

c. Reset Linux passwords if necessary.

d. Check that access policies and firewalls are still enforced.

Other Tasks

If you have access to an anything on demand (XOD) application, do the following:

1. Check the inspect-live log for excessive errors.

2. Check the inspect-live log for excessive communication times with the BMS or CDS.