Table Of Contents
System Monitoring
System Level Monitoring
System Health
System Snapshot
Monitoring Content Objects
Ingests
Viewing and Deleting Completed Ingests
Viewing Other Ingests
Package Expirations
Publish Failures
Monitoring Stream Objects
Stream Monitor
Stream Failures
Array Level Monitoring
Array Snapshot
Barker Stream Monitor
Server Level Monitoring
Disk Monitor
S.M.A.R.T
NIC Monitor
Server Vitals
Cache/Fill Bandwidth
Services Monitor
Recommended Monitoring Schedule
Daily Tasks
Weekly Tasks
Monitoring Tasks for Streamers and Vaults
Monitoring Tasks for Vaults
Monitoring Tasks for Streamers
Monthly Tasks
Other Tasks
System Monitoring
The CDSM provides tools that can be used for system monitoring and system diagnostics. The topics covered in this chapter include:
•
System Level Monitoring
•Monitoring Content Objects
•Monitoring Stream Objects
•Array Level Monitoring
•Server Level Monitoring
•Recommended Monitoring Schedule
Note If Virtual Video Infrastructure (VVI) with split-domain management is enabled, the CDSM pages associated with the Vaults and Caching Nodes display only on the VVI Manager (VVIM), and the CDSM pages associated with the Streamers display only on the Stream Manager. For more information, see the "Virtual Video Infrastructure" section on page E-6.
System Level Monitoring
The System Level Monitoring pages provide an overall view of the health and activity of the CDS. The System Level links are:
•System Health
•System Snapshot
To view the System Level Monitoring pages, click Monitor from any page in the CDSM, and then click System Health or System Snapshot, as appropriate.
System Health
The System Health page provides a top-level view of the overall health of each group in the CDS and each server in each group. Any time there is an alarmed event, an alarm is displayed in the CDSM banner. The Alarms menu is displayed when you roll your mouse over the alarm icon. See Figure 5-1. Clicking the alarmed event in the Alarm menu takes you to the CDSM page that has more information. For example, in Figure 5-1, clicking System health problems reported takes you to the System Health page.
Figure 5-1 CDSM Banner—System Health Alarm
To view the System Health page, choose Monitor > System Health. See Figure 5-2.
Figure 5-2 System Health Page
The colored boxes on the System Health Monitor page have the following meaning:
•Green—All components are operating.
•Yellow—Some components are not operational.
•Red—All components have failed.
You can view the details of a monitored area of a server by clicking the box in the appropriate column.
•When you click the Network check box you are taken to the NIC Monitor page. See the "NIC Monitor" section for more information.
•When you click the Disk check box you are taken to the Disk Monitor page. See the "Disk Monitor" section for more information.
•When you click the Services check box you are taken to the Services Monitor page. See the "Services Monitor" section for more information.
•When you click the Vitals check box you are taken to the Server Vitals Monitor page. See the "Server Vitals" section for more information.
Note The Vitals column is displayed only if the CDSM Health Monitor feature is enabled. For more information, see the "CDSM or VVIM Health Monitoring" section on page E-8.
The time shown at the bottom of the left-panel menu is not the current time, but rather the CDSM time that is used for the health status and monitoring the system.
System Snapshot
The System Snapshot page provides an overview of the current activity on the CDS. A summary of the state of all streams, content ingests, and disk usage is displayed. See Figure 5-3.
Figure 5-3 System Snapshot Page
Table 5-1 describes the information displayed on the System Snapshot page.
Table 5-1 System Snapshot Page
Field
|
Description
|
Data Refresh Rate
|
How often the information is refreshed. The default is 10 seconds. The range is 10 to 300. All field values that are updated, based on the refresh rate, are initially shown in a green colored font.
|
Total Streams
|
Total number of stream objects the CDS is currently streaming.
|
HD Streams
|
Total number of high-definition stream objects the CDS is currently streaming.
|
SD Streams
|
Total number of standard-definition stream objects the CDS is currently streaming.
|
Active Stream Bandwidth
|
Total bandwidth, in megabits, used for active streams.
|
Active Fill Bandwidth
|
Total bandwidth, in megabits, used for caching content among Vaults and Streamers.
|
Total Content
|
Total number of content objects currently stored, ingested, provisioned for ingest, and failed ingest on the CDS.
|
Active Ingests
|
Total number of content objects currently being ingested on the CDS.
|
Completed Ingests
|
Total number of content objects currently stored on the CDS.
|
Prov. (push) Ingests
|
Total number of content objects that have been requested for ingestion, but have not yet begun active ingestion.
|
Failed Ingests
|
Total number of content objects that failed to complete the ingest process.
|
Temp Out of Service
|
Total number of content objects that are in a Temp Out of Service state. The backoffice may put a content object into this state for a certain amount of time.
|
Total Disk
|
Total disk space, in bytes, on the CDS.
|
Disk Used
|
Total used disk space, in bytes, on the CDS.
|
Disk Available
|
Total available disk space, in bytes, on the CDS.
|
Monitoring Content Objects
The content objects links on the Monitor System Level page provides information on the status of content ingests. Table 5-2 describes the different ingest states that are monitored.
Table 5-2 Monitored Ingest States
Completed Ingests
|
Lists content objects that have been fully ingested.
|
Active Ingests
|
Lists content objects that are in the process of being ingested.
|
Provisioned Ingests
|
Lists content objects that have been requested for ingestion, by way of the BMS administrator creating an entry, but have not yet begun active ingestion.
|
Unprovisioned Ingests
|
Lists content objects that were terminated by the BMS administrator or have been created but do not yet contain any information (in other words, they are blank shells).
|
Failed Ingests
|
Lists content objects that failed to complete the ingest process. All failed ingests are reported back to the OpenStream system.
|
Package Expiration
|
Lists Package Expiration information, including expiration dates and all associated metadata. Allows for package expiration adjustments.
|
Publish Failures
|
Lists the packages that were not able to be published to the backoffice and provides a mechanism to republish the package.
|
In a VVI with split-domain management, the Stream Manager displays the following completed ingest details: Content Name, File Size, Rate, Create Time, and Last Modified. For the other completed ingests fields, see the same content asset on the VVIM.
Ingests
Viewing Completed Ingests is a different procedure than viewing the other types of monitored ingests. This section contains the following topics:
•Viewing and Deleting Completed Ingests
•Viewing Other Ingests
Viewing and Deleting Completed Ingests
To view the details of completed ingests, do the following:
Step 1 Choose Monitor > System Level > Completed Ingests.
Step 2 The following methods can be used to display a list of content objects:
•Enter the first character of the content object name in the text box. A drop-down list of content objects is displayed. If there are more than 25 content objects that start with that first character you entered, you are prompted to continue entering the next character of the content object name or click Display. You can continue to enter characters to reduce the list (you can also delete characters to increase the list) and at any point click Display. After you click Display, a list of content objects is displayed that has the same beginning characters that you entered in the text box.
•In the Browse Content box, click one of the characters. A list of content objects that begin with that character is displayed.
•In the Quick Lists box, the following options are offered:
–Most Recent Ingests (max 100)—Lists the 100 most recent completed ingests sorted by ingest date.
–List All Contents—Lists all completed ingests sorted by content name. This option is available only if the number of completed ingests is less than 100.
–Content Status—Lists status information for each completed ingest.
–Content Status (Damaged Only)—Lists status information only for damaged completed ingests.
After you perform one of these methods, a list is displayed. The list of content objects can span several pages. To view the next page, click the page number.
Figure 5-4 shows an example of the Completed Ingests list generated with any of the methods, except the Content Status options. The content names and the date they were ingested are displayed.
Figure 5-4 Completed Ingests List
Figure 5-5 shows an example of the Content Status information that displays when you choose Content Status or Content Status (Damaged Only).
Figure 5-5 Completed Ingests List—Content Status
Table 5-3 Content Status Fields
Field
|
Description
|
Content Name
|
Name of the content.
|
Duration
|
Duration of the content.
|
GOID
|
Global Object ID for the content object associated with the content.
|
Version
|
Trick-play speed of the content object. The value, IGate, refers to an index file, which allows for the jumps between trick speeds, and so on. The value, redo, is an undo file. When the content is ingested, if there are any changes made during the ingest, the changes are recorded in the redo file. If the content is sent by using FTP Out, the changes are undone and the original file is sent.
|
Server ID
|
Server ID of the Vault that is storing the content object.
|
Status
|
Status of the storing process of the content object, either complete or partial.
|
Step 3 To view the details of a content object, click the content name. The Ingest Details are displayed (Figure 5-6).
Figure 5-6 Completed Ingests—Ingest Details
Table 5-4 describes the content object details that are displayed for each type of ingest.
Table 5-4 Content Object Details
Field
|
Description
|
Content Name
|
Name of the content object.
|
Factory ID
|
Factory responsible for this content object.
|
Ingest IP
|
TIP address for the ingest interface on the Vault used to download the content.
|
File Size
|
File size, in bytes, of this content object.
|
Rate
|
Rate of ingest in bits per second (3750000 = 3.75 Mbps).
|
Create Time
|
Time and date this content object was created.
|
Last Modified
|
Time and date this content object was last modified.
|
Op State
|
Operational state of this content object. The possible operational states are:
•Created—Content is loading.
•In Service—Content is available for streaming.
•Out of Service—Content is not available for streaming.
|
Admin State
|
Administrative state of this content object. The possible administrative states are:
•Unprovisioned—Content is loading.
•In Service—Content is available for streaming.
•Out of Service—Content is not available for streaming.
|
Push Provision
|
Type of FTP provisioned. The provision types are:
•FTP pull
•FTP push
|
Encrypted
|
Whether the content object is encrypted or not; Yes means encrypted and No means not encrypted.
|
Content Copies
|
These fields display the following information about the copies of the content:
Speed/Direction—The trick-mode speed and direction (fast forward or rewind).
Server ID—The server ID where the copy is stored.
Status—The status of the stored content.
|
Trick Speeds [1-8]
|
Trick speeds 1 through 8 show the trick-mode speeds for this content object.
|
Full Content ID
|
Full identification of this content object.
|
Ior1
|
Interoperable Object Reference (IOR) for this content object.
|
URL1
|
Uniform Resource Locator (URL) address of the content has the following:
•Protocol used (for example, FTP)
•Username and password (for example, videolan:mpeg4ftp)
•IP address of the content provider server (for example, 192.168.100.184)
•Directory where the content is stored on the provider server (for example, videolan)
•Name of the file (for example, long_encore_3.75.mpg)
|
Asset Ior1
|
Asset IOR associated with this content object.
|
To delete the completed ingest, click Delete.
Viewing Other Ingests
To view the details of active, provisioned, unprovisioned, and failed ingests, do the following:
Step 1 Choose Monitor > System Level from any page in the CDSM, and then click the link for the type of content object you want to view:
•Active Ingests
•Provisioned Ingests
•Unprovisioned Ingests
•Failed Ingests
Step 2 Choose a content object from the drop-down list and click Display. The details of the content object are displayed. Figure 5-6 shows an example of the ingest details.
By typing the first character of the content object name, you can jump to that section of the list.
In addition, you can perform a text string search by typing the text string you want to search for in the Search Ingests field and clicking Search. A list of content objects that contain the text string are listed. To see the content object details, click the content object name listed. To return to the previous page without selecting a content object, click Back.
Table 5-4 describes the content object details that are displayed for each type of ingest.
Note The Unprovisioned Ingests page displays only the content name and the date the content object was considered unprovisioned.
Package Expirations
Note Package Expirations are part of the optional Ingest Manager feature. This option is listed only on the Monitoring System Level left-panel menu if the Ingest Manager is included in your deployment.
To view the details or adjust the license expiration of a package expiration, do the following:
Step 1 Choose Monitor > Package Expiration. The Package Expiration page is displayed.
Step 2 From the Available Packages drop-down list, choose a package and click Display. The Package Expiration details are displayed.
By typing the first character of the package name, you can jump to that section of the list.
Figure 5-7 shows an example of the Package Expiration details.
Figure 5-7 Package Expiration Page
Table 5-5 describes the package expiration details that are displayed.
Table 5-5 Package Expiration Details
Field
|
Description
|
Package Name
|
Name of the package.
|
License Expiration
|
Date the package expires.
|
Additional Package Window
|
Additional time added to the package.
|
Actual Package Expiration
|
Actual Package Expiration is the License Expiration plus the Additional Package Window.
|
Step 3 To adjust the license expiration, enter the number of days (positive or negative) in the Adjust License Expiration field and click Update.
The license expiration is adjusted by the number of days you entered. The Additional Package Window is not affected and is still applied to create the Actual Package Expiration.
Step 4 In the Asset Details section, to view the metadata associated with the package expiration, click the plus sign (+) next to the metadata you want to view.
To delete a package, choose the package from the Available Packages drop-down list, click Display, and then click Delete in the Package Expiration Details section.
Publish Failures
Note Publish Failures are part of the optional Ingest Manager feature. This option is listed only on the Monitoring System Level left-panel menu if the Ingest Manager is included in your deployment.
The Publish Failures page lists the packages that were not able to be published to the backoffice and provides a mechanism to republish the package.
To publish an unpublished package, or delete an unpublished package, do the following:
Step 1 Choose Monitor > Publish Failures. The Publish Failures page is displayed.
Step 2 From the Unpublished Packages drop-down list, choose a package and click Display. The Unpublished Package details are displayed.
By typing the first character of the package name, you can jump to that section of the list.
Figure 5-7 shows an example of the Publish Failures details.
Figure 5-8 Publish Failures Page
Table 5-6 describes the publish failures details that are displayed.
Table 5-6 Publish Failures Details
Field
|
Description
|
Package Name
|
Name of the package.
|
Source URL
|
Location of the original package information.
|
Target URL
|
Location where to place the package information.
|
Module Type
|
Internal identifier for what failed. In the example in Figure 5-8, the publishing of the package failed.
|
Creation Date
|
Date the database record was created for this failure.
|
Step 3 In the Assets section, to view the metadata associated with the unpublished package, click the plus sign (+) next to the metadata you want to view.
Step 4 To restart the publishing process and set the creation date to today, click Publish.
To delete an unpublished package, click Delete.
Monitoring Stream Objects
The monitored stream objects consist of:
•Stream Monitor
•Stream Failures
Stream Monitor
To view the details of stream objects, do the following:
Step 1 Choose Monitor > Stream Monitor. The Stream Monitor page is displayed.
Step 2 The following methods can be used to display a stream object or a list of stream objects:
•Enter the first character of the session ID in the text box. A drop-down list of stream objects is displayed. If there are more than 25 objects that start with that first character you entered, you are prompted to continue entering the next character of the object name or choose one that is listed. You can continue to enter characters to reduce the list (you can also delete characters to increase the list) and at any point choose one. After you choose one, the stream details are displayed (Figure 5-10).
•In the Quick Lists box, the following options are offered:
–Most Recent 100—Lists the 100 most recent stream objects.
–All Streams—Lists all streams. This option is available only if the number of streams is less than 100.
–Subnet Address (or QAM IP) and STB MAC—If you do not know the session ID, you can perform a text string search by selecting the field you want to search on, entering the complete text string (for example, the full IP address) in the Search String field, and clicking Search. A list of stream objects that match the text string in the field you selected are listed. Click the linked field (session ID as seen in Figure 5-9) to see the stream monitor details for the selected object, or click Back to return to the previous page.
After you perform one of the Quick List methods, a list is displayed. The list of stream objects can span several pages. To view the next page, click the page number.
Figure 5-9 shows an example of the stream object list generated by clicking Most Recent 100 or All Streams. The Session handles and the stream start time are displayed.
The stream object list generated by entering the Subnet Address, QAM IP, or STB MAC in the Quick List box, displays the session handle, stream start time, QAM IP and port (or subnet address if Stream Destination is enabled), service group, an TSID in and out if applicable.
Figure 5-9 Stream Monitor—Stream List
Step 3 To view the details of a stream object, click the session ID. The Stream Details are displayed.
Figure 5-10 shows an example of the stream object details.
Figure 5-10 Stream Monitor—Stream Details
Table 5-7 describes the stream details.
Table 5-7 Stream Details
Field
|
Description
|
Stream ID
|
Internal unique identifier assigned to the stream session by the Streamer.
|
Content ID
|
Content identifier of the stream object.
|
LSCP IP
|
Source IP address of the LSCP transaction (set-top box).
|
Session ID
|
Session ID of the stream object.
|
Create Time
|
Date and time the stream object was created.
|
Last Modified
|
Date and time the stream object was last modified.
|
TSID out
|
The output transport stream identification on the associated MQAM device.
Note This field is applicable only when Streaming Mode is set to ASI.
|
Service Group
|
Service group that the stream object is transmitting on.
|
QAM IP
|
IP address of the QAM device delivering this stream object.
Note If Stream Destination is enabled, this field displays the subnet address.
|
QAM Port
|
Port the QAM device is using to receive the stream object.
Note If Stream Destination is enabled, this field does not display.
|
Program Number
|
This field is applicable only when Streaming Mode is set to ASI.
The numerical MPEG program number for this stream object.
|
Bandwidth Used
|
Transport stream bandwidth, in bytes, required for this stream object.
|
Op State
|
Operational State indicates the state of the object. The possible states are:
•InService—Stream object is functioning.
•OutOfService—Occurs when the entity using the object wants to temporarily stop the object.
•Created—Stream object not yet provisioned.
•Destroyed—Stream object is destroyed.
|
Admin State
|
Administrative state of this stream object. The possible administrative states are:
•Unprovisioned—Stream is loading.
•InService—Stream is available for streaming.
•OutOfService—Stream is not available for streaming.
|
Stream State
|
Stream state originates from the LSCP server and has the following modes:
•Open—The server is not transporting a media stream.
•Pause—The server is not transporting a media stream.
•Search Transport—The server is searching for start normal play time (NPT). When at start NPT, it enters Transport mode.
•Transport—The server is transporting the media stream and pauses at the end of the stream. If scale is positive, indicating a forward direction, end of stream is the end of media. If scale is negative, indicating a reverse direction, end of stream is the beginning of media.
•Transport Pause—The server is transporting the media stream and pauses at stop NPT.
•Search Transport Pause—The server is searching for start NPT. When at start NPT, it enters transport pause mode.
•Pause Search Transport—The server is transporting the media stream. It does so until stop NPT, and then transitions to search transport mode.
•End of Stream—The server is not transporting a media stream.
|
Speed Direction
|
Speed direction is as follows:
•Play
•Not playing (Pause)
•n fast-forward, where n means n times fast-forward
•-n means, where n means n times rewind
|
Provision Multiple
|
Provision Multiple is enabled if this field is "yes" and disabled if this field is "no."
|
The Graph Stream button displays the trick-mode activity of the stream (Figure 5-11).
Note If Trick Mode Capture is disabled, the Graph Stream is not displayed. For information on enabling the Trick Mode Capture, see the "Trick Mode Capture" section on page E-5.
Figure 5-11 Stream Activity Report
To delete a stream object, display the object and click Delete.
Stream Failures
To view the details of Failed Streams, do the following:
Step 1 Choose Monitor > Failed Streams. The Stream Failures page is displayed.
Each failed stream is listed by the date and time the stream failed, followed by the session ID of the failed stream.
Note Stream Failure monitoring displays only the failed streams for the current day. To view past stream failures, see the "Stream Failures" section on page 6-19.
Step 2 From the Stream Failures drop-down list, choose the timestamp and session ID of the stream object and click Display. The stream failure details are displayed. See Figure 5-12.
To delete a failed stream, display the object and click Delete.
Figure 5-12 Stream Failures Page
Table 5-8 describes the stream failure details.
Table 5-8 Stream Failure Details
Field
|
Description
|
Session ID
|
Session ID of the failed stream.
|
Failure Date
|
Date and time the failure occurred.
|
QAM IP
|
IP address of the QAM device associated with the failure.
|
Service Group
|
Service group associated with the failure.
|
Server ID
|
Server responsible for streaming this stream object. To view the IP address associated with the Server ID, see the "Configuring the Servers" section on page 4-81.
|
Group ID
|
All servers that are part of the same CDS system (managed by one CDSM) have the same Group ID. This Group ID corresponds to the CDSM GUI array ID and should be unique across an enterprise. Table 5-9 describes the ID mapping between the CDSM GUI and the CServer.
|
Failed Operation
|
Operation that was taking place when the stream failed, for example, createStream, LSCP Command(), or createServant, destroy. These are the measurement points or transactional states of the system at the time of the failure.
|
Failed Task
|
Failed task is the event category that provides the type of execution sequence that the call stack was currently in at the time of the failure. The list of the high-level categories are:
|
| |
•Tune In
•Load Application
•Load Catalog
•Eligibility Check
•Select Subscription
•Purchase Subscription
•Select Movie
•Purchase Check
|
•Play Movie
•Movie Setup
•Movie Control
•Movie Confirm
•Purchase Confirm
•Purchase Log
•Stop Movie
•Movie Release
|
Error Code
|
Error code provides a description of the event that caused an error. See Table 5-10, Table 5-11, and Table 5-13 for descriptions of the error codes.
|
Table 5-9 lists the CDSM GUI ID names and maps them to the CServer names in the setupfile and .arroyorc files.
Table 5-9 ID Names in the CDSM GUI and CServer Files
CDSM GUI ID Name
|
CServer Files ID Name
|
Array ID on the Array Name page
|
groupid
|
Group ID on the Server-Level pages
|
groupid
|
Stream Group ID on the Server Setup page
|
arrayid
|
Cache Group ID on the Server Setup page
|
arrayid
|
Vault Group ID on the Server Setup page
|
arrayid
|
Stream Group ID on the Configuration Generator page
|
arrayid
|
Table 5-10 lists the Managed Services Architecture (MSA) error codes. Some MSA monitored events are monitored for the CDS as well, and are prefaced by "AVS_" instead of "MSA_." They are denoted with a footnote in the table. Some MSA monitored events are not errors, but rather information about an event.
Table 5-10 MSA Error Codes
Numeric Error Code
|
Error Code
|
Description
|
5001
|
MSA_INT_ERR1
|
There is an internal error. INT_ERR has a subset of error codes that specifically describe where the error occurred. See Table 5-11.
|
5002
|
MSA_FLOW1
|
Entry or exit of a measured or tracked flow, or some other important check point, and is recorded as non-realtime.
|
5003
|
MSA_CMPT_NOT_EXIST1
|
Component does not exist.
|
5004
|
MSA_REQ_TIMEOUT1
|
Client timed out waiting for a response to a request.
|
5005
|
MSA_CMPT_OUT_OF_SVC1
|
Component is unavailable.
|
5006
|
MSA_REQ_NOT_IMPL1
|
Requested item is not implemented.
|
5007
|
MSA_RES_INVALID1
|
Resource is invalid.
|
5008
|
MSA_RES_DUP1
|
Duplicate resource is being added to the session.
|
5009
|
MSA_CMPT_DUP
|
It was determined that a component that was being added has the same name as a previously created component.
|
5010
|
MSA_REQ_CREATE
|
Attempt to create a request failed.
|
5011
|
MSA_UNKNOWN
|
Unclassified or undetermined error occurred.
|
5012
|
MSA_REAP
|
Component is destroyed outside of the normal expected flows.
|
5013
|
MSA_VS_LSC_TIMEOUT
|
Video server timed out waiting for the client to issue a resume or play command after the initial creation of the stream.
|
5014
|
MSA_SVC_GROUP_MISSING
|
Request contains a missing service group.
|
5015
|
MSA_RES_NO_CAPACITY
|
Resource is currently out of capacity and cannot satisfy the request.
|
5016
|
MSA_RES_NO_BANDWIDTH
|
Resource does not have the bandwidth to deliver the stream.
|
5017
|
MSA_REQ_FAIL
|
Request failed.
|
5018
|
MSA_RES_UNAVAIL
|
Response is not available.
|
5019
|
MSA_FLOW_RT
|
Entry or exit of a measured or tracked flow, and is recorded in real-time.
|
5020
|
MSA_LSC_SERVER_FAILURE
|
LSC response; server failed.
|
5021
|
MSA_LSC_NO_MEMORY
|
LSC response; dynamic memory allocation failure.
|
5022
|
MSA_LSC_IMPL_LIMIT
|
LSC response; implementation limit exceeded.
|
5023
|
MSA_LSC_NO_RESOURCES
|
LSC response; no resources.
|
5024
|
MSA_LSC_SERVER_ERROR
|
LSC response; server error.
|
5025
|
MSA_LSC_MPEG_DELIVERY
|
LSC response; unable to deliver MPEG stream.
|
5026
|
MSA_LSC_ERR
|
Generic DSM-CC error event.
|
5027
|
MSA_LSC_BAD_REQUEST
|
LSC response; invalid request.
|
5028
|
MSA_LSC_BAD_STREAM
|
LSC response; invalid stream handle.
|
5029
|
MSA_LSC_WRONG_STATE
|
LSC response; wrong state.
|
5030
|
MSA_LSC_UNKNOWN
|
LSC response; unknown error.
|
5031
|
MSA_LSC_NO_PERMISSION
|
LSC response; client does not have permission for the request.
|
5032
|
MSA_LSC_BAD_PARAM
|
LSC response; invalid parameter.
|
5033
|
MSA_LSC_NO_IMPL
|
LSC response; not implemented.
|
5034
|
MSA_LSC_TRANSIENT
|
LSC response; transient error.
|
5035
|
MSA_LSC_BAD_SCALE
|
LSC response; incorrect scale value.
|
5036
|
MSA_LSC_BAD_START
|
LSC response; stream start time does not exist.
|
5037
|
MSA_LSC_BAD_STOP
|
LSC response; stream stop time does not exist.
|
Table 5-11 lists the error codes for internal errors and external errors. Internal errors are errors that occurred in the CDS and specifically describe where the error occurred. External errors are errors that occurred in the network or network components, which includes the ContentStore, StreamService, and so on. The error codes listed in Table 5-11 provide more detail to the MSA_INT_ERR or AVS_INT_ERR error code.
Table 5-11 INT_ERR Error Codes
Numeric Error Code
|
Error Code
|
Description
|
1001
|
INGEST_THREADS_NOT_RUNNING
|
Cache server threads are not running.
|
1002
|
INGEST_NIC_DOWN
|
Ingest interface is disabled.
|
1003
|
INGEST_DATA_BLOCKAGE
|
Ingest data read is backlogged, causing data socket blockage.
|
1004
|
BAD_CONTENT
|
Content data is not recoverable.
|
1005
|
NOT_ENOUGH_NIC_BANDWIDTH
|
Not enough bandwidth left over on NICs to perform the operation.
|
1006
|
NOT_ENOUGH_SYSTEM_RESOURCES
|
Not enough system resources left to perform the operation.
|
1007
|
NOT_ENOUGH_DISK_SPACE_AVAILABLE
|
Not enough disk space available.
|
1008
|
STREAMER_MAX_SLOTS_LIMIT_EXCEEDED
|
No stream slot available to allocate the stream.
|
1009
|
REMOTE_VAULT_DOWN
|
Remote Vault is not responding.
|
1010
|
REMOTE_STREAMER_DOWN
|
Remote Streamer is down.
|
1011
|
VAULT_DISK_BAD
|
Disk is bad on a Vault.
|
1012
|
STREAMER_DISK_BAD
|
Disk is bad on a Streamer.
|
1013
|
CONTENT_LOCATE_FAILED
|
Cannot locate the content on any Vault.
|
1014
|
CONTENT_FILL_FAILED
|
Cannot push content to the Streamer from Vault.
|
1015
|
NOT_ENOUGH_FILL_BANDWIDTH
|
Not enough fill bandwidth available.
|
1016
|
FILL_LINKS_DOWN
|
Fill links are down.
|
1017
|
STREAMING_LINKS_DOWN
|
Stream links are down.
|
1018
|
VAULT_MIRRORING_SITE_DOWN
|
Mirroring site of the Vault array is down.
|
1019
|
SET_CONTENT_BUNDLE_FAILED
|
Set content bundle descriptor array failed.
|
1020
|
SET_DESTINATION_FAILED
|
Set destination of stream failed.
|
1021
|
DESTROY_STREAM_FAILED
|
Destroy stream failed.
|
1022
|
PLAY_STREAM_FAILED
|
Play stream failed.
|
1023
|
FILLCB_FAILED
|
Fill CB failed.
|
1024
|
WAIT_FOR_FTP_DATA_DONE_FAILED
|
Wait for FTP data done failed.
|
1025
|
GET_CURRENT_NPT_FAILED
|
Get current NPT for LSCP status failed.
|
2001
|
CAN_NOT_CONNECT_TO_NAME_SERVICE
|
Cisco ISA cannot connect to the BMS Naming Server.
|
2002
|
CAN_NOT_CONNECT_TO_NOTIFY_SERVICE
|
Cisco ISA cannot connect to the Notify Server.
|
2003
|
CAN_NOT_CREATE_EVENT_CHANNELS
|
Cisco ISA cannot create event channels.
|
2004
|
NO_CONTENT_EVENT_CHANNEL_FOUND
|
Naming server does not have content event channel.
|
2005
|
NO_STREAM_EVENT_CHANNEL_FOUND
|
Naming server does not have stream event channel.
|
2006
|
EVENT_CHANNEL_OBJECT_NOT_EXISTS
|
Event channel object does not exist in Notify Server.
|
2007
|
CORBA_CONNECTION_FAILED
|
CORBA System exception while connecting to other entity.
|
2008
|
CORBA_BROKEN_PIPE
|
CORBA system exception with broken pipe with other entity.
|
2009
|
CORBA_CONTENT_STORE_BIND_FAILED
|
CORBA bind exception while starting Cisco ContentStoreFactory.
|
2010
|
CORBA_STREAM_SERVICE_BIND_FAILED
|
CORBA bind exception while starting Cisco StreamService.
|
2011
|
CORBA_TRANSIENT_ERROR
|
CORBA system exception with object being transient.
|
2012
|
CORBA_TIMEOUT_ERROR
|
CORBA timeout exception.
|
2013
|
CORBA_IOR_NIL
|
Orb object is nil.
|
2014
|
CORBA_IOR_NIL_AFTER_NARROW
|
Orb object is nil after narrow.
|
2015
|
CORBA_SYSTEM_ERROR
|
CORBA system exception while connecting to the servant.
|
2021
|
SERVICE_GROUP_NOT_SUPPORTED
|
Service group is not supported.
|
2022
|
REMOTE_CONTENT_STORE_FACTORY_DOWN
|
Remote Cisco ContentStoreFactory is down.
|
2023
|
VAULT_HAS_FULL_LOAD
|
Vault is running with full load.
|
2024
|
FTP_CONNECTION_FAILED
|
Connection to FTP server failed.
|
2025
|
FTP_SERVER_BIND_FAILED
|
FTP server can not bind to the port.
|
2026
|
FTP_PUSH_TIMEOUT
|
FTP push timeout (PASV is not served fast enough).
|
2027
|
FTP_QUIT_RECEIVED_DURING_INGEST
|
FTP server received QUIT request.
|
2028
|
NO_LSCP_SET_TOP_CONNECTION
|
LSCP server to set-top box connection is down.
|
2029
|
LSCP_SERVER_BIND_FAILD
|
LSCP server cannot bind to the running port.
|
2030
|
LSCP_PROXY_BIND_FAILED
|
LSCP proxy cannot bind to the running port.
|
2031
|
STREAMER_GROUP_MAX_LIMIT_EXCEEDED
|
Stream count is exceeding the limit for the Streamer group.
|
2032
|
STREAMER_MAX_LIMIT_EXCEEDED
|
Stream count is exceeding the limit for a Streamer.
|
2033
|
REMOTE_STREAMER_NOT_RESPONDING
|
Remote Streamer is down.
|
2034
|
NOT_ENOUGH_MQAM_BANDWIDTH
|
Not enough MQAM bandwidth.
|
2035
|
NO_QAM_FOR_SERVER_ID
|
Server is not connected to any QAM.
|
2036
|
NOT_ENOUGH_QAM_BANDWIDTH
|
Not enough QAM bandwidth.
|
2037
|
STREAMER_IS_NOT_IN_THE_SERVICE_GROUP
|
Streamer is not in the service group.
|
2038
|
STREAMER_HAS_FULL_LOAD
|
Streamer is running with full load.
|
2039
|
STREAMER_IS_NOT_CONNECTED_THAT_QAM
|
Server is not connected to the QAM.
|
2040
|
INVALID_SERVICE_GROUP
|
Service group is not returned by session gateway.
|
2041
|
CONTENT_CAN_NOT_BE_LOCATED
|
Content is not found in the related content stores.
|
2042
|
CONTENT_OBJECT_NOT_YET_PROVISIONED
|
Content object is not yet provisioned.
|
2043
|
STREAM_OBJECT_NOT_YET_PROVISIONED
|
Stream object is not yet provisioned.
|
2044
|
STREAM_OBJECT_IS_OUT_OF_SERVICE
|
Stream object is out of service.
|
2045
|
STREAM_OBJECT_IS_ALREADY_PROVISIONED
|
Stream object is already in service.
|
2046
|
CONTENT_OBJECT_IS_ALREADY_PROVISIONED
|
Content object is already in service.
|
2047
|
STREAM_SERVANT_OBJECT_NOT_EXIST
|
Remote streamer does not have a servant for stream object.
|
2048
|
NO_DESTINATION_QAM_IP_FOUND
|
No QAM IP Address is received for the stream destination.
|
2049
|
NO_DESTINATION_QAM_PORT_FOUND
|
No QAM port is received for the stream destination.
|
2050
|
FAILED_TO_SET_STREAM_DESTINATION
|
Some error occurred while setting the stream destination.
|
2051
|
UNABLE_TO_ACCEPT_CONNECTION
|
Cannot accept more TCP connections.
|
2052
|
UNABLE_TO_REGISTER_EVENT_HANDLER
|
Cannot register event handle to serve the TCP connection.
|
2053
|
CAN_NOT_LOCATE_QAM_IP_FOR_TSID_IN
|
Cannot locate the QAM IP addresses associated with TSID IN.
|
2099
|
NS_LOG_MONITOR_ERROR
|
Ns_log file is not updating. Restart the ISA service.
|
2100
|
AVS_ISA_GENERIC_ERROR
|
Some unknown error occurred during execution of the operation.
|
3001
|
CACHE2APP_INITIALIZE_ERROR
|
Failed to initialize Cache2App library.
|
3002
|
FILLCB_FAILED
|
FillCB failed during content ingest.
|
3003
|
DESTROYCB_FAILED
|
Failed while destroying the content bundle descriptor.
|
3004
|
WAIT_FOR_FTP_DATA_DONE_FAILED
|
API wait for FTP data done has returned an error.
|
3005
|
SET_CONTENT_BUNDLE_DESCRIPTOR_FAILED
|
Failed while setting content bundle descriptor for a stream.
|
3006
|
SET_DESTINATION_FAILED
|
Failed while setting destination of the stream.
|
3007
|
SET_ENCRYPTION_KEY_FAILED
|
Failed while setting ECM keys for the stream.
|
3008
|
CREATE_STREAM_FAILED
|
AVS cache server cannot allocate the stream handle.
|
3009
|
DESTROY_STREAM_FAILED
|
AVS cache server cannot tear down the stream.
|
3010
|
DESTROY_REMOTE_STREAM_FAILED
|
AVS cache server cannot tear down stream on remote server.
|
3011
|
PLAY_STREAM_FAILED
|
AVS cache server cannot play the stream.
|
4001
|
DATABASE_DOWN
|
Database is down.
|
4002
|
DATABASE_SYNCHRONIZING_REPLICATION_Q
|
Database is synchronizing with replication queue.
|
4003
|
DATA_IS_NOT_IN_SYNC
|
Database is not in sync with master.
|
4004
|
DATABASE_RETURNED_ERROR
|
Database has returned an error; may be there is no record found.
|
4005
|
DATABASE_RECORD_NOT_FOUND
|
Record is not found in the database.
|
4006
|
DATABASE_CAN_NOT_INSERT_RECORD
|
Record cannot be inserted into the database.
|
4007
|
DATABASE_CAN_NOT_DELETE_RECORD
|
Record cannot be deleted from the database.
|
4008
|
DATABASE_CAN_NOT_UPDATE_RECORD
|
Record cannot be updated.
|
4009
|
DATABASE_QUERY_SEND_ERROR
|
Failed to make a query to the database.
|
Table 5-12 lists the error codes for errors that could occur during ingest or during trick-mode file creation, which cause stream failures.
Table 5-12 MPEG Error Codes
Numeric Error Code
|
Error Code
|
Description
|
8001
|
TRICK_INGEST_NO_INGEST_OBJECTS
|
Ingest fails. Check available system memory.
|
8002
|
TRICK_INGEST_TOO_MANY_SPEEDS
|
Too many trick speeds. Change trick speed configuration.
|
8003
|
TRICK_INGEST_NULL_INGEST_OBJECTS
|
Ingest fails. Check available system memory.
|
8004
|
TRICK_INGEST_INVALID_SPEED_DENOMINATOR
|
Ingest fails. Change trick speed configuration.
|
8005
|
TRICK_INGEST_INVALID_SPEED_LT_2X
|
Ingest fails. Change trick speed configuration.
|
8006
|
TRICK_INGEST_NULL_DERIVED_INGEST_OBJECTS
|
Ingest fails. Check available system memory.
|
8007
|
TRICK_RSDVR_DYNAMIC_TRICK_CREATION
_FAILS
|
RS-DVR trick-mode file creation fails. Change trick speed configuration.
|
8008
|
TRICK_INGEST_CDN_AVC_UNSUPPORTED
|
Do not attempt to ingest an Advanced Video Coding (AVC) stream on a VVI system.
|
8009
|
TRICK_RSDVR_BAD_STREAM_TYPE
|
RS-DVR trick-mode file creation fails. Stream should already have failed ingest.
|
8010
|
TRICK_INGEST_ABORTED
|
General ingest failure. Check ingest feed.
|
8011
|
TRICK_INGEST_INSUFFICIENT_DATA
|
Ingest fails. Check ingest feed.
|
8012
|
TRICK_INGEST_STREAM_TOO_BIG
|
Ingest fails. The limit is 162 GB or about 12 hours at a known bitrate.
|
8013
|
TRICK_RSDVR_WRITE_OVERFLOW
|
RS-DVR trick-mode file creation fails. Check ingest feed.
|
8014
|
TRICK_INGEST_VBR_UNSUPPORTED
|
Ingest fails. Check ingest feed.
|
8015
|
TRICK_INGEST_RATE_FORCED
|
Streaming rate may be incorrect. Check ingest feed.
|
8016
|
TRICK_INGEST_PAT_NOT_FOUND
|
Program association table (PAT) not found. Check ingest feed.
|
8017
|
TRICK_INGEST_DEFAULTING_PMT_PID
|
Program map table (PMT) process ID (PID) not determined. Check ingest feed.
|
8018
|
TRICK_INGEST_DEFAULTING_PROGRAM_NUMBER
|
Program number not determined. Check ingest feed.
|
8019
|
TRICK_INGEST_DEFAULTING_VIDEO_PID_AND_
TYPE
|
Video PID or type not determined. Check ingest feed.
|
8020
|
TRICK_INGEST_BITRATE_INDETERMINATE
|
Bitrate cannot be determined. Check ingest feed or adjust ingest configuration parameters.
|
8021
|
TRICK_INGEST_FIRST_PTS_NOT_FOUND
|
First presentation time stamp (PTS) not determined. Check ingest feed.
|
8022
|
TRICK_INGEST_CANNOT DETERMINE_FRAMERATE
|
Frame rate not determined. Check ingest feed.
|
8023
|
TRICK_INGEST_PMT_NOT_FOUND
|
PMT not found. Check ingest feed.
|
8024
|
TRICK_INGEST_MULTIPLE_VIDEO_PIDS
|
Multiple video PIDs found. Check ingest feed.
|
8025
|
TRICK_INGEST_PID_REPLACEMENT_CANCELLED
|
PIDs could not be standardized. Check ingest feed.
|
8026
|
TRICK_INGEST_OVERFLOW
|
Ingest fails because of ring buffer overflow. Check ingest feed.
|
8027
|
TRICK_INGEST_WRITE_ERROR
|
Ingest fails because of a 1x write problem. Check ingest feed.
|
8028
|
TRICK_INGEST_OVERFLOW_ON_RETRY
|
Ingest fails even after a retry (ring buffer overflow). Check ingest feed.
|
8029
|
TRICK_INGEST_KNOBS_FAILURE
|
Ingest fails. Check ingest feed or adjust ingest configuration parameters.
|
8030
|
TRICK_INGEST_KNOBS_FAILURE_PAT_PMT
|
Ingest fails. No PAT or PMT found. Check ingest feed or adjust ingest configuration parameters.
|
8031
|
TRICK_INGEST_KNOBS_FAILURE_BITRATE
|
Ingest fails. Bitrate cannot be computed. Check ingest feed or adjust ingest configuration parameters.
|
8032
|
TRICK_INGEST_KNOBS_FAILURE
_DISCONTINUITIES
|
Ingest fails. Too many discontinuities. Check ingest feed or adjust ingest configuration parameters.
|
8033
|
TRICK_INGEST_KNOBS_FAILURE_CONTINUITY
_COUNTERS
|
Ingest fails. Too many continuity counter errors. Check ingest feed or adjust ingest configuration parameters.
|
8034
|
TRICK_INGEST_KNOBS_FAILURE_SYNC
|
Ingest fails. Too many sync errors. Check ingest feed or adjust ingest configuration parameters.
|
8035
|
TRICK_INGEST_KNOBS_FAILURE_SYNC_TIME
|
Ingest fails. Sync loss too long. Check ingest feed or adjust ingest configuration parameters.
|
8036
|
TRICK_INGEST_KNOBS_FAILURE_PIC_GAPS
|
Ingest fails. Too many picture gaps. Check ingest feed or adjust ingest configuration parameters.
|
8037
|
TRICK_INGEST_KNOBS_FAILURE_PIC_GAP_TIME
|
Ingest fails. Picture gap too long. Check ingest feed or adjust ingest configuration parameters.
|
8038
|
TRICK_INGEST_SEQUENCE_HEADER_NOT_FOUND
|
Ingest fails. Could not find a Sequence Header. Check ingest feed.
|
8039
|
TRICK_INGEST_SPS_NOT_FOUND
|
Ingest fails. Could not find an SPS. Check ingest feed.
|
8040
|
TRICK_INGEST_CDN_SEQ_WRITE_FAILED
|
Ingest fails because of a Sequence Header write error. Check ingest feed.
|
8041
|
TRICK_INGEST_CDN_NONCONFORMAL
_FRAME_START
|
VVI: Ingest fails. Invalid frame start. Check ingest feed.
|
8042
|
TRICK_INGEST_SPLIT_SEQEND_SEQ_PAIR
|
Ingest fails. SequenceEnd/SequenceHeader pair not consecutive. Check ingest feed.
|
8043
|
TRICK_INGEST_PIC_SIZE_CHANGED
|
Ingest fails. Picture size changed. Check ingest feed.
|
8044
|
TRICK_INGEST_PIC_SIZE_H_OR_V_ZERO
|
Ingest fails. Picture size H or V zero. Check ingest feed.
|
8045
|
TRICK_INGEST_HORIZONTAL_PIC_SIZE_EXCEEDS_MAX
|
Ingest fails. Horizontal size exceeds max (1920). Check ingest feed.
|
8046
|
TRICK_INGEST_VERTICAL_PIC_SIZE_EXCEEDS
_MAX
|
Ingest fails. Vertical size exceeds max (1088). Check ingest feed.
|
8047
|
TRICK_INGEST_SEQUENCE_HEADER_CHANGED
|
VVI: Ingest fails. Sequence Header changed. Check ingest feed.
|
8048
|
TRICK_INGEST_SEQUENCE_HEADER_CHANGE_NO_SEQEND
|
Ingest fails. Sequence Header changed with no preceding Sequence End.
|
8049
|
TRICK_INGEST_SEQUENCE_HEADER_CHANGE
_BAD_PRIOR_STARTCODE
|
Ingest fails. Sequence Header changed with no immediately preceding Sequence End.
|
8050
|
TRICK_INGEST_SEQUENCE_HEADER_CHANGE
_NO_PRIOR_STARTCODE
|
Ingest fails. Sequence Header changed with no preceding start code
|
8051
|
TRICK_INGEST_ILLEGAL_FRAMERATE
|
Illegal frame rate code. Check ingest feed.
|
8052
|
TRICK_INGEST_CDN_ILLEGAL PES_PACKETISATION
|
VVI: Ingest fails. Illegal PES packetization. Check ingest feed.
|
8053
|
TRICK_INGEST_CDN_STREAM_STARTS_WITH
_P_FRAME
|
VVI: Ingest fails: Stream begins with a P-frame. Check ingest feed.
|
8054
|
TRICK_INGEST_CDN_STREAM_STARTS_WITH
_B_FRAME
|
VVI: Ingest fails: Stream begins with a B-frame. Check ingest feed.
|
8055
|
TRICK_INGEST_ZERO_BITRATE
|
Check ingest feed. Bitrate indeterminate.
|
8056
|
TRICK_INGEST_CDN_STREAM_STARTS_WITH
_BAD_I_FRAME
|
VVI: Ingest fails: Stream begins with a malformed I-frame. Check ingest feed.
|
Table 5-13 lists the Managed Services Architecture (MSA) error codes for the optional Ingest Manager feature.
Table 5-13 MSA Error Codes for the Optional Ingest Manager Feature
Numeric Error Code
|
Error Code
|
Description
|
7000
|
MSA_BAD_XML
|
There was an XML parsing error. Check the ADI XML for errors.
|
7001
|
MSA_BAD_REQUEST
|
Request for content was bad. Check the target backoffice URL.
|
7002
|
MSA_UNKNOWN_HOST
|
Host is unknown. Check the target backoffice URL.
|
7003
|
MSA_CONNECTION_DROP
|
The connection was dropped. Check the URL. The Ingest Manager possibly misformatted the ADI XML.
|
7004
|
MSA_BACKOFFICE_TIMEOUT
|
The backoffice did not respond within the allowed time interval.
|
7005
|
MSA_UNKNOWN
|
Unknown error occurred. Check the /home/isa/bss/log/aim.log.
|
7006
|
MSA_FAILED_POST
|
Ingest Manager failed to post the ADI to the backoffice.
|
7007
|
MSA_PKG_EXPIRED
|
Package has expired and the retry record is removed.
|
Array Level Monitoring
The Array Level Monitoring pages provide an overall view of the health and activity of a specified array, as well as provide monitoring and deletion of barker streams. The Array Level links are:
•Array Snapshot
•Barker Stream Monitor
Array Snapshot
The Array Snapshot page provides an overview of the current activity for the specified array of servers on the CDS. A summary of the state of all streams, content ingests, and disk usage is displayed.
The fields displayed on the Array Snapshot page are the same fields that are displayed on the System Snapshot page, with the active stream bandwidth and active fill bandwidth shown for each Stream Group. For descriptions of the fields, see Table 5-1.
Barker Stream Monitor
Note The Barker Stream feature is optional and is not listed on the Array Level left-panel menu if it is not included in your deployment. The Barker Stream feature is also not available if the Stream Destination is set to IPTV. For more information, see the "Stream Destination" section on page E-4.
The Barker Stream Monitor page lists the barker streams currently configured. Figure 5-13 shows an example of barker streams for ASI streaming.
To delete a barker stream configuration, click the Delete button next to the associated barker stream.
Figure 5-13 Barker Stream Monitor Page
Server Level Monitoring
The Server Level Monitoring pages provide detail information on the health and activity of a Vault or Streamer server.
To view the Server Level Monitoring pages, do the following:
Step 1 Choose Monitor > Server Level, and then click one of the following as applicable:
•Disk Monitor
•NIC Monitor
•Server Vitals
•Cache/Fill Bandwidth
•Services Monitor
Step 2 Choose the IP address of the server from the drop-down list and click Display.
Disk Monitor
The Disk Monitor page provides real-time information on the status of a disk.
To view the current status of a disk, choose the IP address of the server from the drop-down list on the Disk page, click Display, and roll your mouse over one of the disks displayed in the graphic. If the server is a Lindenhurst CDE, click one of the disks displayed in the graphic. Figure 5-14 shows an example of a Streamer server.
To change how often the information is refreshed, enter the number of seconds in the Data Refresh Rate field. The default is 10 seconds. The range is 10 to 300. All field values that are updated, based on the refresh rate, are initially shown in a green font.
Figure 5-14 Disk Monitor Page—Streamer
For Vault servers, the Disk Availability line graph shows the percentage of disk space available. The gigabytes displayed for "Total Space," "Available Space," and "% Used" are the sum of all the disks installed on the Vault server.
The Linux File System Stats table shows the combined total storage space for the partions of the disk drives, the combined available storage space for the partions of the disk drives, and the percentage of used storage for each combined partition. In Figure 5-14, the hda2 partition has an alarm indicator because the usage has exceeded the user-defined threshold of 40 percent. For information on setting thresholds, see the "Setting System Thresholds" section on page 7-7.
Table 5-14 describes the information displayed when a disk is selected.
Table 5-14 Disk Status Fields
Field
|
Description
|
Current Temp
|
Current temperature of the hard disk.
|
Smart Status
|
Self-Monitoring, Analysis and Reporting Technology (S.M.A.R.T.) status of a disk as determined by the manufacturer in accordance with the relevant ATA/SCSI standards. S.M.A.R.T. is logic embedded in the firmware that determines when a disk is going bad.
|
Number of Reads
|
Number of bytes read from the disk since it was powered on.
|
Number of Writes
|
Number of bytes written to the disk since it was powered on.
|
S.M.A.R.T
The CDS incorporates S.M.A.R.T. to monitor the reliability of a hard drive, predict drive failures, and to carry out different types of drive self-tests. S.M.A.R.T is firmware, native to most disk drives, that monitors disk attributes over time, making it possible to perform predictive failure analysis. Advanced warning of predictive failures allows the operator to perform preventative maintenance.
To view the current read/write activity that has occurred in the last five seconds on the selected disk, click Graph Disks. Figure 5-15 shows an example of the Disk Activity graph.
Figure 5-15 Disk Monitor—Streamer Disk Activity Graph
The Disk Activity graph displays an average calculation of the amount of data read (in megabytes per second) and data writes over a five-second period.
NIC Monitor
The NIC Monitor page displays the status of each network interface card (NIC) on the server.
To view the current status of a NIC on a server, choose the IP address of the server from the drop-down list on the NIC Monitor page, click Display, and click one of the NIC ports displayed in the graphic. Figure 5-16 shows an example of the eth4 interface on a Streamer server.
To change how often the information is refreshed, enter the number of seconds in the Data Refresh Rate field. The default is 10 seconds. The range is 10 to 300. All field values that are updated, based on the refresh rate, are initially shown in a green font.
Figure 5-16 NIC Monitor Page—Streamer
Table 5-15 describes the information displayed for a NIC port.
Table 5-15 NIC Port Status Fields
Field
|
Description
|
Port Speed
|
Speed of the interface in megabits per second (Mbps).
|
Admin State
|
Administrative state of the port interface. The administrative state is determined at the time the server is booted. The possible administrative states are up or down.
|
Op State
|
Operational state of the port interface. The operational state is either up or down. If the port is not connected to the network or is malfunctioning, the operational state displayed is down.
|
Media Type
|
Physical conduit of the interface. The physical type is either copper or fiber optic.
|
Poll Interval
|
Number of seconds between each disk polling.
|
Transmit
|
Total number of bytes transmitted since this port has been operational and configured as administratively up.
|
Received
|
Total number of bytes received since this port has been operational and configured as administratively up.
|
To view the average transmit and receive activity that has occurred in the last two seconds for each port on this server, click Graph Ports. Figure 5-17 shows an example of the Port Activity graph.
Figure 5-17 NIC Monitor—Streamer Port Activity Graph
Server Vitals
The Server Vitals page provides current values for monitored system components. Server components are monitored and if a threshold is exceeded, the System Health Monitor page reports the event and an SNMP trap is sent to the Network Management System (NMS).
Note The Server Vitals page is displayed only if the CDSM Health Monitor feature is enabled. For more information, see the "CDSM or VVIM Health Monitoring" section on page E-8.
To view the current values of the monitored components, as well as the threshold settings, choose the IP address of the server from the drop-down list on the Server Vitals page and click Display.
To change how often the information is refreshed, enter the number of seconds in the Data Refresh Rate field. The default is 10 seconds. The range is 10 to 300. All field values that are updated, based on the refresh rate, are initially shown in a green colored font.
To change the temperature format to Fahrenheit, choose ºF for the Temperature Format. The default is Celsius (ºC).
The monitored components are different for each Content Delivery Engine (CDE) model. Figure 5-18 shows an example of the Server Vitals page for a Vault (CDE420).
Figure 5-18 Server Vitals Page
Cache/Fill Bandwidth
The Cache/Fill Bandwidth page displays details on the content caching activity on a Streamer.
To view the caching activity on a server, choose the IP address of the server from the drop-down list on the Cache/Fill Bandwidth page and click Display. Figure 5-19 shows an example.
To change how often the information is refreshed, enter the number of seconds in the Data Refresh Rate field. The default is 10 seconds. The range is 10 to 300. All field values that are updated, based on the refresh rate, are initially shown in a green font.
Figure 5-19 Cache/Fill Bandwidth Page
Table 5-16 describes the services listed in the Cache/Fill Bandwidth page.
Table 5-16 Cache/Fill Bandwidth Fields
Service
|
Description
|
Active Stream Count
|
Number of active streams on this Streamer.
|
Active Stream Bandwidth
|
Bandwidth (in Mbps) used for streaming on this Streamer.
|
Unique Stream Count
|
Number of unique streams on this Streamer.
|
Unique Stream Bandwidth
|
Bandwidth (in Mbps) used for serving unique streams on this Streamer.
|
Fill Receive Stream Count
|
Number of streams on this Streamer that are retrieving content from the Vault to fulfill requests for content.
|
Actual Fill Stream Bandwidth
|
Bandwidth (in Mbps) used on this Streamer for retrieving content from the Vault.
|
Disk Read Stream Count
|
Number of streams on this Streamer sending content that was retrieved from the hard drives on the Streamer.
|
Disk Read Bandwidth
|
Bandwidth (in Mbps) used on this Streamer for retrieving locally stored content (content on the Streamer hard drives).
|
Services Monitor
The Services Monitor page displays whether specific processes are running on a server.
To view the current status of the services running on a server, choose the IP address of the server from the drop-down list on the Services Monitor page and click Display. Figure 5-20 shows an example of a Streamer server.
Figure 5-20 Services Monitor Page—Streamer
Table 5-17 describes the services listed on the Services Monitor page.
Table 5-17 CDS Services
Service
|
Server
|
Description
|
Cisco Cache Server
|
Both
|
Cache server runs on all servers. The Cache server is responsible for the core functions of the CDS.
|
Cisco Content Store Master
|
Vault
|
Content Store Master process is running if you are looking at a master Vault server. The Content Store Master serves as the master Vault process for accepting inbound OpenStream connections.
|
Cisco Content Store Slave
|
Vault
|
Content Store Slave process is running if you are looking at either a master or a slave Vault server. The Content Store Slave handles requests proxied by the Content Store Master.
|
Cisco Stream Service Master
|
Streamer
|
Streamer Service Master process is running if you are looking at a master Streamer server. The Stream Service Master serves as the master Streamer process for accepting inbound OpenStream connections.
|
Cisco Primary Setup Service
|
Streamer
|
Primary Setup Service is running if you are looking at the Streamer server designated as the primary Setup server. The Setup server handles setting up stream sessions.
|
Cisco Stream Control Service
|
Streamer
|
Stream Control Service accepts set-top box play stream commands, for example, LSCP.
|
Cisco Resource Manager
|
Streamer
|
Resource Manager runs on a Streamer server. The Resource Manager handles orphaned streams.
|
Cisco DB Server
|
Both
|
DB (database) server runs on all servers and is responsible for keeping track of all data objects in the CDS.
|
DB Synchronization Status
|
Both
|
Displays the status of the database synchronization among all servers. The states are "OK" and "not OK."
|
Cisco SNMP Server
|
Both
|
SNMP server shows as running when the SNMP agent is running.
|
Cisco System Manager
|
Both
|
System Manager runs on each server and facilitates communication with the CDSM.
|
Cisco Ingest Manager
|
Vault
|
Ingest Manager process is running if you are looking at a master Vault server and the optional Ingest Manager feature is part of your deployment.
|
Recommended Monitoring Schedule
This monitoring schedule is recommended to ensure that the CDS is functioning as expected and identify potential issues that may cause down time.
Caution Do not attempt to access the Linux command line unless you are familiar with the CDS, the Linux operating system, and the Linux command line.
Note Some error warnings in the logs are only informational and no action is necessary.
Daily Tasks
The following tasks should be performed daily:
•Choose Monitor > System Level > System Health and check the System Health Monitor page for red or yellow states on any of the servers. Click any red or yellow boxes to see detail information on disk, NIC, or services. See the "System Health" section for more information.
•Choose Monitor > System Level > Failed Ingests to check for any failed ingests. See the "Ingests" section for more information.
•Choose Monitor > System Level > Failed Streams to check for any failed streams. See the "Stream Failures" section for more information. It is also possible to run a report for the previous day. See the "Stream Failures" section on page 6-19 for more information.
Weekly Tasks
The following tasks should be performed weekly:
•Monitoring Tasks for Streamers and Vaults
•Monitoring Tasks for Vaults
•Monitoring Tasks for Streamers
Note All commands require that you log into each Linux operating system as root. Some tasks have a CDSM option.
Monitoring Tasks for Streamers and Vaults
To monitor the Streamer and Vaults weekly, do the following:
Step 1 Recover used disk space. Log in to each server using the root logon and run the following command:
Filesystem Size Used Avail Use% Mounted on
/dev/hda1 13G 5.2G 7.0G 43% /
/dev/hda6 20G 16G 4.3G 78% /arroyo/log
If the disk usage is greater than 75 percent, recover the disk space using the following methods:
a. Search and remove any core files.
find /home/isa -name core*
b. Copy any archived logs to an external device and delete them from the /arroyo/archive directory.
c. Check for the presence of old install or upgrade ISO files in the /root directory and delete them.
Step 2 Verify the services are running. Choose Monitor > Server Level > Services Monitor to check the services for each server, or log in to each server and run the following commands:
Step 3 Check the CServer interfaces to verify the status of the Ethernet adapters. Choose Monitor > Server Level NIC Monitor, or log in to each server and use the following commands
a. Use the grep -i Link command to verify that all adapters should have a status of "link up," except those adapters that are not being used.
grep -i Link /proc/net/PRO_LAN_Adapters/*.info
b. Use the grep -i Speed command to verify that each adapter that has a "link up" status should have a speed of 1000.
grep -i Speed /proc/net/PRO_LAN_Adapters/*.info
c. Use the grep -i State command to verify that all adapters should have an "up" state, except those adapters that are not being used.
grep -i State /proc/net/PRO_LAN_Adapters/*.info
Step 4 Check the CServer streaming and cache-fill interfaces using the following command:
Step 5 Check the database thread count using the following command:
Two connections for each Vault and Streamer should be listed with a status of "ESTABLISHED."
Step 6 Check the protocol timing logs for errors or problems. Also, look at the protocol timing logs for packet retransmissions.
tail -f /arroyo/log/protocoltiming.log.{date} | grep retransmissions
Step 7 Look for warning messages.
grep -i warning /arroyo/log/protocoltiming.log.<date> | more
Note The "WARNING" messages can sometimes be misleading; for example, "datawait" and "slow disk" messages occur normally and do not indicate a problem.
Step 8 The number of GOIDs for a particular content object must be the same on all servers (Vaults and Streamers) that are supposed to have the content. The number of Vaults that must have the same number of GOIDs for a particular content object is determined by the mirrored copy configuration (see the "Configuring the Servers" section on page 4-81). The number of GOIDs is also dependent on the trick speeds configured you configured (see the "Configuring Ingest Tuning" section on page 4-28). If the GOID is different between a Vault and a Streamer, session setup is not created properly because of an issue of "no content available." This is because there is no content on the Vault that matches the GOID of the Streamer has.
Monitoring Tasks for Vaults
In addition to the weekly monitoring tasks for both the Vaults and Streamers, the Vaults can also be monitored in the following ways:
1. Check the available space on the Vault hard drives. Choose Monitor > Server Level > Disk Monitor. The disk availability is shown as a percentage and as a number of gigabytes. Alternatively, view the protocol timing logs by running the following command:
tail -f /arroyo/log/protocoltiming.<date> | grep "Capacity Disk:"
The number returned indicates the percentage of the disk space available on this server. If the number is 5 or lower, then steps need to be taken to increase storage space by adding more Vaults, replacing drives with higher capacity drives, or removing unused content.
2. Check the /home/isa/ContentStore/server/ContentStore.log for ingest errors on each Vault. The master Vault has an additional log.
Monitoring Tasks for Streamers
In addition to the weekly monitoring tasks for both the Vaults and Streamers, the Streamers can also be monitored in the following ways:
1. In an ISA environment, look for any errors in the /Streaming/lscp_server/LSCPService.log on the primary Control server, and /Streaming/master/StreamService.log on the primary Setup server.
2. Look at the streaming log.
tail -f /arroyo/log/streamevent.log.<date>
Monthly Tasks
The monthly monitoring tasks consist of the following:
1. Choose Monitor > System Level > System Snapshot and check that the "Disk Available" amount meets the requirements for the expected movie storage in the next three to six months.
2. Run the reports for the last month that are suitable for your requirements and save them as comma-separated value (CSV) files.
3. Using the bandwidth and streaming reports, check that the CDS is not exceeding required usage per service area.
4. Run a quick security check.
a. Ensure that the CDSM changes can be attributed to individual users and not to a generic admin account.
b. Reset CDSM passwords if necessary.
c. Reset Linux passwords if necessary.
d. Check that access policies and firewalls are still enforced.
Other Tasks
If you have access to an anything on demand (XOD) application, do the following:
1. Check the inspect-live log for excessive errors.
2. Check the inspect-live log for excessive communication times with the BMS or CDS.
Feedback
