Table Of Contents
System Monitoring
System Level Monitoring
System Health
System Snapshot
Monitoring Content Objects
Ingests
List Content
Package Expirations
Publish Failures
Monitoring Stream Objects
Stream Monitor
Stream Failures
Array Level Monitoring
Array Snapshot
Barker Stream Monitor
Server Level Monitoring
Disk Monitor
S.M.A.R.T
NIC Monitor
Cache/Fill Bandwidth
Services Monitor
Recommended Monitoring Schedule
Daily Tasks
Weekly Tasks
Monitoring Tasks for Streamers and Vaults
Monitoring Tasks for Vaults
Monitoring Tasks for Streamers
Monthly Tasks
Other Tasks
System Monitoring
The CDSM provides tools that can be used for system monitoring and system diagnostics. The topics covered in this chapter include:
•
System Level Monitoring
•Monitoring Content Objects
•Monitoring Stream Objects
•Array Level Monitoring
•Server Level Monitoring
•Recommended Monitoring Schedule
Note If Virtual Video Infrastructure (VVI) with split-domain management is enabled, the CDSM pages associated with the Vaults and Caching Nodes display only on the VVI Manager (VVIM), and the CDSM pages associated with the Streamers display only on the Stream Manager. For more information, see the "Virtual Video Infrastructure" section on page D-5.
System Level Monitoring
The System Level Monitoring pages provide an overall view of the health and activity of the CDS. The System Level links are:
•System Health
•System Snapshot
To view the System Level Monitoring pages, click Monitor from any page in the CDSM, and then click System Health or System Snapshot, as appropriate.
System Health
The System Health page provides a top-level view of the overall health of each server in the CDS. Any time there is an alarmed event, an alarm is displayed in the CDSM banner. The Alarms menu is displayed when you roll your mouse over the alarm icon. See Figure 5-1. Clicking the alarmed event in the Alarm menu takes you to the CDSM page that has more information. For example, in Figure 5-1, clicking System health problems reported takes you to the System Health page.
Figure 5-1 CDSM Banner—System Health Alarm
To view the System Health page, choose Monitor > System Health. See Figure 5-2.
Figure 5-2 System Health Page
The colored boxes on the System Health Monitor page have the following meaning:
•Green—All components are operating.
•Yellow—Some components are not operational.
•Red—All components have failed.
You can view the details of a monitored area of a server by clicking the box in the appropriate column.
•When you click the Network check box you are taken to the NIC Monitor page. See the "NIC Monitor" section for more information.
•When you click the Disk check box you are taken to the Disk Monitor page. See the "Disk Monitor" section for more information.
•When you click the Services check box you are taken to the Services Monitor page. See the "Services Monitor" section for more information.
The time shown at the bottom of the left-panel menu is not the current time, but rather the CDSM time that is used for the health status and monitoring the system.
System Snapshot
The System Snapshot page provides an overview of the current activity on the CDS. A summary of the state of all streams, content ingests, and disk usage is displayed. See Figure 5-3.
Figure 5-3 System Snapshot Page
Table 5-1 describes the information displayed on the System Snapshot page.
Table 5-1 System Snapshot Page
Field
|
Description
|
Data Refresh Rate
|
How often the information is refreshed. The default is 10 seconds. The range is 10 to 300. All field values that are updated, based on the refresh rate, are initially shown in a green colored font.
|
Total Streams
|
Total number of stream objects the CDS is currently streaming.
|
HD Streams
|
Total number of high-definition stream objects the CDS is currently streaming.
|
SD Streams
|
Total number of standard-definition stream objects the CDS is currently streaming.
|
Active Stream Bandwidth
|
Total bandwidth, in megabits, used for active streams.
|
Active Fill Bandwidth
|
Total bandwidth, in megabits, used for caching content among Vaults and Streamers.
|
Total Content
|
Total number of content objects currently stored, ingested, provisioned for ingest, and failed ingest on the CDS.
|
Active Ingests
|
Total number of content objects currently being ingested on the CDS.
|
Completed Ingests
|
Total number of content objects currently stored on the CDS.
|
Prov. (push) Ingests
|
Total number of content objects that have been requested for ingestion, but have not yet begun active ingestion.
|
Failed Ingests
|
Total number of content objects that failed to complete the ingest process.
|
Temp Out of Service
|
Total number of content objects that are in a Temp Out of Service state. The backoffice may put a content object into this state for a certain amount of time.
|
Total Disk
|
Total disk space, in bytes, on the CDS.
|
Disk Used
|
Total used disk space, in bytes, on the CDS.
|
Disk Available
|
Total available disk space, in bytes, on the CDS.
|
Monitoring Content Objects
The content objects links on the Monitor System Level page provides information on the status of content ingests. Table 5-2 describes the different ingest states that are monitored.
Table 5-2 Monitored Ingest States
Completed Ingests
|
Lists content objects that have been fully ingested.
|
Active Ingests
|
Lists content objects that are in the process of being ingested.
|
Provisioned Ingests
|
Lists content objects that have been requested for ingestion, by way of the BMS administrator creating an entry, but have not yet begun active ingestion.
|
Unprovisioned Ingests
|
Lists content objects that were terminated by the BMS administrator or have been created but do not yet contain any information (in other words, they are "blank shells").
|
Failed Ingests
|
Lists content objects that failed to complete the ingest process. All failed ingests are reported back to the OpenStream system.
|
Package Expiration
|
Lists Package Expiration information, including expiration dates and all associated metadata. Allows for package expiration adjustments.
|
Publish Failures
|
Lists the packages that were not able to be published to the backoffice and provides a mechanism to republish the package.
|
In a VVI with split-domain management, the System Manager displays the following completed ingest details: Content Name, File Size, Rate, Create Time, and Last Modified. For the other completed ingests fields, see the same content asset on the VVIM.
Ingests
To view the details of content object ingests, do the following:
Step 1 Choose Monitor > Server Level from any page in the CDSM, and then click the link for the type of content object you want to view:
•Completed Ingests
•Active Ingests
•Provisioned Ingests
•Unprovisioned Ingests
•Failed Ingests
Step 2 Select a content object from the drop-down list and click Display. The details of the content object are displayed. Figure 5-4 shows an example of the ingest details.
By typing the first character of the content object name, you can jump to that section of the list.
In addition, you can perform a text string search by typing the text string you want to search for in the Search Ingests field and clicking Search. A list of content objects that contain the text string are listed. To see the content object details, click the content object name listed. To return to the previous page without selecting a content object, click Back.
Figure 5-4 Completed Ingests Page
Table 5-3 describes the content object details that are displayed for each type of ingest.
Table 5-3 Content Object Details
Field
|
Description
|
Content Name
|
The name of the content object.
|
Factory ID
|
The factory responsible for this content object.
|
Ingest IP
|
The IP address of the Vault's ingest interface used to download the content.
|
File Size
|
The file size, in bytes, of this content object.
|
Rate
|
The rate of ingest in bits per second (3750000 = 3.75 Mbps).
|
Create Time
|
The time and date this content object was created.
|
Last Modified
|
The time and date this content object was last modified.
|
Op State
|
The operational state of this content object. The possible operational states are:
•Created—Content is loading.
•In Service—Content is available for streaming.
•Out of Service—Content is not available for streaming.
|
Admin State
|
The administrative state of this content object. The possible administrative states are:
•Unprovisioned—Content is loading.
•In Service—Content is available for streaming.
•Out of Service—Content is not available for streaming.
|
Push Provision
|
The type of FTP provisioned. The provision types are:
•FTP pull
•FTP push
|
Content Copies
|
These fields display the following information about the copies of the content:
Speed/Direction—The trick-mode speed and direction (fast forward or rewind).
Server ID—The server ID where the copy is stored.
Status—The status of the stored content.
|
Trick Speeds [1-8]
|
Trick speeds 1 through 8 show the trick-mode speeds for this content object.
|
Full Content ID
|
The full identification of this content object.
|
Ior1
|
The Interoperable Object Reference (IOR) for this content object.
|
URL1
|
The Uniform Resource Locator (URL) address of the content has the following:
•Protocol used (for example, FTP)
•Username and password (for example, videolan:mpeg4ftp)
•IP address of the content provider's server (for example, 192.168.100.184)
•Directory where the content is stored on the provider's server (for example, videolan)
•Name of the file (for example, long_encore_3.75.mpg)
|
Asset Ior1
|
The Asset IOR associated with this content object.
|
List Content
The List Content option on the Completed Ingests page lists all content objects that meet the criteria (Only Damaged or List All) and that are associated with each content.
To list all associated content objects, do the following:
Step 1 Choose Monitor > System Level > Completed Ingests. The Completed Ingest page is displayed.
Step 2 In the List Content area, choose Only Damaged to list only the damaged content objects, or choose List All to list all content objects, and click List. The Content List is displayed (Figure 5-5).
Figure 5-5 Completed Ingests—Content List
Table 5-4 describes the information displayed in the Content List.
Table 5-4 Content List Fields
Field
|
Description
|
Content Name
|
The name of the content.
|
Duration
|
The duration of the content.
|
GOID
|
The Global Object ID for the content object associated with the content.
|
Version
|
The trick-play speed of the content object. The value, IGate, refers to an index file, which allows for the jumps between trick speeds, and so on. The value, redo, is an undo file. When the content is ingested, if there are any changes made during the ingest, the changes are recorded in the redo file. If the content is sent by using FTP Out, the changes are undone and the original file is sent.
|
Server ID
|
The server ID of the Vault that is storing the content object.
|
Status
|
The status of the storing process of the content object, either complete or partial.
|
Package Expirations
Note Package Expirations are part of the optional Ingest Manager feature. This option is only listed on the Monitoring System Level left-panel menu if the Ingest Manager is included in your deployment.
To view the details or adjust the license expiration of a package expiration, do the following:
Step 1 Choose Monitor > Package Expiration. The Package Expiration page is displayed.
Step 2 From the Available Packages drop-down list, choose a package and click Display. The Package Expiration details are displayed.
By typing the first character of the package name, you can jump to that section of the list.
Figure 5-6 shows an example of the Package Expiration details.
Figure 5-6 Package Expiration Page
Table 5-5 describes the package expiration details that are displayed.
Table 5-5 Package Expiration Details
Field
|
Description
|
Package Name
|
The name of the package.
|
License Expiration
|
The date the package will expire.
|
Additional Package Window
|
The additional time added to the package.
|
Actual Package Expiration
|
The Actual Package Expiration is the License Expiration plus the Additional Package Window.
|
Step 3 To adjust the license expiration, enter the number of days (positive or negative) in the Adjust License Expiration field and click Update.
The license expiration is adjusted by the number of days you entered. The Additional Package Window is not affected and is still applied to create the Actual Package Expiration.
Step 4 In the Asset Details section, to view the metadata associated with the package expiration, click the plus sign (+) next to the metadata you want to view.
To delete a package, choose the package from the Available Packages drop-down list, click Display, and then click Delete in the Package Expiration Details section.
Publish Failures
Note Publish Failures are part of the optional Ingest Manager feature. This option is only listed on the Monitoring System Level left-panel menu if the Ingest Manager is included in your deployment.
The Publish Failures page lists the packages that were not able to be published to the backoffice and provides a mechanism to republish the package.
To publish an unpublished package, or delete an unpublished package, do the following:
Step 1 Choose Monitor > Publish Failures. The Publish Failures page is displayed.
Step 2 From the Unpublished Packages drop-down list, choose a package and click Display. The Unpublished Package details are displayed.
By typing the first character of the package name, you can jump to that section of the list.
Figure 5-6 shows an example of the Publish Failures details.
Figure 5-7 Publish Failures Page
Table 5-6 describes the publish failures details that are displayed.
Table 5-6 Publish Failures Details
Field
|
Description
|
Package Name
|
The name of the package.
|
Source URL
|
The location of the original package information.
|
Target URL
|
The location where to place the package information.
|
Module Type
|
An internal identifier for what failed. In the example in Figure 5-7, the publishing of the package failed.
|
Creation Date
|
The date the database record was created for this failure.
|
Step 3 In the Assets section, to view the metadata associated with the unpublished package, click the plus sign (+) next to the metadata you want to view.
Step 4 To restart the publishing process and set the creation date to today, click Publish.
To delete an unpublished package, click Delete.
Monitoring Stream Objects
The monitored stream objects consist of:
•Stream Monitor
•Stream Failures
Stream Monitor
To view the details of stream objects, do the following:
Step 1 Choose Monitor > Stream Monitor. The Stream Monitor page is displayed.
Step 2 From the Browse By Session ID drop-down list, choose the Session ID of the stream object and click Display. The stream object details are displayed.
If you do not know the session ID, you can perform a text string search by selecting the field you want to search on, entering the text string in the Search String field, and clicking Search. A list of stream objects that contain the text string in the field you selected are listed. Click the linked field (Session Handle in Figure 5-8) to see the stream monitor details for the selected object, or click Back to return to the previous page.
Figure 5-8 Stream Search Results
Figure 5-9 shows an example of the stream object details.
Figure 5-9 Stream Monitor Page
Table 5-7 describes the stream details.
Table 5-7 Stream Details
Field
|
Description
|
Stream ID
|
The internal unique identifier assigned to the stream session by the Streamer.
|
Content ID
|
The content identifier of the stream object.
|
LSCP IP
|
The source IP address of the LSCP transaction (set-top box).
|
Session ID
|
The session ID of the stream object.
|
Create Time
|
The date and time the stream object was created.
|
Last Modified
|
The date and time the stream object was last modified.
|
TSID out
|
This field is only applicable when Streaming Mode is set to ASI (0).
The output transport stream identification on the associated MQAM device.
|
Service Group
|
The service group that the stream object is transmitting on.
|
QAM IP
|
The IP address of the QAM device delivering this stream object.
|
QAM Port
|
The port the QAM device is using to receive the stream object.
|
Program Number
|
This field is only applicable when Streaming Mode is set to ASI.
The numerical MPEG program number for this stream object.
|
Bandwidth Used
|
The transport stream bandwidth, in bytes, required for this stream object.
|
Op State
|
Operational State indicates the state of the object. The possible states are:
•InService—Stream object is functioning.
•OutOfService—Occurs when the entity using the object wants to temporarily stop the object.
•Created —Stream object not yet provisioned.
•Destroyed—Stream object is destroyed.
|
Admin State
|
The administrative state of this stream object. The possible administrative states are:
•Unprovisioned—Stream is loading.
•InService—Stream is available for streaming.
•OutOfService—Stream is not available for streaming.
|
Stream State
|
Stream state originates from the LSCP server and has the following modes:
•Open—The server is not transporting a media stream.
•Pause—The server is not transporting a media stream.
•Search Transport—The server is searching for start normal play time (NPT). When at start NPT, it enters Transport mode.
•Transport—The server is transporting the media stream and will pause at the end of the stream. If scale is positive, indicating a forward direction, end of stream is the end of media. If scale is negative, indicating a reverse direction, end of stream is the beginning of media.
•Transport Pause—The server is transporting the media stream and will pause at stop NPT.
•Search Transport Pause—The server is searching for start NPT. When at start NPT, it enters transport pause mode.
•Pause Search Transport—The server is transporting the media stream. It does so until stop NPT, and then transitions to search transport mode.
•End of Stream—The server is not transporting a media stream.
|
Speed Direction
|
Speed direction is as follows:
•Play
•Not playing (Pause)
•n fast-forward, where n means n times fast-forward
•-n means, where n means n times rewind
|
Provision Multiple
|
Provision Multiple is enabled if this field is" yes" and disabled if this field is "no."
|
The Graph This Stream button displays the trick-mode activity of the stream (Figure 5-10).
Figure 5-10 Stream Activity Report
To delete a stream object, display the object and click Delete.
Stream Failures
To view the details of Failed Streams, do the following:
Step 1 Choose Monitor > Failed Streams. The Stream Failures page is displayed.
Each failed stream is listed by the date and time the stream failed, followed by the session ID of the failed stream.
Note Stream Failure monitoring displays only the failed streams for the current day. To view past stream failures, see the "Stream Failures" section on page 6-18.
Step 2 From the Stream Failures drop-down list, choose the timestamp and session ID of the stream object and click Display. The stream failure details are displayed. See Figure 5-11.
To delete a failed stream, display the object and click Delete.
Figure 5-11 Stream Failures Page
Table 5-8 describes the stream failure details.
Table 5-8 Stream Failure Details
Field
|
Description
|
Session ID
|
The session ID of the failed stream.
|
Failure Date
|
The date and time the failure occurred.
|
QAM IP
|
The IP address of the QAM device associated with the failure.
|
Service Group
|
The service group associated with the failure.
|
Server ID
|
The server responsible for streaming this stream object. To view the IP address associated with the Server ID, see the "Configuring the Servers" section on page 4-78.
|
Group ID
|
All servers that are part of the same CDS system (managed by one CDSM) have the same Group ID. This Group ID corresponds to the CDSM GUI array ID and should be unique across an enterprise. Table 5-9 describes the ID mapping between the CDSM GUI and the CServer.
|
Failed Operation
|
The operation that was taking place when the stream failed, for example, createStream, LSCP Command(), or createServant, destroy. These are the measurement points or transactional states of the system at the time of the failure.
|
Failed Task
|
A failed task is the event category that provides the type of execution sequence that the call stack was currently in at the time of the failure. The list of the high-level categories are:
|
| |
•Tune In
•Load Application
•Load Catalog
•Eligibility Check
•Select Subscription
•Purchase Subscription
•Select Movie
•Purchase Check
|
•Play Movie
•Movie Setup
•Movie Control
•Movie Confirm
•Purchase Confirm
•Purchase Log
•Stop Movie
•Movie Release
|
Error Code
|
The error code provides a description of the event that caused an error. See Table 5-10, Table 5-11, and Table 5-12 for descriptions of the error codes.
|
Table 5-9 lists the CDSM GUI ID names and maps them to the CServer names in the setupfile and .arroyorc files.
Table 5-9 ID Names in the CDSM GUI and CServer Files
CDSM GUI ID Name
|
CServer Files ID Name
|
Array ID on the Array Name page
|
groupid
|
Group ID on the Server-Level pages
|
groupid
|
Stream Group ID on the Server Setup page
|
arrayid
|
Cache Group ID on the Server Setup page
|
arrayid
|
Vault Group ID on the Server Setup page
|
arrayid
|
Stream Group ID on the Configuration Generator page
|
arrayid
|
Table 5-10 lists the Managed Services Architecture (MSA) error codes. Some MSA monitored events are monitored for the CDS as well, and are prefaced by "AVS_" instead of "MSA_." They are denoted with a footnote in the table. Some MSA monitored events are not errors, but rather information about an event.
Table 5-10 MSA Error Codes
Error Code
|
Description
|
MSA_INT_ERR1
|
There is an internal error. INT_ERR has a subset of error codes that specifically describe where the error occurred. See Table 5-11.
|
MSA_FLOW1
|
An entry or exit of a measured or tracked flow, or some other important check point, and is recorded as non-realtime.
|
MSA_CMPT_NOT_EXIST1
|
A component does not exist.
|
MSA_REQ_TIMEOUT1
|
A client timed out waiting for a response to a request.
|
MSA_CMPT_OUT_OF_SVC1
|
A component is unavailable.
|
MSA_REQ_NOT_IMPL1
|
The requested item is not implemented.
|
MSA_RES_INVALID1
|
A resource is invalid.
|
MSA_RES_DUP1
|
A duplicate resource is being added to the session.
|
MSA_CMPT_DUP
|
It was determined that a component that was being added has the same name as a previously created component.
|
MSA_REQ_CREATE
|
An attempt to create a request failed.
|
MSA_UNKNOWN
|
An unclassified or undetermined error occurred.
|
MSA_REAP
|
A component is destroyed outside of the normal expected flows.
|
MSA_VS_LSC_TIMEOUT
|
The video server timed out waiting for the client to issue a resume or play command after the initial creation of the stream.
|
MSA_SVC_GROUP_MISSING
|
A request contains a missing service group.
|
MSA_RES_NO_CAPACITY
|
A resource is currently out of capacity and cannot satisfy the request.
|
MSA_RES_NO_BANDWIDTH
|
A resource does not have the bandwidth to deliver the stream.
|
MSA_REQ_FAIL
|
A request failed.
|
MSA_RES_UNAVAIL
|
A response is not available.
|
MSA_FLOW_RT
|
An entry or exit of a measured or tracked flow, and is recorded in real-time.
|
MSA_LSC_SERVER_FAILURE
|
LSC response; server failed.
|
MSA_LSC_NO_MEMORY
|
LSC response; dynamic memory allocation failure.
|
MSA_LSC_IMPL_LIMIT
|
LSC response; implementation limit exceeded.
|
MSA_LSC_NO_RESOURCES
|
LSC response; no resources.
|
MSA_LSC_SERVER_ERROR
|
LSC response; server error.
|
MSA_LSC_MPEG_DELIVERY
|
LSC response; unable to deliver MPEG stream.
|
MSA_LSC_ERR
|
A generic DSM-CC error event.
|
MSA_LSC_BAD_REQUEST
|
LSC response; invalid request.
|
MSA_LSC_BAD_STREAM
|
LSC response; invalid stream handle.
|
MSA_LSC_WRONG_STATE
|
LSC response; wrong state.
|
MSA_LSC_UNKNOWN
|
LSC response; unknown error.
|
MSA_LSC_NO_PERMISSION
|
LSC response; client does not have permission for the request.
|
MSA_LSC_BAD_PARAM
|
LSC response; invalid parameter.
|
MSA_LSC_NO_IMPL
|
LSC response; not implemented.
|
MSA_LSC_TRANSIENT
|
LSC response; transient error.
|
MSA_LSC_BAD_SCALE
|
LSC response; incorrect scale value.
|
MSA_LSC_BAD_START
|
LSC response; stream start time does not exist.
|
MSA_LSC_BAD_STOP
|
LSC response; stream stop time does not exist.
|
Table 5-11 lists the error codes for internal errors and external errors. Internal errors are errors that occurred in the CDS and specifically describe where the error occurred. External errors are errors that occurred in the network or network components, which includes the ContentStore, StreamService, and so on. The error codes listed in Table 5-11 provide more detail to the MSA_INT_ERR or AVS_INT_ERR error code.
Table 5-11 INT_ERR Error Codes
Error Code
|
Description
|
INGEST_THREADS_NOT_RUNNING
|
Cache server threads are not running.
|
INGEST_NIC_DOWN
|
Ingest interface is disabled.
|
INGEST_DATA_BLOCKAGE
|
Ingest data read is backlogged, causing data socket blockage.
|
BAD_CONTENT
|
Content data is not recoverable.
|
NOT_ENOUGH_NIC_BANDWIDTH
|
Not enough bandwidth left over on NICs to perform the operation.
|
NOT_ENOUGH_SYSTEM_RESOURCES
|
Not enough system resources left to perform the operation.
|
NOT_ENOUGH_DISK_SPACE_AVAILABLE
|
Not enough disk space available.
|
STREAMER_MAX_SLOTS_LIMIT_EXCEEDED
|
No stream slot available to allocate the stream.
|
REMOTE_VAULT_DOWN
|
The remote Vault is not responding.
|
REMOTE_STREAMER_DOWN
|
The remote Streamer is down.
|
VAULT_DISK_BAD
|
The disk is bad on a Vault.
|
STREAMER_DISK_BAD
|
The disk is bad on a Streamer.
|
CONTENT_LOCATE_FAILED
|
Cannot locate the content on any Vault.
|
CONTENT_FILL_FAILED
|
Cannot push content to the Streamer from Vault.
|
NOT_ENOUGH_FILL_BANDWIDTH
|
Not enough fill bandwidth available.
|
FILL_LINKS_DOWN
|
The fill links are down.
|
STREAMING_LINKS_DOWN
|
The stream links are down.
|
VAULT_MIRRORING_SITE_DOWN
|
The mirroring site of the Vault array is down.
|
SET_CONTENT_BUNDLE_FAILED
|
Set content bundle descriptor array failed.
|
SET_DESTINATION_FAILED
|
Set destination of stream failed.
|
DESTROY_STREAM_FAILED
|
Destroy stream failed.
|
PLAY_STREAM_FAILED
|
Play stream failed.
|
FILLCB_FAILED
|
Fill CB failed.
|
WAIT_FOR_FTP_DATA_DONE_FAILED
|
Wait for FTP data done failed.
|
GET_CURRENT_NPT_FAILED
|
Get current NPT for LSCP status failed.
|
CAN_NOT_CONNECT_TO_NAME_SERVICE
|
Cisco ISA cannot connect to the BMS Naming Server.
|
CAN_NOT_CONNECT_TO_NOTIFY_SERVICE
|
Cisco ISA cannot connect to the Notify Server.
|
CAN_NOT_CREATE_EVENT_CHANNELS
|
Cisco ISA cannot create event channels.
|
NO_CONTENT_EVENT_CHANNEL_FOUND
|
Naming server does not have content event channel.
|
NO_STREAM_EVENT_CHANNEL_FOUND
|
Naming server does not have stream event channel.
|
EVENT_CHANNEL_OBJECT_NOT_EXISTS
|
Event channel object does not exist in Notify Server.
|
CORBA_CONNECTION_FAILED
|
CORBA System exception while connecting to other entity.
|
CORBA_BROKEN_PIPE
|
CORBA system exception with broken pipe with other entity.
|
CORBA_CONTENT_STORE_BIND_FAILED
|
CORBA bind exception while starting Cisco ContentStoreFactory.
|
CORBA_STREAM_SERVICE_BIND_FAILED
|
CORBA bind exception while starting Cisco StreamService.
|
CORBA_SYSTEM_ERROR
|
CORBA system exception while connecting to the servant.
|
CORBA_TRANSIENT_ERROR
|
CORBA system exception with object being transient.
|
CORBA_TIMEOUT_ERROR
|
CORBA timeout exception.
|
CORBA_IOR_NIL
|
The orb object is nil.
|
CORBA_IOR_NIL_AFTER_NARROW
|
The orb object is nil after narrow.
|
SERVICE_GROUP_NOT_SUPPORTED
|
Service group is not supported.
|
REMOTE_CONTENT_STORE_FACTORY_DOWN
|
Remote Cisco ContentStoreFactory is down.
|
VAULT_HAS_FULL_LOAD
|
Vault is running with full load.
|
FTP_CONNECTION_FAILED
|
Connection to FTP server failed.
|
FTP_SERVER_BIND_FAILED
|
FTP server can not bind to the port.
|
FTP_PUSH_TIMEOUT
|
FTP push timeout (PASV is not served fast enough).
|
FTP_QUIT_RECEIVED_DURING_INGEST
|
FTP server received QUIT request.
|
NO_LSCP_SET_TOP_CONNECTION
|
LSCP server to set-top box connection is down.
|
LSCP_SERVER_BIND_FAILD
|
LSCP server cannot bind to the running port.
|
LSCP_PROXY_BIND_FAILED
|
LSCP proxy cannot bind to the running port.
|
STREAMER_GROUP_MAX_LIMIT_EXCEEDED
|
The stream count is exceeding the limit for the Streamer group.
|
STREAMER_MAX_LIMIT_EXCEEDED
|
The stream count is exceeding the limit for a Streamer.
|
REMOTE_STREAMER_NOT_RESPONDING
|
The remote Streamer is down.
|
NOT_ENOUGH_MQAM_BANDWIDTH
|
Not enough MQAM bandwidth.
|
NO_QAM_FOR_SERVER_ID
|
Server is not connected to any QAM.
|
NOT_ENOUGH_QAM_BANDWIDTH
|
Not enough QAM bandwidth.
|
STREAMER_IS_NOT_IN_THE_SERVICE_GROUP
|
Streamer is not in the service group.
|
STREAMER_HAS_FULL_LOAD
|
Streamer is running with full load.
|
STREAMER_IS_NOT_CONNECTED_THAT_QAM
|
Server is not connected to the QAM.
|
INVALID_SERVICE_GROUP
|
The service group is not returned by session gateway.
|
CONTENT_CAN_NOT_BE_LOCATED
|
Content is not found in the related content stores.
|
CONTENT_OBJECT_NOT_YET_PROVISIONED
|
The content object is not yet provisioned.
|
STREAM_OBJECT_NOT_YET_PROVISIONED
|
The stream object is not yet provisioned.
|
STREAM_OBJECT_IS_OUT_OF_SERVICE
|
The stream object is out of service.
|
STREAM_OBJECT_IS_ALREADY_PROVISIONED
|
The stream object is already in service.
|
CONTENT_OBJECT_IS_ALREADY_PROVISIONED
|
The content object is already in service.
|
STREAM_SERVANT_OBJECT_NOT_EXIST
|
The remote streamer does not have a servant for stream object.
|
NO_DESTINATION_QAM_IP_FOUND
|
No QAM IP Address is received for the stream destination.
|
NO_DESTINATION_QAM_PORT_FOUND
|
No QAM port is received for the stream destination.
|
FAILED_TO_SET_STREAM_DESTINATION
|
Some error occurred while setting the stream destination.
|
UNABLE_TO_ACCEPT_CONNECTION
|
Cannot accept more TCP connections.
|
UNABLE_TO_REGISTER_EVENT_HANDLER
|
Cannot register event handle to serve the TCP connection.
|
CAN_NOT_LOCATE_QAM_IP_FOR_TSID_IN
|
Cannot locate the QAM IP addresses associated with TSID IN.
|
NS_LOG_MONITOR_ERROR
|
The ns_log file is not updating.
|
AVS_ISA_GENERIC_ERROR
|
Some unknown error occurred during execution of the operation.
|
CACHE2APP_INITIALIZE_ERROR
|
Failed to initialize Cache2App library.
|
FILLCB_FAILED
|
FillCB failed during content ingest.
|
DESTROYCB_FAILED
|
Failed while destroying the content bundle descriptor.
|
WAIT_FOR_FTP_DATA_DONE_FAILED
|
API wait for FTP data done has returned an error.
|
SET_CONTENT_BUNDLE_DESCRIPTOR_FAILED
|
Failed while setting content bundle descriptor for a stream.
|
SET_DESTINATION_FAILED
|
Failed while setting destination of the stream.
|
SET_ENCRYPTION_KEY_FAILED
|
Failed while setting ECM keys for the stream.
|
CREATE_STREAM_FAILED
|
AVS cache server cannot allocate the stream handle.
|
DESTROY_STREAM_FAILED
|
AVS cache server cannot tear down the stream.
|
DESTROY_REMOTE_STREAM_FAILED
|
AVS cache server cannot tear down stream on remote server.
|
PLAY_STREAM_FAILED
|
AVS cache server cannot play the stream.
|
DATABASE_DOWN
|
Database is down.
|
DATABASE_SYNCHRONIZING_REPLICATION_Q
|
Database is synchronizing with replication queue.
|
DATA_IS_NOT_IN_SYNC
|
Database is not in sync with master.
|
DATABASE_RETURNED_ERROR
|
The database has returned an error; may be there is no record found.
|
DATABASE_RECORD_NOT_FOUND
|
The record is not found in the database.
|
DATABASE_CAN_NOT_INSERT_RECORD
|
The record cannot be inserted into the database.
|
DATABASE_CAN_NOT_DELETE_RECORD
|
The record cannot be deleted from the database.
|
DATABASE_CAN_NOT_UPDATE_RECORD
|
The record cannot be updated.
|
DATABASE_QUERY_SEND_ERROR
|
Failed to make a query to the database.
|
Table 5-12 lists the Managed Services Architecture (MSA) error codes for the optional Ingest Manager feature.
Table 5-12 MSA Error Codes for the Optional Ingest Manager Feature
Error Code
|
Description
|
MSA_BAD_XML
|
There was an XML parsing error. Check the ADI XML for errors.
|
MSA_BAD_REQUEST
|
The request for content was bad. Check the target backoffice URL.
|
MSA_UNKNOWN_HOST
|
The host is unknown. Check the target backoffice URL.
|
MSA_CONNECTION_DROP
|
The connection was dropped. Check the URL. The Ingest Manager possibly misformatted the ADI XML.
|
MSA_BACKOFFICE_TIMEOUT
|
The backoffice did not respond within the allowed time interval.
|
MSA_UNKNOWN
|
An unknown error occurred. Check the /home/isa/bss/log/aim.log.
|
MSA_FAILED_POST
|
The Ingest Manager failed to post the ADI to the backoffice.
|
MSA_PKG_EXPIRED
|
The package has expired and the retry record will be removed.
|
Array Level Monitoring
The Array Level Monitoring pages provide an overall view of the health and activity of a specified array, as well as provide monitoring and deletion of barker streams. The Array Level links are:
•Array Snapshot
•Barker Stream Monitor
Array Snapshot
The Array Snapshot page provides an overview of the current activity for the specified array of servers on the CDS. A summary of the state of all streams, content ingests, and disk usage is displayed.
The fields displayed on the Array Snapshot page are the same fields that are displayed on the System Snapshot page, with the active stream bandwidth and active fill bandwidth shown for each Stream Group. For descriptions of the fields, see Table 5-1.
Barker Stream Monitor
Note The Barker Stream feature is optional and is not listed on the Array Level left-panel menu if it is not included in your deployment. The Barker Stream feature is also not available if the Stream Destination is set to IPTV. For more information, see the "Stream Destination" section on page D-4.
The Barker Stream Monitor page lists the barker streams currently configured. Figure 5-12 shows an example of barker streams for ASI streaming.
To delete a barker stream configuration, click the Delete button next to the associated barker stream.
Figure 5-12 Barker Stream Monitor Page
Server Level Monitoring
The Server Level Monitoring pages provide detail information on the health and activity of a Vault or Streamer server.
To view the Server Level Monitoring pages, do the following:
Step 1 Choose Monitor > Server Level, and then click one of the following as applicable:
•Disk Monitor
•NIC Monitor
•Cache/Fill Bandwidth
•Services Monitor
Step 2 Select a server's IP address from the drop-down list and click Display.
Disk Monitor
The Disk Monitor page provides real-time information on the status of a disk.
To view the current status of a disk, select a server's IP address from the drop-down list on the Disk page, click Display, and roll your mouse over one of the disks displayed in the graphic. If the server is a Lindenhurst CDE, click one of the disks displayed in the graphic. Figure 5-13 shows an example of a Streamer server.
Figure 5-13 Disk Monitor—Streamer Page
For Vault servers, the Disk Availability line graph shows the percentage of disk space available. The gigabytes displayed for "Total Space," "Available Space," and "% Used" are the sum of all the disks installed on the Vault server.
The Linux File System Stats table shows the combined total storage space for the partions of the disk drives, the combined available storage space for the partions of the disk drives, and the percentage of used storage for each combined partition. In Figure 5-13, the hda2 partition has an alarm indicator because the usage has exceeded the user-defined threshold of 40 percent. For information on setting thresholds, see the "Setting System Thresholds" section on page 7-10.
Table 5-13 describes the information displayed when a disk is selected.
Table 5-13 Disk Status Fields
Field
|
Description
|
Current Temp
|
The current temperature of the hard disk.
|
Smart Status
|
The Self-Monitoring, Analysis and Reporting Technology (S.M.A.R.T.) status of a disk as determined by the manufacturer in accordance with the relevant ATA/SCSI standards. S.M.A.R.T. is logic embedded in the firmware that determines when a disk is going bad.
|
Number of Reads
|
Number of bytes read from the disk since it was powered on.
|
Number of Writes
|
Number of bytes written to the disk since it was powered on.
|
S.M.A.R.T
The CDS incorporates S.M.A.R.T. to monitor the reliability of a hard drive, predict drive failures, and to carry out different types of drive self-tests. S.M.A.R.T is firmware, native to most disk drives, that monitors disk attributes over time, making it possible to perform predictive failure analysis. Advanced warning of predictive failures allows the operator to perform preventative maintenance.
To view the current read/write activity that has occurred in the last five seconds on the selected disk, click Graph Disks. Figure 5-14 shows an example of the Disk Activity graph.
Figure 5-14 Disk Monitor—Vault Disk Activity Graph
The Disk Activity graph displays an average calculation of the amount of data read (in megabytes per second) and data writes over a five-second period.
NIC Monitor
The NIC Monitor page displays the status of each network interface card (NIC) on the server.
To view the current status of a NIC on a server, select a server's IP address from the drop-down list on the NIC Monitor page, click Display, and click one of the NIC ports displayed in the graphic. Figure 5-15 shows an example of the eth4 interface on a Vault server.
Figure 5-15 NIC Monitor—Vault Page
Table 5-14 describes the information displayed for a NIC port.
Table 5-14 NIC Port Status Fields
Field
|
Description
|
Port Speed
|
The speed of the interface in megabits per second (Mbps).
|
Admin State
|
The administrative state of the port interface. The administrative state is determined at the time the server is booted. The possible administrative states are up or down.
|
Op State
|
The operational state of the port interface. The operational state is either up or down. If the port is not connected to the network or is malfunctioning, the operational state displayed is down.
|
Media Type
|
The physical conduit of the interface. The physical type is either copper or fiber optic.
|
Poll Interval
|
The number of seconds between each disk polling.
|
Transmit
|
The total number of bytes transmitted since this port has been operational and configured as administratively up.
|
Received
|
The total number of bytes received since this port has been operational and configured as administratively up.
|
To view the average transmit and receive activity that has occurred in the last two seconds for each port on this server, click Graph Ports. Figure 5-16 shows an example of the Port Activity graph.
Figure 5-16 NIC Monitor—Vault Port Activity Graph
Cache/Fill Bandwidth
The Cache/Fill Bandwidth page displays details on the content caching activity on a Streamer.
To view the caching activity on a server, select a server's IP address from the drop-down list on the Cache/Fill Bandwidth page and click Display. Figure 5-17 shows an example.
Figure 5-17 Cache/Fill Bandwidth Page
Table 5-15 describes the services listed in the Cache/Fill Bandwidth page.
Table 5-15 Cache/Fill Bandwidth Fields
Service
|
Description
|
Data Refresh Rate
|
How often the information is refreshed. The default is 10 seconds. The range is 10 to 300. All field values that are updated, based on the refresh rate, are initially shown in a green-colored font.
|
Active Stream Count
|
The number of active streams on this Streamer.
|
Active Stream Bandwidth
|
The bandwidth (in Mbps) used for streaming on this Streamer.
|
Unique Stream Count
|
The number of unique streams on this Streamer.
|
Unique Stream Bandwidth
|
The bandwidth (in Mbps) used for serving unique streams on this Streamer.
|
Fill Receive Stream Count
|
The number of streams on this Streamer that are retrieving content from the Vault in order to fulfill requests for content.
|
Actual Fill Stream Bandwidth
|
The bandwidth (in Mbps) used on this Streamer for retrieving content from the Vault.
|
Disk Read Stream Count
|
The number of streams on this Streamer sending content that was retrieved from the Streamer's disks.
|
Disk Read Bandwidth
|
The bandwidth (in Mbps) used on this Streamer for retrieving locally stored content (content on the Streamer's disks).
|
Services Monitor
The Services Monitor page displays whether specific processes are running on a server.
To view the current status of the services running on a server, select a server's IP address from the drop-down list on the Services Monitor page and click Display. Figure 5-17 shows an example of a Streamer server.
Figure 5-18 Services Monitor Page—Streamer
Table 5-16 describes the services listed on the Services Monitor page.
Table 5-16 CDS Services
Service
|
Server
|
Description
|
Cisco Cache Server
|
Both
|
The Cache server runs on all servers. The Cache server is responsible for the core functions of the CDS.
|
Cisco Content Store Master
|
Vault
|
The Content Store Master process will be running if you are looking at a master Vault server. The Content Store Master serves as the master Vault process for accepting inbound OpenStream connections.
|
Cisco Content Store Slave
|
Vault
|
The Content Store Slave process will be running if you are looking at either a master or a slave Vault server. The Content Store Slave handles requests proxied by the Content Store Master.
|
Cisco Stream Service Master
|
Streamer
|
The Streamer Service Master process will be running if you are looking at a master Streamer server. The Stream Service Master serves as the master Streamer process for accepting inbound OpenStream connections.
|
Cisco Primary Setup Service
|
Streamer
|
The Primary Setup Service will be running if you are looking at the Streamer server designated as the primary Setup server. The Setup server handles setting up stream sessions.
|
Cisco Stream Control Service
|
Streamer
|
The Stream Control Service accepts set-top box play stream commands, for example, LSCP.
|
Cisco Resource Manager
|
Streamer
|
The Resource Manager runs on a Streamer server. The Resource Manager handles orphaned streams.
|
Cisco DB Server
|
Both
|
The DB (database) server runs on all servers and is responsible for keeping track of all data objects in the CDS.
|
DB Synchronization Status
|
Both
|
Displays the status of the database synchronization among all servers. The states are "OK" and "not OK."
|
Cisco SNMP Server
|
Both
|
The SNMP server shows as running when the SNMP agent is running.
|
Cisco System Manager
|
Both
|
The System Manager runs on each server and facilitates communication with the CDSM.
|
Cisco Ingest Manager
|
Vault
|
The Ingest Manager process will be running if you are looking at a master Vault server and the optional Ingest Manager feature is part of your deployment.
|
Recommended Monitoring Schedule
This monitoring schedule is recommended to ensure that the CDS is functioning as expected and identify potential issues that may cause down time.
Caution Do not attempt to access the Linux command line unless you are familiar with the CDS, the Linux operating system, and the Linux command line.
Note Some error warnings in the logs are only informational and no action is necessary.
Daily Tasks
The following tasks should be performed daily:
•Choose Monitor > System Level > System Health and check the System Health Monitor page for red or yellow states on any of the servers. Click any red or yellow boxes to see detail information on disk, NIC, or services. See the "System Health" section for more information.
•Choose Monitor > System Level > Failed Ingests to check for any failed ingests. See the "Ingests" section for more information.
•Choose Monitor > System Level > Failed Streams to check for any failed streams. See the "Stream Failures" section for more information. It is also possible to run a report for the previous day. See the "Stream Failures" section on page 6-18 for more information.
Weekly Tasks
The following tasks should be performed weekly:
•Monitoring Tasks for Streamers and Vaults
•Monitoring Tasks for Vaults
•Monitoring Tasks for Streamers
Note All commands require that you log into each server's Linux operating system as root. Some tasks have a CDSM option.
Monitoring Tasks for Streamers and Vaults
To monitor the Streamer and Vaults weekly, do the following:
Step 1 Recover used disk space. Log in to each server using the root logon and run the following command:
Filesystem Size Used Avail Use% Mounted on
/dev/hda1 13G 5.2G 7.0G 43% /
/dev/hda6 20G 16G 4.3G 78% /arroyo/log
If the disk usage is greater than 75 percent, recover the disk space using the following methods:
a. Search and remove any core files.
find /home/isa -name core*
b. Copy any archived logs to an external device and delete them from the /arroyo/archive directory.
c. Check for the presence of old install or upgrade ISO files in the /root directory and delete them.
Step 2 Verify the services are running. Choose Monitor > Server Level > Services Monitor to check the services for each server, or log in to each server and run the following commands:
Step 3 Check the CServer interfaces to verify the status of the Ethernet adapters. Choose Monitor > Server Level NIC Monitor, or log in to each server and use the following commands
a. Use the grep -i Link command to verify that all adapters should have a status of "link up," except those adapters that are not being used.
grep -i Link /proc/net/PRO_LAN_Adapters/*.info
b. Use the grep -i Speed command to verify that each adapter that has a "link up" status should have a speed of 1000.
grep -i Speed /proc/net/PRO_LAN_Adapters/*.info
c. Use the grep -i State command to verify that all adapters should have an "up" state, except those adapters that are not being used.
grep -i State /proc/net/PRO_LAN_Adapters/*.info
Step 4 Check the CServer streaming and cache-fill interfaces using the following command:
Step 5 Check the database thread count using the following command:
Two connections for each Vault and Streamer should be listed with a status of "ESTABLISHED."
Step 6 Check the protocol timing logs for errors or problems.
a. Look at the protocol timing logs for packet retransmissions.
tail -f /arroyo/log/protocoltiming.log.{date} | grep retransmissions
Step 7 Look for warning messages.
grep -i warning /arroyo/log/protocoltiming.log.<date> | more
Note The "WARNING" messages can sometimes be misleading; for example, "datawait" and "slow disk" messages occur normally and do not indicate an problem.
Step 8 The number of GOIDs for a particular content object must be the same on all servers (Vaults and Streamers) that are supposed to have the content. The number of Vaults that must have the same number of GOIDs for a particular content object is determined by the mirrored copy configuration (see the "Configuring the Servers" section on page 4-78). The number of GOIDs is also dependent on the trick speeds configured you configured (see the "Configuring Ingest Tuning" section on page 4-27). If the GOID is different between a Vault and a Streamer, session setup is not created properly because of an issue of "no content available." This is because there is no content on the Vault that matches the GOID of the Streamer has.
Monitoring Tasks for Vaults
1. Check the available space on the Vault hard drives. Choose Monitor > Server Level > Disk Monitor. The disk availability is shown as a percentage and as a number of gigabytes. Alternatively, view the protocol timing logs by running the following command:
tail -f /arroyo/log/protocoltiming.<date> | grep "Capacity Disk:"
The number returned indicates the percentage of the disk space available on this server. If the number is 5 or lower, then steps need to be taken to increase storage space by adding more Vaults, replacing drives with higher capacity drives, or removing unused content.
2. Check the /home/isa/ContentStore/server/ContentStore.log for ingest errors on each Vault. The master Vault has an additional log.
Monitoring Tasks for Streamers
1. In an ISA environment, look for any errors in the /Streaming/lscp_server/LSCPService.log on the primary Control server, and /Streaming/master/StreamService.log on the primary Setup server.
2. Look at the streaming log.
tail -f /arroyo/log/streamevent.log.<date>
Monthly Tasks
1. Choose Monitor > System Level > System Snapshot and check that the "Disk Available" amount meets the requirements for the expected movie storage in the next three to six months.
2. Run the reports for the last month that are suitable for your requirements and save them as comma-separated value (CSV) files.
3. Using the bandwidth and streaming reports, check that the CDS is not exceeding required usage per service area.
4. Run a quick security check.
a. Ensure that the CDSM changes can be attributed to individual users and not to a generic admin account.
b. Reset CDSM passwords if necessary.
c. Reset Linux passwords if necessary.
d. Check that access policies and firewalls are still enforced.
Other Tasks
If you have access to an anything on demand (XOD) application, do the following:
1. Check the inspect-live log for excessive errors.
2. Check the inspect-live log for excessive communication times with the BMS or CDS.
Feedback
