Contents

Introduction

Prerequisites

Requirements

Components Used

Background Information

Configure

Network Diagram

Configurations

Verify

Troubleshooting

Introduction

This document describes the steps to configure and troubleshoot Cisco Meeting Server (CMS) integration with Next Generation Streamer and Uploader. The Next Generation Streamer was introduced from CMS version 3.0 and is Session Initiation Protocol(SIP)-Based.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

- CMS Callbridge(s) Version 3.0 or later with Recording/Streaming license(s). (one recording license will allow one streaming call)

- Vbrick Distributed Media Engine (DME) (used for publishing the live stream from CMS Streaming service)

- Vbrick Rev (optional: only required if Live Streaming need to be shared outside internal network or multicast

- Network File System (NFS) directory is needed, and it can be setup on Windows Server or Linux.

- For Windows server, follow the steps toDeploy Network File system (NFS) on Windows

- For Linux, follow the steps toDeploy Network File systemon Linux

Components Used

The information in this document is based on these software and hardware versions:

- CMS 3.2 and above with 'Recorder' and/or 'Streamer' license(s). (recorder license will also allow you to stream)

- VBrick Distributed Media Engine (DME) 3.15.0 Rhel7

- Vbrick Rev (Uploader is used with vBrick Rev server. No manual importing of recordings is required)

- Windows Server 2012 R2 with NFS

Background Information

CMS Version 2.1 and later introduced support for live streaming with the CMS streamer using standard Real-Time Messaging Protocol (RTMP). In CMS 3.0, the Next Generation Streamer was introduced which is a SIP-based component. Prior version than 3.0 used Extensible Messaging and Presence Protocol (XMPP)  . CMS version 3.1 and above support RTMPS protocol and thus communcation between the CMS streamer component and external server can be encrypted. This allows for the CMS streamer to integrate with any streaming platform that supports RTMP(S) (Youtube, Facebook, Wowza, and so on). Currently the CMS Streamer has been tested with Vbrick DME as an external streaming server and is the recommended platform for integration.

Live Streaming (Webcast) integration with VBrick DME allow users to watch any live streamed CMS conference anywhere inside the network from different devices. Additionally, when VBrick Rev is used along VBrick DME, this extends this capability for viewing from outside the internal network for every VBrick Rev authorized user.Also, CMS Uploader component simplifies the work flow for uploading Meeting Server recordings to the video content manager, Vbrick, from a configured NFS connected to a Meeting Server. No manual importing of recordings is required. Once the Uploader component is configured and enabled, recordings are pushed from the NFS to Vbrick.

Streamer

The information in this document was created from the devices in a specific lab environment. All of the devices used in here started with cleared (default) configurations. If your network is live, make sure that you understand the potential impact of any command.

Uploader

The Uploader component can be installed on the same server as the Recorder component, or on a separate server. If installed on the same server as the Recorder, then add a couple of vCPUs for it to use. If run on a different server, then use the same server specification as for the Recorder: dedicated VM with a minimum of 4 physical cores and 4GB of RAM.

The Meeting Server running the Uploader will require  Read and Write permissions for the Network File Sharing (NFS). The Uploader must run on a different Meeting Server and not on Call Bridge hosting the conferences.

Configure

Network Diagram

There are several scenarios supported to deploy Streamer and Uploader with CMS such as: single callbridge with multiple streaming servers, a callbridge cluster with a single streaming server and callbridge cluster with multiple streaming servers. This document is based on a basic deployment with a cluster of callbridge connecting to a single streamer, Uploader server as all the configuration steps with this scenario apply to other scenarios as well.

As shown in the above image

CMS CallBridge Cluster

CMS Streamer/Recorder

CMS Uploader

VBrick DME For Streaming

VBrick Rev For Streaming

Configurations

Streamer

It is assumed that the callbridge is already setup and accepting calls.

Step 1. Certificates

The new streamer components do not require to listen https connections, however, it listen to SIP connections , the streamer server must have a valid certificate for TLS communication.

streamer> pki csr tac CN:.*.tptac9.com subjectAltName:streamer.tptac9.com

..............

......

Created key file tac.key and CSR tac.csr

CSR file tac.csr ready for download via SFTP

Get the certificate signed from local Certificate Authority (CA). Upload the generate certificate using Secure File Transfer Protocol (SFTP) to the Streamer server. Checked to confirm the certificates uploaded successfully.

In this document wildcard certificates are used for streamer. Please use the certificate guide for reference.

https://www.cisco.com/c/dam/en/us/td/docs/conferencing/ciscoMeetingServer/Deployment_Guide/Version-3-1/Certificate-Guidelines-for-all-Deployments-3-1.pdf

streamer> pki list

User supplied certificates and keys:

tac.key

tac.cer

ROOTCA.cer

example.key

example.csr

tac.csr

Step 2. MMP/SSH Configuration

- Configure the listening interface of the streamer and the SIP TCP and TLS ports to listen on using the MMP command

streamer sip listen <interface> <tcp-port|none> <tls-port|none>

streamer> streamer sip listen a 7000 7001

To use only TLS sip connection. Configure TCP sip connection as "none" the command is below

streamer> streamer sip listen a none 7001

- Apply certificates for the streamer server

streamer> streamer sip certs tac.key tac.cer

- Select the quality for streaming

streamer> streamer sip resolution 720p

- Enable streamer

streamer> streamer enable

- Optionally, if TLS is configured, you can perform TLS verification for SIP on the streamer

streamer> tls sip trust ROOTCA.cer

Note: For the TLS connection to be secure we recommend enabling TLS verification.

streamer> tls sip verify enable

- Verfiy the configuration information entered above is correct

streamer> streamer

Enabled                 : false

SIP interfaces          : tcp a:7000, tls a:7001

SIP key file            : tac.key

SIP certificate file    : tac.cer

SIP CA Bundle file      : none

SIP Resolution          : 720p

SIP traffic trace       : Disabled

Call Limit              : none

- Enable the streamer by using command: "streamer enable". All message must show "SUCCESS" as below

streamer> streamer enable

SUCCESS: Key and certificate pair match

SUCCESS: Streamer enabled

Step 3. API configuration

This configuration is performed in the CMS hosting CallBridge. Below, API on Webadmin interface of CMS. Any REST client like Postman, Poster can also be used to perform this.

Once the new SIP streamer is enabled, it can be configured and used in the Call Bridge under /callProfiles for sipStreamerUri.

To use API on CMS server. Go To  Webadmin > Configuration > API

- Create Call Profile

- Configure sipStreamuri=demo@streamer.com & StreamingMode= Automatic/Manual

The user part of the configured "sipStreamuri" (i.e. the part before '@' symbol) has no significant meaning, and for the new SIP streamer component, although required, it can usually be anything, e.g. "streamer@streamer.com".The important part of the URI is the "domain" part.

Configure sipStreamuri=demo@streamer.com & StreamingMode= Automatic/Manual

- Add the created streamer callProfile above to the /system/profiles. This is a global configuration and the configured "sipStreamerUri" will be used for streamer operation.

Add the callProfile to system/profiles

- Add the VBrick 'streamURL' to the space(s) used for streaming. For reference, a space called 'Stream' was created using the CMS web interface

Space used for Streaming

- Modify the space to add “StreamURL”. The 'streamURL' in the following format:  rtmp://<VBrickBroadcastUsername>:<VBrickBroadcastPassword>@<VBrick IP or FQDN>/live/NameoftheStream. In example I configured it as “rtmp://broadcast:broadcast@10.106.81.40/live/SpaceStream77”

Step 4. Create "Outbound Rules"

Configure, a custom URI that maps to an outboundDialPlan rule (the domain can be anything for example "streamer.com"). Please configure an outboundDialPlan rule to match the domain used in streamerUri to route.

Create outbound rules

As shown in the above image, for SIP streamer, if default ports for SIP (5060,5061) are not used, then It is mandatory to specify ports in the configuration of the streamer and include the following port number to connect to the "sip proxy to use" field when outboundDialPlanRule is configured for the service.

streamer> streamer

Enabled                 : true

SIP interfaces          : tcp a:7000, tls a:7001

SIP key file            : tac.key

SIP certificate file    : tac.cer

SIP CA Bundle file      : none

SIP Resolution          : 720p

SIP traffic trace       : Disabled

Call Limit              : none

Uploader

- Specify the NFS and directory where recordings will be stored that the Uploader will monitor

streamer> uploader nfs 192.168.15.38:Recording

- Specify the Meeting Server that the Uploader will query for recording information

streamer> uploader cms host join.mextp.local

- Specify the Web Admin port on the Meeting Server running the Call Bridge

streamer> uploader cms port 445

- Specify the user with API access on the Meeting Server running the Call Bridge

streamer> uploader cms user apiadmin

streamer> uploader cms password

Please enter password:

- Add the certificate bundle from CMS to the Meeting Server trust store

Create a certificate bundle (crt-bundle) holding a copy of the Root CA’s certificate and all intermediate certificates in the chain for the Web Admin on the Meeting Server running the Call Bridge.

streamer> uploader cms trust ROOTCA.cer

- Configure the Vbrick host and the port to which the Uploader will connect

streamer> uploader rev host ciscotac.rev-na.demo.vbrick.com

streamer> uploader rev port 443

Note: The port defaults to 443 unless otherwise specified

- Add a Vbrick Rev user who has API permission to upload video recordings

streamer> uploader rev user tacuser

streamer> uploader rev password

Please enter password:

- Add the certificate bundle to the Vbrick Rev trust store

Create a certificate bundle (crt-bundle) holding a copy of the Root CA’s certificate and all intermediate certificates in the chain for the Vbrick Rev serve

streamer> uploader rev trust vbrickbundle.cer

- Check uploader configuration and enable uploader

streamer> uploader

Enabled                          : false

NFS hostname                     : 192.168.15.38

NFS directory                    : Recording

CMS host                         : join.mextp.local

CMS port                         : 445

CMS user                         : apiadmin

CMS trust bundle                 : ROOTCA.cer

Vbrick Rev hostname              : ciscotac.rev-na.demo.vbrick.com

Vbrick Rev port                  : 443

Vbrick Rev username              : tacuser

Vbrick Rev trust bundle          : brick.cer

View access                      : Public

cospace_member_access            : edit

recording_owned_by_cospace_owner : false

fallback_owner                   : admin

comments_enabled                 : true

ratings_enabled                  : true

downloads_enabled                : true

active_upon_upload               : true

delete_after_upload              : false

If configuration is correct, use "uploader enable" command to enable the Uploader component. All message must show "SUCCESS" as shown below.

streamer> uploader enable

SUCCESS: uploader enabled

Verify

Streamer

Streaming working and a sip streaming call connected

Uploader

You can see log for a successful event in syslog follow of uploader.

Jun 17 22:24:41.867 user.info cms-02 Uploader[1]: scanning directory: /mnt/recordings/forwardedCalls

Jun 17 22:24:41.867 user.info cms-02 Uploader[1]: scanning directory: /mnt/recordings/spaces

Jun 17 22:24:41.869 user.info cms-02 Uploader[1]: checking the status of /mnt/recordings/spaces/8a7076e2-6db6-47e9-98ee-3bd063e32559/20210618032309+0000_vid-id=c4605aaf-dc49-4cd7-9174-c46185ba1983@vbrick.mp4

Jun 17 22:24:41.870 user.info cms-02 Uploader[1]: Getting from: https://ciscotac.rev-na.demo.vbrick.com:443/api/v1/videos/c4605aaf-dc49-4cd7-9174-c46185ba1983/status

Jun 17 22:24:42.035 user.info cms-02 Uploader[1]: Received vbrick response status code: 200

Jun 17 22:24:42.035 user.info cms-02 Uploader[1]: vbrick response: main.vbrickStatusResp{Status:"Ready"}

Jun 17 22:24:42.035 user.info cms-02 Uploader[1]: file 20210618032309+0000_vid-id=c4605aaf-dc49-4cd7-9174-c46185ba1983@vbrick.mp4 vid c4605aaf-dc49-4cd7-9174-c46185ba1983 status Ready

Jun 17 22:24:42.035 user.info cms-02 Uploader[1]: Getting from: https://ciscotac.rev-na.demo.vbrick.com:443/api/v1/videos/c4605aaf-dc49-4cd7-9174-c46185ba1983/playback-url

Jun 17 22:24:42.200 user.info cms-02 Uploader[1]: Received vbrick response 200

Troubleshooting

Streamer

1. No License

Streamer required "recorder" licnese on the server having callbridge component. If that is not there, or insufficient license is present, then errors as shown below will be seen in Event Logs.

Make sure to add required license. Status of license can be checked CLI using the command "license"

cms1> license

Feature: callbridge status: Activated expiry: 2023-Apr-28 (690 days remain)

Feature: turn status: Activated expiry: 2023-Apr-28 (690 days remain)

Feature: webbridge status: Activated expiry: 2023-Apr-28 (690 days remain)

Feature: customizations status: Activated expiry: 2023-Apr-28 (690 days remain)

Feature: local_license_mode status: Activated expiry: 2023-Apr-28 (690 days remain)

Feature: recording status: Activated expiry: 2023-Apr-28 (690 days remain)

Feature: personal status: Activated expiry: 2023-Apr-28 (690 days remain)

Feature: shared status: Activated expiry: 2023-Apr-28 (690 days remain)

2.TLS port

• If TLS port configured and no certificates are applied. Configure certs for streamer to use TLS

• If certificate are not available. Configure the TCP port only

streamer> streamer sip listen a 7000 7001

streamer> streamer enable

FAILURE: TLS port set but no certificates configured

FAILURE: Streamer configuration not complete

Now you have 2 options, either to remove TLS Port, or to add SIP TLS Trust and Streamer Certificate

Cisco recommends to have TLS Port enabled.

3. RTMP Sream not configured correctly

You will see error in logs

daemon.info streamer streamer-sip[2280]:  144500.368 : INFO    : call 3: retrieved stream URL from RTCP: "rtmp://broadcast:broadcast@10.106.81.40/test"

daemon.info streamer streamer-sip[2280]:  144500.368 : INFO    : call 3: parsing rtmp://broadcast:broadcast@10.106.81.40/test

daemon.info streamer streamer-sip[2280]:  144500.368 : INFO    : call 3: RTMP stream="test"

daemon.info streamer streamer-sip[2280]:  144500.368 : INFO    : call 3: RTMP server="rtmp://10.106.81.40:1935/test"

daemon.info streamer streamer-sip[2280]:  144500.370 : INFO    : call 3: Connected to RTMP server

daemon.info streamer streamer-sip[2280]:  144500.370 : INFO    : call 3: C2 pending - len 1536

daemon.info streamer streamer-sip[2280]:  144500.370 :         : call 3: snd: create new chunk stream 2

daemon.info streamer streamer-sip[2280]:  144500.370 :         : call 3: snd: create new chunk stream 3

daemon.info streamer streamer-sip[2280]:  144500.370 : INFO    : call 3: RTMP sent chunk size of 4096 and connect message

daemon.info streamer streamer-sip[2280]:  144500.410 : ERROR   : call 3: connection : far end closed connection 5

Please check the procedure in streamer configuration and configure RTMP URL correctly in the  format "rtmp://<VBrickBroadcastUsername>:<VBrickBroadcastPassword>@<VBrick IP or FQDN>/live/NameoftheStream"

4.Call Routing Related Issues

Because the CMS streamer is a SIP-based client and as discussed earlier, it requires routing to be in place. This could cause scenarios where calls might fail. Consider this example, where the CMS Callbridge sent an outbound call, but it failed with the following 'transaction timeout - no provisional responses sending INVITE' error

2021-06-28  17:37:02.412  Info     user 'guest300535034' starting streaming (space 'test')

2021-06-28  17:37:02.413  Info     API call leg bc0917df-589c-4628-887d-79481d322fed in call 63f0b174-831e-4a12-b4ee-27186d4162af (API call 00286960-9af9-4d5d-9ca7-20dd40425292)

2021-06-28  17:37:02.413  Info     call 44: outgoing SIP call to "demo@streamer.com" from space "test"

2021-06-28  17:37:02.413  Info     call 44: configured - API call leg bc0917df-589c-4628-887d-79481d322fed with SIP call ID "7d37a80e-7996-4e8d-aa87-77c9d4729cec"

2021-06-28  17:37:04.482  Info     call 42: receiver report 1 interval for rx video 0 = 6113ms (period 6108ms) 00000000

2021-06-28  17:37:22.074  Info     call 44: falling back to unencrypted control connection...

2021-06-28  17:37:54.075  Info     call 44: ending; local SIP teardown with reason 7 (transaction timeout - no provisional responses sending INVITE) - not connected after 0:52

2021-06-28  17:37:54.075  Info     call 44: destroying API call leg bc0917df-589c-4628-887d-79481d322fed

2021-06-28  17:37:54.076  Info     streaming call leg for space 'test' disconnected with reason 7 (transaction timeout - no provisional responses sending INVITE)

Review the Outbound calls settings on the CMS Callbridge servers to validate the location it is being sent to and if being set correctly. Also check if the callprofile is configured with correct streamer URI and same is associated with Cospace.

Uploader

1. Vbrick detials not correct.

You can see error in Uploader log

Jun 27 11:29:27.864 user.info streamer Uploader[1]: Received vbrick response 500

Jun 27 11:29:27.864 user.info streamer Uploader[1]: posting to: https://sales.vbrick.com:443/api/v1/user/login

Jun 27 11:29:47.870 user.info streamer Uploader[1]: Received vbrick response 500

Jun 27 11:29:47.870 user.err streamer Uploader[1]: Failed to initialise Vbrick Client

Jun 27 11:29:47.870 user.err streamer Uploader[1]: vbrick returned status code: 500

Make sure to have configured the correct credentials and port for the vbrick server.  Also make sure Uploader should be able to reach CMS callbridge webadmin port.