Configure Land Mobile Radio (LMR) / Hoot and Holler Over IP on IOS-XE Voice Gateways

Introduction

This document describes Land Mobile Radio (LMR) or Hoot and Holler (Hootie) feature which allows analog devices to communicate with other (analog and IP) endpoints across a multicast enabled LAN.

The Voice Gateway acts as a demarcation point between the IP Network and the Analog endpoints and facilitates the conversation between analog audio and multicast Real-time Transport Protocol (RTP).

Contributed by Kyzer Davis and Matt Snow, Cisco TAC Engineers.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• Digital Signal Processor (DSP)
• Analog Cards
• Applicable Licenses for the feature

!
license boot level appxk9
license boot level uck9
! or
license boot suite FoundationSuiteK9
license boot suite AdvUCSuiteK9

• Multicast Enabled LAN or WAN

Note: This document does not cover the many facets of Multicast configuration on the LAN or WAN. Refer to applicable documentation to enable Multicast on LAN or WAN devices in the network path.

Components Used

• 4451-X
• NIM-4E/M
• IOS-XE 16.3 or above. (Release Notes) [Recommended: IOS-XE 16.7 or above]

ISR4451# show inventory
NAME: "Chassis", DESCR: "Cisco ISR4451 Chassis"
PID: ISR4451-X/K9      , VID: V03  , SN: XXXXXXXXX

NAME: "NIM subslot 0/3", DESCR: "NIM-4E/M Voice Analog Module"
PID: NIM-4E/M          , VID: V01  , SN: XXXXXXXXX

Note: Analog NIM cards on with ISR 4000 Voice Gatways utlize on-NIM DSP(s). Thus no motherboard DSP is required.

Background Information

Potential Use Cases:

• Radio equipment and alert which includes push-to-talk devices
• Multicast informational announcements (Radio Broadcasts)
• Analog Turret Systems

Note: These are a few sample use cases. The application is not limited to these functions.

The original design guide for LMR does not cover the required items for the latest generation of Cisco Voice Gateways. Hence, this document aims describes the LMR / Hootie feature in regards to IOS-XE devices such as ISR 4300 and 4400 series voice gateways.

Here is a Sample Topology;

Layer 7 Signaling and Media

Analog Endpoint <> Ear and Mouth (E&M) Port <> Cisco Voice Gateway (4451-X) <> Multicast Enabled LAN <> IP Endpoint.

Tip: Remember, since the IP backbone uses multicast, the Voice Gateway only needs to be able to join the desired multicast group successfully. The voice gateway does not know about the other endpoints nor does it communicate with them directly as a result this document details a sample configuration, debugs, show commands, and troubleshooting in one LMR/Hootie Voice Gateway.

Configuration

Step 1. You must first configure the IOS-XE licenses required to operate voice and the multicast feature.

config t
!
license boot level appxk9
license boot level uck9
! or
license boot suite FoundationSuiteK9
license boot suite AdvUCSuiteK9
!
exit
!
wr
!
reload
!

When the device has been powered back verify the license status matches this show command output:

ISR4451# show license feature
Feature name             Enforcement  Evaluation  Subscription   Enabled  RightToUse
appxk9                   yes          yes         no             yes      yes
uck9                     yes          yes         no             yes      yes

Step 2. Next you define a Multicast Voice over IP Dial-peer which contains the desired multicast IP and port ;

!
dial-peer voice 33333 voip
 destination-pattern 33333
 session protocol multicast
 session target ipv4:239.X.X.X:21000
 codec g711ulaw
 vad aggressive
!

Dial-peer Command Syntax:

Step 3. In order to facilitate a permanent (always up) connection between the multicast group and this voice gateway for the analog port you must define a voice-class permanent and then apply this to the voice-port.

!
voice class permanent 1
 signal timing oos timeout disabled
 signal keepalive disabled
!
voice-port 0/1/0
 voice-class permanent 1
!

voice class permanent command syntax

The voice-port is then configured for the desired type of connection for the E&M Port and then the command. (E&M or other analog specific configurations not covered in this document Refer to E&M Configuration Guide for more information.)

Step 4. Cisco hoot and holler over IP provides an Always-On communications bridge. End users do not need to dial any phone numbers to contact the other members of a hoot group. In order to simulate this functionality, Cisco IOS provides a feature called Connection Trunk. Connection trunk provides a permanent voice call, which does not require any input from the end user, because all the digits are internally dialed by the router/gateway.

This connection trunk ties the voice-port to a multicast address you configured in the dial-peer configuration step.

!
voice-port 0/1/0
 connection trunk 33333
!

Analog Port command Syntax

Step 5. Once the voice configuration is complete you need to define the multicast configuration.

!
ip multicast-routing distributed
!
interface GigabitEthernet0/0/1
 ip address Y.Y.Y.Y 255.255.255.0
 ip pim sparse-mode
!
interface Vif1
 ip address 192.0.2.2 255.255.255.0
 ip pim sparse-mode
! 
interface Service-Engine0/1/0
 ip pim sparse-mode
!
ip pim rp-address 2.x.x.x
!

Notes about the Multicast configuration:

• The Service-Engine interfaces is the layer 3 interface for the PVDM on the Analog NIM. This needs to be configured with a Protocol Independent Multicast (PIM) command like any other ingress / egress layer 3 interface
• The Service-Engine does not require an IP address
• The type of PIM configure depends on the type of Multicast implementation on your LAN
• Multicast routing MUST be enabled, even if all traffic is within the same VLAN
• For Multicast RTP sourced from the router, the IP must be the VIF IP minus 1. So our source must be 192.0.2.1 because we have configured 192.0.2.2 on the VIF
  • In some scenarios this may be VIF plus 1 but for this configuration the VIF assumed minus 1. Always check show ip mroute to see what VIF is being used by the router.
• The Multicast PIM RP can be the same Voice Gateway however for this lab the Multicast PIM RP is on another device in the network (2.x.x.x) which is learned by way of EIGRP (Not shown)

Verify

Use this section to confirm that your configuration works properly.

Voice Verification

When the configurations are complete a permanent connection stoods up. You can use this show command output to verify it;

ISR4451# show call active voice compact
 <callID>  A/O FAX T<sec> Codec       type        Peer Address       IP R<ip>:<udp>        VRF
Total call-legs: 2
       115 ANS     T24    g711ulaw    TELE        P
       116 ORG     T0     g711ulaw    VOIP        P33333      239.X.X.X:21000                   

ISR4451# show voip rtp connections
VoIP RTP Port Usage Information:
Max Ports Available: 19999, Ports Reserved: 101, Ports in Use: 0
Port range not configured
                                        Min   Max   Ports     Ports     Ports  
Media-Address Range                     Port  Port  Available Reserved  In-use 
------------------------------------------------------------------------------
Global Media Pool                       8000  48198 19999     101       0      
------------------------------------------------------------------------------
VoIP RTP active connections :
No. CallId     dstCallId  LocalRTP RmtRTP   LocalIP                                RemoteIP                                  MPSS VRF 
1     116        115        15986    21000    192.0.2.1                           239.X.X.X                               NO   NA                  
Found 1 active RTP connections

ISR4451# show voice port summary
                                           IN       OUT
PORT            CH   SIG-TYPE   ADMIN OPER STATUS   STATUS   EC
=============== == ============ ===== ==== ======== ======== ==
0/3/1           --  e&m-imd     up    up   trunked  trunked  y 

ISR4451# show voice call summary
PORT           CODEC     VAD VTSP STATE            VPM STATE
============== ========= === ===================== =====================
0/3/1          g711ulaw   y  S_CONNECT             S_TRUNKED                

ISR4451# show voice call status
CallID     CID  ccVdb      Port        Slot/Bay/DSP:Ch  Called #   Codec    MLPP Dial-peers
0x73       12D0 0x7F7475CF8C08 0/3/1            0/3/1:1 33333      g711ulaw  4 777 33333777/33333
1 active call found

ISR4451# show voice trunk-conditioning supervisory 
FAST SCAN
0/3/1 : state : TRUNK_SC_CONN_DEFAULT_OOS, voice : off , signal : on ,master
status: lost keepalive, trunk connected
sequence oos : idle and oos
pattern :rx_idle = 0000 rx_oos = 1111 
timeout timing : idle = 0, idle_off = 0, restart = 120, standby = 0, timeout = 30
supp_all = 0, supp_voice = 0, keep_alive = 5
timer: oos_ais_timer = 46, timer = 43


ISR4451# show voice trunk-conditioning signaling 

0/3/1 : 
hardware-state ACTIVE signal type is NorthamericanCAS
status : lost keepalive,
forced playout pattern = 0xF
idle monitoring : disabled
tx_idle = FALSE, rx_idle = FALSE, tx_oos = FALSE, lost_keepalive = TRUE
trunk_down_timer = 0, rx_ais_duration = 0, idle_timer = 0,tx_oos_timer = 0

In order to verify IP to Analog replication first check the new IOS-XE Command:

ISR4451# show platform hardware qfp active feature sbc hootie group
SBC Hootie structure :
---------------------------------------

  VRF                                   = 0
  IP                                    = 239.X.X.X
  Port                                  = 21000
  Protocol                              = 1
  Calls in group                        = 1

SBC Hootie group Statistics
---------------------------------------

  Total RTP packets received                  = 2873
  Total RTP octects received                  = 573520
  Total RTP packets replicated                = 2873
  Total RTP octects replicated                = 573520
  Total RTP packets dropped                   = 0
  Total RTP octects dropped                   = 0


ISR4451# show platform hardware qfp active feature sbc hootie group
SBC Hootie structure :
---------------------------------------

  VRF                                   = 0
  IP                                    = 239.X.X.X
  Port                                  = 21000
  Protocol                              = 1
  Calls in group                        = 1

SBC Hootie group Statistics
---------------------------------------

  Total RTP packets received                  = 3111
  Total RTP octects received                  = 621032
  Total RTP packets replicated                = 3111
  Total RTP octects replicated                = 621032
  Total RTP packets dropped                   = 0
  Total RTP octects dropped                   = 0

Multicast Verification

Verify PIM Neighbors:

ISR4451# show ip pim neighbor 
PIM Neighbor Table
Mode: B - Bidir Capable, DR - Designated Router, N - Default DR Priority,
      P - Proxy Capable, S - State Refresh Capable, G - GenID Capable,
      L - DR Load-balancing Capable
Neighbor          Interface                Uptime/Expires    Ver   DR
Address                                                            Prio/Mode
Y.Y.Y.Y      GigabitEthernet0/0/1     00:20:13/00:01:41 v2    1 / DR S P G

Verify the mroute output is correct:

ISR4451# show ip mroute
[snip]

(192.0.2.1, 239.X.X.X), 00:01:08/00:02:20, flags: FT
  Incoming interface: Vif1, RPF nbr 0.0.0.0
  Outgoing interface list:
    GigabitEthernet0/0/1, Forward/Sparse, 00:01:08/00:03:19

Verify we have the Multicast RP in the list:

ISR4451# show ip igmp member
Flags: A  - aggregate, T - tracked
       L  - Local, S - static, V - virtual, R - Reported through v3 
       I - v3lite, U - Urd, M - SSM (S,G) channel 
       1,2,3 - The version of IGMP, the group is in
Channel/Group-Flags: 
       / - Filtering entry (Exclude mode (S,G), Include mode (G))
Reporter:
       <mac-or-ip-address> - last reporter if group is not explicitly tracked
       <n>/<m>      - <n> reporter in include mode, <m> reporter in exclude

 Channel/Group                  Reporter        Uptime   Exp.  Flags  Interface 
 *,239.X.X.X                  192.0.2.2    00:01:16 01:43 2VA    Vi1

Verify multicast packet replication:

RP# show ip mroute count
[snip]
Group: 239.X.X.X, Source count: 1, Packets forwarded: 2107, Packets received: 2108
  RP-tree: Forwarding: 2/0/56/0, Other: 2/0/0
  Source: 192.168.19.1/32, Forwarding: 2105/50/158/80, Other: 2106/0/1


RP# show ip mroute count
[snip]
Group: 239.X.X.X, Source count: 1, Packets forwarded: 2190, Packets received: 2191
  RP-tree: Forwarding: 2/0/56/0, Other: 2/0/0
  Source: 192.168.19.1/32, Forwarding: 2188/50/159/80, Other: 2189/0/1

The Cisco CLI Analyzer (registered customers only) supports certain show commands. Use the Cisco CLI Analyzer in order to view an analysis of show command output.

Troubleshoot

This section provides information you can use in order to troubleshoot your configuration.

Call Setup Issues

 If the connection is not established, first verify the signaling through these debugs:

debug vpm signal
debug voip vtsp session
debug voip ccapi inout

Debug Sample:

123165: Oct  XX 13:21:55.563: htsp_process_event: [0/3/1, S_DOWN, E_HTSP_IF_INSERVICE]
123166: Oct  XX 13:21:55.564: %LINK-3-UPDOWN: Interface recEive and transMit 0/3/1, changed state to up
123167: Oct  XX 13:21:55.564: recEive and transMit 0/3/1 rx_signal_map:
 0 0 0 0
 0 0 0 0
 8 8 8 8
 8 8 8 8
123168: Oct  XX 13:21:55.564: recEive and transMit 0/3/1 tx_signal_map:
 0 0 0 0
 0 0 0 0
 C C C C
 C C C C
123169: Oct  XX 13:21:55.564: htsp_process_event: [0/3/1, S_OPEN_PEND, E_HTSP_GO_TRUNK]em_trunk_null_init 
123170: Oct  XX 13:21:55.564: flex_set_Legerity_impedance: [0/3/1] impedance = 0
123171: Oct  XX 13:21:55.704: htsp_process_event: [0/3/1, S_TRUNK_NULL, E_HTSP_INSERVE]default_trunk_down
123172: Oct  XX 13:21:55.704: htsp_timer - 6204 msec
123173: Oct  XX 13:21:55.919: %SYS-5-CONFIG_I: Configured from console by vty3 (192.168.19.2)
123174: Oct  XX 13:22:01.908: htsp_process_event: [0/3/1, S_TRUNK_PEND, E_HTSP_EVENT_TIMER]
123175: Oct  XX 13:22:01.908: htsp_timer_stop htsp_setup_ind
123176: Oct  XX 13:22:01.908: [0/3/1] get_local_station_id calling num= calling name= calling time=10/08 13:22  orig called=
123177: Oct  XX 13:22:01.908: htsp_timer - 2000 msec

123181: Oct  XX 13:22:01.909: //-1/80F08D0180E8/CCAPI/cc_api_call_setup_ind_common:
   Interface=0x7F7475CF8C08, Call Info(
   Calling Number=,(Calling Name=)(TON=Unknown, NPI=Unknown, Screening=Not Screened, Presentation=Allowed),
   Called Number=33333(TON=Unknown, NPI=Unknown),
   Calling Translated=FALSE, Subscriber Type Str=RegularLine, FinalDestinationFlag=TRUE,
   Incoming Dial-peer=777, Progress Indication=ORIGINATING SIDE IS NON ISDN(3), Calling IE Present=FALSE,
   Source Trkgrp Route Label=, Target Trkgrp Route Label=, CLID Transparent=FALSE), Call Id=-1

123203: Oct  XX 13:22:01.911: //115/80F08D0180E8/CCAPI/ccCallSetupRequest:
   Calling Number=(TON=Unknown, NPI=Unknown, Screening=Not Screened, Presentation=Allowed),
   Called Number=33333(TON=Unknown, NPI=Unknown),
   Redirect Number=, Display Info=
   Account Number=, Final Destination Flag=TRUE,
   Guid=80F08D01-CA55-11E8-80E8-8E0AC3C8E4C4, Outgoing Dial-peer=33333

123252: Oct  XX 13:22:01.914: //116/80F08D0180E8/CCAPI/cc_api_caps_ack:
   Destination Interface=0x7F7475CF8C08, Destination Call Id=115, Source Call Id=116,
   Caps(Codec=g711ulaw(0x1), Fax Rate=FAX_RATE_VOICE(0x2), Fax Version:=0, Vad=AGGRESSIVE(0x4),
   Modem=OFF(0x0), Codec Bytes=160, Signal Type=2, Seq Num Start=2165)
123253: Oct  XX 13:22:01.914: //115/80F08D0180E8/CCAPI/cc_api_caps_ack:
   Destination Interface=0x7F7471175B68, Destination Call Id=116, Source Call Id=115,
   Caps(Codec=g711ulaw(0x1), Fax Rate=FAX_RATE_VOICE(0x2), Fax Version:=0, Vad=AGGRESSIVE(0x4),
   Modem=OFF(0x0), Codec Bytes=160, Signal Type=2, Seq Num Start=2165)


123255: Oct  XX 13:22:01.914: //115/80F08D0180E8/VTSP:(0/3/1):-1:1:1/vtsp_call_connect: Connected Name 
123256: Oct  XX 13:22:01.914: //115/80F08D0180E8/VTSP:(0/3/1):-1:1:1/vtsp_call_connect: Connected Number 33333
123257: Oct  XX 13:22:01.914: //115/80F08D0180E8/VTSP:(0/3/1):-1:1:1/vtsp_call_connect: Connected oct3a  0
123258: Oct  XX 13:22:01.914: //115/80F08D0180E8/CCAPI/ccCallConnect:
   Call Entry(Connected=TRUE, Responsed=TRUE)

123265: Oct  XX 13:22:01.916: htsp_process_event: [0/3/1, S_TRUNK_W_CUTTHRU, E_HTSP_VOICE_CUT_THROUGH]
123266: Oct  XX 13:22:01.916: send_trunk_dsp_voice_chnl_mapping:[0/3/1], 1/0/0 
123267: Oct  XX 13:22:01.916: send_trunk_dsp_sig_chnl_mapping:[0/3/1], 129/0/0 
123268: Oct  XX 13:22:01.916: recEive and transMit 0/3/1 rx_signal_map:
 0 0 0 0
 0 0 0 0
 0 0 0 0
 0 0 0 8 default_trunk_up
123269: Oct  XX 13:22:01.916: recEive and transMit 0/3/1 tx_signal_map:
 0 0 0 0
 0 0 0 0
 F F F F
 F F F F default_trunk_updefault_trunk_up
123270: Oct  XX 13:22:01.916: recEive and transMit 0/3/1 rx_signal_map:
 0 0 0 0
 0 0 0 0
 0 0 0 0
 0 0 0 8 default_trunk_up
123271: Oct  XX 13:22:01.916: recEive and transMit 0/3/1 tx_signal_map:
 0 0 0 0
 0 0 0 0
 F F F F
 F F F F default_trunk_up
123272: Oct XX 13:22:01.916: %HTSP-5-UPDOWN: Trunk port(channel) [0/3/1] is up

If you see this error,  it is due to session protocol multicast command not available on the dial-peer.

%VOICE_IEC-3-GW: H323: Internal Error (H225 chn, sock fail in RAS): IEC=1.1.186.5.81.0

Audio Issues

If the problem lies in no audio, verify that the Voice gateway has correctly joined the multicast group. Refer to the command outputs in the verification section of this document for a baseline output of a working device. The outgoing interface of the show ip mroute command for the specific multicast group must never be Null. If you see a Null outgoing interface review applicable network configurations for the multicast LAN becuase this indicates the voice gateway could not properly join the multicast group.

Sample Null Outgoing Interface:

Router# show ip mroute 239.X.X.X
(*, 239.X.X.X), 00:22:02/stopped, RP 10.188.0.1, flags: SJCF
  Incoming interface: GigabitEthernet0/0/1, RPF nbr X.X.X.X
  Outgoing interface list:
    Vif1, Forward/Sparse-Dense, 00:18:27/00:02:32

(A.B.C.D, 239.X.X.X), 00:20:34/00:01:23, flags: PFT
  Incoming interface: Vif1, RPF nbr 0.0.0.0
  Outgoing interface list: Null 

If the device is correctly in the multicast group but audio issues still persist, use command show platform hardware qfp active feature sbc hootie group a few times to verify if the device is able to receive and replicate packets. The counters must increase each time the command is run. Alternatively, the command show platform hardware qfp active statistics drop can be run to see if the voice gateway drops the traffic. In order to clear these counters run the command show platform hardware qfp active statistics drop clear.

If IP multicast-routing is not configured the drop reason of Ipv4mcNoRoute increments as shown:

4451# show platform hardware qfp active statistics drop
-------------------------------------------------------------------------
Global Drop Stats                         Packets                  Octets
-------------------------------------------------------------------------
Ipv4mcNoRoute                                 728                  145272

Other audio issues such as ones where the gateway is unable to replicate multicast RTP packets recieved on analog side  to IP side, can occur due to a problem with the multicast configuration. These issues can manifest themselves as the drop reason FIAError when these drops are observed. When these are observed, review the applicable multicast configurations and ensure the gateway can properly join the multicast group and that the show ip mroute command has a valid output interface. See the multicast section of this document for baseline command outputs.

4451# show platform hardware qfp active statistics drop
-------------------------------------------------------------------------
Global Drop Stats                         Packets                  Octets
-------------------------------------------------------------------------
FIAError                                      724                  144800

If multicast routing is not enabled the output of show ip mroute states as such.

ISR4451# sh ip mroute
IP Multicast Forwarding is not enabled.
[snip]

PCM Capture

In order to verify if analog audio is sent or received on a voice-port, you can take a PCM capture. Full PCM Documentation

conf t
voice pcm capture buffer 200000
voice pcm capture destination bootflash:
exit
!
test voice port 0/1/0 pcm-dump caplog ffffff duration 255
! send audio
test voice port 0/1/0 pcm-dump disable
!
copy flash:/<filename>.dat [ftp://user:pass@ip.address/filename.pcap | tftp://a.b.c.d/filename]
! TAC is required to decode the binary .dat file into SIN/SOUT/RIN audio streams

Packet Capture (PCAP)

In order to verify if multicast RTP is sent or received, you can take a Packet Capture (PCAP)  on the physical interface. Full EPC Documentation.

! NOT IN CONFIGURATION TERMINAL
monitor capture TAC int gig0/0/1 both
monitor capture TAC match any
!
monitor capture TAC start
! send audio
monitor capture TAC stop
!
monitor capture TAC export [flash:/filename.pcap | ftp://user:pass@ip.address/filename.pcap | tftp://a.b.c.d/filename]
!
monitor capture TAC clear

DSP Test Tone

If required a test tone can be generated by the DSP / PVDM on the voice gateway in the desired direction (Network-IP-LAN side or Local-Analog-Port side).

This tone can be directed to the DSP towards the IP LAN Multicast address. These commands can be used to enable/disable. The connection must be active and you must specify the analog port for the test.

test voice port 0/1/0 inject-tone network 1000
! A 1000hz tone is now being generated from the analog port to the IP LAN Multicast Address
test voice port 0/1/0 inject-tone network disable

In order to generate a tone from the DSP out the analog port these commands can be used to enable/disable. The connection must be active and you must specify the analog port for the test.

test voice port 0/1/0 inject-tone local 1000
! A 1000hz tone is now being generated out of the analog port.
test voice port 0/1/0 inject-tone local disable

Test Multicast Reception with VLC Media Player

Download VLC Media Player and navigate to Media > Open Network Stream

Enter the multicast RTP IP address in this format and hit play

rtp://@239.X.X.X:21000

Next download and open Wireshark. Then select the specifc interface desired for packet capture.

Start a capture with the filter of rtp.

If all went well you must be joined to the multicast RP. (The same multicast commands can be run from the RP to verify the PC joined the multicast group).

Either generate a tone through the tone commands or have an analog endpoint speak.

You must now see packets in wireshark. Remember, the source IP must be the VIF IP minus 1 so for our test it must be 192.0.2.2 -1 = 192.0.2.1.

The Cisco CLI Analyzer (registered customers only) supports certain show commands. Use the Cisco CLI Analyzer in order to view an analysis of show command output.

Note: Refer to Important Information on Debug Commands before you use debug commands.

Related Information

• Known Defects

CSCvd18792 - ISR4K - Hoot and Holler E&M port cannot be co-located with multicast hub

CSCve66876 - ISR4K - multicast RP registration is dropped for packets from DSP

CSCve71893 - ISR4K - Hoot and Holler multicast replication issue

• Technical Support & Documentation - Cisco System