Troubleshooting Guide for 3G Module for CGR 1000 Series Router (Cisco CG-OS)

Introduction

This document describes the steps to troubleshoot 3G module for Cisco Connected Grid Router (CGR) 1000 series router with CG-OS. The article also discuss the issues, cause and remedial steps to resolve the problems. 

When you troubleshoot modem issues, the first step is to determine if the CGR can recognize the module. This document explains the steps necessary to determine this and how to proceed with the process.

Contributed by Sumedha Phatak and Abhishek Kumar, Cisco TAC Engineers.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• CLI access to the CGR 1000 series router installed with 3G module and SIM card with active data plan.
• 3G wireless network coverage at the router's physical location.

Components Used

The information in this document is based on these software and hardware versions:

• CGR 1000 series routers
• Cisco CG-OS software

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Troubleshoot

When you troubleshoot modem issues, the first step is to determine if the CGR can recognize the module.

Verify that CGR can Recognize the 3G Module

After the 3G module is installed and configured, sometimes the CGR is not able to recognize the module.

1. Run the show cellular 3/1 all command to verify if the CGR can recognize the module and display all the related information. If all the fields in the output show "unknown", reload the module and wait for upto two mins before you verify again.

2. If all the fields show "unknown" even after reload, shut/ no shut the cellular interface and wait for upto two mins before you issue the show cellular 3/1 all command again.

Note: If all the field values continue to show "unknown", contact Cisco TAC for replacement of the module. Attach logs with the above troubleshooting and output of "show inventory" for serial number information. 

Reload 3G Module

Use this command to reload just the 3G module.

ast03-cgr1120-cgosB# reload module 3
reloading module 3 ...

Bounce the Cellular Interface

Use this command to shut/ no shut the cellular interface. 

ast03-cgr1120-cgosB# conf t
Enter configuration commands, one per line.  End with CNTL/Z.
ast03-cgr1120-cgosB(config)# interface cellular 3/1
ast03-cgr1120-cgosB(config-if)# shut   <<<<<<<<<<<<<<<<<   Wait for 30 sec
ast03-cgr1120-cgosB(config-if)# no shut

Sample Output when 3G Module is UP and Working

Output of "show cellular 3/1 all" when the module is working.

ast03-cgr1120-cgosB# sh cellular 3/1 all

Hardware Information
====================

Modem Firmware Version = T1_0_3_2AP R361 CNSZXD00000061 2011/04/15 17:40:48
Modem Firmware Built = 04/15/11
Modem Hardware Version = 1.0
International Mobile Subscriber Identity (IMSI) = 867941410
International Mobile Equipment Identity (IMEI) = 357115040246303
Factory Serial Number (FSN) = CC3159202581007 04/15/11
Modem status = Online
Current Modem Temperature = 35 deg C
Current Temperature State = Normal


Profile Information
===================

--- Profile 1 details ---
Status = Activated
Default: Yes
PDP Type: IPv4
Profile IP: 192.0.2.1
Profile APN: broadband
Authentication: None
Profile UserName:
Profile Password:


Data Connection Information
===========================

Data Transmitted = 460 bytes
Data Received = 420 bytes
Profile 1, Packet Session Status = ACTIVATED
IP Address = 192.0.2.1


Network Information
===================

Modem status = Online
Service status = Normal
Service type = Combined
Current Roaming Status = Home
Country Initials = USA
Network name = AT&T
Mobile Country Code (MCC) = 310
Mobile Network Code (MNC) = 410
Location Area Code (LAC) = 28986
Routing Area Code (RAC) = 1
Cell ID = 50434
Primary Scrambling Code = 2
PLMN Selection = Automatic


Radio Information
=================

Current band = WCDMA_V_850
Radio power mode = On
Channel number = 4359
Band selected = Auto
Current RSSI = -92 dBm


Band Information
================

  Number of bands:   5
  Band 1: WCDMA: NA
  Band 2: GSM PCS 850
  Band 3: GSM ALL
  Band 4: WCDMA ALL
  Band 5: AUTO BAND


Current LED values
==================
WWAN: green
RSSI: amber
SIM Slot 0: off
SIM Slot 1: green
Service: green
GPS: off

Check if the 3G Module is Recognized but There is No Connectivity

Check Cellular Interface

Issue the show ip interface brief command and verify that the cellular interface received an IP and the status is UP. If the interface was not able to get IP, continue to trpubleshoot.

ast03-cgr1120-cgosB# sh ip int b
IP Interface Status for VRF "default"(1)
Interface            IP Address      Interface Status
Cell3/1              192.0.2.1       protocol-up/link-up/admin-up

Check Signal Strength

The received signal strength indication (RSSI) indicates the strength of signal received by the 3G module. This can be checked with the show cellular 3/1 radio command.

ast03-cgr1120-cgosB# sh cellular 3/1 radio

Current band = WCDMA_V_850
Radio power mode = On
Channel number = 4359
Band selected = Auto
Current RSSI = -92 dBm

RSSI Values:

1. RSSI should be at least greater than -90 for reliable connectivity. RSSI greater than -60 indicates strong signal strength.

2. If RSSI is less than -100, module will not be able to connect to network. This maybe due to these:

• Cellular interface is administratively down; no shut the cellular interface and check if the RSSI value changed.
• Module is stuck in low power mode; shut/no shut the cellular interface and check if the RSSI value changed. Wait for 30 sec before you issue no shut.
• Antenna missing or not properly connected.
• No 3G service coverage in the area.

3. If the RSSI is greater than -60, more troubleshooting is needed. Enable debugs to further triage the issue.

Check IP Address Connectivity

1. Verify if an outbound call is initiated and an IP address is received by the module with the show ip interface brief and show cellular 3/1 all command. Sample debug outputs to verify this:

2009 Feb  5 22:57:51.479132 dialer: Dialer1 is in READY state

2009 Feb  5 22:57:52 ast03-cgr1120-cgosB %DIALER-5-DIALER_DIAL_OUT: Dialer1 dial out cellular3/1 successfully 

2009 Feb  5 22:57:52.572770 dialer: Dialer1 dial out cellular3/1 successfully 

2009 Feb  5 22:57:55 ast03-cgr1120-cgosB %CELLPM-5-CELLPM_CON_EST: Connection Established 

2009 Feb  5 22:57:55.317420 cellpm: Get Ip from modem 192.0.2.1 

2009 Feb  5 22:57:55.995243 cellpm: interface cellular 3/1 IP address 192.0.2.1

2. If the outbound call is made but no IP address is received, collect the logs to see the disconnect reason with show log | no command and attach it to the TAC case.

3. If there is no outbound call request, execute Initialize Over the Air (IOTA) to verify if connection can be made (for CDMA module only).

cellular3/1 cdma activate otasp *22899

(i) If IOTA connection is successful, check for hardware or configuration issues.

• show interface cellular 3/1
• show controllers cellular 3/1
• show tech

(ii) If IOTA connection is not successful, verify the logs for the cause and issue this command:

• show log | no 

Sample debug output when a successful connection is made:

2009 Feb  5 22:57:37 ast03-cgr1120-cgosB %DIALER-5-DIALER_MODEM_UP: Modem active 

2009 Feb  5 22:57:37.758585 dialer: modem changes from DISABLE to ENABLE 
2009 Feb  5 22:57:37.758652 dialer: CHAT SCRIPT CHAT profile 1 
2009 Feb  5 22:57:37.758722 dialer: Dialer1 is in READY state 
2009 Feb  5 22:57:37.758764 dialer: Dial out interface service is not ready yet 
2009 Feb  5 22:57:37.762968 cellpm: Plmn Ready: 0 
2009 Feb  5 22:57:37 ast03-cgr1120-cgosB %CELLPM-5-CELLPM_SIM_STATUS: SIM Status: SIM Okay 

2009 Feb  5 22:57:37.763128 cellpm: SIM Status: SIM Okay 
2009 Feb  5 22:57:37.763231 cellpm: Rec'd Modem Radio Power Enable 
2009 Feb  5 22:57:37.763520 cellpm: Receive modem enabled notification 
2009 Feb  5 22:57:37 ast03-cgr1120-cgosB %CELLPM-5-CELLPM_MODEM_UP: Modem enabled 

2009 Feb  5 22:57:37.766192 dialer: Modem Enable on interface 29100000 
2009 Feb  5 22:57:37.766257 dialer: Rec'd cellular3/1 modem ENABLE notification  
2009 Feb  5 22:57:37.766303 dialer: current modem state is ENABLE 
2009 Feb  5 22:57:39.485643 cellpm: Network Service Status: Emergency 
2009 Feb  5 22:57:39.485697 cellpm: Network Service Type: Invalid Service 
2009 Feb  5 22:57:39.485932 cellpm: Send MTS opcode 183321 to sap 776 if_index 29100000 
2009 Feb  5 22:57:39.488392 dialer: Modem service is out for interface 29100000 
2009 Feb  5 22:57:39.488686 cellpm: Service Type: 0 
2009 Feb  5 22:57:39.488787 cellpm: Registration Indication: 
2009 Feb  5 22:57:39 ast03-cgr1120-cgosB %CELLPM-5-CELLPM_NETWORK_NAME:   

2009 Feb  5 22:57:39.488851 cellpm:  
2009 Feb  5 22:57:41.360863 cellpm: Network Service Status: Emergency 
2009 Feb  5 22:57:41.360928 cellpm: Network Service Type: Invalid Service 
2009 Feb  5 22:57:41.361704 cellpm: Send MTS opcode 183321 to sap 776 if_index 29100000 
2009 Feb  5 22:57:41.364084 cellpm: Service Type: 0 
2009 Feb  5 22:57:41.364157 dialer: Modem service is out for interface 29100000 
2009 Feb  5 22:57:41.364938 cellpm: Registration Indication: 
2009 Feb  5 22:57:41.365040 cellpm:  
2009 Feb  5 22:57:41.466717 cellpm: Network Service Status: Emergency 
2009 Feb  5 22:57:41.466770 cellpm: Network Service Type: Invalid Service 
2009 Feb  5 22:57:41.467005 cellpm: Send MTS opcode 183321 to sap 776 if_index 29100000 
2009 Feb  5 22:57:41.469358 cellpm: Service Type: 0 
2009 Feb  5 22:57:41.469444 dialer: Modem service is out for interface 29100000 
2009 Feb  5 22:57:41.469478 cellpm: Registration Indication: 
2009 Feb  5 22:57:41.469568 cellpm:  
2009 Feb  5 22:57:41.469645 cellpm: Network Service Status: Normal 
2009 Feb  5 22:57:41.469689 cellpm: Network Service Type: Combined 
2009 Feb  5 22:57:41.469733 cellpm: Modem Status Normal notify dialer 
2009 Feb  5 22:57:41.469929 cellpm: Send MTS opcode 183299 to sap 776 if_index 29100000 
2009 Feb  5 22:57:41.472329 dialer: Modem Enable on interface 29100000 
2009 Feb  5 22:57:41.472382 dialer: Rec'd cellular3/1 modem ENABLE notification  
2009 Feb  5 22:57:41.472427 dialer: current modem state is ENABLE 
2009 Feb  5 22:57:41.477311 cellpm: MTS_OPC_CELLPM_SERVICE_READY send at port 3 
2009 Feb  5 22:57:41.477369 cellpm: Service Type: 6 
2009 Feb  5 22:57:41.477405 dialer: Modem service is ready for interface 29100000 
2009 Feb  5 22:57:41.477456 dialer: Rec'd cellular3/1 AT port 3 
2009 Feb  5 22:57:41.477466 cellpm: Registration Indication: 
2009 Feb  5 22:57:41.477504 cellpm:  Network Name: AT&T 
2009 Feb  5 22:57:41.477512 dialer: timer start #0 if_index 29100000 
2009 Feb  5 22:57:41 ast03-cgr1120-cgosB last message repeated 2 times

2009 Feb  5 22:57:41 ast03-cgr1120-cgosB %CELLPM-5-CELLPM_NETWORK_NAME:  AT&T 

2009 Feb  5 22:57:41.477560 dialer: Create time for msg 0 
2009 Feb  5 22:57:41.477589 cellpm:  
2009 Feb  5 22:57:41.477601 dialer: dialer_timer_start start timer 
2009 Feb  5 22:57:48.636548 cellpm: cellpm_timer_msg_handler 0xb610052c Message type is 0 
2009 Feb  5 22:57:48.636609 cellpm: timer expired if_index 29100000 
2009 Feb  5 22:57:48.637429 cellpm: The onboard temp is 30 
2009 Feb  5 22:57:48.639603 cellpm: temperature timer start if_index 29100000 
2009 Feb  5 22:57:48.639666 cellpm: Create time for msg 0 
2009 Feb  5 22:57:48.639711 cellpm: cellpm_timer_start start timer 
2009 Feb  5 22:57:48.655443 cellpm: Modem Status Online 
2009 Feb  5 22:57:48.655495 cellpm: temp 30 low crit -26 norm -23 admin state 1 
2009 Feb  5 22:57:51.479034 dialer: timer expired if_index 29100000 
2009 Feb  5 22:57:51.479098 dialer: CHAT SCRIPT CHAT profile 1 
2009 Feb  5 22:57:51.479132 dialer: Dialer1 is in READY state 
2009 Feb  5 22:57:52.572492 dialer: timeout_val 60 dialer_retry_num 1 
2009 Feb  5 22:57:52.572572 dialer: timer start #1 if_index 29100000 
2009 Feb  5 22:57:52.572627 dialer: Create time for msg 0 
2009 Feb  5 22:57:52.572673 dialer: dialer_timer_start start timer 
2009 Feb  5 22:57:52 ast03-cgr1120-cgosB %DIALER-5-DIALER_DIAL_OUT: Dialer1 dial out cellular3/1 successfully 

2009 Feb  5 22:57:52.572770 dialer: Dialer1 dial out cellular3/1 successfully 
2009 Feb  5 22:57:52.573620 cellpm: Plmn Ready: 0 
2009 Feb  5 22:57:54.497230 cellpm: Network Service Status: Normal 
2009 Feb  5 22:57:54.497286 cellpm: Network Service Type: Combined 
2009 Feb  5 22:57:54.497329 cellpm: Modem Status Normal notify dialer 
2009 Feb  5 22:57:54.497553 cellpm: Send MTS opcode 183299 to sap 776 if_index 29100000 
2009 Feb  5 22:57:54.497771 dialer: Modem Enable on interface 29100000 
2009 Feb  5 22:57:54.497831 dialer: Rec'd cellular3/1 modem ENABLE notification  
2009 Feb  5 22:57:54.497880 dialer: current modem state is ENABLE 
2009 Feb  5 22:57:54.508855 dialer: Modem service is ready for interface 29100000 
2009 Feb  5 22:57:54.508924 dialer: Rec'd cellular3/1 AT port 3 
2009 Feb  5 22:57:54.508972 dialer: Dial out timer started already 
2009 Feb  5 22:57:54.509194 cellpm: MTS_OPC_CELLPM_SERVICE_READY send at port 3 
2009 Feb  5 22:57:54.509259 cellpm: Service Type: 6 
2009 Feb  5 22:57:55 ast03-cgr1120-cgosB %CELLPM-5-CELLPM_GSM_SESSION_ACTIVE: GSM Profile 1 is activated 

2009 Feb  5 22:57:55.295970 cellpm: GSM profile 1 is activated 
2009 Feb  5 22:57:55.296018 cellpm: Activation Status:             1 
2009 Feb  5 22:57:55.296062 cellpm: Negotiated QoS:                1 
2009 Feb  5 22:57:55.296103 cellpm: Negotiated Precedence:         2 
2009 Feb  5 22:57:55.296334 cellpm: Send MTS - PKT SESS ACTIVATED to dialer 
2009 Feb  5 22:57:55.296695 dialer: Session activated. Dialer1 stop dialing 
2009 Feb  5 22:57:55 ast03-cgr1120-cgosB %DIALER-5-DIALER_CON_EST: Connection Established 

2009 Feb  5 22:57:55.297250 cellpm: Rec'd session active notification 
2009 Feb  5 22:57:55 ast03-cgr1120-cgosB %CELLPM-5-CELLPM_CON_EST: Connection Established 

2009 Feb  5 22:57:55.317420 cellpm: Get Ip from modem 192.0.2.1 
2009 Feb  5 22:57:55 ast03-cgr1120-cgosB %URIB-4-ZERO_NEXTHOP:  urib [3445] (default-base) Client "static" added 0.0.0.0 next hop and interface Cellular3/1 is not point-to-point

2009 Feb  5 22:57:55 ast03-cgr1120-cgosB %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by root on vsh.28455

2009 Feb  5 22:57:55.995243 cellpm: interface cellular 3/1 IP address 192.0.2.1 
2009 Feb  5 22:57:55.998937 cellpm: Send MTS opcode 229378 to sap 776 if_index 29100000 
2009 Feb  5 22:57:56.006669 cellpm: Service Type: 6 
2009 Feb  5 22:57:56 ast03-cgr1120-cgosB %CELLPM-5-CELLPM_STATE_CHANGE: Moving to State: Connected 

2009 Feb  5 22:58:03.100719 cellpm: Network Service Status: Normal 
2009 Feb  5 22:58:03.100788 cellpm: Network Service Type: Combined 
2009 Feb  5 22:58:03.100835 cellpm: Modem Status Normal notify dialer 
2009 Feb  5 22:58:03.101097 cellpm: Send MTS opcode 183299 to sap 776 if_index 29100000 
2009 Feb  5 22:58:03.103525 cellpm: Service Type: 6 
2009 Feb  5 22:58:03.103560 dialer: Modem Enable on interface 29100000 
2009 Feb  5 22:58:03.103610 dialer: Rec'd cellular3/1 modem ENABLE notification  
2009 Feb  5 22:58:03.103654 dialer: current modem state is ENABLE 
2009 Feb  5 22:58:03.732185 cellpm: Service Type: 6 
2009 Feb  5 22:58:38.918415 cellpm: Get Ip from modem 192.0.2.1 
2009 Feb  5 22:58:38.969173 cellpm: Get Ip from modem 192.0.2.1 

Verify Pings

If the cellular interface received an IP and the interface status is UP, verify that pings to public IP are successful. If pings are not successful, check that default route is configured correctly. 

ast03-cgr1120-cgosB# conf t
Enter configuration commands, one per line.  End with CNTL/Z.
ast03-cgr1120-cgosB(config)#ip route 0.0.0.0 0.0.0.0 Cellular3/1

ast03-cgr1120-cgosB# show ip route
0.0.0.0/0, ubest/mbest: 1/0
    *via Cellular3/1, [1/0], 00:06:05, static


ast03-cgr1120-cgosB# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=54 time=1786.39 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=54 time=89.716 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=54 time=89.699 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=54 time=99.691 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=54 time=119.689 ms

--- 8.8.8.8 ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 89.699/437.037/1786.39 ms

Information to Collect when you Open a TAC Service Request

If you open a TAC Service Request, attach this information to your request to troubleshoot 3G modem issues:

• Details of troubleshooting performed with related logs
• show tech
• show log | no
• debug chat trace
• debug cellular trace
• debug dialer trace

Related Information

• http://www.cisco.com/c/en/us/td/docs/routers/connectedgrid/modules/3g/Cisco_Connected_Grid_3G_Module_Installation_and_Configuration_Guide.html
• Technical Support & Documentation - Cisco Systems