Table Of Contents

Troubleshooting

Introduction

Using the Troubleshooting

General Errors

SM Not Running

SM in Failure Mode

SM User Log Messages

General

HTML Adaptor

FTP Server

Domains Messages

SCE Messages

SM-SCE Synchronization Messages

Auto-logout Messages

Inactive Subscribers Removal Messages

PRPC Messages

Subscriber Messages

Cable-related Messages

High-Availability Messages

Configuration File Messages

Quota Manager Messages

Lease Query LEG Messages

SOAP LEG Messages

VLM Messages

RADIUS Listener LEG Messages

BGP LEG Messages

General Setup Errors

Cannot Run this Script-/etc/motd File Exists

install-sm.sh Script-User is not Root

Cisco SCE Pull Rate Error—Output Queue Full

install-sm.sh Script-User pcube Exists

TimesTen Database Setup Errors

Introduction

System (Kernel) Configuration File

Configuration File /var/TimesTen/sys.odbc.ini

Configuration File ~pcube/.odbc.ini

TimesTen DSN Configuration—Cannot Find Requested DSN

TimesTen DSN Configuration—Data Source Name Not Found

TimesTen Database Settings—Cannot Connect to Data Source

TimesTen Configuration Error—Not Enough Memory

TimesTen Configuration Error—Incorrect Memory Definitions

TimesTen Configuration Error—Cannot Create Semaphores

TimesTen Configuration Error—Cannot Read Data Store File

TimesTen Configuration Error—Data Store Space Exhausted

TimesTen Setup Error—Database Not Setup Correctly

Network Management Command Line Utility (p3net) Errors

First Connection—Operation Timed Out

Status Error—Connection Down

Status Error—Subscriber Management Down

Subscriber Database Command Line Utility (p3subsdb) Errors

CSV File Validation Error

Clear-all Error in Cluster Mode

Subscriber Database is Not Replicated on Standby Server

Cable Support Command Line Utility (p3cable) Errors

CSV File Import Error

Configuration Errors

Network Management Errors

Domain Errors

PRPC Errors

RADIUS Listener Errors

Common Validation Errors

Troubleshooting

---

Revised: March 07, 2013, OL-21080-06

Introduction

This module describes how to troubleshoot the Subscriber Manager (SM) installation and daily operation.

•Using the Troubleshooting

•SM User Log Messages

•General Errors

•General Setup Errors

•TimesTen Database Setup Errors

•Network Management Command Line Utility (p3net) Errors

•Subscriber Database Command Line Utility (p3subsdb) Errors

•Cable Support Command Line Utility (p3cable) Errors

•Configuration Errors

Using the Troubleshooting

Each entry in this module consists of an error message, probable cause(s), and solution. Note that the same error message may appear in more than one section of this module.

When an unexpected error occurs during the system's installation or daily operation, search for the error message throughout this module (the message may appear in more than one place). When you find the error message, read the section below the message and try the recommended solution. If the message appears more than once, try to correct the most probable cause first.

General Errors

•SM Not Running

•SM in Failure Mode

SM Not Running

Error message:	The following sequence of output appears (in response to the command p3sm --sm-status): >p3sm --sm-status Could not connect to SM.
Probable cause:	The SM server has not been started.
Solution:	Use the following command to start/restart the SM server: >p3sm --start

SM in Failure Mode

Error message:	The following sequence of output appears (in response to the command p3sm --sm-status): >p3sm --sm-status SM is running. SM operational state is Failure Command terminated successfully
Probable cause:	The SM server restarted three times in 30 minutes due to an internal error.
Solution:	This error can happen only in a cluster setup. Check the pcube user log and the Veritas Cluster Server log for the reason for the failure that caused the reboots. Act according to the problem in the logs. Additional operations that can be taken are: •Use the following command to extract a support file: >p3sm --extract-support-file --output=support.zip •Send the support file to Cisco's customer support •Use the following command to start/restart the SM server and get out of the Failure state: >p3sm --restart

SM User Log Messages

SM maintains log files that can provide an additional information about the SM activity. The files can be extracted using the p3log CLU (add link to the p3log). The SM user log is located under: {p-cube home directory/sm/}

The following is a list of the SM Log Messages that may appear in the User Log:

General

1. EM Agent Started. Version: {Version number}

2. EM Agent shutdown

3. Running {operational name} operation due to management link failure

4. Failure in setting up SM database : {exception message and class name}

5. Creating support file { the name of the support file }

6. Restarting EM Agent in {number of seconds} seconds.

7. Detected LEG {leg name} connection failure.

8. The database is full

HTML Adaptor

9. HTML adaptor server started on port {port}

10. Unable to start HTML adaptor server on port {port}

11. HTML adaptor server stopped

12. HTML adaptor server port changed from{old port} to {new port }

FTP Server

13. FTP server started on port {port number}. Passive ports are: {first passive port - last passive port}.

14. Unable to start FTP server on port {error message exception}

15. FTP server stopped.

16. FTP server port changed from{old port} to {new port}

17. FTP server passive ports changed from{old first port - old last port} to {new first port- new last port}

Domains Messages

18. Domain {domain name} added

19. Errors occurred while adding network elements{elements logical names} to domain {domain name}

20. Domain {domain name} removed.

21. All of the domains were removed.

22. All of the elements were removed from domain {domain name}

23. Elements {logical names of elements (one or more)} were removed from domain {domain name}

24. Domain {domain name} was moved to {target domain}

25. Domain {old name} was renamed to {new name}.

26. Domains were reset back to defaults.

27. All SM domains reset to defaults

28. Domain {domain name} was cleared from subscribers.

SCE Messages

29. Network element {logical name-super user-description} at host {host} added.

30. Connection to {logical name} is ready.

31. Connection to {logical name} is lost, trying to establish connection...

32. Connection to {logical name} lost.

33. Network element {logical name} reset back to defaults.

34. All network elements were reset back to defaults.

35. All network elements were removed.

36. Network element {logical name} was removed.

37. All network elements were removed

38. Network element {logical name} was removed.

39. SCE {logical name} has been put in quarantine for the next {quarantine time} seconds - {quarantine reason}

40. Quarantine of SCE { logical name} ended

41. Quarantine of SCE { logical name} was stopped by the user

42. Post-quarantine timeout has expired for SCE { logical name}

SM-SCE Synchronization Messages

43. Started SM synchronization of {SE logical name} in {domain name} domain

44. Started SM pull-mode synchronization of {SE logical name} in {domain name} domain

45. Finished SM synchronization of {SE logical} in {domain name} domain ({logins} subscribers

46. Finished SM pull-mode synchronization of {SE logical} in {domain name} domain.

47. Aborted SM synchronization of {SE logical name} in {domain name} domain

48. {subscriber/VPN name} - operation failure {description} on {SE logical name} domain{domain}

49. Started SM VPN synchronization of {logical name} in {domain name} domain.

50. Finished SM VPN synchronization of {logical name} in {domain name} domain ({successful logins} VPNs, {failed logins} failures).

51. Aborted SM VPN synchronization of {logical name} in {domain name} domain.

Auto-logout Messages

52. Starting task of logging out expired subscribers

53. Finished task of logging out expired subscribers. Task ran for {number of seconds task ran} seconds. Logged out { number of subscriber logged out} IP mappings

54. Finished task of logging out expired subscribers. Task ran for {number of seconds task ran} seconds. Sent { number of subscriber sent to external pull } IP mappings to external MBean

Inactive Subscribers Removal Messages

55. Removed subscriber {subscriber ID} due to inactivity

PRPC Messages

56. PRPC connection to {Peer inet address} established, client name is {client name}, session name is {session encoding}

57. PRPC connection to {peer inet address} closed ok: {message}

58. PRPC connection to {peer inet address} closed BAD: {message}

59. PRPC server started on port {port}

60. PRPC server port changed from {old port} to {new port}

61. PRPC server stopped.

Subscriber Messages

1. Removing all subscribers from SM

2. VLAN {removed VLAN} was removed from {logged out subscriber} due to conflict with VLAN {removed} of {logged in subscriber} on domain {domain}.

3. IP mapping {removed IP range} was removed from {logged out subscriber} due to conflict with {gained IP range} of {logged in subscriber} on domain {domain}.

4. {subscriber name} - Operation failure {description} on {SE logical name} (domain: {domain}).

5. Clearing subscribers mapping in domain {domain name}, due to LEG {leg name} connection failure.

6. Failed {import\export operation name}: {error string}

7. Could not find configuration file { import\export file name}

8. Reloading configuration

9. Getting all subscriber names to file {file name}

10. Importing subscribers from file {file name}

11. Exporting subscribers to file {file name}

12. Adding subscriber {Subscriber name}

13. Removing subscriber {Subscriber name}

14. Showing subscriber by name {Subscriber name}

15. Showing subscriber by ip-range {Subscriber name}

16. Showing subscriber by VLAN {Subscriber name}

17. Setting domain of subscribers{Subscriber name} to {domain name}

18. Setting mapping of {subscriber name} to {mapping}

19. Getting mapping of {subscriber name}

20. Clearing mapping of {subscriber name}

21. Setting property {property name + value} of {subscriber name} to {property name + value}

22. Getting property {property name} of {subscriber name}

23. Setting properties of {Subscriber name}

24. Getting properties of {Subscriber name}

25. Clearing all subscribers.

26. Getting the number of subscribers.

27. Login: {login info}

28. Introduce: {subscriber info}

29. Unintroduce: {subscriber name} from domain {domain name}

30. Logout: {Subscriber info}

31. Logout by IP: {Subscriber IP}

32. Logout by VLAN: {Subscriber VLAN}

33. Remove: {Subscriber name}

34. Put: {Subscriber info}

35. Rename: {Subscriber name} to {Subscriber name}

36. Add IP: added IP {IP address} to {Subscriber name}

37. Add VLAN: added VLAN {VLAN ID} to {Subscriber name}

38. Extended the lease of IP {IP address} to {date}}

39. Pull response for {Subscriber name}: pulled IP: {IP mapping}, returned IP {IP mapping}

Cable-related Messages

40. Importing cable modems from file {file name}

41. Exporting cable modems to file {file name}

42. Cleared all cable modems

43. Reload configuration called.

High-Availability Messages

44. The agent's Operational state has changed to failure.

45. The Agent's Operational state has changed to fatal. {Operation performed : restart or shutdown}

46. Agent {agent name}: {operation fail message}. details: {error details}

47. Agent {agent name} timeout expire during operation {operation}.Agent {agent name} failed while performing {operation}. details: {error details}

48. SM operational state switched from {mode} to {mode}

Configuration File Messages

49. Loaded p3sm.cfg configuration file

50. Warning : {Warning name}

51. Unknown property {property name} found in section [{section name}] in configuration file {configuration file name}.

52. Unknown section [{section name}] found in configuration file {configuration file name}.

53. Error value {property value} for property {property name} in section [{section name}]. Optional values: [{allowed values}]

54. Property key {property name} does not match expected type in section [{section name}].

55. Missing mandatory property {property name} in section [{section name}].

56. Error value {property value} for property {property name} in [{section name}] section. Valid format: [0..255].[0..255].[0..255].[0..255]

57. Error empty value for { property name} property in [{section name}] section - must have at least one character

58. Configuration file loading failed due to { num of errors} errors {num of warnings } warnings

59. Configuration file loaded successfully - {warning message}

Quota Manager Messages

60. Got quota notification {quota notification name } for subscriber {subscriber ID}

61. Subscriber {Subscriber ID} has more quota in SCE than expected (bucket ID= {bucket ID}, last quota= {last quota}, current quota= {current quota}). Possible causes: Another server provisions quota to this subscriber, QM restart

62. Quota profile ID { quota profile ID } does not exist in configuration - Ignoring quota notification { quota notification name } for subscriber { subscriber ID }

63. Subscriber {subscriber ID} was moved from SCE {old SCE IP} to SCE {new SCE IP}

64. Got quota notification { quota notification name } at {provision time} for subscriber {subscriber ID} that is too adjacent to last provision time ({last provision time })

65. Subscriber { subscriber ID } changed quota profile from { old quota profile ID } to { new quota profile ID} - Quota is replenished (with or without reset history)

66. Subscriber {subscriber ID} changed quota profile from {old quota profile ID} to {new quota profile ID}. Number of configured buckets is not equal between profiles. Therefore - replenish quota

67. Subscriber {subscriber ID} changed quota profile from {old quota profile ID} to {new quota profile ID}. Slice period are not equal between profiles. Therefore - replenish quota

68. Quota profile { quota profile ID } configuration was changed - Quota is replenished for subscriber { subscriber ID }

69. New aggregation period started for subscriber { subscriber ID } - Quota is replenished

70. Calling quota update for subscriber { subscriber ID }: { quota operation }

71. Quota Update operation for subscriber {subscriber ID} failed, error message = { error message}

72. Get Quota Status operation for subscriber { subscriber ID } succeeded

73. Get Quota Status operation for subscriber { subscriber ID } failed, error message = {error message}

74. Subscriber {subscriber ID} has negative quota in SCE (bucket ID={ bucket ID }, last quota={last quota}, current quota={current quota}).

75. Quota bucket { subscriber ID } has depleted for subscriber {bucket ID} till {end of aggregation period}

76. Quota bucket { bucket ID } has depleted for subscriber { subscriber ID }

77. Cannot update quota bucket {bucket ID} for subscriber {subscriber ID} due to large update value {update value} - setting the dosage value instead

78. Identified breach of bucket {bucket ID} for subscriber {subscriber ID} - moving the subscriber to {penalty profile} profile for {penalty period} minutes period.

79. Subscriber {subscriber ID} usage was lower than {post penalty threshold} - going to move the subscriber to quota profile {profile ID}

80. Identify breach within bucket {bucket ID} for subscriber {subscriber ID} while moving to penalty profile {penalty profile}. Remaining quota is {remaining quota}.

Lease Query LEG Messages

81. Lease-Query LEG: Session {session Id}: sending query of IP {queried IP} to the DHCP server ({DHCP server IP})

82. Lease-Query LEG: Session {session Id}: queried IP ({queried IP}) and CIADDR ({CIADDR field}) do not match, reply is discarded

83. Lease-Query LEG: Session {session Id}: the lease is not active for IP {queried IP}

84. Lease-Query LEG: Session {session Id}: Lease reply received for IP {queried IP}, but there is no session in the repository - maybe the session has expired

85. Lease-Query LEG: Session {session Id}: no modem MAC address (option 82) was provided for IP {queried IP}, reply is discarded

86. Lease-Query LEG: Session {session Id}: successful query, subscriber info: {subscriber Info}

87. Lease-Query LEG: Session {session Id}: subscriber properties customization failed. DHCP info: {DHCP packet info}

88. Lease-Query LEG: Session {session Id}: query of IP {queried IP} to the DHCP server ({DHCP server IP}) has timed-out, releasing session

89. Lease-Query LEG: Server {DHCP server IP} failure detected, but there is no redundant server to switch to

90. Lease-Query LEG: Server {failed DHCP server IP} failure detected, failing-over to {1}

91. Lease-Query LEG: LEG initialization failure: {message}

92. Failed to assign default package to {subscriber ID} (package ID {package ID}).

93. Assigned a default package to {subscriber ID} (package ID {package ID}).

94. No package ID was found for {subscriber ID} under the package-name of { package ID }

95. Failed to assign a package to {subscriber ID} (package ID {package ID}).

96. Assigned a package to {subscriber ID} (package ID {package ID}).

97. Successfully logged CPE: subscriber name {subscriber ID}, mapping {IP}, lease time {lease-time}

98. Successfully logged CM: subscriber name { subscriber ID }

99. Successfully logged out CPE: mapping {logged out IP address }

100. Failed to login CPE: subscriber name { subscriber ID }, mapping {IP} ({exception name })

101. Failed to login CM: subscriber name {subscriber ID} ({exception name})

102. Failed to logout CPE: mapping { logged out IP address } ({exception name })

103. Dropped RDR due to missing option 51. Assigned IP { Assigned IP }

104. Dropped RDR due to missing option 82. Assigned IP { Assigned IP }

105. Dropped RDR due to missing package information for subscriber { subscriber ID } (no default package is configured)

SOAP LEG Messages

106. SOAP-LEG query operation for : { subscriber ID }

107. SOAP-LEG query failed for : { subscriber ID }

108. SOAP-LEG query success for : { subscriber ID }

109. SOAP-LEG convert fail for : { subscriber ID }

110. SOAP-LEG login operation for : { subscriber ID }

111. SOAP-LEG login operation failed for { subscriber ID } : {error description}

VLM Messages

For VLM messages please refer the Cisco Service Control for Managing Remote Cable MSO Links Solution Guide - Userlog Messages section.

RADIUS Listener LEG Messages

112. Login of subscriber { subscriber name } failed, { error message }

113. Logout of subscriber { subscriber name } failed, { error message }

114. Login of subscriber {subscriber name} failed, cannot translate attribute value { attribute value } to property { property name }. Check mapping table configuration.

115. Missing data to login {subscriber name} - ignoring. NAS {NAS}

116. Accounting-Stop received from NAS {NAS} with no subscriber-Id attribute and no Network-Id

117. Accounting-{ Start/Interim } received from NAS {NAS} with no subscriber-Id attribute and no Network-Id

118. Invalid Framed-IP-Address attribute received - {error description}

119. Invalid attribute { attribute } received: { attribute data}

120. Invalid attribute {attribute} received: {attribute data} - {error description}

121. Invalid Framed-IP-Address attribute received: {attribute data}

122. Invalid IP Address, IP address {attribute} is reserved by the application.

123. Invalid VPN attribute {attribute} received: {attribute data}

BGP LEG Messages

124. Received an Add-Route message from BGP LEG for a known VPN { VPN: PE@RD } with IP range {bgpRange} and MPLS tag {label}

125. Received a Remove-Route message from BGP LEG for an existing route with IP range { bgpRange } and MPLS tag {label} for VPN { VPN: PE@RD }

126. Database format is not compatible with MPLS/VPN mappings, see release notes to fix that

General Setup Errors

•Cannot Run this Script-/etc/motd File Exists

•install-sm.sh Script-User is not Root

•install-sm.sh Script-User pcube Exists

•Cisco SCE Pull Rate Error—Output Queue Full

Cannot Run this Script-/etc/motd File Exists

Error message:	The following sequence of output appears when running one of the installation/upgrade scripts: >./<script-name> <script-name>: Cannot run this script - /etc/motd file exists <script-name>: Please remove the file or move it to another location <script-name>: and run the script again This error applies to the following scripts: •install-sm.sh •upgrade-sm.sh •cluster-upgrade.sh •install-vcs-agents.sh
Probable cause:	The /etc/motd file exists or has a size greater than 0.
Solution:	Remove the file or move it to another location.

install-sm.sh Script-User is not Root

Error message:	The following sequence of output appears (in response to the command ./install-sm.sh): >./install-sm.sh install-sm.sh: Starting SM installation sequence install-sm.sh: Error - this script must be run by root - exiting.
Probable cause:	You started the installation sequence as user and not as super user.
Solution:	Run the install-sm.sh script as super user.

Cisco SCE Pull Rate Error—Output Queue Full

Error message:	The following output appears in the SM log when the pull rate between Cisco SCE and SM is more than 360 per second per SCE: Output queue is full - ignoring pull request
Probable cause:	If the Cisco Subscriber Manager receives pull requests from a Cisco SCE at a higher rate, typically more than 360 requests per second, the pull queue becomes full and the it cannot add incoming pull-requests to the queue.
Solution:	Although this will not impact introducing of subscribers as the Cisco SCE will send pull requests again for the ignored pull requests, we recommend that you set a maximum of 360 pull requests per second. If external LEGs are enabled please check the supported rate of individual LEGs.

install-sm.sh Script-User pcube Exists

Error message:	The following sequence of output appears (in response to the command ./install-sm.sh): # ./install-sm.sh install-sm.sh: Starting SM installation sequence install-sm.sh: Error - pcube user exists and has home /export/home/pcube, not /opt/pcube - remove it or use -o - exiting. #
Probable cause:	Your machine already has the user pcube.
Solution:	Run the installation using the -o option (overwrite), as follows: # ./install-sm.sh -o

TimesTen Database Setup Errors

•Introduction

•TimesTen DSN Configuration—Cannot Find Requested DSN

•TimesTen DSN Configuration—Data Source Name Not Found

•TimesTen Database Settings—Cannot Connect to Data Source

•TimesTen Configuration Error—Not Enough Memory

•TimesTen Configuration Error—Incorrect Memory Definitions

•TimesTen Configuration Error—Cannot Create Semaphores

•TimesTen Configuration Error—Cannot Read Data Store File

•TimesTen Configuration Error—Data Store Space Exhausted

•TimesTen Setup Error—Database Not Setup Correctly

Introduction

The TimesTen configuration consists of several configuration files. This section explains the purpose and scope of each of these files. When troubleshooting the TimesTen, you will be requested to edit these configuration files and reboot the machine or restart the SM. In most cases, the defaults applied by the SM installation procedure are satisfactory.

---

Caution Changing the TimesTen configuration files should be done with extreme care, and it is best to consult Cisco technical support prior to making any changes. See Obtaining Documentation and Submitting a Service Request for more information.

---

•System (Kernel) Configuration File

•Configuration File /var/TimesTen/sys.odbc.ini

•Configuration File ~pcube/.odbc.ini

System (Kernel) Configuration File

The kernel configuration file is a system configuration file, which affects system-wide configuration parameters:

•For Solaris, it is file /etc/system.

•For RedHat, it is file /etc/sysctl.conf.

The SM installation procedure configures this file to add extra semaphores and shared memory to the system. After editing this file, you have to reboot the machine for the changes to take effect.

If you are running other applications that require changes in this file's semaphore and shared memory values, take care that the TimesTen configuration does not override the other application's configuration, or vice versa. You should consult with the Cisco technical support for the proper values to use for the file configuration parameters.

Configuration File /var/TimesTen/sys.odbc.ini

The file /var/TimesTen/sys.odbc.ini is a TimesTen configuration file that configures system DSNs. Any user on the machine on which the system DSN is defined can use this file. The SM DSNs are system DSNs that are named PCube_SM_Repository and PCube_SM_Local_Repository, and which have the following system DSN configuration parameters:

•LogFileSize—The size of the TimesTen log file, in megabytes.

•PermSize—The size of the permanent memory region for the data store, in megabytes. You may increase PermSize but not decrease it.

The data stored in the permanent memory region includes tables and indexes that make up a TimesTen data store. The permanent data partition is written to the disk periodically.

•TempSize—The size of the memory allocated to the temporary region, in megabytes.

Temporary data includes locks, cursors, compiled commands, and other structures needed for command execution and query evaluation. The temporary data partition is created when a data store is loaded into memory and is destroyed when the data store is unloaded.

---

Note For additional information, see Memory Configuration Parameters Versus Number of Subscribers: without Quota Manager or Memory Configuration Parameters Versus Number of Subscribers: with Quota Manager.

---

•SMPOptLevel—Optimizes the database operation on multi-processor machines. If the machine is a multi-processor platform, set parameter SMPOptLevel to 1 (default is 0).

Configuration File ~pcube/.odbc.ini

The file ~pcube/.odbc.ini is a TimesTen configuration file that configures user DSNs.

TimesTen DSN Configuration—Cannot Find Requested DSN

Error message:	The following sequence of output appears (in response to the command p3sm --sm-status): >p3sm --sm-status SM is running. SM operational state is XXX Error - Times-Ten DB is not setup correctly: [TimesTen][TimesTen 5.0.35 CLIENT]Cannot find the requested DSN (PCube_SM_Repository_CS) in ODBCINI
Probable cause:	The TimesTen Client DSN is not configured correctly in file ~pcube/.odbc.ini.
Solution:	Ensure that file ~pcube/.odbc.ini contains the following: [ODBC Data Sources] PCube_SM_Repository_CS=TimesTen 5.0 Client Driver

TimesTen DSN Configuration—Data Source Name Not Found

Error message:	The following sequence of output appears (in response to the command p3sm --sm-status): >p3sm --sm-status SM is running. SM operational state is XXX Error - Times-Ten DB is not setup correctly: [TimesTen][TimesTen 5.0.35 ODBC Driver]Data source name not found and no default driver specified
Probable cause:	The TimesTen Client DSN is not configured correctly in file ~pcube/.odbc.ini.
Solution:	Ensure that file ~pcube/.odbc.ini contains the following: [PCube_SM_Repository_CS] TTC_SERVER_DSN=PCube_SM_Repository Ensure that file /var/TimesTen/sys.odbc.ini contains the following: [ODBC Data Sources] PCube_SM_Repository=TimesTen 5.0 Driver [PCube_SM_Repository] Driver=__TTDIR__/TimesTen/pcubesm22/lib/libtten.so DataStore=__VARDIR__/pcube_SM_Repository

TimesTen Database Settings—Cannot Connect to Data Source

Error message:	The following sequence of output appears (in response to the command p3sm --sm-status): >p3sm --sm-status SM is running. SM operational state is XXX Error - Times-Ten DB is not set up correctly: [TimesTen][TimesTen 5.0.35 CLIENT]Unable to connect to data source (DSN: pcube_SM_Repository_CS; Network Address: X.X.X.X; Port Number: XXX): This operation has Timed Out. Try increasing your ODBC timeout attribute or check to make sure the target TimesTen Server is running
Probable cause:	The following causes are possible: •The address of the Server DS is incorrect. •The port of the Server DS is incorrect. •TimesTen is not active.
Solution:	The Service Control solutions for the above causes are: •(The address of the Server DS is incorrect.) Ensure that file ~pcube/.odbc.ini contains the following: TTC_SERVER=127.0.0.1 •(The port of the Server DS is incorrect.) On a default installation, ensure that file ~pcube/.odbc.ini does not contain "TTC_SERVER_PORT". On a non-default installation, ensure that file ~pcube/.odbc.ini does contain "TTC_SERVER_PORT=Non-default-port". •(TimesTen is not active.) Run the following command: ~pcube/lib/tt/TimesTen/pcubesm22/bin/ttStatus If TimesTen is not working, re-install TimesTen. If the above solutions do not work, please refer to the TimesTen manual.

TimesTen Configuration Error—Not Enough Memory

Error message:	The following sequence of output appears (in response to the command p3sm --sm-status): >p3sm --sm-status SM is running. SM operational state is XXX Error - Times-Ten DB is not setup correctly: [TimesTen][TimesTen 5.0.35 ODBC Driver][TimesTen]TT0836: Cannot create data store shared-memory segment, error 1455 -- file "db.c", lineno 6289, procedure "sbDbConnect()"
Probable cause:	There is not enough memory for creating TimesTen's in-memory database.
Solution:	Do all of the following: •Ensure that the Unix machine has at least 1024 MB of memory installed. •Ensure that the configured memory size parameters (PermSize and TempSize) specified in file /var/TimesTen/sys.odbc.ini are less than the total amount of memory installed in the machine. •For Solaris, ensure that the maximum shared memory (parameter shmsys:shminfo_shmmax) specified in file /etc/system is less than the total amount of memory installed in the machine. •For Red Hat, ensure that the maximum shared memory (parameter kernel.shmmax) specified in file /etc/sysctl.conf is less than the total amount of memory installed in the Linux Machine.

TimesTen Configuration Error—Incorrect Memory Definitions

Error message:	The following sequence of output appears (in response to the command p3sm --sm-status): >p3sm --sm-status SM is running. SM operational state is XXX. Error - Times-Ten DB is not setup correctly: [TimesTen][TimesTen 5.0.35 ODBC Driver]Overflow in converting data store or log file size from megabytes to bytes, or in converting log buffer size from kilobytes to bytes
Probable cause:	The memory definitions of DSN are incorrect.
Solution:	Ensure that the configured permanent memory size and log file size (parameters PermSize and LogSize) are less than the total amount of memory and of disk space specified in file /var/TimesTen/sys.odbc.ini.

TimesTen Configuration Error—Cannot Create Semaphores

Error message:	The following sequence of output appears (in response to the command p3sm --sm-status): >p3sm --sm-status SM is running. SM operational state is XXX Error - Times-Ten DB is not setup correctly: [TimesTen][TimesTen 5.0.35 ODBC Driver][TimesTen]TT0925: Cannot create data store semaphores (Invalid argument) - file "db.c", lineno 5124, procedure "sbDbCreate()", sqlState: 08001, errorCode: 925
Probable cause:	TimesTen was unable to create the data store semaphores that are defined in the kernel configuration file (Solaris: /etc/system for Solaris; for Red Hat: /etc/sysctl.conf for Red Hat).
Solution:	Do all of the following: •Ensure that the machine has at least 1024 MB of memory installed. •Reboot the machine after the first time that TimesTen is installed. •Verify the contents of the system (kernel) configuration file: –For Solaris, ensure that file /etc/system contains the following: semsys:seminfo_semmni = 20 semsys:seminfo_semmsl = 100 semsys:seminfo_semmns = 2000 semsys:seminfo_semmnu = 2000 –For Red Hat, ensure that file /etc/sysctl.conf contains the following: *---- Begin settings for TimesTen kernel.sem = "SEMMSL_250 SEMMNS_32000 SEMOPM_100 SEMMNI_100 *---- End of settings for TimesTen

TimesTen Configuration Error—Cannot Read Data Store File

Error message:	The following sequence of output appears (in response to the command p3sm --sm-status): >p3sm --sm-status SM is running. SM operational state is XXX. Error - Times-Ten DB is not setup correctly: [TimesTen][TimesTen 5.0.35 ODBC Driver][TimesTen]TT0845: Cannot read data store file. OS-detected error: Error 0 -- file "db.c", lineno 6320, procedure "sbDbConnect()"
Probable cause:	TimesTen was unable to read the data store file, probably due to an error during the installation. This error occurs when installing a TimesTen application on top of an existing TimesTen, without having first uninstalled the old TimesTen database.
Solution:	Do the following: 1. Remove the database, by using the SM p3db CLU with the following commands: >p3db -destroy-rep-db >p3db -destroy-local-db 2. Uninstall TimesTen with the following commands: >su Password: # ~pcube/lib/tt/TimesTen4.5/32/bin/setup.sh -uninstall 3. Re-install TimesTen, either by running the SM install-tt.sh script or by using the installation files supplied by TimesTen.

TimesTen Configuration Error—Data Store Space Exhausted

Error message:	The following sequence of output appears in the SM log (while using the SM APIs): java.io.IOException: Failure in putting subscriber 45977166__00:50:bf:97:c1:b2 : [TimesTen][TimesTen 5.0.35 ODBC Driver][TimesTen]TT0802: Data store space exhausted -- file "blk.c", lineno 1571, procedure "sbBlkAlloc"
Probable cause:	The TimesTen database has already reached its maximum capacity, which caused the operation of adding a new subscriber to the database to fail.
Solution:	Usually, doing just one of the following is sufficient: •Reduce the number of the subscribers handled by the SM (of course, this solution is not always possible). •Configure the system to support a larger number of subscribers. Note that this solution may require editing one or more of the TimesTen configuration files discussed in Introduction as well as rebooting the machine. •Move the SM to a more powerful machine; this could be a faster CPU (or more CPUs), a larger disk, more RAM, etc. For help and guidance in implementing the last two solutions, please contact Cisco Technical Support. See Obtaining Documentation and Submitting a Service Request.

TimesTen Setup Error—Database Not Setup Correctly

Error message:	The following error message appears when the Subscriber Manager database is not started properly: Unable to connect to daemon; check daemon status
Probable cause:	This error message appears when the Subscriber Manager database is not started properly.
Solution:	This is a generic error message from the Subscriber Manager database. Please check the umlog file at ~pcube/sm/server/root/ for the origin of the error.

Network Management Command Line Utility (p3net) Errors

•First Connection—Operation Timed Out

•Status Error—Connection Down

•Status Error—Subscriber Management Down

First Connection—Operation Timed Out

Error message:	The following sequence of output appears (in response to the command p3net --connect): >p3net --connect --ne-name=YYYY Error - failed to connect to element 'YYYY' Operation timed out: connect
Probable cause:	The following causes are possible: •The IP address is incorrect. •The element YYYY is down.
Solution:	The Service Control solutions for the above causes are: •Ensure that the IP address is correct. •Ensure that the element YYYY is online and is connected via its management port.

Status Error—Connection Down

Error message:	The following sequence of output appears (in response to the command p3net --show-ne): >p3net --show --ne-name=se0 Network Element Information: ============================ Name: YYY Description: testing element Host: X.X.X.X Ip: X.X.X.X Port: 14374 Status: Connection down (Failure in connecting to agent on host, Connection refused: connect) Type: SCE1000 Domain: smartNET.policy.unitTestSubscribers Subscriber Management: Not Active
Probable cause:	Following causes are possible: •The IP address is incorrect. •The element YYYY is down.
Solution:	Service Control solutions for the above causes are: •Ensure that the IP address is correct. •Ensure that the element YYYY is online and is connected via its management port. •Ensure that the PRPC adapter is online on the port that the status indicates.

Status Error—Subscriber Management Down

Error message:	In response to the p3net--show-ne command, the following sequence of output appears: >p3net --show --ne-name=se0 Network Element Information: ============================ Name: YYY Description: testing element Host: X.X.X.X Ip: X.X.X.X Port: 14374 Status: Connection ready Type: SCE1000 Domain: smartNET.policy.unitTestSubscribers Subscriber Management: Not Active
Probable cause:	Subscriber Management field indicates whether the SM successfully performed SM-SCE subscriber synchronization. If the value of the field is Not Active, it is possible that the SM failed to synchronize the SCE.
Solution:	One possible solution is to force SM-SCE resynchronization by using the CLU command p3sm --resync.

Subscriber Database Command Line Utility (p3subsdb) Errors

CSV File Validation Error

Error message:	In response to the command p3subsdb --import, the following sequence of output appears: >p3subsdb --import --file=/export/home/pcube/XXX.csv Error - Failed to validate the file XXX.csv See import-results.txt for detailed errors description. >cat import-results.txt x.csv:1: expected 2 items but got 4 items. 1 subscribers, 1 errors. NO APPLICATION INSTALLED, MAKE SURE TO INSTALL PQI BEFORE IMPORTING CM WITH TUNEABLES.
Probable cause:	You tried to import a four-field csv file to the SM, but no application (SCA BB) was installed. For example, the following csv file for a SCA BB application contains four fields: # CSV line format: subscriber-id, domain, mappings, package-id JerryS,subscribers,80.179.152.159,0 ElainB,subscribers,194.90.12.2,3 However, the default definition file that defines csv file parsing rules contains only two fields: name and ip mapping.
Solution:	Do one of the following: •Install an application (SCA BB) on the SM (for details, see Cisco Service Control Application for Broadband User Guide). •Import a csv file that has just two fields.

Clear-all Error in Cluster Mode

Error message:	In response to the p3db -rep-status command, the following output appears on the standby machine: -bash-3.00$ p3db --rep-status (of standby machine) Peer name            Host name                Port    State   Proto ----------------     ------------------------ ------  ------- ----- PCUBE_SM_REPOSITORY  SM_REP1                  15111   Start   20 Last Msg Sent Last Msg Recv Latency TPS     RecordsPS Logs ------------- ------------- ------- ------- --------- ---- 15:16:10      00:00:20      -1.00   -1      -1        2 Warning 8025: Replication Log Threshold limit reached at master. The Replication state for this subscriber data store is now set to FAILED. This store is not in sync with the master the subscriber DB is in failed state.
Probable cause:	This problem indicates that the replication NICs bandwidth is not set high enough for the required DB changes. The DB replication log file, which is responsible for sending the replication messages to the standby machine, is overloaded. This occurs when there are a large number of DB changes (usually caused by the p3subsdb --clear-all command when there are a large number of subscribers).
Solution	To resolve this problem, the operator needs to increase the replication log file size from 20MB to a higher value. To increase the replication log file size on the standby machine: 1. Open the /etc/TimesTen/odbc.ini file. 2. In the [Pcube_SM_Repository] section, increase the value of LogFileSize attribute from 20MB to a higher value. 3. Save updated the /etc/TimesTen/odbc.ini file. 4. Restart the TimesTen process: ~pcube/lib/tt/TimesTen/pcubesm22/bin/ttDaemonAdmin -restart 5. Perform a SM machine switchover. 6. Execute Steps 1 through 4 for the second SM machine.

Subscriber Database is Not Replicated on Standby Server

Error message:	In response to the p3db --rep-status command, the following output appears on the active and standby SM servers: __ACTIVE SM__ [pcube@smwv01 ~]$ p3db --rep-status Peer name Host name Port State Proto ---------------- ------------------------ ------ ------- ----- PCUBE_SM_REPOSITORY SM_REP2 15111 Stop 24 Last Msg Sent Last Msg Recv Latency TPS RecordsPS Logs ------------- ------------- ------- ------- --------- ---- 00:00:00 00:00:00 273.49 617 1040 0 the subscriber DB is ok Command terminated successfully STANDBY SM___ -bash-3.2$ p3db --rep-status Peer name Host name Port State Proto ---------------- ------------------------ ------ ------- ----- PCUBE_SM_REPOSITORY SM_REP1 15111 Start 24 Last Msg Sent Last Msg Recv Latency TPS RecordsPS Logs ------------- ------------- ------- ------- --------- ---- 00:00:02 00:00:00 -1.00 -1 -1 1 the subscriber DB is ok Command terminated successfully The following timers are set to 0 every few seconds: •Last Msg Sent •Last Msg Recv
Probable cause:	In the SM cluster, the standby server has been replaced, reinstalled, or the database of the server has been deleted.
Solution	This error indicates that the subscriber database is not replicated on the standby server. To resolve this problem, follow these steps: 1. Stop Veritas Cluster on both servers as root user. # hastop -local 2. Stop replication on the standby server as pcube user. ($ p3db --rep-stop) 3. Pause and continue on the active server. $ p3db --rep-pause p3db --rep-continue) 4. Drop replication scheme on the standby server. $ p3db --drop-rep-scheme 5. Set replication scheme on the standby server. $ p3db --set-rep-scheme 6. Start replication scheme on the standby server. $ p3db --rep-start 7. Start Veritas Cluster on both systems as root user. # hastart 8. Verify if VCS is running. # hastatus -sum 9. Verify if replication is running. $ p3db --rep-status

Cable Support Command Line Utility (p3cable) Errors

CSV File Import Error

Error message:	In response to the p3cable --import-cm command, the following sequence appears: >p3cable --import-cm --file=/export/home/pcube/XXX.csv Importing cable modems ... 0% Importing cable modems ... 100% Error - Errors during import from 'H:\work\Mng\dev\install\ems\bin\win32\x.csv': Imported 1 CM(s). 1 Error(s). See cm-import-results.txt for detailed errors description. >cat cm-import-results.txt x.csv:1: expected 2 items but got 4 items. 1 cable modem(s); 1 error(s). NO APPLICATION INSTALLED, MAKE SURE TO INSTALL PQI BEFORE IMPORTING CM WITH TUNEABLES.
Probable cause:	You tried to import a four-field csv file to the SM, but no application (SCA BB) was installed. For example, the following csv file for a SCA BB application contains four fields: # CSV line format: subscriber-id, domain, mappings, package-id JerryS,subscribers,80.179.152.159,0 ElainB,subscribers,194.90.12.2,3 However, the default definition file that defines csv file parsing rules contains only two fields: name and ip mapping.
Solution:	Do one of the following: •Install an application (SCA BB) on the SM (for details, see Cisco Service Control Application for Broadband User Guide). •Import a csv file that has just two fields.

Configuration Errors

•Network Management Errors

•Domain Errors

•PRPC Errors

•RADIUS Listener Errors

•Common Validation Errors

Network Management Errors

Error message (1):	Error section <section name>: cannot contain white spaces.
Probable cause:	[SCE.XXX] section cannot contain white spaces (SCE name cannot contain white space).
Solution:	Remove the white spaces.

Error message (2):	Error in section <section name>: host <ip address> already exists in section <section name>
Probable cause:	Configuration cannot contain two SCEs with the same IP address.
Solution:	Change the IP address of one of the SCEs.

Error message (3):	Unknown NE <name> found in domain <domain name> section: it does not have [SCE.<name>] section
Probable cause:	The section <domain name> includes, under the elements property, an SCE that is not defined in an [SCE.XXX] section.
Solution:	Add the missing [SCE.XXX] section to the file.

Error message (4):	Duplicate NE <name> found in domain <domain name> section: it already appears in <domain name > domain section.
Probable cause:	Same SCE cannot belong to more than one domain.
Solution:	Remove the SCE from all but one of the domains.

Domain Errors

Error message (1):	Error in section <section name>: cannot contain white spaces.
Probable cause:	[Domain.XXX] section cannot contain white spaces (Domain name cannot contain white space).
Solution:	Remove the white spaces.

Error message (2):	Error <alias name> value - alias name should not start with `CM_'.
Probable cause:	The alias name cannot start with CM_ because this is the prefix of hidden domain generated by the SM when working with CMs (see CPE as Subscriber in Cable Environment).
Solution:	Use a different prefix for the alias.

Error message (3):	Alias <alias name> already exists in [<domain name>] section
Probable cause:	Each alias can appear in only one [Domain.XXX] section.
Solution:	Alias mentioned in the error message should be removed from all but one [Domain.XXX] section.

Error message (4):	Unknown domain <domain name> found in [LEG-Domains Association]. It does not appear as a section.
Probable cause:	Domain mentioned in given section does not have a [Domain.XXX] section.
Solution:	Domain mentioned in error message should be given a [Domain.XXX] section.

Error message (5):	Invalid non-integer value: <value> for property '<property name>' in section [section name].
Probable cause:	Properties in [Domain.XXX] sections do not have integer values.
Solution:	Properties mentioned in error message should be given integer values.

Error message (6):	Error in section <domain name>: Property - <name> not found: <list of application properties>
Probable cause:	Property defined in [Domain.XXX] section is not found in properties list defined by installed application.
Solution:	Delete the properties mentioned in the error message (or define them in the installed application).

Error message (7):	New configuration was not applied due to the following warnings: Warning - Cannot remove domain <domain name> with <num of SCEs>SCEs. Note that all subscribers will be removed from domain db. Please use '--ignore-warnings' option to complete the operation.
Probable cause:	This is actually a warning: its purpose is to warn that the user removed from the p3sm.cfg file a domain that contained SCEs (which probably contained subscribers, etc.), and that the user will lose all subscriber data relevant for that domain. This warning appears only after the CLU command p3sm --load-config is activated.
Solution:	To avoid this warning, use the --ignore-warnings option.

PRPC Errors

Error message:	New configuration was not applied due to the following warnings: Warning - PRPC configuration was changed. Note: Reloading may take up to 5 seconds. Please use '--ignore-warnings' option to complete the operation.
Probable cause:	This is actually a warning: it is displayed after the CLU command p3sm -load-config is activated when the PRPC configuration in the p3sm.cfg file has been changed.
Solution:	Use the --ignore-warnings option to complete the operation.

RADIUS Listener Errors

Error message:	Duplicate NAS identifier <nasID> found in section [NAS name]: already exists in <other NAS name>
Probable cause:	<nasID> is not unique.
Solution:	Change one of the <nasID> so that both are unique.

Common Validation Errors

The following configuration errors are relevant for all sections/parameters of the p3sm.cfg file.

Error message (1):	Unknown property <property name> found in section [<section name>] in configuration file <file name>.
Probable cause:	Property written in the p3sm.cfg file is unknown to the SM. Maybe the name is misspelled or the property belongs in a different section.
Solution:	Ensure that the name is spelled correctly and that the property resides in the correct section.

Error message (2):	Unknown section [<section name>] found in configuration file {2}.
Probable cause:	The section written in the p3sm.cfg file is unknown to the SM. Maybe the name is misspelled.
Solution:	Ensure that the section name is spelled correctly.

Error message (3):	Error value <value> for property <property name> in section [<section name>]. Optional values: [<values range>]
Probable cause:	Value of the property is invalid. The <values range> field contains the valid values.
Solution:	Specify any valid value for the property.

Error message (4):	Missing mandatory property <property name> in section [<section name>].
Probable cause:	The property <property name> is mandatory and must appear in the section <section name>.
Solution:	Set a value for the requested property in the specified section.

Error message (5):	Error value <property value> for property <property name> in [<section name>] section. Valid format: [0..255].[0..255].[0..255].[0..255]
Probable cause:	The value is an invalid IP address.
Solution:	Specify a valid IP address.

Error message (6):	Error empty value for <property name> property in [<section name>] section - must have at least one character
Probable cause:	Value of the <property name> is empty; for example, prop=
Solution:	Specify a non-empty value for the property.

Error message (7):	Section <section name> added when already exists
Probable cause:	The section with <section name> appears more than once. This error is most likely to occur for the [SCE.XXX] and [Domain.XXX] sections.
Solution:	Use the specified section name only once.