gNMI - gRPC Network Management Interface

This chapter contains the following topics:

About gNMI

gNMI uses gRPC (Google Remote Procedure Call) as its transport protocol.

Cisco NX-OS supports gNMI for dial-in subscription to telemetry applications running on the Cisco Nexus 3000 Series switches. Although past release supported telemetry events over gRPC, the switch pushed the telemetry data to the telemetry receivers. This method was called dial out.

With gNMI, applications can pull information from the switch. They subscribe to specific telemetry services by learning the supported telemetry capabilities and subscribing to only the telemetry services that it needs.

Table 1. Supported gNMI RPCs

gNMI RPC

Supported

Capabilities

Yes

Get

Yes

Set

Yes

Subscribe

Yes

gNMI RPC and SUBSCRIBE

The NX-OS 9.3(1) release supports gNMI version 0.5.0. Cisco NX-OS Release 9.3(1) supports the following parts of gNMI version 0.5.0.

Table 2. SUBSCRIBE Options

Type

Sub Type

Supported?

Description

Once

Yes

Switch sends current values only once for all specified paths

Poll

Yes

Whenever the switch receives a Poll message, the switch sends the current values for all specified paths.

Stream

Sample

Yes

Once per stream sample interval, the switch sends the current values for all specified paths. The supported sample interval range is from 1 through 604800 seconds.

The default sample interval is 10 seconds.

On_Change Yes The switch sends current values as its initial state, but then updates the values only when changes, such as create, modify, or delete occur to any of the specified paths.
Target_Defined No

Optional SUBSCRIBE Flags

For the SUBSCRIBE option, some optional flags are available that modify the response to the options listed in the table. In release 9.3(1), the updates_only optional flag is supported, which is applicable to ON_CHANGE subscriptions. If this flag is set, the switch suppresses the initial snapshot data (current state) that is normally sent with the first response.

The following flags are not supported:

  • aliases

  • allow_aggregation

  • extensions

  • heart-beat interval

  • prefix

  • qos

  • suppress_redundant

Guidelines and Limitations for gNMI

Following are the guidelines and limitations for gNMI:

  • Beginning with Cisco NX-OS Release 9.3(5), Get and Set are supported.

  • gNMI queries do not support wildcards in paths.

  • If you configure a prefix-list in CLI mode and issue an Edit, Get, or Get-Config, for the OpenConfig model, it is not supported.

    If you configure a prefix-list in CLI Mode and issue a NETCONF/gNMI Notification or Subscription from the OpenConfig model, it is not supported.

  • When you attempt to subscribe an OpenConfig routing policy with a preexisting CLI configuration like the following, it returns empty values due to the current implementation of the OpenConfig model. 

    ip prefix-list bgp_v4_drop seq 5 deny 125.2.0.0/16 le 32
ipv6 prefix-list bgp_v6_drop seq 5 deny cafe:125:2::/48 le 128

using the xpath

openconfig-routing-policy:/routing-policy/defined-sets/prefix-sets/prefix-set[name=bgp_v4_drop]/config
openconfig-routing-policy:/routing-policy/defined-sets/prefix-sets/prefix-set[name=bgp_v6_drop]/config

  • When you enable gRPC on both the management VRF and default VRF and later disable on the default VRF, the gNMI notifications on the management VRF stop working.

    As a workaround, disable gRPC completely by entering the no feature grpc command and reprovision it by entering the feature grpc command and any existing gRPC configuration commands. For example, grpc certificate or grpc port . You must also resubscribe to any existing notifications on the management VRF.

  • When you attempt to subscribe an OpenConfig routing policy with a preexisting CLI configuration like the following, it returns empty values due to the current implementation of the OpenConfig model. 

    ip prefix-list bgp_v4_drop seq 5 deny 125.2.0.0/16 le 32
ipv6 prefix-list bgp_v6_drop seq 5 deny cafe:125:2::/48 le 128

using the xpath

openconfig-routing-policy:/routing-policy/defined-sets/prefix-sets/prefix-set[name=bgp_v4_drop]/config
openconfig-routing-policy:/routing-policy/defined-sets/prefix-sets/prefix-set[name=bgp_v6_drop]/config

  • Beginning with Cisco NX-OS Release 9.3(3), if you have configured a custom gRPC certificate, upon entering the reload ascii command the configuration is lost. It reverts to the default day-1 certificate. After entering the reload ascii command, the switch will reload. Once the switch is up again, you must reconfigure the gRPC custom certificate.


    Note

    This applies when entering the grpc port and grpc certificate commands.

  • Use of origin, use_models, or both, is optional for gNMI subscriptions.

  • Beginning with Cisco NX-OS Release 9.3(3), Subscribe supports the OpenConfig model.

  • The gNMI feature supports Subscribe and Capability as options of the gNMI service.

  • The feature supports JSON and gnmi.proto encoding. The feature does not support protobuf.any encoding.

  • Each gNMI message has a maximum size of 12 MB. If the amount of collected data exceeds the 12 MB maximum, the collected data is dropped.

    You can avoid this situation by creating more focused subscriptions that handle smaller, more granular data-collection sets. So, instead of subscribing to one higher-level path, create multiple subscriptions for different, lower-level parts of the path.

  • Across all subscriptions, there is support of up to 150K aggregate MOs. Subscribing to more MOs can lead to collection data drops.

  • All paths within the same subscription request must have the same sample interval. If the same path requires different sample intervals, create multiple subscriptions.

  • The gRPC process that supports gNMI uses the HIGH_PRIO cgroup, which limits the CPU usage to 75% of CPU and memory to 1.5 GB.

  • The show grpc gnmi command has the following considerations:

    • The gRPC agent retains gNMI calls for a maximum of 1 hour after the call has ended.

    • If the total number of calls exceeds 2000, the gRPC agent purges ended calls based an internal cleanup routine.

The gRPC server runs in the management VRF. As a result, the gRPC process communicates only in this VRF forcing the management interface to support all gRPC calls.

gRPC functionality now includes the default VRF for a total of two gRPC servers on each Cisco Nexus 3000 Series switch. You can run one gRPC server in each VRF, or run only one gRPC server in the management VRF. Supporting a gRPC in the default VRF adds flexibility to offload processing gRPC calls from the management VRF, where significant traffic load might not be desirable.

If two gRPC servers are configured, be aware of the following:

  • VRF boundaries are strictly enforced, so each gRPC server processes requests independent of the other. Requests do not cross between VRFs.

  • The two servers are not HA or fault tolerant. One gRPC server does not back up the other, and there is no switchover or switchback between them.

  • Any limits for the gRPC server are per VRF.

Configuring gNMI

Configure the gNMI feature through the grpc gnmi commands.

Procedure

  Command or Action Purpose
Step 1

configure terminal

Example:

switch-1# configure terminal
switch-1(config)#

Enters global configuration mode.
Step 2

feature grpc

Example:

switch-1# feature grpc
switch-1(config)#

Enables the gRPC agent, which supports the gNMI interface for dial-in.

Step 3

grpc gnmi max-concurrent-call number

Example:

switch-1(config)# grpc gnmi max-concurrent-call 16
switch-1(config)#

Sets the limit of simultaneous dial-in calls to the gNMI server on the switch. Configure a limit from 1 through 16. The default limit is 8.

The maximum value that you configure is for each VRF. If you set a limit of 16 and gNMI is configured for both management and default VRFs, each VRF supports 16 simultaneous gNMI calls.

This command does not affect and ongoing or in-progress gNMI calls. Instead, gRPC enforces the limit on new calls, so any in-progress calls are unaffected and allowed to complete.

Note 
The configured limit does not affect the gRPCConfigOper service.

Configuring Server Certificate

gNMI communicates over gRPC and uses TLS to secure the channel between the switch and the client. The default hard-coded gRPC certificate is no longer shipped with the switch. The default behavior is a self-signed key and certificate which is generated on the switch as shown below with an expiration date of one day. 

switch# show grpc gnmi service statistics

=============
gRPC Endpoint
=============

Vrf : management
Server address : [::]:50051

Cert notBefore : Fri Jun 21 10:54:42 PDT 2019
Cert notAfter : Sat Jun 22 10:54:42 PDT 2019

Max concurrent calls : 8
Listen calls : 1
Active calls : 0

Number of created calls : 1
Number of bad calls : 0

Subscription stream/once/poll : 0/0/0

The keys are generated in the location below

switch# pwd
/opt/mtx/etc
switch# ls -al
total 20
drwxr-xr-x 4 root floppy 140 Jun 21 10:54 .
drwxr-xr-x 7 root floppy 140 Jun 20 02:43 ..
-rw------- 1 root root 1704 Jun 21 10:54 grpc.key
-rw------- 1 root root 1107 Jun 21 10:54 grpc.pem
drwxr-xr-x 2 root floppy 60 Jun 21 10:54 init.d
-rw-r--r-- 1 root floppy 10712 Jun 20 02:43 nxos_grpc.proto
drwxr-xr-x 2 root floppy 60 Jun 21 10:54 proto-1.0
switch#

With an expiration of one day, you can use this temporary certificate for quick testing. For long term a new key/certificate must be generated.


Note

After the certificate expires, there are two ways to have the key/certificate to regenerate:

  • Reload the switch.

  • Manually delete the key/certificate in the /opt/mtx/etc folder and enter the no feature grpc and feature grpc commands.

Generating and Configuring Key/Certificate Examples for Cisco NX-OS Release 9.3(2) and Earlier

The following is an example for generating key/certificate:

For more information on generating identify certificates, see the Installing Identity Certificates section of the Cisco Nexus 3000 Series NX-OS Security Configuration Guide, Release 9.3(x).

Procedure

Step 1

Generate the selfsigned key and pem files.

  1. switch# openssl req -x509 -newkey rsa:2048 -keyout self_sign2048.key -out self_sign2048.pem -days 365 -nodes
Step 2

After generating the key and pem files, modify the mtx.conf.user files in the Bash shell to have the gRPC service pick up the certificates. 

[grpc]
key = /bootflash/self-sign2048.key
cert = /bootflash/self-sign2048.pem
Step 3

Reload the box to have the gRPC service pick up the certificate.

Step 4

Verify gRPC is now using the certificate. 

switch# show grpc gnmi service statistics

=============
gRPC Endpoint
=============

Vrf : management
Server address : [::]:50051

Cert notBefore : Nov 5 16:48:58 2015 GMT
Cert notAfter : Nov 5 16:48:58 2035 GMT

Max concurrent calls : 16
Listen calls : 1
Active calls : 0

Number of created calls : 953
Number of bad calls : 0

Subscription stream/once/poll : 476/238/238

Max gNMI::Get concurrent : 5
Max grpc message size : 8388608
gNMI Synchronous calls : 10
gNMI Synchronous errors : 0
gNMI Adapter errors : 0
gNMI Dtx errors : 0

Generating and Configuring Key/Certificate Examples for Cisco NX-OS Release 9.3(3) and Later

The following is an example for generating key/certificate.


Note

We recommend using this method for configuring certificates.

For more information on generating identify certificates, see the Installing Identity Certificates section of the Cisco Nexus 3000 Series NX-OS Security Configuration Guide, Release 9.3(x).

Procedure

Step 1

Generate the selfsigned key and pem files.

  1. switch# openssl req -x509 -newkey rsa:2048 -keyout self_sign2048.key -out self_sign2048.pem -days 365 -nodes
Step 2

After generating the key and pem files, you must bundle the key and pem files for use in the trustpoint CA Association.

switch# run bash sudo su
bash-4.3# cd /bootflash/
bash-4.3# openssl pkcs12 -export -out self_sign2048.pfx -inkey self_sign2048.key -in self_sign2048.pem -certfile self_sign2048.pem -password pass:Ciscolab123!
bash-4.3# exit
Step 3

Set up the trustpoint CA Association by inputting the pkcs12 bundle into the trustpoint. 

switch(config)# crypto ca trustpoint mytrustpoint
switch(config-trustpoint)# crypto ca import mytrustpoint pkcs12 self_sign2048.pfx Ciscolab123!  <- input the pkcs12 bundle into the trustpoint.
Step 4

Verify the setup. 

switch(config)# show crypto ca certificates
Trustpoint: mytrustpoint
certificate:
subject= /C=US/O=Cisco Systems, Inc./OU=CSG/L=San Jose/ST=CA/street=3700 Cisco Way/postalCode=95134/CN=ems.cisco.com/serialNumber=FGE18420K0R
issuer= /C=US/O=Cisco Systems, Inc./OU=CSG/L=San Jose/ST=CA/street=3700 Cisco Way/postalCode=95134/CN=ems.cisco.com/serialNumber=FGE18420K0R
serial=0413
notBefore=Nov  5 16:48:58 2015 GMT
notAfter=Nov  5 16:48:58 2035 GMT
SHA1 Fingerprint=2E:99:2C:CE:2F:C3:B4:EC:C7:E2:52:3A:19:A2:10:D0:54:CA:79:3E
purposes: sslserver sslclient
 
CA certificate 0:
subject= /C=US/O=Cisco Systems, Inc./OU=CSG/L=San Jose/ST=CA/street=3700 Cisco Way/postalCode=95134/CN=ems.cisco.com/serialNumber=FGE18420K0R
issuer= /C=US/O=Cisco Systems, Inc./OU=CSG/L=San Jose/ST=CA/street=3700 Cisco Way/postalCode=95134/CN=ems.cisco.com/serialNumber=FGE18420K0R
serial=0413
notBefore=Nov  5 16:48:58 2015 GMT
notAfter=Nov  5 16:48:58 2035 GMT
SHA1 Fingerprint=2E:99:2C:CE:2F:C3:B4:EC:C7:E2:52:3A:19:A2:10:D0:54:CA:79:3E
purposes: sslserver sslclient
Step 5

Configure gRPC to use the trustpoint.

switch(config)# grpc certificate mytrustpoint
switch(config)# show run grpc
 
!Command: show running-config grpc
!Running configuration last done at: Thu Jul  2 12:24:02 2020
!Time: Thu Jul  2 12:24:05 2020
 
version 9.3(5) Bios:version 05.38
feature grpc
 
grpc gnmi max-concurrent-calls 16
grpc use-vrf default
grpc certificate mytrustpoint
Step 6

Verify gRPC is now using the certificate.

switch# show grpc gnmi service statistics

=============
gRPC Endpoint
=============

Vrf : management
Server address : [::]:50051

Cert notBefore : Nov 5 16:48:58 2015 GMT
Cert notAfter : Nov 5 16:48:58 2035 GMT

Max concurrent calls : 16
Listen calls : 1
Active calls : 0

Number of created calls : 953
Number of bad calls : 0

Subscription stream/once/poll : 476/238/238

Max gNMI::Get concurrent : 5
Max grpc message size : 8388608
gNMI Synchronous calls : 10
gNMI Synchronous errors : 0
gNMI Adapter errors : 0
gNMI Dtx errors : 0

Verifying gNMI

To verify the gNMI configuration, enter the following command:

Command

Description

show grpc gnmi service statistics

Displays a summary of the agent running status, respectively for the management VRF, or the default VRF (if configured). It also displays:

  • Basic overall counters

  • Certificate expiration time

    Note 

    If the certificate is expired, the agent cannot accept requests.

show grpc gnmi rpc summary

Displays the following:

  • Number of capability RPCs received.

  • Capability RPC errors.

  • Number of Get RPCs received.

  • Get RPC errors.

  • Number of Set RPCs received.

  • Set RPC errors.

  • More error types and counts.

show grpc gnmi transactions

The show grpc gnmi transactions command is the most dense and contains considerable information. It is a history buffer of the most recent 50 gNMI transactions that are received by the switch. As new RPCs come in, the oldest history entry is removed from the end. The following explains what is displayed:

  • RPC – This shows the type of RPC that was received (Get, Set, Capabilities)

  • DataType – For a Get only. Has values ALL, CONFIG, and STATE.

  • Session – shows the unique session-id that is assigned to this transaction. It can be used to correlate data that is found in other log files.

  • Time In -- shows timestamp of when the RPC was received by the gNMI handler.

  • Duration – time delta in ms from receiving the request to giving response.

  • Status – the status code of the operation returned to the client (0 = Success, !0 == error)

This section is data that is kept per path within a single gNMI transaction. For example, a single Get or Set

  • subtype – for a Set RPC, shows the specific operation that is requested per path (Delete, Update, Replace). For Get, there is no subtype.

  • dtx – shows that this path was processed in DTX “fast” path or not. A dash ‘-‘ means no, an asterisk ‘*’ means yes.

  • st – Status for this path. The meaning is as follows:

    • OK: path is valid and processed by infra successfully.

    • ERR: path is either invalid or generated error by infra

    • --: path not processed yet, might or might not be valid and has not been sent to infra yet.

  • path – the path

show grpc gnmi service statistics Example 

=============
gRPC Endpoint
=============

Vrf : management
Server address : [::]:50051

Cert notBefore : Mar 13 19:05:24 2020 GMT
Cert notAfter : Nov 20 19:05:24 2033 GMT

Max concurrent calls : 8
Listen calls : 1
Active calls : 0

Number of created calls : 1
Number of bad calls : 0

Subscription stream/once/poll : 0/0/0

Max gNMI::Get concurrent : 5
Max grpc message size : 8388608
gNMI Synchronous calls : 74
gNMI Synchronous errors : 0
gNMI Adapter errors : 0
gNMI Dtx errors : 0

show grpc gnmi rpc summary Example 

=============
gRPC Endpoint
=============
 
Vrf            : management
Server address : [::]:50051
 
Cert notBefore : Mar 31 20:55:02 2020 GMT
Cert notAfter  : Apr  1 20:55:02 2020 GMT
 
Capability rpcs    : 1   
Capability errors  : 0     
Get rpcs           : 53   
Get errors         : 19   
Set rpcs           : 23   
Set errors         : 8    
Resource Exhausted : 0     
Option Unsupported : 6
Invalid Argument   : 18
Operation Aborted  : 1
Internal Error     : 2
Unknown Error      : 0
 
RPC Type        State      Last Activity  Cnt Req    Cnt Resp   Client
--------------- ---------- -------------- ---------- ---------- -----------------
-----------------------
Subscribe       Listen     04/01 07:39:21          0          0

show grpc gnmi transactions Example 

=============
gRPC Endpoint
=============
 
Vrf            : management
Server address : [::]:50051
 
Cert notBefore : Mar 31 20:55:02 2020 GMT
Cert notAfter  : Apr  1 20:55:02 2020 GMT
 
RPC          DataType   Session         Time In              Duration(ms) Status
------------ ---------- --------------- -------------------- ------------ ------
Set          -          2361443608      04/01 07:43:49       173          0    
subtype: dtx:  st: path:                                                       
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo789]          
 
Set          -          2293989720      04/01 07:43:45       183          0
subtype: dtx:  st: path:                                                   
Replace  -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo6]         
 
Set          -          2297110560      04/01 07:43:41       184          0
subtype: dtx:  st: path:                                                   
Update   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo7]        
 
Set          -          0               04/01 07:43:39       0            10
 
Set          -          3445444384      04/01 07:43:33       3259         0
subtype: dtx:  st: path:                                                   
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo789]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo790]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo791]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo792]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo793]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo794]       
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo795]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo796]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo797]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo798]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo799]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo800]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo801]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo802]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo803]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo804]       
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo805]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo806]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo807]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo808]      
 
Set          -          2297474560      04/01 07:43:26       186          0
subtype: dtx:  st: path:                                                   
Update   -     OK  /System/ipv4-items/inst-items/dom-items/Dom-list[name=foo]/rt-
items/Route-list[prefix=0.0.0.0/0]/nh-items/Nexthop-list[nhAddr=192.168.1.1/32][n
hVrf=foo][nhIf=unspecified]/tag                                                  
 
 
Set          -          2294408864      04/01 07:43:17       176          13
subtype: dtx:  st: path:                                                    
Delete   -     ERR /System/intf-items/lb-items/LbRtdIf-list/descr           
 
 
Set          -          0               04/01 07:43:11       0            3
subtype: dtx:  st: path:                                                   
Update   -     --  /System/intf-items/lb-items/LbRtdIf-list[id=lo4]/descr  
Update   -     ERR /system/processes                                       
 
Set          -          2464255200      04/01 07:43:05       708          0
subtype: dtx:  st: path:                                                   
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo2]        
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo777]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo778]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo779]       
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo780]      
Replace  -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo3]/descr  
Replace  -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo4]/descr  
Replace  -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo5]/descr  
Update   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo3]/descr  
Update   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo4]/descr  
Update   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo5]/descr  
 
 
Set          -          3491213208      04/01 07:42:58       14           0
subtype: dtx:  st: path:                                                   
Replace  -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo3]/descr  
 
Set          -          3551604840      04/01 07:42:54       35           0
subtype: dtx:  st: path:                                                   
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo1]        
 
Set          -          2362201592      04/01 07:42:52       13           13
subtype: dtx:  st: path:                                                    
Delete   -     ERR /System/intf-items/lb-items/LbRtdIf-list[id=lo3]/lbrtdif-items
/operSt                                                                         
 
 
Set          -          0               04/01 07:42:47       0            3
subtype: dtx:  st: path:                                                   
Delete   -     ERR /System/*                                               
 
Set          -          2464158360      04/01 07:42:46       172          3
subtype: dtx:  st: path:                                                   
Delete   -     ERR /system/processes/shabang                               
 
Set          -          2295440864      04/01 07:42:46       139          3
subtype: dtx:  st: path:                                                   
Delete   -     ERR /System/invalid/path                                     
 
Set          -          3495739048      04/01 07:42:44       10           0
 
 
Get          ALL        3444580832      04/01 07:42:40       3            0
subtype: dtx:  st: path:                                                   
-        -     OK  /System/bgp-items/inst-items/disPolBatch                
 
Get          ALL        0               04/01 07:42:36       0            3
subtype: dtx:  st: path:                                                    
-        -     --  /system/processes/process[pid=1]                        
 
Get          ALL        3495870472      04/01 07:42:36       2            0
subtype: dtx:  st: path:                                                    
-        *     OK  /system/processes/process[pid=1]                        
 
Get          ALL        2304485008      04/01 07:42:36       33           0
subtype: dtx:  st: path:                                                   
-        *     OK  /system/processes                                       
 
Get          ALL        2464159088      04/01 07:42:36       251          0
subtype: dtx:  st: path:                                                   
-        -     OK  /system                                                  
 
Get          ALL        2293232352      04/01 07:42:35       258          0
subtype: dtx:  st: path:                                                   
-        -     OK  /system                                                  
 
Get          ALL        0               04/01 07:42:33       0            12
subtype: dtx:  st: path:                                                    
-        -     --  /intf-items

Sample DME Subscription - PROTO Encoding

gnmi-console --host >iip> --port 50051 -u <user> -p <pass> --tls --
operation=Subscribe --rpc /root/gnmi-console/testing_bl/once/61_subscribe_bgp_dme_gpb.json

[Subscribe]-------------------------------
### Reading from file ' /root/gnmi-console/testing_bl/once/61_subscribe_bgp_dme_gpb.json '
Wed Jun 26 11:49:17 2019
### Generating request : 1 -----------
### Comment : ONCE request
### Delay : 2 sec(s) ...
### Delay : 2 sec(s) DONE
subscribe {
subscription {
path {
origin: "DME"
elem {
name: "sys"
}
elem {
name: "bgp"
}
}
mode: SAMPLE
}
mode: ONCE
use_models {
name: "DME"
organization: "Cisco Systems, Inc."
version: "1.0.0"
}
encoding: PROTO
}
Wed Jun 26 11:49:19 2019
Received response 1 --------------------------
update {
timestamp: 1561574967761
prefix {
elem {
name: "sys"
}
elem {
name: "bgp"
}
}
update {
path {
elem {
}
elem {
name: "version_str"
}
}
val {
string_val: "1.0.0"
}
}
update {
path {
elem {
}
elem {
name: "node_id_str"
}
}
val {
string_val: "n9k-tm2"
}
}
update {
path {
elem {
}
elem {
name: "encoding_path"
}
}
val {
string_val: "sys/bgp"
}
}
update {
path {
elem {
}
elem {
/Received -------------------------------------
Wed Jun 26 11:49:19 2019
Received response 2 --------------------------
sync_response: true
/Received -------------------------------------
(_gnmi) [root@tm-ucs-1 gnmi-console]#

Capabilities

About Capabilities

The Capabilities RPC returns the list of capabilities of the gNMI service. The response message to the RPC request includes the gNMI service version, the versioned data models, and data encodings supported by the server.

Guidelines and Limitations for Capabilities

Following are the guidelines and limitations for Capabilities:

  • Beginning with Cisco NX-OS Release 9.3(3), Capabilities supports the OpenConfig model.

  • The gNMI feature supports Subscribe and Capability as options of the gNMI service.

  • The feature supports JSON and gnmi.proto encoding. The feature does not support protobuf.any encoding.

  • Each gNMI message has a maximum size of 12 MB. If the amount of collected data exceeds the 12-MB maximum, the collected data is dropped.

    You can avoid this situation by creating more focused subscriptions that handle smaller, more granular data-collection sets. So, instead of subscribing to one higher-level path, create multiple subscriptions for different, lower-level parts of the path.

  • All paths within the same subscription request must have the same sample interval. If the same path requires different sample intervals, create multiple subscriptions.

  • The feature does not support a path prefix in the Subscription request, but the Subscription can contain an empty prefix field.

  • The feature supports Cisco DME and Device YANG data models. Openconfig YANG is not supported.

  • The gRPC process that supports gNMI uses the HIGH_PRIO cgroup, which limits the CPU usage to 75% of CPU and memory to 1.5 GB.

  • The show grpc gnmi command has the following considerations:

    • The commands are not XMLized in this release.

    • The gRPC agent retains gNMI calls for a maximum of 1 hour after the call has ended.

    • If the total number of calls exceeds 2000, the gRPC agent purges ended calls based an internal cleanup routine.

The gRPC server runs in the management VRF. As a result, the gRPC process communicates only in this VRF forcing the management interface to support all gRPC calls.

gRPC functionality now includes the default VRF for a total of 2 gRPC servers on each Cisco Nexus 3000 Series switch. You can run one gRPC server in each VRF, or run only one gRPC server in the management VRF. Supporting a gRPC in the default VRF adds flexibility to offload processing gRPC calls from the management VRF, where significant traffic load might not be desirable.

If two gRPC servers are configured, be aware of the following:

  • VRF boundaries are strictly enforced, so each gRPC server processes requests independent of the other, and requests do not cross between VRFs.

  • The two servers are not HA or fault tolerant. One gRPC server does not back up the other, and there is no switchover or switchback between them.

  • Any limits for the gRPC server are per VRF.

Example Client Output for Capabilities

In this example, all the OpenConfig model RPMs have been installed on the switch.

The following is an example of client output for Capabilities. 

hostname user$ ./gnmi_cli -a 172.19.193.166:50051 -ca_crt ./grpc.pem -insecure -capabilities
supported_models: <
  name: "Cisco-NX-OS-device"
  organization: "Cisco Systems, Inc."
  version: "2019-11-13"
> 
supported_models: <
  name: "openconfig-acl"
  organization: "OpenConfig working group"
  version: "1.0.0"
> 
supported_models: <
  name: "openconfig-bgp-policy"
  organization: "OpenConfig working group"
  version: "4.0.1"
> 
supported_models: <
  name: "openconfig-interfaces"
  organization: "OpenConfig working group"
  version: "2.0.0"
> 
supported_models: <
  name: "openconfig-if-aggregate"
  organization: "OpenConfig working group"
  version: "2.0.0"
> 
supported_models: <
  name: "openconfig-if-ethernet"
  organization: "OpenConfig working group"
  version: "2.0.0"
> 
supported_models: <
  name: "openconfig-if-ip"
  organization: "OpenConfig working group"
  version: "2.3.0"
> 
supported_models: <
  name: "openconfig-if-ip-ext"
  organization: "OpenConfig working group"
  version: "2.3.0"
> 
supported_models: <
  name: "openconfig-lacp"
  organization: "OpenConfig working group"
  version: "1.0.2"
> 
supported_models: <
  name: "openconfig-lldp"
  organization: "OpenConfig working group"
  version: "0.2.1"
> 
supported_models: <
  name: "openconfig-network-instance"
  organization: "OpenConfig working group"
  version: "0.11.1"
> 
supported_models: <
  name: "openconfig-network-instance-policy"
  organization: "OpenConfig working group"
  version: "0.1.1"
> 
supported_models: <
  name: "openconfig-ospf-policy"
  organization: "OpenConfig working group"
  version: "0.1.1"
> 
supported_models: <
  name: "openconfig-platform"
  organization: "OpenConfig working group"
  version: "0.12.2"
> 
supported_models: <
  name: "openconfig-platform-cpu"
  organization: "OpenConfig working group"
  version: "0.1.1"
> 
supported_models: <
  name: "openconfig-platform-fan"
  organization: "OpenConfig working group"
  version: "0.1.1"
> 
supported_models: <
  name: "openconfig-platform-linecard"
  organization: "OpenConfig working group"
  version: "0.1.1"
> 
supported_models: <
  name: "openconfig-platform-port"
  organization: "OpenConfig working group"
  version: "0.3.2"
> 
supported_models: <
  name: "openconfig-platform-psu"
  organization: "OpenConfig working group"
  version: "0.2.1"
> 
supported_models: <
  name: "openconfig-platform-transceiver"
  organization: "OpenConfig working group"
  version: "0.7.0"
> 
supported_models: <
  name: "openconfig-relay-agent"
  organization: "OpenConfig working group"
  version: "0.1.0"
> 
supported_models: <
  name: "openconfig-routing-policy"
  organization: "OpenConfig working group"
  version: "2.0.1"
> 
supported_models: <
  name: "openconfig-spanning-tree"
  organization: "OpenConfig working group"
  version: "0.2.0"
> 
supported_models: <
  name: "openconfig-system"
  organization: "OpenConfig working group"
  version: "0.3.0"
> 
supported_models: <
  name: "openconfig-telemetry"
  organization: "OpenConfig working group"
  version: "0.5.1"
> 
supported_models: <
  name: "openconfig-vlan"
  organization: "OpenConfig working group"
  version: "3.0.2"
> 
supported_models: <
  name: "DME"
  organization: "Cisco Systems, Inc."
> 
supported_models: <
  name: "Cisco-NX-OS-Syslog-oper"
  organization: "Cisco Systems, Inc."
  version: "2019-08-15"
> 
supported_encodings: JSON
supported_encodings: PROTO
gNMI_version: "0.5.0"
 
hostname user$

Get

About Get

The purpose of the Get RPC is to allow a client to retrieve a snapshot of the data tree from the device. Multiple paths may be requested in a single request. A simplified form of XPATH according to the gNMI Path Conventions, Schema path encoding conventions for gNMI are used for the path.

For detailed information on the Get operation, refer to the Retrieving Snapshots of State Information section in the gNMI specification: gRPC Network Management Interface (gNMI)

Guidelines and Limitations for Get

The following are guidelines and limitations for Get and Set:

  • GetRequest.encoding supports only JSON.

  • For GetRequest.type, only DataType CONFIG and STATE have direct correlation and expression in YANG. OPERATIONAL is not supported.

  • A single request cannot have both OpenConfig (OC) YANG and device YANG paths. A request must have only OC YANG paths or device YANG paths, but not both.

  • GetRequest for root path (“/”: everything from all models) is not allowed.

  • GetRequest for the top level of the device model (“/System”) is not allowed.

  • gNMI Get returns all default values (ref. report-all mode in RFC 6243 [4]).

  • Subscribe supports the model Cisco-NX-OS-syslog-oper.

  • Get does not support the model Cisco-NX-OS-syslog-oper.

  • Query from the path /system does not return data from the path /system/processes. The specific path /system/processes should be used to query openconfig-procmon data.

  • The following optional items are not supported:

    • Path prefix

    • Path alias

    • Wildcards in path

  • A single GetRequest can have up to 10 paths.

  • If the size of value field to be returned in GetResponse is over 12 MB, the system returns error status grpc::RESOURCE_EXHAUSTED.

  • The maximum gRPC receive buffer size is set to 8 MB.

  • The number of total concurrent sessions for Get is limited to five.

  • Performing a Get operation when a large configuration is applied to the switch might cause the gRPC process to consume all available memory. If a memory exhaustion condition is hit, the following syslog is generated: 

    MTX-API: The memory usage is reaching the max memory resource limit (3072) MB

    If this condition is hit several times consecutively, the following syslog is generated:

    The process has become unstable and the feature should be restarted.

    We recommend that you restart the gRPC feature at this point to continue normal processing of gNMI transactions.

Set

About Set

The Set RPC is used by a client to change the configuration of the device. The operations, which may be applied to the device data, are (in order) delete, replace, and update. All operations in a single Set request are treated as a transaction, meaning that all operations are successful or the device is rolled-back to the original state. The Set operations are applied in the order that is specified in the SetRequest. If a path is mentioned multiple times, the changes are applied even if they overwrite each other. The final state of the data is achieved with the final operation in the transaction. It is assumed that all paths specified in the SetRequest::delete, replace, update fields are CONFIG data paths and writable by the client.

For detailed information on the Set operation, refer to the Modifying State section of the gNMI Specification https://github.com/openconfig/reference/blob/1cf43d2146f9ba70abb7f04f6b0f6eaa504cef05/rpc/gnmi/gnmi-specification.md.

Guidelines and Limitations for Set

The following are guidelines and limitations for Set:

  • SetRequest.encoding supports only JSON.

  • A single request cannot have both OpenConfig (OC) YANG and device YANG paths. A request must have only OC YANG paths or device YANG paths, but not both.

  • Subscribe supports the model Cisco-NX-OS-syslog-oper.

  • Query from the path /system does not return data from the path /system/processes. The specific path /system/processes should be used to query openconfig-procmon data.

  • The following optional items are not supported:

    • Path prefix

    • Path alias

    • Wildcards in path

  • A single SetRequest can have up to 20 paths.

  • The maximum gRPC receive buffer size is set to 8 MB.

  • The number of total concurrent sessions for Get is limited to five.

  • Performing a Set operation when a large configuration is applied to the switch might cause the gRPC process to consume all available memory. If a memory exhaustion condition is hit, the following syslog is generated: 

    MTX-API: The memory usage is reaching the max memory resource limit (3072) MB

    If this condition is hit several times consecutively, the following syslog is generated:

    The process has become unstable and the feature should be restarted.

    We recommend that you restart the gRPC feature at this point to continue normal processing of gNMI transactions.

  • For the Set::Delete RPC, an MTX log message warns if the configuration being operated on may be too large:

    Configuration size for this namespace exceeds operational limit. Feature may become unstable and require restart.

Subscribe

Guidelines and Limitations for Subscribe

Following are the guidelines and limitations for Subscribe:

  • Beginning with Cisco NX-OS Release 9.3(3), Subscribe supports the OpenConfig model.

  • The gNMI feature supports Subscribe and Capability as options of the gNMI service.

  • The feature supports JSON and gnmi.proto encoding. The feature does not support protobuf.any encoding.

  • Each gNMI message has a maximum size of 12 MB. If the amount of collected data exceeds the 12-MB maximum, the collected data is dropped.

    You can avoid this situation by creating more focused subscriptions that handle smaller, more granular data-collection sets. So, instead of subscribing to one higher-level path, create multiple subscriptions for different, lower-level parts of the path.

  • All paths within the same subscription request must have the same sample interval. If the same path requires different sample intervals, create multiple subscriptions.

  • The feature does not support a path prefix in the Subscription request, but the Subscription can contain an empty prefix field.

  • The feature supports Cisco DME and Device YANG data models. Openconfig YANG is not supported.

  • The gRPC process that supports gNMI uses the HIGH_PRIO cgroup, which limits the CPU usage to 75% of CPU and memory to 1.5 GB.

  • The show grpc gnmi command has the following considerations:

    • The commands are not XMLized in this release.

    • The gRPC agent retains gNMI calls for a maximum of 1 hour after the call has ended.

    • If the total number of calls exceeds 2000, the gRPC agent purges ended calls based an internal cleanup routine.

The gRPC server runs in the management VRF. As a result, the gRPC process communicates only in this VRF forcing the management interface to support all gRPC calls.

gRPC functionality now includes the default VRF for a total of 2 gRPC servers on each Cisco Nexus 3000 Series switch. You can run one gRPC server in each VRF, or run only one gRPC server in the management VRF. Supporting a gRPC in the default VRF adds flexibility to offload processing gRPC calls from the management VRF, where significant traffic load might not be desirable.

If two gRPC servers are configured, be aware of the following:

  • VRF boundaries are strictly enforced, so each gRPC server processes requests independent of the other, and requests do not cross between VRFs.

  • The two servers are not HA or fault tolerant. One gRPC server does not back up the other, and there is no switchover or switchback between them.

  • Any limits for the gRPC server are per VRF.

gNMI Payload

gNMI uses a specific payload format to subscribe to:

  • DME Streams

  • YANG Streams

Subscribe operations are supported with the following modes:

  • ONCE: Subscribe and receive data once and close session.

  • POLL: Subscribe and keep session open, client sends poll request each time data is needed.

  • STREAM: Subscribe and receive data at specific cadence. The payload accepts values in nanoseconds 1 second = 1000000000.

  • ON_CHANGE: Subscribe, receive a snapshot, and only receive data when something changes in the tree.

Setting modes:

  • Each mode requires 2 settings, inside sub and outside sub

  • ONCE: SAMPLE, ONCE

  • POLL: SAMPLE, POLL

  • STREAM: SAMPLE, STREAM

  • ON_CHANGE: ON_CHANGE, STREAM

Origin

  • DME: Subscribing to DME model

  • device: Subscribing to YANG model

Name

  • DME = subscribing to DME model

  • Cisco-NX-OS-device = subscribing to YANG model

Encoding

  • JSON = Stream will be send in JSON format.

  • PROTO = Stream will be sent in protobuf.any format.

Sample gNMI Payload for DME Stream


Note

Different clients have their own input format. 

{
    "SubscribeRequest":
    [
        {
            "_comment" : "ONCE request",
            "_delay" : 2,
            "subscribe":
            {
                "subscription":
                [
                    {
                        "_comment" : "1st subscription path",
                        "path":
                        {
                            "origin": "DME",
                            "elem":
                            [
                                {
                                    "name": "sys"
                                },
                                {
                                    "name": "bgp"
                                 }
                             ]
                        },
                        "mode": "SAMPLE"
                    }
                ],
                "mode": "ONCE",
                "allow_aggregation" : false,
                "use_models":
                [
                    {
                        "_comment" : "1st module",
                        "name": "DME",
                        "organization": "Cisco Systems, Inc.",
                        "version": "1.0.0"
                    }
                ],
                "encoding": "JSON"
            }
        }
    ]
}

Sample gNMI Payload YANG Stream


{
    "SubscribeRequest":
    [
        {
            "_comment" : "ONCE request",
            "_delay" : 2,
            "subscribe":
            {
                "subscription":
                [
                    {
                        "_comment" : "1st subscription path",
                        "path":
                        {
                            "origin": "device",
                            "elem":
                            [
                                {
                                    "name": "System"
                                 },
                                 {
                                     "name": "bgp-items"
                                  }
                              ]
                          },
                                                  "mode": "SAMPLE"
                      }
                  ],
                  "mode": "ONCE",
                  "allow_aggregation" : false,
                  "use_models":
                  [
                      {
                          "_comment" : "1st module",
                          "name": "Cisco-NX-OS-device",
                          "organization": "Cisco Systems, Inc.",
                          "version": "0.0.0"
                      }
                  ],
                  "encoding": "JSON"
              }
          }
      ]
}

Streaming Syslog

About Streaming Syslog for gNMI

gNMI Subscribe is a new way of monitoring the network as it provides a real-time view of what's going on in your system by pushing the structured data as per gNMI Subscribe request.

Beginning with the Cisco NX-OS Release 9.3(3), support is added for gNMI Subscribe functionality.

gNMI Subscribe Support Detail

  • Syslog-oper model streaming

    • stream_on_change

This feature applies to Cisco Nexus 3000 Series switches with 8 GB or more of memory.

Guidelines and Limitations for Streaming Syslog - gNMI

The following are guidelines and limitations for Streaming Syslog:

  • An invalid syslog is not supported. For example, a syslog with a filter or query condition

  • Only the following paths are supported:

    • Cisco-NX-OS-Syslog-oper:syslog

    • Cisco-NX-OS-Syslog-oper:syslog/messages

  • The following modes are not supported:

    • Stream sample

    • POLL

  • A request must be in the YANG model format.

  • You can use the internal application or write your own application.

  • The payload comes from the controller and gNMI sends a response.

  • Encoding formats are JSON and PROTO.

Syslog Native YANG Model

The YangModels are located here.


Note

The time-zone field is set only when the clock format show-timezone syslog is entered. By default, it's not set, therefore the time-zone field is empty. 

PYANG Tree for Syslog Native Yang Model:
>>> pyang -f tree Cisco-NX-OS-infra-syslog-oper.yang
module: Cisco-NX-OS-syslog-oper
+--ro syslog
+--ro messages
+--ro message* [message-id]
+--ro message-id int32
+--ro node-name? string
+--ro time-stamp? uint64
+--ro time-of-day? string
+--ro time-zone? string
+--ro category? string
+--ro group? string
+--ro message-name? string
+--ro severity? System-message-severity
+--ro text? string

Subscribe Request Example

The following is an example of a Subscribe request:

{
    "SubscribeRequest":
    [
        {
            "_comment" : "STREAM request",
            "_delay"   : 2,
            "subscribe":
            {
                "subscription":
                [
                    {
                        "_comment" : "1st subscription path",
                        "path":
                        {
                            "origin": "syslog-oper",
                            "elem":
                              [
                                {
                                    "name": "syslog"
                                },
                                {
                                    "name":"messages"
                                }
                            ]
                        },
                        "mode": "ON_CHANGE"
                    }
                ],
                "mode": "ON_CHANGE",
             "allow_aggregation" : false,
                "use_models":
                [
                    {
                        "_comment" : "1st module",
                        "name": "Cisco-NX-OS-Syslog-oper",
                        "organization": "Cisco Systems, Inc.",
                        "version": "0.0.0"
                    }
                ],
                "encoding":"JSON"
            }
        }
    ]
}

Sample PROTO Output

This is a sample of PROTO output. 

############################

[Subscribe]-------------------------------

### Reading from file ' /root/gnmi-console/testing_bl/stream_on_change/OC_SYSLOG.json '

Sat Aug 24 14:38:06 2019

### Generating request : 1 -----------

### Comment : STREAM request

### Delay : 2 sec(s) ...

### Delay : 2 sec(s) DONE

subscribe {

subscription {

path {

origin: "syslog-oper"

elem {

name: "syslog"

}

elem {

name: "messages"

}

}

mode: ON_CHANGE

}

use_models {

name: "Cisco-NX-OS-Syslog-oper"

organization: "Cisco Systems, Inc."

version: "0.0.0"

}

encoding: PROTO

}

Thu Nov 21 14:26:41 2019
Received response 3 --------------------------
update {
timestamp: 1574375201665688000
prefix {
origin: "Syslog-oper"
elem {
name: "syslog"
}
elem {
name: "messages"
}
}
update {
path {
elem {
name: "message-id"
}
}
val {
uint_val: 529
}
}
update {
path {
elem {
name: "node-name"
}
}
val {
string_val: "task-n9k-1"
}
}
update {
path {
elem {
name: "message-name"
}
}
val {
string_val: "VSHD_SYSLOG_CONFIG_I"
}
}
update {
path {
elem {
name: "text"
}
}
val {
string_val: "Configured from vty by admin on console0"
}
}
update {
path {
elem {
name: "group"
}
}
val {
string_val: "VSHD"
}
}
update {
path {
elem {
name: "category"
}
}
val {
string_val: "VSHD"
}
}
update {
path {
elem {
name: "time-of-day"
}
}
val {
string_val: "Nov 21 2019 14:26:40"
}
}
update {
path {
elem {
name: "time-zone"
}
}
val {
string_val: ""
}
}
update {
path {
elem {
name: "time-stamp"
}
}
val {
uint_val: 1574375200000
}
}
update {
path {
elem {
name: "severity"
}
}
val {
uint_val: 5
}
}
}

/Received -------------------------------------

Sample JSON Output

This is a sample JSON output.

[Subscribe]-------------------------------
### Reading from file ' testing_bl/stream_on_change/OC_SYSLOG.json '


Tue Nov 26 11:47:00 2019
### Generating request : 1 -----------
### Comment : STREAM request
### Delay : 2 sec(s) ...
### Delay : 2 sec(s) DONE
subscribe {
subscription {
path {
origin: "syslog-oper"
elem {
name: "syslog"
}
elem {
name: "messages"
}
}
mode: ON_CHANGE
}
use_models {
name: "Cisco-NX-OS-Syslog-oper"
organization: "Cisco Systems, Inc."
version: "0.0.0"
}
}

Tue Nov 26 11:47:15 2019
Received response 5 --------------------------
update {
timestamp: 1574797636002053000
prefix {
}
update {
path {
origin: "Syslog-oper"
elem {
name: "syslog"
}
}
val {
json_val: "[ { \"messages\" : [[ {\"message-id\":657},{\"node-name\":\"task-n9k-1\",\"time-stamp\":\"1574797635000\",\"time-of-day\":\"Nov 26 2019 11:47:15\",\"severity\":3,\"message-name\":\"HDR_L2LEN_ERR\",\"category\":\"ARP\",\"group\":\"ARP\",\"text\":\"arp [30318] Received packet with incorrect layer 2 address length (8 bytes), Normal pkt with S/D MAC: 003a.7d21.d55e ffff.ffff.ffff eff_ifc mgmt0(9), log_ifc mgmt0(9), phy_ifc mgmt0(9)\",\"time-zone\":\"\"} ]] } ]"
}
}
}

/Received -------------------------------------

Troubleshooting

Gathering TM-Trace Logs

1. tmtrace.bin -f gnmi-logs gnmi-events gnmi-errors following are available
2. Usage:

bash-4.3# tmtrace.bin -d gnmi-events | tail -30 Gives the last 30
}
}
}
[06/21/19 15:58:38.969 PDT f8f 3133] [3981658944][tm_transport_internal.c:43] dn: Cisco-NX-OS-device:System/cdp-items, sub_id: 0,
sub_id_str: 2329, dc_start_time: 0, length: 124, sync_response:1
[06/21/19 15:58:43.210 PDT f90 3133] [3621780288][tm_ec_yang_data_processor.c:93] TM_EC: [Y] Data received for 2799743488: 49
{
"cdp-items" : {
"inst-items" : {
"if-items" : {
"If-list" : [
{
"id" : "mgmt0",
"ifstats-items" : {
"v2Sent" : "74",
"validV2Rcvd" : "79"
}
}
]
}
}
}
}
[06/21/19 15:58:43.210 PDT f91 3133] [3981658944][tm_transport_internal.c:43] dn: Cisco-NX-OS-device:System/cdp-items, sub_id: 0,
sub_id_str: 2329, dc_start_time: 0, length: 141, sync_response:1
[06/21/19 15:59:01.341 PDT f92 3133] [3981658944][tm_transport_internal.c:43] dn: Cisco-NX-OS-device:System/intf-items, sub_id:
4091, sub_id_str: , dc_start_time: 1561157935518, length: 3063619, sync_response:0
[06/21/19 15:59:03.933 PDT f93 3133] [3981658944][tm_transport_internal.c:43] dn: Cisco-NX-OS-device:System/cdp-items, sub_id:
4091, sub_id_str: , dc_start_time: 1561157940881, length: 6756, sync_response:0
[06/21/19 15:59:03.940 PDT f94 3133] [3981658944][tm_transport_internal.c:43] dn: Cisco-NX-OS-device:System/lldp-items, sub_id:
4091, sub_id_str: , dc_start_time: 1561157940912, length: 8466, sync_response:1
bash-4.3#

Gathering MTX-Internal Logs

1. Modify the following file with below /opt/mtx/conf/mtxlogger.cfg

<config name="nxos-device-mgmt">
  <container name="mgmtConf">
    <container name="logging">
      <leaf name="enabled" type="boolean" default="false">true</leaf>
      <leaf name="allActive" type="boolean" default="false">true<
/leaf>
      <container name="format">
        <leaf name="content" type="string" default="$DATETIME$
$COMPONENTID$ $TYPE$: $MSG$">$DATETIME$ $COMPONENTID$ $TYPE$
$SRCFILE$ @ $SRCLINE$ $FCNINFO$:$MSG$</leaf>
            <container name="componentID">
          <leaf name="enabled" type="boolean" default="true"></leaf>
            </container>
            <container name="dateTime">
          <leaf name="enabled" type="boolean" default="true"></leaf>
          <leaf name="format" type="string" default="%y%m%d.%H%M%S"><
/leaf>
             </container>
             <container name="fcn">
           <leaf name="enabled" type="boolean" default="true"></leaf>        
           <leaf name="format" type="string"
default="$CLASS$::$FCNNAME$($ARGS$)@$LINE$"></leaf>
             </container>
      </container>
      <container name="facility">
          <leaf name="info" type="boolean" default="true">true</leaf>
          <leaf name="warning" type="boolean" default="true">true<
/leaf>
          <leaf name="error" type="boolean" default="true">true</leaf>
          <leaf name="debug" type="boolean" default="false">true<
/leaf>
        </container>
        <container name="dest">
          <container name="console">
            <leaf name="enabled" type="boolean" default="false">true<
/leaf>
          </container>
          <container name="file">
         <leaf name="enabled" type="boolean" default="false">true<
/leaf>
    <leaf name="name" type="string" default="mtx-internal.log"><
/leaf>
                                                                                                                                                                                            <leaf name="location" type="string" default="./mtxlogs">
/volatile</leaf>
              <leaf name="mbytes-rollover" type="uint32" default="10"
>50</leaf>
              <leaf name="hours-rollover" type="uint32" default="24"
>24</leaf>
              <leaf name="startup-rollover" type="boolean" default="
false">true</leaf>
            <leaf name="max-rollover-files" type="uint32" default="10"
>10</leaf>
        </container>
      </container>
      <list name="logitems" key="id">
        <listitem>
                <leaf name="id" type="string">*</leaf>
                    <leaf name="active" type="boolean" default="false"
>false</leaf>
         </listitem>
          <listitem>
                 <leaf name="id" type="string">MTX-EvtMgr</leaf>
                     <leaf name="active" type="boolean" default="true"
>true</leaf>
        </listitem>
        <listitem>
               <leaf name="id" type="string">TM-ADPT</leaf>
                   <leaf name="active" type="boolean" default="true"
>false</leaf>
        </listitem>
        <listitem>
              <leaf name="id" type="string">TM-ADPT-JSON</leaf>
                  <leaf name="active" type="boolean" default="true"
>false</leaf> 
        </listitem  >
        <listitem>
                <leaf name="id" type="string">SYSTEM</leaf>
                    <leaf name="active" type="boolean" default="true"
>true</leaf>
        </listitem>
        <listitem>
                <leaf name="id" type="string">LIBUTILS</leaf>
                     <leaf name="active" type="boolean" default="true"
>true</leaf>
        </listitem>
        <listitem>
                <leaf name="id" type="string">MTX-API</leaf>
                    <leaf name="active" type="boolean" default="true"
>true</leaf>
        </listitem>
         <listitem>
                 <leaf name="id" type="string">Model-*</leaf>
                     <leaf name="active" type="boolean" default="true"
>true</leaf>
        </listitem>
        <listitem>
                <leaf name="id" type="string">Model-Cisco-NX-OS-
device</leaf>
                 <leaf name="active" type="boolean" default="true"
>false</leaf>         
        </listitem>
        <listitem>
                 <leaf name="id" type="string">Model-openconfig-bgp<
/leaf>
                     <leaf name="active" type="boolean" default="true"
>false</leaf>
        </listitem>
        <listitem>
               <leaf name="id" type="string">INST-MTX-API</leaf>
                   <leaf name="active" type="boolean" default="true"
>true</leaf>
        </listitem>
        <listitem>
                <leaf name="id" type="string">INST-ADAPTER-NC</leaf>
                    <leaf name="active" type="boolean" default="true"
>true</leaf>
        </listitem>
        <listitem>
               <leaf name="id" type="string">INST-ADAPTER-RC</leaf>
                   <leaf name="active" type="boolean" default="true"
>true</leaf>
        </listitem>
        <listitem>
               <leaf name="id" type="string">INST-ADAPTER-GRPC</leaf>
                   <leaf name="active" type="boolean" default="true"
>true</leaf>
         </listitem>
      </list>
    </container>
  </container>
</config>

2. Run "no feature grpc" / "feature grpc"
3. The /volataile directory houses the mtx-internal.log, the log rolls over over time so be sure to grab what  you need before thenbash-4.3# cd /volatile/

bash-4.3# cd /volaiflels -al
total 148
drwxrwxrwx 4 root root 340 Jun 21 15:47 .
drwxrwxr-t 64 root network-admin 1600 Jun 21 14:45 ..
-rw-rw-rw- 1 root root 103412 Jun 21 16:14 grpc-internal-log
-rw-r--r-- 1 root root 24 Jun 21 14:44 mtx-internal-19-06-21-14-46-21.log
-rw-r--r-- 1 root root 24 Jun 21 14:46 mtx-internal-19-06-21-14-46-46.log
-rw-r--r-- 1 root root 175 Jun 21 15:11 mtx-internal-19-06-21-15-11-57.log
-rw-r--r-- 1 root root 175 Jun 21 15:12 mtx-internal-19-06-21-15-12-28.log
-rw-r--r-- 1 root root 175 Jun 21 15:13 mtx-internal-19-06-21-15-13-17.log
-rw-r--r-- 1 root root 175 Jun 21 15:13 mtx-internal-19-06-21-15-13-42.log
-rw-r--r-- 1 root root 24 Jun 21 15:13 mtx-internal-19-06-21-15-14-22.log
-rw-r--r-- 1 root root 24 Jun 21 15:14 mtx-internal-19-06-21-15-19-05.log
-rw-r--r-- 1 root root 24 Jun 21 15:19 mtx-internal-19-06-21-15-47-09.log
-rw-r--r-- 1 root root 24 Jun 21 15:47 mtx-internal.log
-rw-rw-rw- 1 root root 355 Jun 21 14:44 netconf-internal-log
-rw-rw-rw- 1 root root 0 Jun 21 14:45 nginx_logflag
drwxrwxrwx 3 root root 60 Jun 21 14:45 uwsgipy
drwxrwxrwx 2 root root 40 Jun 21 14:43 virtual-instance
bash-4.3#.