gNMI - gRPC Network Management Interface

This chapter contains the following topics:

About gNMI

gNMI uses gRPC (Google Remote Procedure Call) as its transport protocol.

Cisco NX-OS supports gNMI for dial-in subscription to telemetry applications running on switches. Although the past release supported telemetry events over gRPC, the switch pushed the telemetry data to the telemetry receivers. This method was called dial out.

With gNMI, applications can pull information from the switch. They subscribe to specific telemetry services by learning the supported telemetry capabilities and subscribing to only the telemetry services that it needs.

Cisco NX-OS Release 9.3(1) and later supports gNMI version 0.5.0.

Cisco NX-OS Release 9.3(5) and later supports gNMI version 0.5.0.

Table 1. Supported gNMI RPCs

gNMI RPC

Supported

Capabilities

Yes

Get

Yes

Set

Yes

Subscribe

Yes

gNMI RPC and SUBSCRIBE

The NX-OS 9.3(1) release supports gNMI version 0.5.0. Cisco NX-OS Release 9.3(1) supports the following parts of gNMI version 0.5.0.

Table 2. SUBSCRIBE Options

Type

Sub Type

Supported?

Description

Once

Yes

Switch sends current values only once for all specified paths

Poll

Yes

Whenever the switch receives a Poll message, the switch sends the current values for all specified paths.

Stream

Sample

Yes

Once per stream sample interval, the switch sends the current values for all specified paths. The supported sample interval range is from 1 through 604800 seconds.

The default sample interval is 10 seconds.

On_Change Yes The switch sends current values as its initial state, but then updates the values only when changes, such as create, modify, or delete occur to any of the specified paths.
Target_Defined No

Optional SUBSCRIBE Flags

For the SUBSCRIBE option, some optional flags are available that modify the response to the options listed in the table. In release 9.3(1), the updates_only optional flag is supported, which is applicable to ON_CHANGE subscriptions. If this flag is set, the switch suppresses the initial snapshot data (current state) that is normally sent with the first response.

The following flags are not supported:

  • aliases

  • allow_aggregation

  • extensions

  • heart-beat interval

  • prefix

  • qos

  • suppress_redundant

gNMI に関する注意事項と制限事項

gNMI に関する注意事項と制限事項は次のとおりです。

  • Cisco NX-OS リリース 9.3(5) 以降、Get および Set がサポートされています。

  • gNMI クエリは、パス内のワイルドカードをサポートしていません。

  • Nexus デバイス向けの gRPCトラフィックは、デフォルト クラスのコントロール プレーンポリサー（CoPP）にヒットします。gRPC ドロップの可能性を抑えるには、管理クラスの gRPC 構成ポートを使用して、カスタム CoPP ポリシーを構成してください。

  • 管理 VRF とデフォルト VRF の両方で gRPC をイネーブルにし、後でデフォルト VRF でディセーブルにすると、管理 VRF の gNMI 通知が機能しなくなります。

    gNMI 通知を有効にするには、 コマンドを入力して gRPC を有効にする必要があります。feature grpc これにより、管理 VRF で gNMI 通知が有効になります。管理 VRF に加えてデフォルト VRF で gNMI 通知が必要な場合は、 コマンドを適用する必要があります。grpc use-vrf default コマンドを入力してデフォルト VRF で無効になっている場合、管理 VRF の gNMI 通知は機能しません。no grpc use-vrf default

    回避策として、 コマンドを入力して gRPC を完全に無効にし、 コマンドと既存の gRPC コンフィギュレーション コマンドを入力して再プロビジョニングします。no feature grpc feature grpc たとえば、grpc certificate または grpc port 。また、管理 VRF の既存の通知に再登録する必要もあります。

  • 次のような既存の CLI 設定を使用して OpenConfig ルーティングポリシーをサブスクライブしようとすると、OpenConfig モデルの現在の実装により空の値が返されます。 

    ip prefix-list bgp_v4_drop seq 5 deny 125.2.0.0/16 le 32
ipv6 prefix-list bgp_v6_drop seq 5 deny cafe:125:2::/48 le 128

using the xpath

openconfig-routing-policy:/routing-policy/defined-sets/prefix-sets/prefix-set[name=bgp_v4_drop]/config
openconfig-routing-policy:/routing-policy/defined-sets/prefix-sets/prefix-set[name=bgp_v6_drop]/config

  • サーバー証明書認証のみが実行されます。クライアント証明書がサーバーによって認証されません。

  • gRPC 証明書が明示的に設定されている場合、保存されたスタートアップ コンフィギュレーションを使用して以前の Cisco NX-OS 9.3(x) イメージにリロードした後、gRPC 機能は接続を受け入れません。この問題を確認するには、 コマンドを入力します。ステータス行に次のようなエラーが表示されます。show grpc gnmi service statistics 

    Status: Not running - Initializing...Port not available or certificate invalid.

    サービスを復元するには、適切な証明書コマンドを設定解除して設定します。

  • Cisco NX-OS リリース 9.3(3) 以降では、カスタム gRPC 証明書を設定している場合、 コマンドを入力すると設定が失われます。reload ascii デフォルトの day-1 証明書に戻ります。reload ascii コマンドを入力した後には、スイッチをリロードします。スイッチが再び起動したら、gRPC カスタム証明書を再設定する必要があります。


    （注）  

    これは、 コマンドを入力した場合に適用されます。grpc certificate

  • gRPC のデフォルト以外の VRF の到達可能性は、L3VNI/EVPN および IP 経由でのみサポートされます。ただし、デフォルト以外の VRF および VXLAN フラッドおよびラーニングでの MPLS を介した到達可能性はサポートされていません。

  • origin、use_models、またはその両方の使用は、gNMI サブスクリプションではオプションです。

  • gNMI サブスクリプションは、Cisco DME およびデバイス YANG データモデルをサポートします。Cisco NX-OS リリース 9.3(3)以降、サブスクライブは OpenConfig モデルをサポートします。

  • 9.3(x) より前の Cisco NX-OS においてサポートされるプラットフォームの詳細については、そのリリース向けガイドのプログラマビリティ機能のプラットフォーム サポートを参照してください。Cisco NX-OS リリース 9.3(x) 以降、サポートされるプラットフォームの詳細については、Nexus Switch Platform Matrixを参照してください。

  • この機能は、JSON および gnmi.proto エンコーディングをサポートします。この機能は、protobuf.any エンコーディングをサポートしていません。

  • 各 gNMI メッセージの最大サイズは 12 MB です。収集されたデータの量が最大値 12 MB を超えると、収集されたデータはドロップされます。gNMI ON_CHANGE モードにのみ適用されます。

    この状況は、より小規模で詳細なデータ収集セットを処理する、焦点を絞ったサブスクリプションを作成することで回避できます。したがって、1 つの上位レベルのパスにサブスクライブする代わりに、パスの異なる下位レベルの部分に対して複数のサブスクリプションを作成してください。

  • すべてのサブスクリプションで、最大 150K の集約 MO がサポートされます。より多くのMOに登録すると、収集データがドロップする可能性があります。

  • この機能はサブスクリプション要求の単一パス プレフィックスをサポートしていませんが、サブスクリプションには空のプレフィックス フィールドを含めることができます。

  • この機能は、Cisco DME およびデバイス YANG データモデルをサポートします。Openconfig YANG はサポートされていません。

  • gNMI をサポートする gRPC プロセスは、CPU 使用率を CPU の 75% に、メモリを 1.5 GB に制限する HIGH_PRIO 制御グループを使用します。

  • show grpc gnmi コマンドには、次の考慮事項があります。

    • このリリースでは、コマンドは XML 化されていません。

    • gRPC エージェントは、コールが終了した後、最大 1 時間 gNMI コールを保持します。

    • コールの合計数が 2000 を超えると、gRPC エージェントは、内部クリーンアップルーチンに基づいて終了したコールを消去します。

  • Cisco NX-OS リリース 10.2(3)F 以降では、デバイス YANG エフェメラルデータ（アカウンティングログおよびマルチキャスト）のサブスクリプションの変更がサポートされています。

gRPC サーバーは管理 VRF で実行されます。その結果、gRPC プロセスはこの VRF でのみ通信し、管理インターフェイスはすべての gRPC 呼び出しをサポートする必要があります。

gRPC 機能には、各スイッチの合計 2 つの gRPC サーバーのデフォルト VRF が含まれるようになりました。VRF ごとに 1 つの gRPC サーバーを実行することも、管理 VRF で 1 つの gRPC サーバーのみを実行することもできます。デフォルト VRF で gRPC をサポートすると、大量のトラフィック負荷が望ましくない管理 VRF からの gRPC コールの処理を柔軟にオフロードできます。

2 つの gRPC サーバーを構成する場合は、次の点に注意してください。

  • VRF 境界は厳密に適用されるため、各 gRPC サーバーは相互に独立して要求を処理します。要求は VRF 間を通過しません。

  • 2 台のサーバーは HA またはフォールト トレラントではありません。一方の gRPC サーバーは他方をバックアップせず、それらの間でスイッチオーバーまたはスイッチバックはありません。

  • gRPC サーバーの制限は VRF 単位です。

gNMI の制限事項は次のとおりです。

  • パスでは複数レベルのワイルドカード「...」は使用できません

  • パスの先頭にワイルドカード「*」を使用することはできません

  • キー名でワイルドカード「*」を使用することはできません

  • キーにワイルドカードと値を混在させることはできません

次の表に、gNMI のワイルドカードサポートの詳細を示します。

表 3. gNMI 要求のワイルドカード サポート

リクエストの種類

ワイルドカード サポート

gNMI GET

YES

gNMI 設定

NO

gNMI サブスクリプション（1 回）

YES

gNMI SUBSCRIBE、POLL

YES

gNMI SUBSCRIBE、STREAM、SAMPLE

YES

gNMI SUBSCRIBE、STREAM、TARGET_DEFINED

YES

gNMI SUBSCRIBE、STREAM、ON_CHANGE

NO

Configuring gNMI

Configure the gNMI feature through the grpc gnmi commands.

To import certificates used by the grpc certificate command onto the switch, see the Installing Identity Certificates section of the Cisco Nexus 3500 Series NX-OS Security Configuration Guide, Release 9.3(x).


Note

When modifying the installed identity certificates or grpc port and grpc certificate values, the gRPC server might restart to apply the changes. When the gRPC server restarts, any active subscription is dropped and you must resubscribe.

SUMMARY STEPS

  1. configure terminal
  2. feature grpc
  3. (Optional) grpc port port-id
  4. (Optional) grpc certificate certificate-id
  5. grpc gnmi max-concurrent-call number

DETAILED STEPS

  Command or Action Purpose

Step 1

configure terminal

Example:

switch-1# configure terminal
switch-1(config)#

Enters global configuration mode.

Step 2

feature grpc

Example:

switch-1# feature grpc
switch-1(config)#

Enables the gRPC agent, which supports the gNMI interface for dial-in.

Step 3

(Optional) grpc port port-id

Example:

switch-1(config)# grpc port 50051
(Optional)

Configure the port number. The range of port-id is 1024–65535. 50051 id the default.

Note

 

This command is available beginning with Cisco NX-OS Release 9.3(3).

Step 4

(Optional) grpc certificate certificate-id

Example:

switch-1(config)# grpc certificate cert-1
(Optional)

Specify the certificate trustpoint ID. For more information, see the Installing Identity Certificates section of the Cisco Nexus 30009000 Series NX-OS Security Configuration Guide, Release 9.3(x) for importing the certificate to the switch.

Note

 

This command is available beginning with Cisco NX-OS Release 9.3(3).

Step 5

grpc gnmi max-concurrent-call number

Example:

switch-1(config)# grpc gnmi max-concurrent-call 16
switch-1(config)#

Sets the limit of simultaneous dial-in calls to the gNMI server on the switch. Configure a limit from 1 through 16. The default limit is 8.

The maximum value that you configure is for each VRF. If you set a limit of 16 and gNMI is configured for both management and default VRFs, each VRF supports 16 simultaneous gNMI calls.

This command does not affect and ongoing or in-progress gNMI calls. Instead, gRPC enforces the limit on new calls, so any in-progress calls are unaffected and allowed to complete.

Note

 
The configured limit does not affect the gRPCConfigOper service.

gNMI - gRPC Network Management Interface

This section defines a gRPC-based protocol for the modification and retrieval of the configuration from a target device, as well as, the control and generation of telemetry streams from a target device to a data collection system. The intention is that a single gRPC service definition can cover both configuration and telemetry - allowing a single implementation on the target, as well as a single NMS element to interact with the device via telemetry and configuration RPCs.

サーバー証明書の構成

TLS 証明書を設定し、スイッチに正常にインポートした場合の show grpc gnmi service statistics コマンドの出力例を次に示します。 

switch(config)# sh grpc gnmi service statistics

============= gRPC Endpoint
Vrf : management
Server address : [::]:50051

Cert notBefore : Nov 5 16:48:58 2015 GMT
Cert notAfter : Nov 5 16:48:58 2035 GMT
Client Root Cert notBefore : n/a
Client Root Cert notAfter : n/a

Max concurrent calls : 8
Listen calls : 1
Active calls : 0
KeepAlive Timeout : 120

Number of created calls : 1
Number of bad calls : 0

Subscription stream/once/poll : 0/0/0

Max gNMI::Get concurrent : 6
Max grpc message size : 25165824
gNMI Synchronous calls : 3
gNMI Synchronous errors : 3
gNMI Adapter errors : 3
gNMI Dtx errors : 0

gNMI は gRPC を介して通信し、TLS を使用してスイッチとクライアント間のチャネルをセキュアにします。デフォルトのハードコードされた gRPC 証明書は、スイッチに同梱されなくなりました。デフォルトの動作は、次に示すように、スイッチで生成される有効期限が 1 日の自己署名キーと証明書です。

証明書の有効期限が切れているか、正常にインストールできなかった場合は、1-D デフォルト証明書が表示されます。次に、show grpc gnmi service statistics コマンドの出力を示します。 

#show grpc gnmi service statistics

=============
gRPC Endpoint
=============

Vrf            : management
Server address : [::]:50051

Cert notBefore : Wed Mar 11 19:43:01 PDT 2020
Cert notAfter  : Thu Mar 12 19:43:01 PDT 2020

Max concurrent calls            :  8
Listen calls                    :  1
Active calls                    :  0

Number of created calls         :  1
Number of bad calls             :  0

Subscription stream/once/poll   :  0/0/0

有効期限は 1 日ですが、この一時証明書を使用してテストを簡単に行えます。長期的には、新しいキー/証明書を生成する必要があります。

Generating Key/Certificate Examples


Note

Use this procedure for Cisco NX-OS Release 9.3(2) and earlier.

For Cisco NX-OS Release 9.3(3) and later, see

Typically, there are two possible scenarios:

  1. The server and client can use a Self-Signed Certificate. Self-Signed Certificates are less secure and are not to be used in production environments.

    1. Generating self-signed certificates on the switch:

      
1. Generating self-signed certificates on the switch:
        1. Login to the Bash shell:
        2. cd to the location where you want to store the key/cert
        3. switch# openssl req -x509 -newkey rsa:2048 -keyout self_sign2048.key -out self_sign2048.pem -days 365 -nodes
               Generating a 2048 bit RSA private key
                ...............................................
                ........
                writing new private key to 'self_sign2048.key'
                -----
                You are about to be asked to enter information that will be incorporated
                into your certificate request.
                What you are about to enter is what is called a Distinguished Name or a DN.
                There are quite a few fields but you can leave some blank
                For some fields there will be a default value,
                If you enter '.', the field will be left blank.
                -----
                Country Name (2 letter code) [AU]:US
                string is too long, it needs to be less than 2 bytes long
                Country Name (2 letter code) [AU]:US
                State or Province Name (full name) [Some-State]:CA
                Locality Name (eg, city) []:San Jose
                Organization Name (eg, company) [Internet Widgits Pty Ltd]:Cisco Systems
                Organizational Unit Name (eg, section) []:
                Common Name (e.g. server FQDN or YOUR name) []:benton.cisco.com
                Email Address []:
                switch#
                switch# ls -al | grep self
                -rw-r--r-- 1 root root 912 Jun 6 15:16 self_sign2048.key
                -rw-r--r-- 1 root root 952 Jun 6 15:16 self_sign2048.pem
                switch#

2. Modify /etc/mtx.conf file. The mtx.conf file is not persistent after reloads but persistent through "no feature grpc" & "feature grpc"
3. Under the "grpc" section in the file add the key and cert location
   [grpc]
   key = /bootflash/self_sign2048.key
   cert = /bootflash/self_sign2048.pem
4. Go to cli and run
        1. no feature grpc
        2. feature grpc
        3. show grpc internal gnmi service statistics
                1. Verify that the new cert and key are used by grpc process.

5. Optional: If you want the changes to be persistent across reloads
        1. Create a new file /etc/mtx.conf.user
        2. Add the following
        [grpc]
        key = /bootflash/self_sign2048.key
        cert = /bootflash/self_sign2048.pem
        3. To get the mtx.conf file to accept the changes either
                1. Install, activate or de-activate a MTX RPM
                2. Or reload the box

  2. You can generate a CA signed certificate.

Cisco NX-OS リリース 9.3(3) 以降のキー/証明書の生成と構成の例

次に、キー/証明書を生成する例を示します。


（注）  

このタスクは、スイッチで証明書を生成する方法の例です。任意の Linux 環境で証明書を生成することもできます。実稼働環境では、CA 署名付き証明書の使用を検討する必要があります。

アイデンティティ証明書の生成の詳細については、『Cisco Nexus 9000 Series NX-OS Security Configuration Guide, Release 9.3(x)』の「Installing Identity Certificates」セクションを参照してください。https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/93x/security/configuration/guide/b-cisco-nexus-9000-nx-os-security-configuration-guide-93x/b-cisco-nexus-9000-nx-os-security-configuration-guide-93x_chapter_011010.html#task_2088148

手順

ステップ 1

自己署名キーと pem ファイルを生成します。 

switch# run bash sudo su
bash-4.3# openssl req -x509 -newkey rsa:2048 -keyout self_sign2048.key -out self_sign2048.pem -days 365 -nodes

  1. switch# openssl req -x509 -newkey rsa:2048 -keyout self_sign2048.key -out self_sign2048.pem -days 365 -nodes

ステップ 2

キー ファイルと pem ファイルを生成した後、トラストポイント CA アソシエーションで使用するためにキー ファイルと pem ファイルをバンドルする必要があります。

switch# run bash sudo su
bash-4.3# cd /bootflash/
bash-4.3# openssl pkcs12 -export -out self_sign2048.pfx -inkey self_sign2048.key -in self_sign2048.pem -certfile self_sign2048.pem -password pass:Ciscolab123!
bash-4.3# exit

ステップ 3

pkcs12 バンドルをトラストポイントに入力して、トラストポイント CA アソシエーションを設定します。 

switch(config)# crypto ca trustpoint mytrustpoint
switch(config-trustpoint)# crypto ca import mytrustpoint pkcs12 self_sign2048.pfx Ciscolab123!

ステップ 4

セットアップを確認します。 

switch(config)# show crypto ca certificates
Trustpoint: mytrustpoint
certificate:
subject= /C=US/O=Cisco Systems, Inc./OU=CSG/L=San Jose/ST=CA/street=3700 Cisco Way/postalCode=95134/CN=ems.cisco.com/serialNumber=FGE18420K0R
issuer= /C=US/O=Cisco Systems, Inc./OU=CSG/L=San Jose/ST=CA/street=3700 Cisco Way/postalCode=95134/CN=ems.cisco.com/serialNumber=FGE18420K0R
serial=0413
notBefore=Nov  5 16:48:58 2015 GMT
notAfter=Nov  5 16:48:58 2035 GMT
SHA1 Fingerprint=2E:99:2C:CE:2F:C3:B4:EC:C7:E2:52:3A:19:A2:10:D0:54:CA:79:3E
purposes: sslserver sslclient
 
CA certificate 0:
subject= /C=US/O=Cisco Systems, Inc./OU=CSG/L=San Jose/ST=CA/street=3700 Cisco Way/postalCode=95134/CN=ems.cisco.com/serialNumber=FGE18420K0R
issuer= /C=US/O=Cisco Systems, Inc./OU=CSG/L=San Jose/ST=CA/street=3700 Cisco Way/postalCode=95134/CN=ems.cisco.com/serialNumber=FGE18420K0R
serial=0413
notBefore=Nov  5 16:48:58 2015 GMT
notAfter=Nov  5 16:48:58 2035 GMT
SHA1 Fingerprint=2E:99:2C:CE:2F:C3:B4:EC:C7:E2:52:3A:19:A2:10:D0:54:CA:79:3E
purposes: sslserver sslclient

ステップ 5

トラストポイントを使用するように gRPC を構成します。

switch(config)# grpc certificate mytrustpoint
switch(config)# show run grpc
 
!Command: show running-config grpc
!Running configuration last done at: Thu Jul  2 12:24:02 2020
!Time: Thu Jul  2 12:24:05 2020
 
version 9.3(5) Bios:version 05.38
feature grpc
 
grpc gnmi max-concurrent-calls 16
grpc use-vrf default
grpc certificate mytrustpoint

ステップ 6

gRPC が証明書を使用していることを確認します。

switch# show grpc gnmi service statistics

=============
gRPC Endpoint
=============

Vrf : management
Server address : [::]:50051

Cert notBefore : Nov 5 16:48:58 2015 GMT
Cert notAfter : Nov 5 16:48:58 2035 GMT

Max concurrent calls : 16
Listen calls : 1
Active calls : 0

Number of created calls : 953
Number of bad calls : 0

Subscription stream/once/poll : 476/238/238

Max gNMI::Get concurrent : 5
Max grpc message size : 8388608
gNMI Synchronous calls : 10
gNMI Synchronous errors : 0
gNMI Adapter errors : 0
gNMI Dtx errors : 0

Enabling gNMI

This procedure starts the gNMI agent which runs on the management VRF and port 50051.

You can configure it to run on the additional default VRF or configure the maximal concurrent connection number.

Before you begin

You might need to configure the key/cert. If you do not, the server generates a pair of temporary key/certs under /opt/mtx/etc/grpc.pem and /opt/mtx/etc/grpc.key. You need to download the key/cert to its clients. It is valid for one-day use.

SUMMARY STEPS

  1. configure terminal
  2. feature grpc
  3. grpc gnmi max-concurrent-calls number
  4. grpc use-vrf default

DETAILED STEPS

  Command or Action Purpose

Step 1

configure terminal

Example:

switch# configure terminal

Enters global configuration mode.

Step 2

feature grpc

Example:

switch(config)# feature grpc

Enables the gRPC agent, which supports the gNMI interface.

Step 3

grpc gnmi max-concurrent-calls number

Example:

switch(config)# grpc gnmi max-concurrent-calls 16

Sets the limit of simultaneous dial-in calls to the gNMI server on the switch. Configure a limit from 1 through 16. The default limit is 8.

The maximum value that you configure is for each VRF. If you set a limit of 16 and gNMI is configured for both the management and default VRF, each VRF supports 16 simultaneous gNMI calls.

This command does not affect ongoing or in-progress gNMI calls. Instead, gRPC enforces the limit on new calls. Any in-progress calls are unaffected and allowed to complete.

Note

 

The configured limit does not affect the gRPCConfigOper service.

Step 4

grpc use-vrf default

Example:

switch(config)# grpc use-vrf default

Enables the gRPC agent to accept incoming (dial-in) RPC requests from the default VRF. This step enables the default VRF to process incoming RPC requests. By default, the management VRF processes incoming RPC requests when the gRPC feature is enabled.

Note

 

Both VRFs process requests individually, so that requests do not cross between VRFs.

Verifying gNMI

To verify the gNMI configuration, enter the following command:

Command

Description

show grpc gnmi service statistics

Displays a summary of the agent running status, respectively for the management VRF, or the default VRF (if configured). It also displays:

  • Basic overall counters

  • Certificate expiration time

    Note

     

    If the certificate is expired, the agent cannot accept requests.

show grpc gnmi rpc summary

Displays the following:

  • Number of capability RPCs received.

  • Capability RPC errors.

  • Number of Get RPCs received.

  • Get RPC errors.

  • Number of Set RPCs received.

  • Set RPC errors.

  • More error types and counts.

show grpc gnmi transactions

The show grpc gnmi transactions command is the most dense and contains considerable information. It is a history buffer of the most recent 50 gNMI transactions that are received by the switch. As new RPCs come in, the oldest history entry is removed from the end. The following explains what is displayed:

  • RPC – This shows the type of RPC that was received (Get, Set, Capabilities)

  • DataType – For a Get only. Has values ALL, CONFIG, and STATE.

  • Session – shows the unique session-id that is assigned to this transaction. It can be used to correlate data that is found in other log files.

  • Time In -- shows timestamp of when the RPC was received by the gNMI handler.

  • Duration – time delta in ms from receiving the request to giving response.

  • Status – the status code of the operation returned to the client (0 = Success, !0 == error).

This section is data that is kept per path within a single gNMI transaction. For example, a single Get or Set

  • subtype – for a Set RPC, shows the specific operation that is requested per path (Delete, Update, Replace). For Get, there is no subtype.

  • dtx – shows that this path was processed in DTX “fast” path or not. A dash ‘-‘ means no, an asterisk ‘*’ means yes.

  • st – Status for this path. The meaning is as follows:

    • OK: path is valid and processed by infra successfully.

    • ERR: path is either invalid or generated error by infra

    • --: path not processed yet, might or might not be valid and has not been sent to infra yet.

  • path – the path

show grpc gnmi service statistics Example 

=============
gRPC Endpoint
=============

Vrf : management
Server address : [::]:50051

Cert notBefore : Mar 13 19:05:24 2020 GMT
Cert notAfter : Nov 20 19:05:24 2033 GMT

Max concurrent calls : 8
Listen calls : 1
Active calls : 0

Number of created calls : 1
Number of bad calls : 0

Subscription stream/once/poll : 0/0/0

Max gNMI::Get concurrent : 5
Max grpc message size : 8388608
gNMI Synchronous calls : 74
gNMI Synchronous errors : 0
gNMI Adapter errors : 0
gNMI Dtx errors : 0

show grpc gnmi rpc summary Example 

=============
gRPC Endpoint
=============
 
Vrf            : management
Server address : [::]:50051
 
Cert notBefore : Mar 31 20:55:02 2020 GMT
Cert notAfter  : Apr  1 20:55:02 2020 GMT
 
Capability rpcs    : 1   
Capability errors  : 0     
Get rpcs           : 53   
Get errors         : 19   
Set rpcs           : 23   
Set errors         : 8    
Resource Exhausted : 0     
Option Unsupported : 6
Invalid Argument   : 18
Operation Aborted  : 1
Internal Error     : 2
Unknown Error      : 0
 
RPC Type        State      Last Activity  Cnt Req    Cnt Resp   Client
--------------- ---------- -------------- ---------- ---------- -----------------
-----------------------
Subscribe       Listen     04/01 07:39:21          0          0

show grpc gnmi transactions Example 

=============
gRPC Endpoint
=============
 
Vrf            : management
Server address : [::]:50051
 
Cert notBefore : Mar 31 20:55:02 2020 GMT
Cert notAfter  : Apr  1 20:55:02 2020 GMT
 
RPC          DataType   Session         Time In              Duration(ms) Status
------------ ---------- --------------- -------------------- ------------ ------
Set          -          2361443608      04/01 07:43:49       173          0    
subtype: dtx:  st: path:                                                       
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo789]          
 
Set          -          2293989720      04/01 07:43:45       183          0
subtype: dtx:  st: path:                                                   
Replace  -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo6]         
 
Set          -          2297110560      04/01 07:43:41       184          0
subtype: dtx:  st: path:                                                   
Update   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo7]        
 
Set          -          0               04/01 07:43:39       0            10
 
Set          -          3445444384      04/01 07:43:33       3259         0
subtype: dtx:  st: path:                                                   
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo789]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo790]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo791]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo792]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo793]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo794]       
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo795]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo796]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo797]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo798]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo799]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo800]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo801]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo802]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo803]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo804]       
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo805]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo806]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo807]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo808]      
 
Set          -          2297474560      04/01 07:43:26       186          0
subtype: dtx:  st: path:                                                   
Update   -     OK  /System/ipv4-items/inst-items/dom-items/Dom-list[name=foo]/rt-
items/Route-list[prefix=0.0.0.0/0]/nh-items/Nexthop-list[nhAddr=192.168.1.1/32][n
hVrf=foo][nhIf=unspecified]/tag                                                  
 
 
Set          -          2294408864      04/01 07:43:17       176          13
subtype: dtx:  st: path:                                                    
Delete   -     ERR /System/intf-items/lb-items/LbRtdIf-list/descr           
 
 
Set          -          0               04/01 07:43:11       0            3
subtype: dtx:  st: path:                                                   
Update   -     --  /System/intf-items/lb-items/LbRtdIf-list[id=lo4]/descr  
Update   -     ERR /system/processes                                       
 
Set          -          2464255200      04/01 07:43:05       708          0
subtype: dtx:  st: path:                                                   
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo2]        
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo777]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo778]      
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo779]       
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo780]      
Replace  -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo3]/descr  
Replace  -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo4]/descr  
Replace  -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo5]/descr  
Update   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo3]/descr  
Update   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo4]/descr  
Update   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo5]/descr  
 
 
Set          -          3491213208      04/01 07:42:58       14           0
subtype: dtx:  st: path:                                                   
Replace  -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo3]/descr  
 
Set          -          3551604840      04/01 07:42:54       35           0
subtype: dtx:  st: path:                                                   
Delete   -     OK  /System/intf-items/lb-items/LbRtdIf-list[id=lo1]        
 
Set          -          2362201592      04/01 07:42:52       13           13
subtype: dtx:  st: path:                                                    
Delete   -     ERR /System/intf-items/lb-items/LbRtdIf-list[id=lo3]/lbrtdif-items
/operSt                                                                         
 
 
Set          -          0               04/01 07:42:47       0            3
subtype: dtx:  st: path:                                                   
Delete   -     ERR /System/*                                               
 
Set          -          2464158360      04/01 07:42:46       172          3
subtype: dtx:  st: path:                                                   
Delete   -     ERR /system/processes/shabang                               
 
Set          -          2295440864      04/01 07:42:46       139          3
subtype: dtx:  st: path:                                                   
Delete   -     ERR /System/invalid/path                                     
 
Set          -          3495739048      04/01 07:42:44       10           0
 
 
Get          ALL        3444580832      04/01 07:42:40       3            0
subtype: dtx:  st: path:                                                   
-        -     OK  /System/bgp-items/inst-items/disPolBatch                
 
Get          ALL        0               04/01 07:42:36       0            3
subtype: dtx:  st: path:                                                    
-        -     --  /system/processes/process[pid=1]                        
 
Get          ALL        3495870472      04/01 07:42:36       2            0
subtype: dtx:  st: path:                                                    
-        *     OK  /system/processes/process[pid=1]                        
 
Get          ALL        2304485008      04/01 07:42:36       33           0
subtype: dtx:  st: path:                                                   
-        *     OK  /system/processes                                       
 
Get          ALL        2464159088      04/01 07:42:36       251          0
subtype: dtx:  st: path:                                                   
-        -     OK  /system                                                  
 
Get          ALL        2293232352      04/01 07:42:35       258          0
subtype: dtx:  st: path:                                                   
-        -     OK  /system                                                  
 
Get          ALL        0               04/01 07:42:33       0            12
subtype: dtx:  st: path:                                                    
-        -     --  /intf-items

gRPC クライアント証明書認証

10.1(1) リリース以降、gRPC に追加の認証方式が提供されます。 10.1(1) リリースより前の gRPC サービスは、サーバー証明書のみをサポートしていました。10.1(1) 以降では、クライアント証明書のサポートも追加するように認証が拡張され、gRPC でサーバー証明書とクライアント証明書の両方を検証できるようになっています。この機能拡張により、さまざまなクライアントにパスワードなしの認証が提供されます。

新しいクライアント ルート CA 証明書の生成

次に、クライアント ルートに新しい証明書を生成する例を示します。

  • 信頼できる認証局（CA）

DigiCert などの信頼できる CA を使用する場合は、次の手順を実行します。

手順の概要

  1. CA 証明書ファイルをダウンロードします。
  2. Cisco NX-OS セキュリティ構成ガイドの手順に従って、NX-OS にインポートします。

手順の詳細

  コマンドまたはアクション 目的

ステップ 1

CA 証明書ファイルをダウンロードします。

ステップ 2

Cisco NX-OS セキュリティ構成ガイドの手順に従って、NX-OS にインポートします。

（注）  

 

cat [CA_cert_file] からの CA 証明書を使用します。

NX-OS デバイスでの生成されたルート CA 証明書の構成

クライアント root に対する新しい証明書が正常に生成されたときの、スイッチで証明書を構成するためのコマンド例とその出力を次に示します。

switch(config)# crypto ca trustpoint my_client_trustpoint
enticate my_client_trustpoint
switch(config-trustpoint)# crypto ca authenticate my_client_trustpoint
input (cut & paste) CA certificate (chain) in PEM format;
end the input with a line containing only END OF INPUT :
-----BEGIN CERTIFICATE-----
MIIDUDCCAjigAwIBAgIJAJLisBKCGjQOMA0GCSqGSIb3DQEBCwUAMD0xCzAJBgNV
BAYTAlVTMQswCQYDVQQIDAJDQTERMA8GA1UEBwwIU2FuIEpvc2UxDjAMBgNVBAoM
BUNpc2NvMB4XDTIwMTAxNDIwNTYyN1oXDTQwMTAwOTIwNTYyN1owPTELMAkGA1UE
BhMCVVMxCzAJBgNVBAgMAkNBMREwDwYDVQQHDAhTYW4gSm9zZTEOMAwGA1UECgwF
Q2lzY28wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDEX7qZ2EdogZU4
EW0NSpB3EjY0nSlFLOw/iLKSXfIiQJD0Qhaw16fDnnYZj6vzWEa0ls8canqHCXQl
gUyxFOdGDXa6neQFTqLowSA6UCSQA+eenN2PIpMOjfdFpaPiHu3mmcTI1xP39Ti3
/y548NNORSepApBNkZ1rJSB6Cu9AIFMZgrZXFqDKBGSUOf/CPnvIDZeLcun+zpUu
CxJLA76Et4buPMysuRqMGHIX8CYw8MtjmuCuCTHXNN31ghhgpFxfrW/69pykjU3R
YOrwlSUkvYQhtefHuTHBmqym7MFoBEchwrlC5YTduDzmOvtkhsmpogRe3BiIBx45
AnZdtdi1AgMBAAGjUzBRMB0GA1UdDgQWBBSh3IqRrm+mtB5GNsoLXFb3bAVg5TAf
BgNVHSMEGDAWgBSh3IqRrm+mtB5GNsoLXFb3bAVg5TAPBgNVHRMBAf8EBTADAQH/
MA0GCSqGSIb3DQEBCwUAA4IBAQAZ4Fpc6lRKzBGJQ/7oK1FNcTX/YXkneXDk7Zrj
8W0RS0Khxgke97d2Cwl5P5reXO27kvXsnsz/VZn7JYGUvGSlxTlcCb6x6wNBr4Qr
t9qDBu+LykwqNOFe4VCAv6e4cMXNbH2wHBVS/NSoWnM2FGZ10VppjEGFm6OM+N6z
8n4/rWslfWFbn7T7xHH+Nl0Ffc+8q8h37opyCnb0ILj+a4rnyus8xXJPQb05DfJe
ahPNfdEsXKDOWkrSDtmKwtWDqdtjSQC4xioKHoshnNgWBJbovPlMQ64UrajBycwV
z9snWBm6p9SdTsV92YwF1tRGUqpcI9olsBgH7FUVU1hmHDWE
-----END CERTIFICATE-----
END OF INPUT
Fingerprint(s): SHA1 Fingerprint=0A:61:F8:40:A0:1A:C7:AF:F2:F7:D9:C7:12:AE:29:15:52:9D:D2:AE


Do you accept this certificate? [yes/no]:yes
switch(config)# 

NOTE: Use the CA Certificate from the .pem file content.

switch# show crypto ca certificates
Trustpoint: my_client_trustpoint
CA certificate 0:
subject=C = US, ST = CA, L = San Jose, O = Cisco
issuer=C = US, ST = CA, L = San Jose, O = Cisco
serial=B7E30B8F4168FB87
notBefore=Oct  1 17:29:47 2020 GMT
notAfter=Sep 26 17:29:47 2040 GMT
SHA1 Fingerprint=E4:91:4E:D4:41:D2:7D:C0:5A:E8:F7:2D:32:81:B3:37:94:68:89:10
purposes: sslserver sslclient

gRPC へのトラストポイントの関連付け

クライアント ルートに新しい証明書を正常に構成した場合に、スイッチ上でトラストポイントを gRPC に関連付ける出力例を次に示します。


（注）  

クライアント認証用のルート証明書を構成または削除すると、gRPCプロセスが再起動します。

 
# switch(config)# feature grpc
 
switch(config)# grpc client root certificate my_client_trustpoint
switch(config)# show run grpc
 
!Command: show running-config grpc
!Running configuration last done at: Wed Dec 16 20:18:35 2020
!Time: Wed Dec 16 20:18:40 2020
 
version 10.1(1) Bios:version N/A
feature grpc
 
grpc gnmi max-concurrent-calls 14
grpc use-vrf default
grpc certificate my_trustpoint            
grpc client root certificate my_client_trustpoint   
grpc port 50003

証明書の詳細の検証

スイッチの gRPC にトラストポイントを正常に関連付けられた場合の、証明書の詳細を検証するための出力例を次に示します。

switch# show grpc gnmi service statistics
 
=============
gRPC Endpoint
=============

Vrf : management
Server address : [::]:50003

Cert notBefore : Mar 13 19:05:24 2020 GMT
Cert notAfter : Nov 20 19:05:24 2033 GMT
Client Root Cert notBefore : Oct 1 17:29:47 2020 GMT
Client Root Cert notAfter : Sep 26 17:29:47 2040 GMT

Max concurrent calls : 14
Listen calls : 1
Active calls : 0

Number of created calls : 1
Number of bad calls : 0

Subscription stream/once/poll : 0/0/0

Max gNMI::Get concurrent : 5
Max grpc message size : 8388608
gNMI Synchronous calls : 0
gNMI Synchronous errors : 0
gNMI Adapter errors : 0
gNMI Dtx errors : 0

任意の gNMI クライアントのクライアント証明書認証を使用した接続の確認

クライアント証明書は、秘密キー（pkey）と CA チェーン（cchain）を使用して要求を行います。現在では、パスワードはオプションです。

Performing GetRequest, encoding = JSON to 172.19.199.xxx with the following gNMI Path
 -------------------------
 [elem {
  name: "System"
}
elem {
  name: "bgp-items"
}
]
The GetResponse is below
-------------------------
 
notification {
  timestamp: 1608071208072199559
  update {
    path {
      elem {
        name: "System"
      }
      elem {
        name: "bgp-items"
      }
    }
    val {
      json_val: ""
    }
  }
}

gRPC からトラストポイント参照を削除するには（no コマンド）、次のコマンドを使用します。

 [no] grpc client root certificate <my_client_trustpoints> 
switch(config)# no grpc client root certificate my_client_trustpoint

コマンドは、gRPC エージェントのトラストポイント参照だけを削除します。トラストポイント CA 証明書は削除されません。スイッチ上の gRPC サーバーへのクライアント証明書認証を使用する接続は確立されませんが、ユーザー名とパスワードによる基本認証は通過します。


（注）  

クライアントの証明書が中間 CA によって署名されているが、上記の構成からインポートされたルート CA によって直接署名されていない場合、grpc クライアントは、ユーザー、中間 CA 証明書、およびルート CA 証明書を含む完全な証明書チェーンを提供する必要があります。

Sample DME Subscription - JSON Encoding

gnmi-console --host <iip> --port 50051 -u <user> -p <pass> --tls --
operation=Subscribe --rpc bl_paylod/once/01_subscribe_bgp_dme.json

[Subscribe]-------------------------------
### Reading from file ' bl_paylod/once/01_subscribe_bgp_dme.json '

### Generating request : 1 -----------
### Comment : ONCE request
### Delay : 2 sec(s) ...
### Delay : 2 sec(s) DONE
subscribe {
subscription {
path {
origin: "DME"
elem {
name: "sys"
}
elem {
name: "bgp"
}
}
}
mode: ONCE
use_models {
name: "DME"
organization: "Cisco Systems, Inc."
version: "1.0.0"
}
}

Received response 1 --------------------------
update {
timestamp: 1549061991079
update {
path {
elem {
name: "sys"
}
elem {
name: "bgp"
}
}
\"\" } } } ] } } ] } } ] } } ] } } }"
}
duplicates: 1093487956
}
}
/Received -------------------------------------
Received response 2 --------------------------
sync_response: true
/Received -------------------------------------
(_gnmi) [root@tm-ucs-1 gnmi-console]#

Sample DME Subscription - PROTO Encoding

gnmi-console --host >iip> --port 50051 -u <user> -p <pass> --tls --
operation=Subscribe --rpc /root/gnmi-console/testing_bl/once/61_subscribe_bgp_dme_gpb.json

[Subscribe]-------------------------------
### Reading from file ' /root/gnmi-console/testing_bl/once/61_subscribe_bgp_dme_gpb.json '
Wed Jun 26 11:49:17 2019
### Generating request : 1 -----------
### Comment : ONCE request
### Delay : 2 sec(s) ...
### Delay : 2 sec(s) DONE
subscribe {
subscription {
path {
origin: "DME"
elem {
name: "sys"
}
elem {
name: "bgp"
}
}
mode: SAMPLE
}
mode: ONCE
use_models {
name: "DME"
organization: "Cisco Systems, Inc."
version: "1.0.0"
}
encoding: PROTO
}
Wed Jun 26 11:49:19 2019
Received response 1 --------------------------
update {
timestamp: 1561574967761
prefix {
elem {
name: "sys"
}
elem {
name: "bgp"
}
}
update {
path {
elem {
}
elem {
name: "version_str"
}
}
val {
string_val: "1.0.0"
}
}
update {
path {
elem {
}
elem {
name: "node_id_str"
}
}
val {
string_val: "n9k-tm2"
}
}
update {
path {
elem {
}
elem {
name: "encoding_path"
}
}
val {
string_val: "sys/bgp"
}
}
update {
path {
elem {
}
elem {
/Received -------------------------------------
Wed Jun 26 11:49:19 2019
Received response 2 --------------------------
sync_response: true
/Received -------------------------------------
(_gnmi) [root@tm-ucs-1 gnmi-console]#

Capabilities

About Capabilities

The Capabilities RPC returns the list of capabilities of the gNMI service. The response message to the RPC request includes the gNMI service version, the versioned data models, and data encodings supported by the server.

Guidelines and Limitations for Capabilities

Following are the guidelines and limitations for Capabilities:

  • Beginning with Cisco NX-OS Release 9.3(3), Capabilities supports the OpenConfig model.

  • The gNMI feature supports Subscribe and Capability as options of the gNMI service.

  • The feature supports JSON and gnmi.proto encoding. The feature does not support protobuf.any encoding.

  • Each gNMI message has a maximum size of 12 MB. If the amount of collected data exceeds the 12-MB maximum, the collected data is dropped.

    You can avoid this situation by creating more focused subscriptions that handle smaller, more granular data-collection sets. So, instead of subscribing to one higher-level path, create multiple subscriptions for different, lower-level parts of the path.

  • All paths within the same subscription request must have the same sample interval. If the same path requires different sample intervals, create multiple subscriptions.

  • The feature does not support a path prefix in the Subscription request, but the Subscription can contain an empty prefix field.

  • The feature supports Cisco DME and Device YANG data models. Openconfig YANG is not supported.

  • The gRPC process that supports gNMI uses the HIGH_PRIO cgroup, which limits the CPU usage to 75% of CPU and memory to 1.5 GB.

  • The show grpc gnmi command has the following considerations:

    • The commands are not XMLized in this release.

    • The gRPC agent retains gNMI calls for a maximum of 1 hour after the call has ended.

    • If the total number of calls exceeds 2000, the gRPC agent purges ended calls based an internal cleanup routine.

The gRPC server runs in the management VRF. As a result, the gRPC process communicates only in this VRF forcing the management interface to support all gRPC calls.

gRPC functionality now includes the default VRF for a total of 2 gRPC servers on each switch. You can run one gRPC server in each VRF, or run only one gRPC server in the management VRF. Supporting a gRPC in the default VRF adds flexibility to offload processing gRPC calls from the management VRF, where significant traffic load might not be desirable.

If two gRPC servers are configured, be aware of the following:

  • VRF boundaries are strictly enforced, so each gRPC server processes requests independent of the other, and requests do not cross between VRFs.

  • The two servers are not HA or fault tolerant. One gRPC server does not back up the other, and there is no switchover or switchback between them.

  • Any limits for the gRPC server are per VRF.

Example Client Output for Capabilities

In this example, all the OpenConfig model RPMs have been installed on the switch.

The following is an example of client output for Capabilities. 

hostname user$ ./gnmi_cli -a 172.19.193.166:50051 -ca_crt ./grpc.pem -insecure -capabilities
supported_models: <
  name: "Cisco-NX-OS-device"
  organization: "Cisco Systems, Inc."
  version: "2019-11-13"
> 
supported_models: <
  name: "openconfig-acl"
  organization: "OpenConfig working group"
  version: "1.0.0"
> 
supported_models: <
  name: "openconfig-bgp-policy"
  organization: "OpenConfig working group"
  version: "4.0.1"
> 
supported_models: <
  name: "openconfig-interfaces"
  organization: "OpenConfig working group"
  version: "2.0.0"
> 
supported_models: <
  name: "openconfig-if-aggregate"
  organization: "OpenConfig working group"
  version: "2.0.0"
> 
supported_models: <
  name: "openconfig-if-ethernet"
  organization: "OpenConfig working group"
  version: "2.0.0"
> 
supported_models: <
  name: "openconfig-if-ip"
  organization: "OpenConfig working group"
  version: "2.3.0"
> 
supported_models: <
  name: "openconfig-if-ip-ext"
  organization: "OpenConfig working group"
  version: "2.3.0"
> 
supported_models: <
  name: "openconfig-lacp"
  organization: "OpenConfig working group"
  version: "1.0.2"
> 
supported_models: <
  name: "openconfig-lldp"
  organization: "OpenConfig working group"
  version: "0.2.1"
> 
supported_models: <
  name: "openconfig-network-instance"
  organization: "OpenConfig working group"
  version: "0.11.1"
> 
supported_models: <
  name: "openconfig-network-instance-policy"
  organization: "OpenConfig working group"
  version: "0.1.1"
> 
supported_models: <
  name: "openconfig-ospf-policy"
  organization: "OpenConfig working group"
  version: "0.1.1"
> 
supported_models: <
  name: "openconfig-platform"
  organization: "OpenConfig working group"
  version: "0.12.2"
> 
supported_models: <
  name: "openconfig-platform-cpu"
  organization: "OpenConfig working group"
  version: "0.1.1"
> 
supported_models: <
  name: "openconfig-platform-fan"
  organization: "OpenConfig working group"
  version: "0.1.1"
> 
supported_models: <
  name: "openconfig-platform-linecard"
  organization: "OpenConfig working group"
  version: "0.1.1"
> 
supported_models: <
  name: "openconfig-platform-port"
  organization: "OpenConfig working group"
  version: "0.3.2"
> 
supported_models: <
  name: "openconfig-platform-psu"
  organization: "OpenConfig working group"
  version: "0.2.1"
> 
supported_models: <
  name: "openconfig-platform-transceiver"
  organization: "OpenConfig working group"
  version: "0.7.0"
> 
supported_models: <
  name: "openconfig-relay-agent"
  organization: "OpenConfig working group"
  version: "0.1.0"
> 
supported_models: <
  name: "openconfig-routing-policy"
  organization: "OpenConfig working group"
  version: "2.0.1"
> 
supported_models: <
  name: "openconfig-spanning-tree"
  organization: "OpenConfig working group"
  version: "0.2.0"
> 
supported_models: <
  name: "openconfig-system"
  organization: "OpenConfig working group"
  version: "0.3.0"
> 
supported_models: <
  name: "openconfig-telemetry"
  organization: "OpenConfig working group"
  version: "0.5.1"
> 
supported_models: <
  name: "openconfig-vlan"
  organization: "OpenConfig working group"
  version: "3.0.2"
> 
supported_models: <
  name: "DME"
  organization: "Cisco Systems, Inc."
> 
supported_models: <
  name: "Cisco-NX-OS-Syslog-oper"
  organization: "Cisco Systems, Inc."
  version: "2019-08-15"
> 
supported_encodings: JSON
supported_encodings: PROTO
gNMI_version: "0.5.0"
 
hostname user$

Get

About Get

The purpose of the Get RPC is to allow a client to retrieve a snapshot of the data tree from the device. Multiple paths may be requested in a single request. A simplified form of XPATH according to the gNMI Path Conventions, Schema path encoding conventions for gNMI are used for the path.

For detailed information on the Get operation, refer to the Retrieving Snapshots of State Information section in the gNMI specification: gRPC Network Management Interface (gNMI)

Guidelines and Limitations for Get

The following are guidelines and limitations for Get and Set:

  • GetRequest.encoding supports only JSON.

  • For GetRequest.type, only DataType CONFIG and STATE have direct correlation and expression in YANG. OPERATIONAL is not supported.

  • A single request cannot have both OpenConfig (OC) YANG and device YANG paths. A request must have only OC YANG paths or device YANG paths, but not both.

  • GetRequest for root path (“/”: everything from all models) is not allowed.

  • GetRequest for the top level of the device model (“/System”) is not allowed.

  • gNMI Get returns all default values (ref. report-all mode in RFC 6243 [4]).

  • Subscribe supports the model Cisco-NX-OS-syslog-oper.

  • Get does not support the model Cisco-NX-OS-syslog-oper.

  • Query from the path /system does not return data from the path /system/processes. The specific path /system/processes should be used to query openconfig-procmon data.

  • The following optional items are not supported:

    • Path prefix

    • Path alias

    • Wildcards in path

  • A single GetRequest can have up to 10 paths.

  • If the size of value field to be returned in GetResponse is over 12 MB, the system returns error status grpc::RESOURCE_EXHAUSTED.

  • The maximum gRPC receive buffer size is set to 8 MB.

  • The number of total concurrent sessions for Get is limited to five.

  • Performing a Get operation when a large configuration is applied to the switch might cause the gRPC process to consume all available memory. If a memory exhaustion condition is hit, the following syslog is generated: 

    MTX-API: The memory usage is reaching the max memory resource limit (3072) MB

    If this condition is hit several times consecutively, the following syslog is generated:

    The process has become unstable and the feature should be restarted.

    We recommend that you restart the gRPC feature at this point to continue normal processing of gNMI transactions.

Set

About Set

The Set RPC is used by a client to change the configuration of the device. The operations, which may be applied to the device data, are (in order) delete, replace, and update. All operations in a single Set request are treated as a transaction, meaning that all operations are successful or the device is rolled-back to the original state. The Set operations are applied in the order that is specified in the SetRequest. If a path is mentioned multiple times, the changes are applied even if they overwrite each other. The final state of the data is achieved with the final operation in the transaction. It is assumed that all paths specified in the SetRequest::delete, replace, update fields are CONFIG data paths and writable by the client.

For detailed information on the Set operation, refer to the Modifying State section of the gNMI Specification https://github.com/openconfig/reference/blob/1cf43d2146f9ba70abb7f04f6b0f6eaa504cef05/rpc/gnmi/gnmi-specification.md.

Guidelines and Limitations for Set

The following are guidelines and limitations for Set:

  • SetRequest.encoding supports only JSON.

  • A single request cannot have both OpenConfig (OC) YANG and device YANG paths. A request must have only OC YANG paths or device YANG paths, but not both.

  • Subscribe supports the model Cisco-NX-OS-syslog-oper.

  • Query from the path /system does not return data from the path /system/processes. The specific path /system/processes should be used to query openconfig-procmon data.

  • The following optional items are not supported:

    • Path prefix

    • Path alias

    • Wildcards in path

  • A single SetRequest can have up to 20 paths.

  • The maximum gRPC receive buffer size is set to 8 MB.

  • The number of total concurrent sessions for Get is limited to five.

  • Performing a Set operation when a large configuration is applied to the switch might cause the gRPC process to consume all available memory. If a memory exhaustion condition is hit, the following syslog is generated: 

    MTX-API: The memory usage is reaching the max memory resource limit (3072) MB

    If this condition is hit several times consecutively, the following syslog is generated:

    The process has become unstable and the feature should be restarted.

    We recommend that you restart the gRPC feature at this point to continue normal processing of gNMI transactions.

  • For the Set::Delete RPC, an MTX log message warns if the configuration being operated on may be too large:

    Configuration size for this namespace exceeds operational limit. Feature may become unstable and require restart.

Subscribe

Guidelines and Limitations for Subscribe

Following are the guidelines and limitations for Subscribe:

  • Beginning with Cisco NX-OS Release 9.3(3), Subscribe supports the OpenConfig model.

  • The gNMI feature supports Subscribe and Capability as options of the gNMI service.

  • The feature supports JSON and gnmi.proto encoding. The feature does not support protobuf.any encoding.

  • Each gNMI message has a maximum size of 12 MB. If the amount of collected data exceeds the 12-MB maximum, the collected data is dropped.

    You can avoid this situation by creating more focused subscriptions that handle smaller, more granular data-collection sets. So, instead of subscribing to one higher-level path, create multiple subscriptions for different, lower-level parts of the path.

  • All paths within the same subscription request must have the same sample interval. If the same path requires different sample intervals, create multiple subscriptions.

  • The feature does not support a path prefix in the Subscription request, but the Subscription can contain an empty prefix field.

  • The feature supports Cisco DME and Device YANG data models. Openconfig YANG is not supported.

  • The gRPC process that supports gNMI uses the HIGH_PRIO cgroup, which limits the CPU usage to 75% of CPU and memory to 1.5 GB.

  • The show grpc gnmi command has the following considerations:

    • The commands are not XMLized in this release.

    • The gRPC agent retains gNMI calls for a maximum of 1 hour after the call has ended.

    • If the total number of calls exceeds 2000, the gRPC agent purges ended calls based an internal cleanup routine.

The gRPC server runs in the management VRF. As a result, the gRPC process communicates only in this VRF forcing the management interface to support all gRPC calls.

gRPC functionality now includes the default VRF for a total of 2 gRPC servers on each switch. You can run one gRPC server in each VRF, or run only one gRPC server in the management VRF. Supporting a gRPC in the default VRF adds flexibility to offload processing gRPC calls from the management VRF, where significant traffic load might not be desirable.

If two gRPC servers are configured, be aware of the following:

  • VRF boundaries are strictly enforced, so each gRPC server processes requests independent of the other, and requests do not cross between VRFs.

  • The two servers are not HA or fault tolerant. One gRPC server does not back up the other, and there is no switchover or switchback between them.

  • Any limits for the gRPC server are per VRF.

gNMI Payload

gNMI uses a specific payload format to subscribe to:

  • DME Streams

  • YANG Streams

Subscribe operations are supported with the following modes:

  • ONCE: Subscribe and receive data once and close session.

  • POLL: Subscribe and keep session open, client sends poll request each time data is needed.

  • STREAM: Subscribe and receive data at specific cadence. The payload accepts values in nanoseconds 1 second = 1000000000.

  • ON_CHANGE: Subscribe, receive a snapshot, and only receive data when something changes in the tree.

Setting modes:

  • Each mode requires 2 settings, inside sub and outside sub

  • ONCE: SAMPLE, ONCE

  • POLL: SAMPLE, POLL

  • STREAM: SAMPLE, STREAM

  • ON_CHANGE: ON_CHANGE, STREAM

Origin

  • DME: Subscribing to DME model

  • device: Subscribing to YANG model

Name

  • DME = subscribing to DME model

  • Cisco-NX-OS-device = subscribing to YANG model

Encoding

  • JSON = Stream will be send in JSON format.

  • PROTO = Stream will be sent in protobuf.any format.

Sample gNMI Payload for DME Stream


Note

Different clients have their own input format. 

{
    "SubscribeRequest":
    [
        {
            "_comment" : "ONCE request",
            "_delay" : 2,
            "subscribe":
            {
                "subscription":
                [
                    {
                        "_comment" : "1st subscription path",
                        "path":
                        {
                            "origin": "DME",
                            "elem":
                            [
                                {
                                    "name": "sys"
                                },
                                {
                                    "name": "bgp"
                                 }
                             ]
                        },
                        "mode": "SAMPLE"
                    }
                ],
                "mode": "ONCE",
                "allow_aggregation" : false,
                "use_models":
                [
                    {
                        "_comment" : "1st module",
                        "name": "DME",
                        "organization": "Cisco Systems, Inc.",
                        "version": "1.0.0"
                    }
                ],
                "encoding": "JSON"
            }
        }
    ]
}

Sample gNMI Payload YANG Stream


{
    "SubscribeRequest":
    [
        {
            "_comment" : "ONCE request",
            "_delay" : 2,
            "subscribe":
            {
                "subscription":
                [
                    {
                        "_comment" : "1st subscription path",
                        "path":
                        {
                            "origin": "device",
                            "elem":
                            [
                                {
                                    "name": "System"
                                 },
                                 {
                                     "name": "bgp-items"
                                  }
                              ]
                          },
                                                  "mode": "SAMPLE"
                      }
                  ],
                  "mode": "ONCE",
                  "allow_aggregation" : false,
                  "use_models":
                  [
                      {
                          "_comment" : "1st module",
                          "name": "Cisco-NX-OS-device",
                          "organization": "Cisco Systems, Inc.",
                          "version": "0.0.0"
                      }
                  ],
                  "encoding": "JSON"
              }
          }
      ]
}

Streaming Syslog

About Streaming Syslog for gNMI

gRPC Network Management Interface (gNMI) defines a gRPC-based protocol for the modification and retrieval of configuration from a target device, Also, it defines the control and generation of telemetry streams from a target device to a data collection system. The intention is that a single gRPC service definition can cover both configuration and telemetry - allowing a single implementation on the target, as well as a single NMS element to interact with the device via telemetry and configuration RPCs.

gNMI Subscribe is a new way of monitoring the network as it provides a real-time view of what's going on in your system by pushing the structured data as per gNMI Subscribe request.

Beginning with the Cisco NX-OS Release 9.3(3), support is added for gNMI Subscribe functionality.

gNMI Subscribe Support Detail

  • Syslog-oper model streaming

    • stream_on_change

Telemetry Support Detail

  • YANG model

    • Events only

This feature applies to Cisco Nexus 3500 platform switches with 8 GB or more of memory.

Guidelines and Limitations for Streaming Syslog - gNMI

The following are guidelines and limitations for Streaming Syslog:

  • An invalid syslog is not supported. For example, a syslog with a filter or query condition

  • Only the following paths are supported:

    • Cisco-NX-OS-Syslog-oper:syslog

    • Cisco-NX-OS-Syslog-oper:syslog/messages

  • The following modes are not supported:

    • Stream sample

    • POLL

  • A request must be in the YANG model format.

  • You can use the internal application or write your own application.

  • The payload comes from the controller and gNMI sends a response.

  • Encoding formats are JSON and PROTO.

Syslog Native YANG Model

The YangModels are located here.


Note

The time-zone field is set only when the clock format show-timezone syslog is entered. By default, it's not set, therefore the time-zone field is empty. 

PYANG Tree for Syslog Native Yang Model:
>>> pyang -f tree Cisco-NX-OS-infra-syslog-oper.yang
module: Cisco-NX-OS-syslog-oper
+--ro syslog
+--ro messages
+--ro message* [message-id]
+--ro message-id int32
+--ro node-name? string
+--ro time-stamp? uint64
+--ro time-of-day? string
+--ro time-zone? string
+--ro category? string
+--ro group? string
+--ro message-name? string
+--ro severity? System-message-severity
+--ro text? string

Subscribe Request Example

The following is an example of a Subscribe request:

{
    "SubscribeRequest":
    [
        {
            "_comment" : "STREAM request",
            "_delay"   : 2,
            "subscribe":
            {
                "subscription":
                [
                    {
                        "_comment" : "1st subscription path",
                        "path":
                        {
                            "origin": "syslog-oper",
                            "elem":
                              [
                                {
                                    "name": "syslog"
                                },
                                {
                                    "name":"messages"
                                }
                            ]
                        },
                        "mode": "ON_CHANGE"
                    }
                ],
                "mode": "ON_CHANGE",
             "allow_aggregation" : false,
                "use_models":
                [
                    {
                        "_comment" : "1st module",
                        "name": "Cisco-NX-OS-Syslog-oper",
                        "organization": "Cisco Systems, Inc.",
                        "version": "0.0.0"
                    }
                ],
                "encoding":"JSON"
            }
        }
    ]
}

Sample PROTO Output

This is a sample of PROTO output. 

############################

[Subscribe]-------------------------------

### Reading from file ' /root/gnmi-console/testing_bl/stream_on_change/OC_SYSLOG.json '

Sat Aug 24 14:38:06 2019

### Generating request : 1 -----------

### Comment : STREAM request

### Delay : 2 sec(s) ...

### Delay : 2 sec(s) DONE

subscribe {

subscription {

path {

origin: "syslog-oper"

elem {

name: "syslog"

}

elem {

name: "messages"

}

}

mode: ON_CHANGE

}

use_models {

name: "Cisco-NX-OS-Syslog-oper"

organization: "Cisco Systems, Inc."

version: "0.0.0"

}

encoding: PROTO

}

Thu Nov 21 14:26:41 2019
Received response 3 --------------------------
update {
timestamp: 1574375201665688000
prefix {
origin: "Syslog-oper"
elem {
name: "syslog"
}
elem {
name: "messages"
}
}
update {
path {
elem {
name: "message-id"
}
}
val {
uint_val: 529
}
}
update {
path {
elem {
name: "node-name"
}
}
val {
string_val: "task-n9k-1"
}
}
update {
path {
elem {
name: "message-name"
}
}
val {
string_val: "VSHD_SYSLOG_CONFIG_I"
}
}
update {
path {
elem {
name: "text"
}
}
val {
string_val: "Configured from vty by admin on console0"
}
}
update {
path {
elem {
name: "group"
}
}
val {
string_val: "VSHD"
}
}
update {
path {
elem {
name: "category"
}
}
val {
string_val: "VSHD"
}
}
update {
path {
elem {
name: "time-of-day"
}
}
val {
string_val: "Nov 21 2019 14:26:40"
}
}
update {
path {
elem {
name: "time-zone"
}
}
val {
string_val: ""
}
}
update {
path {
elem {
name: "time-stamp"
}
}
val {
uint_val: 1574375200000
}
}
update {
path {
elem {
name: "severity"
}
}
val {
uint_val: 5
}
}
}

/Received -------------------------------------

Sample JSON Output

This is a sample JSON output.

[Subscribe]-------------------------------
### Reading from file ' testing_bl/stream_on_change/OC_SYSLOG.json '


Tue Nov 26 11:47:00 2019
### Generating request : 1 -----------
### Comment : STREAM request
### Delay : 2 sec(s) ...
### Delay : 2 sec(s) DONE
subscribe {
subscription {
path {
origin: "syslog-oper"
elem {
name: "syslog"
}
elem {
name: "messages"
}
}
mode: ON_CHANGE
}
use_models {
name: "Cisco-NX-OS-Syslog-oper"
organization: "Cisco Systems, Inc."
version: "0.0.0"
}
}

Tue Nov 26 11:47:15 2019
Received response 5 --------------------------
update {
timestamp: 1574797636002053000
prefix {
}
update {
path {
origin: "Syslog-oper"
elem {
name: "syslog"
}
}
val {
json_val: "[ { \"messages\" : [[ {\"message-id\":657},{\"node-name\":\"task-n9k-1\",\"time-stamp\":\"1574797635000\",\"time-of-day\":\"Nov 26 2019 11:47:15\",\"severity\":3,\"message-name\":\"HDR_L2LEN_ERR\",\"category\":\"ARP\",\"group\":\"ARP\",\"text\":\"arp [30318] Received packet with incorrect layer 2 address length (8 bytes), Normal pkt with S/D MAC: 003a.7d21.d55e ffff.ffff.ffff eff_ifc mgmt0(9), log_ifc mgmt0(9), phy_ifc mgmt0(9)\",\"time-zone\":\"\"} ]] } ]"
}
}
}

/Received -------------------------------------

Troubleshooting

Gathering TM-Trace Logs

1. tmtrace.bin -f gnmi-logs gnmi-events gnmi-errors following are available
2. Usage:

bash-4.3# tmtrace.bin -d gnmi-events | tail -30 Gives the last 30
}
}
}
[06/21/19 15:58:38.969 PDT f8f 3133] [3981658944][tm_transport_internal.c:43] dn: Cisco-NX-OS-device:System/cdp-items, sub_id: 0,
sub_id_str: 2329, dc_start_time: 0, length: 124, sync_response:1
[06/21/19 15:58:43.210 PDT f90 3133] [3621780288][tm_ec_yang_data_processor.c:93] TM_EC: [Y] Data received for 2799743488: 49
{
"cdp-items" : {
"inst-items" : {
"if-items" : {
"If-list" : [
{
"id" : "mgmt0",
"ifstats-items" : {
"v2Sent" : "74",
"validV2Rcvd" : "79"
}
}
]
}
}
}
}
[06/21/19 15:58:43.210 PDT f91 3133] [3981658944][tm_transport_internal.c:43] dn: Cisco-NX-OS-device:System/cdp-items, sub_id: 0,
sub_id_str: 2329, dc_start_time: 0, length: 141, sync_response:1
[06/21/19 15:59:01.341 PDT f92 3133] [3981658944][tm_transport_internal.c:43] dn: Cisco-NX-OS-device:System/intf-items, sub_id:
4091, sub_id_str: , dc_start_time: 1561157935518, length: 3063619, sync_response:0
[06/21/19 15:59:03.933 PDT f93 3133] [3981658944][tm_transport_internal.c:43] dn: Cisco-NX-OS-device:System/cdp-items, sub_id:
4091, sub_id_str: , dc_start_time: 1561157940881, length: 6756, sync_response:0
[06/21/19 15:59:03.940 PDT f94 3133] [3981658944][tm_transport_internal.c:43] dn: Cisco-NX-OS-device:System/lldp-items, sub_id:
4091, sub_id_str: , dc_start_time: 1561157940912, length: 8466, sync_response:1
bash-4.3#

Gathering MTX-Internal Logs

1. Modify the following file with below /opt/mtx/conf/mtxlogger.cfg

<config name="nxos-device-mgmt">
  <container name="mgmtConf">
    <container name="logging">
      <leaf name="enabled" type="boolean" default="false">true</leaf>
      <leaf name="allActive" type="boolean" default="false">true<
/leaf>
      <container name="format">
        <leaf name="content" type="string" default="$DATETIME$
$COMPONENTID$ $TYPE$: $MSG$">$DATETIME$ $COMPONENTID$ $TYPE$
$SRCFILE$ @ $SRCLINE$ $FCNINFO$:$MSG$</leaf>
            <container name="componentID">
          <leaf name="enabled" type="boolean" default="true"></leaf>
            </container>
            <container name="dateTime">
          <leaf name="enabled" type="boolean" default="true"></leaf>
          <leaf name="format" type="string" default="%y%m%d.%H%M%S"><
/leaf>
             </container>
             <container name="fcn">
           <leaf name="enabled" type="boolean" default="true"></leaf>        
           <leaf name="format" type="string"
default="$CLASS$::$FCNNAME$($ARGS$)@$LINE$"></leaf>
             </container>
      </container>
      <container name="facility">
          <leaf name="info" type="boolean" default="true">true</leaf>
          <leaf name="warning" type="boolean" default="true">true<
/leaf>
          <leaf name="error" type="boolean" default="true">true</leaf>
          <leaf name="debug" type="boolean" default="false">true<
/leaf>
        </container>
        <container name="dest">
          <container name="console">
            <leaf name="enabled" type="boolean" default="false">true<
/leaf>
          </container>
          <container name="file">
         <leaf name="enabled" type="boolean" default="false">true<
/leaf>
    <leaf name="name" type="string" default="mtx-internal.log"><
/leaf>
                                                                                                                                                                                            <leaf name="location" type="string" default="./mtxlogs">
/volatile</leaf>
              <leaf name="mbytes-rollover" type="uint32" default="10"
>50</leaf>
              <leaf name="hours-rollover" type="uint32" default="24"
>24</leaf>
              <leaf name="startup-rollover" type="boolean" default="
false">true</leaf>
            <leaf name="max-rollover-files" type="uint32" default="10"
>10</leaf>
        </container>
      </container>
      <list name="logitems" key="id">
        <listitem>
                <leaf name="id" type="string">*</leaf>
                    <leaf name="active" type="boolean" default="false"
>false</leaf>
         </listitem>
          <listitem>
                 <leaf name="id" type="string">MTX-EvtMgr</leaf>
                     <leaf name="active" type="boolean" default="true"
>true</leaf>
        </listitem>
        <listitem>
               <leaf name="id" type="string">TM-ADPT</leaf>
                   <leaf name="active" type="boolean" default="true"
>false</leaf>
        </listitem>
        <listitem>
              <leaf name="id" type="string">TM-ADPT-JSON</leaf>
                  <leaf name="active" type="boolean" default="true"
>false</leaf> 
        </listitem  >
        <listitem>
                <leaf name="id" type="string">SYSTEM</leaf>
                    <leaf name="active" type="boolean" default="true"
>true</leaf>
        </listitem>
        <listitem>
                <leaf name="id" type="string">LIBUTILS</leaf>
                     <leaf name="active" type="boolean" default="true"
>true</leaf>
        </listitem>
        <listitem>
                <leaf name="id" type="string">MTX-API</leaf>
                    <leaf name="active" type="boolean" default="true"
>true</leaf>
        </listitem>
         <listitem>
                 <leaf name="id" type="string">Model-*</leaf>
                     <leaf name="active" type="boolean" default="true"
>true</leaf>
        </listitem>
        <listitem>
                <leaf name="id" type="string">Model-Cisco-NX-OS-
device</leaf>
                 <leaf name="active" type="boolean" default="true"
>false</leaf>         
        </listitem>
        <listitem>
                 <leaf name="id" type="string">Model-openconfig-bgp<
/leaf>
                     <leaf name="active" type="boolean" default="true"
>false</leaf>
        </listitem>
        <listitem>
               <leaf name="id" type="string">INST-MTX-API</leaf>
                   <leaf name="active" type="boolean" default="true"
>true</leaf>
        </listitem>
        <listitem>
                <leaf name="id" type="string">INST-ADAPTER-NC</leaf>
                    <leaf name="active" type="boolean" default="true"
>true</leaf>
        </listitem>
        <listitem>
               <leaf name="id" type="string">INST-ADAPTER-RC</leaf>
                   <leaf name="active" type="boolean" default="true"
>true</leaf>
        </listitem>
        <listitem>
               <leaf name="id" type="string">INST-ADAPTER-GRPC</leaf>
                   <leaf name="active" type="boolean" default="true"
>true</leaf>
         </listitem>
      </list>
    </container>
  </container>
</config>

2. Run "no feature grpc" / "feature grpc"
3. The /volataile directory houses the mtx-internal.log, the log rolls over over time so be sure to grab what  you need before thenbash-4.3# cd /volatile/

bash-4.3# cd /volaiflels -al
total 148
drwxrwxrwx 4 root root 340 Jun 21 15:47 .
drwxrwxr-t 64 root network-admin 1600 Jun 21 14:45 ..
-rw-rw-rw- 1 root root 103412 Jun 21 16:14 grpc-internal-log
-rw-r--r-- 1 root root 24 Jun 21 14:44 mtx-internal-19-06-21-14-46-21.log
-rw-r--r-- 1 root root 24 Jun 21 14:46 mtx-internal-19-06-21-14-46-46.log
-rw-r--r-- 1 root root 175 Jun 21 15:11 mtx-internal-19-06-21-15-11-57.log
-rw-r--r-- 1 root root 175 Jun 21 15:12 mtx-internal-19-06-21-15-12-28.log
-rw-r--r-- 1 root root 175 Jun 21 15:13 mtx-internal-19-06-21-15-13-17.log
-rw-r--r-- 1 root root 175 Jun 21 15:13 mtx-internal-19-06-21-15-13-42.log
-rw-r--r-- 1 root root 24 Jun 21 15:13 mtx-internal-19-06-21-15-14-22.log
-rw-r--r-- 1 root root 24 Jun 21 15:14 mtx-internal-19-06-21-15-19-05.log
-rw-r--r-- 1 root root 24 Jun 21 15:19 mtx-internal-19-06-21-15-47-09.log
-rw-r--r-- 1 root root 24 Jun 21 15:47 mtx-internal.log
-rw-rw-rw- 1 root root 355 Jun 21 14:44 netconf-internal-log
-rw-rw-rw- 1 root root 0 Jun 21 14:45 nginx_logflag
drwxrwxrwx 3 root root 60 Jun 21 14:45 uwsgipy
drwxrwxrwx 2 root root 40 Jun 21 14:43 virtual-instance
bash-4.3#.