Log in to your Multi-Site Orchestrator GUI.

Open the System Logs screen.

1. In the main menu, select Operations > Tech Support.

2. In the top right corner of the System Logs frame, click the edit button.

Download the logs.

1. Select which logs you want to download.

2. Click the Download button.

   An archive of the selected items will be downloaded to your system. The report contains the following information:

   • All schemas in JSON format

   • All sites definitions in JSON format

   • All tenants definitions in JSON format

   • All users definitions in JSON format

   • All logs of the containers in the infra_logs.txt file

Check if there are any stale containers in the system.

# docker ps -a
CONTAINER ID    IMAGE                              COMMAND                  CREATED         STATUS
5bd87a6a6813    [...]/msc-kong:3.0.1i              "/docker-entrypoint.…"   23 hours ago    Up 23 hours (healthy)
bf73df31ff51    [...]/msc-ui:3.0.1i                "/nginx.sh"              25 hours ago    Up 25 hours (healthy)
77c96b515e63    [...]/msc-ui:3.0.1i                "/nginx.sh"              25 hours ago    Exited (1) 25 hours ago
dfedfd82233a    [...]/msc-ui:3.0.1i                "/nginx.sh"              25 hours ago    Exited (1) 25 hours ago
d70aa6262396    [...]/msc-kong:3.0.1i              "/docker-entrypoint.…"   25 hours ago    Exited (143) 23 hours ago
7c15db4db6eb    [...]/msc-endpointservice:3.0.1i   "python3 main.py"        25 hours ago    Up 25 hours (healthy)

Collect system logs using the Orchestrator GUI.

Log in to one of the nodes and execute the following commands to remove stale containers.

1. Ensure that the exited containers are stopped.

   # docker ps --filter "status=exited" --format '{{.ID}}' | xargs --no-run-if-empty docker container stop

2. Remove the containers.

   # docker ps --filter "status=exited" --format '{{.ID}}' | xargs --no-run-if-empty docker container rm -f
   # docker container prune -f

Repeat the previous step for the other two nodes.

Log in to one of the Multi-Site Orchestrator nodes via SSH.

Check if the MongoDB containers are properly replicated on all nodes.

# docker service ls
ID                  NAME              MODE            REPLICAS    IMAGE
[...]
jvztl0waek4c        msc_mongodb1      replicated      1/1         msc-mongo:3.6
x1tkpwf1q1df        msc_mongodb2      replicated      1/1         msc-mongo:3.6
zbi376btmjbg        msc_mongodb3      replicated      0/1         msc-mongo:3.6
[...]

Find the hostnames of all the nodes.

# docker node ls
ID                            HOSTNAME    STATUS      AVAILABILITY    MANAGER STATUS  ENGINE VERSION
z3b6sqc38gfgoerte8cx1w17r     node1       Ready       Active          Reachable       18.06.1-ce
mb3hqe1g0r55oa2zoe32yyfiw *   node2       Ready       Active          Leader          18.06.1-ce
ur5vq2gli8zfc8ngafjn8plej     node3       Ready       Active          Reachable       18.06.1-ce

Inspect each node.

# docker inspect <node-name>

# docker inspect node3
[
    {
        "ID": "ur5vq2gli8zfc8ngafjn8plej",
        "Version": {
            "Index": 317093
        },
        "CreatedAt": "2018-01-19T11:00:41.522951756Z",
        "UpdatedAt": "2019-03-17T07:38:35.487509349Z",
        "Spec": {
            "Labels": {},
            "Role": "manager",
            "Availability": "active"
        },
[...]

Restore the missing label for the node.

# docker node update --label-add "msc-node=<node-label>" <node-name>

# docker node update --label-add "msc-node=msc-node3" node3

Verify that the label was added correctly.

# docker inspect node3
[
    {
        "ID": "ur5vq2gli8zfc8ngafjn8plej",
        "Version": {
            "Index": 317093
        },
        "CreatedAt": "2018-01-19T11:00:41.522951756Z",
        "UpdatedAt": "2019-03-17T07:38:35.487509349Z",
        "Spec": {
            "Labels": {
                "msc-node": "msc-node3"
            },
            "Role": "manager",
            "Availability": "active"
        },
[...]

On the site5 ingress leaf switch (leaf101 in this case), use the NX-OS style CLI show endpoint mac mac-address command to determine whether the system has learned both the source and destination EP, as in the following example:

leaf101# show endpoint mac 0000.6666.1716
Legend:
s - arp              O - peer-attached    a - local-aged       S - static
V - vpc-attached     p - peer-aged        M - span             L - local
B - bounce           H - vtep
+-----------------+---------------+-----------------+--------------+-------------+
      VLAN/         Encap           MAC Address       MAC Info/       Interface
      Domain        VLAN            IP Address        IP Info
+-----------------+---------------+-----------------+--------------+-------------+
3                    vlan-201       0000.6666.1716 L                 eth1/40
msite-site:vrf1      vlan-201       172.17.15.106 L                  eth1/40

If both the local and remote EPs have been learned, and the policy permits the EPs to communicate (using the default contract permitting intra-EPG traffic), site5-leaf101 encapsulates the ICMP packet with the following data, and forwards the packet out through the fabric uplink port towards the spine switch:

If there is a concern about the packet being forwarded, run ERSPAN in fabric mode on the APIC, in the NX-OS style CLI, to capture the outgoing packet from the uplink interface, using commands such as in the following example.

apic1# configure terminal 
apic1(config)# monitor access session mySession
apic1(config-monitor-fabric)# description "This is my fabric ERSPAN session"
apic1(config-monitor-fabric)# destination tenant t1 application app1 epg epg1 destination-ip 192.0.20.123 source-ip-prefix 10.0.20.1
apic1(config-monitor-fabric-dest)# erspan-id 100
apic1(config-monitor-fabric-dest)# ip dscp 42
apic1(config-monitor-fabric-dest)# ip ttl 16
apic1(config-monitor-fabric-dest)# mtu 9216
apic1(config-monitor-fabric-dest)# exit
apic1(config-monitor-fabric)# source interface eth 1/1 switch 101
apic1(config-monitor-fabric-source)# direction tx
apic1(config-monitor-fabric-source)# filter tenant t1 bd bd1
apic1(config-monitor-fabric-source)# filter tenant t1 vrf vrf1
apic1(config-monitor-fabric-source)# exit
apic1(config-monitor-fabric)# no shut

To verify whether the routing table contains an explicit entry for 6.6.6.1, use the NX-OS style command, show ip route 6.6.6.1 vrf overlay-1.

When the next-hop interface has been found, use the show lldp neighbor command to determine whether the next hop of the interface , where 6.6.6.1 is learned from, is the expected IPN interface.

When the packet arrives at IPN, because it is a unicast packet, IPN forwards the IP packet based on the routing table. To confirm the routing table has the correct/expected next-hop interface, use the command show ip route 6.6.6.1.

To verify the VNID translation bewteen site5 and site6, on spine601 enter the show dcimgr repo vnid-maps verbose command.

To verify the sclass translation between site5 and site6, on spine601 enter the show dcimgr repo sclass-maps command.

To determine if the packet was forwarded properly, go to the expected leaf switch (site6-leaf101) run Fabric ERSPAN to capture the packet.

If the packet arrives at site6-leaf101, leaf101 performs a local lookup, based on the destination MAC within the scope of VNID, 16449430, to determine the egress interface. To determine the egress interface, enter the show endpoint mac mac-address command.

To determine if the packet was forwarded properly, use access SPAN to capture the outgoing packet on the expected interface, using commands such as in the following example.

apic1# configure terminal 
apic1(config)# monitor access session mySession
apic1(config-monitor-access)# description "This is my SPAN session"
apic1(config-monitor-access)# destination interface eth 1/2 leaf 101
apic1(config-monitor-access)# source interface eth 1/1 leaf 101
apic1(config-monitor-access-source)# direction tx
apic1(config-monitor-access-source)# filter tenant t1 application app1 epg epg1
apic1(config-monitor-access-source)# exit
apic1(config-monitor-access)# no shut
apic1(config-monitor-access)# show run

In a supported browser, enter the URL of the spine switch followed by /visore.html, as in the following example:

When prompted, log in using the same credentials you would use to log in to the spine CLI interface.

Enter a query for l3LbRtdIf to verify that the type is inter-site.

Enter a query for Ipv4IF to verify that the mode is cp-etep and the modeExtn is mscp-etep.

Enter a query for Intersite BgpPeers to verify it was created with the inter-site type and uses the CP-TEP loopback address as the source interface.

If any of these values is incorrect, go to the APIC for the site and correct the values. Return to the Sites tab in Multi-Site and click CONFIGURE INFRA, and then click Apply.

Log in to the Cisco ACI Multi-Site Orchestrator GUI.

Refresh site connectivity information.

1. In the Main menu, select Infrastructure > Infra Configuration.

2. In the top right of the main Infra Configuration view, click the Configure Infra button.

3. In the left pane, under Sites, select the site with the recommissioned spine switch.

4. In the main window, click the Reload Site Data button to pull fabric information from the APIC.

5. In the Confirmation dialog, ensure that you check the Remove config for decommissioned spine nodes checkbox.

   When you select the checkbox, the old configuration information for the decommissioned spine switch will be removed from the Multi-Site Orchestrator database.

6. Finally, click Yes to confirm and load the connectivity information.

   This will update the site connectivity information including the recommissioned switch by re-importing it from the APIC.

Verify spine switches configuration.

In the top right of the main Fabric Connectivity Infra view, click the Deploy button.

Browse to the spine switch's Visore page.

https://<spine-ip-address>/visore.html

Log in using the same credentials you would use to log in to the spine CLI interface.

Enter a query to verify the fvIntersiteConnPDef and fvIntersiteMcastConnPDef MOs are under fvSiteConnPDef.

Enter a query to verify that the tunnelIf MO was created with the type dci-ucast or dci-mcast-hrep, and the destination is the same as the remote site DP TEPs.

Check the local site unicast and multicast DP TEPs. Enter a query for fvIntersiteConnPDef under fvPodConnPDef and fvIntersiteMcastConnPDef under fvFabricExtConnPDef.

Enter a query for the SiteLocal ipv4If MOs to verify they were created with the mode dci-ucast and dci-mcast-hrep and the ipv4Addr MO was configured under it with the same address as the DP TEP's.

If any of these values is incorrect, go to the APIC for the site and correct the values. Return to the Sites tab in Multi-Site and click CONFIGURE INFRA, and then click Apply.

To verify the locally configured Multi-Site TEP IP addresses, log on to the Supervisor module, and enter a command such as the following example:

swmp11-spine6# show ip interface vrf overlay-1 
loopback11, Interface status: protocol-up/link-up/admin-up, iod: 126, mode: dci-ucast, vrf_vnid: 16777199
  IP address: 33.20.1.1, IP subnet: 33.20.1.1/32
  IP primary address route-preference: 1, tag: 0
loopback12, Interface status: protocol-up/link-up/admin-up, iod: 127, mode: mcast-hrep, vrf_vnid: 16777199
  IP address: 33.30.1.1, IP subnet: 33.30.1.1/32

To confirm the MFDM on the spine switch, log on to the Supervisor module and enter a command such as the following example:

swmp11-spine6# show forwarding distribution multicast hrep
MFDM HREP NODE TABLE
--------------------
IP Address: 0xb1e0101
Table Id: 2
Flags: 0x0
IfIndex: 0x18010009
Internal BD 0x1001
Internal encap 0xb54
Nexthop Information: (num: 5)
Address               Ifindex                   Dvif
0x14950a02        0x1a018019        0x1eb (Selected)  <== Selected NH to reach the HREP TEP IP
0x14950602        0x1a00e00f        0x0
0x14950802        0x1a010011        0x0
0x14950902        0x1a011012        0x0
0x14950b02        0x1a01901a        0x0

To verify HREP TEP IP address reachability, log on to the Supervisor module, and enter a command such as the following example:

swmp11-spine6# show ip route 11.30.1.1 vrf overlay-1
11.30.1.1/32, ubest/mbest: 5/0
    *via 20.149.6.2, Eth1/15.15, [110/9], 1d21h, ospf-default, intra
    *via 20.149.8.2, Eth1/17.17, [110/9], 1d21h, ospf-default, intra
    *via 20.149.9.2, Eth1/18.18, [110/9], 1d21h, ospf-default, intra
    *via 20.149.10.2, Eth1/25.25, [110/9], 1d21h, ospf-default, intra
    *via 20.149.11.2, Eth1/26.26, [110/9], 1d21h, ospf-default, intra
     via 10.0.112.95, Eth2/21.77, [115/65], 1d21h, isis-isis_infra, L1
     via 10.0.112.95, Eth1/24.35, [115/65], 1d21h, isis-isis_infra, L1
     via 10.0.112.92, Eth2/19.76, [115/65], 1d21h, isis-isis_infra, L1
     via 10.0.112.92, Eth1/21.36, [115/65], 1d21h, isis-isis_infra, L1
     via 10.0.112.90, Eth2/17.75, [115/65], 1d21h, isis-isis_infra, L1
     via 10.0.112.90, Eth1/23.33, [115/65], 1d21h, isis-isis_infra, L1

To verify the MFIB on the spine switch line card module, log on to the module as root and use an (vsh_lc) command such as the following example:

root@module-1# show forwarding multicast hrep tep_routes

****HREP TEP ROUTES****
------------------------------------------------------------------------------------------------------------
| Tep Ip    |   Tep If    |    NH Ip     |       NH If     |     NH dmac    |    NH dvif | Vlan Id | Bd Id |
------------------------------------------------------------------------------------------------------------
|22.30.1.1    |0x1801000b   |20.149.11.2    |0x1a01901a     |00c8.8bba.54bc |490  |2901   |4098   |
|11.30.1.1    |0x18010009   |20.149.10.2  |0x1a018019   |00c8.8bba.54bc |491  |2900   |4097   |

To verify the multicast HREP TEP details for remote sites, log on to the spine switch line card module to investigate the SDK, by entering a command such as the following example:

root@module-1# show platform internal hal objects mcast hreptep
## Get Objects for mcast hreptep for Asic 0
  OBJECT 1:
Handle                                                    : 52303
tepifindex                                                : 0x18010009
tepipaddr                                                 : 11.30.1.1/0
intbdid                                                   : 0x1001
intvlanid                                                 : 0xb54
nexthopipaddr                                             : 20.149.10.2/0
nexthopifindex                                            : 0x1a018019
nexthopmacaddr                                            : 00:c8:8b:ba:54:bc

To verify the GIPo route having HREP tunnels for the remote sites, log on to the spine switch Supervisor module and examine IS-IS details on the spine switch with a command, such as the following example:

swmp11-spine6# show isis internal mcast routes gipo
GIPo: 225.0.6.176 [TRANSIT]
    OIF List:
      Ethernet1/21.36
      Ethernet1/23.33
      Ethernet1/24.35
      Tunnel9         <== Multicast HREP tunnel for Remote Site 1
      Tunnel1         <== Multicast HREP tunnel for Remote Site 2
      Ethernet2/17.75
      Ethernet2/19.76
      Ethernet2/21.77

To verify the GIPo route having HREP tunnels for the remote sites, examine the MRIB on the spine switches using a command, such as the following example:

swmp11-spine6# show ip mroute 225.0.6.176 vrf overlay-1
IP Multicast Routing Table for VRF "overlay-1"
(*, 225.0.6.176/32), uptime: 1d02h, isis
  Incoming interface: Null, RPF nbr: 0.0.0.0
  Outgoing interface list: (count: 8)
    Tunnel9, uptime: 1d01h
    Tunnel11, uptime: 1d02h
    Ethernet2/21.77, uptime: 1d02h
    Ethernet2/19.76, uptime: 1d02h
    Ethernet2/17.75, uptime: 1d02h
    Ethernet1/24.35, uptime: 1d02h
    Ethernet1/23.33, uptime: 1d02h
    Ethernet1/21.36, uptime: 1d02h

To verify the MFIB on an FC, log on to the module as root, and use a command such as the following example:

root@module-24# show forwarding multicast route group 225.0.6.176 vrf all
  (*, 225.0.6.176/32), RPF Interface: NULL, flags: Dc
    Received Packets: 0 Bytes: 0
    Number of Outgoing Interfaces: 8
    Outgoing Interface List Index: 484
      Ethernet1/21.36 Outgoing Packets:N/A Bytes:N/A
      Ethernet1/23.33 Outgoing Packets:N/A Bytes:N/A
      Ethernet1/24.35 Outgoing Packets:N/A Bytes:N/A
      Tunnel9 Outgoing Packets:0 Bytes:0
      Tunnel11 Outgoing Packets:0 Bytes:0
      Ethernet2/17.75 Outgoing Packets:N/A Bytes:N/A
      Ethernet2/19.76 Outgoing Packets:N/A Bytes:N/A
      Ethernet2/21.77 Outgoing Packets:N/A Bytes:N/A

To verify the GIPo route having HREP tunnels for the remote sites, examine the SDL on FC, using a command such as the following example:

root@module-24# show platform internal hal objects mcast l3mcastroute groupaddr 225.0.6.176/32 extensions
## Get Extended Objects for mcast l3mcastroute for Asic 0
  OBJECT 0:
Handle                                                    : 78705
groupaddr                                                 : 225.0.6.176/32
grpprefixlen                                              : 0x20
sourceaddr                                                : 0.0.0.0/32
ispimbidir                                                : Enabled
ctrlflags                                                 : UseMetFlag,
rtflags                                                   : none, UseMetEntry,
acirtpolicy                                               : none
 - - - - - - - - 
Relation Object repllistnextobj :
  rel-repllistnextobj-mcast-mcast_mcast_repl_list-handle  : 78702
  rel-repllistnextobj-mcast-mcast_mcast_repl_list-id      : 0x600001e4

To verify the GIPo route to the remote sites, examine the replication list identified in the last step, using a command such as the following example:

root@module-24# show platform internal hal objects mcast mcastrepllist id 0x600001e4
## Get Objects for mcast mcastrepllist for Asic 0
- - - - - - - - - - 
Repl-List Asicpd Debug :
Entry-Num 0
Repl Entry Id:      0x1e5 Hw Epg Id:    4050 Hw Bd Id:   4050
Mc Id:                484 Met Id:        485 Encap Id:     -1
Sh Grp:                 0 Next Met Id:   749
Entry-Num 1
Repl Entry Id:      0x2ed Hw Epg Id:    4098 Hw Bd Id:   4098
Mc Id:                490 Met Id:        749 Encap Id:     -1
Sh Grp:                 0 Next Met Id:  1191
Entry-Num 2
Repl Entry Id:      0x3a4 Hw Epg Id:    4097 Hw Bd Id:   4097
Mc Id:                491 Met Id:       1191 Encap Id:     -1
Sh Grp:                 0 Next Met Id:     0

To verify the VNID and GIPo mappings on the local (TX) and remote (RX) sites, enter a command such as the following example:

root@module-2# show platform internal hal objects dci vnidmap extensions | grep -B 5 -A 5 225.1.148.0

  OBJECT 182:
Handle                                                    : 26456
isbdvnid                                                  : Enabled
localvnid                                                 : 0xe78007
localgipo                                                 : 225.1.148.0/32
remotevnid                                                : 0xe1000c
remotevrfvnid                                             : 0x208019
islocalbdctrl                                             : Enabled
siteid                                                    : 0x3

 OBJECT 1285:
Handle                                                    : 29468
isbdvnid                                                  : Enabled
localvnid                                                 : 0xe78007
localgipo                                                 : 225.1.148.0/32
remotevnid                                                : 0xee7fa8
remotevrfvnid                                             : 0x2e000e
islocalbdctrl                                             : Enabled
siteid                                                    : 0x2