How to Collect Logs from VM Bootup

Introduction

This document describes how to collect logs when a Virtual Machine (VM) of Virtualized Packet Core (VPC) in Cisco Ultra Services Platform (Ultra M) boots up and points to multiple boot ups.

Contributed by Dennis Lanov, Cisco TAC Engineer.

VM Bootup

In order to boot a VM with Control Function (CF) or Service Function (SF), you can include multiple steps and checks that are covered here. When you monitor a VM, it has to be via Serial1 because that includes all debug logs. 

Identify Instance of VM to Monitor.

Option 1. Via GUI log in to Dashboard.

Navigate to Admin > Instances, find instance that for example, _c1  and find compute host, in the example here, C1 is on compute 1 and C2 is on compute 2.

Navigate to the console side and check QEMU instance as shown in this image.

Option 2. Search for each instance from the 'virsh list' to find out instance name.

source from undercloud: source stackrc

identify compute node's control IP: nova list

SSH to control plane of the compute node with heat-admin: ssh heat-admin@<IP address>.

Change to root: sudo su

List all instances: virsh list

Console to serial 1 of the instance: virsh console instance-<number> serial1

Logs here contain multiple major items at the time of boot up of CF in Slot 1. SF has very similar boot up process.

This card was restarted manually:

[  811.235666] Restarting system.

[  811.235950] machine restart

Identify card type:

platform_get_card_info CARDTYPE Read in 0x40010100 --> 0x40010100

Read disk and system parameters:

"QEMU HARDDISK"

 

Reading Boot priority. Notice ~7 seconds. If you see more than 30 seconds, then it has issues in order to fetch the image. Possible issues: image issue etc.

Indication: that compute does not compute access to the location where the image located. Septh or Cinder.

Booting priority 1

   image : /flash/qvpc-vchitlur.bin

   config: /flash/day-N.cfg

   flags : 0x0

Entry at 0x000000000c8f66f0

--------------------------------------------------------------------------------********************************************************************************

Total bytes read: 145289216 in 7.972 Sec (17797 KBytes/Sec)

 Fetch all information and start the boot process:

 Scale BootStrap RAM Image (32bit,SP,LE,X86)

Start StarOS:

 Invoking StarOS Image...

Set up environment:

[    0.000000] Linux version 2.6.38-staros-v3-scale-64 (yuel@bxb-mitg6-dev10) (gcc version 4.7.2 (GCC) ) #1 SMP PREEMPT Thu Feb 23 16:10:46 EST 2017

Boxer process gets instantiated:

Boxer /etc/rc beginning.

Identify this hosted environment QEMU and add DVD-ROM:

[    8.308582] scsi 0:0:0:0: Direct-Access     ATA      QEMU HARDDISK    2.3. PQ: 0 ANSI: 5

[    8.309031] ata2.01: ATAPI: QEMU DVD-ROM, 2.3.0, max UDMA/100

[    8.309521] ata2.01: configured for MWDMA2

[    8.311612] sd 0:0:0:0: [sda] 8388608 512-byte logical blocks: (4.29 GB/4.00 GiB)

[    8.312090] scsi 0:0:1:0: Direct-Access     ATA      QEMU HARDDISK    2.3. PQ: 0 ANSI: 5

[    8.312878] sd 0:0:0:0: [sda] Write Protect is off

[    8.312978] sd 0:0:1:0: [sdb] 33554432 512-byte logical blocks: (17.1 GB/16.0 GiB)

[    8.313011] sd 0:0:1:0: [sdb] Write Protect is off

[    8.313021] sd 0:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

[    8.314286] scsi 1:0:1:0: CD-ROM            QEMU     QEMU DVD-ROM     2.3. PQ: 0 ANSI: 5

Search for parameters file on configuration drive:

   ...Looking for staros_param.cfg on config driveInitial card type is 64  ...Looking for param.cfg on boot1.
[    8.414031] usb 1-1: new full speed USB device using uhci_hcd and address 2

Map the file Parameters in staros_param.cfg file and take precedence if there are any conflicts with values stored in the /boot1/param.cfg:

 Found param.cfg in local disk
Set 0x40010100 into sn_cardtype
: Found staros_param.cfg in config drive

Mount:

  ...mounting /var/crash from tmpfs

  ...Detected KVM Guest

  ...UUID DD2C2139-9E98-4C1B-B87F-83BBD9E8270B

Add NIC cards:

   ...loading networking kernel modules

     ...virtio net

[    9.661076] Selected 1 Queues, Max-Queue = 1, Online CPUs=8

[    9.663552] Selected 1 Queues, Max-Queue = 1, Online CPUs=8

     ...vmxnet3

[    9.669130] VMware vmxnet3 virtual NIC driver - version 1.0.25.0-k-NAPI

     ...e1000

[    9.677388] e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI

[    9.677909] e1000: Copyright (c) 1999-2006 Intel Corporation.

     ...e1000e

[    9.687631] e1000e: Intel(R) PRO/1000 Network Driver - 1.2.20-k2

[    9.688079] e1000e: Copyright(c) 1999 - 2011 Intel Corporation.

     ...mdio

     ...ixgbe

                 

Set up Network Interfaces (NI):

   ...setting up network interfaces

DI Internal changing MTU size on VM, SR-IOV is supposed to be enabled:

[   10.399271] ixgbevf: cpeth1: ixgbevf_change_mtu: changing MTU from 1500 to 7020

...create vlan interface cpeth1.2111

Start iftask:

waiting for iftask to start.........

waiting for iftask to start.........

Start masterd to decide master CF role:

 start masterd 1 to decide master CF role

Broadcards to figure out Master/Standby mode:

  ...Broadcasting presence to master CF

Check Jumbo Packets: First small size ping, mid size and jumbo:

Pinging(size=56) master slot : card2

Pinging(size=1472) master slot : card2

Pinging(size=6992) master slot : card2

Virtual network connectivity OK!