更换服务器UCS C240 M4 - vEPC上的故障组件

下载选项

  • PDF     (458.8 KB)
    在各种设备上使用 Adobe Reader 查看
  • ePub     (205.5 KB)
    在 iPhone、iPad、Android、Sony Reader 或 Windows Phone 上使用各种应用查看
  • Mobi (Kindle)     (178.0 KB)
    在 Kindle 设备上查看或在多个设备上使用 Kindle 应用查看
已更新: 2018 年 7 月 2 日
文档 ID:213464

非歧视性语言

此产品的文档集力求使用非歧视性语言。在本文档集中，非歧视性语言是指不隐含针对年龄、残障、性别、种族身份、族群身份、性取向、社会经济地位和交叉性的歧视的语言。由于产品软件的用户界面中使用的硬编码语言、基于 RFP 文档使用的语言或引用的第三方产品使用的语言，文档中可能无法确保完全使用非歧视性语言。 深入了解思科如何使用包容性语言。

关于此翻译

思科采用人工翻译与机器翻译相结合的方式将此文档翻译成不同语言，希望全球的用户都能通过各自的语言得到支持性的内容。 请注意：即使是最好的机器翻译，其准确度也不及专业翻译人员的水平。 Cisco Systems, Inc. 对于翻译的准确性不承担任何责任，并建议您总是参考英文原始文档（已提供链接）。

    Introduction

    本文描述要求的步骤替换在设置的Ultra-M的统一计算系统(UCS)服务器提及的这里有故障的组件主机StarOS虚拟网络作用(VNFs)。 

    • 双列直插存储器模块(DIMM)更换MOP
    • FlexFlash控制器故障
    • 固体推进(SSD)故障 
    • 委托的平台模块(TPM)故障
    • 袭击高速缓冲存储器故障
    • 袭击控制器热总线适配器(HBA)故障
    • PCI造反者故障
    • PCIe适配器Intel X520 10G故障
    • 模块化LAN在主板(MLOM)故障
    • 风扇盘RMA
    • CPU故障

    背景信息

    Ultra-M是设计为了简化VNFs的配置的一个被预先包装的和被验证的虚拟化的便携信息包核心解决方案。 OpenStack是虚拟化的基础设施管理器(精力) Ultra-M的并且包括这些节点类型：

    • 估计
    • 对象存储磁盘-估计(OSD -估计)
    • 控制器
    • OpenStack平台-导向器(OSPD)

    Ultra-M高级体系结构和介入的组件在此镜像表示：

    本文供熟悉Cisco Ultra-M平台的Cisco人员使用，并且选派要求的步骤被执行在OpenStack和StarOS VNF级别在服务器的组分更换时。 

    Note:超M 5.1.x版本考虑为了定义在本文的程序。

    简称

    VNF   虚拟网络功能
    CF    控制功能
    SF    服务功能
    ESC   有弹性服务控制器
    MOP   程序方法
    OSD   对象存储磁盘
    HDD   硬盘驱动器
    SSD   固体驱动
    精力   虚拟基础设施管理器
    VM    虚拟机
    EM    网元管理
    UAS   超自动化服务
    UUID   通用唯一标识符

    Mop的工作流 

    213464-replacement-of-faulty-components-on-serv-00.png

    Prerequisites

    备份

    在您替换一个有故障的组件前，检查您的红帽子OpenStack平台环境当前状态是重要的。它是推荐的您检查当前状态为了避免复杂化，当替换过程打开时。它可以由更换此流达到。

    在恢复的情况下， Cisco推荐使用OSPD数据库的备份与使用的这些步骤：

    [root@director ~]# mysqldump --opt --all-databases > /root/undercloud-all-databases.sql
    [root@director ~]# tar --xattrs -czf undercloud-backup-`date +%F`.tar.gz /root/undercloud-all-databases.sql 
    /etc/my.cnf.d/server.cnf /var/lib/glance/images /srv/node /home/stack
    tar: Removing leading `/' from member names


    此进程保证节点可以被替换，无需影响所有实例的可用性。 并且，推荐备份StarOS配置，特别是如果将被替换的compute/OSD估计节点主机控制功能(CF)虚拟机。

    Note:如果服务器是控制器节点，请进行对部分""，否则继续下个部分。


    组件RMA - Compute/OSD估计节点 

    识别在Compute/OSD估计节点主机的VMs

    识别在服务器主机的VMs。可以有两种可能性：

    • 服务器包含仅服务功能(SF) VM ：
    [stack@director ~]$ nova list --field name,host | grep compute-10
    | 49ac5f22-469e-4b84-badc-031083db0533 |  VNF2-DEPLOYM_s9_0_8bc6cc60-15d6-4ead-8b6a-10e75d0e134d     |  
    pod1-compute-10.localdomain    |

      

    • 服务器包含控制功能(CF) /Elastic服务控制器(ESC)/网元管理(EM)/超自动化VMs的服务(UAS)组合：

    [stack@director ~]$ nova list --field name,host | grep compute-8
    | 507d67c2-1d00-4321-b9d1-da879af524f8 | VNF2-DEPLOYM_XXXX_0_c8d98f0f-d874-45d0-af75-88a2d6fa82ea | pod1-compute-8.localdomain     |
    | f9c0763a-4a4f-4bbd-af51-bc7545774be2 | VNF2-DEPLOYM_c1_0_df4be88d-b4bf-4456-945a-3812653ee229     | pod1-compute-8.localdomain     |
    | 75528898-ef4b-4d68-b05d-882014708694 | VNF2-ESC-ESC-0                                             | pod1-compute-8.localdomain     |
    | f5bd7b9c-476a-4679-83e5-303f0aae9309 | VNF2-UAS-uas-0                                             | pod1-compute-8.localdomain     |

    Note: 在显示的输出中这里，第一列全体地对应于唯一标识符(UUID)，第二列是VM名字，并且第三列是主机名- VM存在的地方。从此输出的参数用于随后部分。

    优美停电

    案例仅1.估计节点主机SF VM

    移植SF卡到备用状态

    • 登陆对StarOS VNF并且识别对应于SF VM的卡。请使用从部分识别的SF VM的UUID “识别在Compute/OSD估计节点主机的VMs”，并且识别对应于UUID的卡：
    [local]VNF2# show card hardware
    Tuesday might 08 16:49:42 UTC 2018
    <snip>
    Card 8:
      Card Type               : 4-Port Service Function Virtual Card
      CPU Packages            : 26 [#0, #1, #2, #3, #4, #5, #6, #7, #8, #9, #10, #11, #12, #13, #14, #15, #16, #17, #18, #19, #20, #21, #22, #23, #24, #25]
      CPU Nodes               : 2
      CPU Cores/Threads       : 26
      Memory                  : 98304M (qvpc-di-large)
      UUID/Serial Number      :  49AC5F22-469E-4B84-BADC-031083DB0533
    • 检查卡的状态：
    [local]VNF2# show card table
    Tuesday might 08 16:52:53 UTC 2018
    Slot         Card Type                               Oper State     SPOF  Attach
    -----------  --------------------------------------  -------------  ----  ------
     1: CFC      Control Function Virtual Card           Active         No         
     2: CFC      Control Function Virtual Card           Standby        -          
     3: FC       4-Port Service Function Virtual Card    Active         No         
     4: FC       4-Port Service Function Virtual Card    Active         No         
     5: FC       4-Port Service Function Virtual Card    Active         No         
     6: FC       4-Port Service Function Virtual Card    Active         No         
     7: FC       4-Port Service Function Virtual Card    Active         No         
    8: FC       4-Port Service Function Virtual Card    Active         No         
     9: FC       4-Port Service Function Virtual Card    Active         No         
    10: FC       4-Port Service Function Virtual Card    Standby        -  
    • 如果卡在激活状态，请迁移卡向备用状态：
      [local]VNF2# card migrate from 8 to 10

    关闭从ESC的SF VM

    • 登陆对对应于VNF的ESC节点并且检查SF VM的状态：

    [admin@VNF2-esc-esc-0 ~]$ cd /opt/cisco/esc/esc-confd/esc-cli
    [admin@VNF2-esc-esc-0 esc-cli]$ ./esc_nc_cli get esc_datamodel | egrep --color "<state>|<vm_name>|<vm_id>|<deployment_name>"
    <snip>
    <state>SERVICE_ACTIVE_STATE</state>
                        VNF2-DEPLOYM_c1_0_df4be88d-b4bf-4456-945a-3812653ee229
                        VM_ALIVE_STATE
                         VNF2-DEPLOYM_s9_0_8bc6cc60-15d6-4ead-8b6a-10e75d0e134d
                        VM_ALIVE_STATE</state>
    <snip>
    • 终止与使用的SF VM其VM名字。(VM命名要注意从部分“识别在Compute/OSD估计节点主机的VMs”) ：

    [admin@VNF2-esc-esc-0 esc-cli]$ ./esc_nc_cli vm-action STOP VNF2-DEPLOYM_s9_0_8bc6cc60-15d6-4ead-8b6a-10e75d0e134d
    •  一旦它被终止， VM必须进入切断状态：
    [admin@VNF2-esc-esc-0 ~]$ cd /opt/cisco/esc/esc-confd/esc-cli
    [admin@VNF2-esc-esc-0 esc-cli]$ ./esc_nc_cli get esc_datamodel | egrep --color "<state>|<vm_name>|<vm_id>|<deployment_name>"
    <snip>
    <state>SERVICE_ACTIVE_STATE</state>
                        VNF2-DEPLOYM_c1_0_df4be88d-b4bf-4456-945a-3812653ee229
                        VM_ALIVE_STATE
                        VNF2-DEPLOYM_c3_0_3e0db133-c13b-4e3d-ac14-
                        VM_ALIVE_STATE
                        VNF2-DEPLOYM_s9_0_8bc6cc60-15d6-4ead-8b6a-10e75d0e134d
                        VM_SHUTOFF_STATE</state>

    案例2. Compute/OSD估计节点主机CF/ESC/EM/UAS

    迁移对备用状态的CF卡

    • 登陆对StarOS VNF并且识别对应于CF VM的卡。请使用从部分识别的CF VM的UUID “识别在节点主机的VMs”，并且查找对应于UUID的卡：

    [local]VNF2# show card hardware
    Tuesday might 08 16:49:42 UTC 2018
    <snip>
    Card 2:
      Card Type               : Control Function Virtual Card
      CPU Packages            : 8 [#0, #1, #2, #3, #4, #5, #6, #7]
      CPU Nodes               : 1
      CPU Cores/Threads       : 8
      Memory                  : 16384M (qvpc-di-large)
      UUID/Serial Number      : F9C0763A-4A4F-4BBD-AF51-BC7545774BE2
    <snip>
    • 检查卡的状态：

    [local]VNF2# show card table
    Tuesday might 08 16:52:53 UTC 2018
    Slot         Card Type                               Oper State     SPOF  Attach
    -----------  --------------------------------------  -------------  ----  ------
     1: CFC      Control Function Virtual Card           Standby        -
     2: CFC      Control Function Virtual Card           Active         No          
     3: FC       4-Port Service Function Virtual Card    Active         No         
     4: FC       4-Port Service Function Virtual Card    Active         No         
     5: FC       4-Port Service Function Virtual Card    Active         No         
     6: FC       4-Port Service Function Virtual Card    Active         No         
     7: FC       4-Port Service Function Virtual Card    Active         No         
     8: FC       4-Port Service Function Virtual Card    Active         No         
     9: FC       4-Port Service Function Virtual Card    Active         No         
    10: FC       4-Port Service Function Virtual Card    Standby        -
    • 如果卡在激活状态，请迁移卡向备用状态：

    [local]VNF2# card migrate from 2 to 1

    关闭CF和EM VM从ESC

    • 登陆对对应于VNF的ESC节点并且检查VMs的状态：

    [admin@VNF2-esc-esc-0 ~]$ cd /opt/cisco/esc/esc-confd/esc-cli
    [admin@VNF2-esc-esc-0 esc-cli]$ ./esc_nc_cli get esc_datamodel | egrep --color "<state>|<vm_name>|<vm_id>|<deployment_name>"
    <snip>
    <state>SERVICE_ACTIVE_STATE</state>
                        VNF2-DEPLOYM_c1_0_df4be88d-b4bf-4456-945a-3812653ee229
                        VM_ALIVE_STATE</state>
                        VNF2-DEPLOYM_c3_0_3e0db133-c13b-4e3d-ac14-
                        VM_ALIVE_STATE
    <deployment_name>VNF2-DEPLOYMENT-em</deployment_name>
                      507d67c2-1d00-4321-b9d1-da879af524f8
                      dc168a6a-4aeb-4e81-abd9-91d7568b5f7c
                      9ffec58b-4b9d-4072-b944-5413bf7fcf07
                    SERVICE_ACTIVE_STATE
                        VNF2-DEPLOYM_XXXX_0_c8d98f0f-d874-45d0-af75-88a2d6fa82ea
                        VM_ALIVE_STATE</state>
    <snip>
    • 逐个终止CF和EM VM与使用其VM名字。(VM命名要注意从部分“识别在Compute/OSD估计节点主机的VMs”) ：

    [admin@VNF2-esc-esc-0 esc-cli]$ ./esc_nc_cli vm-action STOP VNF2-DEPLOYM_c1_0_df4be88d-b4bf-4456-945a-3812653ee229

    [admin@VNF2-esc-esc-0 esc-cli]$ ./esc_nc_cli vm-action STOP VNF2-DEPLOYM_XXXX_0_c8d98f0f-d874-45d0-af75-88a2d6fa82ea

      

    • 在它终止后， VMs必须进入切断状态：

    [admin@VNF2-esc-esc-0 ~]$ cd /opt/cisco/esc/esc-confd/esc-cli
    [admin@VNF2-esc-esc-0 esc-cli]$ ./esc_nc_cli get esc_datamodel | egrep --color "<state>|<vm_name>|<vm_id>|<deployment_name>"
    <snip>
    <state>SERVICE_ACTIVE_STATE</state>
                        VNF2-DEPLOYM_c1_0_df4be88d-b4bf-4456-945a-3812653ee229</vm_name>
                        VM_SHUTOFF_STATE</state>
                        VNF2-DEPLOYM_c3_0_3e0db133-c13b-4e3d-ac14-
                        VM_ALIVE_STATE
    <deployment_name>VNF2-DEPLOYMENT-em</deployment_name>
                      507d67c2-1d00-4321-b9d1-da879af524f8
                      dc168a6a-4aeb-4e81-abd9-91d7568b5f7c
                      9ffec58b-4b9d-4072-b944-5413bf7fcf07
                    SERVICE_ACTIVE_STATE
                        VNF2-DEPLOYM_XXXX_0_c8d98f0f-d874-45d0-af75-88a2d6fa82ea</vm_name>
                        <state>VM_SHUTOFF_STATE</state>
    <snip>

    对备用方式的迁移ESC

    • 登陆对在节点主机的ESC并且检查是否在重要的状态。如果是，请换成ESC备用方式：

    [admin@VNF2-esc-esc-0 esc-cli]$ escadm status
    0 ESC status=0 ESC Master Healthy


    [admin@VNF2-esc-esc-0 ~]$ sudo service keepalived stop
    Stopping keepalived:                                       [  OK  ]

    [admin@VNF2-esc-esc-0 ~]$ escadm status
    1 ESC status=0 In SWITCHING_TO_STOP state. Please check status after a while.

    [admin@VNF2-esc-esc-0 ~]$ sudo reboot
    Broadcast message from admin@vnf1-esc-esc-0.novalocal
           (/dev/pts/0) at 13:32 ...
    The system is going down for reboot NOW!

    Note:如果有故障的组件将被替换在OSD估计节点，请放Ceph到维护在服务器，在您继续进行组分更换前。 

    [admin@osd-compute-0 ~]$ sudo ceph osd set norebalance
    set norebalance

    [admin@osd-compute-0 ~]$ sudo ceph osd set noout
    set noout

    [admin@osd-compute-0 ~]$ sudo ceph status
        cluster eb2bb192-b1c9-11e6-9205-525400330666
         health HEALTH_WARN
                noout,norebalance,sortbitwise,require_jewel_osds flag(s) set
         monmap e1: 3 mons at {tb3-ultram-pod1-controller-0=11.118.0.40:6789/0,tb3-ultram-pod1-controller-1=11.118.0.41:6789/0,tb3-ultram-pod1-controller-2=11.118.0.42:6789/0}
                election epoch 58, quorum 0,1,2 tb3-ultram-pod1-controller-0,tb3-ultram-pod1-controller-1,tb3-ultram-pod1-controller-2
         osdmap e194: 12 osds: 12 up, 12 in
                flags noout,norebalance,sortbitwise,require_jewel_osds
          pgmap v584865: 704 pgs, 6 pools, 531 GB data, 344 kobjects
                1585 GB used, 11808 GB / 13393 GB avail
                     704 active+clean
      client io 463 kB/s rd, 14903 kB/s wr, 263 op/s rd, 542 op/s wr

    替换从Compute/OSD估计节点的有故障的组件

    停电指定的服务器。步骤为了替换在UCS C240 M4服务器的一个有故障的组件可以参考从： 

    替换服务器组件

    恢复VMs

     案例仅1.估计节点主机SF VM

     SF从ESC的VM恢复

    • SF VM在新星列表的错误状态：

    [stack@director  ~]$ nova list |grep VNF2-DEPLOYM_s9_0_8bc6cc60-15d6-4ead-8b6a-10e75d0e134d
    | 49ac5f22-469e-4b84-badc-031083db0533 | VNF2-DEPLOYM_s9_0_8bc6cc60-15d6-4ead-8b6a-10e75d0e134d     | ERROR  | -          | NOSTATE     | 

      

    • 从ESC恢复SF VM ：

    [admin@VNF2-esc-esc-0 ~]$ sudo /opt/cisco/esc/esc-confd/esc-cli/esc_nc_cli recovery-vm-action DO VNF2-DEPLOYM_s9_0_8bc6cc60-15d6-4ead-8b6a-10e75d0e134d
    [sudo] password for admin: 

    Recovery VM Action
    /opt/cisco/esc/confd/bin/netconf-console --port=830 --host=127.0.0.1 --user=admin --privKeyFile=/root/.ssh/confd_id_dsa --privKeyType=dsa --rpc=/tmp/esc_nc_cli.ZpRCGiieuW
    <?xml version="1.0" encoding="UTF-8"?>
    <rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="1">
      <ok/>
    </rpc-reply>
    • 监控yangesc.log ：

    admin@VNF2-esc-esc-0 ~]$ tail -f /var/log/esc/yangesc.log
    …
    14:59:50,112 07-Nov-2017 WARN  Type: VM_RECOVERY_COMPLETE
    14:59:50,112 07-Nov-2017 WARN  Status: SUCCESS
    14:59:50,112 07-Nov-2017 WARN  Status Code: 200
    14:59:50,112 07-Nov-2017 WARN  Status Msg: Recovery: Successfully recovered VM [VNF2-DEPLOYM_s9_0_8bc6cc60-15d6-4ead-8b6a-10e75d0e134d].
    • 保证SF卡过来作为在VNF的暂挂SF

    案例2. Compute/OSD估计节点主机CF、ESC、EM和UAS

      UAS VM恢复

    • 检查UAS VM的状态在新星列表的并且删除它：
    [stack@director ~]$ nova list | grep VNF2-UAS-uas-0
    | 307a704c-a17c-4cdc-8e7a-3d6e7e4332fa | VNF2-UAS-uas-0                                                 | ACTIVE | -          | Running     | VNF2-UAS-uas-orchestration=172.168.11.10; VNF2-UAS-uas-management=172.168.10.3
    [stack@tb5-ospd ~]$ nova delete VNF2-UAS-uas-0
    Request to delete server VNF2-UAS-uas-0 has been accepted.
    • 为了恢复autovnf UAS VM，请运行UAS检查脚本为了检查状态。它必须报告错误。然后再请运行以--修正选项为了再创缺少UAS VM ：

    [stack@director ~]$ cd /opt/cisco/usp/uas-installer/scripts/
    [stack@director scripts]$ ./uas-check.py auto-vnf VNF2-UAS
    2017-12-08 12:38:05,446 - INFO: Check of AutoVNF cluster started
    2017-12-08 12:38:07,925 - INFO: Instance 'vnf1-UAS-uas-0' status is 'ERROR'
    2017-12-08 12:38:07,925 - INFO: Check completed, AutoVNF cluster has recoverable errors

    [stack@director scripts]$ ./uas-check.py auto-vnf VNF2-UAS --fix
    2017-11-22 14:01:07,215 - INFO: Check of AutoVNF cluster started
    2017-11-22 14:01:09,575 - INFO: Instance VNF2-UAS-uas-0' status is 'ERROR'
    2017-11-22 14:01:09,575 - INFO: Check completed, AutoVNF cluster has recoverable errors
    2017-11-22 14:01:09,778 - INFO: Removing instance VNF2-UAS-uas-0'
    2017-11-22 14:01:13,568 - INFO: Removed instance VNF2-UAS-uas-0'
    2017-11-22 14:01:13,568 - INFO: Creating instance VNF2-UAS-uas-0' and attaching volume ‘VNF2-UAS-uas-vol-0'
    2017-11-22 14:01:49,525 - INFO: Created instance ‘VNF2-UAS-uas-0'
    • 登陆对autovnf UAS。等待几分钟，并且UAS必须返回到好状态：
    VNF2-autovnf-uas-0#show uas
    uas version 1.0.1-1
    uas state ha-active
    uas ha-vip 172.17.181.101
    INSTANCE IP   STATE  ROLE
    -----------------------------------
    172.17.180.6  alive  CONFD-SLAVE
    172.17.180.7  alive  CONFD-MASTER
    172.17.180.9  alive  NA

    Note:如果UAScheck.py--修正发生故障，您也许需要复制此文件和再运作。

    [stack@director ~]$ mkdir –p /opt/cisco/usp/apps/auto-it/common/uas-deploy/
    [stack@director ~]$ cp /opt/cisco/usp/uas-installer/common/uas-deploy/userdata-uas.txt /opt/cisco/usp/apps/auto-it/common/uas-deploy/

      

    ESC VM恢复

    • 检查ESC VM的状态从新星列表的并且删除它：
    stack@director scripts]$ nova list |grep ESC-1
    | c566efbf-1274-4588-a2d8-0682e17b0d41 | VNF2-ESC-ESC-1                                                 | ACTIVE | -          | Running     | VNF2-UAS-uas-orchestration=172.168.11.14; VNF2-UAS-uas-management=172.168.10.4                                                                                                 |
    [stack@director scripts]$ nova delete VNF2-ESC-ESC-1
    Request to delete server VNF2-ESC-ESC-1 has been accepted.
    • 从AutoVNF UAS，请查找ESC配置处理和在处理的日志查找boot_vm.py line命令为了创建ESC实例：

    ubuntu@VNF2-uas-uas-0:~$ sudo -i
    root@VNF2-uas-uas-0:~# confd_cli -u admin -C
    Welcome to the ConfD CLI    
    admin connected from 127.0.0.1 using console on VNF2-uas-uas-0
    VNF2-uas-uas-0#show transaction
    TX ID                                 TX TYPE          DEPLOYMENT ID    TIMESTAMP                         STATUS
    -----------------------------------------------------------------------------------------------------------------------------
    35eefc4a-d4a9-11e7-bb72-fa163ef8df2b  vnf-deployment   VNF2-DEPLOYMENT  2017-11-29T02:01:27.750692-00:00  deployment-success
    73d9c540-d4a8-11e7-bb72-fa163ef8df2b  vnfm-deployment  VNF2-ESC         2017-11-29T01:56:02.133663-00:00  deployment-success


    VNF2-uas-uas-0#show logs 73d9c540-d4a8-11e7-bb72-fa163ef8df2b | display xml
    <config xmlns="http://tail-f.com/ns/config/1.0">
      <logs xmlns="http://www.cisco.com/usp/nfv/usp-autovnf-oper">
        <tx-id>73d9c540-d4a8-11e7-bb72-fa163ef8df2b</tx-id>
        <log>2017-11-29 01:56:02,142 - VNFM Deployment RPC triggered for deployment: VNF2-ESC, deactivate: 0
    2017-11-29 01:56:02,179 - Notify deployment
    ..
    2017-11-29 01:57:30,385 - Creating VNFM 'VNF2-ESC-ESC-1' with [python //opt/cisco/vnf-staging/bootvm.py VNF2-ESC-ESC-1 --flavor VNF2-ESC-ESC-flavor --image 3fe6b197-961b-4651-af22-dfd910436689 --net VNF2-UAS-uas-management --gateway_ip 172.168.10.1 --net VNF2-UAS-uas-orchestration --os_auth_url http://10.1.2.5:5000/v2.0 --os_tenant_name core --os_username ****** --os_password ****** --bs_os_auth_url http://10.1.2.5:5000/v2.0 --bs_os_tenant_name core --bs_os_username ****** --bs_os_password ****** --esc_ui_startup false --esc_params_file /tmp/esc_params.cfg --encrypt_key ****** --user_pass ****** --user_confd_pass ****** --kad_vif eth0 --kad_vip 172.168.10.7 --ipaddr 172.168.10.6 dhcp --ha_node_list 172.168.10.3 172.168.10.6 --file root:0755:/opt/cisco/esc/esc-scripts/esc_volume_em_staging.sh:/opt/cisco/usp/uas/autovnf/vnfms/esc-scripts/esc_volume_em_staging.sh --file root:0755:/opt/cisco/esc/esc-scripts/esc_vpc_chassis_id.py:/opt/cisco/usp/uas/autovnf/vnfms/esc-scripts/esc_vpc_chassis_id.py --file root:0755:/opt/cisco/esc/esc-scripts/esc-vpc-di-internal-keys.sh:/opt/cisco/usp/uas/autovnf/vnfms/esc-scripts/esc-vpc-di-internal-keys.sh

    保存boot_vm.py线路对shell script程序文件(esc.sh)并且更新与正确的信息(典型地core/<PASSWORD>)的所有用户名*****和密码*****线路。您需要取消– encrypt_key选项。对于user_pass和user_confd_pass，您需要使用格式–用户名：密码(示例- admin ：<PASSWORD>)。

    • 查找URL为了从running-config和wget的bootvm.py bootvm.pyfile对autovnf UAS VM。  在这种情况下， 10.1.2.3是自动IT VM的IP ：
    root@VNF2-uas-uas-0:~# confd_cli -u admin -C
    Welcome to the ConfD CLI
    admin connected from 127.0.0.1 using console on VNF2-uas-uas-0
    VNF2-uas-uas-0#show running-config autovnf-vnfm:vnfm
    …
    configs bootvm
      value http:// 10.1.2.3:80/bundles/5.1.7-2007/vnfm-bundle/bootvm-2_3_2_155.py
    !
    root@VNF2-uas-uas-0:~# wget http://10.1.2.3:80/bundles/5.1.7-2007/vnfm-bundle/bootvm-2_3_2_155.py
    --2017-12-01 20:25:52--  http://10.1.2.3 /bundles/5.1.7-2007/vnfm-bundle/bootvm-2_3_2_155.py
    Connecting to 10.1.2.3:80... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 127771 (125K) [text/x-python]
    Saving to: ‘bootvm-2_3_2_155.py’
    100%[=====================================================================================>] 127,771  --.-K/s   in 0.001s
    2017-12-01 20:25:52 (173 MB/s) - ‘bootvm-2_3_2_155.py’ saved [127771/127771]
    • 创建一个/tmp/esc_params.cfg文件：

    root@VNF2-uas-uas-0:~# echo "openstack.endpoint=publicURL" > /tmp/esc_params.cfg
    • 执行shell script程序为了配置从UAS节点的ESC ：

    root@VNF2-uas-uas-0:~# /bin/sh esc.sh
    + python ./bootvm.py VNF2-ESC-ESC-1 --flavor VNF2-ESC-ESC-flavor --image 3fe6b197-961b-4651-af22-dfd910436689
     --net VNF2-UAS-uas-management --gateway_ip 172.168.10.1 --net VNF2-UAS-uas-orchestration --os_auth_url 
    http://10.1.2.5:5000/v2.0 --os_tenant_name core --os_username core --os_password <PASSWORD> --bs_os_auth_url 
    http://10.1.2.5:5000/v2.0 --bs_os_tenant_name core --bs_os_username core --bs_os_password <PASSWORD> 
    --esc_ui_startup false --esc_params_file /tmp/esc_params.cfg --user_pass admin:<PASSWORD> --user_confd_pass 
    admin:<PASSWORD> --kad_vif eth0 --kad_vip 172.168.10.7 --ipaddr 172.168.10.6 dhcp --ha_node_list 172.168.10.3
    172.168.10.6 --file root:0755:/opt/cisco/esc/esc-scripts/esc_volume_em_staging.sh:/opt/cisco/usp/uas/autovnf/vnfms/esc-scripts/esc_volume_em_staging.sh 
    --file root:0755:/opt/cisco/esc/esc-scripts/esc_vpc_chassis_id.py:/opt/cisco/usp/uas/autovnf/vnfms/esc-scripts/esc_vpc_chassis_id.py 
    --file root:0755:/opt/cisco/esc/esc-scripts/esc-vpc-di-internal-keys.sh:/opt/cisco/usp/uas/autovnf/vnfms/esc-scripts/esc-vpc-di-internal-keys.sh
    • 登陆对新的ESC并且验证备份的状态：

    ubuntu@VNF2-uas-uas-0:~$ ssh admin@172.168.11.14
    …
       ####################################################################
       #   ESC on VNF2-esc-esc-1.novalocal is in BACKUP state.
       ####################################################################

    [admin@VNF2-esc-esc-1 ~]$ escadm status
    0 ESC status=0 ESC Backup Healthy

    [admin@VNF2-esc-esc-1 ~]$ health.sh
    ============== ESC HA (BACKUP) ===================================================
    ESC HEALTH PASSED

    从ESC恢复CF和EM VMs

    • 检查CF和EM从新星列表的VMs的状态。他们必须在错误状态：

    [stack@director ~]$ source corerc
    [stack@director ~]$ nova list --field name,host,status |grep -i err   
    | 507d67c2-1d00-4321-b9d1-da879af524f8 | VNF2-DEPLOYM_XXXX_0_c8d98f0f-d874-45d0-af75-88a2d6fa82ea | None                                 | ERROR|
    | f9c0763a-4a4f-4bbd-af51-bc7545774be2 | VNF2-DEPLOYM_c1_0_df4be88d-b4bf-4456-945a-3812653ee229     |None                                 | ERROR

         

    • 登陆对ESC主设备，运行每个被影响的EM和CF VM的恢复VM动作。耐心。ESC将安排恢复动作，并且也许几分钟不发生。监控yangesc.log ：
    sudo /opt/cisco/esc/esc-confd/esc-cli/esc_nc_cli recovery-vm-action DO <VM-Name>

    [admin@VNF2-esc-esc-0 ~]$ sudo /opt/cisco/esc/esc-confd/esc-cli/esc_nc_cli recovery-vm-action DO VNF2-DEPLOYMENT-_VNF2-D_0_a6843886-77b4-4f38-b941-74eb527113a8
    [sudo] password for admin: 

    Recovery VM Action
    /opt/cisco/esc/confd/bin/netconf-console --port=830 --host=127.0.0.1 --user=admin --privKeyFile=/root/.ssh/confd_id_dsa --privKeyType=dsa --rpc=/tmp/esc_nc_cli.ZpRCGiieuW
    <?xml version="1.0" encoding="UTF-8"?>
    <rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="1">
      <ok/>
    </rpc-reply>

    [admin@VNF2-esc-esc-0 ~]$ tail -f /var/log/esc/yangesc.log
    …
    14:59:50,112 07-Nov-2017 WARN  Type: VM_RECOVERY_COMPLETE
    14:59:50,112 07-Nov-2017 WARN  Status: SUCCESS
    14:59:50,112 07-Nov-2017 WARN  Status Code: 200
    14:59:50,112 07-Nov-2017 WARN  Status Msg: Recovery: Successfully recovered VM [VNF2-DEPLOYMENT-_VNF2-D_0_a6843886-77b4-4f38-b941-74eb527113a8]
    • 登陆对新的EM并且验证EM状态是UP ：

    ubuntu@VNF2vnfddeploymentem-1:~$ /opt/cisco/ncs/current/bin/ncs_cli -u admin -C
    admin connected from 172.17.180.6 using ssh on VNF2vnfddeploymentem-1
    admin@scm# show ems
    EM            VNFM
    ID  SLA  SCM  PROXY
    ---------------------
    2   up   up   up
    3   up   up   up
    • 日志到StarOS VNF里和验证CF卡在备用状态

       

    把柄ESC恢复故障

    在ESC不能开始VM由于一个意外的状态处， Cisco推荐如何通过重新启动主设备ESC执行ESC切换。ESC切换将花费一分钟。运行脚本“health.sh”在新的主设备ESC为了检查状态是否是UP。掌握ESC开始VM和修理VM状态。此恢复任务将花费5分钟完成。

    您能监控/var/log/esc/yangesc.log和/var/log/esc/escmanager.log。如果看不到在5-7分钟之后被恢复的VM获得，用户将需要去执行被影响的VM的手工的恢复。

    自动配置配置更新

    • 从AutoDeploy VM，请编辑autodeploy.cfg并且用新的替换老估计服务器。然后负荷在confd_cli替换。此步骤对于成功的部署去活后是必需的：
    root@auto-deploy-iso-2007-uas-0:/home/ubuntu# confd_cli -u admin -C
    Welcome to the ConfD CLI
    admin connected from 127.0.0.1 using console on auto-deploy-iso-2007-uas-0
    auto-deploy-iso-2007-uas-0#config
    Entering configuration mode terminal
    auto-deploy-iso-2007-uas-0(config)#load replace autodeploy.cfg
    Loading.     14.63 KiB parsed in 0.42 sec (34.16 KiB/sec)

    auto-deploy-iso-2007-uas-0(config)#commit
    Commit complete.
    auto-deploy-iso-2007-uas-0(config)#end
    • 在配置更改以后重新启动UASconfd和autodeploy服务：
    root@auto-deploy-iso-2007-uas-0:~# service uas-confd restart
    uas-confd stop/waiting
    uas-confd start/running, process 14078

    root@auto-deploy-iso-2007-uas-0:~# service uas-confd status
    uas-confd start/running, process 14078

    root@auto-deploy-iso-2007-uas-0:~# service autodeploy restart
    autodeploy stop/waiting
    autodeploy start/running, process 14017
    root@auto-deploy-iso-2007-uas-0:~# service autodeploy status
    autodeploy start/running, process 14017

    组件RMA -控制器节点

    预先校验

                                                       

    • 从OSPD，对控制器的登录和验证个人计算机是在好州全部三个控制器联机，并且加莱拉角显示全部三个控制器作为主设备。 

    Note:健康簇要求2个激活控制器，因此请验证保持的两个控制器是联机和活跃的。

    [heat-admin@pod1-controller-0 ~]$ sudo pcs status
    Cluster name: tripleo_cluster
    Stack: corosync
    Current DC: pod1-controller-2 (version 1.1.15-11.el7_3.4-e174ec8) - partition with quorum
    Last updated: Mon Dec  4 00:46:10 2017                        Last change: Wed Nov 29 01:20:52 2017 by hacluster via crmd on pod1-controller-0
    3 nodes and 22 resources configured

    Online: [ pod1-controller-0 pod1-controller-1 pod1-controller-2 ]

    Full list of resources:
     ip-11.118.0.42  (ocf::heartbeat:IPaddr2):           Started pod1-controller-1
     ip-11.119.0.47  (ocf::heartbeat:IPaddr2):           Started pod1-controller-2
     ip-11.120.0.49  (ocf::heartbeat:IPaddr2):           Started pod1-controller-1
     ip-192.200.0.102          (ocf::heartbeat:IPaddr2):           Started pod1-controller-2
     Clone Set: haproxy-clone [haproxy]
         Started: [ pod1-controller-0 pod1-controller-1 pod1-controller-2 ]
     Master/Slave Set: galera-master [galera]
         Masters: [ pod1-controller-0 pod1-controller-1 pod1-controller-2 ]
     ip-11.120.0.47  (ocf::heartbeat:IPaddr2):           Started pod1-controller-2
     Clone Set: rabbitmq-clone [rabbitmq]
         Started: [ pod1-controller-0 pod1-controller-1 pod1-controller-2 ]
     Master/Slave Set: redis-master [redis]
         Masters: [ pod1-controller-2 ]
         Slaves: [ pod1-controller-0 pod1-controller-1 ]
     ip-10.84.123.35            (ocf::heartbeat:IPaddr2):           Started pod1-controller-1
     openstack-cinder-volume          (systemd:openstack-cinder-volume):            Started pod1-controller-2
     my-ipmilan-for-pod1-controller-0        (stonith:fence_ipmilan):  Started pod1-controller-0
     my-ipmilan-for-pod1-controller-1        (stonith:fence_ipmilan):  Started pod1-controller-0
     my-ipmilan-for-pod1-controller-2        (stonith:fence_ipmilan):  Started pod1-controller-0

    Daemon Status:
      corosync: active/enabled
      pacemaker: active/enabled
      pcsd: active/enabled

    移动控制器簇向维护模式

    • 请使用在暂挂更新的控制器的个人计算机簇：
    [heat-admin@pod1-controller-0 ~]$ sudo pcs cluster standby
    • 再检查个人计算机状态并且保证个人计算机在此节点集群终止：
    [heat-admin@pod1-controller-0 ~]$ sudo pcs status
    Cluster name: tripleo_cluster
    Stack: corosync
    Current DC: pod1-controller-2 (version 1.1.15-11.el7_3.4-e174ec8) - partition with quorum
    Last updated: Mon Dec  4 00:48:24 2017                        Last change: Mon Dec  4 00:48:18 2017 by root via crm_attribute on pod1-controller-0
    3 nodes and 22 resources configured

    Node pod1-controller-0: standby

    Online: [ pod1-controller-1 pod1-controller-2 ]

    Full list of resources:
     ip-11.118.0.42  (ocf::heartbeat:IPaddr2):           Started pod1-controller-1
     ip-11.119.0.47  (ocf::heartbeat:IPaddr2):           Started pod1-controller-2
     ip-11.120.0.49  (ocf::heartbeat:IPaddr2):           Started pod1-controller-1
     ip-192.200.0.102          (ocf::heartbeat:IPaddr2):           Started pod1-controller-2
     Clone Set: haproxy-clone [haproxy]
         Started: [ pod1-controller-1 pod1-controller-2 ]
         Stopped: [ pod1-controller-0 ]
    Master/Slave Set: galera-master [galera]
         Masters: [ pod1-controller-1 pod1-controller-2 ]
         Slaves: [ pod1-controller-0 ]
     ip-11.120.0.47  (ocf::heartbeat:IPaddr2):           Started pod1-controller-2
     Clone Set: rabbitmq-clone [rabbitmq]
         Started: [ pod1-controller-0 pod1-controller-1 pod1-controller-2 ]
     Master/Slave Set: redis-master [redis]
         Masters: [ pod1-controller-2 ]
         Slaves: [ pod1-controller-1 ]
         Stopped: [ pod1-controller-0 ]
     ip-10.84.123.35            (ocf::heartbeat:IPaddr2):           Started pod1-controller-1
     openstack-cinder-volume          (systemd:openstack-cinder-volume):            Started pod1-controller-2
     my-ipmilan-for-pod1-controller-0        (stonith:fence_ipmilan):  Started pod1-controller-1
     my-ipmilan-for-pod1-controller-1        (stonith:fence_ipmilan):  Started pod1-controller-1
     my-ipmilan-for-pod1-controller-2        (stonith:fence_ipmilan):  Started pod1-controller-2

    Daemon Status:
      corosync: active/enabled
      pacemaker: active/enabled
      pcsd: active/enabled               

    并且，在其他2个控制器的个人计算机状态应该显示节点作为暂挂。

    替换从控制器节点的有故障的组件

    停电指定的服务器。 步骤为了替换在UCS C240 M4服务器的一个有故障的组件可以参考从： 

    替换服务器组件

    在服务器的功率

    • 在服务器的功率和验证服务器出现：
    [stack@tb5-ospd ~]$ source stackrc
    [stack@tb5-ospd ~]$ nova list |grep pod1-controller-0
    | 1ca946b8-52e5-4add-b94c-4d4b8a15a975 | pod1-controller-0  | ACTIVE | -          | Running     | ctlplane=192.200.0.112 |
    • 登陆到被影响的控制器，去除备用方式与使用unstandby。验证控制器来联机与簇，并且加莱拉角显示全部三个控制器作为主设备。这也许花费几分钟：
    [heat-admin@pod1-controller-0 ~]$ sudo pcs cluster unstandby

    [heat-admin@pod1-controller-0 ~]$ sudo pcs status
    Cluster name: tripleo_cluster
    Stack: corosync
    Current DC: pod1-controller-2 (version 1.1.15-11.el7_3.4-e174ec8) - partition with quorum
    Last updated: Mon Dec  4 01:08:10 2017                        Last change: Mon Dec  4 01:04:21 2017 by root via crm_attribute on pod1-controller-0
    3 nodes and 22 resources configured

    Online: [ pod1-controller-0 pod1-controller-1 pod1-controller-2 ]

    Full list of resources:
     ip-11.118.0.42  (ocf::heartbeat:IPaddr2):           Started pod1-controller-1
     ip-11.119.0.47  (ocf::heartbeat:IPaddr2):           Started pod1-controller-2
     ip-11.120.0.49  (ocf::heartbeat:IPaddr2):           Started pod1-controller-1
     ip-192.200.0.102          (ocf::heartbeat:IPaddr2):           Started pod1-controller-2
     Clone Set: haproxy-clone [haproxy]
         Started: [ pod1-controller-0 pod1-controller-1 pod1-controller-2 ]
    Master/Slave Set: galera-master [galera]
         Masters: [ pod1-controller-0 pod1-controller-1 pod1-controller-2 ]
     ip-11.120.0.47  (ocf::heartbeat:IPaddr2):           Started pod1-controller-2
     Clone Set: rabbitmq-clone [rabbitmq]
         Started: [ pod1-controller-0 pod1-controller-1 pod1-controller-2 ]
     Master/Slave Set: redis-master [redis]
         Masters: [ pod1-controller-2 ]
         Slaves: [ pod1-controller-0 pod1-controller-1 ]
     ip-10.84.123.35            (ocf::heartbeat:IPaddr2):           Started pod1-controller-1
     openstack-cinder-volume          (systemd:openstack-cinder-volume):            Started pod1-controller-2
     my-ipmilan-for-pod1-controller-0        (stonith:fence_ipmilan):  Started pod1-controller-1
     my-ipmilan-for-pod1-controller-1        (stonith:fence_ipmilan):  Started pod1-controller-1
     my-ipmilan-for-pod1-controller-2        (stonith:fence_ipmilan):  Started pod1-controller-2

    Daemon Status:
      corosync: active/enabled
      pacemaker: active/enabled
      pcsd: active/enabled

    • 您能检查某些监控程序服务例如ceph他们是在进行状态：
    [heat-admin@pod1-controller-0 ~]$ sudo ceph -s
        cluster eb2bb192-b1c9-11e6-9205-525400330666
         health HEALTH_OK
         monmap e1: 3 mons at {pod1-controller-0=11.118.0.10:6789/0,pod1-controller-1=11.118.0.11:6789/0,pod1-controller-2=11.118.0.12:6789/0}
                election epoch 70, quorum 0,1,2 pod1-controller-0,pod1-controller-1,pod1-controller-2
         osdmap e218: 12 osds: 12 up, 12 in
                flags sortbitwise,require_jewel_osds
          pgmap v2080888: 704 pgs, 6 pools, 714 GB data, 237 kobjects
                2142 GB used, 11251 GB / 13393 GB avail
                     704 active+clean
      client io 11797 kB/s wr, 0 op/s rd, 57 op/s wr
    TAC Authored

    由思科工程师提供

    • Prashanth Shetty
      Cisco Advanced Services
    • Padmaraj Ramanoudjam
      Cisco Advanced Services

    此文档是否有帮助?

    Feedback反馈

    联系我们

    本文档适用于以下产品