更換OSPD伺服器UCS 240M4 - CPS

簡介

本文檔介紹在Ultra-M設定中替換託管OpenStack平台導向器(OSPD)的故障伺服器所需的步驟。

背景資訊

Ultra-M是經過預打包和驗證的虛擬化移動資料包核心解決方案，旨在簡化VNF的部署。 OpenStack是適用於Ultra-M的虛擬化基礎架構管理器(VIM)，包含以下節點型別：

• 計算
• 對象儲存磁碟 — 計算（OSD — 計算）
• 控制器
• OpenStack平台 — 導向器(OSPD)

Ultra-M的高級體系結構及涉及的元件如圖所示。

UltraM體系結構  

附註：Ultra M 5.1.x版本用於定義本文檔中的過程。本文檔面向熟悉Cisco Ultra-M平台的思科人員，詳細說明了更換OSPD伺服器時在OpenStack級別上需要執行的步驟。

縮寫

VNF	虛擬網路功能
ESC	彈性服務控制器
澳門幣	程式方法
OSD	對象儲存磁碟
硬碟	硬碟驅動器
固態硬碟	固態驅動器
VIM	虛擬基礎架構管理員
虛擬機器	虛擬機器
EM	元素管理器
UAS	Ultra自動化服務
UUID	通用唯一識別符號

MoP的工作流程

替換過程的高級工作流   

必要條件

狀態檢查

在更換OSPD伺服器之前，請務必檢查Red Hat OpenStack平台環境的當前狀態，並確保其處於正常狀態，以避免更換過程啟動時出現問題。 

1.檢查OpenStack堆疊和節點清單的狀態。

[stack@director ~]$ source stackrc 
[stack@director ~]$ openstack stack list --nested
[stack@director ~]$ ironic node-list
[stack@director ~]$ nova list

2.從OSPD節點檢查所有底層雲服務是否處於已載入、活動和運行狀態。 

[stack@director ~]$ systemctl list-units "openstack*" "neutron*" "openvswitch*"

UNIT                                       LOAD   ACTIVE SUB     DESCRIPTION

neutron-dhcp-agent.service                 loaded active running OpenStack Neutron DHCP Agent
neutron-openvswitch-agent.service          loaded active running OpenStack Neutron Open vSwitch Agent
neutron-ovs-cleanup.service                loaded active exited  OpenStack Neutron Open vSwitch Cleanup Utility
neutron-server.service                     loaded active running OpenStack Neutron Server
openstack-aodh-evaluator.service           loaded active running OpenStack Alarm evaluator service
openstack-aodh-listener.service            loaded active running OpenStack Alarm listener service
openstack-aodh-notifier.service            loaded active running OpenStack Alarm notifier service
openstack-ceilometer-central.service       loaded active running OpenStack ceilometer central agent
openstack-ceilometer-collector.service     loaded active running OpenStack ceilometer collection service
openstack-ceilometer-notification.service  loaded active running OpenStack ceilometer notification agent
openstack-glance-api.service               loaded active running OpenStack Image Service (code-named Glance) API server
openstack-glance-registry.service          loaded active running OpenStack Image Service (code-named Glance) Registry server
openstack-heat-api-cfn.service             loaded active running Openstack Heat CFN-compatible API Service
openstack-heat-api.service                 loaded active running OpenStack Heat API Service
openstack-heat-engine.service              loaded active running Openstack Heat Engine Service
openstack-ironic-api.service               loaded active running OpenStack Ironic API service
openstack-ironic-conductor.service         loaded active running OpenStack Ironic Conductor service
openstack-ironic-inspector-dnsmasq.service loaded active running PXE boot dnsmasq service for Ironic Inspector
openstack-ironic-inspector.service         loaded active running Hardware introspection service for OpenStack Ironic
openstack-mistral-api.service              loaded active running Mistral API Server
openstack-mistral-engine.service           loaded active running Mistral Engine Server
openstack-mistral-executor.service         loaded active running Mistral Executor Server
openstack-nova-api.service                 loaded active running OpenStack Nova API Server
openstack-nova-cert.service                loaded active running OpenStack Nova Cert Server
openstack-nova-compute.service             loaded active running OpenStack Nova Compute Server
openstack-nova-conductor.service           loaded active running OpenStack Nova Conductor Server
openstack-nova-scheduler.service           loaded active running OpenStack Nova Scheduler Server
openstack-swift-account-reaper.service     loaded active running OpenStack Object Storage (swift) - Account Reaper
openstack-swift-account.service            loaded active running OpenStack Object Storage (swift) - Account Server
openstack-swift-container-updater.service  loaded active running OpenStack Object Storage (swift) - Container Updater
openstack-swift-container.service          loaded active running OpenStack Object Storage (swift) - Container Server
openstack-swift-object-updater.service     loaded active running OpenStack Object Storage (swift) - Object Updater
openstack-swift-object.service             loaded active running OpenStack Object Storage (swift) - Object Server
openstack-swift-proxy.service              loaded active running OpenStack Object Storage (swift) - Proxy Server
openstack-zaqar.service                    loaded active running OpenStack Message Queuing Service (code-named Zaqar) Server
openstack-zaqar@1.service                  loaded active running OpenStack Message Queuing Service (code-named Zaqar) Server Instance 1
openvswitch.service                        loaded active exited  Open vSwitch

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

37 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

備份

1.在執行備份過程之前，請確認是否有足夠的可用磁碟空間。此彈珠應至少為3.5 GB。

[stack@director ~]$df -h

2.以root使用者身份運行這些命令，以將資料從底層雲節點備份到名為undercloud-backup-[timestamp].tar.gz的檔案中。

[root@director ~]# mysqldump --opt --all-databases > /root/undercloud-all-databases.sql 
[root@director ~]# tar --xattrs -czf undercloud-backup-`date +%F`.tar.gz /root/undercloud-all-databases.sql 
/etc/my.cnf.d/server.cnf /var/lib/glance/images /srv/node /home/stack 
tar: Removing leading `/' from member names 

安裝新的OSPD節點

UCS伺服器安裝

1.有關安裝新UCS C240 M4伺服器的步驟和初始設定步驟，請參閱此處：Cisco UCS C240 M4伺服器安裝和服務指南

2.使用CIMC IP登入到伺服器。

3.如果韌體與以前使用的推薦版本不一致，請執行BIOS升級。此處提供了BIOS升級步驟：Cisco UCS C系列機架式伺服器BIOS升級指南

5.從RAID級別為1的物理驅動器建立虛擬驅動器。導航到Storage > Cisco 12G SAS Modular Raid Controller(SLOT-HBA)> Controller Info > Create Virtual Drive from Unused Physical Drives，如下圖所示。

6.選擇VD並配置「設定為引導驅動器」，如下圖所示。

7.啟用IPMI over LAN。導覽至Admin > Communication Services > Communication Services，如下圖所示。

8.禁用超執行緒。導覽至Compute > BIOS > Configure BIOS > Advanced > Processor Configuration，如下圖所示。

 

附註：此處顯示的影象和本節中提到的配置步驟是參考韌體版本3.0(3e)，如果您使用其他版本，可能會有細微的變化。

  

Redhat安裝

安裝Red Hat ISO映像

1.登入OSP-D伺服器

2.啟動KVM控制檯

3.選擇Virtual Media > Activate Virtual Devices。接受會話並啟用記住您的設定以便將來連線。

4.選擇Virtual Media > Map CD/DVD並對映Red Hat ISO映像。

5.選擇Power > Reset System(Warm Boot)以重新啟動系統。

6.重新啟動時，按F6並選擇Cisco vKVM-Mapped vDVD1.22，然後按Enter。

7.安裝RHEL。

恢復Undercloud

根據備份準備雲下安裝

在電腦安裝有RHEL 7.3並處於乾淨狀態後，重新啟用安裝和運行director所需的所有訂閱/儲存庫。

1.主機名配置。

[root@director ~]$sudo hostnamectl set-hostname <FQDN_hostname> 
[root@director ~]$sudo hostnamectl set-hostname --transient <FQDN_hostname>

2.編輯/etc/hosts文件。

[root@director ~]$ vi /etc/hosts

<ospd_external_address>  <server_hostname>  <FQDN_hostname>
10.225.247.142  pod1-ospd pod1-ospd.cisco.com

3.驗證主機名。

[root@director ~]$ cat /etc/hostname
pod1-ospd.cisco.com

4.驗證DNS配置。

[root@director ~]$ cat /etc/resolv.conf

#Generated by NetworkManager
nameserver <DNS_IP>

5.修改調配nic介面。

[root@director ~]$ cat /etc/sysconfig/network-scripts/ifcfg-eno1

DEVICE=eno1
ONBOOT=yes
HOTPLUG=no
NM_CONTROLLED=no
PEERDNS=no
DEVICETYPE=ovs
TYPE=OVSPort
OVS_BRIDGE=br-ctlplane
BOOTPROTO=none
MTU=1500

 完成Redhat註冊

  

1.下載此軟體包以配置subscription-manager以使用rh-satellite。

[root@director ~]$ rpm -Uvh http://
     
     
       /pub/katello-ca-consumer-latest.noarch.rpm 
     
[root@director ~]$ subscription-manager config

2.使用此針對RHEL 7.3的啟用金鑰向rh衛星註冊。 

[root@director ~]$subscription-manager register --org="<ORG>" --activationkey="<KEY>"

3.檢視訂閱。

[root@director ~]$ subscription-manager list –consumed

4.啟用與舊的OSPD重定位相同的儲存庫。

[root@director ~]$ sudo subscription-manager repos --disable=*
[root@director ~]$ subscription-manager repos --enable=rhel-7-server-rpms --enable=rhel-7-server-extras-rpms --enable=rh
el-7-server-openstack-10-rpms --enable=rhel-7-server-rh-common-rpms --enable=rhel-ha-for-rhel-7-server-rpm

5.對系統進行更新，以確保您擁有最新的基本系統軟體包並重新啟動系統。

[root@director ~]$sudo yum update -y
[root@director ~]$sudo reboot

Undercloud Restoration

啟用訂閱後，將備份的undercloud tar檔案undercloud-backup-'date +%F'.tar.gz匯入新的OSPD伺服器根目錄/root。

1.安裝mariadb伺服器。

[root@director ~]$ yum install -y mariadb-server

2.提取MariaDB配置檔案和資料庫備份。以root使用者身份執行此操作。

[root@director ~]$ tar -xzC / -f undercloud-backup-$DATE.tar.gz etc/my.cnf.d/server.cnf
[root@director ~]$ tar -xzC / -f undercloud-backup-$DATE.tar.gz root/undercloud-all-databases.sql

3.編輯/etc/my.cnf.d/server.cnf，註釋掉bind-address條目（如果存在）。

[root@tb3-ospd ~]# vi /etc/my.cnf.d/server.cnf

4.啟動MariaDB服務並暫時更新max_allowed_packet設定：

[root@director ~]$ systemctl start mariadb
[root@director ~]$ mysql -uroot -e"set global max_allowed_packet = 16777216;"

5.清理某些許可權（稍後重新建立）：

[root@director ~]$ for i in ceilometer glance heat ironic keystone neutron nova;do mysql -e "drop user $i";done
[root@director ~]$ mysql -e 'flush privileges'

附註：如果之前已在設定中禁用了ceilometer服務，請執行上述命令刪除「ceilometer」。

6.建立stackuser帳戶。

[root@director ~]$ sudo useradd stack
[root@director ~]$ sudo passwd stack << specify a password
[root@director ~]$ echo "stack ALL=(root) NOPASSWD:ALL" | sudo tee -a /etc/sudoers.d/stack
[root@director ~]$ sudo chmod 0440 /etc/sudoers.d/stack 

7.還原堆疊使用者主目錄。

[root@director ~]$ tar -xzC / -f undercloud-backup-$DATE.tar.gz home/stack

8.安裝swift和一覽式基礎程式包，然後恢復其資料。

[root@director ~]$ yum install -y openstack-glance openstack-swift
[root@director ~]$ tar --xattrs -xzC / -f undercloud-backup-$DATE.tar.gz srv/node var/lib/glance/images

9.確認資料為正確使用者所有。

[root@director ~]$ chown -R swift: /srv/node
[root@director ~]$ chown -R glance: /var/lib/glance/images

10.還原雲下SSL證書（可選 — 僅在安裝程式使用SSL證書時完成）。

[root@director ~]$ tar -xzC / -f undercloud-backup-$DATE.tar.gz etc/pki/instack-certs/undercloud.pem
[root@director ~]$ tar -xzC / -f undercloud-backup-$DATE.tar.gz etc/pki/ca-trust/source/anchors/ca.crt.pem

11.以stackuser身份重新運行底層雲安裝，確保它在堆疊使用者主目錄中運行。

[root@director ~]$ su - stack
[stack@director ~]$ sudo yum install -y python-tripleoclient

12.確認已在/etc/hosts中正確設定主機名。

13.重新安裝雲下。

[stack@director ~]$ openstack undercloud install

<snip>
#############################################################################
Undercloud install complete.

The file containing this installation's passwords is at
/home/stack/undercloud-passwords.conf.

There is also a stackrc file at /home/stack/stackrc.

These files are needed to interact with the OpenStack services, and must be
secured.

#############################################################################

將已恢復的Undercloud重新連線到Overcloud

完成這些步驟後，可以預期底層雲自動恢復其與上層雲的連線。節點將繼續輪詢協調（加熱）以查詢待執行任務，使用每隔幾秒發出的簡單HTTP請求。

驗證已完成還原

使用這些命令對新恢復的環境執行運行狀況檢查。

[root@director ~]$ su - stack
Last Log in: Tue Nov 28 21:27:50 EST 2017 from 10.86.255.201 on pts/0

[stack@director ~]$ source stackrc
[stack@director ~]$ nova list
+--------------------------------------+--------------------+--------+------------+-------------+------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------------------+--------+------------+-------------+------------------------+
| b1f5294a-629e-454c-b8a7-d15e21805496 | pod1-compute-0 | ACTIVE | - | Running | ctlplane=192.200.0.119 |
| 9106672e-ac68-423e-89c5-e42f91fefda1 | pod1-compute-1 | ACTIVE | - | Running | ctlplane=192.200.0.120 |
| b3ed4a8f-72d2-4474-91a1-b6b70dd99428 | pod1-compute-2 | ACTIVE | - | Running | ctlplane=192.200.0.124 |
| 677524e4-7211-4571-ac35-004dc5655789 | pod1-compute-3 | ACTIVE | - | Running | ctlplane=192.200.0.107 |
| 55ea7fe5-d797-473c-83b1-d897b76a7520 | pod1-compute-4 | ACTIVE | - | Running | ctlplane=192.200.0.122 |
| c34c1088-d79b-42b6-9306-793a89ae4160 | pod1-compute-5 | ACTIVE | - | Running | ctlplane=192.200.0.108 |
| 4ba28d8c-fb0e-4d7f-8124-77d56199c9b2 | pod1-compute-6 | ACTIVE | - | Running | ctlplane=192.200.0.105 |
| d32f7361-7e73-49b1-a440-fa4db2ac21b1 | pod1-compute-7 | ACTIVE | - | Running | ctlplane=192.200.0.106 |
| 47c6a101-0900-4009-8126-01aaed784ed1 | pod1-compute-8 | ACTIVE | - | Running | ctlplane=192.200.0.121 |
| 1a638081-d407-4240-b9e5-16b47e2ff6a2 | pod1-compute-9 | ACTIVE | - | Running | ctlplane=192.200.0.112 |
<snip>

[stack@director ~]$ ssh heat-admin@192.200.0.107
[heat-admin@pod1-controller-0 ~]$ sudo pcs status

Cluster name: tripleo_cluster
Stack: corosync
Current DC: pod1-controller-0 (version 1.1.15-11.el7_3.4-e174ec8) - partition with quorum

3 nodes and 22 resources configured

Online: [ pod1-controller-0 pod1-controller-1 pod1-controller-2 ]
Full list of resources:

 ip-10.1.10.10 (ocf::heartbeat:IPaddr2): Started pod1-controller-0
 ip-11.120.0.97 (ocf::heartbeat:IPaddr2): Started pod1-controller-1

 Clone Set: haproxy-clone [haproxy]
 Started: [ pod1-controller-0 pod1-controller-1 pod1-controller-2 ]
 Master/Slave Set: galera-master [galera]

 Masters: [ pod1-controller-0 pod1-controller-1 pod1-controller-2 ]

 ip-192.200.0.106 (ocf::heartbeat:IPaddr2): Started pod1-controller-0
 ip-11.120.0.95 (ocf::heartbeat:IPaddr2): Started pod1-controller-1
 ip-11.119.0.98 (ocf::heartbeat:IPaddr2): Started pod1-controller-0
 ip-11.118.0.92 (ocf::heartbeat:IPaddr2): Started pod1-controller-1

 Clone Set: rabbitmq-clone [rabbitmq]
 Started: [ pod1-controller-0 pod1-controller-1 pod1-controller-2 ]
 Master/Slave Set: redis-master [redis]
 Masters: [ pod1-controller-0 ]
 Slaves: [ pod1-controller-1 pod1-controller-2 ]

 openstack-cinder-volume (systemd:openstack-cinder-volume): Started pod1-controller-0
 my-ipmilan-for-controller-0 (stonith:fence_ipmilan): Stopped
 my-ipmilan-for-controller-1 (stonith:fence_ipmilan): Stopped
 my-ipmilan-for-controller-2 (stonith:fence_ipmilan): Stopped

Failed Actions:

* my-ipmilan-for-controller-0_start_0 on pod1-controller-1 'unknown error' (1): call=190, status=Timed Out, exitreason='none',
 last-rc-change='Wed Nov 22 13:52:45 2017', queued=0ms, exec=20005ms

* my-ipmilan-for-controller-1_start_0 on pod1-controller-1 'unknown error' (1): call=192, status=Timed Out, exitreason='none',
 last-rc-change='Wed Nov 22 13:53:08 2017', queued=0ms, exec=20005ms

* my-ipmilan-for-controller-2_start_0 on pod1-controller-1 'unknown error' (1): call=188, status=Timed Out, exitreason='none',
 last-rc-change='Wed Nov 22 13:52:23 2017', queued=0ms, exec=20004ms

* my-ipmilan-for-controller-0_start_0 on pod1-controller-0 'unknown error' (1): call=210, status=Timed Out, exitreason='none',
 last-rc-change='Wed Nov 22 13:53:08 2017', queued=0ms, exec=20005ms

* my-ipmilan-for-controller-1_start_0 on pod1-controller-0 'unknown error' (1): call=207, status=Timed Out, exitreason='none',
 last-rc-change='Wed Nov 22 13:52:45 2017', queued=0ms, exec=20004ms

* my-ipmilan-for-controller-2_start_0 on pod1-controller-0 'unknown error' (1): call=206, status=Timed Out, exitreason='none',
 last-rc-change='Wed Nov 22 13:52:45 2017', queued=0ms, exec=20006ms

* ip-192.200.0.106_monitor_10000 on pod1-controller-0 'not running' (7): call=197, status=complete, exitreason='none',
 last-rc-change='Wed Nov 22 13:51:31 2017', queued=0ms, exec=0ms

* my-ipmilan-for-controller-0_start_0 on pod1-controller-2 'unknown error' (1): call=183, status=Timed Out, exitreason='none',
 last-rc-change='Wed Nov 22 13:52:23 2017', queued=1ms, exec=20006ms

* my-ipmilan-for-controller-1_start_0 on pod1-controller-2 'unknown error' (1): call=184, status=Timed Out, exitreason='none',
 last-rc-change='Wed Nov 22 13:52:23 2017', queued=0ms, exec=20005ms

* my-ipmilan-for-controller-2_start_0 on pod1-controller-2 'unknown error' (1): call=177, status=Timed Out, exitreason='none',
 last-rc-change='Wed Nov 22 13:52:02 2017', queued=0ms, exec=20005ms

Daemon Status:

 corosync: active/enabled
 pacemaker: active/enabled
 pcsd: active/enabled

[heat-admin@pod1-controller-0 ~]$ sudo ceph status

 cluster eb2bb192-b1c9-11e6-9205-525400330666
 health HEALTH_OK
 monmap e1: 3 mons at {pod1-controller-0=11.118.0.40:6789/0,pod1-controller-1=11.118.0.41:6789/0,pod1-controller-2=11.118.0.42:6789/0}
 election epoch 58, quorum 0,1,2 pod1-controller-0,pod1-controller-1,pod1-controller-2
 osdmap e1398: 12 osds: 12 up, 12 in
 flags sortbitwise,require_jewel_osds
 pgmap v1245812: 704 pgs, 6 pools, 542 GB data, 352 kobjects
 1625 GB used, 11767 GB / 13393 GB avail
 704 active+clean
 client io 21549 kB/s wr, 0 op/s rd, 120 op/s wr

檢查身份服務(Keystone)操作

此步驟通過查詢使用者清單來驗證Identity Service操作。

[stack@director ~]$ source stackrc 
[stack@director ~]$ openstack user list
+----------------------------------+------------------+
| ID | Name |
+----------------------------------+------------------+
| 69ac2b9d89414314b1366590c7336f7d | admin |
| f5c30774fe8f49d0a0d89d5808a4b2cc | glance |
| 3958d852f85749f98cca75f26f43d588 | heat |
| cce8f2b7f1a843a08d0bb295a739bd34 | ironic |
| ce7c642f5b5741b48a84f54d3676b7ee | ironic-inspector |
| a69cd42a5b004ec5bee7b7a0c0612616 | mistral |
| 5355eb161d75464d8476fa0a4198916d | neutron |
| 7cee211da9b947ef9648e8fe979b4396 | nova |
| f73d36563a4a4db482acf7afc7303a32 | swift |
| d15c12621cbc41a8a4b6b67fa4245d03 | zaqar |
| 3f0ed37f95544134a15536b5ca50a3df | zaqar-websocket |
+----------------------------------+------------------+
[stack@director ~]$ 
[stack@director ~]$ source <overcloudrc>
[stack@director ~]$ openstack user list
+----------------------------------+------------+
| ID | Name |
+----------------------------------+------------+
| b4e7954942184e2199cd067dccdd0943 | admin |
| 181878efb6044116a1768df350d95886 | neutron |
| 6e443967ee3f4943895c809dc998b482 | heat |
| c1407de17f5446de821168789ab57449 | nova |
| c9f64c5a2b6e4d4a9ff6b82adef43992 | glance |
| 800e6b1163b74cc2a5fab4afb382f37d | cinder |
| 4cfa5a2a44c44c678025842f080e5f53 | heat-cfn |
| 9b222eeb8a58459bb3bfc76b8fff0f9f | swift |
| 815f3f25bcda49c290e1b56cd7981d1b | core |
| 07c40ade64f34a64932129175150fa4a | gnocchi |
| 0ceeda0bc32c4d46890e53adef9a193d | aodh |
| f3caab060171468592eab376a94967b8 | ceilometer |
+----------------------------------+------------+
[stack@director ~]$ 

上傳影象以供未來節點內省

驗證/httpboot所有這些檔案inspector.ipxe、agent.kernel和agent.ramdisk。如果沒有，請按照以下步驟操作，以在映像中更新。

[stack@director ~]$ ls /httpboot
inspector.ipxe
[stack@director ~]$ source stackrc 
[stack@director ~]$ cd images/
[stack@director images]$ openstack overcloud image upload --image-path /home/stack/images
Image "overcloud-full-vmlinuz" is up-to-date, skipping.
Image "overcloud-full-initrd" is up-to-date, skipping.
Image "overcloud-full" is up-to-date, skipping.
Image "bm-deploy-kernel" is up-to-date, skipping.
Image "bm-deploy-ramdisk" is up-to-date, skipping.
[stack@director images]$ ls /httpboot
agent.kernel agent.ramdisk inspector.ipxe
[stack@director images]$ 

重新啟動圍欄

在OSPD恢復後，圍欄將處於停止狀態。此過程將啟用圍欄。

[heat-admin@pod1-controller-0 ~]$ sudo pcs property set stonith-enabled=true
[heat-admin@pod1-controller-0 ~]$ sudo pcs status
[heat-admin@pod1-controller-0 ~]$sudo pcs stonith show 

相關資訊

• 備份和恢復Director Undercloud

• 隔離控制器節點

• 技術支援與文件 - Cisco Systems