Contents

Introduction

Prerequisites

Requirements

Components Used

Background Information

Problem

Error Logs

Solution

Introduction

This document describes how to recover the MGMTPOSTGRES_SLAVE when it does not form a cluster with the MGMTPOSTGRESS_MASTER.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

- Linux Interface

- Virtual Machine Environment

- postgresql

- Pacemaker/Corosync Configuration System (PCS)

Components Used

The information in this document is based on these software versions:

- CloudCenter version 4.8.1.1

- MGMTPOSTGRES_SLAVE Component

- MGMTPOSTGRES_MASTER Component

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

If there is a failure on both MGMTPOSTGRES components, the MGMTPOSTGRES_SLAVE no longer forms a cluster with the MGMTPOSTGRES_MASTER.

Problem

The MGMTPOSTGRES_SLAVE does not form a cluster with the MGMTPOSTGRES_MASTER. In order to get both MGMTPOSTGRES to form a cluster, MGMTPOSTGRES_SLAVE database needs to be deleted. Then, the database will be recovered from the MGMTPOSTGRES_MASTER.

Error Logs

[root@mgmtpostgres_master etc]# pcs status

Cluster name: cliqrdbcluster

Stack: corosync

Current DC: dbmaster (version 1.1.15-11.e174ec8) – partition with quorum

Last updated: Mon Nov 13 19:15:30 2017                 Last changed: Mon Nov 13 16:59:51 2017 by root via crm_attribute on db master

2 nodes and 3 resources configured

Online: [ dbmaster dbslave ]

Full list of resrouces:

Resrouce Group: VIPGroup

PGMasterVIP             (ocf::heartbeat:IPaddr2):                    Started dbmaster

Master/Slave Set: mspostgresql [pgsql]

Masters: [ dbmaster ]

Stopped: [ dbslave ]

Failed Actions:

* pgsql_start_0 on dbslave ‘unknown error’ (1): call=11, status=Timed Out, exitreason=’none’,

last-rc-change=’Mon Nov 13 18:15:25 2017’, queued-0ms, exec=60003ms

Daemon Status:

corosyn: active/disabled

pacemaker: active/enabled

pcsd: inactive/disabled

Solution

Proceed to recover the MGMTPOSTGRES_SLAVE database in order for the MGMTPOSTGRES to form a cluster.

Step 1. In the MGMTPOSTGRES_MASTER, ensure that the cluster is stopped.

pcs cluster stop

pcs status

Step 2. In MGMTPOSTGRES_SLAVE, delete the existing database.

rm -rf /var/lib/pgsql/9.5/data/*

Step 3. In the MGMTPOSTGRES_MASTER, start the cluster again.

pcs cluster start

pcs status

Step 4. In MGMTPOSTGRES_SLAVE, recover the database from the MGMTPOSTGRES_MASTER.

/usr/pgsql-9.5/bin/pg_basebackup -h <MGMTPOSTGRES_MASTER-IP> -D /var/lib/pgsql/9.5/data/ -U replication -v -P --xlog-method=stream

Step 5. In MGMTPOSTGRES_SLAVE, change the ownership of the recovered database.

chown postgres:postgres -R /var/lib/pgsql/9.5/data/*

Step 6. In MGMTPOSTGRES_SLAVE, start the cluster.

pcs cluster start

pcs cluster status

Step 7. In the MGMTPOSTGRES_MASTER, clean up the resources and check the cluster status.

pcs resource cleanup

pcs cluster status

Step 8. In the MGMTPOSTGRES_MASTER, verify that there is replication (look for the IP in the MGMTPOSTGRES_SLAVE IP).

ps –ef | grep postgr