Crie um cluster Kubernetes multi-master com Centos 7 na plataforma de nuvem do Google

4. Ative crony para serviços NTP com este comando.

systemctl enable --now chronyd

2. Verifique o status com este comando.

systemctl status chronyd

3. Verifique as fontes NTP com este comando.

chronyc sources -v

Servidor Bastion

Etapa 1. Verifique as atualizações.

sudo yum check-update

Etapa 2. Instale atualizações se não estiverem atualizadas.

yum update -y

Etapa 3. Instale os utilitários do yum.

yum -y install yum-utils

Etapa 4. Instale o pacote haproxy.

yum install haproxy -y

Etapa 5. Adicione esta configuração aos padrões em /etc/haproxy/haproxy.cfg :

frontend kubernetes
    bind 10.160.x.19:6443
    option tcplog
    mode tcp
    default_backend kubernetes-master-nodes
frontend http_front
    mode http
    bind 10.160.x.19:80
    default_backend http_back
frontend https_front
    mode http
    bind 10.160.x.19:443
    default_backend https_back
backend kubernetes-master-nodes
    mode tcp
    balance roundrobin
    option tcp-check
    server master-1 10.160.x.9:6443 check fall 3 rise 2
    server master-2 10.160.x.10:6443 check fall 3 rise 2
    server master-3 10.160.x.11:6443  check fall 3 rise 2
backend http_back
    mode http
    server master-1 10.160.x.9:6443 check fall 3 rise 2
    server master-2 10.160.x.10:6443 check fall 3 rise 2
    server master-3 10.160.x.11:6443  check fall 3 rise 2
backend https_back
    mode http
    server master-1 10.160.x.9:6443 check fall 3 rise 2
    server master-2 10.160.x.10:6443 check fall 3 rise 2
    server master-3 10.160.x.11:6443  check fall 3 rise 2

listen stats
    bind 10.160.x.19:8080
    mode http
    stats enable
    stats uri /

Etapa 6. Verifique o arquivo de configuração de forma que ele se pareça com este comando vi /etc/haproxy/haproxy.cfg:

---------------------------------------------------------------------
# Example configuration for a possible web application.  See the
# full configuration options online.
#
#   http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
#
#---------------------------------------------------------------------

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    #
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    #
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

frontend kubernetes
    bind 10.160.x.19:6443
    option tcplog
    mode tcp
    default_backend kubernetes-master-nodes
frontend http_front
    mode http
    bind 10.160.x.19:80
    default_backend http_back
frontend https_front
    mode http
    bind 10.160.x.19:443
    default_backend https_back
backend kubernetes-master-nodes
    mode tcp
    balance roundrobin
    option tcp-check
    server master-1 10.160.x.9:6443 check fall 3 rise 2
    server master-2 10.160.x.10:6443 check fall 3 rise 2
    server master-3 10.160.x.11:6443  check fall 3 rise 2
backend http_back
    mode http
    server master-1 10.160.x.9:6443 check fall 3 rise 2
    server master-2 10.160.x.10:6443 check fall 3 rise 2
    server master-3 10.160.x.11:6443  check fall 3 rise 2
backend https_back
    mode http
    server master-1 10.160.x.9:6443 check fall 3 rise 2
    server master-2 10.160.x.10:6443 check fall 3 rise 2
    server master-3 10.160.x.11:6443  check fall 3 rise 2

listen stats
    bind 10.160.x.19:8080
    mode http
    stats enable
    stats uri /

Passo 7. Verifique o status de haproxy:

[root@k8-loadbalancer vapadala]# systemctl status haproxy 
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2022-10-26 08:33:17 UTC; 6s ago
 Main PID: 30985 (haproxy-systemd)
   CGroup: /system.slice/haproxy.service
           ├─30985 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
           ├─30986 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
           └─30987 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds

Oct 26 08:33:17 k8-loadbalancer systemd[1]: Started HAProxy Load Balancer.
Oct 26 08:33:17 k8-loadbalancer haproxy-systemd-wrapper[30985]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /r...pid -Ds
Oct 26 08:33:17 k8-loadbalancer haproxy-systemd-wrapper[30985]: [WARNING] 298/083317 (30986) : config : 'option forwardfor' ignored for frontend 'kube...P mode.
Oct 26 08:33:17 k8-loadbalancer haproxy-systemd-wrapper[30985]: [WARNING] 298/083317 (30986) : config : 'option forwardfor' ignored for backend 'kuber...P mode.
Hint: Some lines were ellipsized, use -l to show in full.
[root@k8-loadbalancer vapadala]# 

Possíveis erros

1. O serviço HAProxy estará em um estado de falha depois que você fizer alterações de configuração no haproxy.cfg. Por exemplo;

[root@k8-loadbalancer vapadala]# systemctl status haproxy 
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2022-10-26 08:29:23 UTC; 3min 44s ago
  Process: 30951 ExecStart=/usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid $OPTIONS (code=exited, status=1/FAILURE)
 Main PID: 30951 (code=exited, status=1/FAILURE)

Oct 26 08:29:23 k8-loadbalancer systemd[1]: Started HAProxy Load Balancer.
Oct 26 08:29:23 k8-loadbalancer haproxy-systemd-wrapper[30951]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /r...pid -Ds
Oct 26 08:29:23 k8-loadbalancer haproxy-systemd-wrapper[30951]: [WARNING] 298/082923 (30952) : config : 'option forwardfor' ignored for frontend 'kube...P mode.
Oct 26 08:29:23 k8-loadbalancer haproxy-systemd-wrapper[30951]: [WARNING] 298/082923 (30952) : config : 'option forwardfor' ignored for backend 'kuber...P mode.
Oct 26 08:29:23 k8-loadbalancer haproxy-systemd-wrapper[30951]: [ALERT] 298/082923 (30952) : Starting frontend kubernetes: cannot bind socket [10.160.x.19:6443].
Oct 26 08:29:23 k8-loadbalancer haproxy-systemd-wrapper[30951]: haproxy-systemd-wrapper: exit, haproxy RC=1.
Oct 26 08:29:23 k8-loadbalancer systemd[1]: haproxy.service: main process exited, code=exited, status=1/FAILURE.
Oct 26 08:29:23 k8-loadbalancer systemd[1]: Unit haproxy.service entered failed state.
Oct 26 08:29:23 k8-loadbalancer systemd[1]: haproxy.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

Resolução

1. Set the boolean value for haproxy_connect_any to true.
2. Restart the haproxy service.
3. Verify the status.

Execução:

[root@k8-loadbalancer vapadala]# setsebool -P haproxy_connect_any=1
[root@k8-loadbalancer vapadala]# systemctl restart haproxy 
[root@k8-loadbalancer vapadala]# systemctl status haproxy 
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2022-10-26 08:33:17 UTC; 6s ago
 Main PID: 30985 (haproxy-systemd)
   CGroup: /system.slice/haproxy.service
           ├─30985 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
           ├─30986 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
           └─30987 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds

Oct 26 08:33:17 k8-loadbalancer systemd[1]: Started HAProxy Load Balancer.
Oct 26 08:33:17 k8-loadbalancer haproxy-systemd-wrapper[30985]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /r...pid -Ds.
Oct 26 08:33:17 k8-loadbalancer haproxy-systemd-wrapper[30985]: [WARNING] 298/083317 (30986) : config : 'option forwardfor' ignored for frontend 'kube...P mode.
Oct 26 08:33:17 k8-loadbalancer haproxy-systemd-wrapper[30985]: [WARNING] 298/083317 (30986) : config : 'option forwardfor' ignored for backend 'kuber...P mode.
Hint: Some lines were ellipsized, use -l to show in full.
[root@k8-loadbalancer vapadala]# 

Instalar Docker nos Nós Mestre e de Trabalho

Etapa 1. Verifique as atualizações.

sudo yum check-update

Etapa 2. Instalar atualizações se não estiverem atualizadas.

yum update -y

Etapa 3. Instale os utilitários do yum.

yum -y install yum-utils

Etapa 4. Instale o Docker.

curl -fsSL https://get.docker.com/ | sh

Etapa 5. Habilitar encaixe.

systemctl enable --now docker

Etapa 6. Iniciar serviço de encaixe.

sudo systemctl start docker

Passo 7. Verifique o status do encaixe.

sudo systemctl status docker

Saída:

[root@kube-master1 vapadala]# sudo systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2022-10-25 10:44:28 UTC; 40s ago
     Docs: https://docs.docker.com
 Main PID: 4275 (dockerd)
    Tasks: 21
   Memory: 35.2M
   CGroup: /system.slice/docker.service
           └─4275 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

Oct 25 10:44:26 kube-master1 dockerd[4275]: time="2022-10-25T10:44:26.128233809Z" level=info msg="scheme \"unix\" not registered, fallback to defaul...dule=grpc.
Oct 25 10:44:26 kube-master1 dockerd[4275]: time="2022-10-25T10:44:26.128251910Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:/...dule=grpc.
Oct 25 10:44:26 kube-master1 dockerd[4275]: time="2022-10-25T10:44:26.128260953Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc.
Oct 25 10:44:27 kube-master1 dockerd[4275]: time="2022-10-25T10:44:27.920336293Z" level=info msg="Loading containers: start."
Oct 25 10:44:28 kube-master1 dockerd[4275]: time="2022-10-25T10:44:28.104357517Z" level=info msg="Default bridge (docker0) is assigned with an IP ad... address".
Oct 25 10:44:28 kube-master1 dockerd[4275]: time="2022-10-25T10:44:28.163830549Z" level=info msg="Loading containers: done."
Oct 25 10:44:28 kube-master1 dockerd[4275]: time="2022-10-25T10:44:28.182833703Z" level=info msg="Docker daemon" commit=03df974 graphdriver(s)=overl...=20.10.20.
Oct 25 10:44:28 kube-master1 dockerd[4275]: time="2022-10-25T10:44:28.182939545Z" level=info msg="Daemon has completed initialization".
Oct 25 10:44:28 kube-master1 systemd[1]: Started Docker Application Container Engine.
Oct 25 10:44:28 kube-master1 dockerd[4275]: time="2022-10-25T10:44:28.208492147Z" level=info msg="API listen on /var/run/docker.sock".
Hint: Some lines were ellipsized, use -l to show in full.
[root@kube-master1 vapadala]# 

Instalar o Kubernetes nos nós mestre e de trabalho

Etapa 1. Adicionar repositório Kubernetes.

cat <
      
      

"gpgcheck = 0" will not verify the authenticity of the package if unsigned. Production environment it is recommended to set "gpgcheck = 1"

Etapa 2. Desabilitado SELinux.

sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

Etapa 3. Instalação Kubernetes.

sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

Etapa 4. Enable Kubelet.

sudo systemctl enable --now kubelet

Etapa 5. Configurar Pod Network.

kubeadm init --control-plane-endpoint "10.160.x.19:6443" --upload-certs

Etapa 6. Geração de token:

Saída para os nós mestre (plano de controle) e de trabalho.

Nó mestre

Token: O plano de controle do Kubernetes foi inicializado com êxito!

Para usar o cluster, execute-o como um usuário regular:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Como alternativa, se você for o usuário root, poderá executar o.

export KUBECONFIG=/etc/kubernetes/admin.conf

Agora você pode implantar uma rede pod no cluster.

Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Agora você pode unir qualquer número dos nós do plano de controle ou dos nós mestres que executam esse comando em cada um como raiz.

Consulte: fluxo de trabalho de união do kubeadm

kubeadm join 10.160.0.19:6443 --token 5fv6ce.h07kyelronuy0mw2 \
        --discovery-token-ca-cert-hash sha256:b5509b6fe784561f3435bf6b909dc8877e567c70921b21e35c464eb61d9527d0 \
        --control-plane --certificate-key 66773b960199ef4530461ef4014e1432066902d4a3dee01669d8622579731701

Observação: observe que a chave do certificado dá acesso a dados sensíveis ao cluster, mantenha-os em segredo!

Como proteção, os certificados carregados são excluídos em duas horas; se necessário, você pode usá-los.

"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Em seguida, você pode unir qualquer número de nós de trabalho e executar isso em cada um como raiz.

kubeadm join 10.160.0.19:6443 --token 5fv6ce.h07kyelronuy0mw2 \
        --discovery-token-ca-cert-hash sha256:b5509b6fe784561f3435bf6b909dc8877e567c70921b21e35c464eb61d9527d0 

Passo 7. Verifique o serviço Core DNS.

[root@kube-master1 vapadala]# kubectl get pods --all-namespaces
NAMESPACE     NAME                                       READY   STATUS    RESTARTS         AGE
kube-system   calico-kube-controllers-59697b644f-v2f22   1/1     Running   0                32m
kube-system   calico-node-gdwr6                          1/1     Running   0                5m54s
kube-system   calico-node-zszbc                          1/1     Running   11 (5m22s ago)   32m
kube-system   calico-typha-6944f58589-q9jxf              1/1     Running   0                32m
kube-system   coredns-565d847f94-cwgj8                   1/1     Running   0                58m
kube-system   coredns-565d847f94-tttpq                   1/1     Running   0                58m
kube-system   etcd-kube-master1                          1/1     Running   0                59m
kube-system   kube-apiserver-kube-master1                1/1     Running   0                59m
kube-system   kube-controller-manager-kube-master1       1/1     Running   0                59m
kube-system   kube-proxy-flgvq                           1/1     Running   0                5m54s
kube-system   kube-proxy-hf5qv                           1/1     Running   0                58m
kube-system   kube-scheduler-kube-master1                1/1     Running   0                59m
[root@kube-master1 vapadala]# 

Etapa 8. Verifique o status do nó mestre.

[root@kube-master1 vapadala]# kubectl get nodes
NAME           STATUS     ROLES           AGE   VERSION
kube-master1   Ready   control-plane   11m   v1.25.3

Para unir vários nós mestres, este comando join é executado nos nós mestres.

kubeadm join 10.160.x.19:6443 --token 5fv6ce.h07kyelronuy0mw2 \
        --discovery-token-ca-cert-hash sha256:b5509b6fe784561f3435bf6b909dc8877e567c70921b21e35c464eb61d9527d0 \
        --control-plane --certificate-key 66773b960199ef4530461ef4014e1432066902d4a3dee01669d8622579731701

Possíveis erros encontrados no momento da geração do token

Erro de CRI

[root@kube-master1 vapadala]# kubeadm init --control-plane-endpoint "10.160.x.19:6443" --upload-certs
[init] Using Kubernetes version: v1.25.3
[preflight] Running pre-flight checks
        [WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR CRI]: container runtime is not running: output: time="2022-10-25T08:56:42Z" level=fatal msg="unable to determine runtime API version: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/run/containerd/containerd.sock: connect: no such file or directory\"", error: exit status 1.
        [ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

Erro de resolução de CRI

Etapa 1. Remova o arquivo config.toml e reinicie o container.

rm -rf /etc/containerd/config.toml
systemctl restart containerd
kubeadm init

Etapa 2. Instalação contida:
Instale o pacote containerd.io dos repositórios oficiais do Docker com esses comandos.

yum install -y yum-utils
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo.
yum install -y containerd.io

Etapa 3. Configurar contêineres:

sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml

Etapa 4. Reinicialização contida:

systemctl restart containerd

Erro FileContent—proc-sys-net-ipv4-ip_forward

[ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1

Erro na resolução FileContent—proc-sys-net-ipv4-ip_forward

Defina o valor ip_forward como 1.

echo 1 > /proc/sys/net/ipv4/ip_forward

O serviço DNS principal não é executado

[root@kube-master1 vapadala]# kubectl get nodes
NAME           STATUS     ROLES           AGE   VERSION
kube-master1   NotReady   control-plane   11m   v1.25.3
[root@kube-master1 vapadala]# kubectl get pods --all-namespaces
NAMESPACE     NAME                                                       READY   STATUS    RESTARTS   AGE
kube-system   coredns-565d847f94-cwgj8                       0/1         Pending   0                12m
kube-system   coredns-565d847f94-tttpq                        0/1         Pending   0                12m
kube-system   etcd-kube-master1                                    1/1         Running   0                12m
kube-system   kube-apiserver-kube-master1                   1/1         Running   0                12m
kube-system   kube-controller-manager-kube-master1  1/1         Running   0                12m
kube-system   kube-proxy-hf5qv                                      1/1        Running   0                 12m
kube-system   kube-scheduler-kube-master1                  1/1        Running   0                 12m
[root@kube-master1 vapadala]# 

Resolução

O DNS central está pendente, o que indica um problema com a rede. Portanto, o Calico precisa ser instalado.

Referência: Calico

Execute estes dois comandos:

curl https://raw.githubusercontent.com/projectcalico/calico/v3.24.3/manifests/calico-typha.yaml -o calico.yaml
kubectl apply -f calico.yaml

Nó do trabalhador

No Nó de Trabalho quando o token é obtido do mestre:

Etapa 1. Habilitar serviço kubelet.

sudo systemctl daemon-reload
sudo systemctl restart docker
sudo systemctl restart kubelet
systemctl enable kubelet.service

Etapa 2. Associe todos os nós de trabalho com o nó mestre com este comando join.

kubeadm join 10.160.x.19:6443 --token 5fv6ce.h07kyelronuy0mw2 \
        --discovery-token-ca-cert-hash sha256:b5509b6fe784561f3435bf6b909dc8877e567c70921b21e35c464eb61d9527d0 

Etapa 3. Se forem encontrados erros relacionados a arquivos ou portas, redefina o kubeadm com este comando.

kubeadm reset

Depois de redefinir, insira o token do nó mestre.

kubeadm join 10.160.x.19:6443 --token 5fv6ce.h07kyelronuy0mw2 \
        --discovery-token-ca-cert-hash sha256:b5509b6fe784561f3435bf6b909dc8877e567c70921b21e35c464eb61d9527d0

Saída final

O cluster agora está formado, verifique-o com o comando kubectl get nodes.

saída de cluster kubernetes de alta disponibilidade

Observação: no momento da formação do cluster, somente o controle dos nós principais é configurado. O host bastion não está configurado como um servidor centralizado para monitorar todos os Kubes no cluster.