Contents

Introduction

Problem

Analysis

Solution

Introduction

This document describes how to address FailedGetResourceMetric events generated by istiod pods.

Problem

After deploying Subscriber Microservices Infrastructure (SMI) cluster using 2023.01 Cloud Native Deployment Platform (CNDP) release it has been noticed istiod pods keep generating FailedGetResourceMetric events:

cloud-user@ndce-smi-1:~$ kubectl get events -A -o yaml

apiVersion: v1

items:

- apiVersion: v1

count: 467057

eventTime: null

firstTimestamp: "2023-12-31T03:57:42Z"

involvedObject:

apiVersion: autoscaling/v2beta2

kind: HorizontalPodAutoscaler

name: istiod

namespace: istio-system

resourceVersion: "4374"

uid: c85a05a1-b9ec-4241-8347-0b3c9eec1193

kind: Event

lastTimestamp: "2024-03-22T14:42:49Z"

message: 'unable to get metrics for resource cpu: unable to fetch metrics from resource

metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)'

metadata:

creationTimestamp: "2023-12-31T03:57:42Z"

managedFields:

- apiVersion: v1

fieldsType: FieldsV1

fieldsV1:

f:count: {}

f:firstTimestamp: {}

f:involvedObject:

f:apiVersion: {}

f:kind: {}

f:name: {}

f:namespace: {}

f:resourceVersion: {}

f:uid: {}

f:lastTimestamp: {}

f:message: {}

f:reason: {}

f:source:

f:component: {}

f:type: {}

manager: kube-controller-manager

operation: Update

time: "2023-12-31T03:57:42Z"

name: istiod.17a5ce7b410bceb6

namespace: istio-system

resourceVersion: "55452121"

selfLink: /api/v1/namespaces/istio-system/events/istiod.17a5ce7b410bceb6

uid: 9616bdc6-90d1-4475-b6af-6d41e185b434

reason: FailedGetResourceMetric

reportingComponent: ""

reportingInstance: ""

source:

component: horizontal-pod-autoscaler

type: Warning

kind: List

metadata:

resourceVersion: ""

selfLink: ""

Analysis

The issue appears to be due to Horizontal Pod Autoscaler (HPA) is enabled in istiod pods during the deployment. Since HPA requires Metrics API to determine CPU utilization for scaling pods up/down, and Metrics API is not currently used in CNDP managed Kubernetes cluster, then this is a design flaw in istiod deployment.

Solution

HPA has been removed starting 2023.03 CNDP release onwards.