# 06. How to configure HI GIO Kunernetes cluster autoscale

## <mark style="color:green;">**Overview**</mark> <a href="#overview" id="overview"></a>

*Step-by-step guide on how to configure HI GIO Kubernetes cluster autoscale*

* *Install tanzu-cli*
* *Create cluster-autoscaler deployment from tanzu package using tanzu-cli*
* *Enable cluster autoscale for your cluster*
* *Test cluster autoscale*
* *Delete cluster-autoscaler deployment and clean up test resource*

## <mark style="color:green;">**Procedure**</mark> <a href="#procedure" id="procedure"></a>

**Pre-requisites:**

* Ubuntu bastion can connect to your Kubernetes cluster
* Permission for access to your Kubernetes cluster

{% stepper %}
{% step %}
**Step 1: Install tanzu-cli**

```
#Install tanzu-cli to ubuntu
sudo apt update
sudo apt install -y ca-certificates curl gpg
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://storage.googleapis.com/tanzu-cli-installer-packages/keys/TANZU-PACKAGING-GPG-RSA-KEY.gpg | sudo gpg --dearmor -o /etc/apt/keyrings/tanzu-archive-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/tanzu-archive-keyring.gpg] https://storage.googleapis.com/tanzu-cli-installer-packages/apt tanzu-cli-jessie main" | sudo tee /etc/apt/sources.list.d/tanzu.list
sudo apt update
sudo apt install -y tanzu-cli
#Verify tanzu-cli installation
tanzu version
```

<div align="left"><figure><img src="/files/6z1teipsR67Pk2WLDSpd" alt=""><figcaption></figcaption></figure></div>

{% hint style="warning" %}
To install tanzu-cli in other environments, please refer to the documentation below:

[![](https://techdocs.broadcom.com/favicon.ico)Installing and Using VMware Tanzu CLI v1.5.x](https://techdocs.broadcom.com/us/en/vmware-tanzu/cli/tanzu-cli/1-5/cli/index.html)
{% endhint %}

{% hint style="info" %}
(Optional) If you want to configure tanzu completion, please run the command below and follow the instructions output

`tanzu completion --help`
{% endhint %}

<div align="left"><figure><img src="/files/XwbeyHafmSFbm6RWUbls" alt=""><figcaption></figcaption></figure></div>
{% endstep %}

{% step %}
**Step 2: Create cluster-autoscaler deployment from tanzu package using tanzu-cli**

* Switched to your Kubernetes context

```
kubectl config use-context <your context name>
```

<div align="left"><figure><img src="/files/jAfnjjBXXRT5HZlo8Ja8" alt=""><figcaption></figcaption></figure></div>

* List available cluster-autoscaler in tanzu package and note the version name

```
tanzu package available list cluster-autoscaler.tanzu.vmware.com
```

<figure><img src="/files/e5GqoCBF7nEpBXvgp0fH" alt=""><figcaption></figcaption></figure>

* Create kubeconfig secret name `cluster-autoscaler-mgmt-config-secret` in cluster `kube-system` namespace

```
kubectl create secret generic cluster-autoscaler-mgmt-config-secret \
--from-file=value=<path to your kubeconfig file> \
-n kube-system
```

<div align="left"><figure><img src="/files/iPEFtdtlgFZTp4Hf1yLa" alt=""><figcaption></figcaption></figure></div>

{% hint style="warning" %}
Please do not change the secret name (cluster-autoscaler-mgmt-config-secret) and namespace (kube-system)
{% endhint %}

* Create `cluster-autoscaler-values.yaml` file

```
arguments:
  ignoreDaemonsetsUtilization: true
  maxNodeProvisionTime: 15m
  maxNodesTotal: 0 #Leave this value as 0. We will define the max and min number of nodes later.
  metricsPort: 8085
  scaleDownDelayAfterAdd: 10m
  scaleDownDelayAfterDelete: 10s
  scaleDownDelayAfterFailure: 3m
  scaleDownUnneededTime: 10m
clusterConfig:
  clusterName: "demo-autoscale-tkg" #adjust here
  clusterNamespace: "demo-autoscale-tkg-ns" #adjust here
paused: false
```

{% hint style="warning" %}
Required values:

* `clusterName`: your cluster name

* `clusterNamespace`: your cluster namespace
  {% endhint %}

* Install cluster-autoscaler

```
tanzu package install cluster-autoscaler \
--package cluster-autoscaler.tanzu.vmware.com \
--version <version available> \ #adjust the version listed above to match your kubernetes version
--values-file 'cluster-autoscaler-values.yaml' \
--namespace tkg-system #please do not change, this is default namespace for tanzu package
```

<div align="left"><figure><img src="/files/KLoEHacPdImN9r9cYwTt" alt=""><figcaption></figcaption></figure></div>

{% hint style="warning" %}
The cluster-autoscaler will deploy into the `kube-system` namespace.

Run the command below to verify cluster-autoscaler deployment:

`kubectl get deployments.apps -n kube-system cluster-autoscaler`
{% endhint %}

<div align="left"><figure><img src="/files/zeFeDxyCR5wBL64FlsLz" alt=""><figcaption></figcaption></figure></div>

* Configure the minimum and maximum number of nodes in your cluster
  * Get `machinedeployments` name and namespace

```
kubectl get machinedeployments.cluster.x-k8s.io -A
```

<figure><img src="/files/wdkQS4c31hj7bL4HJeX2" alt=""><figcaption></figcaption></figure>

* Set `cluster-api-autoscaler-node-group-min-size` and `cluster-api-autoscaler-node-group-max-size`

```
kubectl annotate machinedeployment <machinedeployment name> cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size=<number min> -n <machinedeployment namespace>
kubectl annotate machinedeployment <machinedeployment name> cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size=<number max> -n <machinedeployment namespace>
```

* Enable cluster autoscale for your cluster

{% hint style="warning" %}
Because this step requires provider permission to perform, please notify the cloud provider to perform this step.
{% endhint %}
{% endstep %}

{% step %}

#### Step 3: **Test cluster autoscale**

* Get the current number of nodes

`kubectl get nodes`

{% hint style="warning" %}
There is currently only one worker node.
{% endhint %}

* Create `test-autoscale.yaml` file

```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
      topologySpreadConstraints: #Spreads pods across different nodes (ensures no node has more pods than others)
      - maxSkew: 1 
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: nginx
```

* Apply `test-autoscale.yaml` file to deploy 2 replicas of nginx pod in the default namespace (it will trigger to create a new worker node)

```
kubectl apply -f test-autoscale.yaml
```

<div align="left"><figure><img src="/files/Da3DLvvKlaC1WqLM3TFu" alt=""><figcaption></figcaption></figure></div>

* Get nginx deployment

```
kubectl get pods
```

<div align="left"><figure><img src="/files/zjXJr3dhdHEsjoNuNjyD" alt=""><figcaption></figcaption></figure></div>

```
kubectl describe pod nginx-589656b9b5-mcm5j | grep -A 10 Events
```

<figure><img src="/files/wmB9QRdwGEWii6l6W6cF" alt=""><figcaption></figcaption></figure>

{% hint style="warning" %}
You can see there is a new nginx pod with a status of Pending and the events shown `FailedScheduling` and `TriggeredScaleUp`:
{% endhint %}

```
Warning  FailedScheduling  2m53s  default-scheduler   0/2 nodes are available: 1 node(s) didn't match pod topology spread constraints, 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }. preemption: 0/2 nodes are available: 1 No preemption victims found for incoming pod, 1 Preemption is not helpful for scheduling.
Normal   TriggeredScaleUp  2m43s  cluster-autoscaler  pod triggered scale-up: [{MachineDeployment/demo-autoscale-tkg-ns/demo-autoscale-tkg-worker-node-pool-1 1->2 (max: 5)}]
```

* Waiting for a new node to be provisioned, then you can see a new worker node has been provisioned and new nginx pod status is running

  <figure><img src="/files/48zQ3JOi1NtCM4L1Xzlh" alt=""><figcaption></figcaption></figure>
* Clean up test resource

```
kubectl delete -f test-autoscale.yaml
```

<div align="left"><figure><img src="/files/EWXJ5JwNviQGm20NFRGb" alt=""><figcaption></figcaption></figure></div>

{% hint style="warning" %}
After deleting the nginx deployment test. The cluster waits a few minutes to delete the unneeded node (please see `scaleDownUnneededTime` value in `cluster-autoscaler-values.yaml` file)
{% endhint %}

* Delete cluster-autoscaler deployment (Optional)

{% hint style="info" %}
In case you don't want your cluster to auto-scale anymore. You can delete cluster-autoscaler deployment using tanzu-cli:

```
tanzu package installed delete cluster-autoscaler -n tkg-system -y
```

{% endhint %}

<figure><img src="/files/x8YmlIGaYZadtgagoI0i" alt=""><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.higiocloud.vn/hi-gio-kubernetes/06.-how-to-configure-hi-gio-kunernetes-cluster-autoscale.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.