Prerequisites

• Find the number of cores on the instance

  To find the number of cores, run the following command:

  aws ec2 describe-instance-types \
    --instance-type m6in.16xlarge \
    --query "InstanceTypes[0].VCpuInfo.DefaultCores" \
    --output text

  This value must be substituted for <cpu-cores> in the EC2 run-instances command.

• Find cluster information

  To find the required cluster information, use the following commands:

  • To find the API endpoint:

    aws eks describe-cluster \
      --name xrd-cluster \
      --query "cluster.endpoint" \
      --output text

  • To find the certificate:

    aws eks describe-cluster \
      --name xrd-cluster \
      --query "cluster.certificateAuthority.data" \
      --output text

  • To find the Classless Inter-Domain Routing (CIDR) IP address:

    aws eks describe-cluster \
      --name xrd-cluster \
      --query "cluster.kubernetesNetworkConfig.serviceIpv4Cidr" \
      --output text

• Create a user data file

  Create the user data file by copying the following contents into a file named worker-user-data.yaml.

  
  apiVersion: node.eks.aws/v1alpha1
  kind: NodeConfig
  spec:
    cluster:
      name: xrd-cluster
      apiServerEndpoint: <api-endpoint>
      certificateAuthority: <certificate>
      cidr: <cidr>

  Use the values obtained for <api-endpoint>, <certificate>, and <cidr>.

Bring Up the Worker Node

Bring up the worker node by running the following command:

aws ec2 run-instances \
  --image-id <xrd-ami-id> \
  --count 1 \
  --instance-type m6in.16xlarge \
  --key-name <key-pair-name> \
  --block-device-mappings "DeviceName=/dev/xvda,Ebs={VolumeSize=56}" \
  --iam-instance-profile "Arn=<node-profile-arn>" \
  --network-interfaces "DeleteOnTermination=true,DeviceIndex=0,Groups=<sg-id>, SubnetId=<private-subnet-1>,PrivateIpAddress=10.0.0.10" \
  --cpu-options CoreCount=<cpu-cores>,ThreadsPerCore=1 \
  --tag-specifications "ResourceType=instance,Tags=[{Key=kubernetes.io/cluster/xrd-cluster,Value=owned}]" \
  --user-data file://worker-user-data.yaml

Make a note of the instance id, <worker-instance-id>.

This command brings up an EC2 instance with the following settings:

• A 56-GB primary partition - required to store any process cores that the XRd generates.

• A single interface in the first private subnet with permissions to communicate with the EKS control plane. This interface is used for cluster control plane communications. The assigned IP address is 10.0.0.10.

• One thread per core (SMT or Hyper-Threading turned off). This is to prevent the "noisy neighbor effect" (where processes scheduled on a different logical but same physical core hampers the performance of high priority processes) for the high-performance packet processing threads.

• A tag that is required by EKS to display the node should be allowed to join the cluster.

• A user data file that contains the required NodeConfig for EKS.

The requirements for XRd Control Plane are as follows:

• You can use a smaller (and cheaper) instance type, for example, m5.2xlarge.

• The --cpu-options line is not required.

When using the base Amazon Linux 2023 for EKS AMI, ensure that the user data file is in MIME format with a bash section that runs:

echo "fs.inotify.max_user_instances=64000" >> /etc/sysctl.conf
echo "fs.inotify.max_user_watches=64000" >> /etc/sysctl.conf

Turn off the source or destination check for the instance, by running the following command:

aws ec2 modify-instance-attribute \
  --instance-id <worker-instance-id> \
  --no-source-dest-check

When the worker node is up, check if the worker node is added to the cluster.

# kubectl get nodes
NAME                        STATUS   ROLES    AGE    VERSION
ip-10-0-0-10.ec2.internal   Ready    <none>   1m     v1.31.3-eks-48e63af

Note	If you do not see the worker node, check the EKS configuration steps.

Add Subnets for Data Traffic

The setup for data subnets and security groups vary with each deployment. In this example two subnets are created, and isolated from the internet. One security group is created, which allows all communication inside the security group, but rejects any traffic from outside the security group.

These subnets and the security group can be used for interfaces on multiple worker nodes.

Note	The data traffic subnets must be created in the same Availability Zone (AZ) as the worker node.

Create two subnets for data traffic using the following command:

aws ec2 create-subnet \
  --vpc-id "<vpc-id>" \
  --cidr-block "10.0.100.0/24" \
  --availability-zone "<region>a"
aws ec2 create-subnet \
  --vpc-id <vpc-id> \
  --cidr-block "10.0.101.0/24" \
  --availability-zone "<region>a"

Note	The a after the <region> in the availability zone specification indicates the first subnet in the region.

Make a note of the subnet IDs, <data-subnet-1> and <data-subnet-2>.

Create the Security Group

Create a security group for the data network traffic using the following command:

aws ec2 create-security-group --group-name "xrd-data" \
  --description "Data traffic for XRd EKS Cluster" \
  --vpc-id <vpc-id>

Make a note of the output group ID, <data-sg-id>.

A single egress rule that allows all traffic egress is added when a security group is created, but no ingress rules are added. So, all incoming traffic is dropped. You must add an ingress rule for all traffic originating from the same security group, so that all traffic is allowed within the security group.

aws ec2 authorize-security-group-ingress \
  --group-id <data-sg-id> \
  --source-group <data-sg-id> \
  --protocol all

Create and Attach Interfaces

The following example creates two interfaces on the worker node to be used by XRd, one in each of the two subnets created. Run the following commands to create these interfaces:

aws ec2 create-network-interface \
  --description "XRd worker 1 data 1" \
  --groups <data-sg-id> \
  --private-ip-address "10.0.100.10" \
  --subnet-id <data-subnet-1> \
  --tag-specifications "ResourceType=network-interface,Tags=[{Key=node.k8s.amazonaws.com/no_manage,Value=true}]"
aws ec2 create-network-interface \
  --description "XRd worker 1 data 2" \
  --groups <data-sg-id> \
  --private-ip-address "10.0.101.10" \
  --subnet-id <data-subnet-2> \
  --tag-specifications "ResourceType=network-interface,Tags=[{Key=node.k8s.amazonaws.com/no_manage,Value=true}]"

Make a note of both the network interface IDs, <data-interface-1> and <data-interface-2>.

To attach the interfaces to the worker node, run the following commands:

aws ec2 attach-network-interface \
  --device-index 1 \
  --instance-id <worker-instance-id> \
  --network-interface-id <data-interface-1>
aws ec2 attach-network-interface \
  --device-index 2 \
  --instance-id <worker-instance-id> \
  --network-interface-id <data-interface-2>