Amazon EKS Operations: Difference between revisions

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
 
(95 intermediate revisions by the same user not shown)
Line 7: Line 7:


=Overview=
=Overview=
=Create and Delete Cluster=
{{Internal|Amazon EKS Create and Delete Cluster|Create and Delete Cluster}}


=Cluster Information=
=Cluster Information=


==Cluster Status==
<syntaxhighlight lang='bash'>
aws eks [--region us-east-1] describe-cluster --name example-cluster --query "cluster.status"
"ACTIVE"
</syntaxhighlight>
If the right region is configured in the profile, there is no need to be specified.
==Cluster Endpoint==
<syntaxhighlight lang='bash'>
aws eks [--region us-east-1] describe-cluster --name example-cluster --query "cluster.endpoint" --output text
https://FDXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.gr0.us-east-1.eks.amazonaws.com
</syntaxhighlight>
If the right region is configured in the profile, there is no need to be specified.
==Cluster Certificate Authority==
<syntaxhighlight lang='bash'>
<syntaxhighlight lang='bash'>
aws eks --region us-east-1 describe-cluster --name example-cluster --query "cluster.status"
aws eks [--region us-east-1] describe-cluster --name example-cluster --query "cluster.certificateAuthority.data" --output text
 
LS0t...LQo=
</syntaxhighlight>
</syntaxhighlight>
If the right region is configured in the profile, there is no need to be specified.


=Connect to an EKS Cluster with kubectl=
=<span id='Connect_to_an_EKS_Cluster_with_kubectl'></span>kubectl Context=
{{Internal|Amazon EKS kubectl Context|Amazon EKS kubectl Context}}


Update [[Amazon_EKS_Concepts#.kube.2Fconfig_Configuration|.kube/config]] with the EKS cluster definition as follows:
=Allowing Additional Users to Access the Cluster=
{{External|https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html}}
==<span id='Allow_Role_Access'></span>Allow IAM Role Access==
Individual AWS users (authenticating as particular IAM Users) can be allowed access if an IAM role is "allowed" access to the Kubernetes cluster by associating it with RBAC roles or groups, and then the IAM role is configured to allow IAM users to assume it. This is the preferred solution, because different roles can be associated with different cluster permissions, and the same user can access the cluster with different permissions, by just using a different role.
 
1. Create an IAM role dedicated to cluster access, as described here: [[AWS_Security_Operations#Create_a_Role_to_Delegate_Permission_to_an_IAM_User|Create a Role to Delegate Permission to an IAM User]].
 
2. Update [[Amazon_EKS_Concepts#aws-auth_ConfigMap|aws-auth ConfigMap]] to allow the IAM role to access the Kubernetes cluster. This is done by associating it with a specific set of RBAC permissions, denoted by a group or Kubernetes role:


<syntaxhighlight lang='bash'>
<syntaxhighlight lang='bash'>
aws eks --region us-east-1 update-kubeconfig --name example-eks-cluster
kubectl -n kube-system edit cm aws-auth
</syntaxhighlight>
 
<syntaxhighlight lang='yaml'>
apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: arn:aws:iam::...
      username: system:node:{{EC2PrivateDNSName}}
      groups:
      - system:bootstrappers
      - system:nodes
    - rolearn: arn:aws:iam::999999999999:role/playground-eks-cluster-admin
      groups:
      - system:masters
</syntaxhighlight>
 
3. Edit the trust relationship of the IAM role:
<syntaxhighlight lang='yaml'>
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::999999999999:user/some.user",
          "arn:aws:iam::999999999999:user/some.otheruser"
        ]
      },
      "Action": "sts:AssumeRole",
      "Condition": {}
    }
  ]
}
</syntaxhighlight>
 
==Allow Individual IAM User Access==
Configuring individual user access directly in [[Amazon_EKS_Concepts#aws-auth_ConfigMap|aws-auth ConfigMap]] is less preferable than [[#Allow_Role_Access|using an IAM role]] for access, for reasons explained in that section.
 
<syntaxhighlight lang='yaml'>
apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    ...
  mapUsers: |
    - userarn: arn:aws:iam::999999999999:user/some.user
      username: some.user
      groups:
      - system:masters
</syntaxhighlight>
 
==Associate an IAM Role with a Kubernetes User==
This procedure describe defining a [[Kubernetes_Security_Concepts#User|Kubernetes User]] from an IAM Role.
 
1. Create an IAM role dedicated to cluster access, as described here: [[AWS_Security_Operations#Create_a_Role_to_Delegate_Permission_to_an_IAM_User|Create a Role to Delegate Permission to an IAM User]]. Use the following convention when naming it:
<syntaxhighlight lang='text'>
<cluster-name>-eks-namespaced-edit-role
</syntaxhighlight>
</syntaxhighlight>


The result is to add a new context to ./kube/config:
2. Edit [[Amazon_EKS_Concepts#aws-auth_ConfigMap|aws-auth ConfigMap]] and associate the IAM role with a Kubernetes User:


<syntaxhighlight lang='bash'>
<syntaxhighlight lang='bash'>
Added new context arn:aws:eks:us-east-1:999999999999:cluster/example-eks-cluster to /Users/testuser/.kube/config
kubectl -n kube-system edit cm aws-auth
</syntaxhighlight>
 
<syntaxhighlight lang='yaml'>
kind: ConfigMap
data:
  mapRoles: |
    - rolearn: arn:aws:iam::999999999999:role/blue-experimental-role
      username: blue-experimental-user
</syntaxhighlight>
</syntaxhighlight>


The default name of the context is the name of the IAM role, but it can be changed to make it more wieldy.
=EFS CSI Operations=
{{Internal|Amazon EFS CSI Operations|EFS CSI Operations}}
=EKS Webhook Token Authentication=
{{Internal|EKS Webhook Token Authentication|EKS Webhook Token Authentication}}
=Node Group Operations=
==Create a Node IAM Role==
{{External|https://docs.aws.amazon.com/eks/latest/userguide/create-node-role.html#create-worker-node-role}}
This is the procedure to create a [[Amazon_EKS_Concepts#EKS_Worker_Node_IAM_Role|Node IAM Role]], required during the creation of a Node Group.
 
IAM Console → Roles → Create Role → AWS Service → Choose a use case → Common use cases → EC2 → Next: Permissions
 
* Filter policies: AmazonEKSWorkerNodePolicy → Check "AmazonEKSWorkerNodePolicy"
* Filter policies: AmazonEC2ContainerRegistryReadOnly → Check "AmazonEC2ContainerRegistryReadOnly"
 
The "AmazonEKS_CNI_Policy" must be attached either to this role or to a different role that is mapped to the <code>was-node</code> Kubernetes service account. Assigning the role to the service account is recommended, instead of attaching it to the Node IAM Role. <font color=darkkhaki>Develop this. For the time being, filter by "AmazonEKS_CNI_Policy" and select it.</font>
 
Next: Tags  → Next: Review
 
Role name: blue-node-iam-role.
==Create a New Node Group==
EKS console → Cluster → Configuration → Compute → Add Node Group
 
Name:
 
Node IAM Role - select the role created according to the [[#Create_a_Node_IAM_Role|Create a Node IAM Role]] procedure.
 
Next.
 
'''Node Group compute configuration'''
 
AMI type:
 
Capacity Type: On-Demand
 
Instance type: t3.micro
 
Disk size: 20
 
'''Node Group scaling configuration'''
 
Minimum size: 2
 
Maximum size: 2
 
Desired size: 2
 
'''Node Group update configuration'''
 
Maximum unavailable: Number 1.
 
Next:
 
'''Node Group network configuration'''
 
Subnets: private subnets.
 
If the creation fails and new nodes fail to join, see: {{Internal|EKS Node Group Nodes Not Able to Join the Cluster|EKS Node Group Nodes Not Able to Join the Cluster }}
 
==Scale Up Node Group==
{{External|https://docs.aws.amazon.com/eks/latest/userguide/update-managed-node-group.html}}
 
EKS Console → Amazon EKS Clusters → Select cluster → Compute → Select group → Edit → Minimum/Maximum/Desired size.
 
Scale minimum and desired up.
 
The current nodes will not be removed.
 
==Delete Node Group==
 
Deleting a node group should preserve the state of the cluster and allow the pods to be rescheduled as soon a new node group and new nodes are avaialble.
 
=Node Operations=
==ssh Tunnel into an EKS NodePort Service==
{{Internal|Amazon EKS Operations ssh Tunnel into an EKS NodePort Service|ssh Tunnel into an EKS NodePort Service}}
 
=Troubleshooting=
==General Troubleshooting==
* EKS Troubleshooting: https://docs.aws.amazon.com/eks/latest/userguide/troubleshooting.html
* <span id='https://kb.novaordis.com/index.php/Amazon_EKS_Operations#Load_Balancer_Troubleshooting'></span>Load balancer troubleshooting: https://aws.amazon.com/premiumsupport/knowledge-center/eks-load-balancers-troubleshooting/
* Pod connection troubleshooting: https://aws.amazon.com/premiumsupport/knowledge-center/eks-pod-connections/
* How can I get my worker nodes to join my Amazon EKS cluster? https://aws.amazon.com/premiumsupport/knowledge-center/eks-worker-nodes-cluster/
==Node Group Nodes Not Able to Join the Cluster==
{{Internal|EKS Node Group Nodes Not Able to Join the Cluster|Node Group Nodes Not Able to Join the Cluster}}
=="Your current user or role does not have access to Kubernetes objects on this EKS cluster" Message in AWS Console==
The behavior is caused the fact that the user accessing the AWS Console <font color=darkgray>(or any of the roles it is associated with)</font> is not listed in the cluster's [[Amazon_EKS_Concepts#aws-auth_ConfigMap|aws-auth ConfigMap]]. The behavior can be fixed by listing the IAM user in aws-auth ConfigMap as described here: [[#Allow_Individual_IAM_User_Access|Amazon EKS Operations &#124; Allow Individual IAM User Access]].

Latest revision as of 03:09, 10 January 2022

External

Internal

Overview

Create and Delete Cluster

Create and Delete Cluster

Cluster Information

Cluster Status

aws eks [--region us-east-1] describe-cluster --name example-cluster --query "cluster.status"

"ACTIVE"

If the right region is configured in the profile, there is no need to be specified.

Cluster Endpoint

aws eks [--region us-east-1] describe-cluster --name example-cluster --query "cluster.endpoint" --output text

https://FDXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.gr0.us-east-1.eks.amazonaws.com

If the right region is configured in the profile, there is no need to be specified.

Cluster Certificate Authority

aws eks [--region us-east-1] describe-cluster --name example-cluster --query "cluster.certificateAuthority.data" --output text

LS0t...LQo=

If the right region is configured in the profile, there is no need to be specified.

kubectl Context

Amazon EKS kubectl Context

Allowing Additional Users to Access the Cluster

https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html

Allow IAM Role Access

Individual AWS users (authenticating as particular IAM Users) can be allowed access if an IAM role is "allowed" access to the Kubernetes cluster by associating it with RBAC roles or groups, and then the IAM role is configured to allow IAM users to assume it. This is the preferred solution, because different roles can be associated with different cluster permissions, and the same user can access the cluster with different permissions, by just using a different role.

1. Create an IAM role dedicated to cluster access, as described here: Create a Role to Delegate Permission to an IAM User.

2. Update aws-auth ConfigMap to allow the IAM role to access the Kubernetes cluster. This is done by associating it with a specific set of RBAC permissions, denoted by a group or Kubernetes role:

kubectl -n kube-system edit cm aws-auth
apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: arn:aws:iam::...
      username: system:node:{{EC2PrivateDNSName}}
      groups:
      - system:bootstrappers
      - system:nodes
    - rolearn: arn:aws:iam::999999999999:role/playground-eks-cluster-admin
      groups:
      - system:masters

3. Edit the trust relationship of the IAM role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::999999999999:user/some.user",
          "arn:aws:iam::999999999999:user/some.otheruser"
        ]
      },
      "Action": "sts:AssumeRole",
      "Condition": {}
    }
  ]
}

Allow Individual IAM User Access

Configuring individual user access directly in aws-auth ConfigMap is less preferable than using an IAM role for access, for reasons explained in that section.

apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    ...
  mapUsers: |
    - userarn: arn:aws:iam::999999999999:user/some.user
      username: some.user
      groups:
      - system:masters

Associate an IAM Role with a Kubernetes User

This procedure describe defining a Kubernetes User from an IAM Role.

1. Create an IAM role dedicated to cluster access, as described here: Create a Role to Delegate Permission to an IAM User. Use the following convention when naming it:

<cluster-name>-eks-namespaced-edit-role

2. Edit aws-auth ConfigMap and associate the IAM role with a Kubernetes User:

kubectl -n kube-system edit cm aws-auth
kind: ConfigMap
data:
  mapRoles: |
    - rolearn: arn:aws:iam::999999999999:role/blue-experimental-role
      username: blue-experimental-user

EFS CSI Operations

EFS CSI Operations

EKS Webhook Token Authentication

EKS Webhook Token Authentication

Node Group Operations

Create a Node IAM Role

https://docs.aws.amazon.com/eks/latest/userguide/create-node-role.html#create-worker-node-role

This is the procedure to create a Node IAM Role, required during the creation of a Node Group.

IAM Console → Roles → Create Role → AWS Service → Choose a use case → Common use cases → EC2 → Next: Permissions

  • Filter policies: AmazonEKSWorkerNodePolicy → Check "AmazonEKSWorkerNodePolicy"
  • Filter policies: AmazonEC2ContainerRegistryReadOnly → Check "AmazonEC2ContainerRegistryReadOnly"

The "AmazonEKS_CNI_Policy" must be attached either to this role or to a different role that is mapped to the was-node Kubernetes service account. Assigning the role to the service account is recommended, instead of attaching it to the Node IAM Role. Develop this. For the time being, filter by "AmazonEKS_CNI_Policy" and select it.

Next: Tags → Next: Review

Role name: blue-node-iam-role.

Create a New Node Group

EKS console → Cluster → Configuration → Compute → Add Node Group

Name:

Node IAM Role - select the role created according to the Create a Node IAM Role procedure.

Next.

Node Group compute configuration

AMI type:

Capacity Type: On-Demand

Instance type: t3.micro

Disk size: 20

Node Group scaling configuration

Minimum size: 2

Maximum size: 2

Desired size: 2

Node Group update configuration

Maximum unavailable: Number 1.

Next:

Node Group network configuration

Subnets: private subnets.

If the creation fails and new nodes fail to join, see:

EKS Node Group Nodes Not Able to Join the Cluster

Scale Up Node Group

https://docs.aws.amazon.com/eks/latest/userguide/update-managed-node-group.html

EKS Console → Amazon EKS Clusters → Select cluster → Compute → Select group → Edit → Minimum/Maximum/Desired size.

Scale minimum and desired up.

The current nodes will not be removed.

Delete Node Group

Deleting a node group should preserve the state of the cluster and allow the pods to be rescheduled as soon a new node group and new nodes are avaialble.

Node Operations

ssh Tunnel into an EKS NodePort Service

ssh Tunnel into an EKS NodePort Service

Troubleshooting

General Troubleshooting

Node Group Nodes Not Able to Join the Cluster

Node Group Nodes Not Able to Join the Cluster

"Your current user or role does not have access to Kubernetes objects on this EKS cluster" Message in AWS Console

The behavior is caused the fact that the user accessing the AWS Console (or any of the roles it is associated with) is not listed in the cluster's aws-auth ConfigMap. The behavior can be fixed by listing the IAM user in aws-auth ConfigMap as described here: Amazon EKS Operations | Allow Individual IAM User Access.