Amazon EKS Operations: Difference between revisions

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
 
(71 intermediate revisions by the same user not shown)
Line 8: Line 8:
=Overview=
=Overview=


=Create a Cluster=
=Create and Delete Cluster=


{{Internal|Amazon EKS Create Cluster|Create Cluster}}
{{Internal|Amazon EKS Create and Delete Cluster|Create and Delete Cluster}}


=Cluster Information=
=Cluster Information=
Line 38: Line 38:
If the right region is configured in the profile, there is no need to be specified.
If the right region is configured in the profile, there is no need to be specified.


=Connect to an EKS Cluster with kubectl=
=<span id='Connect_to_an_EKS_Cluster_with_kubectl'></span>kubectl Context=
{{External|https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html}}
{{Internal|Amazon EKS kubectl Context|Amazon EKS kubectl Context}}


Update [[Amazon_EKS_Concepts#.kube.2Fconfig_Configuration|.kube/config]] with the EKS cluster definition as follows:
=Allowing Additional Users to Access the Cluster=
{{External|https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html}}
==<span id='Allow_Role_Access'></span>Allow IAM Role Access==
Individual AWS users (authenticating as particular IAM Users) can be allowed access if an IAM role is "allowed" access to the Kubernetes cluster by associating it with RBAC roles or groups, and then the IAM role is configured to allow IAM users to assume it. This is the preferred solution, because different roles can be associated with different cluster permissions, and the same user can access the cluster with different permissions, by just using a different role.
 
1. Create an IAM role dedicated to cluster access, as described here: [[AWS_Security_Operations#Create_a_Role_to_Delegate_Permission_to_an_IAM_User|Create a Role to Delegate Permission to an IAM User]].
 
2. Update [[Amazon_EKS_Concepts#aws-auth_ConfigMap|aws-auth ConfigMap]] to allow the IAM role to access the Kubernetes cluster. This is done by associating it with a specific set of RBAC permissions, denoted by a group or Kubernetes role:


<syntaxhighlight lang='bash'>
<syntaxhighlight lang='bash'>
aws eks [--region us-east-1] update-kubeconfig --name example-eks-cluster [--alias <context-alias>] [--role-arn arn:aws:iam::999999999999:role/some-role]
kubectl -n kube-system edit cm aws-auth
</syntaxhighlight>
 
<syntaxhighlight lang='yaml'>
apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: arn:aws:iam::...
      username: system:node:{{EC2PrivateDNSName}}
      groups:
      - system:bootstrappers
      - system:nodes
    - rolearn: arn:aws:iam::999999999999:role/playground-eks-cluster-admin
      groups:
      - system:masters
</syntaxhighlight>
 
3. Edit the trust relationship of the IAM role:
<syntaxhighlight lang='yaml'>
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::999999999999:user/some.user",
          "arn:aws:iam::999999999999:user/some.otheruser"
        ]
      },
      "Action": "sts:AssumeRole",
      "Condition": {}
    }
  ]
}
</syntaxhighlight>
</syntaxhighlight>


This command constructs a [[.kube_config#Contexts|Kubernetes context]] with pre-populated server and certificate authority data values for the cluster specified by name. If the right region is configured in the profile, there is no need to be specified. If no alias is used, the default name of the context is the cluster ARN.
==Allow Individual IAM User Access==
The result is to add a new context to .kube/config:
Configuring individual user access directly in [[Amazon_EKS_Concepts#aws-auth_ConfigMap|aws-auth ConfigMap]] is less preferable than [[#Allow_Role_Access|using an IAM role]] for access, for reasons explained in that section.


<syntaxhighlight lang='bash'>
<syntaxhighlight lang='yaml'>
Added new context arn:aws:eks:us-east-1:999999999999:cluster/example-eks-cluster to /Users/testuser/.kube/config
apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    ...
  mapUsers: |
    - userarn: arn:aws:iam::999999999999:user/some.user
      username: some.user
      groups:
      - system:masters
</syntaxhighlight>
</syntaxhighlight>


If no --role-arn option is specified for the aws eks command, kubectl context is configured to accesses the EKS cluster with the default AWS CLI IAM user identity at the time of aws eks execution. This identity can be obtained with [[AWS_Security_Operations#IAM_Information|aws sts get-caller-identity]]. The IAM identity associated with the context can be changed with the --role-arn option. If the --role-arn option is specified, the [[.kube_config#Contexts|Kubernetes context]] will be configured as such that it will not be necessary to explicitly assume the role; kubectl operations in the correct context will simply work. Note that the IAM role used for --role-arn is NOT the [[Amazon_EKS_Concepts#Cluster_Service_Role|cluster service role]], but a completely different role altogether.
==Associate an IAM Role with a Kubernetes User==
This procedure describe defining a [[Kubernetes_Security_Concepts#User|Kubernetes User]] from an IAM Role.


For more details on how the IAM user or role identity is linked to a specific set of RBAC permissions, see [[Amazon_EKS_Concepts#API_Server_User_Management_and_Access_Control|EKS API Server User Management and Access Control]]. Building upon this capability, it is possible to create two different Kuberenetes context that imply to different sets of RBAC permission on the Kubernetes clusters:
1. Create an IAM role dedicated to cluster access, as described here: [[AWS_Security_Operations#Create_a_Role_to_Delegate_Permission_to_an_IAM_User|Create a Role to Delegate Permission to an IAM User]]. Use the following convention when naming it:
<syntaxhighlight lang='text'>
<cluster-name>-eks-namespaced-edit-role
</syntaxhighlight>
 
2. Edit [[Amazon_EKS_Concepts#aws-auth_ConfigMap|aws-auth ConfigMap]] and associate the IAM role with a Kubernetes User:


<syntaxhighlight lang='bash'>
<syntaxhighlight lang='bash'>
aws eks update-kubeconfig --name example-eks-cluster --alias access-with-cluster-admin-permissions --role-arn arn:aws:iam::999999999999:role/eks-clusterrole-cluster-admin
kubectl -n kube-system edit cm aws-auth
aws eks update-kubeconfig --name example-eks-cluster --alias access-with-limited-permissions --role-arn arn:aws:iam::999999999999:role/eks-clusterrole-limited-permissions
</syntaxhighlight>
 
<syntaxhighlight lang='yaml'>
kind: ConfigMap
data:
  mapRoles: |
    - rolearn: arn:aws:iam::999999999999:role/blue-experimental-role
      username: blue-experimental-user
</syntaxhighlight>
</syntaxhighlight>


Switching between Kubernetes contexts is done with [[Kubectl_config#use-context|kubectl config use-context]]:
=EFS CSI Operations=
<syntaxhighlight lang='bash'>
{{Internal|Amazon EFS CSI Operations|EFS CSI Operations}}
kubectl config use-context access-with-cluster-admin-permissions
=EKS Webhook Token Authentication=
{{Internal|EKS Webhook Token Authentication|EKS Webhook Token Authentication}}
=Node Group Operations=
==Create a Node IAM Role==
{{External|https://docs.aws.amazon.com/eks/latest/userguide/create-node-role.html#create-worker-node-role}}
This is the procedure to create a [[Amazon_EKS_Concepts#EKS_Worker_Node_IAM_Role|Node IAM Role]], required during the creation of a Node Group.
 
IAM Console → Roles → Create Role → AWS Service → Choose a use case → Common use cases → EC2 → Next: Permissions
 
* Filter policies: AmazonEKSWorkerNodePolicy → Check "AmazonEKSWorkerNodePolicy"
* Filter policies: AmazonEC2ContainerRegistryReadOnly → Check "AmazonEC2ContainerRegistryReadOnly"
 
The "AmazonEKS_CNI_Policy" must be attached either to this role or to a different role that is mapped to the <code>was-node</code> Kubernetes service account. Assigning the role to the service account is recommended, instead of attaching it to the Node IAM Role. <font color=darkkhaki>Develop this. For the time being, filter by "AmazonEKS_CNI_Policy" and select it.</font>
 
Next: Tags  → Next: Review
 
Role name: blue-node-iam-role.
==Create a New Node Group==
EKS console → Cluster → Configuration → Compute → Add Node Group
 
Name:
 
Node IAM Role - select the role created according to the [[#Create_a_Node_IAM_Role|Create a Node IAM Role]] procedure.
 
Next.
 
'''Node Group compute configuration'''
 
AMI type:
 
Capacity Type: On-Demand
 
Instance type: t3.micro
 
Disk size: 20
 
'''Node Group scaling configuration'''
 
Minimum size: 2
 
Maximum size: 2
 
Desired size: 2
 
'''Node Group update configuration'''
 
Maximum unavailable: Number 1.
 
Next:
 
'''Node Group network configuration'''
 
Subnets: private subnets.
 
If the creation fails and new nodes fail to join, see: {{Internal|EKS Node Group Nodes Not Able to Join the Cluster|EKS Node Group Nodes Not Able to Join the Cluster }}
 
==Scale Up Node Group==
{{External|https://docs.aws.amazon.com/eks/latest/userguide/update-managed-node-group.html}}
 
EKS Console → Amazon EKS Clusters → Select cluster → Compute → Select group → Edit → Minimum/Maximum/Desired size.
 
Scale minimum and desired up.
 
The current nodes will not be removed.


kubectl config current-context
==Delete Node Group==
access-with-cluster-admin-permissions


kubectl config use-context access-with-limited-permissions
Deleting a node group should preserve the state of the cluster and allow the pods to be rescheduled as soon a new node group and new nodes are avaialble.


kubectl config current-context
=Node Operations=
access-with-limited-permissions
==ssh Tunnel into an EKS NodePort Service==
</syntaxhighlight>
{{Internal|Amazon EKS Operations ssh Tunnel into an EKS NodePort Service|ssh Tunnel into an EKS NodePort Service}}


=Load Balancer Troubleshooting=
=Troubleshooting=
{{External|https://aws.amazon.com/premiumsupport/knowledge-center/eks-load-balancers-troubleshooting/}}
==General Troubleshooting==
* EKS Troubleshooting: https://docs.aws.amazon.com/eks/latest/userguide/troubleshooting.html
* <span id='https://kb.novaordis.com/index.php/Amazon_EKS_Operations#Load_Balancer_Troubleshooting'></span>Load balancer troubleshooting: https://aws.amazon.com/premiumsupport/knowledge-center/eks-load-balancers-troubleshooting/  
* Pod connection troubleshooting: https://aws.amazon.com/premiumsupport/knowledge-center/eks-pod-connections/
* How can I get my worker nodes to join my Amazon EKS cluster? https://aws.amazon.com/premiumsupport/knowledge-center/eks-worker-nodes-cluster/
==Node Group Nodes Not Able to Join the Cluster==
{{Internal|EKS Node Group Nodes Not Able to Join the Cluster|Node Group Nodes Not Able to Join the Cluster}}
=="Your current user or role does not have access to Kubernetes objects on this EKS cluster" Message in AWS Console==
The behavior is caused the fact that the user accessing the AWS Console <font color=darkgray>(or any of the roles it is associated with)</font> is not listed in the cluster's [[Amazon_EKS_Concepts#aws-auth_ConfigMap|aws-auth ConfigMap]]. The behavior can be fixed by listing the IAM user in aws-auth ConfigMap as described here: [[#Allow_Individual_IAM_User_Access|Amazon EKS Operations &#124; Allow Individual IAM User Access]].

Latest revision as of 03:09, 10 January 2022

External

Internal

Overview

Create and Delete Cluster

Create and Delete Cluster

Cluster Information

Cluster Status

aws eks [--region us-east-1] describe-cluster --name example-cluster --query "cluster.status"

"ACTIVE"

If the right region is configured in the profile, there is no need to be specified.

Cluster Endpoint

aws eks [--region us-east-1] describe-cluster --name example-cluster --query "cluster.endpoint" --output text

https://FDXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.gr0.us-east-1.eks.amazonaws.com

If the right region is configured in the profile, there is no need to be specified.

Cluster Certificate Authority

aws eks [--region us-east-1] describe-cluster --name example-cluster --query "cluster.certificateAuthority.data" --output text

LS0t...LQo=

If the right region is configured in the profile, there is no need to be specified.

kubectl Context

Amazon EKS kubectl Context

Allowing Additional Users to Access the Cluster

https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html

Allow IAM Role Access

Individual AWS users (authenticating as particular IAM Users) can be allowed access if an IAM role is "allowed" access to the Kubernetes cluster by associating it with RBAC roles or groups, and then the IAM role is configured to allow IAM users to assume it. This is the preferred solution, because different roles can be associated with different cluster permissions, and the same user can access the cluster with different permissions, by just using a different role.

1. Create an IAM role dedicated to cluster access, as described here: Create a Role to Delegate Permission to an IAM User.

2. Update aws-auth ConfigMap to allow the IAM role to access the Kubernetes cluster. This is done by associating it with a specific set of RBAC permissions, denoted by a group or Kubernetes role: 

kubectl -n kube-system edit cm aws-auth

apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: arn:aws:iam::...
      username: system:node:{{EC2PrivateDNSName}}
      groups:
      - system:bootstrappers
      - system:nodes
    - rolearn: arn:aws:iam::999999999999:role/playground-eks-cluster-admin
      groups:
      - system:masters

3. Edit the trust relationship of the IAM role: 

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::999999999999:user/some.user",
          "arn:aws:iam::999999999999:user/some.otheruser"
        ]
      },
      "Action": "sts:AssumeRole",
      "Condition": {}
    }
  ]
}

Allow Individual IAM User Access

Configuring individual user access directly in aws-auth ConfigMap is less preferable than using an IAM role for access, for reasons explained in that section. 

apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    ...
  mapUsers: |
    - userarn: arn:aws:iam::999999999999:user/some.user
      username: some.user
      groups:
      - system:masters

Associate an IAM Role with a Kubernetes User

This procedure describe defining a Kubernetes User from an IAM Role.

1. Create an IAM role dedicated to cluster access, as described here: Create a Role to Delegate Permission to an IAM User. Use the following convention when naming it: 

<cluster-name>-eks-namespaced-edit-role

2. Edit aws-auth ConfigMap and associate the IAM role with a Kubernetes User: 

kubectl -n kube-system edit cm aws-auth

kind: ConfigMap
data:
  mapRoles: |
    - rolearn: arn:aws:iam::999999999999:role/blue-experimental-role
      username: blue-experimental-user

EFS CSI Operations

EFS CSI Operations

EKS Webhook Token Authentication

EKS Webhook Token Authentication

Node Group Operations

Create a Node IAM Role

https://docs.aws.amazon.com/eks/latest/userguide/create-node-role.html#create-worker-node-role

This is the procedure to create a Node IAM Role, required during the creation of a Node Group.

IAM Console → Roles → Create Role → AWS Service → Choose a use case → Common use cases → EC2 → Next: Permissions

  • Filter policies: AmazonEKSWorkerNodePolicy → Check "AmazonEKSWorkerNodePolicy"
  • Filter policies: AmazonEC2ContainerRegistryReadOnly → Check "AmazonEC2ContainerRegistryReadOnly"

The "AmazonEKS_CNI_Policy" must be attached either to this role or to a different role that is mapped to the was-node Kubernetes service account. Assigning the role to the service account is recommended, instead of attaching it to the Node IAM Role. Develop this. For the time being, filter by "AmazonEKS_CNI_Policy" and select it.

Next: Tags → Next: Review

Role name: blue-node-iam-role.

Create a New Node Group

EKS console → Cluster → Configuration → Compute → Add Node Group

Name:

Node IAM Role - select the role created according to the Create a Node IAM Role procedure.

Next.

Node Group compute configuration

AMI type:

Capacity Type: On-Demand

Instance type: t3.micro

Disk size: 20

Node Group scaling configuration

Minimum size: 2

Maximum size: 2

Desired size: 2

Node Group update configuration

Maximum unavailable: Number 1.

Next:

Node Group network configuration

Subnets: private subnets.

If the creation fails and new nodes fail to join, see:

EKS Node Group Nodes Not Able to Join the Cluster

Scale Up Node Group

https://docs.aws.amazon.com/eks/latest/userguide/update-managed-node-group.html

EKS Console → Amazon EKS Clusters → Select cluster → Compute → Select group → Edit → Minimum/Maximum/Desired size.

Scale minimum and desired up.

The current nodes will not be removed.

Delete Node Group

Deleting a node group should preserve the state of the cluster and allow the pods to be rescheduled as soon a new node group and new nodes are avaialble.

Node Operations

ssh Tunnel into an EKS NodePort Service

ssh Tunnel into an EKS NodePort Service

Troubleshooting

General Troubleshooting

Node Group Nodes Not Able to Join the Cluster

Node Group Nodes Not Able to Join the Cluster

"Your current user or role does not have access to Kubernetes objects on this EKS cluster" Message in AWS Console

The behavior is caused the fact that the user accessing the AWS Console (or any of the roles it is associated with) is not listed in the cluster's aws-auth ConfigMap. The behavior can be fixed by listing the IAM user in aws-auth ConfigMap as described here: Amazon EKS Operations | Allow Individual IAM User Access.

Retrieved from "https://kb.novaordis.com/index.php?title=Amazon_EKS_Operations&oldid=88283"