Amazon S3 Command Line Operations

Internal

• Amazon S3 Operations

Commands

cp

https://docs.aws.amazon.com/cli/latest/reference/s3/cp.html

Create a Bucket

aws s3 mb s3://bucket-name

List all Buckets

aws s3api list-buckets --query "Buckets[].Name"

List Objects in a Bucket

aws s3

ls

aws s3 ls s3://some-bucket

If a "some-prefix" exists in the bucket, to list all objects under that prefix it must end with a "/"

aws s3 ls s3://some-bucket/some-prefix/

Listing multiple object names that match a name pattern:

aws s3 ls s3://some-bucket/some-prefix/dir1/someth

The following will match something-1, something-2, somethingelse, etc.

aws s3api

aws s3api list-objects --bucket bucket-name

Example:

aws s3api list-objects --bucket thalarion-release-pipeline

{
   "Contents": [
       {
           "LastModified": "2019-03-17T20:26:01.000Z",
           "ETag": "\"901056c645f90d6aa4edf0a98720a2a3\"",
           "StorageClass": "STANDARD",
           "Key": "thalarion/BuildspecP/Uew1P4u",
           "Owner": {
               "DisplayName": "aws-admin-dv1",
               "ID": "ca887c9c7e3f48cf499d19b960e5ee436cb4ce34e035463003b51c21ea6b4499"
           },
           "Size": 395
       },
       {
           "LastModified": "2019-03-17T20:25:40.000Z",
           "ETag": "\"bbaa592d88d0d5024c776eed46fb6370-1\"",
           "StorageClass": "STANDARD",
           "Key": "thalarion/Repository/5CfRwn8.zip",
           "Owner": {
               "DisplayName": "aws-admin-dv1",
               "ID": "1f887c9c7e3f48cf499d195960e5ee436cb4ce3be035463003b51c21ea6b4493"
           },
           "Size": 3156
       }
   ]
}

To return specific attributes:

 aws s3api list-objects --bucket bucket-name --query 'Contents[].{Key: Key}'

Equivalent:

aws s3 ls s3://bucket-name

Copy a File into a Bucket

aws s3 cp file.txt s3://bucket-name[/optional-different-key] [--grants read=uri=http://acs.amazonaws.com/groups/global/AllUsers full=emailaddress=user@example.com]

"file.txt" will become the key of the S3 object corresponding to the file content, unless "optional-different-key" is used as part of the S3 URI.

Copy a File into a Folder

aws s3 cp file.txt s3://bucket-name/folder-name/

aws s3 cp file.txt s3://devops-bucket/jenkins/develop/

Trailing slash is required.

Copy a File from a Bucket

aws [--profile <some-profile>] s3 cp s3://<bucket-name>/path-element-1/path-element-2/<file-name> ./local-file.txt

Copy Recursively the Content of a Folder

aws s3 cp s3://<bucket-name>/<path>/<folder-name> ./local/dir --recursive

The command recursively downloads the content of the specified S3 folder into the specified local directory.

Deleting or Emptying a Bucket

Deleting or Emptying a Bucket

Empty a Bucket

aws s3 rm s3://bucket-name --recursive