Amazon S3 Concepts: Difference between revisions

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
 
(11 intermediate revisions by the same user not shown)
Line 15: Line 15:
{{External|[https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingObjects.html Working with Amazon S3 Objects]}}
{{External|[https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingObjects.html Working with Amazon S3 Objects]}}


An object is the fundamental entity stored in Amazon S3. Each object consists of [[#Data|data]], a [[#Key|key]], and [[#Metadata|metadata]].
An object is the fundamental entity stored in Amazon S3. Each object consists of [[#Data|data]], a [[#Key|key]], and [[#Metadata|metadata]]. Every S3 object can be uniquely addressed through a combination of the web service endpoint, bucket name, [[#Key|key]] and optionally a [[#Version_ID|version]]. Assuming that "blue" is the bucket name, and "2021-01-01/picture.jpg" is the key, the object can be accessed as https://blue.s3.amazonaws.com/2021-01-01/picture.jpg


==Data==
==Data==
Line 34: Line 34:
{{External|[https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html#object-metadata Object Metadata]}}
{{External|[https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html#object-metadata Object Metadata]}}


The object metadata is a set of key/value pairs. After the object was uploaded, the object metadata cannot be modified. The only way to modify the metadata is to make a copy of the object and set the metadata.
The object metadata is a set of key/value pairs that describe the object. Metadata include some default pairs: date last modified, and standard HTTP metadata such as Content-Type. Custom metadata can be specified at the time the object is stored. After the object was stored, the object metadata cannot be modified. The only way to modify the metadata is to make a copy of the object and set the metadata.


=<span id='S3_Bucket'></span>Bucket=
=<span id='S3_Bucket'></span>Bucket=


The Amazon S3 data model is a flat structure: you create a bucket, and the bucket stores [[#Objects|objects]]. There is no hierarchy of subbuckets or subfolders; however, you can infer logical hierarchy using key name prefixes and delimiters. The Amazon S3 console supports a concept of folders. The console uses the key name prefixes and the delimiter ('/') to present a folder structure.
The Amazon S3 data model is a flat structure: you create a bucket, and the bucket stores [[#Objects|objects]]. A bucket organizes the Amazon S3 namespace at the highest level. <font color=darkgray>A bucket seems to be associated with a specific AWS account - an unrelated AWS account cannot see the bucket in the console. However, I cannot seem to get the AWS account from the bucket.</font>
 
Buckets are created in a [[#Regions_and_Buckets|specific region]].
 
There is no hierarchy of sub-buckets or subfolders; however, you can infer logical hierarchy using key name prefixes and delimiters. The Amazon S3 console supports a concept of folders. The console uses the key name prefixes and the delimiter ('/') to present a folder structure.


==Bucket ARN==
==Bucket ARN==
Line 57: Line 61:


===Bucket CORS Configuration===
===Bucket CORS Configuration===
=Regions and Buckets=
A [[#Bucket|bucket]] lives in a [[Amazon_AWS_Concepts#Region|region]]. The region may be chosen to optimize latency, minimize costs and address regulatory requirements. Objects stored in a region never leave the region unless they are explicitly transferred them to another region.
=Data Consistency Model=
{{External|https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#CoreConcepts}}
=Presigned URL=
{{External|https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html}}
{{External|https://medium.com/@aidan.hallett/securing-aws-s3-uploads-using-presigned-urls-aa821c13ae8d}}
Example:  https://novaordis.s3.us-west-2.amazonaws.com/testobject?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20211003T065621Z&X-Amz-SignedHeaders=host&X-Amz-Expires=43&X-Amz-Credential=AKIA3999999999999999%2F20211003%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=523a48af48039871145651306016a937c988172578722c48c6f0b5b845bcd109
{{Internal|Generate an AWS S3 Presigned URL with Java JDK|Generate an AWS S3 Presigned URL with Java JDK}}
==Uploading Objects with Presigned URL==
{{External|https://docs.aws.amazon.com/AmazonS3/latest/userguide/PresignedUrlUploadObject.html}}
==Sharing an Object with Presigned URL==
{{External|https://docs.aws.amazon.com/AmazonS3/latest/userguide/ShareObjectPreSignedURL.html}}
=REST Authentication for S3 Requests=
{{External|http://s3.amazonaws.com/doc/s3-developer-guide/RESTAuthentication.html}}

Latest revision as of 21:03, 3 October 2021

External

Internal

Overview

Amazon Simple Storage Service (S3) is an object storage service. It can be used to store and protect any amount of data for a range of use cases, such as websites, mobile applications, backup and restore, archive, enterprise applications, IoT devices and big data analytics. Amazon S3 is designed for 99.999999999% (11 9's) of durability. The S3 service can be integrated and exposed externally via the Amazon API Gateway.

The fundamental storage entity in S3 is the object. Objects are contained in buckets.

Object

Working with Amazon S3 Objects

An object is the fundamental entity stored in Amazon S3. Each object consists of data, a key, and metadata. Every S3 object can be uniquely addressed through a combination of the web service endpoint, bucket name, key and optionally a version. Assuming that "blue" is the bucket name, and "2021-01-01/picture.jpg" is the key, the object can be accessed as https://blue.s3.amazonaws.com/2021-01-01/picture.jpg

Data

The data portion of an object is opaque to Amazon S3.

Key

Object Keys

The object key and its version ID uniquely identify the object in a bucket, and the combination of a bucket, key and version ID uniquely identify globally each object stored in Amazon S3. Every object in a bucket has exactly one key.

Version ID

https://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html

Metadata

Object Metadata

The object metadata is a set of key/value pairs that describe the object. Metadata include some default pairs: date last modified, and standard HTTP metadata such as Content-Type. Custom metadata can be specified at the time the object is stored. After the object was stored, the object metadata cannot be modified. The only way to modify the metadata is to make a copy of the object and set the metadata.

Bucket

The Amazon S3 data model is a flat structure: you create a bucket, and the bucket stores objects. A bucket organizes the Amazon S3 namespace at the highest level. A bucket seems to be associated with a specific AWS account - an unrelated AWS account cannot see the bucket in the console. However, I cannot seem to get the AWS account from the bucket.

Buckets are created in a specific region.

There is no hierarchy of sub-buckets or subfolders; however, you can infer logical hierarchy using key name prefixes and delimiters. The Amazon S3 console supports a concept of folders. The console uses the key name prefixes and the delimiter ('/') to present a folder structure.

Bucket ARN

arn:aws:s3:::bucket-name

Bucket Permissions

By default, all S3 buckets are private and can be accessed only by users that are explicitly granted access.

Bucket Public Access

Bucket Access Control List

Bucket Policy

Operations:

Grant an AWS Account Permissions on a Bucket

Bucket CORS Configuration

Regions and Buckets

A bucket lives in a region. The region may be chosen to optimize latency, minimize costs and address regulatory requirements. Objects stored in a region never leave the region unless they are explicitly transferred them to another region.

Data Consistency Model

https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#CoreConcepts

Presigned URL

https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html
https://medium.com/@aidan.hallett/securing-aws-s3-uploads-using-presigned-urls-aa821c13ae8d

Example: https://novaordis.s3.us-west-2.amazonaws.com/testobject?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20211003T065621Z&X-Amz-SignedHeaders=host&X-Amz-Expires=43&X-Amz-Credential=AKIA3999999999999999%2F20211003%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=523a48af48039871145651306016a937c988172578722c48c6f0b5b845bcd109

Generate an AWS S3 Presigned URL with Java JDK

Uploading Objects with Presigned URL

https://docs.aws.amazon.com/AmazonS3/latest/userguide/PresignedUrlUploadObject.html

Sharing an Object with Presigned URL

https://docs.aws.amazon.com/AmazonS3/latest/userguide/ShareObjectPreSignedURL.html

REST Authentication for S3 Requests

http://s3.amazonaws.com/doc/s3-developer-guide/RESTAuthentication.html