Knowledgebase

1.Organization of data storage in S3 object storage

S3 is not a hierarchical file system, but merely a store of keys and values, although a key is often used as a file path to organize data. To host a file in S3 object storage, you should put it in a bucket. A bucket is an entity for organizing storage.

1.1 The following requirements apply to the name of the bucket

• name length from 3 to 63 characters
• contain only Latin characters
• consist only of lowercase letters, digits, dots (.) and hyphens (-)
• begin and end only with a letter or a number
• does not contain two adjacent dots
• is not in IP address format
• does not begin with the prefix xn--
• does not end with -s3alias

Dots and hyphens are allowed, but not in all cases. This may cause compatibility problems with some advanced functions. We recommend avoiding the use of dots (.)

Examples of bucket names:

Acceptable and compliant:

• my-bucket-name
• mybucket1
• my-bucket-name-2022

Accepted but not recommended:

• docexamplewebsite.com
• www.docexamplewebsite.com
• my.example.s3.bucket

Not allowed:

• doc_example_bucket (содержит подчеркивание)
• DocExampleBucket (содержит заглавные буквы)
• doc-example-bucket- (заканчивается дефисом)

1.2 Bucket restrictions:

• the name must be unique in the entire Object Storage cluster
• for each user in the group, up to 100 bucket can be created
• the maximum size of one upload object is 4 GB; multipart upload can be used for uploading large objects.

1.3 Accessing the bucket

For access you can use HTTP (s3.objstor.cloud4u.com) or HTTPS (s3.objstor.cloud4u.com) protocols

• https://s3.objstor.cloud4u.com/<bucket>?<parameters> 
• http://<bucket>.s3.objstor.cloud4u.com?<parameters>

Access via HTTPS is available only by URI format  

1.4. Organization of data storage in the bucket

S3 storage does not have a hierarchical file system as in UNIX. The file system in the S3 bucket is flat. Also, there is no such thing as a file or directory in S3 storage, there are "objects" here.

However, it is possible to create some sort of file system in S3 bucket by using prefixes and delimiters to organize the stored data.

Each object can be assigned a key name. The key name has a prefix. A prefix is a string of characters at the beginning of the object key name. The prefix can be of any length, depending on the maximum length of the object key name. However, prefixes are not directories. 

Assigning a prefix and a delimiter will help you organize your keys and then review them hierarchically. First, choose a delimiter that will not occur in any object key name you create. You can use any delimiter. For example, " \ ". This is not a unique delimiter, just a common one.

Create object key names hierarchically by connecting all contained levels of the hierarchy, labelling each level with a delimiter. For example, if you store information about cities, you might naturally organise them into continents, countries, provinces. Since city names don't usually have the "\" symbol in their name. You can use it as a separator:

• Country1/Region1/City1
• Country1/Region1/City2
• Country2/Region1/City1

To list only root-level objects in the bucket, you send a GET request to the bucket with a slash delimiter character ( / ).

There is no limit to the number of prefixes in a bucket. A prefix has no fixed number of characters. It is any string between the bucket name and the object name, for example:

bucket/folder1/sub1/file

bucket/folder1/sub2/file

bucket/1/file

bucket/2/file

The prefixes of the object "file" will be as follows: /folder1/sub1/, /folder1/sub2/, /1/, /2/

You can find the requirements for the object name in the article.  

Managing access control in S3 to the bucket and objects

2.1 Creating a bucket

When creating a bucket, you can choose a storage policy that suits your needs ( don't confuse this with bucket access policies, which we will discuss later). The bucket belongs to the USER (owner) account that created it. It is impossible to change the owner of an already created bucket. Only by creating a new bucket and moving data between them. Policies are available in the control panel https://cmc.objstor.cloud4u.com:8443/ 

Let's highlight the main functionalities provided by S3 for controlling access to the bucket:

Acess Control List (ACL)

S3 bucket policies

user-based policies

IAM policies

As well as user entities. 

USER (owner)

IAM USER

Now let's look at their usage

2.2. ACCESS CONTROL LIST (ACL)

It can be applied both to the bucket itself and to an object (file) in the bucket. Assigning public access to the bucket itself is a bad practice and is not recommended. It is more reasonable to apply accesses granularly, at the object (file) level. Rights to a bucket do not prohibit access to an object in the bucket. Before the introduction of Identity and Access Management (IAM), this was the primary means of access control.

S3 has predefined groups:

Authenticated users group

All users group

Log delivery group

And access types:

READ

WRITE

READ_ACP (view current permission settings for btsket)

WRITE_ACP (change the permission settings for the bucket).

FULL_CONTROL (full access)

There are also pre-configured access lists (canned ACLs) that can be assigned to the bucket.

An object can only have one canned ACL attached to it simultaneously. If you apply one canned ACL to an object and then apply another canned ACL, the first canned ACL will be removed.

The table shows the available canned ACLs.

Additional Canned ACL (HyperStore extensions for S3 API)

Also, refer to the AWS documentation on ACLs.

2.3. S3 BUCKET POLICIES

Allow users to easily grant access between accounts without creating IAM roles. CMC does not support the creation of bucket policies; they can only be created via the S3 API or a third party client application.

Suppose we want to allow s3:GetObject on mybucket for the root account and also for the user Alice. This can be done using the mybucket policy below:

{

  "Version": "2012-10-17",

  "Statement": [

    {

      "SID": "Allow",

      "Effect": "Allow",

      "Principal": {

        "AWS": ["arn:aws:iam:: 6161616165:user/Alice",

                "arn:aws:iam:: 6161616161:root"]

      },

      "Action": " s3:GetObject ",

      "Resource": "arn:aws:s3::: mybucket /*"

    }

  ]

}

S3 bucket policies are attached only to S3 bucket policies. They define which actions are allowed or denied to which users in the bucket that the bucket policy is attached to. s3 bucket policies are a type of access control list (in a general sense, don't confuse it with S3 ACL, which is a separate S3 feature). You attach S3 bucket policies at the bucket level, i.e. you cannot attach a bucket policy to an S3 object, but the permissions specified in the bucket policy apply to all objects in the bucket.

Bucket policies are a simple and powerful means of managing access control to your bucket, but it is very important to understand and define the Principal in the example above. If you define it incorrectly and use a wildcard (*), you risk making your entire bucket public when using the example below.

{

  "Version": "2012-10-17",

  "Statement": [

    {

      "SID": "Allow",

      "Effect": "Allow",

      "Principal": “*”,

      "Action": " s3:GetObject ",

      "Resource": "arn:aws:s3::: mybucket /*"

    }

  ]

} 

2.4. USER-BASED POLICIES

User-based policies are often used to control access in S3. For example, to grant user or role access to certain buckets of S3, a user-based policy can be easily defined and attached to the corresponding entity.

For example, the following custom policy allows any entity to which the policy is attached to perform any actions with mybucket. 

{

  "Version": "2012-10-17",

  "Statement": [

    {

      "Effect": "Allow",

      "Action": " s3:* ",

      "Resource": ["arn:aws:s3::: mybucket ",

                   "arn:aws:s3::: mybucket /*"]

    }

  ]

}

2.5. IAM POLICIES

IAM policies define the actions that are allowed or prohibited and on which resources. You attach IAM policies to IAM users, groups, or roles that grant the permissions you define. 

The IAM example in the policy below grants the IAM entity (user, group or role) to which it is attached permission to perform any S3 operation on a bucket named mybucket as well as its contents.

{

  "Version": "2012-10-17",

  "Statement":[{

    "Effect": "Allow",

    "Action": "s3:*",

    "Resource": ["arn:aws:s3:::mybucket",

                 "arn:aws:s3:::mybucket/*"]

    }

  ]

}

Below is an example of a simple IAM policy document granting permission to list the contents of a bucket named mybucket:

{

  "Version": "2012-10-17",

  "Statement": {

    "Effect": "Allow",

    "Action": "s3:ListBucket",

    "Resource": "arn:aws:s3:::mybucket"

  }

}

Note that the S3 bucket policy includes a Principal element that lists the participants for which the bucket policy controls access. The Principal element is not needed in an IAM policy, since Principal is by default the entity to which the IAM policy is attached. S3 bucket policies (as the name implies) only control access to S3 resources, while IAM policies can be applied to other resources in the AWS cloud. 

2.6 When and which policies to apply

Use IAM POLICIES in the following cases:

You need to control access to services, not just S3. It will be easier to manage IAM policies because you can centrally manage all your permissions in IAM, rather than distributing them between IAM and S3.

You have a lot of btsockets in S3 that have different permission requirements. Managing IAM policies will be easier because you don't need to define a large number of S3 btsocket policies, but instead can rely on a smaller number of more detailed IAM policies.

You prefer to store access control policies in an IAM environment.

Use s3 bucket policies, in the following cases:

You need an easy way to grant another account access in your S3 system without roles, or IAM policies.

Your IAM policies are large and run into a size limitation (up to 2 KB for users, 5 KB for groups, and 10 KB for roles). S3 supports bucket policies up to 20 KB.

You prefer to store access control policies in the S3 environment.   

Limitations on policy size

2.7 Supported IAM solutions in Cloudian's S3 system

In https://cmc.objstor.cloud4u.com:8443/(CMC), the creation of IAM policies using JSON or a visual editor is supported.  

In CMC, created IAM users do not have any permissions until you join them to the policy or IAM group to which the policy is applied. IAM users and IAM groups can be created not only through CMC, but also through third-party software that runs on the S3 API. The CMC supports the standard Amazon Web Services IAM policies and most policy elements for granting S3 or IAM service permissions.

IAM support is not 100%, but supports most of the policy elements, actions, resources, and/or condition keys specified in AWS documentation for IAM policy generation.

Note that IAM actions are case sensitive.

For instructions on creating IAM policies for S3 or IAM service permissions, also see the AWS documentation:

Policies and Permissions

IAM JSON Policy Elements Reference

Actions, Resources, and Condition Keys for Amazon S3

Actions, Resources, and Condition Keys for Identity And Access Management

2.8 Policy Ideology and Implementation

Now that we have covered the basic ways to manage access to S3 resources, let us look at how access control decisions are made. Whenever a request to S3 is received, the authorization decision depends on combining all applicable policies: ACLs, S3 bucket policies, user-based policies. 

By default, all S3 resources are private. Only the owner of the resource (the account that created it) can access the resource. The resource owner can additionally grant access permissions to other users by writing an access policy.

In fact, the system proceeds from the ideology of least privilege. Access is granted only if there is no explicit denial and there is an explicit permission. Simply put, if you don't have any permissions, you will not be granted access. And explicit denial always trumps explicit permission.

2.9 Example of granting full access to your IAM bucket to a user with access only from a specific IP using IAM Policies

When you get access to S3 object storage from Cloud4Y, you get your group name (demo-grp-1) as well as the group administrator (demo-user-1). From under the admin account you can create additional users (USER (owner)) in your group.

Log in to https://cmc.objstor.cloud4u.com:8443/ 

From our user, we'll create:  

two buckets: demo-s3-bucket-1, demo-s3-bucket-2, with storage policy C4Y-R3-K41-V3

two iam users: demo-iam-user1, demo-iam-user2, and create Access Key ID and Secret Key ID data for them

Through Create new key create Access Key ID and Secret Key ID (note, you must remember it, because you can get it only when creating a key pair).
 

For a created IAM user, we can assign several Access Key ID and Secret Key ID pairs to the user

demo-iam-policy-1 for demo-iam-user1 with access to demo-s3-bucket-1, demo-s3-bucket-2 and IP restriction from 176.53.XXX.XXX  

jSON

Using "Attach to user" add the user demo-iam-user1.

Now let's check the results using a client that supports S3 protocol (for example, S3 Browser) and make a request from our server 176.53.XXX.XXX. First, let's configure the connection:

Authorize under demo-iam-user1 and access the bucket described in the policy.
 

Log in as demo-iam-user2. Since we haven't assigned any policies, we don't get access. 

If we repeat the above actions from another server not described in the policy, we will not get access. Thus, we have restricted access to the bucket only from authorized addresses, and provided the necessary IAM users with access to the specified bucket.    

2.10 Example of granting read-only access to your bucket to another user from a specified IP address using S3 Bucket Policies  

In the demo-cmc-grp-1 group, in addition to the existing demo-cmc-user-1, create another user demo-cmc-user-2.

The demo-cmc-user-1 user already has a demo-s3-bucket-1 (created earlier), and for the demo-cmc-user-2 user, we will create a demo-s3-bucket-3.

Let's install the CLI following the article and configure two profiles for demo-cmc-user-1 and demo-cmc-user-2. And see the list of buckets, using the commands:

aws s3api --endpoint-url https://s3.objstor.cloud4u.com:443/ --profile demo-cmc-user-1 list-buckets

aws s3api --endpoint-url https://s3.objstor.cloud4u.com:443/ --profile demo-cmc-user-2 list-buckets

remember users' IDs and the buckets they have:

demo-cmc-user-1, ea25fa386cf9a26db2fd8c9650de065d

demo-cmc-user-2, 86357359a3e5deb3daf86d0ead918d08

Now let's give the demo-cmc-user-1 user access only to read and view the contents of the bucket created by the demo-cmc-user-2 user and allow access only from the IP address: 176.53.XXX.XXX. Since we cannot edit Bucket Policies in CMC, we will use the same S3 Browser using S3 API. Let's connect under demo-cmc-user-2 and for the bucket demo-s3-bucket-3, assign a policy:

   
let's connect as demo-cmc-user-1 using S3 Browser and connect an external bucket:

With assigned Actions, we were able to access the bucket, view its contents, and download objects. 

Similarly, we can assign other actions to the bucket.

However, be careful with policies and first test on test bucket and the operation of your software, and then on product bucket. Be especially careful when applying them to already created bucket with many objects. 

Let's imagine that we granted another user access to the bucket not only for reading, but also for writing. With this scheme, when loading objects into an external bucket, the owner of the bucket will be the same user who created it. But the owner of the object inside will be another one - the one who uploaded it.

For initial preparation of a policy template, it is convenient to use AWS Policy Generator and then adjust the obtained result to your requirements. 

Have you tried Virtual cloud servers by Cloud4Y? Not yet?

Leave a request and get a 10-day free test access.