Using the Object Storage Service – s3cmd

Presentation

The S3cmd is free software (GNU GPL license) developed by TGRMN Software to enable the use of AWS services through commands in a terminal (Linux shell or Terminal under OSX).

Similar to AWS CLI, S3cmd is, however, very limited in terms of IAM (identities and authorizations). It allows you to bypass the complexity of the S3 API to manipulate buckets and objects.

In particular, it offers a directory synchronization feature to/from S3 storage.

Configuration

Prerequisites:

· Have the current version of S3cmd installed (on Linux and macOS)

– See https://s3tools.org/download+and+https://github.com/s3tools/s3cmd/blob/master/INSTALL.md

Minimum configuration procedure:

• Open a command window.
• Run the command s3cmd –configure and enter:
• Key in the Access Key.
• Key in the Secret Access Key.
• Key in EU as the Default region.
• Key in the hostname for S3 Endpoint (see the service endpoint URL).
• “%(bucket)s.hostname” for DNS-style Bucket+hostname (see previous line)
• Confirm (do not enter anything) for Encryption password
• Confirm (do not enter anything) for Path to GPG program
• Confirm for Use HTTPS protocol (Yes by default)
• Confirm (do not enter anything) for HTTP Proxy server name
• Validate (do not enter anything) for Test access with supplied credentials (Y)
• After the message “Success. Your access key and secret key worked fine”, enter: y to Save the configuration (No by default)

NB: using the EU region does not mean that your data is located outside France; Cloud Avenue data is actually stored in mainland France.

Examples of use

In the following examples, url represents the hostname of your service endpoint accessible via HTTPS. You will need to modify it according to your endpoint (see your login information).

The examples correspond to simple use cases (bucket and object management).

Buckets

List buckets :

$ s3cmd ls
2020-03-17 14:31  s3://comp1
2020-03-18 14:58  s3://comp2

In the example, there are two Buckets (comp1 and comp2).

Listing the contents of a Bucket :

$ s3cmd ls s3://comp1
                       DIR   s3://comp1/backup/
2020-03-17 14:33       102   s3://comp1/mon_texte.txt
2020-03-17 14:33         0   s3://comp1/rienàvoir
2020-03-17 14:36        24   s3://comp1/texte2.txt

In the example, the comp1 bucket contains three files and a folder (or prefix) named backup. The contents of the folder are not listed here.

Recursively listing the contents of a bucket :

$ s3cmd ls s3://comp1 --recursive
2020-03-18 15:04      5401   s3://comp1/backup/archive1.tgz
2020-03-17 14:33       102  s3://comp1/mon_texte.txt
2020-03-17 14:33         0  s3://comp1/rienàvoir
2020-03-17 14:36        24  s3://comp1/texte2.txt

In the example, the comp1 Bucket contains four files, one of which is located in a folder named backup. The –recursive option allows you to list all objects, including those located in folders (or prefixes).

Creating a Bucket :

$ s3cmd mb s3://comp3
Bucket 's3://comp3/' created

$ s3cmd ls
2020-03-17 14:31  s3://comp1
2020-03-18 14:58  s3://comp2
2020-03-18 15:15  s3://comp3

In the example, the empty Bucket comp3 is deleted.

Deleting a Bucket containing at least one object :

$ s3cmd rb s3://comp2
ERROR: S3 error: 409 (BucketNotEmpty): The bucket you tried to delete is not empty.

$ s3cmd rb s3://comp2 --recursive
WARNING: Bucket is not empty. Removing all the objects from it first. This may take some time...
delete: 's3://comp2/bigfile.zip'
Bucket 's3://comp2/' removed

In the example, the comp2 partition is not empty; the –recursive option is used to automatically delete all objects and folders present.

Objects

Copying an object to a Bucket :

$ s3cmd put ./archive1.tgz s3://comp3
upload: './archive1.tgz' -> 's3://comp3/archive1.tgz'  [1 of 1]
5401 of 5401   100% in   0s    22.96 kB/s  done

$ s3cmd ls s3://comp3
2020-03-18 15:26      5401   s3://comp3/archive1.tgz

In the example, the local file is copied (uploaded) to the comp3 Bucket.

Copying an object to a folder :

s3cmd put ./help.txt s3://comp3/documentation/
upload: './help.txt' -> 's3://comp3/documentation/help.txt'  [1 of 1]
17946 of 17946   100% in   0s    72.44 kB/s  done

$ s3cmd ls s3://comp3 --recursive
2020-03-18 15:26      5401   s3://comp3/archive1.tgz
2020-03-18 15:34     17946  s3://comp3/documentation/help.txt

In the example, the documentation folder is created if it does not already exist in the Bucket, then the local file is copied (uploaded) to the folder.

Renaming an object :

$ s3cmd mv s3://comp3/documentation/help.txt s3://comp3/documentation/helpv2.txt
move: 's3://comp3/documentation/help.txt' -> 's3://comp3/documentation/helpv2.txt'

In the example, the file fic2.txt is renamed fic2bis.txt.

Moving an object :

$ s3cmd mv s3://comp3/documentation/helpv2.txt s3://comp1
move: 's3://comp3/documentation/helpv2.txt' -> 's3://comp1/helpv2.txt'

Moving an object by renaming it :

$ s3cmd mv s3://comp1/helpv2.txt s3://comp3/documentation/importhelpv2.txt
move: 's3://comp1/helpv2.txt' -> 's3://comp3/documentation/importhelpv2.txt'

Deleting an object :

$ s3cmd del s3://comp3/documentation/importhelpv2.txt
delete: 's3://comp3/documentation/importhelpv2.txt'

$ s3cmd ls s3://comp3 --recursive
2020-03-18 15:26      5401   s3://comp3/archive1.tgz

In the example, the object is deleted; the documentation folder (or prefix) is empty; it is automatically deleted.

Deleting a folder or a prefix:

$ s3cmd del s3://comp1/backup --recursive
delete: 's3://comp1/backup/archive1.tgz'
delete: 's3://comp1/backup/mon_texte.txt'
delele: 's3://comp1/backup/texte2.txt'

$ s3cmd ls s3://comp1 --recursive
2020-03-17 14:33         0   s3://comp1/rienàvoir

In the example, the deletion applies to the folder; the –recursive option causes all objects present to be deleted.