Wiki Cloud Avenue
< All Topics

Object storage – Lifecycle

Functionality overview

The lifecycle functionality allows to store objects cost effectively (because storage isn’t free).

A lifecycle configuration is a set of rules defining actions that will apply automatically and repeatedly to a group of objects in a bucket.


Each rule contains a filter that defines to which objects it will apply.


A rule is enabled or disabled.

It contains one or more actions.


A filter contains one of the following:

  • an empty prefix (meaning no filtering at all)
  • a prefix
  • one or more object’s tag value
  • both a prefix and one or more object’s tag value
  • an upper or lower limit for object’s size


Typical use cases:

  • cleaning old logs
  • keeping no more than n versions of object
  • in general controlling the total amount of used storage and therefore its cost


Note : it’s a best practice to setup lifecycle when a bucket has versioning enabled.

Transition actions

They define when objects will be moved to another storage class.

WARNING : currently not supported

Expiration actions

They define how and when objects will be expired (deleted).


The possible actions are:

  • Expiration i.e. delete current versions of object (including expired delete markers optionally)
    • after n Days or after a Date
    • or only remove expired delete markers with ExpiredObjectDeleteMarker
  • NoncurrentVersionExpiration i.e. delete non-current versions of objects
    • after n Days
    • or by keeping only n non-current versions at maximum with NewerNoncurrentVersions WARNING : currently not supported
  • AbortIncompleteMultipartUpload i.e. delete incomplete multipart uploads
    • after n Days (since upload start)


Notes:

  • expiration actions are applied daily
  • therefore deletion may be delayed
  • once objects are queued for deletion, they will be deleted anyway
  • expiration works whatever the versioning state of a bucket (but results differ)

Use cases with AWS CLI

AWS CLI (Command Line Interface) is an open source tool that enables you to configure and use object storage with commands in a text interface (Linux shell or Windows command line).

A user guide is available at https://docs.aws.amazon.com/cli/latest/userguide/.

Prerequisite : none

You only need a bucket before applying a lifecycle configuration to it.

Deleting current versions

Let’s have a look at how Expiration affects current versions (i.e. where metadata IsLatest is true).

Lifecycle configuration

The following lifecycle configuration (json format) is applied to 3 buckets (each in a different state of versioning):

  • the Filter has an empty prefix so all objects (in a bucket) are managed
  • the Status is enabled so the configuration applies
  • the action is Expiration so current versions are deleted
  • the time criteria for action is a Date so it happens after that date (some delay is to be expected)


A more useful time criteria for Expiration action is Days. It allows (current) version expiration after a time period in (current) version’s lifetime is reached.

This would delete current versions when they are 5 days old :

Nonversioned bucket

Metadata VersionId is always null (not needed).

Metadata IsLatest is always true (i.e. it’s always the current version).

Deleting an object is always permanent.

Before
  • bucket contains only 1 object
    • obj1
After
  • bucket is empty

Versioning-enabled bucket

Deleting an object creates a delete marker (with non null version id).

Before
  • bucket contains 4 objects
    • obj1
      • has 1 version (one upload)
    • obj2
      • has 2 versions (the object was uploaded twice): one current, the other not
    • obj3
      • has 2 versions (the object was uploaded then deleted): one non-current (with data) and the other current as a delete marker (no data)
    • obj4
      • has 1 version (the object was uploaded and deleted, then the non-current version was deleted): a delete marker (no data)
After
  • bucket now contains only 3 objects
    • obj1
      • is now deleted: the current version is a delete marker
      • old data remains unmodified with 1 non current version
    • obj2
      • is now deleted: the current version is a delete marker
      • old data remains unmodified with 2 non current versions
    • obj3
      • is unmodified: object is already deleted
      • there’s nothing to do because the current version is a delete marker and other version exists


obj4, on the contrary, had only a single delete marker (also called an “expired” delete marker).

In this case, the delete marker was deleted (removing any history of the object).

Versioning-suspended bucket

Deleting an object creates a delete marker with version id null.

Before
  • bucket contains 6 objects
  • the first 4 objects were set up when the bucket was versioning-enabled
    • obj1
      • has 1 version (one upload)
    • obj2
      • has 2 versions (the object was uploaded twice): one current, the other not
    • obj3
      • has 2 versions (the object was uploaded then deleted): one non-current (with data) and the other current as a delete marker (no data)
    • obj4
      • has 1 version (the object was uploaded and deleted, then the non-current version was deleted): a delete marker (no data)
  • after versioning was suspended, the last 2 objects were set up (VersionId is always set to null)
    • obj5
      • has 1 version (one upload)
    • obj6
      • has 1 version (the object was uploaded then deleted): a delete marker (no data)
      • when versioning is suspended, delete permanently removes the current version
After
  • bucket contains 6 objects
  • remember the first 4 objects were set up when the bucket was versioning-enabled
    • obj1
      • is now deleted: the current version is a delete marker
      • old data remains unmodified with 1 non current version
    • obj2
      • is now deleted: the current version is a delete marker
      • old data remains unmodified with 2 non current versions
    • obj3
      • is unmodified: object is already deleted
      • there’s nothing to do because the current version is a delete marker and other version exists
    • obj4
      • is unmodified: object is already deleted
      • because this delete marker was created when the bucket was versioning-enabled, it has a non null version id
      • because the bucket is now versioning-suspended, it remains unmodified
  • remember the last 2 objects were set up after versioning was suspended (VersionId is always set to null)
    • obj5
      • is now deleted: the current version is a delete marker
      • previous current version with data was deleted
    • obj6
      • is unmodified: object is already deleted

Delete non-current versions

Let’s have a look at how NoncurrentVersionExpiration affects non current versions (i.e. where metadata IsLatest is false).

Lifecycle configuration

The following lifecycle configuration (json format) is applied to 3 buckets (each in a different state of versioning) :

  • the Filter has an empty prefix so all objects (in a bucket) are managed
  • the Status is enabled so the configuration applies
  • the action is Expiration so current versions are deleted
  • the time criteria for action is Days so it happens after non current version lifetime reach a number of days (some delay is to be expected)

Nonversioned bucket

Expiring non current versions is not applicable.

Versioning-enabled bucket

Before

Please refer to #After_2.

After
  • bucket is almost empty (only delete markers)
    • obj1
      • stays deleted: the current version is a delete marker (no change)
      • non current version containing data was deleted
    • obj2
      • stays deleted: the current version is a delete marker (no change)
      • 2 non current versions containing data were deleted
    • obj3
      • stays deleted: the current version is a delete marker (no change)
      • non current version containing data was deleted

Versioning-suspended bucket

WARNING : currently not supported

Other options

Delete incomplete multipart uploads

A multipart upload (upload one object in parts) is a best practice to upload a large object.

It allows for better throughput and quicker recovery from network issues.


Sometimes a multipart upload is accidentally interrupted with no follow-up (whatever the reason).

You can use the AbortIncompleteMultipartUpload action to remove old incomplete multipart uploads after some time ( DaysAfterInitiation ).


Example (unfinished multipart uploads that are 2 days old) :

Delete expired delete markers

When using Expiration action and parameters Days and Date, expired delete markers are deleted.


In order to always delete expired delete markers whatever the timeline, you can use the parameter ExpiredObjectDeleteMarker.

This parameter must be used alone in a rule.


Example :

About security

Never use a root Access Key

With a root AK, you can disable lifecycle, suspend versioning and delete all object’s versions.

Restrict permissions

For everyday use (PUT objects in a bucket), you should apply a policy removing (deny) the following permissions :

  • s3:PutBucketVersioning (to prevent versioning change)
  • s3:DeleteObjectVersion (to prevent any delete of object’s version and delete marker)
  • s3:PutLifeCycleConfiguration (to prevent deleting with a lifecycle configuration)
  • s3:PutBucketLifecycle (same but deprecated)