What Is The Minimum File Size That I Can Store On S3?

As part of the deject advisory we provide to our customers at DoiT International , I occasionally come across extreme cases where customer attempts at saving costs really lead to an increase in S3 storage costs. The post below explains how to make certain this doesn't happen.

When storing files (objects) in an S3 bucket, the post-obit storage classes are available: Standard(default), Infrequent Admission (also chosen Standard-IA), Intelligent-Tiering, Glacier, and S3 Outposts.

The storage course is fix when uploading, or at any later step. 1 of the mutual practices is to create a lifecycle rule in a bucket that ascertain actions that you want Amazon S3 to take during an object's lifetime (for example, change the storage class of the file from Standard to Infrequent Access after xxx days from the creation of the file).

Exceptional Access storage allows you to shop files at a cheaper cost than a Standard grade. The download speed, storage durability, and latency are the same every bit the Standard class, but, the toll of downloading the file is higher than Standard. As such, the appropriately-named Infrequent Access is suitable for storing files that are not accessed frequently.

1 dd17xhb86cwmsi dskmvaw — Lifecycle rule changing the files from **Standard** storage class to **Intelligent tier**. The transition cost is $2,000. After 5 months from the transition, the customer volition pay less for storage.

Common S3 Cost Pitfalls

2 common issues can increment the storage cost of a bucket:

Well-nigh people are unaware that switching from one storage form to another costs money. AWS charges $0.01 for every k transitions from the Standard storage class to the Exceptional Admission class.
In some S3 storage classes, there is minimum file size. In the instance of the Exceptional Access storage class, the minimum file size is 128Kb.

1 a7plfpy0byihe5 n3yzmug — S3 Storage classes specifications

These two issues can bring about situations where a customer with hundreds of millions of small files in an S3 saucepan will accept to pay a loftier one-time transition fee on tiptop of increasing the saucepan storage size (due to a minimum accuse size per file is 128Kb).

How many files do I take in my bucket?

There are two means to cheque the number of files and the size of the bucket:

At the file level — Using the S3 Inventory, y'all can create a written report (generated on a daily basis) that shows the listing of files in the bucket, file size, and file storage class.
At the saucepan Level — Using Cloudwatch metrics, yous tin see the number of files and the size of the saucepan. These metrics are also available from the S3 console, under Management — > Metrics.

1 0k8q dxa4nojayfern3f5w — S3 bucket size

1 fibo2atjcpv7idxrazzj9q — Number of files in S3 Saucepan

When should I set a lifecycle rule?

Now that we know how to observe the number and size of all the files in our bucket, nosotros need to do some straightforward calculations.

Begin by calculating the (non-weighted) boilerplate file size past dividing the S3 Bucket size by the number of files.

For instance, in the bucket, I used to take the screenshots displayed throughout this article the bucket size is 22.1Tb and at that place are six.3 million files in the saucepan resulting in an average file size of 3.42Mb.

If the saucepan'south average file size is greater than 1Mb, it'due south worthwhile to create a lifecycle dominion that volition transition all the files in the saucepan to the Infrequent Access.

If the boilerplate file size is less than 1Mb, I recommend taking a different approach. Keeping in mind that for the Exceptional Admission storage grade the minimal cost is for a file of 128Kb, if the bucket has a large number of files with each file smaller than 128Kb, switching to Exceptional Access will upshot in an increase in storage costs. In this example, a lifecycle rule is non recommended, and it might be smarter to manually review the S3 Inventory report and modify the storage form independently.

Calculating the lifecycle transition toll and toll-effectiveness

Following the screenshots example given above, storing this 22.1Tb bucket in the u.s.-east-1 region in Standard class would cost (22.1Tb x 1024[converting to Gb]10 $0.023 [per Gb]) $520.v per month.

If we change the storage course using the lifecycle dominion to Infrequent Access for this 22.1Tb bucket, for a change of half dozen.iii million files we will pay a ane-time transition fee of $63 (half-dozen,300,000 files grouped into 1,000 multiplied by $0.01), resulting in s a storage cost of $282.88 per calendar month

From the calculations above it tin can be seen that transitioning between classes will atomic number 82 to a reduction in costs starting from the outset calendar month.