AWS Storage Services in a Nutshell

Getting Started

AWS Storage Services in a Nutshell

AWS Storage Services provide a variety of cost-effective storage solutions to store and automate the lifecycle of the unstructured data such as logs, documents, database backups, archives etc.

AWS Storage Services provide scalable and secure storage solutions to meet the diverse data storage needs for a wide range of applications, from small-scale projects to the enterprise-level solutions.

The Cheatsheet diagram below lists all the key storage services together in one view.

AWS in Nutshell - Storage Services Cheatsheet

AWS in Nutshell - Storage Services Cheatsheet(part-2)

What queries do these storage services address for you ?

In line with the objective of the nutshell module, our focus would be on highlighting the big picture. We will loo...

Simple Storage Service(S3): It's an object storage service with a REST API interface. We can access these storage using S3 console, which addresses many common queries related to our unstructured data as follows:

Effective Management
- Where can I store and manage my application logs , images, videos, documents or a DB snapshots in a cost effectively, securely, reliably ?
- How do I effectively organize and manage my data by projects, applications, clients and accounts ?
- Do I have cheaper options to archive my old but, important, regulatory/compliance related data for years together?

Secure Access Control & Data Protection
- Can I lock and protect my data against any inadvertent or malicious deletions ?
- Do I have proper versioning systems to manage and track the changes to my files ?
- Do I have flexible access control mechanism to easily share my data with 1000s of my clients?
- Sometime I may need to give temporary access to my clients and users?
- Do I have flexible options to encrypt and secure my data at rest or while in transit to this storage?

Cost
- Can AWS help me in optimizing my cost and automate the lifecyle of my data ?
- Can I share my cost with my clients for my shared data in S3?

Elastic Block Store (EBS): You can think of EBS as SSD or HDD volume options which you can attach to EC2 instances and are scoped to a single AZ. Some specific instance of EBS support sharing with multiple EC2 instances using a component called Multi-Attach but, does not inherently support sharing across multiple instances.

Here are some questions the EBS would be addressing for you:

How can I provide additional storage volumes to my EC2 running my database applications and ensure data persistence even after EC2 instance termination?
Do I have an option to create consistent snapshots of my EBS volumes for backup and disaster recovery purposes?
Can I easily encrypt my data at rest to meet security and compliance standards, ensuring sensitive information is protected?

Can I customize the performance of my applications by choosing SSD-based or HDD-based storage as per my needs?
Is there a mechanism to provision more IOPS, if the I/O performance needs of my applications increase ?
What options are available to resize my EBS volumes dynamically as my storage requirements change, ensuring cost-efficiency?

How can I migrate my data between different EBS volumes and EC2 instances efficiently without significant downtime?
Do I have access to detailed performance metrics and monitoring tools to analyze the health and performance of my EBS volumes?

Elastic File System(EFS): Unlike EBS, these storage services are inherently shareable. These storage systems provide file system capabilities and can be shared across multiple EC2 instances in different AZs within the same region.

AWS services provide different types of file systems such as NFS, SMB, NTFS , OpenZFS etc for integration with different supporting OS systems like unix and windows servers.
We can use these as file system storage for hybrid file systems and mountable persistent volumes for EC2 instances.
These systems can be mounted on EC2 instances across multiple availability zones(AZs) and we can use them for providing additional volume requirements for distributed storage systems requirements like HDFS, Cassandra database etc.

AWS Storage Gateway: It provides multiple options to integrate and extend our on-premise storage with AWS cloud storage. Here are some the common questions addressed by this gateway:

Storage Gateway Integration:

How can I seamlessly integrate my on-premises file-based applications with AWS storage services.
- Can my applications read and write data to AWS S3 via file protocols like NFS or SMB?
- Can I integrate my windows based on-premise file system with window based file services?
Do these services handle caching frequently accessed files locally :
- optimizing the performance for the applications and users as well as the cost,
- while ensuring data consistency with AWS S3 or the cloud storage?
Can we utilize cloud storage for the backup and disaster recovery scenarios,
- allowing my organization to store critical data in S3 and
- ensuring seamless recovery in case of site failures or data loss?

AWS Backup and AWS Elastic Disaster Recovery (DR): These services simplify the process of taking backups and prepare us for disaster management cases. These services address some of the common questions like:

What mechanisms are in place to enable easy and reliable restoration of data, ensuring minimal downtime?
Are there any services that simplifies (or automate) the backup of my AWS resources?
Can we customize our policies to manage backup schedules, retention periods, and lifecycle management,
- So, that it allows us to align backup strategies with specific business and compliance requirements?
How can I ensure that my applications and data can be quickly recovered the event of a availability zone or regional disasters?

2. Brief Descriptions, Features & Usages of the Storage Services

Service	Description	Usage scenarios
Object Storage
Amazon Simple Storage Service (S3)	Cost effective object storage service that offers high durability, security, availability, and low latency access for our data. Here are some key features of S3: Store & Organize Unstructured Data: Suitable for storing and organizing unstructured data such as application logs, static website content such as images, videos, documents. Data Lifecycle Management: You can define lifecycle policies to automatically transition objects between storage classes (such as Standard, Intelligent-Tiering, Glacier, and Deep Archive) and expire objects at the end of their lifecycle. Data Protection & Version Control: Provides features like S3 Object Lock and versioning to protect unwanted deletion and track changes through version control. Security and Access Control: You can control access to your data using AWS IAM policies, bucket policies, and Access Control Lists (ACLs). Temporary Access: Supports providing time limited access to specific users to using pre-signed urls.	Besides storing our logs, documents and media file here are some specific use cases for using S3: Static Website Content: We can use for storing and publishing static websites. Data Lake Storage : We can use it for creating data lakes as it supports large data storage and high-throughput for analytics. Data Migration and Transfer: S3 can be used as a staging area for data migration and transfer tasks. It allows seamless movement of data between different AWS regions, on-premises data centers, and other cloud providers, enabling smooth data transfers during migrations. Collaborative Workflows and Media Storage: It's widely used in media and entertainment industries for storing and distributing large media files, facilitating collaborative video editing, and managing media assets.
Block Storage
Amazon Elastic Block Store (EBS)	Provides mountable persistent storage(SSD/HDD/Magnetic) for Amazon EC2 instances, running databases, and applications needing persistent volumes.	Cost-effective and Reliable Block Storage: Amazon EBS provides cost-effective, reliable, scalable block storage for your critical applications like databases running on Amazon EC2 instances. Snapshots for Quick Disaster Recovery: Amazon EBS allows you to take point-in-time snapshots of your EBS volumes. In the event of data loss, corruption, or accidental deletion, you can use these snapshots to restore your volumes to a previous state, facilitating rapid disaster recovery.
File Storage
Amazon Elastic File System (EFS)	Amazon EFS allows you to create a shared file system that can be mounted simultaneously on multiple EC2 instances. This enables seamless collaboration and access to files across all instances. Unlike Amazon S3, which is an object storage service, Amazon EFS presents a familiar file system interface with standard file and directory operations.	A web hosting company might use EFS to store the files for their websites. This would allow them to easily share files between different websites and scale their storage capacity as needed.
Amazon FSx for Windows File Server	AWS managed Windows File Server, specifically designed for Windows-based applications, offering Windows compatibility and integration with Active Directory. It's similar to Amazon EFS which is a multi-platform service and can be used by Windows, Linux, and macOS applications, providing scalable and shared file storage.	Running Windows-based applications that require a file system.
Amazon FSx for Lustre	A managed file storage service designed to provide high-performance, scalable, and parallel file systems for compute-intensive workloads, such as high-performance computing (HPC), machine learning, and data analytics.	Shared file storage for high-performance computing (HPC) that involves large-scale simulations, scientific research, financial modeling, weather forecasting, and other data-intensive tasks that demand high-speed data access and parallel processing.
Backup & Recovery
Amazon S3 Glacier	Provides lowest cost storage suitable for data archiving with different retrieval flexibility. Useful for storing cold data, such as backup data and archives.	S3 Glacier Instant Retrieval : An archive storage class that delivers the lowest cost storage with milliseconds retrieval.For archiving data that needs immediate access, such as medical images, news media assets, or genomics data. S3 Glacier Flexible Retrieval: To archive data that does not require immediate access but needs the flexibility to retrieve large sets of data at no cost, such as backup or disaster recovery use cases. S3 Glacier Deep Archive: To save even more on long-lived archive storage such as compliance archives and digital media preservation. The lowest cost storage in the cloud with data retrieval within 12 hours.
AWS Backup	AWS Backup is a centralized backup service designed to automate and manage data backups across various AWS services, including Amazon EC2 instances, Amazon EBS volumes, and on-premises servers. AWS Backup also offers a variety of features for managing your backups, such as automatic backups, scheduled backups, and retention policies.	Protecting data from accidental deletion, corruption, or disaster.
AWS Elastic Disaster Recovery (DR)	AWS Elastic DR is a disaster recovery service that helps you recover your applications and data in the event of a disaster. It can be used to replicate your applications and data to a different AWS region or to an on-premises location. AWS Elastic DR also offers a variety of features for managing your disaster recovery plan, such as automatic failover and testing.	You can use AWS DRS to recover variety of critical applications and databases running on EC2 instances, EBS volumes, on-premises servers, AWS Lambda functions, Amazon RDS databases, Amazon DynamoDB tables, Amazon S3 buckets and many other.
Hybrid Storage
Storage Gateway	A Hybrid(on-premise plus cloud) Storage Solution. Simplifies storage management(storage/backup/low-latency access) enabling hybrid storage service. Provides four different types of gateways – Tape Gateway, Amazon S3 File Gateway, Amazon FSx File Gateway, and Volume Gateway – that seamlessly connect on-premises applications to cloud storage, caching data locally for low-latency access.	Tape Gateway (VTL and Tape Archiving): Tape Gateway stores your virtual tapes in Amazon S3 and creates new ones automatically, simplifying management and your transition to AWS. File Gateway (NFS and SMB): File Gateway provides on-premises applications with file-based access to objects in Amazon S3 buckets. It supports both Network File System (NFS) and Server Message Block (SMB) protocols. FSx File Gateway: Use Amazon FSx File Gateway for your on-premises file-based business applications and workloads such as user or group file shares, web content management, and media workflows. Stored Volume : We can use Volume Gateway to back up local applications in the cloud as AWS EBS snpashots. We can use Volume Gateway service's snapshot sheduler or Backup service to schedule the snapshots.
Data Transfer
AWS Snowball	Low-Cost, High Volume Data Transfer & Edge Computing AWS Snowball service provides an edge computing and data transfer device known as AWS Snowball Edge. AWS Snowball is available in two device types- Snowball Edge Compute Optimized and Snowball Edge Storage Optimized.	Supports cost effective high volume(80TB or 210TB) data transfer in and out of AWS. Support local data processing and collection in disconnected environments such as ships, windmills, and remote factories.