How to share files between 10,000 servers using AWS Elastic File System

If your company is working on a product that processes files by multiple servers or the files must be accessible by your cloud and on-premises resources, then Amazon Elastic File System (EFS) is a powerful solution to optimize your infrastructure.

AWS EFS is a cloud-based file storage service that allows users to share file data without managing storage capacity and performance. It's a Network File System (NFS) that organizes data in a logical file hierarchy. EFS is designed to be fully elastic and scalable, and can be used with AWS services and on-premises resources. 

Typical EFS use cases

• Big Data Analytics: Storing and processing large datasets for data lakes and data warehouses.
• Machine Learning: Storing training data, model checkpoints, and results.
• Media and Entertainment: Storing and processing media files, such as videos and images.
• High-Performance Computing (HPC): Storing and accessing large datasets for HPC workloads.
• Web Servers: Storing static web content, such as HTML, CSS, and JavaScript files.
• Content Management Systems (CMS): Hosting CMS files.
• Software Development: Sharing code repositories, building artifacts, and project files.

The main benefits of EFS

• Scalability: Automatically scales up or down as your data storage needs change, ensuring optimal performance and cost-efficiency.
• High Availability and Durability: Data is replicated across multiple Availability Zones (AZs) to provide high availability and durability, minimizing downtime and data loss.
• Easy Integration: Easily integrates with various AWS services like EC2, Lambda, and EMR, simplifying your infrastructure and workflows.
• Performance Optimization: Delivers consistent, high-performance access to your files, regardless of the number of concurrent clients or the size of your dataset.
• Security: Offers robust security features, including encryption at rest and in transit, access controls, and integration with AWS Identity and Access Management (IAM).

Setting up AWS EFS is easy. Follow the next steps to create and attach a file system to your EC2 instance.

1) Open the AWS Console and navigate to EFS.

2) Click on "Create file system".

3) Give EFS a name and pick a VPC where your servers are located. Let's call it "demo-efs".

4) Click on "Create". The file system will be created in a few seconds.

5) Click on "demo-efs" and then on "Attach". You will see two options - mount via DNS or IP.

Write down the command line for the EFS mount helper. It should look like

sudo mount -t efs -o tls <FILE_SYSTEM_ID>:/ efs

6) Open your EC2 terminal and run the commands below.

# create folder where EFS will be monted
sudo mkdir /mnt/efs

# install EFS utils
sudo yum install amazon-efs-utils

# try to mount EFS
sudo mount -t efs -o tls <FILE_SYSTEM_ID>:/ /mnt/efs

You will see the timeout error "Mount attempt 1/3 failed due to timeout after 15 sec, wait 0 sec before next attempt." This happens because your EC2 instance profile requires permission to communicate with EFS. Let's fix it.

7) Go back to "demo-efs" but this time open "Network" tab instead of clicking "Attach". The "Security groups" column lists groups that may access the EFS. Click on "Manage" and add/update the group to be the one that your EC2 uses. "Save" the changes.

8) Open your EC2 again and run

sudo mount -t efs -o tls <FILE_SYSTEM_ID>:/ /mnt/efs

This time, the mount point will be created.

If you want to mount the files system automatically, you must update the fstab file.

sudo nano /etc/fstab

# add the line wher
<FILE_SYSTEM_ID>:/ <MOUNT_POINT> efs _netdev,noresvport,tls,iam 0 0

# restart the service
systemctl daemon-reload

# check that auto mount works
sudo mount -fav

---

Are you looking for help? Reach me any time.