S3 Transfer Acceleration vs CloudFront/S3 Presigned URL Upload!
Motivation
In the projects I’ve worked on, setting up AWS CloudFront with an S3 bucket for a static website is basically the DevOps equivalent of making instant noodles — straightforward, and it just works. The Frontend team builds a shiny new Single Page Application (SPA), spits out some static files, and hands them over to me. My job? Toss those files into an S3 bucket, sprinkle in some CloudFront magic, slap on a custom domain, and voilà, the website is alive.
But then, someone always goes, “Hey, can we also upload files to the same bucket?” Sure, why not? Enter S3 Pre-signed URLs combined with CloudFront allowing PUT requests. It’s like handing out VIP tickets for file uploads. And guess what? It works like a charm, just as I hoped.
Now, let’s talk about S3 Transfer Acceleration—a.k.a. the feature I kind of remembered when cramming for my AWS SAA certification. Back then, it was one of those “just memorize it and move on” topics. And like most trivia from certification prep, it vanished from my brain the moment I passed the exam.
Fast forward to today. As I gear up for the AWS SAP certification, guess who’s back? Yep, that mysterious feature. This time, though, I want to actually understand it. No shortcuts, no “because the practice test said so.” Just good old-fashioned learning.
So, I dive into the documentation and read this:
Amazon S3 Transfer Acceleration is a bucket-level feature that enables fast, easy, and secure transfers of files over long distances between your client and an S3 bucket. Transfer Acceleration is designed to optimize transfer speeds from across the world into S3 buckets.
It takes advantage of the globally distributed edge locations in Amazon CloudFront. As data arrives at an edge location, it is routed to Amazon S3 over an optimized network path.
Cool, right? But then a thought hits me: “Wait… is AWS just doing the same thing I’ve been doing with CloudFront and PUT requests? Or am I unknowingly reinventing the wheel here?”
So now I’m stuck in this existential crisis: Which is better? My clever CloudFront + S3 upload solution, or AWS’s S3 Transfer Acceleration sorcery?
Looks like I’ve got some digging to do. But hey, if nothing else, at least I’m getting my money’s worth from this certification prep.
Diagram
First things first, we need to prepare something that will help us stay on the right track - an infrastructure diagram.
Specification
- CloudFront Distribution
- Custom domain: enabled
- Origin: S3
- All other settings: default
- S3 bucket
- Type: General Purpose
- Region: ap-southeast-1
- All other settings: default
Setting Up
Create AWS S3 bucket
Let’s create AWS S3 bucket name bucket-for-transfer-acceleration-demo with default settings.
Done!
Enable S3 Transfer Acceleration
Go to bucket-for-transfer-acceleration-demo > Properties > Transfer acceleration > Edit
Done!
Create CloudFront ditribution
Create new CloudFront distribution with origin is S3 bucket created in previous step.
For Origin access, use Origin access control settings (recommended). Let’s create a new one.
For Allowed HTTP methods, select GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE.
Done enable WAF, I don’t want to lose my money :(
We will add custom domain later. Leave all other settings with default value, and Done.
Update S3 bucket policy to make OAC work
Remember to add s3:PutObject permission to allow upload file.
Create ACM certificate for CloudFront Distribution
Remember switch to us-east-1 region =)).
Domain name is: s3-acc.skillsboost.cloud
Done!
Add CloudFront Alternative domain
Done!
Prepare Javascript script to generate Presign URL
This script will create presigned url for 3 endpoint type:
- CloudFront endpoint
- Transfer Acceleration endpoint
- S3 regional endpoint
import { PutObjectCommand, S3Client } from "@aws-sdk/client-s3";
import {fromIni} from "@aws-sdk/credential-providers";
import {HttpRequest} from "@smithy/protocol-http";
import {getSignedUrl, S3RequestPresigner} from "@aws-sdk/s3-request-presigner";
import {parseUrl} from "@smithy/url-parser";
import {formatUrl} from "@aws-sdk/util-format-url";
import {Hash} from "@smithy/hash-node";
// Generate a presigned URL for a custom endpoint
const region = "ap-southeast-1";
const url = parseUrl(`https://s3-acc.skillsboost.cloud/test.txt`);
const presigner = new S3RequestPresigner({
credentials: fromIni(),
region,
sha256: Hash.bind(null, "sha256"),
});
const signedUrlObject = await presigner.presign(
new HttpRequest({...url, method: "PUT"})
);
console.log("CloudFront endpoint:", formatUrl(signedUrlObject));
console.log();
// Generate a presigned URL for transfer acceleration endpoint
const client2 = new S3Client({useAccelerateEndpoint: true, region: region})
const command2 = new PutObjectCommand({ Bucket: "bucket-for-transfer-acceleration-demo", Key: "test2.txt" });
const signedUrlObject2 = await getSignedUrl(client2, command2, { expiresIn: 3600 });
console.log("Acceleration endpoint:", signedUrlObject2);
console.log();
// Generate a presigned URL using default endpoint
const client3 = new S3Client({useAccelerateEndpoint: false, region: region})
const command3 = new PutObjectCommand({ Bucket: "bucket-for-transfer-acceleration-demo", Key: "test3.txt" });
const signedUrlObject3 = await getSignedUrl(client3, command3, { expiresIn: 3600 });
console.log("Default endpoint:", signedUrlObject3);
Do the Test
This is the fun part!
DNS resolution testing
First, we will determine how many IP addresses we get when resolving the DNS records for the domains.
We will use the dig command to perform the DNS resolution.
CloudFront domain:
dig +noall +answer s3-acc.skillsboost.cloud
s3-acc.skillsboost.cloud. 300 IN CNAME d2vmtkxxkxgoqq.cloudfront.net.
d2vmtkxxkxgoqq.cloudfront.net. 60 IN A 108.157.32.85
d2vmtkxxkxgoqq.cloudfront.net. 60 IN A 108.157.32.56
d2vmtkxxkxgoqq.cloudfront.net. 60 IN A 108.157.32.98
d2vmtkxxkxgoqq.cloudfront.net. 60 IN A 108.157.32.89
Transfer Acceleration domain:
dig +noall +answer bucket-for-transfer-acceleration-demo.s3-accelerate.amazonaws.com
bucket-for-transfer-acceleration-demo.s3-accelerate.amazonaws.com. 60 IN A 3.162.57.183
S3 regional domain:
dig +noall +answer bucket-for-transfer-acceleration-demo.s3.ap-southeast-1.amazonaws.com
bucket-for-transfer-acceleration-demo.s3.ap-southeast-1.amazonaws.com. 60 IN CNAME s3-r-w.ap-southeast-1.amazonaws.com.
s3-r-w.ap-southeast-1.amazonaws.com. 2 IN A 52.219.164.210
s3-r-w.ap-southeast-1.amazonaws.com. 2 IN A 3.5.151.108
s3-r-w.ap-southeast-1.amazonaws.com. 2 IN A 3.5.149.130
s3-r-w.ap-southeast-1.amazonaws.com. 2 IN A 3.5.147.144
s3-r-w.ap-southeast-1.amazonaws.com. 2 IN A 3.5.146.102
s3-r-w.ap-southeast-1.amazonaws.com. 2 IN A 3.5.149.122
s3-r-w.ap-southeast-1.amazonaws.com. 2 IN A 3.5.150.166
s3-r-w.ap-southeast-1.amazonaws.com. 2 IN A 3.5.150.116
And the results are:
| # | Domain | No. IP address(es) |
|---|---|---|
| 1 | s3-acc.skillsboost.cloud | 4 |
| 2 | bucket-for-transfer-acceleration-demo.s3-accelerate.amazonaws.com | 1 |
| 3 | bucket-for-transfer-acceleration-demo.s3.ap-southeast-1.amazonaws.com | 8 |
More IP addresses mean more redundancy is added. However, there is no guarantee that the performance of all servers behind those IPs will be equal.
Let’s Ping it
Now, let’s ping the servers behind the domains 100 times, repeating the test 3 times for each domain. The results will depend on the network your laptop is currently connected to. My Access Point is close to me, so we can safely ignore the latency caused by my network.
CloudFront domain:
ping -c 100 s3-acc.skillsboost.cloud
#1
--- d2vmtkxxkxgoqq.cloudfront.net ping statistics ---
100 packets transmitted, 100 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 18.736/27.995/104.295/11.745 ms
#2
--- d2vmtkxxkxgoqq.cloudfront.net ping statistics ---
100 packets transmitted, 99 packets received, 1.0% packet loss
round-trip min/avg/max/stddev = 17.786/26.227/143.976/13.945 ms
#3
--- d2vmtkxxkxgoqq.cloudfront.net ping statistics ---
100 packets transmitted, 100 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 17.880/25.652/49.138/6.090 ms
Transfer Acceleration domain:
ping -c 100 bucket-for-transfer-acceleration-demo.s3-accelerate.amazonaws.com
#1
--- bucket-for-transfer-acceleration-demo.s3-accelerate.amazonaws.com ping statistics ---
100 packets transmitted, 99 packets received, 1.0% packet loss
round-trip min/avg/max/stddev = 18.010/26.608/73.290/8.460 ms
#2
--- bucket-for-transfer-acceleration-demo.s3-accelerate.amazonaws.com ping statistics ---
100 packets transmitted, 100 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 18.316/24.037/37.018/4.041 ms
#3
--- bucket-for-transfer-acceleration-demo.s3-accelerate.amazonaws.com ping statistics ---
100 packets transmitted, 100 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 18.213/27.100/153.949/14.144 ms
S3 regional domain:
ping -c 100 bucket-for-transfer-acceleration-demo.s3.ap-southeast-1.amazonaws.com
#1
--- s3-r-w.ap-southeast-1.amazonaws.com ping statistics ---
100 packets transmitted, 100 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 47.413/58.297/299.782/25.117 ms
#2
--- s3-r-w.ap-southeast-1.amazonaws.com ping statistics ---
100 packets transmitted, 100 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 45.509/56.717/165.572/17.402 ms
#3
--- s3-r-w.ap-southeast-1.amazonaws.com ping statistics ---
100 packets transmitted, 100 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 45.907/53.702/139.683/10.706 ms
And the results - calculated using average between 3 times.
| # | Domain | Latency Min (ms) | Latency Avg (ms) | Latency Max (ms) |
|---|---|---|---|---|
| 1 | s3-acc.skillsboost.cloud | 18.134 | 26.625 | 99.136 |
| 2 | bucket-for-transfer-acceleration-demo.s3-accelerate.amazonaws.com | 18.179 | 25.915 | 88.085 |
| 3 | bucket-for-transfer-acceleration-demo.s3.ap-southeast-1.amazonaws.com | 46.276 | 56.238 | 201.679 |
Since the domains s3-acc.skillsboost.cloud and bucket-for-transfer-acceleration-demo.s3-accelerate.amazonaws.com leverage edge locations, we benefit from the reduced distance to the server. As a result, the average latency is much lower compared to the domain of the regional endpoint. There is no noticeable difference between the CloudFront domain and the Transfer Acceleration domain.
Give it a big file
I’ll generate a 256Mb binary for testing using this command:
dd if=/dev/urandom of=256Mb.bin bs=64M count=4 iflag=fullblock
Then create presigned urls using script above to upload 256Mb test file.
node s3-presigned-url-generator.js
CloudFront endpoint: https://s3-acc.skillsboost.cloud/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=ASIA3....%2F20241208%2Fap-southeast-1%2Fs3%2Faws4_request&X-Amz-Date=20241208T130513Z&X-Amz-Expires=900&X-Amz-Security-Token=IQoJ...yrV&X-Amz-Signature=5f4d0c9fda9d25d06&X-Amz-SignedHeaders=host
Acceleration endpoint: https://bucket-for-transfer-acceleration-demo.s3-accelerate.amazonaws.com/test2.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=ASIA3R...%2F20241208%2Fap-southeast-1%2Fs3%2Faws4_request&X-Amz-Date=20241208T130514Z&X-Amz-Expires=3600&X-Amz-Security-Token=IQoJ...%3D&X-Amz-Signature=cd916404d122704835e7911&X-Amz-SignedHeaders=host&x-id=PutObject
Default endpoint: https://bucket-for-transfer-acceleration-demo.s3.ap-southeast-1.amazonaws.com/test3.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=ASIA3R...5%2F20241208%2Fap-southeast-1%2Fs3%2Faws4_request&X-Amz-Date=20241208T130515Z&X-Amz-Expires=3600&X-Amz-Security-Token=IQoJb3...LI%3D&X-Amz-Signature=8e9c4b7456cd94edbad4cac8&X-Amz-SignedHeaders=host&x-id=PutObject
AWS CloudShell in us-east-1 region will be used to upload files using a pre-signed URL to simulate far away user. We will upload files 3 times for each pre-signed URL.
# CloudFront endpoint
time wget --no-check-certificate --quiet --method PUT --timeout=0 --header 'Content-Type: application/octet-stream' --body-file='./256Mb.bin' 'https://s3-acc.skillsboost.cloud/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=ASIA3....%2F20241208%2Fap-southeast-1%2Fs3%2Faws4_request&X-Amz-Date=20241208T130513Z&X-Amz-Expires=900&X-Amz-Security-Token=IQoJ...yrV&X-Amz-Signature=5f4d0c9fda9d25d06&X-Amz-SignedHeaders=host'
# Acceleration endpoint
time wget --no-check-certificate --quiet --method PUT --timeout=0 --header 'Content-Type: application/octet-stream' --body-file='./256Mb.bin' 'https://bucket-for-transfer-acceleration-demo.s3-accelerate.amazonaws.com/test2.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=ASIA3R...%2F20241208%2Fap-southeast-1%2Fs3%2Faws4_request&X-Amz-Date=20241208T130514Z&X-Amz-Expires=3600&X-Amz-Security-Token=IQoJ...%3D&X-Amz-Signature=cd916404d122704835e7911&X-Amz-SignedHeaders=host&x-id=PutObject'
# Default endpoint
time wget --no-check-certificate --quiet --method PUT --timeout=0 --header 'Content-Type: application/octet-stream' --body-file='./256Mb.bin' 'https://bucket-for-transfer-acceleration-demo.s3.ap-southeast-1.amazonaws.com/test3.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=ASIA3R...5%2F20241208%2Fap-southeast-1%2Fs3%2Faws4_request&X-Amz-Date=20241208T130515Z&X-Amz-Expires=3600&X-Amz-Security-Token=IQoJb3...LI%3D&X-Amz-Signature=8e9c4b7456cd94edbad4cac8&X-Amz-SignedHeaders=host&x-id=PutObject'
And the results:
| # | Domain | Upload Time Avg (ms) |
|---|---|---|
| 1 | s3-acc.skillsboost.cloud | 23.385 |
| 2 | bucket-for-transfer-acceleration-demo.s3-accelerate.amazonaws.com | 7.493 |
| 3 | bucket-for-transfer-acceleration-demo.s3.ap-southeast-1.amazonaws.com | 22.885 |
You know what? I’m a bit let down.
Turns out, using the CloudFront endpoint for uploads is like trying to sprint in flip-flops—just doesn’t work as well as I’d hoped. It’s actually slower than the default endpoint. Ouch. But hey, here’s the good news: Transfer Acceleration absolutely zooms! It’s over three times faster when uploading files from us-east-1 to ap-southeast-1—basically halfway across the planet.
This speed boost isn’t magic (though it feels like it). Transfer Acceleration taps into Amazon CloudFront’s edge locations to slingshot your data across the globe. Science!
Meanwhile, poor CloudFront is sitting there going, “Uploads? That’s not really my thing. I’m more of a content delivery and caching kind of service.” Fair enough, CloudFront. Stay in your lane.
Moral of the story: when it comes to long-distance uploads, Transfer Acceleration is like the Concorde, while the other options are stuck in economy class.
Conclusion
AWS may have won this round, but don’t get too comfortable. I’ll be back for my revenge! 😏