Stream File Upload to AWS S3 Using Go

Overview

Uploading large files directly to AWS S3 while minimizing memory and disk footprint is an essential task in cloud computing. This guide will demonstrate how to achieve this using the AWS SDK for Go.

Solution

To stream a file upload directly to S3, you can utilize the s3manager package. Here's a step-by-step solution:

1. Configure AWS Credentials and Session:

   • Set your AWS access key and secret, or use the default credentials provider.
   • Initialize an AWS session with the specified configuration.

2. Create an S3 Uploader:

   • Initialize an S3 uploader with the session and optional configuration settings.
   • You can configure parameters such as part size, concurrency, and max upload parts.

3. Open the File:

   • Open the file you want to upload using the os.Open function.

4. Upload the File:

   • Use the uploader.Upload method with the appropriate file information (bucket, key, and file pointer).

Example Code

The following code sample demonstrates how to stream large file upload to AWS S3 using s3manager:

package main

import (
    "fmt"
    "os"

    "github.com/aws/aws-sdk-go/aws/credentials"

    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/s3/s3manager"
)

const (
    filename      = "file_name.zip"
    myBucket      = "myBucket"
    myKey         = "file_name.zip"
    accessKey     = ""
    accessSecret  = ""
)

func main() {
    var awsConfig *aws.Config
    if accessKey == "" || accessSecret == "" {
        //load default credentials
        awsConfig = &aws.Config{
            Region: aws.String("us-west-2"),
        }
    } else {
        awsConfig = &aws.Config{
            Region:      aws.String("us-west-2"),
            Credentials: credentials.NewStaticCredentials(accessKey, accessSecret, ""),
        }
    }

    // The session the S3 Uploader will use
    sess := session.Must(session.NewSession(awsConfig))

    // Create an uploader with the session and default options
    //uploader := s3manager.NewUploader(sess)

    // Create an uploader with the session and custom options
    uploader := s3manager.NewUploader(sess, func(u *s3manager.Uploader) {
        u.PartSize = 5 * 1024 * 1024 // The minimum/default allowed part size is 5MB
        u.Concurrency = 2            // default is 5
    })

    //open the file
    f, err := os.Open(filename)
    if err != nil {
        fmt.Printf("failed to open file %q, %v", filename, err)
        return
    }
    //defer f.Close()

    // Upload the file to S3.
    result, err := uploader.Upload(&s3manager.UploadInput{
        Bucket: aws.String(myBucket),
        Key:    aws.String(myKey),
        Body:   f,
    })

    //in case it fails to upload
    if err != nil {
        fmt.Printf("failed to upload file, %v", err)
        return
    }
    fmt.Printf("file uploaded to, %s\n", result.Location)
}

By following these steps, you can efficiently upload large multipart/form-data files directly to AWS S3 with minimal memory utilization.