"If a worker wants to do his job well, he must first sharpen his tools." - Confucius, "The Analects of Confucius. Lu Linggong"
Front page > Programming > How can I utilize Goroutines in Go to download multiple files concurrently from Dropbox?

How can I utilize Goroutines in Go to download multiple files concurrently from Dropbox?

Published on 2024-11-08
Browse:778

How can I utilize Goroutines in Go to download multiple files concurrently from Dropbox?

Parallel File Downloads in Go Using Goroutines

In Go, it is possible to download and save files concurrently using goroutines. Goroutines allow for lightweight thread execution, facilitating parallel processing and improving performance.

Here's the code you provided, modified to utilize goroutines:

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "io/ioutil"
    "net/http"
    "net/url"
    "os"
    "path/filepath"
    "sync"
)

// Credentials
const app_key string = ""
const app_secret string = ""

// Authorization code
var code string

// Token response
type TokenResponse struct {
    AccessToken string `json:"access_token"`
}

// File structure
type File struct {
    Path string
}

// File list response
type FileListResponse struct {
    FileList []File `json:"contents"`
}

// Download a file using goroutines
func download_file(file File, token TokenResponse, wg *sync.WaitGroup) {
    download_file := fmt.Sprintf("https://api-content.dropbox.com/1/files/dropbox/%s?access_token=%s", file.Path, token.AccessToken)

    resp, _ := http.Get(download_file)
    defer resp.Body.Close()

    filename := filepath.Base(file.Path)
    out, err := os.Create(filename)
    if err != nil {
        panic(err)
    }
    defer out.Close()

    io.Copy(out, resp.Body)
    wg.Done()
}

func main() {
    // Authorization and token retrieval
    authorize_url := fmt.Sprintf("https://www.dropbox.com/1/oauth2/authorize?response_type=code&client_id=%s", app_key)
    fmt.Printf("1. Go to: %s\n", authorize_url)
    fmt.Println("2. Click 'Allow' (you might have to log in first)")
    fmt.Println("3. Copy the authorization code.")
    fmt.Printf("Enter the authorization code here: ")
    fmt.Scanf("%s", &code)

    // Get file list
    file_list_url := fmt.Sprintf("https://api.dropbox.com/1/metadata/dropbox/Camera Uploads?access_token=%s", tr.AccessToken)

    resp2, _ := http.Get(file_list_url)
    defer resp2.Body.Close()

    contents2, _ := ioutil.ReadAll(resp2.Body)

    var flr FileListResponse
    json.Unmarshal(contents2, &flr)

    // WaitGroup to wait for all goroutines to finish
    var wg sync.WaitGroup

    // Download files concurrently
    for i, file := range flr.FileList {
        wg.Add(1)

        go download_file(file, tr, &wg)

        if i >= 2 {
            break
        }
    }
    wg.Wait()
}

In this modified code, we add a sync.WaitGroup called wg to track the number of goroutines executing. We increment wg before starting each goroutine and decrement it when each goroutine finishes using wg.Done(). The main goroutine waits for all the goroutines to finish by calling wg.Wait(). This ensures that the program doesn't exit before all the files are downloaded.

Latest tutorial More>

Disclaimer: All resources provided are partly from the Internet. If there is any infringement of your copyright or other rights and interests, please explain the detailed reasons and provide proof of copyright or rights and interests and then send it to the email: [email protected] We will handle it for you as soon as possible.

Copyright© 2022 湘ICP备2022001581号-3