More Resilient Downloads: Retries and Redirect Support

Besides scanning objects in Amazon S3 and Cloudflare R2, bucketAV can also scan files referenced by a custom download_url. You submit a scan job to the Scan Queue, bucketAV downloads the file via HTTP(S) GET, scans it, and posts the result to your callback_url. See Sending a scan job for the message format.

In practice, downloads don’t always succeed on the first try, and the file you want to scan isn’t always served directly at the URL you started with. Two improvements in the latest release address both situations.

Problem 1: Transient download failures cause unscannable results

When bucketAV fetches a file from a download_url, a lot can go wrong on the way: the remote host might be briefly unreachable, the TCP connection might reset, a load balancer might return a 503, or a presigned URL might race with a backend that returns a transient 5xx. Until now, a single failed request was enough for bucketAV to give up and report the file as unscannable back to your callback_url. Most of these errors are transient, and the right answer is simply to try again a moment later.

Solution: bucketAV retries failed downloads up to 3 times

bucketAV now retries failed downloads up to three times before reporting an unscannable result. A download counts as failed if the remote host cannot be reached (ENOTFOUND, ECONNREFUSED, ECONNRESET, ETIMEDOUT) or if the response status code is in the 4xx or 5xx range. Each retry is delayed (5, 10, 15 seconds) so a momentary outage on your side has time to recover. If all four attempts fail, bucketAV invokes your callback_url with the bucketav:download-failed code as before — the contract for your callback handler does not change.

No configuration is required. Submit a scan job exactly as you did before:

{
  "downloads": [
    {
      "download_url": "https://your-company.com/path/to/file.zip",
      "callback_url": "https://your-company.com/bucketav/webhook"
    }
  ]
}

If your-company.com returns a 502 Bad Gateway on the first attempt and a 200 OK on the second, bucketAV scans the file and reports the result. Before this release, you would have received an unscannable callback and had to re-submit the job yourself.

Problem 2: Files behind HTTP redirects could not be scanned

Many download endpoints return an HTTP redirect (status codes 301, 302, 307, 308) and serve the actual file from a different URL — typically a short-lived, pre-signed URL on object storage. Previously, bucketAV treated the 3xx response as the final response, never reached the file, and reported the job as unscannable. The workaround was to resolve the redirect yourself before sending the scan job.

Solution: bucketAV follows one redirect per download

bucketAV now follows a single HTTP redirect per download. The redirected request is the last hop — if that response is another redirect, the download fails (and is retried, see above). A single hop covers the common “API endpoint redirects to a pre-signed storage URL” pattern without exposing you to redirect loops or unbounded chains.

For the straightforward case, no configuration is needed:

{
  "downloads": [
    {
      "download_url": "https://your-company.com/files/report.pdf",
      "callback_url": "https://your-company.com/bucketav/webhook"
    }
  ]
}

If https://your-company.com/files/report.pdf responds with 302 Found and a Location header pointing at a pre-signed S3 URL, bucketAV follows the redirect and scans the file at the target URL.

Configuring headers for the redirected request

By default, headers from download_headers are not forwarded to the redirect target. That is deliberate: the redirect target is often a different host (for example, your API redirecting to S3), and forwarding an Authorization: Bearer ... token meant for your API to a third party would leak credentials.

For cases where the redirected request does need its own headers, use the new download_redirect_headers field. Each entry maps a header name to one of two shapes:

• {"value": "..."} — send a static value with the redirected request.
• {"response_header": "..."} — copy the value from a header on the 3xx response. If that header is not present, the entry is omitted.

Example: your API issues a redirect to a download host that expects its own bearer token, and also returns a request-signature header on the 3xx response that the target host validates:

{
  "downloads": [
    {
      "download_url": "https://api.your-company.com/files/report.pdf",
      "download_headers": {
        "Authorization": "Bearer api-token-for-your-company"
      },
      "download_redirect_headers": {
        "Authorization": {"value": "Bearer token-for-download-host"},
        "X-Request-Signature": {"response_header": "X-Request-Signature"}
      },
      "callback_url": "https://your-company.com/bucketav/webhook"
    }
  ]
}

In this job, bucketAV sends the API bearer token to api.your-company.com, follows the 302 to the download host, and on the redirected request sends a different bearer token plus the X-Request-Signature value it received on the 3xx response.

Availability

Both improvements are available in the following versions:

• bucketAV for Amazon S3 powered by ClamAV: redirect support and download_redirect_headers in v3.3.0, download retries in v3.4.0.
• bucketAV for Amazon S3 powered by Sophos: redirect support and download_redirect_headers in v3.3.0, download retries in v3.4.0.
• bucketAV for Cloudflare R2 powered by ClamAV: redirect support and download_redirect_headers in v3.3.0, download retries in v3.4.0.
• bucketAV for Cloudflare R2 powered by Sophos: redirect support and download_redirect_headers in v3.3.0, download retries in v3.4.0.

Follow the update guide to roll out the latest version.

Feedback

Do you have any questions? Are you missing any features? Please let us know! hello@bucketav.com