Guides

Scheduled bucket scan

New malware shows up daily, and existing malware can be modified to evade detection. Zero-day attacks are new threats that have yet to be identified or need to be added to the signatures database. Keeping the signature database up-to-date to detect the latest threats is an ongoing fight.

But you can fight back with periodic malware scanning. That’s why we recommend you run regular full bucket scans to ensure that zero-day attackers are detected as soon as the signature database is updated.

Quick Setup (recommended)

Requires bucketAV for Amazon S3 powered by ClamAV® version >= 2.18.0, bucketAV for Amazon S3 powered by Sophos® version >= 2.10.0, bucketAV for Cloudflare R2 powered by ClamAV® version >= 2.0.0, or bucketAV for Cloudflare R2 powered by Sophos® version >= 2.0.0.
To update to the latest version, follow the Update Guide.

For multi-account setups, please perform the setup manually.

  1. Visit the AWS CloudWatch Management Console.
  2. Navigate to Dashboards.
  3. Select the dashboard starting with the name bucketav followed by the name of the AWS region—for example, bucketav-eu-west-1. Step 1
  4. Find the Buckets tile. Enable scheduled bucket scanning for each bucket you want by clicking the Enable button. Step 2

From now on, your S3 bucket or R2 bucket will periodically be scanned for trojans, viruses, and malware.

Setup

If your bucket contains more than 350 million objects/versions (see the NumberOfObjects CloudWatch metric), please send us an email for guidance!

Install Add-On (requires a running bucketAV installation)

  1. Set the Stack name to bucketav-scheduled-bucket-scan.
  2. Set the BucketAVStackName parameter to the stack name of bucketAV (if you followed the docs, the name is bucketav).
  3. Set the BucketName parameter to the name of the S3 bucket that you want to scan. You can also enter multiple bucket names separated by a comma (e.g., bucketa,bucketb) or use a wildcard character (e.g., mycompany-*-prod,bucketa).
  4. Set the ScheduleExpression parameter to a valid expression. E.g., rate(1 day), rate(7 days), or rate(1 hour).
  5. The PagingBatchSize, PagingWaitInSeconds, and TimeoutInSeconds parameters depend on the number of objects/versions in your bucket(s) (see the NumberOfObjects CloudWatch metric of your bucket). The following table provides sample values optimized to enqueue files as steadily as possible over 12 hours. Choose the row that fits your objects (e.g., if your bucket has 6 mio objects, choose the values from the row with 10 mio objects).

For ExcludeScannedObjects set to false (default):

Number of objectsPagingBatchSizePagingWaitInSecondsTimeoutInSeconds
50000503043200 (12 hours)
1000001003043200 (12 hours)
1 mio5001543200 (12 hours)
10 mio50001543200 (12 hours)
50 mio20000743200 (12 hours)
100 mio40000043200 (12 hours)
200 mio60000086400 (24 hours)
350 mio1000000172800 (48 hours)

For ExcludeScannedObjects set to true:

Number of objectsPagingBatchSizePagingWaitInSecondsTimeoutInSeconds
50000503043200 (12 hours)
1000001003043200 (12 hours)
1 mio5001543200 (12 hours)
10 mio5000743200 (12 hours)
50 mio200000172800 (2 days)
100 mio400000259200 (3 days)
200 mio600000360000 (5 days)
  1. If you are interested in a scan report after each scheduled bucket scan:
    1. Install the reporting Add-On.
    2. Set the ReportingAddOnStackName parameter to the stack name of the reporting Add-On (if you followed the docs, the name is bucketav-reporting).
  2. Select I acknowledge that AWS CloudFormation might create IAM resources.
  3. Click on the Create stack button to save.

If your bucket contains more than 350 million objects, please send us an email for guidance!

Install Add-On (requires a running bucketAV installation)

  1. Set the Stack name to bucketav-scheduled-bucket-scan.
  2. Set the BucketAVStackName parameter to the stack name of bucketAV (if you followed the docs, the name is bucketav).
  3. Set the BucketName parameter to the name of the R2 bucket that you want to scan. You can also enter multiple bucket names separated by a comma (e.g., bucketa,bucketb) or use a wildcard character (e.g., mycompany-*-prod,bucketa).
  4. Set the ScheduleExpression parameter to a valid expression. E.g., rate(1 day), rate(7 days), or rate(1 hour).
  5. The PagingBatchSize, PagingWaitInSeconds, and TimeoutInSeconds parameters depend on the number of objects in your bucket(s). The following table provides sample values optimized to enqueue files as steadily as possible over 12 hours. Choose the row that fits your objects (e.g., if your bucket has 6 mio objects, choose the values from the row with 10 mio objects).

For ExcludeScannedObjects set to false (default):

Number of objectsPagingBatchSizePagingWaitInSecondsTimeoutInSeconds
50000503043200 (12 hours)
1000001003043200 (12 hours)
1 mio5001543200 (12 hours)
10 mio50001543200 (12 hours)
50 mio20000743200 (12 hours)
100 mio40000043200 (12 hours)
200 mio60000086400 (24 hours)
350 mio1000000172800 (48 hours)

For ExcludeScannedObjects set to true:

Number of objectsPagingBatchSizePagingWaitInSecondsTimeoutInSeconds
50000503043200 (12 hours)
1000001003043200 (12 hours)
1 mio5001543200 (12 hours)
10 mio5000743200 (12 hours)
50 mio200000172800 (2 days)
100 mio400000259200 (3 days)
200 mio600000360000 (5 days)
  1. If you are interested in a scan report after each scheduled bucket scan:
    1. Install the reporting Add-On.
    2. Set the ReportingAddOnStackName parameter to the stack name of the reporting Add-On (if you followed the docs, the name is bucketav-reporting).
  2. Select I acknowledge that AWS CloudFormation might create IAM resources.
  3. Click on the Create stack button to save.

Multi-account setup

If you have a multi-account setup, you might want to scan buckets created in additional AWS accounts.

IAM role-based access requires version >= 2.14.0 of this Add-On. To update to the latest version, follow the update instructions.

Not available. Please contact us if you have a use case for it!

CloudFormation snippet

# [...]
Resources:
  # [...]
  ScheduledBucketScan:
    Type: 'AWS::CloudFormation::Stack'
    Properties:
      Parameters:
        BucketAVStackName: 'bucketav' # if you followed the docs, the name is bucketav
        BucketName: 'mycompany-*-prod,bucketa,bucketb'
        ScheduleExpression: 'rate(7 days)' # see https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html
        PagingBatchSize: '50' # get value from table above
        PagingWaitInSeconds: '30' # get value from table above
        TimeoutInSeconds: '43200' # get value from table above
      TemplateURL: 'https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/v2.15.0/bucketav-add-on-scheduled-bucket-scan.yaml'

# [...]
Resources:
  # [...]
  ScheduledBucketScan:
    Type: 'AWS::CloudFormation::Stack'
    Properties:
      Parameters:
        BucketAVStackName: 'bucketav' # if you followed the docs, the name is bucketav
        BucketName: 'mycompany-*-prod,bucketa,bucketb'
        ScheduleExpression: 'rate(7 days)' # see https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html
        PagingBatchSize: '50' # get value from table above
        PagingWaitInSeconds: '30' # get value from table above
        TimeoutInSeconds: '43200' # get value from table above
      TemplateURL: 'https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/cloudflare/v2.15.0/bucketav-add-on-scheduled-bucket-scan-cloudflare.yaml'

Terraform snippet

resource "aws_cloudformation_stack" "bucketav_add_on_scheduled_bucket_scan" {
  name         = "bucketav-scheduled-bucket-scan"
  template_url = "https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/v2.15.0/bucketav-add-on-scheduled-bucket-scan.yaml"
  capabilities = ["CAPABILITY_IAM"]
  parameters = {
    BucketAVStackName   = "bucketav" # if you followed the docs, the name is bucketav
    BucketName          = "mycompany-*-prod,bucketa,bucketb"
    ScheduleExpression  = "rate(7 days)" # see https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html
    PagingBatchSize     = "50" # get value from table above
    PagingWaitInSeconds = "30" # get value from table above
    TimeoutInSeconds    = "43200" # get value from table above
  }
}

resource "aws_cloudformation_stack" "bucketav_add_on_scheduled_bucket_scan" {
  name         = "bucketav-scheduled-bucket-scan"
  template_url = "https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/cloudflare/v2.15.0/bucketav-add-on-scheduled-bucket-scan-cloudflare.yaml"
  capabilities = ["CAPABILITY_IAM"]
  parameters = {
    BucketAVStackName   = "bucketav" # if you followed the docs, the name is bucketav
    BucketName          = "mycompany-*-prod,bucketa,bucketb"
    ScheduleExpression  = "rate(7 days)" # see https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html
    PagingBatchSize     = "50" # get value from table above
    PagingWaitInSeconds = "30" # get value from table above
    TimeoutInSeconds    = "43200" # get value from table above
  }
}

Insights

To get insights into running and completed bucket scan runs:

  1. Visit the Step Functions Management Console.
  2. Click on the state machine (if you followed the docs, the name is bucketav-scheduled-bucket-scan-orchestrator).

You will see a list of Executions. The most recent execution is at the top and represents the latest bucket scan. If the status equals Succeeded, the bucket scan is complete. If the status equals Running, the bucket scan is running.

Remember that Succeeded means that all files are enqueued for scanning. It does not mean that all files are already scanned. You can observe the Scan Queue in the CloudWatch Dashboard. An empty (or mostly empty queue if new objects are uploaded in parallel) indicates that all files are scanned.

Update

Which version am I using?

Version 2.7.0 included buckets from all regions using the wildcard character in the BucketName parameter. We fixed this in version 2.7.1!

  1. To update this Add-On to version v2.15.0, go to the AWS CloudFormation Management Console.
  2. Double-check the region at the top right.
  3. Search for bucketav-scheduled-bucket-scan, otherwise search for the name you specified.
  4. Select the stack and click on Update.
  5. Select Replace current template and set the Amazon S3 URL to https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/v2.15.0/bucketav-add-on-scheduled-bucket-scan.yaml Copy
  6. Click on Next.
  7. Scroll to the bottom of the page and click on Next.
  8. Scroll to the bottom of the page and click on Next.
  9. Scroll to the bottom of the page, enable I acknowledge that AWS CloudFormation might create IAM resources, and click on Update stack.
  10. While the update runs, the stack status is UPDATE_IN_PROGRES. Reload the table from time to time and …
  11. … wait until the CloudFormation stack status switches to UPDATE_COMPLETE.
  1. To update this Add-On to version v2.15.0, go to the AWS CloudFormation Management Console.
  2. Double-check the region at the top right.
  3. Search for bucketav-scheduled-bucket-scan, otherwise search for the name you specified.
  4. Select the stack and click on Update.
  5. Select Replace current template and set the Amazon S3 URL to https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/cloudflare/v2.15.0/bucketav-add-on-scheduled-bucket-scan-cloudflare.yaml Copy
  6. Click on Next.
  7. Scroll to the bottom of the page and click on Next.
  8. Scroll to the bottom of the page and click on Next.
  9. Scroll to the bottom of the page, enable I acknowledge that AWS CloudFormation might create IAM resources, and click on Update stack.
  10. While the update runs, the stack status is UPDATE_IN_PROGRES. Reload the table from time to time and …
  11. … wait until the CloudFormation stack status switches to UPDATE_COMPLETE.

Architecture

The following AWS services are used:

  • StepFunction State Machine to orchestrate the S3 bucket scan.
  • Lambda Function to fetch the list of files from the S3 bucket and push them to the Scan Queue.
  • EventBridge Cron Rule to trigger the bucket scan at regular intervals.
  • CloudWatch Alarms to monitor the used AWS services.
  • CloudWatch Logs to store logs.

Limitations

  • When using S3 bucket-policy-based access in multi-account setups, the wildcard character * within the BucketName parameter is not supported. Please use IAM role-based access instead which supports the wildcard character * within the BucketName parameter.
  • If you set PagingWaitInSeconds to a value greater than 0, the Add-On enqueues not more than 2777*PagingBatchSize objects/versions. With the maximum PagingBatchSize of 100000, the Add-On handles up to ~270 million objects/versions.
  • If you set PagingWaitInSeconds to 0, the Add-On enqueues not more than 3570*PagingBatchSize objects/versions. With the maximum PagingBatchSize of 100000, the Add-On handles up to 350 million objects/versions.

Release Notes

Subscribe to our Atom feed or newsletter to stay up-to-date! We also publish a machine-readable JSON file.

v2.15.0

Changes:

  • Reserved Concurrent Execution for Lambda functions

Release date:2024-11-19

Template: https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/v2.15.0/bucketav-add-on-scheduled-bucket-scan.yaml

v2.14.4

Changes:

  • Bug fixes

Release date:2024-10-02

Template: https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/v2.14.4/bucketav-add-on-scheduled-bucket-scan.yaml

v2.14.3

Changes:

  • Bug fixes

Release date:2024-07-09

Template: https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/v2.14.3/bucketav-add-on-scheduled-bucket-scan.yaml

v2.14.2

Changes:

  • Bug fixes

Release date:2024-04-05

Template: https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/v2.14.2/bucketav-add-on-scheduled-bucket-scan.yaml

v2.14.1

Changes:

  • Multi-account with S3 bucket policies was broken

Release date:2024-04-05

Template: https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/v2.14.1/bucketav-add-on-scheduled-bucket-scan.yaml

v2.14.0

Changes:

  • Multi-account with IAM roles

Release date:2024-03-15

Template: https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/v2.14.0/bucketav-add-on-scheduled-bucket-scan.yaml

v2.13.0

Changes:

  • Update Lambda runtime to Node.js 20

Release date:2024-02-14

Template: https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/v2.13.0/bucketav-add-on-scheduled-bucket-scan.yaml

v2.12.0

Changes:

  • Add Service Discovery

Release date:2023-12-07

Template: https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/v2.12.0/bucketav-add-on-scheduled-bucket-scan.yaml

v2.11.0

Changes:

  • Support bucket scan reports

Release date:2023-09-19

Template: https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/v2.11.0/bucketav-add-on-scheduled-bucket-scan.yaml

v2.10.3

Changes:

  • Fix issue in region us-east-1
  • IAM policy missed buckets that are added after installation during an update

Release date:2023-07-21

Template: https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/v2.10.3/bucketav-add-on-scheduled-bucket-scan.yaml

v2.10.2

Changes:

  • Fix issue which prevents an update

Release date:2023-07-19

Template: https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/v2.10.2/bucketav-add-on-scheduled-bucket-scan.yaml

v2.10.1

Changes:

  • Add trace id

Release date:2023-07-14

Template: https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/v2.10.1/bucketav-add-on-scheduled-bucket-scan.yaml

v2.10.0

Changes:

  • Add Lambda function names as outputs

Release date:2023-07-13

Template: https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/v2.10.0/bucketav-add-on-scheduled-bucket-scan.yaml

v2.9.0

Changes:

  • Support exclude scanned files
  • Support object prefix filter

Release date:2023-07-12

Template: https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/v2.9.0/bucketav-add-on-scheduled-bucket-scan.yaml

v2.8.1

Changes:

  • Only enqueue objects that can be downloaded (e.g., exclude Glacier)

Release date:2023-07-03

Template: https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/v2.8.1/bucketav-add-on-scheduled-bucket-scan.yaml

v2.8.0

Changes:

  • Support exclude bucket name filter

Release date:2023-06-28

Template: https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/v2.8.0/bucketav-add-on-scheduled-bucket-scan.yaml

v2.7.3

Changes:

  • Bucket name wildcard were broken in us-east-1

Release date:2023-03-29

Template: https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/v2.7.3/bucketav-add-on-scheduled-bucket-scan.yaml

v2.7.2

Changes:

  • Add retry configuration

Release date:2023-01-20

Template: https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/v2.7.2/bucketav-add-on-scheduled-bucket-scan.yaml

v2.7.1

Changes:

  • Only include buckets of same region when using widlcards

Release date:2022-12-20

Template: https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/v2.7.1/bucketav-add-on-scheduled-bucket-scan.yaml

v2.7.0

Changes:

  • Support scanning of multiple buckets (BucketName parameter accepts comma-separated names of S3 buckets and wildcards)

Release date:2022-09-09

Template: https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/v2.7.0/bucketav-add-on-scheduled-bucket-scan.yaml

v2.6.0

Changes:

  • Support PagingBatchSize of up to 100000

Release date:2022-08-25

Template: https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/v2.6.0/bucketav-add-on-scheduled-bucket-scan.yaml

v2.5.0

Changes:

  • Update Lambda runtime to Node.js 16

Release date:2022-06-08

Template: https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/v2.5.0/bucketav-add-on-scheduled-bucket-scan.yaml

v2.4.0

Changes:

  • Support PagingBatchSize of up to 10000

Release date:2021-11-16

Template: https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/v2.4.0/bucketav-add-on-scheduled-bucket-scan.yaml

v2.3.0

Changes:

  • Validate BucketName parameter

Release date:2021-10-14

Template: https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/v2.3.0/bucketav-add-on-scheduled-bucket-scan.yaml

v2.2.0

Changes:

  • Support configurable timeout

Release date:2021-10-11

Template: https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/v2.2.0/bucketav-add-on-scheduled-bucket-scan.yaml

v2.1.0

Changes:

  • Output IAM role ARN

Release date:2021-10-07

Template: https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/v2.1.0/bucketav-add-on-scheduled-bucket-scan.yaml

v2.0.0

Changes:

  • Initial release

Release date:2021-06-06

Template: https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/v2.0.0/bucketav-add-on-scheduled-bucket-scan.yaml

Subscribe to our Atom feed or newsletter to stay up-to-date! We also publish a machine-readable JSON file.

v2.15.0

Changes:

  • Reserved Concurrent Execution for Lambda functions

Release date:2024-11-19

Template: https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/cloudflare/v2.15.0/bucketav-add-on-scheduled-bucket-scan-cloudflare.yaml

v2.14.4

Changes:

  • Bug fixes

Release date:2024-10-02

Template: https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/cloudflare/v2.14.4/bucketav-add-on-scheduled-bucket-scan-cloudflare.yaml

v2.14.3

Changes:

  • Bug fixes

Release date:2024-07-09

Template: https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/cloudflare/v2.14.3/bucketav-add-on-scheduled-bucket-scan-cloudflare.yaml

Need more help?

Write us, and we'll get back to you as soon as we can.

Send us an email