Scan bucket at regular intervals Add-On

This add-on enqueues all files in a bucket for scanning at regular intervals—the EC2 instances of bucketAV scan the files.

Table of Contents

Setup

Install Add-On

  1. Set the Stack name to bucketav-scheduled-bucket-scan.
  2. Set the BucketAVStackName parameter to the stack name of bucketAV (if you followed our docs, the name is bucketav, or s3-virusscan for older installations).
  3. Set the BucketName parameter to the name of the S3 bucket that you want to scan.
  4. Set the ScheduleExpression parameter to a valid expression . E.g., rate(1 day), rate(7 days), or rate(1 hour).
  5. Select I acknowledge that AWS CloudFormation might create IAM resources.
  6. Click on the Create stack button to save.

Constraints

If your bucket contains more than 100 million objects/versions (see the NumberOfObjects CloudWatch metric of your bucket), please send us an email for guidance!

  • If you set PagingWaitInSeconds a value greater than 0, the add-on enqueues not more than 8333*PagingBatchSize objects/versions. With the maximum PagingBatchSize of 10000, the add-on handles up to ~83 million objects/versions.
  • If you set PagingWaitInSeconds to 0, the add-on enqueues not more than 12500*PagingBatchSize objects/versions. With the maximum PagingBatchSize of 10000, the add-on handles up to 125 million objects/versions.

Insights

To get insights into running and completed bucket scan runs:

  1. Visit the Step Functions Management Console .
  2. Click on the state machine (if you followed our docs, the name is bucketav-scheduled-bucket-scan).

You will see a list of Executions, the most recent execution is at the top and represents the latest bucket scan. If the status equals Succeeded, the bucket scan is complete. If the status equals Running, the bucket scan is running.

Performance optimization

By default, the add-on is configured to protect your bucketAV Scan Queue from overload. Every 30 seconds (controlled by parameter PagingWaitInSeconds), 50 files (controlled by parameter PagingBatchSize) are enqueued for scanning.

We recommend tweaking the two parameters in the following cases:

  1. You want to speed up the bucket scan.
  2. In the Step Functions Management Console , click on the state machine (if you followed our docs, the name is bucketav-scheduled-bucket-scan), and the last execution status is Timed out.
  3. In the Step Functions Management Console , click on the state machine (if you followed our docs, the name is bucketav-scheduled-bucket-scan), and the last execution status is Failed. The last event errored with “The execution reached the maximum number of history events (25000)” error.
    1. To check for this error, open a AWS CloudShell .
    2. Ensure that you are in the correct region where bucketAV runs.
    3. Execute the following commands(replace EXECUTION_ARN with the Execution ARN; In the Step Functions Management Console , click on the last execution to get the value): aws stepfunctions get-execution-history --execution-arn EXECUTION_ARN --reverse-order --max-items 3 Copy

We recommend optimizing by:

  1. Doubling PagingBatchSize (maximum 1,000) and halving PagingWaitInSeconds (minimum 0).
  2. Execute the scan again:
    1. Visit the Step Functions Management Console .
    2. Click on the state machine (if you followed our docs, the name is bucketav-scheduled-bucket-scan.
    3. Click on the last execution.
    4. Click on New Execution.
    5. Confirm with Start execution.
  3. If you still see the error, go to step 1.

Running a full bucket scan only once

To run a full bucket scan only once, set the ScheduleExpression parameter to a value in 10 minutes from now (UTC timezone ) by using the expression cron(mm hh dd MM ? yyyy) filled with:

  1. mm: Minute (0-59).
  2. hh: Hour (0-23).
  3. dd: Day (1-31).
  4. MM: Month (1-12).
  5. ?: Please leave the question mark as it is.
  6. yyyy: Year (1970-2199).

Update

Which version am I using?

  1. To update this add-on to version v2.4.0, go to the AWS CloudFormation Management Console .
  2. Double-check the region at the top right.
  3. Search for bucketav-scheduled-bucket-scan (or s3-virusscan-scheduled-bucket-scan for older installations), otherwise search for the name you specified.
  4. Select the stack and click on Update.
  5. Select Replace current template and set the Amazon S3 URL to https://bucketav-add-ons.s3.eu-west-1.amazonaws.com/scheduled-bucket-scan/v2.4.0/bucketav-add-on-scheduled-bucket-scan.yaml Copy
  6. Click on Next.
  7. Scroll to the bottom of the page and click on Next.
  8. Scroll to the bottom of the page and click on Next.
  9. Scroll to the bottom of the page, enable I acknowledge that AWS CloudFormation might create IAM resources, and click on Update stack.
  10. While the update runs, the stack status is UPDATE_IN_PROGRES. Reload the table from time to time and …
  11. … wait until the CloudFormation stack status switches to UPDATE_COMPLETE.

Architecture

The following AWS services are used:

  • StepFunction State Machine to orchestrate the S3 bucket scan.
  • Lambda Function to fetch the list of files from the S3 bucket and push them to the Scan Queue.
  • EventBridge Cron Rule to trigger the bucket scan at regular intervals.
  • CloudWatch Alarms to monitor the used AWS services.

Need more help?

Write us, and we'll get back to you as soon as we can.

Send us an email