> ## Documentation Index
> Fetch the complete documentation index at: https://braintrust.dev/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# CloudFormation

<Warning>
  Do not use this method for new deployments. For supported methods, see [Cloud providers](/admin/self-hosting/index#cloud-providers).
</Warning>

Follow these steps to set up Braintrust in your AWS account using CloudFormation.

Before following these instructions, make sure you have account access and are able to sign into [Braintrust](https://www.braintrust.dev/app).

You can set up the stack using either:

1. **Braintrust CLI** (recommended): Requires a single command with AWS credentials
2. **AWS console**: Run the process through the AWS web interface if you need more control or cannot use CLI credentials

## Set up the stack

<Tabs>
  <Tab title="CLI">
    <Steps>
      <Step title="Install the CLI">
        Install the latest Braintrust CLI:

        ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
        pip install --upgrade 'braintrust[cli]'

        # verify the installation worked
        braintrust --help
        ```
      </Step>

      <Step title="Create the CloudFormation stack">
        There are just a few relevant parameters you should consider for most use cases:

        * A stack name. This is arbitrary and allows you to refer back to the stack later. A name like "braintrust"
          or "braintrust-dev" should be fine.
        * `--org-name` should be the name of your organization (you can find this in your URL on the app,
          e.g. `https://www.braintrust.dev/app/<YOUR_ORG_NAME>/...`). This will ensure that only users
          with access to your organization can invoke commands on your Braintrust endpoint.
        * `--provisioned-concurrency` the number of lambda workers to keep running in memory.
          This is useful if you expect to have a lot of concurrent users, or if you want to reduce the cold-start
          latency of your API calls. Each increment costs about \$40/month in AWS costs. The default is 0.
        * `--template` if you are deploying in a region other than `us-east-1`, you should specify the template [for that region](#set-up-the-stack).

        ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
        braintrust install api --create <YOUR_STACK_NAME> \
          --org-name <YOUR_ORG_NAME> \
          --provisioned-concurrency 1
        ```

        <Warning>
          This command is idempotent. If it fails, re-run the command without the
          `--create` flag, and it will resume tracking the progress of your
          CloudFormation stack. If your stack enters a failed state, e.g. it fails to
          create, please [reach out to support](mailto:support@braintrust.dev).
        </Warning>

        Once the install completes, you'll see a log statement like

        ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
        Stack with name braintrust has been updated with status: UPDATE_COMPLETE_CLEANUP_IN_PROGRESS
        Universal URL: https://dfwhllz61x709.cloudfront.net
        ```

        Save the endpoint URL. You can now skip ahead to the [Verifying the stack](#verify-the-stack) section.
      </Step>
    </Steps>
  </Tab>

  <Tab title="CloudFormation console">
    <Steps>
      <Step title="Create the CloudFormation stack">
        The latest CloudFormation template for Braintrust is available in the following regions:

        #### US East 1

        {`https://braintrust-cf-us-east-1.s3.amazonaws.com/braintrust-latest.yaml`}

        <br />

        #### US East 2

        {`https://braintrust-cf-us-east-2.s3.amazonaws.com/braintrust-latest.yaml`}

        <br />

        #### US West 2

        {`https://braintrust-cf-us-west-2.s3.amazonaws.com/braintrust-latest.yaml`}

        <br />

        To start in us-east-1, [click here to open up the CloudFormation setup window](https://console.aws.amazon.com/cloudformation/home#/stacks/create/review?templateURL=https://braintrust-cf-us-east-1.s3.amazonaws.com/braintrust-latest.yaml). If you prefer, you can also click “Create Stack” directly in the CloudFormation console and specify one of the URLs above as the S3 template link.

        <Note>
          These instructions walk through how to setup through the AWS UI, but you are
          welcome to install it via the command line too if you prefer.
        </Note>

        You do not need to set any parameters to create a stack; however, there are several you can set to configure behavior. The most important
        ones to consider while creating a stack are:

        * `OrgName` - The name of your organization. This will restrict access to users in your organization. By default it is set to `*` which means all users can query the endpoint. However, rest assured,
          only users in your org will be able to access its resources (due to access control checks).
        * `ProvisionedConcurrency` - The number of lambda workers to keep running in memory. This is useful to set if you want to minimize the cold-start latency of your API calls. Each increment costs
          about \$40/month in AWS costs. The default is 0.

        Once you fill in the parameters, accept the acknowledgments and select **Create stack** to start creating the template. This can take a few minutes (up to \~10) to provision the first time.

        <Note>
          Behind the scenes, the template sets up a few key resources:

          * A VPC with public & private subnets for networking
          * A few lambda functions which contain the logic for executing Braintrust commands
          * An API gateway that runs commands against the lambda functions
          * An auto-scaling group for Brainstore
          * A Postgres database, Redis cache, and S3 buckets

          <Warning>
            **Redis Performance Note**: If you're experiencing performance issues, avoid using burstable Redis instances (t-family instances like `cache.t4g.micro`). These instances use CPU credits that can be exhausted during high-load periods, leading to performance throttling. Consider upgrading to non-burstable instances like `cache.r7g.large` or `cache.r6g.medium` for predictable performance.
          </Warning>
        </Note>
      </Step>

      <Step title="Get the Universal URL">
        Once the stack is provisioned, you should see `UPDATE_COMPLETE` as its status.

        Navigate into the stack, select the **Outputs** tab, and copy the value for `UniversalURL`. We'll use this to test your endpoint and configure Braintrust to access your org through it. For the rest of the doc, we'll refer to it as `<YOUR_UNIVERSAL_URL>`.
      </Step>
    </Steps>
  </Tab>
</Tabs>

## Verify the stack

Run the following command to test that the stack is running. The first time you run it, AWS may do some setup work to provision your lambda function, and it can take up to 30 seconds to run.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
curl -X GET '<YOUR_UNIVERSAL_URL>/ping'
```

You should see a response like `{"id":"80c48a10-4888-4382-a55b-255018e70fe5","email":"___braintrust_anon_user___@braintrustdata.com","organizations":[]}`.

## Configure your organization

Visit your organization's [settings page](http://www.braintrust.dev/app/settings) and set the API URL captured above. You can skip the realtime URL and proxy URL, unless
you have an advanced need that requires it.

Once you configure the URL, select **Save**. The page automatically attempts to test that you're authorized to access the URL.

<Note>
  The `braintrust install api` command tries to install these values for you. So if you see them already filled in, no need to change them!
</Note>

## Test the application

Hooray! At this point you should be able to test the full application. The easiest way to do this is by using the Python SDK.

This simple Python script will run a full loop of using Braintrust and setting up an experiment. You can specify your API key in the `api_key` parameter or set it as the `BRAINTRUST_API_KEY` environment variable. The SDK will automatically use the API URL you configured in the settings page.

```python theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
import braintrust

experiment = braintrust.init(project="SetupTest", org_name="my_org.com", api_key="sk-****")  # Replace with your org name and API key
experiment.log(
    input={"test": 1},
    output="foo",
    expected="bar",
    scores={
        "n": 0.5,
    },
    metadata={
        "id": 1,
    },
)
print(experiment.summarize())
```

## Update the stack

Most new Braintrust releases do not require stack updates. Occasionally, however, you will need to
update the stack to get access to new features and performance enhancements. Like installation, you can
update the stack through either the CLI or AWS console.

<Tabs>
  <Tab title="CLI">
    To update your stack, run the following (replacing `<YOUR_STACK_NAME>`):

    ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
    braintrust install api <YOUR_STACK_NAME> --update-template
    ```

    You can also use this command to change parameters, with or without template updates. For example, if you
    want to allocate provisioned concurrency `8` to your lambda functions, run

    ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
    braintrust install api <YOUR_STACK_NAME> --provisioned-concurrency 8
    ```
  </Tab>

  <Tab title="AWS console">
    You can also update the stack directly through AWS. [Their docs](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-direct.html) walk through how to update through the console
    and the `aws` cli.

    The rest of the guide covers topics only needed for advanced
    configurations.
  </Tab>
</Tabs>

## Advanced

To permit incoming and outgoing traffic between Braintrust's lambda functions and other internal cloud resources, you can either run everything in the same VPC or setup VPC peering.
This is necessary if you want to access resources like an LLM gateway or a database that is not publicly accessible.

### VPC Peering

When you create your Braintrust CloudFormation, it automatically creates a VPC with the same name as your CloudFormation.

You can access the Braintrust VPC's name and ID from the CloudFormation's `Outputs` tab (named `pubPrivateVPCID`).

AWS has a [comprehensive guide](https://docs.aws.amazon.com/vpc/latest/peering/working-with-vpc-peering.html) for configuring peering. Follow the instructions for

* [Create a VPC peering connection](https://docs.aws.amazon.com/vpc/latest/peering/create-vpc-peering-connection.html)
* [Accept a VPC peering connection](https://docs.aws.amazon.com/vpc/latest/peering/accept-vpc-peering-connection.html)
* [View your VPC peering connections](https://docs.aws.amazon.com/vpc/latest/peering/describe-vpc-peering-connections.html)
* [Update your route tables for a VPC peering connection](https://docs.aws.amazon.com/vpc/latest/peering/vpc-peering-routing.html)
  * Make sure to update the route tables in both VPCs.
* [Update your security groups to reference peer security groups](https://docs.aws.amazon.com/vpc/latest/peering/vpc-peering-security-groups.html). We recommend allowing “All Traffic” from Braintrust’s VPC.

### Troubleshooting

* If you continue to see errors after updating the VPC peering group, you may need to update your CloudFormation template (which will effectively reboot your Lambda functions). You can do this by triggering an update on the CloudFormation and letting it run. You may need to change a stack parameter and then change it back to trigger the updates.
* You can manually test network settings by booting up an EC2 machine in the Braintrust VPC to test connectivity. Make sure to assign a public IP to the instance and use the public subnet of the VPC while initializing.