Skip to main content
Applies to:
  • Plan:
  • Deployment:

​
Summary

Issue: Experiment rows were accidentally deleted and need to be restored. Resolution: Use a versioned BTQL query to retrieve rows as they existed before deletion, referencing an identified transaction version (_xact_id), then re-insert them into the experiment.

​
Resolution Steps

​
Step 1: Identify a pre-deletion _xact_id

Each row in Braintrust is stamped with a _xact_id, a monotonically increasing transaction ID. You need a value from before the deletion occurred. If rows still exist in the experiment, query for the maximum _xact_id: 
curl -X POST https://api.braintrust.dev/btql \
  -H "Authorization: Bearer <YOUR_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "SELECT MAX(_xact_id) AS max_xact_id FROM experiment('"'"'<EXPERIMENT_ID>'"'"')"
  }'
Use the max_xact_id value returned as your version in Step 2. If all rows were deleted before capturing a _xact_id, contact Braintrust support so we can help identify the correct version from internal logs.

​
Step 2: Run a versioned query to recover the rows

Pass the _xact_id as the version parameter to retrieve the experiment as it existed at that transaction: 
curl -X POST https://api.braintrust.dev/btql \
  -H "Authorization: Bearer <YOUR_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "SELECT * FROM experiment('"'"'<EXPERIMENT_ID>'"'"') ORDER BY _pagination_key LIMIT 1000",
    "version": "<XACT_ID>"
  }'
If more than 1000 rows were deleted, paginate using the cursor token returned in the response: 
curl -X POST https://api.braintrust.dev/btql \
  -H "Authorization: Bearer <YOUR_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "SELECT * FROM experiment('"'"'<EXPERIMENT_ID>'"'"') ORDER BY _pagination_key LIMIT 1000 OFFSET '"'"'<CURSOR_TOKEN>'"'"'",
    "version": "<XACT_ID>"
  }'
Repeat until no cursor is returned in the response.

​
Step 3: Re-insert the recovered rows

Strip server-managed fields before re-inserting, then POST to the experiment’s insert endpoint: 
import requests

HEADERS = {
    "Authorization": "Bearer <YOUR_API_KEY>",
    "Content-Type": "application/json",
}
STRIP_FIELDS = {"experiment_id", "project_id", "_xact_id", "_pagination_key", "audit_data"}

recovered_rows = [...]  # rows from the Step 2 response

rows_to_insert = [
    {k: v for k, v in row.items() if k not in STRIP_FIELDS}
    for row in recovered_rows
]

resp = requests.post(
    "https://api.braintrust.dev/v1/experiment/<EXPERIMENT_ID>/insert",
    headers=HEADERS,
    json={"events": rows_to_insert},
)
resp.raise_for_status()
print(f"Re-inserted {len(rows_to_insert)} rows")
After re-insertion, rows are immediately visible in the Braintrust UI.

​
Additional Information

​
Why versioned queries can be slow

Versioned queries disable segment elimination, so the query engine must scan all stored data to reconstruct state at a given transaction. Query time scales with the total storage history of the experiment, not just currently visible rows.
Versioned queries are subject to the standard 30-second BTQL timeout and there is no per-request way to extend it. If the scan times out on a very large experiment, contact Braintrust support for assisted recovery.

​
Why certain fields must be stripped before re-insertion

The /v1/experiment/{id}/insert endpoint rejects server-assigned fields with a 400 error. Strip these from each row before posting:
FieldReason
experiment_idSet from the URL path
project_idSet from the URL path
_xact_idAssigned by the server on insert
_pagination_keyAssigned by the server on insert
audit_dataRead-only metadata