Act on findings from Topics

Documentation Index

Fetch the complete documentation index at: https://braintrust.dev/docs/llms.txt

Use this file to discover all available pages before exploring further.

​

Build datasets from topics

1. Go to Logs and click Filter.
2. Select Classifications and choose the classification you want to filter by. Alternately, click SQL and enter a filter clause. See the SQL reference for more query patterns.
   Filter by specific topic

   Filter by multiple topics

   Combine topics across facets

   classifications.Task.label = "Dataset creation"
   
3. Select the logs you want to include.
4. Click + Dataset and choose an existing dataset or create a new one.

• “Error investigation” tasks → test your error handling.
• Negative sentiment interactions → improve responses.
• “Pricing questions” → evaluate your pricing explanations.

​

Score logs based on topics

import braintrust from "braintrust";
import { z } from "zod";

const project = braintrust.projects.create({ name: "my-project" });

project.scorers.create({
  name: "Checkout experience",
  slug: "checkout-experience",
  description: "Flag traces with negative checkout experiences",
  parameters: z.object({
    trace: z.any(),
  }),
  handler: async ({ trace }) => {
    if (!trace) return { score: null };

    const spans = await trace.getSpans();
    const rootSpan = spans.find((s) => s.span_id === s.root_span_id);
    if (!rootSpan) return { score: null };

    const classifications = rootSpan.classifications || {};
    const taskClassification = (classifications.Task || [{}])[0];
    const sentimentClassification = (classifications.Sentiment || [{}])[0];

    if (
      taskClassification.label === "Checkout Flow" &&
      sentimentClassification.label === "NEGATIVE"
    ) {
      return {
        score: 0,
        metadata: { reason: "Negative sentiment during checkout" },
      };
    }

    return { score: 1 };
  },
});

bt functions push topic_scorer.ts

import braintrust
from pydantic import BaseModel

project = braintrust.projects.create(name="my-project")

class TraceParams(BaseModel):
    trace: dict

async def checkout_experience_scorer(trace):
    if not trace:
        return {"score": None}

    spans = await trace.get_spans()
    root_span = next(
        (s for s in spans if s.get("span_id") == s.get("root_span_id")),
        None
    )
    if not root_span:
        return {"score": None}

    classifications = root_span.get("classifications", {})
    task_classification = classifications.get("Task", [{}])[0]
    sentiment_classification = classifications.get("Sentiment", [{}])[0]

    if (
        task_classification.get("label") == "Checkout Flow"
        and sentiment_classification.get("label") == "NEGATIVE"
    ):
        return {
            "score": 0,
            "metadata": {"reason": "Negative sentiment during checkout"},
        }

    return {"score": 1}

project.scorers.create(
    name="Checkout experience",
    slug="checkout-experience",
    description="Flag traces with negative checkout experiences",
    parameters=TraceParams,
    handler=checkout_experience_scorer,
)

bt functions push topic_scorer.py

1. Go to Settings > Automations and click + Rule.
2. Select your scorer, set Scope to Trace, configure the sampling rate, and click Create rule.

​

Assign topics for review

1. Go to Logs and click Filter.
2. Select Classifications and choose the classification you want to filter by. Alternately, click SQL and enter a filter clause. See the SQL reference for more query patterns.
   Filter by specific topic

   Filter by multiple topics

   Combine topics across facets

   classifications.Task.label = "Dataset creation"
   
3. Select the logs you want to assign.
4. Select Assign and choose a team member.

​

Next steps

• Manage Topics to tune the pipeline that produces these classifications.
• Custom facets to define new dimensions to act on.