Debug LangChain Agents with LangSmith CLI

When a LangChain agent fails to retrieve the correct information or loops endlessly through tool calls, pinpointing the issue requires visibility into each decision point. LangSmith CLI provides command-line access to trace data, allowing developers to examine agent behavior without switching to a web interface during active debugging sessions.

Use Cases

The LangSmith CLI excels when debugging agent workflows that involve multiple tool invocations. Developers can filter traces by metadata, search for specific error patterns, or export execution data for offline analysis. This proves particularly valuable when investigating production issues where reproducing the exact conditions in a development environment becomes challenging.

Teams working with continuous integration pipelines benefit from programmatic trace access. The CLI enables automated checks that flag traces exceeding latency thresholds or containing specific error types. A Python script can query recent traces, parse the JSON output, and trigger alerts when agents make unexpected tool selections.

Local development workflows gain efficiency through CLI-based trace inspection. Rather than navigating browser tabs, developers can tail traces in real-time from their terminal, filtering by session ID or custom tags. This approach integrates naturally with existing command-line tools like grep, jq, and awk for sophisticated trace analysis.

Configuration

Installation requires the LangSmith SDK, which includes the CLI tools:

pip install langsmith

Authentication uses an API key from https://smith.langchain.com. Set the environment variable before running CLI commands:

export LANGSMITH_API_KEY="lsv2_pt_your_key_here"

The CLI connects to a specific project within LangSmith. Specify the project name through an environment variable or command-line flag:

export LANGSMITH_PROJECT="agent-debugging"

Verify the setup by listing recent traces:

langsmith trace list --limit 5

This command returns JSON-formatted trace metadata including run IDs, timestamps, and execution status. The output structure allows piping to other tools for further processing.

Advanced Usage

Filtering traces by custom metadata enables targeted debugging. When instrumenting LangChain agents, developers can add tags or metadata fields that the CLI later uses for filtering:

from langsmith import Client

client = Client()
# Tag traces during agent execution
client.create_run(
    name="customer_query",
    run_type="chain",
    inputs={"query": "pricing information"},
    extra={"environment": "staging", "user_id": "12345"}
)

Retrieve only traces matching specific criteria:

langsmith trace list --filter 'metadata.environment = "staging"' --limit 10

The CLI supports exporting complete trace trees for offline analysis. This becomes critical when debugging complex agent behaviors that require comparing multiple execution paths:

langsmith trace export --run-id abc123 --output trace.json

Real-time trace monitoring helps catch issues as they occur. The watch mode refreshes trace listings at specified intervals:

langsmith trace list --watch --interval 5

Combining CLI commands with shell scripting creates powerful debugging workflows. Extract all failed agent runs from the past hour and count tool invocation patterns:

langsmith trace list --filter 'status = "error"' --start-time "-1h" | \
  jq '[.[] | .outputs.tool_calls[].tool] | group_by(.) | map({tool: .[0], count: length})'

Caveats

The CLI requires network connectivity to the LangSmith API, making it unsuitable for completely offline debugging scenarios. Trace data remains on LangSmith servers, so organizations with strict data residency requirements need to evaluate whether this aligns with their policies.

Rate limiting applies to API requests, which can affect scripts that query traces frequently. The exact limits depend on the LangSmith plan tier. Developers should implement exponential backoff when building automated trace analysis tools.

Large trace trees with extensive agent loops can produce JSON output that overwhelms terminal displays or exceeds shell buffer limits. Redirecting output to files becomes necessary for complex debugging sessions.

The CLI does not currently support real-time streaming of trace events as they occur within a running agent. Developers must poll for updates, introducing slight delays between agent execution and trace availability. This gap typically measures in seconds but can complicate debugging of time-sensitive race conditions.

Version compatibility between the LangSmith CLI and the LangChain library matters. Mismatched versions occasionally produce trace formats that the CLI cannot parse correctly. Keeping both packages updated to compatible versions prevents these issues.