AI agents are powerful but they are like black boxes — you send a question and get an answer, but you have no idea what happened in between. Which step was slow? Did the LLM fail? How many tokens did it use?

I built Agent Debugger to solve this problem — a Python AI agent where every single step is visible in SigNoz using OpenTelemetry.

This was built for the Agents of SigNoz hackathon by WeMakeDevs.

What is the Project?

Agent Debugger is an AI agent that answers questions in 3 steps:

1. Plan — The LLM converts the user question into a search query

2. Search — A search tool runs the query

3. Synthesize — The LLM uses the search results to give a final answer

Every step sends a trace to SigNoz so you can see exactly what happened.

Tech Stack

• Python — main programming language

• Groq + LLaMA 3.1 — free and fast AI model

• OpenTelemetry — tracks every step as a span

• SigNoz — visualizes all the traces and spans

• Docker — runs SigNoz locally

Architecture

Setting Up SigNoz

I used the new Foundry CLI to install SigNoz with Docker:

curl -fsSL https://signoz.io/foundry.sh | bash
foundryctl cast -f casting.yaml

Then I opened http://localhost:8080 and created my account. SigNoz was running in minutes!

Instrumenting the Agent with OpenTelemetry

I set up OpenTelemetry to send traces to SigNoz using OTLP over gRPC on port 4317:

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider(resource=resource)
exporter = OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True)
provider.add_span_processor(BatchSpanProcessor(exporter))

Wrapping Each Agent Step in a Span

Each step of the agent is wrapped in its own span:

def _llm_call(span_name, system_prompt, user_prompt):
    with tracer.start_as_current_span(span_name) as span:
        span.set_attribute("gen_ai.system", "groq")
        span.set_attribute("gen_ai.request.model", MODEL)
        span.set_attribute("gen_ai.prompt", user_prompt)
        response = client.chat.completions.create(...)
        span.set_attribute("gen_ai.completion", content)
        return content

This means every LLM call appears as its own span in SigNoz with all the details attached.

What it Looks Like in SigNoz

When I run the agent and open SigNoz Traces Explorer, I can see:

• agent.run — the full request (1.26 seconds total)

• llm.plan — LLM planning step (179ms)

• tool.search — search tool (190ms)

• llm.synthesize — final answer generation (889ms)

The waterfall view shows exactly where time is being spent. If the LLM is slow, I can see it. If the tool fails, I can see the error in the span.

What I Learned

• OpenTelemetry makes it easy to add observability to any Python app

• SigNoz gives a beautiful UI to visualize traces without any complex setup

• AI agents need observability just like microservices do

• The span waterfall is incredibly useful for debugging agent performance

Conclusion

Before this project, AI agents were a black box to me. Now with SigNoz and OpenTelemetry, every step is visible and debuggable. This is the future of AI observability.

GitHub: https://github.com/nimithpoojary/agent-signoz-debugger

Built for Agents of SigNoz Hackathon by WeMakeDevs