AI Observability Is No Joke

A funny request turned into a lesson on AI observability

I thought I was just asking for a joke. Turns out, I stumbled into one of the challenges facing AI development today. And honestly? Here's what unfolded—and why it matters.

Picture this: I'm tinkering with my latest AI agent setup, feeling pretty proud of myself. I've got my simple MCP (Model Context Protocol) server running, connected to jokeapi.com, ready to fetch the freshest jokes from the internet. My agent is configured, the tools are registered, everything looks perfect. So I typed: "Hey agent, tell me a joke."

"Why don't scientists trust atoms? Because they make up everything!"

I chuckle. Mission accomplished, right? My agent used the API, grabbed a joke, delivered the goods. Time to pat myself on the back and move on to the next project. But then that little voice in my head started whispering. "Just where did that joke come from?"

I mean, sure, my agent said it fetched it from jokeapi.com. But did it really? Or did it just pull that gem from the vast repository of dad jokes floating around in its training data? How would I even know?

This is when I realized that while I had written the code, I had no idea of how it worked. Everything looks fine from the outside, but I had no clue as to what was happening under the hood.

Trying to answer the question of what was going on, I decided to dig deeper. I started looking at my logs, my network traffic, my agent's behavior patterns. And what I found was... well, not much. My fancy AI agent was essentially a black box wrapped in promises.

The agent could claim it called the API. It could even format its response to look like it came from an external source. But proving it actually made that HTTP request? That was surprisingly difficult.

It's like asking someone if they actually went to the store or just grabbed leftover groceries from the fridge. Both could produce milk, but the source matters if you're trying to track your spending (or in my case, API costs).

The Bigger Picture

This seemingly simple joke request opened my eyes to a massive problem in AI agent development: observability. Not the most exciting word, I'll admit, but stick with me here.

Think about it this way - when you're driving, you have a dashboard. You can see your speed, fuel level, engine temperature. You know what's happening under the hood (mostly). But with AI agents? We're essentially driving blindfolded, hoping the agent is actually doing what it claims to be doing.

The more I thought about this, the funnier (and scarier) it became. I started imagining all the different ways my agent could be "fetching" that joke without actually calling the API. Imagine these scenarios:

• The Overachiever: Calls five APIs when one would do.
• The Lazy Agent: Fakes it with old data.
• The Confused Agent: Gives up silently but still logs success.
• The Identity Crisis Agent: Thinks it made a call… but didn’t.

Why This Matters

This isn't just about jokes or my particular brand of overthinking. As AI agents become more sophisticated and handle more critical tasks, this observability gap becomes genuinely problematic:

Customer Service:

Did the agent actually check your account status, or is it giving you a generic response?

Financial Applications:

When the agent says it's pulling real-time market data, is it really, or are you making decisions based on stale information?

Healthcare:

If an AI assistant claims to have checked the latest research, you'd better hope it actually did.

Development:

When your coding assistant says it's following best practices from the latest documentation, did it actually access that documentation or just wing it?

AI agents touch everything—from customer service to healthcare. We must know what they actually do, not just what they say they do.

Agents chain actions across tools and APIs. Without logs and proofs, it's a game of telephone with hallucinations.

Observability Best Practices

So, what can we do? Here are some best practices I'm adopting to ensure my AI agents are more transparent and accountable:

• Trust, but Verify: Don't just ask—prove it.
• Embrace Logs: Log intent, attempts, and failures.
• Test Boundaries: Disable tools and see if the agent notices.
• Build for Observability: Make it foundational, not an afterthought.

That joke? Still funny. But if my agent were making business decisions, I'd want proof. The joke's on us if we don't demand it.

Model Context Protocol (MCP) Glossary

Since MCP is central to this discussion, here's a comprehensive glossary of key terms and concepts that will help you understand how this protocol enables AI agent observability.

Code Samples

For those interested in diving deeper, here is the simple C# code snippet that demonstrates how to implement MCP servers and tools for my basic joke-fetching tool. With no observability. Notice the lack of logging or proof of API calls.

MCP Server

// Create a generic host builder for
// dependency injection, logging, and configuration.
var builder = Host.CreateApplicationBuilder(args);

// Configure logging for better integration with MCP clients.
builder.Logging.AddConsole(consoleLogOptions =>
{
  consoleLogOptions.LogToStandardErrorThreshold = LogLevel.Trace;
});

// Register the MCP server and configure it to use stdio transport.
// Scan the assembly for tool definitions.
builder.Services
  .AddMcpServer()
  .WithStdioServerTransport()
  .WithToolsFromAssembly();

// Register HttpClient for API calls
builder.Services.AddHttpClient();

// Build and run the host. This starts the MCP server.
await builder.Build().RunAsync();

Joke Tool

/// <summary>
/// Tool implementations using static methods
/// </summary>
public static class Tools
{

  /// <summary>
  /// Fetches a random joke from JokeAPI
  /// </summary>
  /// <returns>A random programming joke</returns>
  [Description("Fetches a random joke from JokeAPI")]
  public static async Task<string> GetJoke()
  {
    using var client = new HttpClient();
    try
    {
      var response = await client.GetFromJsonAsync<JokeResponse>(
      "https://v2.jokeapi.dev/joke/Programming?safe-mode");

      string joke = response?.Type == "single"
      ? response.Joke ?? "No joke available"
      : $"{response?.Setup}\n{response?.Delivery}";

      return $"JOKE: {joke}";
    }
    catch (Exception ex)
    {
      return $"Error fetching joke: {ex.Message}";
    }
  }
}

Final Thoughts

We need to ask harder questions of our AI systems. Not "can they do it?"—but "did they actually do it?" Because in the end, observability isn’t optional. It’s essential.