Testing and Monitoring LLM Applications in Production

A model gives different wording for the same question. A test that compares the output to a fixed string fails on a good answer and passes on a bad one that happens to match. OpenAI says so in its docs: Chat Completions are non-deterministic by default, and a seed parameter gives only “(mostly) deterministic” output. OpenAI also exposes a system_fingerprint field so you can tell when a change on its side may have altered results. So you cannot stop the variation. You test around it.

This post splits the work into three jobs with different tools: tests for your code, evals for the model’s answers, and logs for production. I ran the Ruby tests and the evaluation example below (Ruby 4.0.6, RSpec 3.13, ActiveSupport 8.1.4; Python 3.12, langsmith 0.14.16). The model client in the Ruby tests is a stub, so none of this measures a real model.

Job 1: Test your code, not the model #

Most bugs in an LLM feature are in your code around the model: what happens on a timeout, an empty reply, a retry. Those are ordinary bugs, and a stubbed client tests them deterministically in milliseconds.

class SupportReplier
  InvalidResponse = Class.new(StandardError)
  Unavailable = Class.new(StandardError)

  def initialize(client:, retries: 2)
    @client = client
    @retries = retries
  end

  def call(question)
    attempts = 0
    begin
      attempts += 1
      response = @client.chat(parameters: { model: "any", messages: [{ role: "user", content: question }] })
    rescue Timeout::Error
      retry if attempts <= @retries
      raise Unavailable, "LLM timed out #{attempts} times"
    end
    text = response.dig("choices", 0, "message", "content")
    raise InvalidResponse, "no content" if text.to_s.strip.empty?
    text
  end
end

RSpec.describe SupportReplier do
  let(:client) { double("client") }
  let(:ok) { { "choices" => [{ "message" => { "content" => "Your order shipped." } }] } }

  it "retries timeouts, then succeeds" do
    calls = 0
    allow(client).to receive(:chat) { (calls += 1) < 3 ? raise(Timeout::Error) : ok }
    expect(described_class.new(client:, retries: 2).call("x")).to eq("Your order shipped.")
    expect(calls).to eq(3)
  end

  it "gives up after the retry budget" do
    allow(client).to receive(:chat).and_raise(Timeout::Error)
    expect { described_class.new(client:, retries: 2).call("x") }
      .to raise_error(SupportReplier::Unavailable, /3 times/)
  end

  it "raises on a response with no content" do
    allow(client).to receive(:chat).and_return({ "choices" => [] })
    expect { described_class.new(client:).call("x") }.to raise_error(SupportReplier::InvalidResponse)
  end
end

The full file has 8 examples, including one that checks the question reaches the client. All pass. To check the tests can fail, I changed the retry line to retry if false: two of the eight failed, the retry and the give-up tests. I put it back and all eight passed again.

The client call shape (chat(parameters: ...), dig("choices", 0, "message", "content")) is the one from the ruby-openai README. If you use another client, stub that client’s own call shape.

Check properties of the output #

When the output has a shape you rely on, such as JSON your code parses, test the shape against a list of replies a model has actually produced. Keep that list in a fixture file and add to it every time production surprises you. I wrote four sample replies and checked that your code either parses them or rejects them cleanly:

outputs = ['{"langs":["ruby"]}', "```json\n{}\n```", "Sure! Here you go", ""]
# result: [:json, :not_json, :not_json, :invalid]

The fenced reply and the chatty reply are the ones that break a bare JSON.parse in production. The test documents that and forces you to decide what to do with them: strip the fence, ask again, or fail.

Job 2: Evaluate answers on a fixed set #

Whether the model’s answers are good is a different question, and a mock cannot answer it. You need a set of real inputs with known good outputs, and a way to score each run.

Anthropic’s guidance on building evaluations lists three ways to grade. It calls code-based grading the fastest and most reliable, and says to choose the fastest, most reliable, most scalable method that fits:

1. Code-based grading: exact match (output == golden_answer) or string match (key_phrase in output). Fast and reliable, but it lacks nuance.
2. Human grading: the most flexible and highest quality, but slow and expensive.
3. LLM-based grading: fast, flexible and suitable for complex judgment. Anthropic’s advice is to test it for reliability before you scale it, and to give the grader a clear rubric and a constrained output such as “correct” or “incorrect”.

Use the cheapest grader that fits the question. If the task is “which team should this ticket go to”, the answer is a label and code-based grading is enough.

Using a model as a grader has known weak spots. Zheng et al. found that strong LLM judges such as GPT-4 reached over 80% agreement with human preferences, about the level humans agree with each other. The same paper examines the judges’ position, verbosity and self-enhancement biases (arXiv 2306.05685). So spot-check a sample of the judge’s grades by hand.

A runnable example #

LangSmith, LangChain’s evaluation and tracing service, runs this kind of eval. Its evaluate function takes your app as a function, a dataset, and evaluator functions (LangSmith docs). A code-based evaluator takes outputs and reference_outputs and returns a bool. I ran it locally with upload_results=False, so no account was needed, on three examples and a stand-in app that routes tickets:

from langsmith import evaluate

def target(inputs):
    return {"route": "refund_agent" if "refund" in inputs["q"] else "qa_agent"}

def correct(outputs, reference_outputs) -> bool:
    return outputs["route"] == reference_outputs["route"]

results = evaluate(target, data=examples, evaluators=[correct], upload_results=False)
print(results.to_pandas().filter(like="feedback").mean(numeric_only=True))

The result was {'feedback.correct': 0.666...}: two of three correct. The third example (“refund? no, recommend tunes”) is one I added to trip the naive keyword rule, and the eval caught it. That is the use: run the set when you change a prompt or a model, and look at which examples flipped.

Two honest notes. First, upload_results is flagged beta in langsmith 0.14.16 and warns on use. Second, in my run the plain list of dicts matched zero examples; I had to pass langsmith.schemas.Example objects for local data. Normally you point data= at a dataset name stored in LangSmith. The dataset and an experiment on the real service were not tested here.

How large should the set be, and what pass mark should gate a deploy? I have no source for a number. Start with the cases that have already gone wrong in production, add every new one, and set the gate at “no example that passed before now fails” until you have enough history to do better.

Job 3: Log every call in production #

You cannot debug a slow or expensive request you did not record. Each response includes token counts, and each call has a duration. Rails already has a bus for this, ActiveSupport::Notifications, so wrap the client once:

class InstrumentedClient
  def initialize(client) = @client = client

  def chat(parameters:)
    ActiveSupport::Notifications.instrument("llm.call", model: parameters[:model]) do |payload|
      response = @client.chat(parameters:)
      usage = response["usage"] || {}
      payload.merge!(prompt_tokens: usage["prompt_tokens"], completion_tokens: usage["completion_tokens"])
      response
    end
  end
end

Two specs ran against it: a call publishes one event with the model and both token counts, and a call that raises still publishes an event, with the exception recorded in the payload. Both pass. When I broke the usage line, the token-count spec failed.

Subscribe to llm.call and send the duration and payload wherever you already send metrics or logs. Add the feature name and a user or tenant id so you can ask “who spent this”. Tracking dollars needs provider pricing, which changes; see the cost post for reading cost from usage fields.

What to alert on is your call. I did not test any alerting rule, and I would not copy a threshold from a blog post. Look at a week of your own latency and error rates first, then alert on the change that would have hurt you.

Order of work #

1. Stub the client and test timeouts, empty replies and retries (an afternoon).
2. Instrument every model call (a small wrapper like the one above).
3. Write 10 to 20 eval cases from real failures and run them on every prompt change.
4. Add an LLM grader only for questions code cannot grade, and check its grades by hand.

Sources #

• OpenAI: reproducible outputs - non-deterministic by default, seed, system_fingerprint
• Anthropic: develop tests - code, human and LLM-based grading
• Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena - agreement with human preferences and judge biases
• LangSmith: evaluate an LLM application - evaluate, datasets, evaluator signature
• ruby-openai - client call shape used in the stubs

Related: evaluating LLM agents in Rails, debugging RubyLLM agents, working with LLMs in Rails.