Debugging RubyLLM Agents in Rails

Debugging RubyLLM agents in Rails keeps landing me on the same question: what is a green test suite actually asserting when the thing under test lives on someone else’s server?

Last week we moved all nine schemas in the talent-matching pipeline we run onto a different schema library. Every request body we send changed shape. All 1549 tests stayed green, and by the rules they had been given, they were right to.

Only one of the three incidents below was ever a test’s job to catch. That test had to measure something strange before it worked.

Nine schemas swapped, 1549 tests green #

VCR’s default request matcher is [:method, :uri]. DEFAULT_MATCHERS = [:method, :uri] sits at line 9 of request_matcher_registry.rb in vcr 6.4.0, and configuration.rb hands that array to every cassette that never sets its own match_requests_on.

Ours never set one.

So moving off RubyLLM::Schema onto the schematist gem was invisible to the suite. ruby_llm duck-types the schema object - with_schema calls to_json_schema on anything that responds to it - so on our side it read as a base-class change with an identical DSL (how the pipeline is wired).

Underneath, the payload changed. schematist emits a bare Draft 2020-12 document, so $schema and title now ride inside the strict schema we send, and the json_schema name collapsed to "response". Same method, same URI, so every cassette replayed and every assertion passed.

We found out the new payload was accepted the slow way, by making live strict-mode calls against two hosted models by hand before merging. The suite had no opinion.

Answer quality is a different measurement entirely; we get that from a per-iteration audit trail.

What the cassette setup still protects #

Two habits in test_helper.rb do hold up.

VCR.configure do |c|
  c.allow_http_connections_when_no_cassette = false

  c.filter_sensitive_data("<OPENROUTER_KEY>") { ENV["OPENROUTER_API_KEY"] }
  c.filter_sensitive_data("<OPENROUTER_KEY>") { |i| i.request.headers.to_s[/sk-or-v1-[0-9a-f]{64}/] }
  c.filter_sensitive_data("<BEARER_JWT>") { |i| i.request.headers.to_s[/Bearer [\w-]+\.[\w-]+\.[\w-]+/] }
  # same shape filter again for sk-ant- and AIza
end

That first line is VCR’s default, and it is worth writing down anyway: no test can quietly reach a paid API, because a request with no cassette raises instead of dialing out. Setting it explicitly means nobody flips it while chasing a red build.

Filtering only on ENV["OPENROUTER_API_KEY"] misses a key that arrived some other way, hardcoded on a branch, pasted into a fixture, or exported in a teammate’s shell. So the scrubbing also matches shapes: sk-or-v1- plus 64 hex, sk-ant- plus 95 characters, AIza plus 35, and the three-segment Bearer JWT.

Cassette names come from the test class name through a small concern (name.underscore.sub("_test", "")), so nobody hand-names a cassette or typos one into a silent re-record.

The model went away and no test noticed #

A separate morning, every LLM feature in production started erroring at once: candidate scoring, query expansion, filter evaluation, reflection, job-description analysis.

I ran the smallest thing the gem has, .ask("say ok") with no schema and no tools (the basics if you haven’t used it), and back came a deprecation notice where the answer should have been. Whether the model was being retired or quietly substituted, the effect was the same: nothing in our code had changed, and every agent was talking to something that no longer answered the way it used to.

No test could have caught that, because every AI test in the suite replays a cassette recorded back when the model still answered.

The model we moved to cost more per token and had a shorter context window, with no like-for-like tier to step across to. Nothing in our code was wrong. The afternoon went into re-tuning prompts for the shorter window and re-recording cassettes.

A cassette is a recording of a conversation that is no longer happening. Nothing in the suite tells you when the recording has gone stale, so re-recording has to sit on somebody’s calendar. Catching a retirement while it happens belongs to monitoring.

Measure something the test harness can’t flatten #

The third incident is the one a test could have owned. A deploy wrapped each scoring call in with_connection, which holds a connection for the whole block, and that block sat waiting on the model for several seconds. The fibers fanning the scoring out drained the pool into ActiveRecord::ConnectionTimeoutError - the outage itself is written up here. Scoring issues no queries of its own, so it should hold no connection at all.

Deleting the wrapper took a minute. I wanted the test that fails the next time someone re-adds it.

Counting busy connections in the pool cannot see this. Transactional tests call pin_connection!, which pins one shared connection for the whole test (connection_adapters/abstract/connection_pool.rb:366), so every checkout inside the test hands back that same connection and the busy count never moves.

ActiveRecord::Base.connection_pool.active_connection? asks a different question: does the current execution context hold a lease? It’s public API, documented at connection_pool.rb:419, and it returns connection_lease.connection - a connection object, or nil.

Two caveats the Rails docs state right there. It only sees connections taken through lease_connection or with_connection, never checkout. And it hands back a connection object rather than a boolean, which is why the assertion below reads assert_nil instead of assert_not.

A third caveat the docs don’t state, and the one that cost me an afternoon: transactional fixtures populate that lease during setup. test_fixtures.rb:201-204 calls pin_connection! and then lease_connection on every fixture pool, which sets the lease and marks it sticky, so active_connection? returns a live adapter before your test body runs a single line. Assert on it as-is and the test is red on a clean codebase. Drop the fixture’s lease first - the transaction stays open, only the lease clears.

Every strategy here gets swapped through config rather than by stubbing gem internals (fakes over mocks), so the test setup does RAG.scorer = RAG::FakeCandidateScorer. The fake is where we hang the observation.

# test/support/rag/fake_candidate_scorer.rb
def score(candidate, role)
  calls << {
    candidate: candidate,
    role: role,
    leased_connection: ActiveRecord::Base.connection_pool.active_connection?
  }
  stubbed_score
end

# test/rag/scoring_test.rb
ActiveRecord::Base.connection_pool.release_connection  # drop the fixture's lease
run_scoring_step

assert_nil scorer.calls.first[:leased_connection]

The commit message says what the test is for: test(rag): pin the invariant that scoring leases no DB connection (f95c0d3ec).

When you don’t need any of this #

One agent making one call needs nothing here.

A .ask with a schema and a stubbed client covers you, plus a calendar reminder to re-record the cassettes every so often.

All of it starts paying off when several agents run inside one request and a failure has more than one plausible cause.

If the request payloads are the thing you actually care about, put :body into match_requests_on and accept the cassette re-recording that follows every prompt tweak. And check model availability on a schedule, because the suite will never raise a hand about it.

If several agents run per request in your Rails app and nobody can name which one changed last Tuesday, that untangling is what our team gets hired for.

Further reading:

• VCR default cassette options - match_requests_on and the rest of the cassette defaults
• VCR on GitHub - source, including request_matcher_registry.rb
• ActiveRecord ConnectionPool API - active_connection?, lease_connection, with_connection
• ruby_llm chat.rb at 1.16.0 - the duck-typed with_schema
• schematist - the JSON Schema DSL we moved onto
• RubyLLM configuration guide - where the model and provider get named, in one place
• Mocks Aren’t Stubs - Martin Fowler on fakes, stubs, and what each one can observe