Building RAG Applications in Rails With pgvector

Keyword search matches strings. A user who types “best pizza spots” finds nothing if your data says “top-rated pizzerias”. Retrieval-augmented generation (RAG) fixes that in two steps: find the stored passages closest in meaning to the question, then hand only those passages to a language model and ask it to answer from them.

If your Rails app already runs PostgreSQL, the first step can live in the same database. This post builds that retrieval layer with the pgvector extension and the neighbor gem, then adds the generation call. Everything database-side below was run against PostgreSQL with pgvector 0.8.7, ActiveRecord 8.1.4 and neighbor 1.2.0.

One honest limit up front: the embeddings in my test run came from a toy hashing function, not from OpenAI. The retrieval code, schema, index and queries are what ran. The OpenAI calls follow the ruby-openai README and were not executed against the live API.

Schema: documents, chunks, one vector column #

A model reads a limited amount of text, and a 10,000-word document is mostly irrelevant to any one question. So you split documents into chunks and store one vector per chunk.

class CreateDocumentChunks < ActiveRecord::Migration[8.1]
  def change
    enable_extension "vector"

    create_table :documents do |t|
      t.string :title
      t.text :content
      t.timestamps
    end

    create_table :document_chunks do |t|
      t.references :document, null: false, foreign_key: true
      t.text :content, null: false
      t.vector :embedding, limit: 1536 # dimensions
      t.integer :position, null: false
      t.timestamps
    end
  end
end

1536 is the default vector length of OpenAI’s text-embedding-3-small; text-embedding-3-large defaults to 3072. Both lengths are in OpenAI’s embeddings guide, which also describes a dimensions parameter for shortening them.

class DocumentChunk < ApplicationRecord
  belongs_to :document
  has_neighbors :embedding
end

has_neighbors comes from the neighbor gem. It builds the SQL for you. In neighbor 1.2.0 (the version I ran) nearest_neighbors validates the query vector’s length (dimensions ||= column_info&.limit, model.rb lines 94-95) and inserts it as a quoted literal via connection.quote (line 102). So has_neighbors :embedding needs no dimensions: option here: the length comes from the column’s limit: 1536, and the same fallback applies to record validation (line 51). The README’s add_column ... :vector, limit: 3 # dimensions shows the same convention. The Arel.sql("... '#{embedding}'") string-interpolation pattern you will see in older tutorials is the thing it replaces.

Chunking by tokens #

Tokens are the units a model counts, roughly word fragments. Chunk by tokens so a chunk never exceeds what the embedding model accepts. tiktoken_ruby gives you the same tokenizer family OpenAI uses:

class ChunkService
  SIZE = 300
  OVERLAP = 50

  def initialize(encoding = Tiktoken.encoding_for_model("text-embedding-3-small"))
    @encoding = encoding
  end

  def call(text)
    tokens = @encoding.encode(text)
    (0...tokens.length).step(SIZE - OVERLAP).map { |i| @encoding.decode(tokens[i, SIZE]) }
  end
end

Run on a 1,000-word test text this produced 5 chunks of up to 300 tokens, each overlapping the next by 50 tokens. The 300/50 split is a starting value, not a recommendation backed by a benchmark. Smaller chunks retrieve more precisely and lose context; larger ones keep context and blur the match. Measure on your own questions.

Embedding and storing #

class Embedder
  def initialize(client = OpenAI::Client.new(access_token: ENV.fetch("OPENAI_API_KEY")))
    @client = client
  end

  # `input` may be a string or an array of strings (OpenAI embeddings API reference).
  def call(input)
    response = @client.embeddings(parameters: { model: "text-embedding-3-small", input: })
    vectors = response["data"].map { _1["embedding"] }
    input.is_a?(Array) ? vectors : vectors.first
  end
end

chunks = ChunkService.new.call(document.content)
chunks.zip(Embedder.new.call(chunks)).each_with_index do |(text, vector), position|
  document.document_chunks.create!(content: text, embedding: vector, position:)
end

Run this in a background job for anything beyond a few documents. Embedding calls are network calls and rate limits depend on your OpenAI account tier, so check your limits page rather than hard-coding a sleep.

Retrieval #

class Retriever
  def initialize(embedder = Embedder.new)
    @embedder = embedder
  end

  def call(question, limit: 3, max_distance: 0.8)
    DocumentChunk
      .nearest_neighbors(:embedding, @embedder.call(question), distance: "cosine")
      .first(limit)
      .select { _1.neighbor_distance <= max_distance }
  end
end

Cosine distance runs from 0 (same direction) to 2 (opposite). OpenAI recommends cosine and notes its embeddings are normalized to length 1, so cosine and Euclidean rankings come out identical (OpenAI embeddings guide, “Which distance function should I use?”).

In my run, a question about database indexes returned the two chunks about indexes at distances 0.522 and 0.528, and a question about pizza dough returned nothing because every chunk was past the cutoff. The 0.8 cutoff is a toy-embedder number. Real embedding models spread distances differently, so choose the cutoff by looking at distances for questions you know the answer to.

I filter in Ruby after fetching the top few rows on purpose. That choice comes from how approximate indexes work.

The index, and the trap in filtering #

Without an index, pgvector does an exact scan: perfect recall, slow on large tables. An index trades some recall for speed. pgvector offers HNSW and IVFFlat; its README lists both, and HNSW needs no training step on existing data.

class AddEmbeddingIndex < ActiveRecord::Migration[8.1]
  def up
    execute <<~SQL
      CREATE INDEX idx_chunks_embedding ON document_chunks
      USING hnsw (embedding vector_cosine_ops)
      WITH (m = 16, ef_construction = 64)
    SQL
  end

  def down = execute("DROP INDEX idx_chunks_embedding")
end

The m = 16, ef_construction = 64 values are pgvector’s defaults. I wrote them out only to show where the knobs are; raise ef_construction for better recall at the cost of build time (pgvector README, HNSW). neighbor’s add_index ... using: :hnsw, opclass: :vector_cosine_ops also works and creates the same index with default options. It rejected a with: option when I tried, hence the raw SQL.

The operator class must match the distance you query with: vector_cosine_ops for distance: "cosine". With a mismatch the planner cannot use the index.

To confirm the index is used, I ran EXPLAIN on the retrieval query. On a four-row table the planner prefers a sequential scan, so I set enable_seqscan = off for the check. The plan then read Index Scan using idx_chunks_embedding. On a real table you should see that without forcing it.

Query-time recall #

HNSW search keeps a candidate list of size hnsw.ef_search, 40 by default. Raise it for better recall, slower queries:

SET hnsw.ef_search = 100;

That also caps how many rows an index query can return. And here is the trap for RAG: pgvector applies WHERE conditions after the index scan. The README’s own example: if a condition matches 10% of rows, then with the default ef_search of 40 only about 4 rows will match on average. A tenant filter or a “published only” filter on a vector query can quietly return fewer than your LIMIT, or nothing. pgvector 0.8.0 added iterative scans (SET hnsw.iterative_scan = strict_order or relaxed_order) that keep scanning until enough rows pass the filter. For highly selective filters, the README suggests an ordinary index on the filter column or partitioning instead (pgvector README, Filtering).

This is why a WHERE similarity > 0.7 clause written into the SQL, as many tutorials do, is the wrong tool. Take the nearest rows first, then apply the cutoff, as above.

If you pick IVFFlat instead, the README’s starting point is lists = rows / 1000 up to 1M rows and sqrt(rows) above that, and the index should be built after the table has data.

Generation #

class RagService
  SYSTEM = "Answer using only the context below. If the context does not contain the answer, say so."

  def initialize(retriever: Retriever.new, client: OpenAI::Client.new(access_token: ENV.fetch("OPENAI_API_KEY")))
    @retriever = retriever
    @client = client
  end

  def call(question)
    chunks = @retriever.call(question)
    return "I couldn't find anything relevant." if chunks.empty?

    context = chunks.map(&:content).join("\n\n")
    response = @client.chat(parameters: {
      model: ENV.fetch("CHAT_MODEL"),
      temperature: 0.2,
      messages: [
        { role: "system", content: SYSTEM },
        { role: "user", content: "Context:\n#{context}\n\nQuestion: #{question}" }
      ]
    })
    response.dig("choices", 0, "message", "content")
  end
end

The model name comes from the environment because model names and prices change faster than blog posts do; look up current ones on the provider’s pricing page. A system prompt that says “use only the context” lowers the chance of invented answers. It does not remove it. Log which chunks went into each answer so you can check a bad answer against what the model was shown.

When pgvector is the wrong tool #

The pgvector README documents the limits that matter:

• Dimensions. The vector type indexes up to 2,000 dimensions; halfvec up to 4,000. The default 3072-dimension text-embedding-3-large vectors therefore need halfvec or a smaller dimensions setting to be indexed.
• Filter-heavy queries need the filtering approaches above, or exact search on a pre-filtered set.
• Not on PostgreSQL. If your app runs MySQL or SQLite, adding PostgreSQL only for vectors is a bigger step than calling a hosted vector service.

I have no benchmark of my own for where pgvector stops scaling, so I will not name a row count. Load your own corpus, run your own queries, and watch recall and latency.

Sources #

• pgvector README - HNSW and IVFFlat options and defaults, dimension limits, filtering after an approximate index, iterative scans
• neighbor gem README - has_neighbors, nearest_neighbors, neighbor_distance, index helpers
• OpenAI embeddings guide - default dimensions per model and the distance-function recommendation
• ruby-openai README - embeddings and chat call shapes

Related: working with LLMs in Rails and token cost management.