File Text Extractor for Uploads | LlamaPress Cookbook
LlamaPress
Sign in Upload
Cookbook
Files Stable

File Text Extractor — Read PDF, Word, Excel and CSV Uploads as Text

One small Ruby service that turns an uploaded PDF, Word, Excel, CSV or text file into plain text, plus a read-only, paged API endpoint so a Leo chat mode can read the files people drop into the chat.

Proven on rsb-dev.leo.llamapress.ai controller
JSON Raw .md

File Text Extractor — Read PDF, Word, Excel and CSV Uploads as Text

⚠️ Cookbook example — not live code. (KEEP THIS CALLOUT.) Every code block below is an example snippet, not part of the llamapress.ai codebase, and not running on this server. This is a reference recipe for a Leo instance (an AI coding agent) to implement in its own app — read it to understand the pattern, then recreate it there.

A file a user drops into the Leo chat is saved to app/imports/<name>, and the agent is told only its name. This guide adds two things: FileTextExtractor, which turns that file into plain text, and Api::ChatUploadsController, which serves the text to a chat agent in pages small enough for its tool. The engineer mode can then read an upload with one call instead of writing a new script, and a read-only custom mode, which has no file tools at all, can read it for the first time.

When to use: any app whose users upload documents to Leo or to a form and want answers from them — summaries, totals, "what does this quote say". Any custom Leo mode (required — see the custom mode guide). When not to: scans and photos. There is no OCR here, so they come back as "no readable text".

The 80/20 in one breath

  1. Add app/services/file_text_extractor.rb. It returns [text, status].
  2. Add app/controllers/api/chat_uploads_controller.rb and two routes in namespace :api.
  3. Tell the agent in its prompt: when a message carries <UPLOADED_FILES>, read the file with GET /api/chat_uploads/<name> before answering.
  4. Test with a real upload and a real chat turn, not only with curl.

Requirements — check before you start

The extractor uses three gems and one program. All four are in the standard Leo Rails image (kody06/llamapress-simple:0.7.15 and later), so no Gemfile change is needed there:

# Run on the box, from ~/Leonardo:
docker compose exec -T llamapress sh -c "grep -E '^    (roo|rubyzip|nokogiri) ' Gemfile.lock; which pdftotext"

You need roo, rubyzip, nokogiri and /usr/bin/pdftotext. You cannot add a gem from inside a Leo box. If one is missing, the base image must be updated first.

Search before you add. If the app already has an extractor (for example, one written for email attachments), call it from the controller instead of adding a second copy:

grep -rn "pdftotext\|Roo::Spreadsheet\|word/document.xml" rails/app/services

Layer 1 — The extractor

# app/services/file_text_extractor.rb
# frozen_string_literal: true

require "open3"
require "tempfile"
require "zip"

# Plain text out of the document types people upload: PDF, Word, Excel, CSV, text.
# Returns [text, status] where status is extracted | empty | unsupported | failed.
#
# Template from .claude/skills/create_custom_leo_mode (mothership). First written as
# Email::AttachmentTextExtractor on rsb-dev. Needs roo, rubyzip, nokogiri and the
# pdftotext program, all in kody06/llamapress-simple:0.7.15. No OCR: a scan or a photo
# comes back "empty". Old binary .doc is best-effort only.
class FileTextExtractor
  MAX_CHARS = 300_000

  def self.call(bytes, extension)
    new(bytes, extension.to_s.downcase).call
  end

  def initialize(bytes, extension)
    @bytes = bytes
    @extension = extension
  end

  def call
    text =
      case @extension
      when "pdf" then pdf
      when "docx" then docx
      when "doc" then doc
      when "xlsx", "xlsm", "xls", "csv" then spreadsheet
      when "txt" then @bytes.to_s.dup.force_encoding("UTF-8").scrub
      else return [nil, "unsupported"]
      end

    text = text.to_s.scrub.gsub(/[ \t]+/, " ").gsub(/\n{3,}/, "\n\n").strip
    return [nil, "empty"] if text.blank?

    [text.truncate(MAX_CHARS), "extracted"]
  rescue StandardError => e
    Rails.logger.warn("[file-text] extraction (#{@extension}) failed: #{e.class}: #{e.message}")
    [nil, "failed"]
  end

  private

  def with_tempfile
    Tempfile.create(["file_text", ".#{@extension}"]) do |file|
      file.binmode
      file.write(@bytes)
      file.flush
      yield file.path
    end
  end

  def pdf
    with_tempfile do |path|
      out, status = Open3.capture2("pdftotext", "-layout", path, "-")
      status.success? ? out : nil
    end
  end

  # A .docx is a zip; the words live in word/document.xml as <w:t> runs.
  def docx
    xml = nil
    Zip::File.open_buffer(StringIO.new(@bytes)) do |zip|
      entry = zip.find_entry("word/document.xml")
      xml = entry&.get_input_stream&.read
    end
    return nil if xml.blank?

    doc = Nokogiri::XML(xml)
    doc.remove_namespaces!
    doc.xpath("//p").map { |p| p.xpath(".//t").map(&:text).join }.join("\n")
  end

  # Old binary Word. There is no converter in the image; pull out the
  # readable runs so at least names, numbers and references are searchable.
  def doc
    utf16 = @bytes.to_s.scan(/(?:[\x20-\x7e]\x00){6,}/n).map { |run| run.force_encoding("UTF-16LE").encode("UTF-8", invalid: :replace, undef: :replace) }
    ascii = @bytes.to_s.scan(/[\x20-\x7e\r\n\t]{8,}/n).map { |run| run.force_encoding("UTF-8") }
    (utf16 + ascii).join("\n")
  end

  def spreadsheet
    with_tempfile do |path|
      book = Roo::Spreadsheet.open(path, extension: @extension.to_sym)
      book.sheets.map do |sheet|
        book.default_sheet = sheet
        rows = []
        begin
          (book.first_row..book.last_row).each do |r|
            cells = (book.first_column..book.last_column).map { |c| book.cell(r, c) }.map { |v| v.to_s.strip }
            rows << cells.reject(&:blank?).join(" | ") if cells.any?(&:present?)
          end
        rescue StandardError
          # an empty sheet has no first_row
        end
        "## #{sheet}\n#{rows.join("\n")}"
      end.join("\n\n")
    end
  end
end

Usage:

text, status = FileTextExtractor.call(File.binread(path), File.extname(path).delete("."))
# status: "extracted" | "empty" (scan, picture) | "unsupported" (.zip, .png …) | "failed"

Layer 2 — The endpoint for a chat agent

The agent's HTTP tool (rails_api_request) cuts every response at 8,000 characters. So this controller returns the text in chunks of up to 6,000 characters, shrinks a chunk when JSON escaping makes it longer, and caps every reply at 7,500 characters. q= returns only the matching lines, which is the quick way through a big spreadsheet.

# app/controllers/api/chat_uploads_controller.rb
# frozen_string_literal: true

module Api
  # Files the user drops into the Leo chat, read back as text for a custom Leo
  # chat mode (langgraph/agents/<mode>). Template from the mothership skill
  # create_custom_leo_mode; first shipped on rsb-dev 2026-10-28.
  #
  # The chat saves an upload to app/imports/<name> on disk and tells the agent
  # only the name. A custom mode reads through this app's API and has no file
  # access of its own, so without this controller it could see that a file
  # arrived but never open it (rsb-dev, 2026-10-28: a CSV, an .xlsx and a
  # .docx all came back "I can't open its contents").
  #
  # Read-only. `llama_bot_allow` names the only actions an agent token may
  # reach. Who may read uploads is `require_upload_reader!` below: any signed-in
  # user by default; narrow it to the mode's own account gate if it has one.
  # A name is matched against the directory listing, never joined onto a path,
  # so nothing outside app/imports can be reached.
  #
  # Text comes from FileTextExtractor (app/services/file_text_extractor.rb).
  # Stopgap until LP-24 ships the same two paths in the llama_bot_rails gem.
  class ChatUploadsController < ApplicationController
    # AgentAuth lets `authenticate_user!` accept the chat's agent token as well
    # as a browser session. (rsb-dev adds its own LlamaBotAgentAuthFix concern
    # here; a stock box does not need it.) GET only, so no CSRF change.
    include LlamaBotRails::AgentAuth

    llama_bot_allow :index, :show

    before_action :authenticate_user!
    before_action :require_upload_reader!

    # The agent's HTTP tool cuts a response off at 8 000 characters. One chunk
    # of text plus its envelope stays under that; the agent pages with offset.
    CHUNK_CHARS = 6_000
    MAX_BODY_CHARS = 7_500
    MAX_MATCHES = 60
    LIST_LIMIT = 50

    # GET /api/chat_uploads?q=&limit=
    # Newest first. q matches the file name.
    def index
      files = upload_paths
      q = params[:q].to_s.strip.downcase
      files = files.select { |path| File.basename(path).downcase.include?(q) } if q.present?
      limit = params[:limit].to_i.positive? ? [params[:limit].to_i, LIST_LIMIT].min : 20

      render json: {
        showing: [files.size, limit].min,
        total: files.size,
        files: files.first(limit).map { |path| file_summary(path) }
      }
    end

    # GET /api/chat_uploads/<name>?offset=&q=
    # The file's text, CHUNK_CHARS at a time from offset. With q, only the
    # lines that contain q (case-insensitive), with their line numbers.
    def show
      path = upload_paths.find { |p| File.basename(p) == params[:name].to_s }
      unless path
        return render json: {
          error: "not_found",
          message: "No uploaded file is called \"#{params[:name]}\". GET /api/chat_uploads lists them."
        }, status: :not_found
      end

      text, status = extract(path)
      unless status == "extracted"
        return render json: file_summary(path).merge(status: status, message: status_message(status))
      end

      q = params[:q].to_s.strip
      if q.present?
        lines = text.lines
        hits = lines.each_with_index.select { |line, _| line.downcase.include?(q.downcase) }
        shown = hits.first(MAX_MATCHES).map { |line, i| { line: i + 1, text: line.strip.truncate(300) } }
        body = nil
        loop do
          body = file_summary(path).merge(
            status: status, q: q, total_lines: lines.size, matches: hits.size,
            showing: shown.size, lines: shown
          ).to_json
          break if body.size <= MAX_BODY_CHARS || shown.empty?

          shown.pop
        end
        render json: body
        return
      end

      offset = [params[:offset].to_i, 0].max
      size = CHUNK_CHARS
      body = nil
      # Escaping (newlines, accents, <, &) can grow a chunk past the tool's
      # cut-off; shrink it until the whole reply fits.
      loop do
        chunk = text[offset, size].to_s
        next_offset = offset + chunk.size
        body = file_summary(path).merge(
          status: status,
          total_chars: text.size,
          offset: offset,
          next_offset: next_offset < text.size ? next_offset : nil,
          text: chunk
        ).to_json
        break if body.size <= MAX_BODY_CHARS || size <= 500

        size = (size * 0.8).to_i
      end
      render json: body
    end

    private

    def upload_dir
      Rails.root.join("app", "imports")
    end

    def upload_paths
      return [] unless upload_dir.directory?

      Dir.children(upload_dir)
         .map { |name| upload_dir.join(name).to_s }
         .select { |path| File.file?(path) && File.readable?(path) }
         .sort_by { |path| -File.mtime(path).to_i }
    end

    def file_summary(path)
      {
        name: File.basename(path),
        type: extension(path),
        bytes: File.size(path),
        uploaded_at: File.mtime(path).iso8601
      }
    end

    def extension(path)
      File.extname(path).delete(".").downcase
    end

    # Cached by name, size and mtime, so paging through a large workbook
    # parses it once rather than once per chunk.
    def extract(path)
      key = ["chat_upload_text", File.basename(path), File.size(path), File.mtime(path).to_i]
      Rails.cache.fetch(key, expires_in: 1.hour) do
        FileTextExtractor.call(File.binread(path), extension(path))
      end
    end

    # Default gate: any signed-in user (an agent token resolves to its user).
    # Narrow it to the mode's own rule if it has one, e.g. RSB's RichAccess
    # (admins plus a list of addresses).
    def require_upload_reader!
      return if current_user

      render json: { error: "not_signed_in", message: "Sign in to the app, then ask again." }, status: :unauthorized
    end

    def status_message(status)
      case status
      when "empty" then "The file has no readable text. It may be a scan or a picture."
      when "unsupported" then "This kind of file cannot be read as text. PDF, Word, Excel, CSV and text files can."
      else "The file could not be read."
      end
    end
  end
end

# config/routes.rb  (inside the existing `namespace :api do ... end`)
get "chat_uploads", to: "chat_uploads#index"
get "chat_uploads/:name", to: "chat_uploads#show", constraints: { name: /[^\/]+/ }, format: false

constraints lets a file name contain dots, and format: false stops Rails from reading .xlsx as a response format.

Replies look like this:

{"name":"Parts_List.csv","type":"csv","bytes":834,"uploaded_at":"2026-10-28T13:08:21+00:00",
 "status":"extracted","total_chars":542,"offset":0,"next_offset":null,
 "text":"## default\nPart mark | No | Profile | Length | Material ..."}

Layer 3 — Tell the agent

Add both blocks to the mode's system prompt. Without the first one, a model sees only a file name and says it cannot open files.

## FILES THE USER UPLOADS

When a message carries an <UPLOADED_FILES> block, the user has just dropped a
file into the chat. You CAN read it. Before you answer, read it with
GET /api/chat_uploads/<name>, where <name> is the file name in that block (the
part before "(saved to ..."). Then answer about what is in it. Never say you
cannot open a file until you have tried.

A long file comes back in pieces: keep reading with offset=<next_offset> until
next_offset is empty, or until you have what you need. To find something in a
big file, pass q=<word> to get only the lines that contain it. If the reply
says the file has no readable text (a scan or a picture) or is a kind that
cannot be read, tell the user that in one sentence and ask them to send a PDF,
Word, Excel or CSV version.

  GET /api/chat_uploads?q=&limit=
      Files the user has uploaded in this chat, newest first: name, type,
      size, when. q matches the file name.

  GET /api/chat_uploads/<name>?offset=&q=
      The text inside one uploaded file (PDF, Word, Excel, CSV, text).
      Excel comes back one sheet at a time as "## <sheet>" then one row per
      line with cells split by " | ". Up to 6 000 characters per reply; read
      on with offset=<next_offset>. q= returns only matching lines.

Verify

  1. Signed out: curl -s -o /dev/null -w "%{http_code}" https://<your-rails-host>/api/chat_uploads should give 302 or 401, never 200.
  2. Real turn: upload a CSV holding a word you made up, select the mode, and ask "Give me a short summary of it". Leo must quote your word. A curl test alone does not prove the agent will call the endpoint.
  3. Big file: a spreadsheet over 6,000 characters should come back with a next_offset, and Leo should page through it or use q=.

Gotchas (the hard-won stuff)

  • A custom mode cannot read uploads by default. It reads only through the app's API, and nothing serves app/imports/. The first production custom mode with this gap answered "I can't open its contents" to a CSV, an Excel quote and a Word file in a row.
  • Match the name against the directory listing; never join it onto a path. upload_paths.find { |p| File.basename(p) == name } makes ..%2F..%2Fconfig%2Fdatabase.yml a plain 404.
  • The 8,000-character cut-off is silent. A reply cut mid-JSON reads to the model as a broken file. Measure real replies (curl -w "%{size_download}"), not the raw text length. Escaped newlines, accents and </& make JSON longer than its text.
  • Cache by name + size + mtime, so paging a large workbook parses it once.
  • Uploads follow the mode's access rule. People drop contracts and pay sheets into chat. If the mode has its own account gate, apply it in require_upload_reader! too.
  • Files in app/imports/ may be owned by root. Reading works because they are world-readable. Do not chmod them from the agent.
  • .doc (old binary Word) is best-effort. It pulls out readable runs of text, which is enough for names and numbers but not for layout.

Files this pattern touches

app/services/file_text_extractor.rb
app/controllers/api/chat_uploads_controller.rb
config/routes.rb
langgraph/agents/<mode>/nodes.py   (prompt sections)

How to adapt this to your app

  1. Reading from a different folder (or from Active Storage): change upload_dir and upload_paths. For Active Storage, look up by attachment id and call FileTextExtractor.call(blob.download, blob.filename.extension).
  2. Using it outside the chat (an import screen, a search index): call the extractor directly. Store [text, status] on the record, so a failed read is visible, not silent.
  3. Adding a type: add one when branch in call that returns a String. Keep the [text, status] contract.

Related

Cookbook Custom Leo Agent Mode Build a custom chat mode. Every mode should serve uploads with this guide's endpoint.