{"slug":"file-text-extractor","meta":{"title":"File Text Extractor — Read PDF, Word, Excel and CSV Uploads as Text","seo_title":"File Text Extractor for Uploads","slug":"file-text-extractor","category":"Files","summary":"One small Ruby service that turns an uploaded PDF, Word, Excel, CSV or text file into plain text, plus a read-only, paged API endpoint so a Leo chat mode can read the files people drop into the chat.","tags":["files","uploads","pdf","docx","xlsx","csv","roo","pdftotext","leo-chat","agent-mode"],"status":"stable","visibility":"public","source_project":"rsb-dev.leo.llamapress.ai","layers":["controller"],"related":[{"title":"Custom Leo Agent Mode","url":"/cookbook/custom-leo-agent-mode","summary":"Build a custom chat mode. Every mode should serve uploads with this guide's endpoint."}]},"body":"# File Text Extractor — Read PDF, Word, Excel and CSV Uploads as Text\n\n\u003e ⚠️ **Cookbook example — not live code.** (KEEP THIS CALLOUT.) Every code block below\n\u003e is an **example snippet**, **not part of the llamapress.ai codebase**, and **not\n\u003e running on this server**. This is a reference recipe for a **Leo instance (an AI coding\n\u003e agent) to implement in its own app** — read it to understand the pattern, then recreate\n\u003e it there.\n\nA file a user drops into the Leo chat is saved to `app/imports/\u003cname\u003e`, and the agent is\ntold only its name. This guide adds two things: `FileTextExtractor`, which turns that\nfile into plain text, and `Api::ChatUploadsController`, which serves the text to a chat\nagent in pages small enough for its tool. The engineer mode can then read an upload with\none call instead of writing a new script, and a read-only custom mode, which has no file\ntools at all, can read it for the first time.\n\n\u003e **When to use:** any app whose users upload documents to Leo or to a form and want\n\u003e answers from them — summaries, totals, \"what does this quote say\". Any custom Leo mode\n\u003e (required — see the custom mode guide). **When not to:** scans and photos. There is no\n\u003e OCR here, so they come back as \"no readable text\".\n\n---\n\n## The 80/20 in one breath\n\n1. Add `app/services/file_text_extractor.rb`. It returns `[text, status]`.\n2. Add `app/controllers/api/chat_uploads_controller.rb` and two routes in `namespace :api`.\n3. Tell the agent in its prompt: when a message carries `\u003cUPLOADED_FILES\u003e`, read the file\n   with `GET /api/chat_uploads/\u003cname\u003e` before answering.\n4. Test with a real upload and a real chat turn, not only with `curl`.\n\n---\n\n## Requirements — check before you start\n\nThe extractor uses three gems and one program. All four are in the standard Leo Rails\nimage (`kody06/llamapress-simple:0.7.15` and later), so no Gemfile change is needed there:\n\n```bash\n# Run on the box, from ~/Leonardo:\ndocker compose exec -T llamapress sh -c \"grep -E '^    (roo|rubyzip|nokogiri) ' Gemfile.lock; which pdftotext\"\n```\n\nYou need `roo`, `rubyzip`, `nokogiri` and `/usr/bin/pdftotext`. You cannot add a gem from\ninside a Leo box. If one is missing, the base image must be updated first.\n\n**Search before you add.** If the app already has an extractor (for example, one written\nfor email attachments), call it from the controller instead of adding a second copy:\n\n```bash\ngrep -rn \"pdftotext\\|Roo::Spreadsheet\\|word/document.xml\" rails/app/services\n```\n\n---\n\n## Layer 1 — The extractor\n\n```ruby\n# app/services/file_text_extractor.rb\n# frozen_string_literal: true\n\nrequire \"open3\"\nrequire \"tempfile\"\nrequire \"zip\"\n\n# Plain text out of the document types people upload: PDF, Word, Excel, CSV, text.\n# Returns [text, status] where status is extracted | empty | unsupported | failed.\n#\n# Template from .claude/skills/create_custom_leo_mode (mothership). First written as\n# Email::AttachmentTextExtractor on rsb-dev. Needs roo, rubyzip, nokogiri and the\n# pdftotext program, all in kody06/llamapress-simple:0.7.15. No OCR: a scan or a photo\n# comes back \"empty\". Old binary .doc is best-effort only.\nclass FileTextExtractor\n  MAX_CHARS = 300_000\n\n  def self.call(bytes, extension)\n    new(bytes, extension.to_s.downcase).call\n  end\n\n  def initialize(bytes, extension)\n    @bytes = bytes\n    @extension = extension\n  end\n\n  def call\n    text =\n      case @extension\n      when \"pdf\" then pdf\n      when \"docx\" then docx\n      when \"doc\" then doc\n      when \"xlsx\", \"xlsm\", \"xls\", \"csv\" then spreadsheet\n      when \"txt\" then @bytes.to_s.dup.force_encoding(\"UTF-8\").scrub\n      else return [nil, \"unsupported\"]\n      end\n\n    text = text.to_s.scrub.gsub(/[ \\t]+/, \" \").gsub(/\\n{3,}/, \"\\n\\n\").strip\n    return [nil, \"empty\"] if text.blank?\n\n    [text.truncate(MAX_CHARS), \"extracted\"]\n  rescue StandardError =\u003e e\n    Rails.logger.warn(\"[file-text] extraction (#{@extension}) failed: #{e.class}: #{e.message}\")\n    [nil, \"failed\"]\n  end\n\n  private\n\n  def with_tempfile\n    Tempfile.create([\"file_text\", \".#{@extension}\"]) do |file|\n      file.binmode\n      file.write(@bytes)\n      file.flush\n      yield file.path\n    end\n  end\n\n  def pdf\n    with_tempfile do |path|\n      out, status = Open3.capture2(\"pdftotext\", \"-layout\", path, \"-\")\n      status.success? ? out : nil\n    end\n  end\n\n  # A .docx is a zip; the words live in word/document.xml as \u003cw:t\u003e runs.\n  def docx\n    xml = nil\n    Zip::File.open_buffer(StringIO.new(@bytes)) do |zip|\n      entry = zip.find_entry(\"word/document.xml\")\n      xml = entry\u0026.get_input_stream\u0026.read\n    end\n    return nil if xml.blank?\n\n    doc = Nokogiri::XML(xml)\n    doc.remove_namespaces!\n    doc.xpath(\"//p\").map { |p| p.xpath(\".//t\").map(\u0026:text).join }.join(\"\\n\")\n  end\n\n  # Old binary Word. There is no converter in the image; pull out the\n  # readable runs so at least names, numbers and references are searchable.\n  def doc\n    utf16 = @bytes.to_s.scan(/(?:[\\x20-\\x7e]\\x00){6,}/n).map { |run| run.force_encoding(\"UTF-16LE\").encode(\"UTF-8\", invalid: :replace, undef: :replace) }\n    ascii = @bytes.to_s.scan(/[\\x20-\\x7e\\r\\n\\t]{8,}/n).map { |run| run.force_encoding(\"UTF-8\") }\n    (utf16 + ascii).join(\"\\n\")\n  end\n\n  def spreadsheet\n    with_tempfile do |path|\n      book = Roo::Spreadsheet.open(path, extension: @extension.to_sym)\n      book.sheets.map do |sheet|\n        book.default_sheet = sheet\n        rows = []\n        begin\n          (book.first_row..book.last_row).each do |r|\n            cells = (book.first_column..book.last_column).map { |c| book.cell(r, c) }.map { |v| v.to_s.strip }\n            rows \u003c\u003c cells.reject(\u0026:blank?).join(\" | \") if cells.any?(\u0026:present?)\n          end\n        rescue StandardError\n          # an empty sheet has no first_row\n        end\n        \"## #{sheet}\\n#{rows.join(\"\\n\")}\"\n      end.join(\"\\n\\n\")\n    end\n  end\nend\n```\n\nUsage:\n\n```ruby\ntext, status = FileTextExtractor.call(File.binread(path), File.extname(path).delete(\".\"))\n# status: \"extracted\" | \"empty\" (scan, picture) | \"unsupported\" (.zip, .png …) | \"failed\"\n```\n\n---\n\n## Layer 2 — The endpoint for a chat agent\n\nThe agent's HTTP tool (`rails_api_request`) cuts every response at **8,000 characters**.\nSo this controller returns the text in chunks of up to 6,000 characters, shrinks a chunk\nwhen JSON escaping makes it longer, and caps every reply at 7,500 characters. `q=` returns\nonly the matching lines, which is the quick way through a big spreadsheet.\n\n```ruby\n# app/controllers/api/chat_uploads_controller.rb\n# frozen_string_literal: true\n\nmodule Api\n  # Files the user drops into the Leo chat, read back as text for a custom Leo\n  # chat mode (langgraph/agents/\u003cmode\u003e). Template from the mothership skill\n  # create_custom_leo_mode; first shipped on rsb-dev 2026-10-28.\n  #\n  # The chat saves an upload to app/imports/\u003cname\u003e on disk and tells the agent\n  # only the name. A custom mode reads through this app's API and has no file\n  # access of its own, so without this controller it could see that a file\n  # arrived but never open it (rsb-dev, 2026-10-28: a CSV, an .xlsx and a\n  # .docx all came back \"I can't open its contents\").\n  #\n  # Read-only. `llama_bot_allow` names the only actions an agent token may\n  # reach. Who may read uploads is `require_upload_reader!` below: any signed-in\n  # user by default; narrow it to the mode's own account gate if it has one.\n  # A name is matched against the directory listing, never joined onto a path,\n  # so nothing outside app/imports can be reached.\n  #\n  # Text comes from FileTextExtractor (app/services/file_text_extractor.rb).\n  # Stopgap until LP-24 ships the same two paths in the llama_bot_rails gem.\n  class ChatUploadsController \u003c ApplicationController\n    # AgentAuth lets `authenticate_user!` accept the chat's agent token as well\n    # as a browser session. (rsb-dev adds its own LlamaBotAgentAuthFix concern\n    # here; a stock box does not need it.) GET only, so no CSRF change.\n    include LlamaBotRails::AgentAuth\n\n    llama_bot_allow :index, :show\n\n    before_action :authenticate_user!\n    before_action :require_upload_reader!\n\n    # The agent's HTTP tool cuts a response off at 8 000 characters. One chunk\n    # of text plus its envelope stays under that; the agent pages with offset.\n    CHUNK_CHARS = 6_000\n    MAX_BODY_CHARS = 7_500\n    MAX_MATCHES = 60\n    LIST_LIMIT = 50\n\n    # GET /api/chat_uploads?q=\u0026limit=\n    # Newest first. q matches the file name.\n    def index\n      files = upload_paths\n      q = params[:q].to_s.strip.downcase\n      files = files.select { |path| File.basename(path).downcase.include?(q) } if q.present?\n      limit = params[:limit].to_i.positive? ? [params[:limit].to_i, LIST_LIMIT].min : 20\n\n      render json: {\n        showing: [files.size, limit].min,\n        total: files.size,\n        files: files.first(limit).map { |path| file_summary(path) }\n      }\n    end\n\n    # GET /api/chat_uploads/\u003cname\u003e?offset=\u0026q=\n    # The file's text, CHUNK_CHARS at a time from offset. With q, only the\n    # lines that contain q (case-insensitive), with their line numbers.\n    def show\n      path = upload_paths.find { |p| File.basename(p) == params[:name].to_s }\n      unless path\n        return render json: {\n          error: \"not_found\",\n          message: \"No uploaded file is called \\\"#{params[:name]}\\\". GET /api/chat_uploads lists them.\"\n        }, status: :not_found\n      end\n\n      text, status = extract(path)\n      unless status == \"extracted\"\n        return render json: file_summary(path).merge(status: status, message: status_message(status))\n      end\n\n      q = params[:q].to_s.strip\n      if q.present?\n        lines = text.lines\n        hits = lines.each_with_index.select { |line, _| line.downcase.include?(q.downcase) }\n        shown = hits.first(MAX_MATCHES).map { |line, i| { line: i + 1, text: line.strip.truncate(300) } }\n        body = nil\n        loop do\n          body = file_summary(path).merge(\n            status: status, q: q, total_lines: lines.size, matches: hits.size,\n            showing: shown.size, lines: shown\n          ).to_json\n          break if body.size \u003c= MAX_BODY_CHARS || shown.empty?\n\n          shown.pop\n        end\n        render json: body\n        return\n      end\n\n      offset = [params[:offset].to_i, 0].max\n      size = CHUNK_CHARS\n      body = nil\n      # Escaping (newlines, accents, \u003c, \u0026) can grow a chunk past the tool's\n      # cut-off; shrink it until the whole reply fits.\n      loop do\n        chunk = text[offset, size].to_s\n        next_offset = offset + chunk.size\n        body = file_summary(path).merge(\n          status: status,\n          total_chars: text.size,\n          offset: offset,\n          next_offset: next_offset \u003c text.size ? next_offset : nil,\n          text: chunk\n        ).to_json\n        break if body.size \u003c= MAX_BODY_CHARS || size \u003c= 500\n\n        size = (size * 0.8).to_i\n      end\n      render json: body\n    end\n\n    private\n\n    def upload_dir\n      Rails.root.join(\"app\", \"imports\")\n    end\n\n    def upload_paths\n      return [] unless upload_dir.directory?\n\n      Dir.children(upload_dir)\n         .map { |name| upload_dir.join(name).to_s }\n         .select { |path| File.file?(path) \u0026\u0026 File.readable?(path) }\n         .sort_by { |path| -File.mtime(path).to_i }\n    end\n\n    def file_summary(path)\n      {\n        name: File.basename(path),\n        type: extension(path),\n        bytes: File.size(path),\n        uploaded_at: File.mtime(path).iso8601\n      }\n    end\n\n    def extension(path)\n      File.extname(path).delete(\".\").downcase\n    end\n\n    # Cached by name, size and mtime, so paging through a large workbook\n    # parses it once rather than once per chunk.\n    def extract(path)\n      key = [\"chat_upload_text\", File.basename(path), File.size(path), File.mtime(path).to_i]\n      Rails.cache.fetch(key, expires_in: 1.hour) do\n        FileTextExtractor.call(File.binread(path), extension(path))\n      end\n    end\n\n    # Default gate: any signed-in user (an agent token resolves to its user).\n    # Narrow it to the mode's own rule if it has one, e.g. RSB's RichAccess\n    # (admins plus a list of addresses).\n    def require_upload_reader!\n      return if current_user\n\n      render json: { error: \"not_signed_in\", message: \"Sign in to the app, then ask again.\" }, status: :unauthorized\n    end\n\n    def status_message(status)\n      case status\n      when \"empty\" then \"The file has no readable text. It may be a scan or a picture.\"\n      when \"unsupported\" then \"This kind of file cannot be read as text. PDF, Word, Excel, CSV and text files can.\"\n      else \"The file could not be read.\"\n      end\n    end\n  end\nend\n```\n\n```ruby\n# config/routes.rb  (inside the existing `namespace :api do ... end`)\nget \"chat_uploads\", to: \"chat_uploads#index\"\nget \"chat_uploads/:name\", to: \"chat_uploads#show\", constraints: { name: /[^\\/]+/ }, format: false\n```\n\n`constraints` lets a file name contain dots, and `format: false` stops Rails from reading\n`.xlsx` as a response format.\n\nReplies look like this:\n\n```json\n{\"name\":\"Parts_List.csv\",\"type\":\"csv\",\"bytes\":834,\"uploaded_at\":\"2026-10-28T13:08:21+00:00\",\n \"status\":\"extracted\",\"total_chars\":542,\"offset\":0,\"next_offset\":null,\n \"text\":\"## default\\nPart mark | No | Profile | Length | Material ...\"}\n```\n\n---\n\n## Layer 3 — Tell the agent\n\nAdd both blocks to the mode's system prompt. Without the first one, a model sees only a\nfile name and says it cannot open files.\n\n```\n## FILES THE USER UPLOADS\n\nWhen a message carries an \u003cUPLOADED_FILES\u003e block, the user has just dropped a\nfile into the chat. You CAN read it. Before you answer, read it with\nGET /api/chat_uploads/\u003cname\u003e, where \u003cname\u003e is the file name in that block (the\npart before \"(saved to ...\"). Then answer about what is in it. Never say you\ncannot open a file until you have tried.\n\nA long file comes back in pieces: keep reading with offset=\u003cnext_offset\u003e until\nnext_offset is empty, or until you have what you need. To find something in a\nbig file, pass q=\u003cword\u003e to get only the lines that contain it. If the reply\nsays the file has no readable text (a scan or a picture) or is a kind that\ncannot be read, tell the user that in one sentence and ask them to send a PDF,\nWord, Excel or CSV version.\n```\n\n```\n  GET /api/chat_uploads?q=\u0026limit=\n      Files the user has uploaded in this chat, newest first: name, type,\n      size, when. q matches the file name.\n\n  GET /api/chat_uploads/\u003cname\u003e?offset=\u0026q=\n      The text inside one uploaded file (PDF, Word, Excel, CSV, text).\n      Excel comes back one sheet at a time as \"## \u003csheet\u003e\" then one row per\n      line with cells split by \" | \". Up to 6 000 characters per reply; read\n      on with offset=\u003cnext_offset\u003e. q= returns only matching lines.\n```\n\n---\n\n## Verify\n\n1. **Signed out:** `curl -s -o /dev/null -w \"%{http_code}\" https://\u003cyour-rails-host\u003e/api/chat_uploads`\n   should give 302 or 401, never 200.\n2. **Real turn:** upload a CSV holding a word you made up, select the mode, and ask\n   \"Give me a short summary of it\". Leo must quote your word. A curl test alone does not\n   prove the agent will call the endpoint.\n3. **Big file:** a spreadsheet over 6,000 characters should come back with a `next_offset`,\n   and Leo should page through it or use `q=`.\n\n---\n\n## Gotchas (the hard-won stuff)\n\n- **A custom mode cannot read uploads by default.** It reads only through the app's API,\n  and nothing serves `app/imports/`. The first production custom mode with this gap answered\n  \"I can't open its contents\" to a CSV, an Excel quote and a Word file in a row.\n- **Match the name against the directory listing; never join it onto a path.**\n  `upload_paths.find { |p| File.basename(p) == name }` makes `..%2F..%2Fconfig%2Fdatabase.yml`\n  a plain 404.\n- **The 8,000-character cut-off is silent.** A reply cut mid-JSON reads to the model as a\n  broken file. Measure real replies (`curl -w \"%{size_download}\"`), not the raw text length.\n  Escaped newlines, accents and `\u003c`/`\u0026` make JSON longer than its text.\n- **Cache by name + size + mtime**, so paging a large workbook parses it once.\n- **Uploads follow the mode's access rule.** People drop contracts and pay sheets into chat.\n  If the mode has its own account gate, apply it in `require_upload_reader!` too.\n- **Files in `app/imports/` may be owned by root.** Reading works because they are\n  world-readable. Do not `chmod` them from the agent.\n- **`.doc` (old binary Word) is best-effort.** It pulls out readable runs of text, which is\n  enough for names and numbers but not for layout.\n\n---\n\n## Files this pattern touches\n\n```\napp/services/file_text_extractor.rb\napp/controllers/api/chat_uploads_controller.rb\nconfig/routes.rb\nlanggraph/agents/\u003cmode\u003e/nodes.py   (prompt sections)\n```\n\n## How to adapt this to your app\n\n1. Reading from a different folder (or from Active Storage): change `upload_dir` and\n   `upload_paths`. For Active Storage, look up by attachment id and call\n   `FileTextExtractor.call(blob.download, blob.filename.extension)`.\n2. Using it outside the chat (an import screen, a search index): call the extractor\n   directly. Store `[text, status]` on the record, so a failed read is visible, not silent.\n3. Adding a type: add one `when` branch in `call` that returns a String. Keep the\n   `[text, status]` contract.\n"}