{"slug":"openwiki-repo-docs-and-papertrail-feed","meta":{"title":"OpenWiki: Repo Docs, Admin Browser, Nightly Cron, and a PaperTrail Feed","slug":"openwiki-repo-docs-and-papertrail-feed","category":"Integrations","summary":"Run OpenWiki so an AI agent keeps a living wiki of your repo, browse it inside your admin at /admin/open-wiki, schedule it with a hardened nightly cron, and extend it past code by feeding it a PaperTrail data-change digest.","tags":["openwiki","documentation","cron","paper_trail","audit","markdown","admin"],"status":"stable","visibility":"public","source_project":"llamapress.ai (mothership)","layers":["view","controller","model","sql"],"related":[{"title":"Rate limiting and IP controls","url":"/cookbook/rate-limiting-and-ip-controls","summary":"Another \"operations feature that lives half in cron, half in Rails\" recipe."},{"title":"OpenWiki on npm","url":"https://www.npmjs.com/package/openwiki","summary":"The CLI this guide drives."},{"title":"paper_trail gem","url":"https://github.com/paper-trail-gem/paper_trail","summary":"The model-versioning gem the data-change feed reads from."}]},"body":"# OpenWiki: Repo Docs, Admin Browser, Nightly Cron, and a PaperTrail Feed\n\n\u003e ⚠️ **Cookbook example — not live code.** (KEEP THIS CALLOUT.) Every code block below\n\u003e is an **example snippet**, **not part of the llamapress.ai codebase**, and **not\n\u003e running on this server**. This is a reference recipe for a **Leo instance (an AI coding\n\u003e agent) to implement in its own app** — read it to understand the pattern, then recreate\n\u003e it there.\n\n**OpenWiki** is a command-line agent that reads a repository and writes a wiki about it\ninto an `openwiki/` folder. You run it once with `--init` to build the wiki, then on a\nschedule with `--update` to keep it current. It writes plain Markdown files with YAML\nfrontmatter, so the wiki is just files in your repo — you can grep it, diff it, and\nserve it.\n\nThis guide has four parts:\n\n1. Install OpenWiki and generate the wiki.\n2. Serve it inside your own app at `/admin/open-wiki` (sidebar, Markdown rendering,\n   working internal links).\n3. Schedule it with a nightly cron that fails loudly instead of silently.\n4. **Feed it more than code.** OpenWiki only reads files. A wiki built from code alone\n   documents what the software *can* do, never what it *did*. Part 4 exports a\n   **PaperTrail** digest — which models real users changed, how often, and which columns\n   — into a Markdown file in the repo, so the next OpenWiki run reads it and writes it\n   into the wiki.\n\n\u003e **When to use:** a codebase big enough that new engineers (or AI agents) can't hold it\n\u003e in their head, where you want documentation that refreshes itself.\n\u003e **When not to:** a small app where a hand-written README stays accurate. OpenWiki costs\n\u003e LLM tokens on every run.\n\n---\n\n## The 80/20 in one breath\n\n1. `npm install --global openwiki`, then authenticate a model provider.\n2. Run `openwiki code --init` in the repo root. It writes `openwiki/*.md`.\n3. Write `openwiki/INSTRUCTIONS.md` — the brief that steers every future run. **This is\n   the file you edit to change the wiki**, not the pages themselves.\n4. Bind-mount `./openwiki` read-only into the Rails container, add two routes, and add a\n   read-only controller that parses frontmatter server-side and renders the body with\n   marked.js.\n5. Add a cron entry running a wrapper script (`flock`, `timeout`, a log, and a loud\n   failure marker) that calls `openwiki code --update --print` nightly.\n6. Optional but high value: a second, earlier cron writes a PaperTrail digest to\n   `docs/audit/data-change-digest.md`, and `INSTRUCTIONS.md` tells OpenWiki to\n   synthesize it.\n\n---\n\n## Layer 1 — Install and generate\n\nOpenWiki is a Node CLI. Install it globally on the machine that will run the cron.\n\n```bash\n# Node 22+. If you use nvm, note WHICH node — the cron section below depends on it.\nnpm install --global openwiki\n\ncd ~/YourRepo\nopenwiki auth openai-chatgpt      # or set OPENWIKI_PROVIDER + an API key\nopenwiki code --init --print      # plans the wiki; see the warning below\n```\n\n\u003e ⚠️ **`--init` does not build the wiki. It plans one.** On any repository big\n\u003e enough to be worth documenting, that single agent turn is spent writing\n\u003e `openwiki/_skeleton.md`, an `index.md`, and a message offering to continue —\n\u003e then it **exits 0**. Nothing tells you the \"wiki\" is a plan: `--print` looks\n\u003e like success and the folder has files in it. Building is a LOOP of steered\n\u003e `--update` passes, each one turn, until the page count stops climbing:\n\u003e\n\u003e ```bash\n\u003e for i in $(seq 1 8); do\n\u003e   openwiki code --update --print \"Continue building this wiki from\n\u003e     openwiki/_skeleton.md. Write the next few pages COMPLETELY, in the\n\u003e     skeleton's priority order. Do not reply with a plan and do not ask whether\n\u003e     to proceed — write the pages now. Delete openwiki/_skeleton.md when every\n\u003e     planned page exists with real content.\"\n\u003e   # stop when the page count stops rising, or _skeleton.md disappears\n\u003e done\n\u003e ```\n\u003e\n\u003e Budget for this: ~25 pages over three passes on a mid-size Rails app.\n\nTwo modes exist and they are easy to confuse:\n\n| Command | What it does | Writes to |\n|---|---|---|\n| `openwiki code` | Documents **the current repository** | `\u003crepo\u003e/openwiki/` |\n| `openwiki personal` | A local personal brain over configured connectors | `~/.openwiki/wiki` |\n\nAlways pass `code` explicitly in scripts. Useful flags:\n\n```bash\nopenwiki code --update --print            # one non-interactive run, prints the summary\nopenwiki code --update \"focus on the jobs directory\"   # steer a single run with a message\nopenwiki code --modelId \u003cmodel-id\u003e --update\n```\n\nAfter a run, OpenWiki records what it did in `openwiki/.last-update.json`:\n\n```json\n{\n  \"updatedAt\": \"2026-08-07T08:03:11.568Z\",\n  \"command\": \"update\",\n  \"gitHead\": \"636f08d71b3992cee07df3f5b2e7279e44289092\",\n  \"model\": \"gpt-5.6-terra\"\n}\n```\n\n**That `gitHead` is the mechanism of the whole system.** An `--update` run diffs the repo\nfrom that commit to `HEAD` and only rewrites the pages the changed files affect. It is\nincremental, not a full rebuild. Two consequences follow, and both bite people:\n\n- A run right after a big merge is slow and expensive. A run with no new commits is\n  nearly free.\n- **Work that never lands in a commit is invisible to the update.** Part 4 is built\n  around this fact.\n\n---\n\n## Layer 2 — Steer the wiki with INSTRUCTIONS.md\n\n`openwiki/INSTRUCTIONS.md` is the standing brief. Every run reads it. Editing a generated\npage is pointless — the next run overwrites it. Editing the brief changes the wiki\npermanently.\n\nA brief that works has five sections:\n\n```markdown\nThis is the internal wiki for \u003csystem\u003e. Its readers are \u003cwho\u003e and the AI agents that\noperate this system.\n\n## Scope — synthesize ALL of these sources\n- `app/`, `lib/`, `bin/` — the code.\n- `docs/dev/`, `docs/incidents/` — designs and postmortems.\n- `docs/audit/` — the PaperTrail data-change digest (see Layer 5).\n- `.claude/skills/*/SKILL.md` — operational playbooks with exact commands.\n\n## HARD RULE — secrets\nNever copy credentials, API keys, tokens, `.env` values, or SSH key material into wiki\npages. Naming the env var is fine; the value never is.\n\n## Required coverage (build pages for these; keep them current)\n1. \u003cTopic\u003e — \u003cthe source files it must synthesize\u003e\n2. …\n\n## Audience convention (maintain on every run)\nEvery page's `tags` list carries exactly ONE audience tag: `audience-engineering`,\n`audience-business`, or `audience-all`. Preserve the tag on every existing page.\n\n## Frontmatter and recall\nAgents find pages by grepping `description:` fields. Write descriptions\ngrep-optimized: include exact command names, class names, error strings, and domain\nterms a searcher would type.\n```\n\nThe \"Required coverage\" list is the highest-leverage part. Without it the agent writes\nwhatever the diff suggested, and important-but-stable subsystems slowly go undocumented\nbecause nobody edits them.\n\n---\n\n## Layer 3 — The admin browser UI\n\nThe wiki is Markdown on disk. Serving it needs three pieces: a **mount**, a **read-only\ncontroller**, and a **client-side renderer**.\n\n### 3a. Mount the folder read-only\n\nThe Rails container cannot see the repository root. Mount just the wiki folder:\n\n```yaml\n# docker-compose.yml\nservices:\n  llamapress:\n    volumes:\n      - ./openwiki:/rails/openwiki:ro   # generated docs, browsed at /admin/open-wiki\n```\n\nThis is a **directory** mount, so regenerated pages appear with no restart. (Single-file\nmounts do not behave this way — see Gotchas.)\n\n### 3b. Routes\n\n```ruby\n# config/routes.rb\nnamespace :admin do\n  # `format: false` keeps a `.md` suffix inside the wildcard instead of Rails\n  # parsing it as a response format.\n  get '/open-wiki',       to: 'open_wiki#index', as: :open_wiki\n  get '/open-wiki/*path', to: 'open_wiki#show',  as: :open_wiki_page, format: false\nend\n```\n\n### 3c. The controller — read-only, path-guarded\n\n```ruby\n# app/controllers/admin/open_wiki_controller.rb\n#\n# Read-only browser for the generated openwiki/ folder. This controller NEVER\n# writes: the pages are generated, and hand-edits get clobbered on regeneration.\nclass Admin::OpenWikiController \u003c ApplicationController\n  before_action :authenticate_user!\n  before_action :ensure_admin\n\n  WIKI_DIR = Rails.root.join(\"openwiki\").freeze\n  # Each path segment maps straight to a filename: leading alphanumeric, then\n  # word chars / dashes / dots. Blocks dotfiles, `..`, and absolute paths.\n  SEGMENT_RE = /\\A[A-Za-z0-9][A-Za-z0-9\\-_.]*\\z/\n\n  def index = render_page(\"index.md\")\n\n  def show\n    rel = resolve_page(params[:path].to_s)\n    return not_found unless rel\n    render_page(rel)\n  end\n\n  private\n\n  def render_page(rel)\n    unless Dir.exist?(WIKI_DIR)\n      redirect_to admin_path, alert: \"The openwiki folder is not mounted into the container.\" and return\n    end\n\n    path = WIKI_DIR.join(rel)\n    return not_found unless File.file?(path)\n\n    raw = File.read(path)\n    render plain: raw, content_type: \"text/markdown\" and return if params[:raw].present?\n\n    @current     = rel\n    @meta        = frontmatter(raw)\n    @body        = strip_frontmatter(raw)\n    @updated     = File.mtime(path)\n    @tree        = page_tree\n    @last_update = wiki_last_update\n    render \"admin/open_wiki/show\"\n  end\n\n  # \"\" → index.md · \"architecture\" → architecture/index.md · \"quickstart\" → quickstart.md\n  def resolve_page(raw_path)\n    segments = raw_path.split(\"/\").reject(\u0026:blank?)\n    return \"index.md\" if segments.empty?\n    return nil unless segments.all? { |s| s.match?(SEGMENT_RE) }\n\n    rel = segments.join(\"/\")\n    candidates = []\n    candidates \u003c\u003c rel if rel.end_with?(\".md\")\n    candidates \u003c\u003c \"#{rel}/index.md\"\n    candidates \u003c\u003c \"#{rel}.md\"\n\n    found = candidates.find { |c| File.file?(WIKI_DIR.join(c)) }\n    return nil unless found\n\n    # Belt-and-braces: the resolved absolute path must stay inside WIKI_DIR.\n    abs = File.expand_path(WIKI_DIR.join(found))\n    abs.start_with?(\"#{File.expand_path(WIKI_DIR)}/\") ? found : nil\n  end\n\n  # All pages grouped by directory (\"\" = root), for the sidebar. index.md sorts\n  # first within a group, then by title.\n  def page_tree\n    Dir.glob(WIKI_DIR.join(\"**/*.md\")).map { |file|\n      rel = Pathname.new(file).relative_path_from(WIKI_DIR).to_s\n      dir = File.dirname(rel)\n      { rel: rel,\n        dir: dir == \".\" ? \"\" : dir,\n        index: File.basename(rel) == \"index.md\",\n        title: frontmatter(File.read(file))[\"title\"].presence ||\n               File.basename(rel, \".md\").tr(\"-\", \" \").capitalize }\n    }.group_by { |p| p[:dir] }\n     .sort_by { |dir, _| dir }.to_h\n     .transform_values { |ps| ps.sort_by { |p| [p[:index] ? 0 : 1, p[:title].downcase] } }\n  end\n\n  # The generator's own metadata: when the wiki was last rebuilt, and by which model.\n  def wiki_last_update\n    file = WIKI_DIR.join(\".last-update.json\")\n    File.file?(file) ? JSON.parse(File.read(file)) : nil\n  rescue JSON::ParserError\n    nil\n  end\n\n  def not_found = redirect_to(admin_open_wiki_path, alert: \"That wiki page doesn't exist.\")\n\n  # A malformed page must not 500 the whole wiki.\n  def frontmatter(raw)\n    m = raw.match(/\\A---\\s*\\n(.*?\\n)---\\s*\\n/m)\n    return {} unless m\n    YAML.safe_load(m[1], permitted_classes: [], aliases: false) || {}\n  rescue Psych::Exception\n    {}\n  end\n\n  def strip_frontmatter(raw) = raw.sub(/\\A---\\s*\\n.*?\\n---\\s*\\n/m, \"\")\nend\n```\n\n### 3d. The view — render Markdown in the browser, and fix the links\n\nTwo things make this view non-trivial. First, there is no server-side Markdown gem in\nmost pinned images, so rendering happens client-side with marked.js. Second, OpenWiki\nwrites **standard relative Markdown links** (`quickstart.md`, `../workflows/foo.md`,\n`architecture/`). Those 404 under an admin route unless you rewrite them.\n\n```erb\n\u003c%# app/views/admin/open_wiki/show.html.erb (body of the page) %\u003e\n\n\u003c%# Raw page body — embedded inertly (Rails-escaped), rendered client-side. %\u003e\n\u003cscript type=\"text/plain\" id=\"openwiki-md-src\"\u003e\u003c%= @body %\u003e\u003c/script\u003e\n\n\u003cscript src=\"https://cdn.jsdelivr.net/npm/marked@12.0.0/marked.min.js\"\u003e\u003c/script\u003e\n\u003cscript\u003e\n(function () {\n  // The blob above was HTML-escaped by \u003c%%= %\u003e; decode back to true markdown.\n  function decodeEntities(s) {\n    var ta = document.createElement('textarea'); ta.innerHTML = s; return ta.value;\n  }\n  var raw     = decodeEntities(document.getElementById('openwiki-md-src').textContent || '');\n  var bodyEl  = document.getElementById('openwiki-body');\n\n  // Relative links resolve against the CURRENT page's directory.\n  var current = \u003c%= @current.to_json.html_safe %\u003e;      // e.g. \"architecture/overview.md\"\n  var baseDir = current.split('/').slice(0, -1).join('/');\n\n  function rewriteHref(href) {\n    // External, absolute, protocol-relative, and pure-anchor links pass through.\n    if (!href || /^([a-z][a-z0-9+.-]*:|\\/\\/|\\/|#)/i.test(href)) return null;\n    var m = href.match(/^([^#?]*)([#?].*)?$/);\n    var p = m[1], suffix = m[2] || '';\n    if (p === '') return null;\n\n    var parts = (baseDir ? baseDir + '/' + p : p).split('/'), out = [];\n    for (var i = 0; i \u003c parts.length; i++) {\n      var s = parts[i];\n      if (s === '' || s === '.') continue;\n      if (s === '..') { out.pop(); continue; }\n      out.push(s);\n    }\n    var joined = out.join('/');\n    if (/\\.md$/i.test(joined)) joined = joined.slice(0, -3);\n    return '/admin/open-wiki/' + joined + suffix;\n  }\n\n  marked.setOptions({ gfm: true, breaks: false });\n  bodyEl.innerHTML = marked.parse(raw);\n  bodyEl.querySelectorAll('a[href]').forEach(function (a) {\n    var to = rewriteHref(a.getAttribute('href'));\n    if (to) a.setAttribute('href', to);\n  });\n})();\n\u003c/script\u003e\n```\n\nRender the sidebar from `@tree`, and show `@last_update['updatedAt']` and\n`@last_update['model']` in the page header. The freshness line matters more than it\nlooks: a wiki that quietly stopped updating three weeks ago reads exactly like a wiki\nthat is current.\n\nAdd a **Raw .md** link (`?raw=1`) on every page. Agents fetch that endpoint directly.\n\n---\n\n## Layer 4 — The nightly cron\n\n`openwiki code --update --print` is the whole job. Everything around it exists because\nthis job runs unattended and can fail in ways that look like success.\n\n```bash\n#!/usr/bin/env bash\n# openwiki-nightly.sh — nightly OpenWiki update for the internal wiki.\n#\n# SCHEDULE: crontab, 0 8 * * * UTC.\n# LOG: /home/ubuntu/openwiki-nightly.log (truncated at 1MB).\n# STEERING: edit openwiki/INSTRUCTIONS.md — NOT this script.\n#\n# HARD-WON NOTES:\n# - openwiki is installed under nvm Node 22, NOT system Node 18 — the PATH export\n#   below is load-bearing; cron does not source ~/.bashrc.\n# - `openwiki --help` prints \"provider: OpenAI\" before loading its env file; that\n#   banner is not evidence of misconfiguration.\n# - AUTH EXPIRES SILENTLY. An OAuth refresh token can be revoked (e.g. the account\n#   is re-signed-in elsewhere). The run then dies instantly. Cron still fires — the\n#   fingerprint is a MISSING \"=== done\" line, not a missing start line. That is why\n#   the trap below writes a greppable \"*** FAILED\" marker.\n\nset -euo pipefail\n\nexport PATH=\"$HOME/.nvm/versions/node/v22.23.1/bin:$PATH\"\nLOG=\"$HOME/openwiki-nightly.log\"\nLOCK=\"/tmp/openwiki-nightly.lock\"\nREPO=\"$HOME/YourRepo\"\n\n# Truncate log if over 1MB\nif [ -f \"$LOG\" ] \u0026\u0026 [ \"$(stat -c%s \"$LOG\")\" -gt 1048576 ]; then\n  tail -c 262144 \"$LOG\" \u003e \"$LOG.tmp\" \u0026\u0026 mv \"$LOG.tmp\" \"$LOG\"\nfi\n\n{\n  echo \"=== openwiki-nightly $(date -u '+%Y-%m-%d %H:%M:%S UTC') ===\"\n  # flock: skip this run entirely if the previous one is still going.\n  flock -n 9 || { echo \"SKIP: previous run still holds lock\"; exit 0; }\n  cd \"$REPO\"\n  trap 'rc=$?; [ \"$rc\" -ne 0 ] \u0026\u0026 echo \"*** FAILED $(date -u \"+%F %T UTC\"), exit $rc — check auth ***\"' EXIT\n  timeout 3600 openwiki code --update --print\n  echo \"=== done $(date -u '+%Y-%m-%d %H:%M:%S UTC'), exit $? ===\"\n} 9\u003e\"$LOCK\" \u003e\u003e \"$LOG\" 2\u003e\u00261\n```\n\nInstall it:\n\n```bash\nchmod +x bin/local/openwiki-nightly.sh\ncrontab -e\n# 0 8 * * * /home/ubuntu/YourRepo/bin/local/openwiki-nightly.sh\n```\n\nCheck health in one command:\n\n```bash\ngrep -c '=== done' ~/openwiki-nightly.log      # successful runs\ngrep -n '\\*\\*\\* FAILED' ~/openwiki-nightly.log # dead runs, with the exit code\n```\n\n**Why nightly and not a git hook.** A per-commit hook looks tempting because the update\nis diff-driven. Do not do it. Each run takes minutes and spends model quota, and the run\nproduces wiki changes that themselves want a commit — so the hook recurses. One batch\nrun per night collapses a day of commits into a single diff.\n\n### The GitHub Actions alternative\n\nIf you would rather run it in CI than on a box, the workflow is small:\n\n```yaml\n# .github/workflows/openwiki-update.yml\nname: OpenWiki Update\non:\n  workflow_dispatch:\n  schedule:\n    - cron: \"0 8 * * *\"\n\npermissions:\n  contents: write\n  pull-requests: write\n\njobs:\n  update:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n      - uses: actions/setup-node@v4\n        with: { node-version: \"22\" }\n      - run: npm install --global openwiki\n      - run: openwiki code --update --print\n        env:\n          OPENWIKI_PROVIDER: openrouter\n          OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}\n          OPENWIKI_MODEL_ID: \u003cmodel-id\u003e\n      - uses: peter-evans/create-pull-request@v7\n        with:\n          add-paths: |\n            openwiki\n            AGENTS.md\n            CLAUDE.md\n            .github/workflows/openwiki-update.yml\n          branch: openwiki/update\n          commit-message: \"docs: update OpenWiki\"\n          title: \"docs: update OpenWiki\"\n```\n\n**Pick one path and mean it.** The Actions path opens a pull request. If your team does\nnot merge those pull requests, the wiki silently stops advancing while the workflow keeps\nreporting green — so the local cron becomes the real path and the workflow is noise.\nOpenWiki also regenerates this workflow file on every run, so never hand-edit it.\n\n---\n\n## Layer 5 — Feed it more than code, with PaperTrail\n\nEverything above documents **the code**. That leaves a real gap. The wiki can tell you\nthat a `ProductionProjectItem` has a `welder` column and which controller writes it. It\ncannot tell you that the column is edited 200 times a week by two people, that imports\ndestroy and recreate every row, or that a table nobody mentions in standup carries the\nmost churn in the system.\n\n**PaperTrail** already records that. It writes one row to a `versions` table for every\ncreate, update, and destroy on a model you opt in. The plan is a small pipeline:\n\n```\nversions table  →  nightly rollup query  →  docs/audit/data-change-digest.md\n                →  git commit  →  OpenWiki --update reads it  →  a wiki page\n```\n\nThe key insight is the arrow in the middle: **OpenWiki reads files in a repository, so\nanything you want it to know must first become a file in the repository.** That makes it\nextensible far past PaperTrail — the same pattern works for error telemetry, job runtimes,\nor support-ticket themes.\n\n### 5a. Turn PaperTrail on\n\n```ruby\n# Gemfile\ngem \"paper_trail\", \"~\u003e 15.2\"\n```\n\n```ruby\n# db/migrate/XXXXXXXX_create_versions.rb\nclass CreateVersions \u003c ActiveRecord::Migration[7.2]\n  def change\n    create_table :versions do |t|\n      t.string   :item_type, null: false\n      t.bigint   :item_id,   null: false\n      t.string   :event,     null: false          # create | update | destroy\n      t.string   :whodunnit\n      t.jsonb    :object                          # the record BEFORE the change\n      t.jsonb    :object_changes                  # {\"col\": [from, to]}\n      t.datetime :created_at\n    end\n\n    add_index :versions, %i[item_type item_id]\n    add_index :versions, :created_at\n  end\nend\n```\n\n```ruby\n# config/initializers/paper_trail.rb\nif defined?(PaperTrail)\n  PaperTrail.config.enabled = true\n  PaperTrail.config.track_associations = false\nend\n```\n\n```ruby\n# app/controllers/application_controller.rb\n# Without this, EVERY version has a null whodunnit and the digest cannot name actors.\nbefore_action :set_paper_trail_whodunnit\n```\n\n```ruby\n# app/models/production_project_item.rb\nclass ProductionProjectItem \u003c ApplicationRecord\n  has_paper_trail          # bare = all columns tracked\n  # has_paper_trail only: [:welder, :weld_date]   # or narrow it\nend\n```\n\n**Opt in deliberately.** Bare `has_paper_trail` on a hot table can double its write\nvolume and grow `versions` past the table it audits. Start with the 5–10 models whose\nhistory someone would actually ask about.\n\n### 5b. The digest script — roll up, never dump\n\nPut the script in `lib/` (or `rails/lib`), not the app root — on Leo boxes the Rails root\nis not bind-mounted, only its subdirectories are, so a script at the root will not exist\ninside the container.\n\n```ruby\n# lib/papertrail_digest.rb — run with: bin/rails runner /rails/lib/papertrail_digest.rb\n#\n# Prints a Markdown digest of the last 7 days of PaperTrail activity to stdout,\n# between two markers so the caller can slice it out of Rails' log noise.\n#\n# HARD RULE: emit COUNTS and COLUMN NAMES only. Never the values in `object` or\n# `object_changes` — those are your users' real data, and this file gets committed\n# and read by a model provider.\n\nActiveRecord::Base.logger = nil\nDAYS = 7\nconn = ActiveRecord::Base.connection\n\ndef rows(conn, sql) = conn.select_all(sql).to_a\n\nputs \"--- BEGIN DIGEST ---\"\nputs \u003c\u003c~HEAD\n  ---\n  type: Data Change Digest\n  title: \"Data change digest (last #{DAYS} days)\"\n  description: \"PaperTrail rollup: which models real users create, update, and destroy, which columns churn, and who the actors are. Generated nightly; counts and column names only, never values.\"\n  tags: [audit, paper_trail, audience-engineering]\n  ---\n\n  # Data change digest — last #{DAYS} days\n\n  Generated #{Time.current.utc.iso8601} from the `versions` table. Counts only.\nHEAD\n\n# 1) Volume by model and event.\nputs \"\\n## Change volume by model\\n\\n| Model | Creates | Updates | Destroys |\\n|---|---:|---:|---:|\"\nrows(conn, \u003c\u003c~SQL).each { |r| puts \"| #{r['item_type']} | #{r['creates']} | #{r['updates']} | #{r['destroys']} |\" }\n  SELECT item_type,\n         count(*) FILTER (WHERE event = 'create')  AS creates,\n         count(*) FILTER (WHERE event = 'update')  AS updates,\n         count(*) FILTER (WHERE event = 'destroy') AS destroys\n  FROM versions\n  WHERE created_at \u003e= now() - interval '#{DAYS} days'\n  GROUP BY 1 ORDER BY count(*) DESC LIMIT 25\nSQL\n\n# 2) Which COLUMNS actually churn. jsonb_each over object_changes gives the keys;\n#    we take the key and throw the value away.\nputs \"\\n## Hottest columns (updates only)\\n\\n| Model | Column | Edits |\\n|---|---|---:|\"\nrows(conn, \u003c\u003c~SQL).each { |r| puts \"| #{r['item_type']} | `#{r['column_name']}` | #{r['edits']} |\" }\n  SELECT v.item_type, c.key AS column_name, count(*) AS edits\n  FROM versions v, jsonb_each(v.object_changes) c\n  WHERE v.event = 'update'\n    AND v.created_at \u003e= now() - interval '#{DAYS} days'\n    AND c.key NOT IN ('updated_at', 'created_at')\n  GROUP BY 1, 2 ORDER BY edits DESC LIMIT 30\nSQL\n\n# 3) Who. whodunnit is a users.id AS A STRING — PaperTrail stores no name. Join it.\nputs \"\\n## Actors\\n\\n| User | Changes |\\n|---|---:|\"\nrows(conn, \u003c\u003c~SQL).each { |r| puts \"| #{r['actor']} | #{r['n']} |\" }\n  SELECT coalesce(u.email, 'unattributed (' || coalesce(v.whodunnit, 'null') || ')') AS actor,\n         count(*) AS n\n  FROM versions v\n  LEFT JOIN users u ON u.id = nullif(v.whodunnit, '')::bigint\n  WHERE v.created_at \u003e= now() - interval '#{DAYS} days'\n  GROUP BY 1 ORDER BY n DESC LIMIT 15\nSQL\n\n# 4) Coverage — the honest denominator. A model with no has_paper_trail is\n#    UNMEASURED, not quiet, and the wiki must say so.\nRails.application.eager_load!\ntracked   = ApplicationRecord.descendants.select { |m| m.respond_to?(:paper_trail_options) }.map(\u0026:name).sort\nuntracked = ApplicationRecord.descendants.map(\u0026:name).sort - tracked\nputs \"\\n## Coverage\\n\"\nputs \"Versioned (#{tracked.size}): #{tracked.join(', ')}\"\nputs \"\\nNOT versioned (#{untracked.size}) — absence of history here means UNMEASURED, not unchanged:\"\nputs untracked.join(', ')\nputs \"--- END DIGEST ---\"\n```\n\n### 5c. The wrapper cron\n\n```bash\n#!/usr/bin/env bash\n# papertrail-digest-nightly.sh — write the data-change digest, then commit it so the\n# OpenWiki run (which is git-diff driven) actually sees it.\n#\n# RUNS BEFORE openwiki-nightly.sh. 15 minutes of headroom is plenty.\nset -euo pipefail\n\nREPO=\"$HOME/YourRepo\"\nOUT=\"$REPO/docs/audit/data-change-digest.md\"\ncd \"$REPO\"\nmkdir -p \"$(dirname \"$OUT\")\"\n\n# `rails runner` interleaves boot output with your stdout. Slice on the markers\n# instead of trusting a clean stdout.\ndocker compose exec -T llamapress bin/rails runner /rails/lib/papertrail_digest.rb \\\n  | sed -n '/--- BEGIN DIGEST ---/,/--- END DIGEST ---/p' \\\n  | sed '1d;$d' \u003e \"$OUT.tmp\"\n\n# Never publish an empty digest over a good one.\nif [ \"$(wc -l \u003c \"$OUT.tmp\")\" -lt 10 ]; then\n  echo \"*** FAILED: digest too short ($(wc -l \u003c \"$OUT.tmp\") lines), keeping previous\"; exit 1\nfi\nmv \"$OUT.tmp\" \"$OUT\"\n\n# The OpenWiki update diffs from the gitHead in .last-update.json. An uncommitted\n# file is not in that diff, so the run will not notice the digest changed.\nif ! git diff --quiet -- \"$OUT\"; then\n  git add \"$OUT\"\n  git commit -m \"chore: nightly data-change digest\"\nfi\n```\n\n```cron\n45 7 * * * /home/ubuntu/YourRepo/bin/local/papertrail-digest-nightly.sh \u003e\u003e /home/ubuntu/papertrail-digest.log 2\u003e\u00261\n 0 8 * * * /home/ubuntu/YourRepo/bin/local/openwiki-nightly.sh\n```\n\n### 5d. Tell OpenWiki the file exists\n\nTwo edits to `openwiki/INSTRUCTIONS.md`. Without them, OpenWiki may treat the digest as\njust another data file and skip it.\n\n```markdown\n## Scope — synthesize ALL of these sources\n- `docs/audit/data-change-digest.md` — a nightly PaperTrail rollup of what real users\n  changed. Treat it as evidence of ACTUAL system usage, distinct from what the code\n  makes possible.\n\n## Required coverage\n11. **Data change patterns** — from `docs/audit/data-change-digest.md`: which models\n    carry real write volume, which columns churn, who the actors are, and which models\n    are NOT versioned (absence of history there means unmeasured, not unchanged). Cross-\n    reference the hot models against the code pages so a reader can jump from \"this\n    table changes constantly\" to the controller that writes it.\n```\n\nNow the wiki says things a code-only wiki never could: *\"`tender_line_item` is the\nhighest-churn model in the system; 94% of edits touch four columns; `Welder` is not\nversioned, so employee-record edits are unknowable.\"* That is the sentence that changes\nwhat an engineer does next.\n\n---\n\n## Gotchas (the hard-won stuff)\n\n**OpenWiki itself**\n\n- **`--init` plans; it does not build.** See the warning in Layer 1. The tell is a\n  `_skeleton.md` in the output folder and a page count in the low single digits.\n- **The first pass writes GENERIC pages, and generic is worse than missing.** Asked\n  to synthesize across dozens of documents in one turn, the agent reaches for\n  plausible themes instead of reading the evidence: a cross-client insight page came\n  back saying clients want \"clear reporting that ties work to outcomes\" — true of\n  every agency on earth, and citing nothing. A page that could have been written\n  without the data looks like insight and is not. Fix it with a **second, narrower\n  pass per deliverable** that demands citations: *\"For every theme you claim, cite at\n  least two specific conversations by name and date and quote the source. Drop any\n  theme you cannot cite.\"* The rewrite came back with six themes, each carrying three\n  or four dated citations and direct quotes.\n- **It reads the FILESYSTEM, not git — so `.gitignore` does not hide `.env` from it.**\n  Add an `.openwikiignore` (same syntax) covering `.env`, `*.pem`, `*.key`, database\n  dumps, `backups/` and `logs/`, or the agent reads your live credentials and may\n  paraphrase them into a page. This is the single most important file in the setup.\n- **The update is git-diff driven from `.last-update.json`'s `gitHead`.** Uncommitted\n  work is largely invisible. If you generate an input file for the wiki, **commit it**\n  (Layer 5c does) — and commit the wiki too, or the first checkpoint rollback on an\n  agent-run box deletes it.\n- **Its model registry goes stale.** v0.3.1 warns that a current model \"is not a known\n  Anthropic model (it belongs to GitHub Copilot)\". The warning is cosmetic — the call\n  is still made — so do not chase it when the real error is underneath.\n- **Never hand-edit `openwiki/index.md` or the GitHub Actions workflow file.** OpenWiki\n  deterministically overwrites both on every run. Same for any generated page: your edit\n  survives until the next run and then vanishes, which is worse than never making it.\n- **`openwiki --help` prints the provider banner before it loads its env file.** Seeing\n  \"provider: OpenAI\" when you configured something else is not a misconfiguration.\n- **OAuth auth expires silently and cron keeps firing.** A revoked refresh token kills\n  the run in seconds. The log then shows a start line and nothing else — identical to\n  \"still running\". Three nights were lost to exactly this. The fix is structural: the\n  `trap` writing `*** FAILED`, and monitoring for a **missing `=== done`** rather than a\n  missing start line.\n- **Under nvm, cron cannot find the binary.** Cron does not source `~/.bashrc`, so\n  `openwiki` installed under Node 22 is not on cron's `PATH`. The explicit `export PATH`\n  is load-bearing, and it hardcodes a Node version — a Node upgrade breaks the cron\n  silently. Re-check it after any nvm change.\n- **Use `flock` and `timeout`.** A slow run overlapping the next night's run produces two\n  agents writing the same files. `timeout 3600` caps a hung run.\n- **The secrets rule belongs in `INSTRUCTIONS.md`, not in your head.** The agent reads\n  your whole repo and writes summaries. Say explicitly that env var *names* are fine and\n  *values* never are.\n- **The wiki folder is generated output, but it is not free to publish.** Decide whether\n  `openwiki/` ships to downstream forks or customer boxes. If it must stay internal, keep\n  it off your deploy allowlist and set `visibility: admin` on the browser.\n\n**The admin browser**\n\n- **`format: false` on the wildcard route.** Without it, a request for\n  `/admin/open-wiki/architecture/overview.md` makes Rails parse `.md` as a response\n  format and the route misses.\n- **Validate every path segment against a regex and re-check the expanded path.** A\n  wildcard route that reads files is a directory-traversal hole by default. The\n  `SEGMENT_RE` check plus the `abs.start_with?` check are both needed — the first blocks\n  `..` in the request, the second catches anything that slips past.\n- **A malformed page must not 500 the wiki.** Frontmatter is model-generated, so it will\n  occasionally be invalid YAML. Rescue `Psych::Exception` and return `{}`.\n- **Use `YAML.safe_load` with `permitted_classes: []`.** Plain `YAML.load` on a\n  generated file is remote code execution waiting for a bad run.\n- **A directory bind-mount hot-reloads; a single-file mount does not.** `./openwiki` as a\n  directory means new pages appear with no restart. If you instead mount an individual\n  file, an atomic-write editor swaps the host inode and detaches it from the mount — the\n  host file changes and the container keeps reading the old one, silently.\n- **Rewrite relative links or the wiki is unnavigable.** OpenWiki writes portable\n  Markdown links. Under `/admin/open-wiki/...` every one of them 404s until you resolve\n  it against the current page's directory and strip the `.md`.\n- **Show the last-updated date and model in the header.** A stale wiki looks exactly like\n  a fresh one. This is the cheapest possible staleness alarm.\n- **Escape the body, then decode it in JS.** Interpolating raw Markdown into a\n  `\u003cscript\u003e` tag unescaped lets a generated page break out of it.\n\n**PaperTrail**\n\n- **Confirm the gem is actually in your image before designing around it.** On this\n  system, the *same* initializer and migration ship everywhere, but the mothership image\n  does not bundle `paper_trail` — so `defined?(PaperTrail)` is `false`, the initializer\n  no-ops, and `versions` sits at 0 rows while looking perfectly configured. The fleet\n  image does bundle it (`paper_trail ~\u003e 15.2`). A table existing is not proof the gem is\n  loaded. Check `defined?(PaperTrail)`, not the schema.\n- **The gem being present is still not tracking.** Nothing is versioned until a model\n  declares `has_paper_trail`. A fresh box has the gem, the table, and zero coverage.\n- **Bulk inserts skip callbacks, so they produce NO version.** `insert_all`,\n  `upsert_all`, and most Excel/CSV importers write rows PaperTrail never sees. One real\n  project had 1,549 versions, every one an `update` and not a single `create` — the\n  values arrived with the insert. **Never read a missing create version as \"nobody set\n  it.\"** Say this in the digest, or the wiki will confidently state the opposite.\n- **Re-imports that destroy and recreate rows break history continuity.** History follows\n  the row id, not the thing the row represents. On that same project, `destroy` was the\n  single largest event type (34,685 of 46,396). A record's story ends at each re-import.\n- **`whodunnit` is a user id stored as a string** — no name, no email. Join it yourself,\n  and expect nulls from anything that runs outside a request (jobs, console, rake).\n- **There is an install-date horizon.** Nothing before the day you added\n  `has_paper_trail` is knowable. Put that date in the digest so nobody mistakes the\n  horizon for a quiet period.\n- **Never let the digest emit `object` or `object_changes` values.** Those columns hold\n  your users' real data verbatim — names, addresses, amounts. Column names and counts\n  answer every question the wiki needs and leak nothing. This matters twice over because\n  the digest gets committed *and* fed to a model provider.\n- **Slice `rails runner` output on markers.** Boot logs and warnings interleave with your\n  stdout; redirecting it straight into a Markdown file gets you a Markdown file with a\n  Docker warning at the top.\n- **Guard against writing an empty digest.** A failed query producing a 3-line file that\n  overwrites a good one turns a monitoring system into a source of false calm.\n\n---\n\n## Files this pattern touches\n\n```\nopenwiki/INSTRUCTIONS.md                          # the standing brief — edit THIS\nopenwiki/**/*.md                                  # generated; never hand-edit\nopenwiki/.last-update.json                        # gitHead + model of the last run\n\ndocker-compose.yml                                # ./openwiki:/rails/openwiki:ro\nconfig/routes.rb                                  # 2 routes, format: false\napp/controllers/admin/open_wiki_controller.rb     # read-only browser\napp/views/admin/open_wiki/show.html.erb           # sidebar + marked.js + link rewriting\n\nbin/local/openwiki-nightly.sh                     # cron wrapper: flock, timeout, markers\nbin/local/papertrail-digest-nightly.sh            # cron wrapper: digest + commit\nlib/papertrail_digest.rb                          # the rollup queries\ndocs/audit/data-change-digest.md                  # generated input to the wiki\n\nGemfile                                           # gem \"paper_trail\"\nconfig/initializers/paper_trail.rb\ndb/migrate/XXXXXXXX_create_versions.rb\napp/controllers/application_controller.rb         # set_paper_trail_whodunnit\napp/models/*.rb                                   # has_paper_trail on chosen models\n.github/workflows/openwiki-update.yml             # optional; regenerated by OpenWiki\n```\n\n## How to adapt to your stack\n\n1. **Swap the model provider.** Set `OPENWIKI_PROVIDER` plus that provider's key. A\n   subscription-billed provider (ChatGPT/Codex-backed) avoids per-token metering on a\n   nightly job; an API key is simpler to automate. Pick before you schedule it — the\n   auth-expiry failure mode in Gotchas is specific to OAuth providers.\n2. **Not on Docker?** Drop the `docker compose exec` from the digest wrapper and run\n   `bin/rails runner lib/papertrail_digest.rb` directly. The controller does not change;\n   only the mount goes away.\n3. **Not Postgres?** The rollup queries use `FILTER (WHERE …)` and `jsonb_each`. On MySQL,\n   use `SUM(event = 'create')` and `JSON_KEYS(object_changes)`. On SQLite, store\n   `object_changes` as text and roll up in Ruby instead.\n4. **Public wiki instead of admin-only?** Drop `ensure_admin`, move the routes out of the\n   `admin` namespace, and re-read the secrets rule in `INSTRUCTIONS.md` first — an agent\n   summarizing your repo into a public page is a disclosure path.\n5. **Feed it something other than PaperTrail.** The pattern is: *roll up a data source\n   into a committed Markdown file, then name that file in `INSTRUCTIONS.md`.* Good\n   candidates are exception counts by class, slowest jobs by runtime, and support-ticket\n   themes. Keep each digest to one file with stable headings so the diff stays readable\n   and the run stays cheap.\n6. **Safe to drop:** the GitHub Actions workflow (pick cron or CI, not both), the\n   `?raw=1` endpoint if no agents read the wiki over HTTP, and the coverage section of\n   the digest once every model you care about is versioned.\n"}