Ask the Archive
How years of books, races, forts, essays and short posts were gathered, moved into one database, and made answerable in a sentence — with readers grading the answers and one admin dashboard controlling all of it.
Every page could search itself. None could search another.
Six filter boxes — books, projects, micro posts, races, treks, tags — each scanning its own array in the browser. They all worked; none talked to each other.
“Which forts did I climb the year I read the most Marathi non-fiction?”
Two tables
Treks store dates as DD-MM-YYYY text; books store the year read as an integer.
A definition
“Most” is an aggregate. No filter box counts.
A conversation
That is not a filter. It is a question — and it needed a place to go.
The data came first — years before the feature
Nothing here was collected for a chatbot. It accumulated: a Tumblr habit, a reading log, race certificates, fort treks, a writing challenge.
Micro posts peaked at 535 in 2020; essays exploded in 2026 with 61 of the 104. Every bar is a row a question can now reach.
Structured, but never in the same shape twice
Each type arrived on its own terms. The quirks are the reason one generic search could never work — and the reason the chatbot reads a hand-written brief on what fields mean.
Micro posts · 1,661
97% imported from Tumblr; many are reblogs — passing thoughts, not positions.
Books · 51
year is the year read, not published — half the dates are reconstructed, so date_precision exists.
Races · 25
Distance is a text label (“21 Kms”, “21K”), dates read “February 22, 2026”.
Treks · 20
Dates are DD-MM-YYYY — a different format from races, for historical reasons.
100,199 words, and most of them recent
104 essays on Substack and WordPress since 2020. Only 60 have a row in the blogs table — the rest lived only on the platforms, until v17.1 pulled in their full text.
By platform
Substack 76 posts · WordPress 28 posts. Busiest month: June 2026, 12 posts. Longest: The Un-Orthodox Ways of Seeing!, 3,529 words.
From files in a repo to one Postgres
For two years the content was JavaScript data files, then markdown behind a CMS. On 13 June 2026 it all moved into Supabase — and every later feature, including the chatbot, stands on that move.
Before · v1 → v7
src/data/*.jshand-edited, then generated from markdown- Decap CMS writing commits; a CI gate failing on drift
- Images in the repo — 114 MB before compression
- 12 posts, 4 books and 1 trek existed only in generated files
After · v9 → today
- Supabase Postgres is the single source of truth
- Row-Level Security: public read, owner-only write
- Images in a public storage bucket, compressed on upload
- A schema-driven admin at
/adminwrites every table
v8.0 (Vite), v9.0 (Supabase), v9.1 (CMS pipeline removed) and v10.0 (1,600-post Tumblr import) all shipped the same day: 13 June 2026.
Fifteen content tables, one tag axis, one search index
Tags and search are both polymorphic — an entity_type plus an entity_id — so a book, a race and a micro post share one tag list and one index.
The database, not React, keeps the promises
Every guarantee the chatbot relies on is enforced below the application, so no code path can forget it.
Drafts stay drafts
The projects read policy is visible or is_owner(): a hidden project never reaches a page, a tag page, a share card — or the index.
Tags are central
213 tags in one table, linked through tag_associations; set_entity_tags() replaces a row's set atomically.
Dates are real
Project dates became a Postgres date; book finish dates carry a precision. Sorting stopped lying.
The log is owner-only
Visitors write the conversation log through a definer function and can never read it back.
A question in, grounded answers out
At /ask and in a launcher on every page. English or Marathi. Sources arrive first, prose streams after, and every name links back to its row.
Across 25 races and 577 km, his half-marathon best is 1:59:57 at NMDC Hyderabad Marathon 2025 Race1, and his marathon best is 05:05:53 at Tata Mumbai Marathon 2026 Race2. He wrote about the training on sankettambare.substack.com ↗3.
Rendered the way v17.1 renders an answer — site links as chips labelled with their type, outside links with their domain.
Six stations, about four seconds
Measured on a logged exchange: retrieval 1.81 s, generation 2.39 s, 4.20 s end to end. Logging happens after the answer is on its way, so it can never slow one down.
Two searches, one ranking
Keyword search knows “Ghangad” is a word. Vector search knows “forts near Pune” means Ghangad. Both run, and Reciprocal Rank Fusion adds up 1 / (k + rank) from each list.
Keyword
tsvector with the simple config — stemming Devanagari is worse than not stemming. Terms ORed, prefix-matched (trek:*).
Meaning
pgvector with HNSW over @cf/baai/bge-m3, multilingual, 1024 dimensions.
Degrades in place
No embedding budget? Pass a null vector: the same function becomes keyword-only. No second code path to rot.
Vectors cannot count
“How many books?” returns eight books that feel like the question. So every prompt carries a card of aggregates — the same numbers the /stats page shows, from one hourly snapshot.
{ counts: { books: 51, microblog: 1661,
blogs: 60, projects: 13,
sports: 25, treks: 20, tags: 213 },
books: { by_language: { English: 30, Marathi: 21 } },
writing: { words: 100199, posts: 104,
current_period: …recomputed today },
site_stats: …every figure on /stats }
One module
siteStats.js computes every /stats figure; the page, the facts card and the index all import it.
Once an hour
GET /api/stats is edge-cached for 3,600 s. The chatbot and the page cannot disagree.
Four rungs, and the last one always holds
Tried top to bottom. Order, model ids, timeouts and switches are rows in ask_settings — when a model was retired mid-build, the fix was editing a field.
- Gemini Flash-Litebest quality · falls over on 429, 5xx or a 6 s timeout
- Workers AI · gpt-oss-20bno key, runs beside the worker
- Workers AI · llama-3.2-3bsame neuron pool, about five times cheaper
- Search-onlythe matching pages and one honest line — never an error page
Every one of the 30 answers logged so far came from rung one.
It cannot link to anything it did not retrieve
The micro-post archive is full of reblogged third-party text pasted into every prompt. So retrieved rows are fenced as data, and any URL or image not in a retrieved row is stripped — in the worker and again in the page.
What the model wrote
PB **1:59:57** at [NMDC Hyderabad Marathon 2025](/sports/15) [1]. Read more at [this link](https://evil.example/x). 
What the reader sees
PB 1:59:57 at NMDC Hyderabad Marathon 2025 Race1. Read more at this link.
The words stay; the invented URL and the tracking pixel are gone.
Caps fail closed
Global and per-visitor daily caps counted in Postgres over a salted IP hash. If the quota check is down, the request is refused.
Turnstile recovers
Every path waits for a fresh token; a rejected one offers Verify and retry.
Data, never instructions
Retrieved items sit inside delimiters the system prompt declares inert.
2,702 chunks across thirteen kinds of content
v17.0 indexed nine types and saw an essay only as its title. The audit found five sources the chatbot could not see — the résumé, full essay text, every stats figure, tag notes, the site map. v17.1 indexes all of them.
Orange rows arrived in v17.1. The blog row grew from 60 metadata chunks to 455 when every tracked post's text was added. A tag source is built too, waiting on tag descriptions.
A new source is one file
Every module in scripts/ask-sources/ is a source. Since migration 0016 the index accepts any well-formed type name, so there is no migration per source and no registry to edit.
// scripts/ask-sources/reading_list.mjs
export default {
type: "reading_list",
load: (ctx) => ctx.fetchAll("reading_list"),
toChunks: (rows, { compose }) => rows.map((r) => ({
entity_type: "reading_list",
entity_id: r.id,
chunk_index: 0,
title: r.title,
url: "/reading-list",
body: compose([["Reading list", r.title], [null, r.note]]),
})),
};A failing source keeps its chunks
If load() throws, that type is skipped for deletion — a Substack outage cannot empty the index.
Unknown types still render
The chat labels a type it has never seen with a title-cased name.
15 sources today
book, blog, microblog, project, sport, trek, instagram, now, page, resume, writing, stats, tag, site.
Only genuinely new text is embedded
Timestamps lied twice: a CI checkout resets file times, and the changelog grows at the top, renumbering every chunk. Hashing the text fixed both.
| Key | Hash | Outcome | Embeds? |
|---|---|---|---|
| same | same | unchanged — skipped | no |
| new | seen elsewhere | moved — copy the vector | no |
| new | new | new — embed, upsert | yes |
| only in DB | — | gone — delete, unless its source failed | no |
The migration run
2,693 chunks planned; 825 embedded (~210K tokens, about $0.003 at list price). The next run: 0.
Every night at 02:00, a quiet night costs nothing
A GitHub Action refreshes the Writing Ledger and the index. No new posts means no downloads, no embeddings, no commit and no deploy.
- Count every post
blogs:wordcountlists Substack + WordPress and rebuilds the ledger JSON. - Download only new bodiesText is cached by URL and version; the first run fetched 34 and reused 42.
- Hash every chunkEmbedded text and metadata get separate hashes.
- Embed only new textMoved text keeps its vector.
- Refresh the facts cardPushed into
ask_settings.context_doc. - Commit only real changesA new date alone never triggers a deploy.
“Indexed” was checked row by row
Production was compared against every source after the v17.1 run.
The one real gap: two micro posts ran past the embedding model's 4,000-character window. Long posts are split on paragraphs now — the fix embedded 15 chunks and left 2,687 alone.
Readers grade every answer
Under each answer: thumbs up, thumbs down, and Suggest. A thumbs down or Suggest opens reason tags and a comment box. The verdict survives a reload.
What could be better?
Stored on the answer
Written onto that answer's ask_messages row, beside the tier, model, timings and sources — so a bad answer can be traced to its cause.
Only its reader can rate it
ask_feedback() updates a row only when the caller's session owns the conversation. Comments cap at 1,000 characters.
Every verdict becomes a test case
The loop is live: 30 exchanges logged so far, 2 of them rated — one up, one down. Small numbers, honest signal.
{"question":"When did he start running?","answer":"The micro posts show the build-up…",
"tier":"gemini-flash-lite","keyword_only":false,"rating":-1,
"tags":["missing data"],"comment":"Doesn't mention the first 10K race."}An illustrative line in the export's shape — one JSON line per rated exchange.
Everything is controlled from /admin
One owner login. Content, tags, the Now page, the résumé and the whole chatbot are edited in one workspace — no redeploy for any of it.
Overview
Quick add: micro post, race, bookIs a Now month current?Recently updated, every tableContent
BooksRacesTreksProjects100 DaysInstagramMicro BlogTagsNow
Months · auto-fill from tablesMeta · rituals, inspirationsAsk
SettingsConversationsRésumé
Experience · EducationCertifications · SkillsSchema-driven forms
Fields declared once in resources.js; date pickers, tag autocomplete, link lists, unsaved-changes guard.
Images compress themselves
Resized to ≤1200 px and ≤150 KB in the browser before upload.
Tags managed centrally
Rename, merge duplicates, recolour, export and import metadata.
Every dial of the chatbot is a form field
Read by the worker with a 60-second cache: a change is live within a minute.
Switches
Caps
Retrieval
Model ladder
Words
Every chat, every filter, one export
Each chat shown in full, newest first — a browser's questions stay together until it goes quiet for 30 minutes. The summary at the top follows whatever is filtered.
Summary for the selection
Questions, chats, satisfaction, comments, latency by stage, tier and model mix, reasons, content cited, questions per day, and index coverage per type.
Pure functions underneath
Grouping, filtering and summarising run client-side over one date-ranged load, all unit-tested.
83 releases, and the curve bends in 2026
Every bar is a changelog version. Three years of occasional updates, then a summer where the site was rebuilt around its data.
June 2026 alone shipped 22 versions; September 2026, 11 — the data migrations and the chatbot.
2023 → 2026: a template became an archive
Major versions from the changelog, coloured by what they changed.
- v1.0Forked a template; resume, stats, routing
- v1.2Instagram and videos pages
- v2.0Sports page — first race data
- v3.5Books page, personal stats data, dark mode
- v4.0Redesign; 100 Days To Offload tracker
- v5.0Changelog page; sports statistics
- v5.1Treks page; Cloudflare middleware
- v6.0First admin panel and Decap CMS
- v7.0Markdown as source of truth; CI drift gate
- v9.0Supabase: Postgres, RLS, storage, live data
- v10.0Micro Blog: 1,600-post Tumblr import, full-text search
- v11.0The Wanderer's Atlas — gamified homepage
- v12.2Blog word counts and the Writing Ledger
- v13.0Admin redesigned as a workspace
- v14.0Centralised tags across every table
- v15.0Projects reborn: real dates, visibility in RLS
- v16.0Books reborn: metadata, statuses, precision
- v17.0Ask the Archive: hybrid search, ladder, log
- v17.1Every source, incremental index, feedback, evals
Data first, then the brain
Read the order: the chatbot is the last layer, not the first. Each step made the next one possible.
Jun 2026 · Move
v8 Vite · v9 Supabase · v9.1 storage, CMS removed · v10 Micro Blog import · v10.1 detail pages · v10.2 share as image
Jul 2026 · Show
v10.3 Substack RSS, monthly digest · v11 Atlas · v12 Stats Almanac · v12.1 Now auto-fill · v12.2 Writing Ledger · v13 admin workspace
Aug 2026 · Tidy
v13.1 images compress on upload · v13.2 canonical domain, classic default
Sep 2026 · Structure, then ask
v14 central tags · v15 projects · v16 books · v17.0 Ask the Archive · v17.0.1–2 streaming + Turnstile fixes · v17.1 sources, feedback, evals
Still $0 a month
Free tiers are the design constraint. At list price one question would cost about $0.00037 — and the real ceiling is requests per day, which is why the caps sit below the free allowance.
What the evals will decide
The foundation is data, not the model. The next moves are chosen by what readers mark as wrong.
Search with context
Prepend the previous question when a follow-up is vague — retrieval currently reads the new message alone.
Describe the tags
The tag source is built and indexes nothing until descriptions are written.
A compiled digest layer
Maintained notes that answer what do I think, not only what did I say.
One search everywhere
Point the six page filters at the same hybrid search.
Try it at sankettambare.in/ask · the build log: Ask the Archive dossier · the audit