How years of books, races, forts, essays and short posts were gathered, moved into one database, and made answerable in a sentence — with readers grading the answers and one admin dashboard controlling all of it.
Six filter boxes — books, projects, micro posts, races, treks, tags — each scanning its own array in the browser. They all worked; none talked to each other.
“Which forts did I climb the year I read the most Marathi non-fiction?”
Treks store dates as DD-MM-YYYY text; books store the year read as an integer.
“Most” is an aggregate. No filter box counts.
That is not a filter. It is a question — and it needed a place to go.
Nothing here was collected for a chatbot. It accumulated: a Tumblr habit, a reading log, race certificates, fort treks, a writing challenge.
Micro posts peaked at 535 in 2020; essays exploded in 2026 with 61 of the 104. Every bar is a row a question can now reach.
Each type arrived on its own terms. The quirks are the reason one generic search could never work — and the reason the chatbot reads a hand-written brief on what fields mean.
97% imported from Tumblr; many are reblogs — passing thoughts, not positions.
year is the year read, not published — half the dates are reconstructed, so date_precision exists.
Distance is a text label (“21 Kms”, “21K”), dates read “February 22, 2026”.
Dates are DD-MM-YYYY — a different format from races, for historical reasons.
104 essays on Substack and WordPress since 2020. Only 60 have a row in the blogs table — the rest lived only on the platforms, until v17.1 pulled in their full text.
Substack 76 posts · WordPress 28 posts. Busiest month: June 2026, 12 posts. Longest: The Un-Orthodox Ways of Seeing!, 3,529 words.
For two years the content was JavaScript data files, then markdown behind a CMS. On 13 June 2026 it all moved into Supabase — and every later feature, including the chatbot, stands on that move.
src/data/*.js hand-edited, then generated from markdown/admin writes every tablev8.0 (Vite), v9.0 (Supabase), v9.1 (CMS pipeline removed) and v10.0 (1,600-post Tumblr import) all shipped the same day: 13 June 2026.
Tags and search are both polymorphic — an entity_type plus an entity_id — so a book, a race and a micro post share one tag list and one index.
Every guarantee the chatbot relies on is enforced below the application, so no code path can forget it.
The projects read policy is visible or is_owner(): a hidden project never reaches a page, a tag page, a share card — or the index.
213 tags in one table, linked through tag_associations; set_entity_tags() replaces a row's set atomically.
Project dates became a Postgres date; book finish dates carry a precision. Sorting stopped lying.
Visitors write the conversation log through a definer function and can never read it back.
At /ask and in a launcher on every page. English or Marathi. Sources arrive first, prose streams after, and every name links back to its row.
Across 25 races and 577 km, his half-marathon best is 1:59:57 at NMDC Hyderabad Marathon 2025 Race1, and his marathon best is 05:05:53 at Tata Mumbai Marathon 2026 Race2. He wrote about the training on sankettambare.substack.com ↗3.
Rendered the way v17.1 renders an answer — site links as chips labelled with their type, outside links with their domain.
Measured on a logged exchange: retrieval 1.81 s, generation 2.39 s, 4.20 s end to end. Logging happens after the answer is on its way, so it can never slow one down.
Keyword search knows “Ghangad” is a word. Vector search knows “forts near Pune” means Ghangad. Both run, and Reciprocal Rank Fusion adds up 1 / (k + rank) from each list.
tsvector with the simple config — stemming Devanagari is worse than not stemming. Terms ORed, prefix-matched (trek:*).
pgvector with HNSW over @cf/baai/bge-m3, multilingual, 1024 dimensions.
No embedding budget? Pass a null vector: the same function becomes keyword-only. No second code path to rot.
“How many books?” returns eight books that feel like the question. So every prompt carries a card of aggregates — the same numbers the /stats page shows, from one hourly snapshot.
{ counts: { books: 51, microblog: 1661,
blogs: 60, projects: 13,
sports: 25, treks: 20, tags: 213 },
books: { by_language: { English: 30, Marathi: 21 } },
writing: { words: 100199, posts: 104,
current_period: …recomputed today },
site_stats: …every figure on /stats }
siteStats.js computes every /stats figure; the page, the facts card and the index all import it.
GET /api/stats is edge-cached for 3,600 s. The chatbot and the page cannot disagree.
Tried top to bottom. Order, model ids, timeouts and switches are rows in ask_settings — when a model was retired mid-build, the fix was editing a field.
Every one of the 30 answers logged so far came from rung one.
The micro-post archive is full of reblogged third-party text pasted into every prompt. So retrieved rows are fenced as data, and any URL or image not in a retrieved row is stripped — in the worker and again in the page.
PB **1:59:57** at [NMDC Hyderabad Marathon 2025](/sports/15) [1]. Read more at [this link](https://evil.example/x). 
PB 1:59:57 at NMDC Hyderabad Marathon 2025 Race1. Read more at this link.
The words stay; the invented URL and the tracking pixel are gone.
Global and per-visitor daily caps counted in Postgres over a salted IP hash. If the quota check is down, the request is refused.
Every path waits for a fresh token; a rejected one offers Verify and retry.
Retrieved items sit inside delimiters the system prompt declares inert.
v17.0 indexed nine types and saw an essay only as its title. The audit found five sources the chatbot could not see — the résumé, full essay text, every stats figure, tag notes, the site map. v17.1 indexes all of them.
Orange rows arrived in v17.1. The blog row grew from 60 metadata chunks to 455 when every tracked post's text was added. A tag source is built too, waiting on tag descriptions.
Every module in scripts/ask-sources/ is a source. Since migration 0016 the index accepts any well-formed type name, so there is no migration per source and no registry to edit.
// scripts/ask-sources/reading_list.mjs
export default {
type: "reading_list",
load: (ctx) => ctx.fetchAll("reading_list"),
toChunks: (rows, { compose }) => rows.map((r) => ({
entity_type: "reading_list",
entity_id: r.id,
chunk_index: 0,
title: r.title,
url: "/reading-list",
body: compose([["Reading list", r.title], [null, r.note]]),
})),
};If load() throws, that type is skipped for deletion — a Substack outage cannot empty the index.
The chat labels a type it has never seen with a title-cased name.
book, blog, microblog, project, sport, trek, instagram, now, page, resume, writing, stats, tag, site.
Timestamps lied twice: a CI checkout resets file times, and the changelog grows at the top, renumbering every chunk. Hashing the text fixed both.
| Key | Hash | Outcome | Embeds? |
|---|---|---|---|
| same | same | unchanged — skipped | no |
| new | seen elsewhere | moved — copy the vector | no |
| new | new | new — embed, upsert | yes |
| only in DB | — | gone — delete, unless its source failed | no |
2,693 chunks planned; 825 embedded (~210K tokens, about $0.003 at list price). The next run: 0.
A GitHub Action refreshes the Writing Ledger and the index. No new posts means no downloads, no embeddings, no commit and no deploy.
blogs:wordcount lists Substack + WordPress and rebuilds the ledger JSON.ask_settings.context_doc.Production was compared against every source after the v17.1 run.
The one real gap: two micro posts ran past the embedding model's 4,000-character window. Long posts are split on paragraphs now — the fix embedded 15 chunks and left 2,687 alone.
Under each answer: thumbs up, thumbs down, and Suggest. A thumbs down or Suggest opens reason tags and a comment box. The verdict survives a reload.
What could be better?
Written onto that answer's ask_messages row, beside the tier, model, timings and sources — so a bad answer can be traced to its cause.
ask_feedback() updates a row only when the caller's session owns the conversation. Comments cap at 1,000 characters.
The loop is live: 30 exchanges logged so far, 2 of them rated — one up, one down. Small numbers, honest signal.
{"question":"When did he start running?","answer":"The micro posts show the build-up…",
"tier":"gemini-flash-lite","keyword_only":false,"rating":-1,
"tags":["missing data"],"comment":"Doesn't mention the first 10K race."}An illustrative line in the export's shape — one JSON line per rated exchange.
One owner login. Content, tags, the Now page, the résumé and the whole chatbot are edited in one workspace — no redeploy for any of it.
Fields declared once in resources.js; date pickers, tag autocomplete, link lists, unsaved-changes guard.
Resized to ≤1200 px and ≤150 KB in the browser before upload.
Rename, merge duplicates, recolour, export and import metadata.
Read by the worker with a 60-second cache: a change is live within a minute.
Each chat shown in full, newest first — a browser's questions stay together until it goes quiet for 30 minutes. The summary at the top follows whatever is filtered.
Questions, chats, satisfaction, comments, latency by stage, tier and model mix, reasons, content cited, questions per day, and index coverage per type.
Grouping, filtering and summarising run client-side over one date-ranged load, all unit-tested.
Every bar is a changelog version. Three years of occasional updates, then a summer where the site was rebuilt around its data.
June 2026 alone shipped 22 versions; September 2026, 11 — the data migrations and the chatbot.
Major versions from the changelog, coloured by what they changed.
Read the order: the chatbot is the last layer, not the first. Each step made the next one possible.
v8 Vite · v9 Supabase · v9.1 storage, CMS removed · v10 Micro Blog import · v10.1 detail pages · v10.2 share as image
v10.3 Substack RSS, monthly digest · v11 Atlas · v12 Stats Almanac · v12.1 Now auto-fill · v12.2 Writing Ledger · v13 admin workspace
v13.1 images compress on upload · v13.2 canonical domain, classic default
v14 central tags · v15 projects · v16 books · v17.0 Ask the Archive · v17.0.1–2 streaming + Turnstile fixes · v17.1 sources, feedback, evals
Free tiers are the design constraint. At list price one question would cost about $0.00037 — and the real ceiling is requests per day, which is why the caps sit below the free allowance.
The foundation is data, not the model. The next moves are chosen by what readers mark as wrong.
Prepend the previous question when a follow-up is vague — retrieval currently reads the new message alone.
The tag source is built and indexes nothing until descriptions are written.
Maintained notes that answer what do I think, not only what did I say.
Point the six page filters at the same hybrid search.
Try it at sankettambare.in/ask · the build log: Ask the Archive dossier · the audit