Static site search, like the index in a reference book
Search that works like the index at the back of a reference book: compile in advance, and consult it yourself rather than pester a librarian.
For a long time I didn't want a search box on this site. Not because I didn't want to make it easier for readers to find a post, but because I was concerned about the responsibility it would entail. I really wanted to avoid signing up for a service, pasting an API key into my build, and sending every word that you type to somebody else's servers.
It boils down to the same preoccupation I've had with every decision concerning the architecture of "Be Useful". Do the benefits of this or that feature outweigh the practicalities of implementation, the ease of use, performance, or respecting the basic obligations for data protection? The clue is in the name.
The sentence I can stand by
This site has a privacy page. It's written in plain English (I think), where every sentence is intended to be as transparent as possible.
I invite you to go and read it. You'll notice that some sentences are hedged. The paragraph about analytics begins "according to Vercel". The paragraph about hosting says that Vercel acts as a data processor under its own terms. Those hedges are deliberate because I personally haven't audited Vercel. I've read what they publish about themselves, found it reasonable, and passed it on to you with an attribution so you can see exactly how much weight it carries.
That's a normal and unavoidable thing to do. Every site you've ever visited is doing the same thing (probably). But each hedge is a sentence I'm relaying rather than verifying, and adding more carries a fresh burden of oversight to a personal project which has no commercial value.
Case in point: adding a hosted third-party search service. Something along the lines of, according to whoever, your search terms are handled thus and retained for so long. Accurate, probably. Can I personally guarantee it? No. And it'd need re-checking every time that company changed their terms, their sub-processors, or the country the data sits in. That's a maintenance job I don't need nor want for the "convenience" of a search bar.
To make a site search possible, the statement I want to share on the privacy page doesn't have "according to" in it at all. No hedging, dear reader.
How a search box normally works
The usual arrangement is that the search box is a messenger. You type some words. Perhaps it's a swearword because you're an edgelord. The words travel to a server, the server consults a database of your site's contents, and it sends back a list of matches.
That round trip is why search has traditionally needed a back end. It's also why the path of least resistance is to rent functionality from Algolia or hand the job to Google.
A prerendered site has nothing that can answer the query. A CDN takes a request for a URL and hands back the file that matches it. It never opens that file, so it has no way to answer a question about what is inside one.
How Pagefind does it instead
Pagefind is a single command, an independent open source project originally built at CloudCannon. It's not a company or a subscription. It lives in this project's development dependencies, which means it's installed on my machine and on the build server and never ships to anyone reading the site.
It runs at the end of the build, after Next.js has finished turning the CMS content into HTML files, and its whole job is to read those files and write an index next to them.
An index here means a compressed map of every word to every page it appears on. The simplest analogy is that it works like the index at the back of a reference book. Somebody compiled it in advance, it travels with the book, and you consult it yourself rather than pestering a librarian.
The index ships as ordinary static files alongside the pages. When you type in the box, your browser fetches the pieces of the index it needs and does the matching locally, using a small WebAssembly module.
Pagefind splits the index up so that a visitor downloads a fraction of it rather than the whole thing, which is what allows sites with tens of thousands of pages to work this way. Mine is about twenty pages, give or take, so deeply inconsequential in terms of scale.
Is this site even static?
A question I had to stop and ponder, because Pagefind runs against the build output and needs the content to exist as HTML on disk before it can index anything. A site rendered fresh on every request would have nothing there to read.
So I checked the response headers. Every page comes back marked as prerendered and served from cache, which means the HTML is a finished file before anyone asks for it.
Underneath that, Next.js regenerates pages on a 24-hour stale window and whenever the CMS tells it to, and there are three small API routes handling revalidation and draft previews. There is server-side code in the deployment. What there isn't, anywhere, is something that answers a search query.
Static enough, then, for the purpose at hand.
The paragraph, and how to check it
The privacy page now says that the search runs entirely in your browser, that the words you type are never sent to this site or to anyone else, and that nothing about your search is stored. No attribution, no according to. Those are descriptions of a mechanism rather than assurances from a company.
You don't need to take my word for it, either. This site sends a Content Security Policy (opens in a new window) header, which is a list of rules the browser itself enforces on the page. One of those rules, connect-src, governs where the page is allowed to open a network connection. Mine is set to this domain plus the two hosts Vercel's analytics use, and anything else is refused by the browser rather than by my good intentions. Open the network tab, type a query, and nothing goes out.
Two Vercel hosts isn't zero, which is why the hedged paragraphs are still on that page and will stay there. Vercel hosts the site, so they already handle every request that reaches it, and the analytics run without cookies. What I avoided was adding another service provider, and another claim I'd have to take on trust.
The same CSP header carries a permission called wasm-unsafe-eval, which is there for one reason. Pagefind's matching runs as WebAssembly, and without that permission the browser would refuse to start it.
What it actually costs
There are some compromises to document, all three of them:
First, loading the search page pulls down about 132KB compressed, split between the search library at 13KB, the WebAssembly module at 72KB, and the interface components at 47KB. That's a fair chunk of code. It only loads on the search page, and every byte comes from this domain, but it's real weight.
Secondly, it needs JavaScript to work. With scripts turned off you get the page and the illustration of a magnifying glass1The search emblem I created is an SVG, so the image is adaptable whether the site is in light or dark mode., but no working search.
Thirdly, the index is only as current as my last deployment. It gets built in a postbuild step, straight after Next.js finishes, but publishing a post doesn't rebuild the site. The CMS pings the site, the affected pages regenerate, and the new post appears on the front page and in the feed within seconds. The search index sits there unchanged.
The two things happen in different places, which is why neither can reach the other. Pagefind runs once inside the build pipeline, on a machine that stops existing the moment the build finishes. Regeneration happens later and somewhere else, on the servers handling live requests, and there is no build machine left for it to hand anything to. So there's a window where a post is live, readable and linked from the home page, and typing its title into the search box returns nothing.
That would be a serious problem for a site publishing several times a day. I publish a couple of times a month, and the window closes the next time I tinker with the code (which is usually sooner than that). The limitation is real, but it's the right size for the channel it applies to.
The archive, and four ways in
Oh, and I added something else in parallel to the search function. An archive page, which lists every post grouped by year with plain titles, dates and categories in a list. I built it because the front page had grown to four pages of pagination, and reaching a post from last autumn meant clicking through all of them.
Between them, the site now has four ways to reach the same library of posts:
Categories answer what kind of thing a post is.
Authors answer who wrote it, which here also means which of the two mascots is responsible.
The archive answers what exists, all of it, at once.
Search answers the case where you already know the word you're looking for.
Not one of these functions is a substitute for another, and it makes me chuckle to think that information architecture has become as exciting to me as content creation. Nerdery!
Nothing to maintain
So, to recap. "Be Useful" has a search function where there's no API key waiting to be rotated, no free tier that becomes a paid one in eighteen months, and no vendor whose terms I need to re-read every year to keep a paragraph accurate. The index rebuilds from the same HTML the CDN is already serving, every time the site deploys.
I put off adding search to this site because I didn't want to layer on more responsibility than necessary for a personal site, and I assumed I'd have to make do without. What it actually took was a build step, 132KB, and three compromises I can accept.
