Dan Hartropp

[Brutally] optimising for cost

Thu, 08 Oct 2026 00:00:00 +0000

Serverless compute is a wonderful deal when you're small. You only pay for what you use, it all scales to zero when not in use, and the free tiers are generous enough that a side project can live on them indefinitely. The catch is that the bill scales linearly with usage, and the number of operations a system performs has a habit of multiplying in ways nobody notices until the invoice arrives.

I've recently spent some time building a frost and rain alerting service at weather.danhartropp.com, and I set myself a deliberately unreasonable target from the start: it had to be able to run a million forecasts every night without the bill becoming a problem ... and the million users should all arrive in the same month. I don't expect anything like that traffic, but designing for that kind of scale now should prevent an annoying bill later. It should also help prevent the far more likely case, which is just enough users to tip me over the free tier and start paying for it out of my own pocket.

The bottom line: if you know where your costs scale (per request, per user per day, or per byte) you can usually arrange the design so the expensive multiplier never arrives, and for bursty/batchy operations you might be able to stay inside a free tier. If you don't know, the default patterns will quietly pick the most expensive option for you. It's mostly just reading the pricing page and doing a few sums before writing the code.

What the app actually does

Each day, a batch job downloads the Met Office's UK forecast, which is published as open data. For every place someone has subscribed to, it samples the forecast at that point, adds a set of terrain descriptors (how sheltered it is, whether cold air pools there, that sort of thing), and runs two small machine learning models that predict how much colder the ground will get than the Met Office forecast says. If the answer crosses the frost threshold, the subscriber gets a push notification. Rain works in a similar way but is simpler, because the Met Office already publishes a probability for 8,667 sites across the UK ... so the job just looks up the nearest one and applies a calibration curve.

This means there are two very different halves to the system. There's a small website where people search for a place, drop a pin on a map and subscribe, and there's a batch job that runs twice a day and does the actual forecasting.

Where the money goes when Hacker News finds you

The (imaginary) scenario I used to stress-test the design was the classic hug of death ... a link lands on the front page of something like Hacker News and a few hours later a very large number of people have all arrived at once. The typical costs of what happens next break down like this:

What a visitor does/gets	What it touches	What you're charged for
loads a page	the web server	requests and CPU-seconds
searches for a place	a geocoding service	requests
looks at the map	map tiles	bytes sent
subscribes	the database	writes
a forecast every night from then on	the batch job	reads, writes and compute, per subscriber

The first four rows are the one-off spike, and they are where things will visibly fail if the budget gets blown. The last row is the one that matters, because a hug of death in this case is a brief spike in the request path but a permanent step in the batch path. Every subscriber you gain is a database read, a forecast and a write every single night from then on.

One big read beats a million small ones

The subscriptions live in Firestore, Google's serverless document database, which charges per document read and written. The obvious design for the nightly job is to read each subscription, forecast it, and write each result back as a document. At a million subscribers that's a million reads and a million writes every night.

Firestore is cheap per operation, so this doesn't sound like much. A nightly full scan is about $9 a month and writing the results back costs about $27 a month more. It's not a lot, but I would somewhat resent paying hundreds a year to send other people weather alerts when there's a better way.

Instead, the job writes its results as one file in cloud storage (Google Cloud Storage, which charges per object rather than per row). A night's forecasts for a million points come to about 40 MB compressed, written as a single object. That's one billable operation instead of a million. It's also a far better format for the job that really needs it, because those files double as the training data for the next version of the model.

The reads take a bit more thought. The job also writes a snapshot of the subscriptions it used, and the next run starts from that snapshot and only asks Firestore for documents that have changed since. Even on a really busy day that's about 30,000 documents rather than a million, so a 97% reduction, and at those volumes it costs essentially nothing. Materialising the data like this is a useful trick, provided that nothing is ever deleted. A hard delete would be invisible to the batch job, and the user would carry on getting alerts after they'd unsubscribed. Soft deleting (marking the row as "deleted") solves this problem.

It's worth noting that the compute isn't the expensive part either. Both frost models together take about 180 microseconds per point, so a million points is about three minutes of a single CPU core. The whole batch fits on one machine, well inside Cloud Run's free allowance of 180,000 vCPU-seconds a month. In fact the job can't usefully be spread across more machines, because the Met Office publishes whole-UK grids (around 500 MB a run), and every extra machine would just download the same files again.

Don't pay for everyone opening the notification at once

The other spike in this system is self-inflicted. When a frost alert goes out, thousands of people get a notification at roughly the same time, and a good number of them tap it. If the page behind that tap looked up the forecast or the subscription, every notification sent would become a database read at exactly the moment of peak load. So it doesn't. The alert page is built entirely from its own URL: the kind of alert, the place name, the probability and the temperature all travel in the link. Opening it touches no database and no storage at all, which means it costs effectively nothing however many people open it.

There's a security cost to this (anything in a URL can be edited by whoever has it), so every value is checked against a hard-coded fixed list or a valid range, and anything that doesn't parse is dropped rather than displayed. Viewing the place on a map does need a database read, so that's a link the reader has to click rather than something every recipient gets automatically.

Put the heavy bytes where they're cheap

Most of the site is tiny. The pages are rendered on the server and weigh between 851 bytes and 2.5 KB, and a first visit including the stylesheet is about 1.6 KB. Everything works with JavaScript switched off, and a single server instance handled about 575 requests a second (that's about 50 million views a day) in load testing at 20% CPU. A massive front-page traffic spike is well within what one small instance can absorb, and the service scales to zero when nobody's using it.

The map is the exception. The mapping library alone is about 300 KB compressed (two orders of magnitude more than the rest of the site) and map tiles are where traffic really adds up. GCP has quite high cloud egress charges, so as well as making sure the map only loads on the one page that needs it, the tiles come from a single file of UK map data hosted on Cloudflare R2, which doesn't charge for data transfer. The browser then uses range requests to read just the parts of that file it needs. There's no map provider, no API key and no per-tile fee. Thank you Cloudflare!

A brief diversion into geocoding

Searching for places is the one part of the site that relies on a third party, and it turned out to be the part with the nastiest surprise in the terms of service. Most geocoding providers have a generous free tier, but it only covers displaying results temporarily. Storing coordinates you got from them, which is exactly what a subscription is, needs a paid licence. For the provider I started with, that was $5 per 1,000 requests with no free tier, which at a million sign-ups a month is roughly $5,000 a month for a service whose entire batch side costs tens of dollars. Again, not likely I'd ever get that much traffic, but it would be a bad month if I did ... and even a fraction of that would be noticeable.

I looked at self-hosting the open-source alternatives (which needed a server running permanently at $30–70 a month), but in the end I built my own UK place search from open data (Ordnance Survey, ONS and OpenStreetMap), which runs on the same scale-to-zero model as everything else. The free tier covers about 25 busy instance-hours a month, which at measured throughput is several million searches, and the residual cost is a pound or two a month for storage and data transfer.

The geosearch engine is still under active development (inferring intent from ambiguous queries is a surprisingly interesting problem), so that's a story for another post.

Giving an LLM a city to run (sort of)

Mon, 07 Sep 2026 00:00:00 +0000

Frontier models are going into jobs that used to need a purpose-built system. The pitch is attractive: just describe the task and plug in reasoning. This is obviously much easier than developing your own systems and models from scratch and drops the cost of deployment to near zero. But does it really work?

Weirdly, there aren't many people who are keen to run experiments on real-world business processes, so I thought I'd try the next best thing ... giving the model a [sim] city to play with.

I also used reinforcement learning to train a specialist model from scratch to see if a frontier model could beat it.

The bottom line: The frontier model can tell you exactly how to run the city. It talks a really good game. But it fails to execute and can't even follow its own plan.

So, to wildly extrapolate a takeaway for the people who won't read any further than this ... if you can define a task accurately, a specially-trained model is likely to beat a frontier LLM. If you can't define the task accurately and you're happy to settle for mediocre performance, then an LLM will work. Just don't trust what it says it's doing.

The setup

I used Micropolis, an open-source SimCity engine. You get a 32×32 patch of land, a million in funds, and 200 decisions. Each turn you place one thing: a house, a road, a power line, a factory. Your score is the population at the end.

It's a good task for this experiment because nothing pays off at once. A zone is dead ground until a road reaches it and power arrives. Growth compounds, so a decision on turn 10 is worth several on turn 80. You have to sequence, not just choose ... and that requires some form of reasoning.

I trained a specialist model on the simulated city using a tiny convolutional network (about 94K parameters) and a hundred thousand games of practice, played inside a reinforcement learning loop on a laptop GPU. The frontier model (Claude Opus 5) was given tools to interact with the city and no training at all.

For the sake of an independent baseline, I also wrote a dumb heuristic rule ... build a power plant, then alternate road, power, residential and industrial (1:3 ratio) zones until the game ends.

What they built

Here is what the specialist builds. Yes, it's ugly.

Dense usage, roads to everything and power everywhere. It's not pretty but it has found a layout that packs the most houses (and other necessary services) between the fewest roads. Note the lack of fire stations. That will become important later.

Here is the city Claude built, on the same map.

It's a logical layout. But most of these plots never became anything. The specialist's map is full of buildings. This one is full of zoning — flat squares marked R, laid out correctly, connected to road, connected to power, and empty. Lots of fire stations though.

The LLM is a mediocre reasoner

The LLM scored 427 against the specialist's 951. A dumb rule scores 71, so the LLM beats that at least, but is only about half as good as the specialist.

The LLM is not failing at the mechanics. It knows a zone needs a road and it gives it one. Just one building in the whole city ends up unpowered. In fact the LLM follows the rules better than the specialist does:

	valid moves	zones built	people per zone
specialist	90%	80	13.2
frontier model	99.5%	59	7.5

Nearly every move it makes is legal and lands. It builds a little less, and gets half as much out of each one.

The LLM can state the strategy. It can't execute it.

This is the most important finding for me.

Before the LLM saw a map, I asked the LLM to write down its plan. It can do this because it will have come across Micropolis (and Sim City) in its training data. Part of the plan read:

"Mix: near-even Residential/Commercial/Industrial early because industry and commerce create residential demand; after turn ~50 skew residential, which holds the most people."

That is exactly right and is basically what the specialist model built. In Micropolis, people arrive because there is work. Residential demand therefore comes from shops and factories. If you just build houses with no economy, the houses stay empty.

Here is what each model actually built:

	houses	shops	factories
specialist	37	20	27
frontier model	39	8	7

The LLM wrote the rule about jobs and then zoned a dormitory suburb. Its city ended with 327 residents and 44 jobs. The plots stayed empty because there was nothing for anyone to do.

It had the same problem with the concept of returns compounding over time. It said growth compounds and early zones are worth more, but it actually built half as many of them early.

	zones down by turn 50	road tiles per zone
specialist	38	0.9
frontier model	19	1.9

It wasn't great at building roads either. The specialist lines development along both sides of one road and lets neighbouring zones pass power to each other. The model ran long spines and hung less off them. Half its turns went on building plumbing.

The gap is not knowledge. It is the distance between describing a plan and following one for two hundred decisions.

To give the LLM a fair chance, I let it play again, giving it a full record of every move it had made and whether each one worked, and allowed it to deviate from its plan. In effect, allowing it to learn as it played. Its execution went to near-perfect, but its score got worse, dropping from 443 to 427.

This is, of course, how today's models work, not a permanent fact about machines. The next generation may not have these problems. But for now, a specialist trained to execute beats a model that sounds like it knows what it is doing — as long as you can build a good enough model of the world to train it against. This is important because ...

What if we set the city on fire?

A specialist is only a specialist in the conditions it trained for. So I lit twelve fires during each game, at fixed times, in random places. The specialist's training didn't include fires at all and it was awful at dealing with them.

951 to 56. Scorched ground and fires still burning, barely a district left standing. The specialist does not fight them - it doesn't even know what they are, so it goes on placing houses beside buildings that are alight.

In contrast, the LLM scored 168 when the fires were turned on. So it fell too, but it did not collapse, and it beat the specialist easily.

This is the real case for general reasoning in production. It's not that the "reasoning" of today's LLMs is better, it's that LLMs have read about the world. They know what a fire is without being told, so what utterly destroys a specialist model merely costs a lot for an LLM. To put it another way, the failure that hurts you in a live system is the one nobody planned for, and that is where the specialist can't help you.

Sixteen minutes later ...

I took the same broken specialist model and trained it again with the fires switched on. It took just sixteen minutes to converge on one laptop GPU.

56 to 579. Three and a half times the LLM's score.

You can see what it learned. The red buildings marked FD are fire stations. They are everywhere. It builds three times as many as before and cuts police to pay for them. It worked that out for itself from two hundred games of burning cities down.

Importantly, the specialist did not become useless the rest of the time. In normal conditions it still scores 833 — twelve per cent below where it started, for a tenfold gain when things go wrong.

What this means if you are deploying

A frontier model is mediocre, but adaptable and easy to deploy. The LLM scored half the performance of a purpose-built specialist system, but was deployable in about 5 minutes, just prompt-and-go.

Saying the answer is not doing it. The LLM wrote the right strategy and then ignored it. Judge a system on what it does over a long run of decisions, not on how well it explains itself - or risk getting bitten in production.

The LLM advantage is robustness, not capability. The LLM wins on the conditions you did not plan for. That is insurance, but it costs performance on the conditions you did plan for.

Closing a gap once you know about it is quick. Just sixteen minutes in this case. That was the cost of teaching a specialist about fire once fire became a thing. Training (and especially re-training) the model is not expensive.

It's worth noting that the specialist was also thousands of times cheaper to run. It played a full 200-turn game in 0.42 seconds. The LLM took 42 minutes. To save you doing the calculation, that's six thousand times slower, every game, forever. Frontier LLM providers typically make you pay per-token to use them as well.

So the real question is not "model or specialist". It's: can you list the conditions your system will meet in the real world?

If you can, train a specialist for them. It will be better, it will be far cheaper to run, and adding a condition you missed costs almost nothing once you've spotted it.

But if you cannot - if the things that might happen are open-ended and you will find out which ones matter only when they do - then an LLM is mediocre everywhere but won't collapse. It will cost you about half your performance and several thousand times your compute budget. Of course, if performance matters you could always go old-school and use people instead of an LLM. Just a thought.

Micropolis is the GPL-3 open-source release of the original SimCity engine

Applied AI ... Who to call first?

Thu, 20 Aug 2026 00:00:00 +0000

AI is going into all sorts of business processes at the moment, with results that range from transformative to embarrassing. Most of that variation isn't in the technology, it's in the choice of where to point it. Operations Research has spent about eighty years on that question, but it almost always answers it from the inside, using a client's private data and the people who already know the business.

I wanted to know whether you could do it from the outside instead. Every UK company files accounts, those accounts are public and free, and there are a couple of million of them ... so could you work out which firms have the most to gain before you've spoken to any of them?

The bottom line: yes, up to a point. I can rank an industry by how much headroom each firm is carrying, and I can name the real competitors that prove the headroom is reachable. What I can't tell you is whether any particular firm will actually get there - you can only see that from the inside.

The tool for the job here is Data Envelopment Analysis, a linear programme that takes the best performers in a peer group, joins them into a frontier, and measures everyone else by the distance back to that line. It's asking how much better a firm could do, given what its peers already manage.

The idea in one picture

Every firm in a given industry is plotted by what it earns from each resource, with return on capital along the bottom and profit per employee up the side. Up and to the right is better.

The blue line is the frontier, and the firms sitting on it beat everyone else on both measures at once. Everything behind the line is a candidate and the arrow is the gap. Because the frontier is built from real companies you can also blend them, which lets you ask a rather more pointed question ... how well would this firm do if you rebuilt it out of the best parts of its competitors?

Here's the answer for one care home operator, straight out of the report. Florence Care Reading, 81%. Grove Care, 11%. Acorn Lodge, 8%. Those are three real firms blended into a single composite, and that composite holds no more assets and employs no more staff than the operator being measured. The composite just earns more - or it would if it were a real firm.

What you actually get from a filing

Companies House publishes around 10,400 filings a day, tagged and machine-readable, and most of them are a balance sheet and very little else. Employee numbers are almost always in there. Operating profit is tagged less than half the time, which is awkward for a model that exists to explain profit.

The bigger problem is that some of the numbers are simply wrong. As an example, one motor group filed 427,000 employees against £273m of turnover, which would make a regional car dealer a larger employer than Tesco. Its own accounts give it away, because an untagged note further down splits that same headcount into 158 sales, 209 aftersales and 60 administration. The filing software had applied the document's thousands scaling to a figure that was already absolute, and about sixty firms in the panel managed the same trick.

A bit of noise matters more than it sounds, because DEA is deterministic. A firm like that doesn't just misreport itself ... it lands on the frontier, and then every honest competitor in the sector gets measured against a fiction. In total, I reckon about 85% of time this little project took was spent exploring, cleaning, reconciling, sanitising and double-checking the raw data.

The model

The output is operating profit and the inputs are total assets and headcount. That's all. Every extra input you add makes each firm look a little more like a special case, until eventually nothing can be compared with anything. Two inputs, one output ... doesn't really get much simpler and its barely a model.

About a third of these firms report a loss and classical DEA needs positive numbers, so I used the Range Directional Model instead. It works in differences rather than ratios, which lets the losses go in as they are and keeps the question the right way up.

Comparability is handled as a constraint inside the programme rather than as a filter in front of it. Every firm gets scored against its whole sector, but only certain firms are allowed to be its benchmark ... not its own subsidiaries, not firms an order of magnitude smaller, and not firms whose margin is so different it suggests they're really in another business. The report then names every benchmark firm and its weight, which matters because SIC codes are self-declared at incorporation and effectively never corrected.

Some industries have nothing to find

I ran the whole register, which came to 19,010 datapoints over 14,460 companies in 156 sectors, out of a panel of about 62,000. Industries turn out to bunch very differently. Some companies declare more than one SIC code and I've scored them in each industry, which is why there are more datapoints than actual companies.

SIC	Sector	Firms	Median eff.	% at frontier
50200	Sea and coastal freight	40	0.96	50%
47300	Retail sale of automotive fuel	41	0.68	32%
87300	Residential care for the elderly	129	0.59	22%
43310	Plastering	47	0.59	36%
…
56103	Take-away food shops	107	0.12	14%
82990	Other business support	790	0.11	3%
62090	Other IT services	359	0.10	6%
90010	Performing arts	80	0.08	11%

Sea and coastal freight sits at 0.96 with half its firms already on the frontier, I guess because everyone owns much the same ships and runs them much the same way. It's an efficient industry and there's nothing to find there, which is worth knowing before you spend a week finding it. Performing arts sits at 0.08 and fans out across the whole range.

Two kinds of gap

The distance back to the frontier splits into two halves, and they want completely different conversations.

The first half is slack, which is just extra resource the benchmark doesn't need in order to perform. One furniture retailer in the panel makes £608k of operating profit on £11.6m of assets, and comparable retailers make that same £608k on ~£5m of assets. Its staffing is fine and the whole excess is sitting on the balance sheet, so that's a conversation with a finance director about capital and nobody needs to go anywhere near the shop floor.

The second half is the radial gap, which is execution, and it's the more interesting of the two from my point of view. One residential care operator holds £14m of assets and employs 110 staff, and so do its peers ... all seventy firms in its benchmark are built the same way. It sits at the 19th percentile with no surplus to shed anywhere, so it has the same resources as its competitors and less profit to show for them.

Whatever explains that must be in the running of the place, which means occupancy, fee mix, rostering and the cost of covering a night shift - or whatever the constraints in care homes are. Those are scheduling and allocation problems, which are precisely what optimisation, and increasingly AI, can help with, once they are sitting inside the process. A large radial gap in an otherwise homogeneous sector is a strong signal we can see from the outside that there's something worth improving on the inside.

What it won't tell you

DEA compares two firms standing still, so it measures the gap rather than the crossing. Some of that gap never reaches the accounts at all ... a theatre company behind the frontier might be badly run, or it might just be short of the one director everybody wants to work with, and talent doesn't get tagged in iXBRL. Headroom means unrealised performance, but it doesn't mean it's realisable.

I also don't know if it works in practice, because I haven't called as bunch of high-headroom companies, persuaded them to let me do "AI" on them and observed the results. I guess that part will have to wait.

Search method shootout

Mon, 10 Aug 2026 00:00:00 +0000

Second of three posts. Part 1 covered building an honest benchmark honest enough to trust. This post is the reference companion: each retrieval method in turn - how it works, where it came from, what it typically scores in the literature, how I implemented it, and how it did on our data. Fair warning ... it's quite a boring post.

A note on the numbers: Results quoted below are MRR@10 on the benchmark set of UK Parliamentary Questions. MRR is Mean Reciprocal Rank … 0.5 means the first genuinely-correct answer typically sits around position two of the 10 items retrieved. 0.8 means it's usually right at the top. For comparison with “our results” I’m also quoting typical results for the various methods on standard datasets … MS MARCO passage dev in this case.

1. BM25 (keyword search)

How it works: BM25 scores a document by how often the query's words appear in it, taking account of two extra factors: saturation (the tenth occurrence of a word adds far less than the second) and length normalisation (a match in a short document counts for more than one buried in a long document).

History: Okapi BM25 came out of Robertson and Walker's work at City University London in the mid-1990s (building on the probabilistic relevance framework of the 1970s). It is still the default ranking function in Lucene, Elasticsearch and OpenSearch - which means it quietly powers a very large fraction of the world's search boxes.

Typical results: Around 0.18–0.19 MRR@10 on MS MARCO.

Implementation notes. I used bm25s, a fast pure-Python implementation (100–500× quicker than rank_bm25, near-Lucene quality). It runs on CPU, the index is a few hundred megabytes, and lookups are sub-millisecond. BM25 is seriously quick, which is exactly why it's the baseline to beat.

Our result: 0.56 MRR@10 [0.54–0.57]. Notably stronger than its MS MARCO reputation, precisely because PQ answers share wording with their questions. A genuinely strong, nearly-free baseline. This is honestly a good enough result for production use on our data, but I suspect we can do better.

2. RM3 query expansion (pseudo-relevance feedback)

How it works. Run the search once, assume the top handful of results are relevant, harvest the words that distinguish them from the rest, add those to the query, and search again. The idea is to bridge vocabulary gaps - if the answer says "clinician" and you asked about "doctors", expansion can pull it in.

History. Relevance models are Lavrenko and Croft (2001); RM3 is the widely-used interpolated variant. Pseudo-relevance feedback is one of the oldest tricks in information retrieval and adds a lot for classic news/legal collections.

Typical results. While it helps on those older collections, it is well documented to hurt on MS MARCO passage, where judgements are sparse and a single wrong assumption in the first pass drags the expanded query off-topic ("query drift").

Implementation notes. A standard PRF pass layered on top of our BM25 index. This needs a second retrieval round, so roughly double the query cost, still CPU-only and still very quick.

Our result: fair 0.50 [0.48–0.51] below plain BM25. On this data the expansion added more drift than signal, exactly matching the MS MARCO pattern. A useful reminder that fancier is not a synonym for "better". This is why we benchmark.

3. SPLADE (learned sparse retrieval)

How it works. SPLADE keeps keyword search's cheap, inspectable inverted index but replaces hand-tuned term statistics with a language model. A model predicts, for every word in its vocabulary, how strongly a document is "about" that word — including words the document never uses (so a passage about pensions can score well for "retirement" without actually containing that word). The result is a sparse, weighted, expanded bag of words that can be searched very efficiently.

History. SPLADE is Formal, Piwowarski and Clinchant at Naver Labs Europe (2021), refined through SPLADE++ to SPLADE-v3 (2024).

Typical results. SPLADE-v3 reaches 0.42 MRR@10 on MS MARCO.

Implementation notes. I used the naver/splade-v3 model from Huggingface. SPLADE is (should be) the best of both worlds … encoding is compute-intensive and happens on GPU; but search is sparse dot-product on CPU.

Our result: 0.58 MRR@10 [0.57–0.60] — statistically tied with BM25 (the intervals overlap). SPLADE's learned expansion is its edge on MS MARCO, but on a corpus where questions already echo their answers, BM25 was getting most of that benefit for free, so the expansion had little left to add. At this point I was starting to question whether this was going anywhere.

4. Dense semantic search (off the shelf model)

How it works. A bi-encoder embeds the question and every candidate answer into the same high-dimensional vector space, trained so that a question lands near its answer. Retrieval is then nearest-neighbour search by cosine similarity, so a question and its answer can match on meaning even with no shared words at all. This is what most people today mean by search using "AI embeddings".

History. DPR (Karpukhin et al., 2020) made dense retrieval competitive; contrastive pre-training then made it general-purpose: Contriever (Izacard et al., 2021), E5 (2022), BGE (2023), and the current crop of instruction-tuned embedding models.

Typical results. Strong bi-encoders land around 0.33–0.40 MRR@10 on MS MARCO

Implementation notes. I used jina-embeddings-v5-text-small — a 0.6B parameter model on a Qwen3 backbone with a retrieval adapter. This was the strongest sub-1B open-weight model on the MTEB multilingual leaderboard at the time, which keeps the fine-tune (next section) affordable. Using embeddings at scale needs an index - I used FAISS flat (exact) search rather than an approximate index. This isn’t necessarily needed for production, but gets rid of a layer of noise in the results.

Our result: 0.74 MRR@10 [0.72–0.75]. Boom. A clear step above every lexical method, even untuned. And because it was being tested on questions published after the model's training cut-off (Part 1's memorisation control), we can be confident that advantage comes from skill, not regurgitated training data.

5. Dense semantic search (fine-tuned model)

How it works. Exactly the model above, given a training pass on our question→answer pairs so the vector space is shaped to this domain. I used use a contrastive objective (MultipleNegativesRankingLoss: pull each question toward its answer, push it away from every other answer in the batch) with specifically chosen hard negatives.

History. In-domain fine-tuning of bi-encoders is standard practice and consistently the highest-return step available — yet it's the one teams most often skip in favour of reaching for a flashier method or getting an existing model into production quickly.

Typical results. Fine-tuning a decent base on in-domain pairs commonly adds 0.05–0.15 MRR over off-the-shelf, depending on how far the domain sits from the model's pre-training.

Implementation notes. a LoRA adapter trained on the frozen backbone, on a rented A100 GPU for a few hours. Crucially the fine-tuning used the train split only and the evaluation was done on the untouched later slice.

Our result: 0.81 MRR@10 [0.80–0.83]. +0.07 over the same model off-the-shelf. A decent jump for a cost you basically only have to pay once when you fine-tune the model. After that it’s the same in production as using the off the shelf version.

6. Hybrid (reciprocal rank fusion)

How it works. Run two retrievers (here: fine-tuned dense and BM25) and merge their ranked lists by Reciprocal Rank Fusion - each document scores the sum of 1/(k + its rank) across the lists, so a document ranked highly by either method floats up. The theory: lexical and semantic search fail on different questions, so their union should be stronger than either.

History. RRF is Cormack, Clarke and Büttcher (2009). It's everywhere in production because it's parameter-light, needs no score calibration between systems, and rarely makes things worse.

Typical results. Fusing complementary rankers usually buys a small, consistent gain — and is often the easy last few points in a competition system.

Implementation notes. Weighted RRF, with the dense/BM25 weight swept on the dev slice and then applied to test exactly once. In production running this costs the sum of both pipelines plus a trivial fusion step.

Our result: 0.82 MRR@10 [0.81–0.84]. Statistically indistinguishable from fine-tuned dense alone. On this data, once the embedding model was tuned it had already captured what BM25 would have contributed, so the hybrid added nothing measurable. Hybrids often earn their keep; here the fine-tune got there first. Worth noting that in some production use-cases the keyword bit carries more weight: if you’re searching for a person’s name, for instance.

7. A frontier API model: (Gemini)

How it works. Instead of hosting an embedding model, you call one. Google's gemini-embedding-001 is a frontier-scale embedding model sitting behind an API: send text, get a 3072-dimensional vector back. It sits at or near the top of the public MTEB leaderboard, and it uses Matryoshka dimensions, which means you can truncate the 3072-vector to 1024 (or 768) with almost no quality loss (I measured 0.798 at 1024 vs 0.802 at full width), so it drops straight into a 1024-dim index.

History. The lineage runs from Google's text-embedding models to the Gemini-family embeddings (generally available in early 2026). Matryoshka Representation Learning (Kusupati et al., 2022) is the trick that lets one model serve many dimensionalities from a single vector.

Typical results. Frontier API embedders (Gemini, OpenAI text-embedding-3-large, Voyage, Cohere) top the public leaderboards, above every open sub-1B model.

Implementation notes. I called it on Vertex AI, batch endpoint, at $0.075 per million tokens (batch). Encoding the entire benchmark (271k documents plus the query sets, ~50M tokens) cost under $4. Two things you cannot do, though, and both matter: you cannot fine-tune it (it's a black box), and every document you encode leaves your infrastructure for a US API.

The fine-tuning thing took me down a bit of a dead end. Since the Gemini model is frozen and closed, one option is a small projection head: take Gemini's 3072-vector and train a little network on our question→answer pairs to reshape it toward our domain. It did nothing - 0.803 with the head versus 0.802 raw. But this could be an interesting technique for “translating” between embedding models … project them all into the same embedding space using a trained adapter and you can swap the backend model. Normally you’re stuck with the one you started with, which creates risk if the model is discontinued etc.

Our result: 0.80 MRR@10 [0.79–0.81]. Read that against the fine-tuned jina model's 0.81 [0.80–0.83]: the confidence intervals overlap almost entirely. A frontier general model, out of the box, matches our domain-fine-tuned specialist. Depending on where you sit that's either deflating or liberating.

The scoreboard

Method	Fair MRR@10	95% CI
Hybrid (tuned dense + BM25)	0.82	[0.81–0.84]
Dense, fine-tuned (jina)	0.81	[0.80–0.83]
Frontier API (Gemini)	0.80	[0.79–0.81]
Dense, off-the-shelf	0.74	[0.72–0.75]
SPLADE (learned sparse)	0.58	[0.57–0.60]
BM25 (keyword)	0.56	[0.54–0.57]
RM3 (query expansion)	0.50	[0.48–0.51]

So, we have two methods that are basically tied at ~0.8 (ignoring hybrid, which added nothing over fine-tuned). Which one to pick depends on what the production workload looks like. If you’ve got enough documents coming in through the day to keep a small GPU spinning then the fine-tuned local model works out pretty cheap and you get to own the model itself with no upstream dependencies … but you do have to manage infrastructure. If your workload is lumpy, or you just want an easy life, then the API route is likely to be the clear winner.

Careful design beats clever tools

Sat, 01 Aug 2026 00:00:00 +0000

This is the first in a series of posts about improving search results using a real-world scenario. Inevitably, as it’s written in 2026, it’s also a bit about the process of using LLMs. This part covers the work that underpins the rest - getting a solid benchmark. Later parts will include a comparison of various information retrieval methods and how to verify the system remains accurate in production.

Let’s start with a little background. This series is about implementing a really good (and really practical) information retrieval system from the ground up. I’ve based it in the context of the political monitoring industry, because that’s what I know best. In the monitoring world, really good search (retrieval) basically is the product, so it literally pays to be good at it.

There are lots of ways to search for and retrieve information, so how can we know which among them is really good? One approach is vibe checking - just look at some results and get a sense of how good they are. In a way this is the only real test, because ultimately it’s the user’s sense of what’s good that matters. But this approach is hard to optimise against and it doesn’t scale well. So we need a benchmark.

This post is all about constructing that benchmark. There are a lot of posts online that put search methods head to head to see which is the best (spoiler alert: it’s usually a hybrid approach), but they tend to use established datasets, which mean they have a pre-ordained “right” answer. That’s not an option in the real world on a greenfield project, so we’re going to spend some time (much more time than I expected when I started this) creating a benchmark we can use to judge how effective various retrieval methods are.

For those who just want the headline: even with a large, clean, public dataset that was almost ideal for the task, getting to a trustworthy scoreboard cost me about £50 of cloud compute, several rounds of human grading over a few days, and a dozen analytical reruns. Modern agentic AI made all of this dramatically faster and easier to build. When I started this project, I did wonder whether a frontier LLM would be a good enough judge to replace the benchmark stage altogether. It was not. If anything, using agentic LLMs made having a reliable, domain-relevant, benchmark even more important.

The test bed

The UK Parliament has the concept of Parliamentary Questions: MPs ask government ministers questions and the official answers are published. It's a public dataset, it's large (hundreds of thousands of question-answer pairs), and it comes in three flavours that conveniently span a difficulty range: written questions (formal, often sharing wording with their answers), oral questions (scripted questions and answers that are read aloud), and supplementary questions (the conversational and more ad-hoc follow-ups, where question and answer can look nothing alike). At a high level, our task is to search a large pool of possible answers and find the one that was actually used for a given question.

The test set

There are some general [anti]patterns that apply to experiments in general and machine learning/information-retrieval projects in particular …

Leakage. Official answers tend to restate the question, which makes plain keyword search look really good … but is it finding the answer or just echoing the question's words back? This is usually seen as a type of overfitting, where the technique is designed (or trained) to work really well on a specific set of data, but it’s really using shortcuts within that data to boost performance. But in our particular case, I’m calling this a feature not a bug. Some of the strongest clues about which answer goes with which question come from matching specific terms between the two. In many cases the name of the constituency is the only thing that changes between otherwise identical answers.

Tuning on the test. A common silent mistake: adjust settings while watching the test score and you're no longer measuring quality, you're fitting to the test's quirks. The more complex a method, the more knobs it will have you can adjust and the more it will benefit from being tuned to the test set. The solution is simple (but quite hard to be disciplined about): only ever optimise on the training set, then use the test set to see whether it generalises. In this case, I also split the training and test sets by time: an earlier slice to tune on, a later slice to test on, after every setting was frozen. This mimics what happens in real life. You develop on yesterday’s data, in production you see tomorrow’s (and you hope they are similar).

Memorisation. Modern embedding models are trained on huge amounts of the public web - almost certainly including our publicly available source. This means the model itself may have already seen our test set date and would perform well until we started using data it hadn’t seen. To avoid this, I chose a test that was not only strictly after the cutoff for our training set, but also after the training cutoff date for the base model itself. I also cross-checked results with keyword methods, retraining from scratch each time to avoid pollution.

The "right answer". The nice thing about this dataset (and the main reason I chose it) is that each question has exactly one official answer that we’re trying to find. But in a haystack of hundreds of thousands of answers, there may be other documents that genuinely answer the question too and a search method should not be punished for finding those answers. Which means I needed a way to find those answers too, so I could judge the method properly. The difficulty here is that finding those answers is exactly the problem we’re trying to solve in the first place. Getting around that turned out to be (by far) the most time consuming part of setting up the benchmark.

What is the right answer anyway?

To know whether the retrieved result genuinely answers a question, someone has to look at it and make an assessment. I didn’t love the idea of hand-verifying thousands of examples from the test set and these days using an LLM as a judge is often seen as a valid solution. It’s very easy to just ask the LLM “Is this a valid answer to the following question …” and accept the result. I’ve seen people do it. But I’m trying to keep at least a thin veneer of scientific method to all this, so I did a quick test. The result was disastrous and I disagreed with the LLM’s judgment about half the time. But the nice thing about LLMs is that they can be calibrated.

The solution was to get a long list of possible answers for a selection of questions and hand-grade them myself. At the risk of skipping ahead to part 2, running all of the retrieval methods across the test set generated a set of answers which were a) not the official “correct” answer but b) were selected by at least two different methods. I hand-graded 300 of these, which was a boring and tedious process but much less boring and tedious than doing tens of thousands would have been. A few rounds of trying different models and prompt-tweaking and the LLM was agreeing with me about 80% of the time. Which perhaps doesn’t seem great but …

I also re-checked 100 of the answers a few days later and found that I only agreed with myself about 80% of the time too. It turns out there are loads of answers that are basically a coin-flip as to whether they answer the question or not. In effect, this means the LLM agrees with me as often as I agree with myself, which is as good a result as we’re ever going to get. The LLM then processed 76,000 judgements for about the price of a London takeaway and now I’ve got a large and validated test set that I can use to judge the performance of retrieval techniques.

There are two key points here. First, an AI judge is a force multiplier, not an oracle. It’s good at heavy lifting, but you have to keep a close eye on it and verify what it’s doing (more on that in another post). Second, if your benchmark comes from human judgment, it’s important to cross-check it to establish an upper limit on what’s actually possible. If your benchmark comes from trusting an LLM without checking, there's a good chance you're doing it wrong.

Where the AI helped, and where a human was needed

This project was built with an agentic AI assistant doing a great deal of the hands-on work, so it's worth being precise about the division of labour, not least because I learned some lessons there.

The AI was genuinely strong at: writing and rewiring the data pipeline and evaluation harness; running dozens of analyses on request; scaling the relevance judgement once it was calibrated; and diagnosing anomalies when it was pointed at them ("this number looks wrong, find out why"). Ultimately, it compressed weeks of coding into days and much of that was done in the background while I got on with other things.

A human was required for: the ground-truth grading (and re-grading) that set the upper limit.I was also needed to keep the LLM going when it would otherwise have stopped.Right now (2026) there are definitely practical limits on the ability of agentic LLMs to reason, particularly over longer horizons and larger problem spaces. The LLM was just as quick to decide that we’d reached a dead end and should abandon the project as it was to take a result at face value and declare success. Equally confidently in both cases.

Unhelpful filters: A large number (about 5% of the test set) of written questions are follow-ups of the form "pursuant to the answer of 6 January to Question 101310, was the representation written or oral?" Claude suggested (and to be fair, my intuition agreed) that, because these reference an unseen answer, they can’t be understood in isolation and therefore should be filtered out. Claude built a filter, tested it and declared the whole thing to be a massive success. Then I actually looked at some examples. Most were perfectly understandable. The question restates its topic and the official answer addresses it, at least to some extent. I suspect this lands right at the edge of what today’s LLMs can reason about … the theory was sound but didn’t actually hold true in reality, despite the LLM convincing itself that it did.

Claude is really bad at: Estimating. Many, many examples of this but the most obvious are around how much time it takes to write code. What is predicted to be “a week’s work” is often done in about 10 minutes. Some more specific examples: a batching optimisation would give a 4-7x speed-up, but when measured it actually gave none (the workload was memory-bound, not compute-bound). On one occasion Claude silently revised an estimate to match observations .. an old-school word-vector baseline was scoring suspiciously low and Claude was happy to report that it was in line with expectations. When prompted to explore, it was clear there was a bug in the pipeline.

The main takeaway for me is to always inspect the data and never trust an assertion that matters. Validate everything by looking at the data yourself.

The Grug Brain Developer

Thu, 31 Oct 2024 00:00:00 +0000

It's quite rare that I read something on the internet and agree with every word of it. grugbrain.dev is an exception. The original is written in a "caveman" style which I quite like, but it makes the content a little harder to digest, so (with a little help from ChatGPT) I've translated and summarised it here. If you like this version, check out the original ... which also has pictures.

The Practical Developer's Guide

Introduction

This is a collection of practical insights on software development from a "Grug Brained" developer: someone who's been in the field for years, has seen a lot, and learned from mistakes (often the hard way).

Complexity: The True Enemy

The biggest challenge is complexity. You can’t always see complexity as it creeps into a codebase, but you can feel its impact as small changes break everything. Avoid complexity by simplifying wherever possible and understanding how each addition could make the system more fragile.

Saying No

The best weapon against complexity is “No.” Saying no to unnecessary features, abstractions, or time-wasting requirements is vital for keeping things simple. While saying yes might be good for your career, saying no keeps your project manageable, understandable, and useful in the long term.

When to Compromise

When “No” isn’t an option, look for the 80/20 solution: build 80% of the feature with 20% of the effort. It might not have every bell and whistle, but it’ll deliver the value and help keep complexity at bay. Managers often won’t even notice if you skip the last 20% of complexity.

Factoring Your Code

When building out a system, don’t over-design it early on. Let natural boundaries emerge as you work, where functionality can be isolated with minimal dependencies on the rest of the system. Waiting for these "cut points" to appear organically will save time and reduce the risk of over-abstracting.

Testing

Testing is crucial, but balance is key. Start with basic unit tests, but don’t overdo it in early stages. Focus more on integration tests as the system grows, and build a few key end-to-end tests that cover the main workflows. Avoid mock-heavy testing unless absolutely necessary.

Agile

Agile methods are fine, but don’t take them too seriously. Agile can be a useful framework, but the real value in development comes from prototyping, good tooling, and hiring talented developers. Agile isn’t a magic solution to every problem.

Refactoring

Refactoring is often necessary but should be done carefully. Small, incremental changes are safer than large, sweeping ones. Keep end-to-end tests updated to catch breaking changes early. And remember, too much abstraction can make refactoring a nightmare.

Chesterton's fence (Respecting Legacy Code)

There’s often a reason why old code is structured a certain way, even if it’s messy. Don’t rip out old code unless you fully understand its purpose. Sometimes code is ugly for a reason, and changing it can introduce unintended bugs.

Microservices

Getting the structure of code right is a hard problem. Microservices require this problem to be solved correctly and then add a network call. (Personal note: the ability to scale some services independtly of others can be very helpful, although it comes at the cost of complexity).

Tools and Debugging

Learning and mastering your development tools is a big productivity boost. IDEs, debuggers, and code completion can speed up development and help you troubleshoot faster. Take the time to understand these tools deeply - indcluding conditional breakpoints and stack travsersal.

Type Systems

Type systems are most useful for autocompleting and catching simple errors, not for building hyper-abstract models. While strong typing has its place, avoid overusing generics and complex abstractions—they can make code hard to work with and maintain.

Expression Complexity

Avoid cramming too much logic into one line. Break up complex conditions into clear, named variables to improve readability and debugging. Although it may mean more lines of code, it’s easier to understand and maintain.

DRY (Don't Repeat Yourself)

While avoiding repeated code is generally good, DRY can go too far. Sometimes, duplicating a small amount of simple code is easier to understand than creating overly complicated abstractions.

Separation of Concerns (SoC)

SoC is a popular principle, but rigidly separating concerns can sometimes create unnecessary complexity. In contrast, the Locality of Behaviour principle says that "The behaviour of a unit of code should be as obvious as possible by looking only at that unit of code". This flies in the face of classical thinking, but makes code easier to read and maintain in the real world. The implementation of logic can still be abstracted away, provided the invocation of that logic is obvious.

Closures

Closures are helpful, especially for abstracting operations over collections, but they can become complex quickly. Use them sparingly, as they can lead to convoluted code that’s hard to debug.

Logging

Logging is essential, especially for production systems. Use structured logs and track request IDs for distributed systems. Well-organised logging can be a lifesaver in debugging but is often overlooked.

Concurrency

Concurrency is hard, so lean toward simpler concurrency models, like stateless request handlers or job queues, which minimize the chance of complex interactions and bugs.

Optimization

Premature optimization wastes time. Profile first: always have data to back up any optimization efforts, focusing on real bottlenecks rather than theoretical improvements.

APIs

Good APIs should be easy to understand and not require deep knowledge of the underlying implementation. Create intuitive methods for simple cases and more complex interfaces for advanced cases where they are needed.

Parsing

Recursive descent parsing is usually simpler and more maintainable than using parser generators. While it might not be the latest trend, it works well for most real-world parsing needs.

The Visitor Pattern

Avoid.

Front-End Development

Avoid unnecessarily splitting frontend and backend, especially for smaller projects. Heavy front-end frameworks can add unneeded complexity, so keep things simple with minimal JavaScript where possible.

Fads

Be cautious with trends, especially in frontend development. New tools and patterns often recycle old ideas and may add more complexity than they’re worth.

Fear of Looking Dumb (FOLD)

FOLD stops people from admitting when things are confusing or complex. Senior developers should lead by example, asking questions openly. Reducing FOLD creates a healthier team environment and better code.

Imposter Syndrome

Most developers feel imposter syndrome at some point. Remember, if you’re learning and improving, you’re on the right path. The feeling of not knowing what you’re doing is more common than you think.

Documentation

Tue, 08 Oct 2024 00:00:00 +0000

Documentation - so often an afterthought in the development process and so often seen as users accepting defeat. There's a lot of subjectivity in this space, not least because of the variation between products and users. Some can legitimately be expected to RTFM before using a product - others should be able to dive and and work it out. There are also legal requirements relevant to some products.

Over all, my view is that, in an ideal world ...

1) high level concepts and the purpose of the product should be explained clearly up-front 2) reference material should be comprehensive, and easy to access from the product itself 3) it should be possible to get meaningful results from the product without having to read the docs at all

Of course, we're not in an ideal world so I've gathered together a few more high-level thoughts that might be useful for anyone trying to write (or re-write) technical documentation.

DDD?

We're familiar with Test Driven Development and getting the User Experience right early in a product's development ... documentation is almost certainly going to be part of that experience. Sadly, we often leave writing the docs until after the product is ready for release. After all, there's no point writing documentation for something that could change during the development process is there?

But what if we flipped that on its head? If we assume that new users will need to consult the documentation early on, then why not write the docs first (or at least the outline), to make sure we're delivering an excellent user experience? Reference material and detailed screenshots can probably wait - but from a UX point of view, what do we want users to experience from their first contact with the documentation ... and how does the product need to work to deliver that?

Start with why?

A common pattern with documentation for software libraries is to include a short example on the first page of the documentation - and often on the first page of the website. It's a good example of the "show, don't tell" communication principle. This helps address an issue with a lot of open source software products, which do a really poor job of explaining why the project exists ... what is the problem they are trying to solve? If I have to learn how to use the product before I can understand what it does (and therefore whether I need it) I'm very unlikely to bother.

Including a short example early on does a few different things. It gives context to a potential user. At the most basic level, if I don't recognise the kind of inputs and outputs the example is dealing with, this probably isn't something I need. It also helps me compare to existing solutions ... does it look more flexible, does it have more features, is it easier to use? Finally, it gives a sense of what using the product will be like - and will it fit with my existing code base? This last point is really important for developer experience and is most easily conveyed with a few short code examples.

Readability and cognitive load

People generally use a product because they want the benefits that product brings. Good documentation is an important, but indirect, part of making those benefits happen - the more time a user has to spend in the documentation, the less time they are spending using the product and the longer it will take for the benefits to arrive.

Make it easy to use the docs and you'll make it easier to use the product. A big part of this (as with code and really any written material) is readability, consistency and reducing cognitive load. These things all help users to navigate the documentation, find what they are looking for and understand it. My advice is to take a pretty rigid approach here - particularly in relation to cognitive load. Don't explain high level concepts in reference material (but linking to them is a great idea) and don't get bogged down in covering every possible parameter when writing a how-to guide. Users generally want to use the docs for as little time as possible before getting back to using the product ... as a documentation author, your job is to make that as easy for them as possible.

Diátaxis

If you're the sort of person who reads a lot of documentation (this applies to most programmers) then you may have noticed a trend for organising documentation into categories such as how-to guides, tutorials, concepts and reference material. This is a very popular structure and has been formalised into a framework called Diátaxis.

This is well worth looking into ... it's essentially a set of guidelines for writing the right kind of documentation for a particular use case. It splits concepts along two axes ... action-cognition and acquisition-application. These in turn lead to the categories listed above. Once you're familiar with the concepts, you'll start to see them in all sorts of documentation.

It's an excellent idea and a very accessible website, full of useful information for anyone involved in writing technical documentation.

Self-documenting code (and products)

It is possible (in theory) to write code that is self-documenting, by using descriptive names and clear structures. The idea is that the code is so easy to follow that additional documentation is unnecessary. This works up to a point, but even very clearly written code can't explain why something is being done a particular way. Comments are very useful for explaining why and if they written in a structured way, can be pulled together and formatted by scripts to make documentation. Typically this approach is used for reference material, rather than detailed conceptual guides. With automated checks in place, it becomes very easy to make sure that your code library is 100% documented.

To be honest, I'm not a huge fan of what tends to come out of these systems. This kind of documentation is great for code inspection tools in modern editors, which can save a lot of time having to flick back and forth between files, but it usually doesn't add much more value than that.

Writing great documentation (including reference material) takes time and effort - there are no short cuts. If you've got great reference documentation, putting the relevant sections in the right place in the code is wonderful and makes it immediately available - but you need to have great docs in the first place. Too often, what actually gets generated is a basic description of a function, listing it's parameters and return type. Using strong typing allows modern editors to do the same thing, without giving the product manager false hope that 100% documentation coverage has been achieved.

Success criteria

As with code, there is always going to a subjective element to assessing the quality of documentation. There are some obvious metrics that can be applied, such as feature/function coverage and word count, but these don't tell us much about whether the documentation is useful.

However, when docs are served over the web (which these days is pretty much all the time) it becomes possible to track their use to identify areas of the product that are under-documented, or where the content isn't clear enough.

It also becomes possible to track use of the docs over time. Typically, a user will consult the documentation more when they first start using a product and less over time. Good documentation should help the user become proficient (and therefore consult the docs less frequently) quicker than bad documentation. There is also likely to be a shift over time from accessing high-level concepts to low-level reference material. If the curve looks different for specific bits of the product, it might be worth looking again at the docs. It might also be worth looking again at the product, but that's a different issue.

Having success metrics in mind makes it possible to apply analytics techniques such as A/B testing to versions of the docs in order to improve them over time. The key point is to define what those metrics are before you start running experiments and before you start writing.

Code Review

Thu, 11 Jul 2024 00:00:00 +0000

I'll start with a definition in case there's anyone reading this who isn't familiar with professional coding practices. A code review is a bit like a peer review in academia. One or more people check code that's been written by someone else, with the aim of ensuring the code is of good quality.

In theory, it's a simple enough process ... person A writes some code and sends it to person B, who reads the code and makes comments/suggestions as to how it can be improved - or asks questions if something is unclear. In practice, there's a lot of devil in the detail and a number of things need to be in place before code reviews are effective.

Much of what follows has been distilled from elsewhere ... mostly here.

Standards and expectations

The person writing the code and the person reviewing it need to have a shared understanding of what good code looks like. A common way of achieving this is with a style guide or coding manual. This sets out the preferred way of doing things, from stylistic issues like spaces-vs-tabs through to security things such as banned packages. Good style guides evolve over time, but it's often helpful to start somewhere such as here. Style guides often specify that certain tools should be used, such as linters, compiler flags etc ... a common configuration for these tools is also then required.

As well as code stuff, it is also helpful to have standards and expectations for things like tests (what types of test, what is the acceptable minimum level of code coverage), the size and content of pull requests (what is mandatory, what is optional) and the circumstances in which the code review process can be skipped (ideally never, but in small teams this isn't always realistic).

Approach and tone

A lot has been written about how to conduct a good code review and practices vary between organisations. In my view, it helps to remember that you are reviewing something that a real person has written - and that a real person will read your review - but you are reviewing the code, not the person who wrote it.

Overall, be kind and constructive. Phrasing something as a question can soften the impact considerably. Take time to explain why you think something and, if possible, refer to a common standard. If there is no agreed standard, this might be a good opportunity to update the style guide. If you're being nitpicky, then say that explicitly ... and it's OK for people to do something differently if a) their way isn't objectively worse or b) it is worse, but it makes very little difference.

Objective

The high level objective of the code review process is to make the whole codebase better over time. This should be the fundamental question each time code is reviewed ... does this code make the overall codebase better, or worse?

Better is obviously a subjective term - the style guide is the ultimate arbiter. Better is also quite a woolly term ... if code runs faster but is harder to read, is it better? HINT: No, unless there's a very strong business reason that the code needs to run faster and there was no other way of doing it.

Knowing that someone else is going to read the code that you're writing is a powerful motivator to write easily readable code. This benefits not only the code reviewer, but also the next person who has to work on that part of the codebase.

It is also worth noting that, in order to make the codebase better, explanations need to be done in the code not just in the code review notes. If something needs explaining, refactor the code or add comments if you have to - don't rely on the code review notes themselves.

What to look for

Is the purpose of the change clear from the pull request?
Can you understand what the code is doing and why it is necessary?
Could the code have been written more clearly/simply/readably?
Do the comments explain why something is necessary?
Are tests in place that cover every line of changed code?
Are there any TODO, NOTE or similar comments that need to be addressed?
Read every line of the change - check for typos and naming conventions.
Take a step back and check things like titles (yes, I know).

Building a team

Fri, 28 Jun 2024 00:00:00 +0000

A few reflections on the various ways of building development team capacity, from hiring freelancers or an agency to hiring staff. Based on my experiences working in bootstrapped startups and as a freelancer - this is not necessarily good advice for a well-funded high-growth firm!

Freelancers

It is tempting to think that you can package up your projects into neat little parcels and get people from Upwork to carry them out. In my experience, it takes much longer than you'd think to create work packages like this and quite a long time to find and vet qualified candidates. After that, if they get it right first time the freelance option can be worthwhile. But in general, longer term relationships are harder to maintain. I've never met anyone who actually enjoyed managing a remote freelance dev team. You're also left with the problem of not having continuous tech support available - and any bug fixing has to fit around the developers' other commitments.

The exception to this is designers. I've had some very good experiences with freelance designers, perhaps because design work lends itself more readily to a freelance "work packaged" format.

Agencies

I've personally never used an agency - by which I mean a development firm that offers a full-spectrum service from coding to tech support - they are often based in Eastern Europe. I have, however, been hired several times for one-off projects that were taking too long for an agency to complete. I suspect the business model of a lot of these outfits is to bid cheap and become indispensable for as long as possible, in the knowledge that whoever hired them probably doesn't have much in-house expertise. This is not a model that encourages them to be responsive and go the extra mile. Depending on your niche, you may come across cultural/language issues that can be frustrating. This is also a risk with hiring overseas talent directly, but I've found the problems tend to recede the longer someone has worked for a UK company - and that's more likely to be true for people who are direct hires.

I'm sure there are good agencies out there so if you can get a personal recommendation then it's worth exploring ... just be aware that the cheap ones are cheap for a reason. You'll also need to factor in the overhead that comes with a remote contractual relationship. Asking an agency to switch priorities and work on an urgent bug isn't as simple as just leaning over to the next desk.

Using a recruiter

Use a recruitment agency. It will make your life so much easier. Find one that specialises in your industry, the level of experience you're looking for and your tech stack. Their fees will be annoyingly high and yes, they mostly just DM people who fit the profile on LinkedIn but the alternative is having to wade through hundreds (yes seriously) of barely credible CVs submitted by bots in India. It's worth having a chat with a few firms when you're early in the process of making your first hire so you can find someone you want to build a long term relationship with - and who wants to work with you.

Hiring cheap

If you've got time to invest in someone's training, then hiring an inexperienced but talented person can be an excellent move. It has certainly worked well for me in the past. The trick is to find someone with exceptional talent ... they need to become productive quickly enough to offset the cost of the initial period - when they won't be adding much value. An opportunistic approach can be sensible ... if you happen to come across someone who is keen and seems to have suitable talents, then move them into the dev team. But don't do it because you want to save money - if you're not paying market rates they will quickly work that out and leave. Make sure you have a clear career and pay progression plan - and stick to it. You'll need to carve out time in the working week for training - and make sure you've got room in the budget for materials/courses/subscriptions.

If all that seems like a pain (and it can be) then just pay the market rate for the most experienced developer you can afford. Don't try to hire experienced people cheaply - you want your team to feel valued, not commoditised. If you beat them down to less than 100% of the market rate, they'll bring less than 100% of their energy and talent ... and will probably leave or renegotiate as soon as a better offer comes along anyway.

Certification and data protection

Fri, 28 Jun 2024 00:00:00 +0000

Aside from internal documentation (which is very important) there are two main kinds of documentation that require attention from the tech lead in a startuppy environment.

NOTE: This is based on UK experience ... it may not apply in other places. It is also not legal advice and you should not reply on it!

The first kind of documentation is what I'll loosely call "quality certification". These are only likely to be of interest to B2B firms - and even then only if you're selling to serious corporate/enterprise outfits with dedicated purchasing departments. At the top end (and less likely to be relevant for a startup) are the ISO accreditations e.g. ISO 9001 (Quality) and ISO 27001 (Data Security). There are also a bunch of certifications you can get that designed to reassure potential clients that you'll be careful with their data. These include things like "Cyber Essentials" (applicable for most firms) and PCI-DSS (for those handling credit card details).

I'm not a fintech expert, but if you're handling card payments you should work out for yourself which bits of PCI-DSS apply to you and what you should do about it - it looks like a total pain and probably something you should just outsource to stripe.

The other certifications only ever seem to come up in tender/qualification documents that have been written by purchasing teams. Over the years, I've been surprised at how often I've been asked whether my firm has these certifications, how often I've said "no" and how little anyone has cared. My strong impression is that the purchasing team want to be able to say they've asked the question, but the budget holder doesn't really care about the answer. That said, I've always been able to tell a compelling story about the security and resilience measures in place, even without a certificate.

My personal view about these schemes is that they are based on sound principles, but the fact you've got a certificate doesn't mean you've applied those principles well. You may have a comprehensive risk and mitigation strategy written down and refresh it every quarter, but if you've misjudged a risk or repeatedly fail to mitigate it, then the register isn't much use.

Your mileage may vary of course ... and if having certification is important for your clients, then you'll just need to suck it up and get certified. There are lots of firms that can help with the process and provide templates to make it all easier.

Which brings us on to data security and privacy. This is a trickier area, not least because it concerns the law, as opposed to contractual requirements (although contracts in this space are important ... see below). Data security and privacy are worth taking seriously for their own sake ... careless handling of sensitive information has the potential to make life very unpleasant for some people and I don't think that's a responsibility that should be taken lightly.

It is almost certainly prudent to get some legal advice in this area - and get contracts and T&C documents drafted by a professional who understands your business. But if something goes wrong, having a contract that's been written by a lawyer isn't a defence in itself - what that contract says will become vitally important. It therefore pays to understand this topic yourself, at least at a high level. The good news is if you're in the UK is that there is some really good (if lengthy) guidance on the ICO website.

What it basically boils down to is this ... you need to know what data you're holding, why you're holding it, what you plan to do with it and who is responsible for it. You then need to make sure you've got a lawful basis for holding it (i.e. permission) and that you've taken sensible precautions to keep it secure. You need to be particularly careful if you are holding (anywhere on systems under your control) data about people's race, religion, political opinions, trade union membership, health, biometrics, sex life or sexual orientation. See the relevant bits of the ICO website for more information. There are separate requirements that relate to marketing activity, particularly around direct marketing aimed at the public. Anyone that you are holding information about has a right to see that information - and there is a requirement for most organisations to register with the ICO.

The requirements are also a bit different depending on whether you are a "data controller" or a "data processor" - the former is responsible for how the data is used, the latter just processes it under instruction. There's a good chance your firm will be both - a controller of your own information and a processor of your clients' information. If that's the case, then make sure your contract is clear about who is responsible for what - ideally make them responsible for their own information as far as possible. Again, this is particularly important if clients are storing sensitive personal information on your systems - and if you're not set up to secure that kind of data, make sure the contract is clear that they shouldn't store it on your systems!

This can all get quite complicated quite quickly, but a good starting point is the ICO checklist for small organisations. Start there and then speak to a lawyer once you've worked out the sort of questions you want them to answer.

Customers and requirements

Fri, 28 Jun 2024 00:00:00 +0000

Over the years, I have become more and more convinced of the business and technical sense of co-designing new products and features with input from customers (or potential customers). Often known as design sprints, Google Ventures has published detailed guidance on how they carry out the process. It's well worth reading.

Away from the heavily-funded world of silicon valley, many startups can't afford the time or money required to run a 5-day sprint (or sprints) and I often come across a general reluctance to reach out to new customers until there is a product to show.

This is a mistake.

In my experience, people are very happy to give up some of their work time in order to help solve a problem ... provided that problem really exists for them. A week is a big ask, but an hour or two is normally fine. If you find yourself reaching out to potential clients and none of them want to spend a couple of hours developing the solution to a problem with their peers, you should probably ask yourself whether this problem (or even the market) really exists. If you don't believe me, do a little role play in your head ... imagine someone emailed to invite you to a short, high-energy session with a group of your peers/competitors to collectively design the solution to a specific problem you all face. You'd get to network and you'd get to shape a solution to the problem ... that someone else would then build for you. The only reason I can think of for saying no is that the problem isn't big enough to justify the time.

So what can you do in two hours?

As long as the problem is well defined, you can cram whole design sprint (up to the prototyping phase) into two hours. The trick is to plan it very carefully, time everything to the minute and pump up the energy in the room. Do it person - this sort of thing works poorly in a virtual environment. A good number of attendees is between six and twelve. A schedule might look something like this (there are lots of variations depending on the problem/context) ...

Define the problem as people arrive ... get them to brainstorm the relevant pain points on sticky notes before the session even starts properly. This builds energy and it means you can be clear about what the problem is from the very start.
Introduce the session for no more than 5 minutes.
Using large-format posters of existing products/solutions, use sticky dots to make heat maps of what people like or dislike. Collectively agree an "elevator pitch" (or pitches) for what the rest of the session should focus on. (20 minutes)
In smaller groups, brainstorm Insights, Questions and Ideas around the agreed pitch from the previous session. Groups feed back to each other at the end. This reduces the risk of groupthink. During the feedback session, have someone write the key ideas on sticky notes and start grouping them into themes. (20 mins)
After a break, assign people to themes for a couple of rounds of crazy fours (a shorter version of crazy eights) - emphasise the goal of quantity over quality! (10 mins)
Vote on everyone's crazy eights with sticky dots (5 mins)
Pick a couple of solutions and story-board them in more detail for 10 minutes each.
Recap to flush out any themes that have emerged unseen during the session.

With breaks, in two hours you'll have a detailed understanding of the pain points your potential users face, an understanding of why they matter and a ranked list of solutions along with sketched UI and storyboards.

Take that away and produce a click-functional mockup solution within a week using something like powerpoint/figma. Best to use a designer at this stage, because people can't usually see past bad design and you want to avoid too many comments about the colour of the buttons etc. Demo the mockup to the people who came to the workshop (invite the relevant decision maker too) and ask them what they'd pay for it.

Assuming you can make the financials stack up, now you just have to build it, sign up the people from the workshop (who are now very warm leads likely to leave stellar reviews) and market it using the information you gathered about pain points!

It's also worth noting the broader marketing benefits to this approach. You are showing off your brand in the context of deep understanding of the problem and developing solutions, you are being useful to potential clients which builds brand equity and you are leaving them with the impression that you are high-energy problem solvers, thought leaders and conveners. What's not to like?

Plan to throw one away

Fri, 28 Jun 2024 00:00:00 +0000

Fred Brooks, in his 1975 book "The Mythical Man Month" makes a lot of sensible points that are still true today (albeit some of writing is quite gendered and could do with an overhaul). On the subject of estimation, he suggests planning to throw the first version of software away ... on the grounds that you'll probably bin it anyway, so you might as well plan for it. With the caveat that the maxim applies to new or unfamiliar technology rather and routine coding work, I think this is very sensible advice.

It's important to be clear about when in the software life cycle this applies. By it's nature, the startup world brings a lot of uncertainty, so it's tempting to think that a product will need to completely re-engineered during it's lifetime. It is important to get a product into the hands of real customers as quickly has possible, to begin the process of feedback and iteration. But if you understand your customers reasonably well and you've been through a process of even rudimentary co-design with them, it's unlikely that your MVP will be completely wrong and destined for the bin.

In my experience, provided you've had input from a few likely customers and you've listened to them, the first few versions of your product are likely to be evolutions, rather than completely new.

But in the very early stages of technical planning for a new project, when you're making the early architectural decisions, it's unlikely that you will know all of the questions you need to ask - let alone the answers. At this stage, it's highly likely that you will make some choices that turn out to be incompatible on a fundamental level, or that something dismissed as an edge case early on turns out to be central to the way a system needs to be structured. This is the process of discovery that Brooks was getting at ... it's only once you've spent some time working on a new project that you understand it well enough to make the right decisions. There's no shortcut to this and there's no shame in throwing away version 0.1 to start again from scratch ... particularly if you've allowed time in the project plan for it!

Refactor for simplicity

Fri, 28 Jun 2024 00:00:00 +0000

For those that aren't familiar, refactoring is like tidying up your code. It’s the process of restructuring existing code without changing its external behaviour. The goal is to make your code simpler, cleaner, and more readable. This can have a huge impact on the maintainability and scalability of your software. Programmers (the good ones anyway) spend quite a lot of time refactoring their code ... and that's a good thing.

But it's not quite that simple.

There are often competing goals in play. The process of refactoring code often throws up opportunities to make it run faster - which is a good thing and can reduce running costs. But if this comes at the cost of greater complexity or reduced readability it's probably not worth doing.

Let's take the example of caching. Ignoring for a moment that caching is famously one of the hard things in computer science, it's a common way to speed up software, basically by reusing common bits of data rather than fetching or calculating them every time they are needed. This saves time, which means more things can be done in a given period and fewer CPU cycles are needed.

However, in order to implement caching, more code needs to be written and more tests are required to make sure the code is working properly. Testing a caching mechanism can be tricky as there needs to be a way of a) making sure the data is cached properly and b) making sure the code still works when the data isn't cached. This doesn't just apply when the cache is being set up - the next developer who comes along is going to have to understand how the caching works - which is going to take some time. It's probably also a good idea to document the caching mechanism - and the problem it's trying to solve - in case a future developer gets frustrated and deletes it. It might be worth doing this kind of refactoring if the overall cost saving from increased performance outweighs the cost of increased developer time - but it probably isn't and you should do the calculation first!

The other side of this coin is refactoring for readability.

Given the choice, I would much rather be presented with code that is readable and doesn't work, than working code that is hard to understand. This sometimes takes people by surprise, as they assume that the bare-minimum standard for code is that it should work. This is true, for code that makes it into production. Although even then there will always be hidden bugs and edge cases, so maybe it's more accurate to say it's mostly true for production code.

But before code makes it into production, at least two people will (should) work on it - the person that wrote it and the person that reviews it. Reviewing readable code can be a very quick process, but reviewing poorly written code can take a very long time. It's also quicker to find and squash bugs in code that is readable ... and this is time that really matters once the code is in production.

Determining how "readable" code is can be something of a subjective exercise. What makes sense to me may not make sense to you. The most important thing is consistency. Things should be where I expect to find them and formatted in a way that my eye expects to see. In short, follow the style guide.

You have got a style guide, haven't you?

Things should be named sensibly and consistently (noting that naming things is the other hard thing in computer science). They should describe the reality of their purpose, variables should be nouns and functions should be verbs. Get those things right and most code becomes self-documenting. It should always be clear from the code itself what is happening and how - but if it isn't clear why that thing is necessary then it might be worth adding a short comment.

Opinions vary about the correct level of abstraction and how much repetition is acceptable. My view is that it should be possible to understand any individual logical unit of code based only on what fits on one screen of text. Utility functions are great and can reduce complexity, but only if what they are doing is obvious from their name and there are no hidden side effects ... otherwise each one adds more cognitive load as the reader has to dig further and further into the code to understand what's happening.

Certainly, typing the same thing out twice is a good indicator that you should consider an abstraction, but I don't treat it as a hard rule. Again, the most important thing is to be consistent in approach so the person who has to read the code in three months' time (that person could well be you) can make sense of it quickly if they are familiar with the overall codebase.

In summary, when refactoring (and you should refactor often) consider readability first, then overall simplicity, then performance if absolutely necessary.

Testing

Fri, 28 Jun 2024 00:00:00 +0000

Before any code goes into production, a set of automated tests should be written and those tests should all pass. Where appropriate (e.g. front-end changes) there should also be a manual check that things work as expected and look right on a range of screen sizes. Where code interacts with existing code or the data layer, the tests for those should also all pass. If the tests are no longer relevant, they should be rewritten.

So far so simple.

Having tests that pass is one thing, but knowing you've got the right tests is another. I take a somewhat pragmatic but rigid approach to this. The bare minimum level of testing as as follows ...

Aim for at least 95% code coverage. In the real world, there are always weird edge cases that need to be accommodated and are very hard (or slow) to test reliably - it isn't always worth writing tests for these. If you can explain in a one-line comment why a specific section of code doesn't need testing, then it's OK to skip it. But make that section of code as small as possible - a couple of lines is about right.
As a minimum, have test cases for a) the happy path, b) handling the most likely failure path (probably arising from dodgy user data) and c) any bugs that have arisen in production.
Unit test and mock everything that it is sensible to unit test. For instance, anything in utils files and the database access layer. Run these tests whenever the relevant files are changed, for any reason.
End to end test everything. Tests should use a copy of the live database in an environment that is as close to production as possible.
Have tests that enforce a data schema across the whole codebase. These don't need to hit the database directly, but should confirm strict typing ... if my code expects a field to be a datetime and yours expects a string, the test should fail. Run these tests for the whole project when the data schema changes for any reason.
Tests should be quick to run. Don't make me wait more than a minute for the whole suite to run. If it's tricky to achieve this while running end to end tests in a close-to-live environment, then refactor the code or the whole architecture to make it work.
Follow a CI/CD process - but keep it as simple as possible. In a small team, a complex CI/CD process can add unhelpful overhead where a simple code review and approval process would have been fine.

The bottom line is that you're always going to find bugs in production, so there's little point in writing hundreds of tests cases for every scenario you can think of. But do test every happy-path scenario because that means you reduce the risk of putting code into production that breaks something else.

A quick word about load testing ...

Periodically (and whenever significant architectural changes are made) it's worth load testing a close-to-live environment to destruction. There are good automated tools for this - put some limits on your auto-scaler and find out how much traffic your system can handle - and where the bottlenecks are. This is useful for a) those times when you get a massive spike in legitimate traffic and b) estimating your future costs given assumptions about growth and traffic.

When things go wrong

Fri, 28 Jun 2024 00:00:00 +0000

In any business, things sometimes go wrong. You should of course try to prevent this, but in the real world sh*t happens. It's often a stressful time for the technical team and for anyone in a customer facing role. It helps to remember that most people (at least in a B2B setting) have been on both sides of things going wrong and are likely to have some sympathy for you, despite expressing frustration.

If your product/service isn't business-and-time-critical for your customers, they may not be all that bothered about a minor outage, even if it feels like a big deal to you. In any event, be proactive and fairly honest in your dealings with customers, including telling them once the incident has been resolved. Thank them for their patience and provide affected customers with regular updates about mitigation measures. Depending on the incident, you may also be able to use the fact that you've got their attention to deepen your relationship with them ... we're taking the opportunity to review the code and would value their views about XYZ features etc.

If you can't turn an incident into an opportunity, then make a honest assessment of how much damage has been done to your reputation in the eyes of any affected customers. Consider how important (financially or in other ways) those customers are to your business and consider putting your energy into maintaining good relationships rather than chasing after those that are irreparably damaged.

Behind the scenes, it really helps to have a clear and simple process in place for dealing with incidents (and things which might be incidents, but you're not sure yet). Don't make it too long or too complicated ... just cover what the escalation mechanism is, how and when to communicate internally and how to make decisions - including deciding when and what to say externally.

The best time to think about these things is before they happen ... ask yourself what information you'll need in order to make useful decisions, who should be responsible for gathering that information and who is responsible for making decisions as the incident unfolds. The second best time to think about these things is after an incident has just occurred - this is when things are fresh in your mind and you can learn useful lessons for next time.

Typically, a standing plan for incident response might look something like ...

Customer or tech team becomes aware of potential incident and notifies x,y,z
Investigate for 30 mins then huddle for 5 mins
Decide on a plan for further investigation / urgent mitigation / further decisions
Execute the plan and adapt as necessary
Wash up <-- THIS IS IMPORTANT

During the washup/lessons learned phase, think about preventing similar problems in the future, how to make incident response easier in the future, what aspects of the incident could have been managed better and whether any changes to the process should be made as a result. Be careful not to just fight yesterday's fire ... any plan needs, above all else, to be simple and flexible.

YAGNI

Fri, 28 Jun 2024 00:00:00 +0000

Developers like developing things. More specifically, they like developing new things. Left to their own devices, developers have a tendency to build many more things than is strictly necessary, in ways that are more complex than needed, in order to handle every conceivable use case and weird edge condition.

YAGNI combats this tendency with a straightforward idea: don't add features, capabilities, or bits of code unless you absolutely need them right now. Instead of planning for every possible future scenario, focus on what's essential at the moment. This approach is a perfect fit with Agile methods, which emphasise delivering small, manageable changes rather than big, speculative ones.

I'm a wholehearted believer in YAGNI. It fits really well with "optimising for speed to market" and the idea that your first (MVP) attempt is going to need rewriting anyway. The less code you've written in the first place, the faster it's going to be to make it work properly the second time round.

YAGNI does, however, require a shift in mindset. Aside from having to watch over development teams like a hawk to make sure they aren't building things that aren't needed, there is inevitably going to be pressure from stakeholders (and therefore the customer and sales teams) pushing for features.

This is tough - and sometimes impossible - to resist. It's always worth asking stakeholders how much they value something (or how quickly they really need it) by getting them to rank any requirements and then asking what would happen if they only got the top two. Chances are they'll still sign up and you'll have saved yourself a whole load of legacy code problems.