Rentisha's WhatsApp self-service bot has to route an inbound tenant message into one of six intents: opt out, check balance, pay, request a receipt, report a maintenance issue, or anything else.
The first implementation was a keyword regex, and on paper it was fine. Tenants were told to send PAY or BAL. In practice tenants send "I need to pay my rent", "what do I owe", "kodi yangu ni ngapi", and "the kitchen tap is leaking", and a keyword matcher classifies most of that as HELP, which is the bucket that means "a human will get to this eventually".
The failure is quiet and expensive. A tenant trying to pay you gets a menu instead of a payment prompt.
Code-switching is the normal case, not the edge case
Kenyan tenants do not write in one language. They write in English, Swahili and Sheng, and they switch inside a single sentence. "Nataka kulipa rent yangu leo" is three languages and one completely unambiguous intent.
This breaks keyword matching structurally rather than incidentally. You cannot enumerate the vocabulary, because Sheng is generational, regional and actively evolving, and a word list current this year is stale next year. And you cannot language-detect first and then branch, because there is no single language to detect.
So the classifier is given the multilingual reality directly. Each intent in the set is described by what the tenant is actually trying to do rather than by the words they might use, and then illustrated with a handful of real phrasings drawn from all three languages at once. The vocabulary lives in the prompt as examples, never as a lookup table, precisely because a lookup table is the thing that goes stale.
The disambiguation rules are where the domain lives
The interesting part of the prompt is not the vocabulary, it is the four rules that resolve the genuinely hard cases:
- A message containing only an attachment is a maintenance ticket. Somebody who photographs their ceiling is documenting a complaint, not asking a general question.
- A bare greeting; "hi", "habari"; is HELP, not an intent to be guessed at.
- The word "stop" inside a longer sentence is not an opt-out. "Don't stop calling me about the leak" must not unsubscribe somebody.
- If two intents are plausible, choose the one the tenant most needs action on. "I want to pay but check my balance first" is PAY.
That third rule is the one with real consequences. Wrongly registering an opt-out is close to irreversible: you have lost a communication channel with a tenant, and you cannot message them to ask whether they meant it.
A classifier that can decline to answer
The model does not return free text. It returns a structured result validated against a schema: one intent from a closed set, a confidence score, and a one-line rationale for the choice. Anything that fails validation is treated exactly like a low-confidence answer. Confidence is load-bearing here rather than decorative.
Below a threshold (0.6 by default, tunable by environment variable), the result is discarded and the old keyword regex answers instead. The regex was never deleted. It is the floor.
It also answers when the API key is absent, when the call errors, when it times out, and when the response fails to parse. Every one of those paths is explicit, and every one of them ends in a working classification rather than an exception. The bot degrades to its previous behaviour rather than degrading to silence.
Storing the rationale costs almost nothing and makes misclassifications diagnosable after the fact. When a tenant complains that the bot misunderstood them, you can read what it thought it was doing.
The honest part: we do not have a benchmark
The obvious question is how we evaluate this, and the honest answer is that we do not have a held-out, labelled, code-switched evaluation set. We should say so plainly rather than describe an accuracy number we did not measure.
Building one properly is harder than it sounds, and the difficulty is instructive. A representative Sheng eval set is real tenant messages, which are personal data governed by the Kenya Data Protection Act, collected for the purpose of operating a tenancy, not for training or benchmarking a classifier. Using them for evaluation is a purpose question before it is an engineering question, and the answer involves consent and minimisation, not a data export. Synthesising the messages instead mostly measures whether we can imagine how tenants write, which is the exact assumption that made the regex fail.
So what we have instead is a design that is conservative in the absence of measurement: a confidence gate, a deterministic fallback that was already in production, a stored rationale for every decision, and one genuinely irreversible action; opt-out; protected by an explicit rule.
That is a defensible interim position, not a finished one. The thing that would actually settle it is a properly consented sample, and that is the piece of work in front of us rather than behind us.
Cost, briefly
The system prompt is cached, so repeat classifications hit the cache and cost a fraction of the first call. The model is a small fast one, capped at 256 output tokens, with extended thinking off; six-way classification of a one-sentence message is not a reasoning problem, and paying for reasoning on it is a good way to make a cheap feature expensive.