Sorting Estate-Agency Enquiries on a Mac Mini: 89 % Right on 235 Synthetic Messages

Estate agencies on the Costa del Sol get enquiries through WhatsApp, e-mail and the portals, in many languages, at any hour. Agents we have talked to name the same problems: telling a real buyer from noise, knowing who to call first, and following up before the lead goes cold.

We tested whether one small local model can do the sorting if it never decides the policy. The model answers closed questions about each message. The agency’s own written rules, in code, turn the answers into an urgency and a next step.

What happened:

• The model answered 11 closed questions per message and wrote no text. Kev-4B, a 4B decision model, returned a probability for each option of questions such as “what does the sender want?” (eight options) and “does the sender ask to be called?” (yes or no). Phone numbers and e-mail addresses were masked before it saw the message. One message took about 1.6 s on a Mac Mini.
• We tested it once on 235 synthetic enquiries in seven languages, with pass marks written and signed before the run.
• It read the request right 89 % of the time, against 24 % for always guessing the most common request. From its answers, the rules picked the agent’s next step right 90 % of the time. It caught 29 of 32 spam messages.
• It also flagged 3 of the 203 genuine (non-spam) enquiries as spam. All three came from one rule, “a business pitching the agency”, and scored just over its cut. One was an owner selling a villa. So spam now goes to a review folder for a person, and nothing is deleted.
• Two fields failed: whether the lead is qualified (78 %) and the timeline (73 %). The rules still use them, combined with other answers, but agents are not shown them.
• The caveat: the enquiries were synthetic, and messages that a second reader found ambiguous were removed. Real enquiries will be harder.

The result in Simplified Technical English

An estate agency gets many messages. Each message comes from a person who wants to buy, rent, sell or ask a question, or it is spam. An agent must read each message and decide what to do.

We used one small model on our own computer. The model reads a message. It does not write text. It answers 11 short questions about the message, for example: “What does the sender want?” and “Does the sender ask for a call?”. A program then applies the agency’s rules to the answers. The program gives the urgency and the next step, for example “book a valuation”.

We tested this on 235 messages in seven languages. A person did not write these messages: a large model wrote them for the test. The model found the correct request in 89 % of the messages. The program gave the correct next step in 90 % of the messages. Three messages from genuine clients were marked as spam. Thus the system must not delete spam. It puts spam in a folder, and a person examines it.

The design: the model answers, the rules decide

We did not ask a model to “triage this lead” in free text. A free-text answer is hard to check, and the model would quietly invent the agency’s policy. Instead we split the job:

• Kev-4B answers closed questions. Decision models such as Kev are trained to choose between fixed options and return a probability for each; they do not write text. Kev-4B is open (Apache-2.0) and runs on llama.cpp’s /v1/systemone endpoint on the Mac Mini. One call answers all 11 questions:
  • three choices: what the sender wants (buy, long-term rent, holiday rent, sell, valuation, investment, a question, spam), the property type, and the timeline;
  • eight yes/no facts: needs a mortgage, from abroad, qualified (a clear goal plus two of budget, area and timing), hot (visiting today or tomorrow, about to make an offer or sign, or talking to another agency), about a specific listing, asks for a call, asks what their property is worth, and a business pitching the agency.
• Code reads the facts a program can read exactly: the language, a budget figure, and whether a phone number or e-mail address is present. Contact details are then masked, so the model never sees them.
• Code applies the agency’s written rules. A business pitch at 0.6 or above becomes spam. Urgency goes from 0 to 3: 0 for spam and questions; otherwise 3 if the sender is hot, 2 if qualified and acting now or asking about a specific listing, 1 if acting now or within months, and 0 for the rest. The next step is the first rule that fits: spam to review (renamed from “discard” after the result: the same decision, nothing deleted); a question to “answer the question”; a valuation, or a seller asking the price, to “book a valuation”; a qualified sender acting now who gives a phone number or asks for a call to “call now”; another qualified sender to “send listings” (a seller to “book a valuation”); and otherwise “ask for details”.

The rules come from a one-page label definition written before any test. If an agency’s policy differs, the rules change in code. The model does not have to relearn anything.

The test (exp_043)

We had no labelled agency messages, so we made a synthetic set and tried to make it hard to game:

• Writing. Eight instances of a strong cloud model (Fable) wrote 440 enquiries: two English batches (one deliberately messy), and one each in Spanish, Polish, German, Swedish, French and Russian. Each batch had the same quotas per request type. About a quarter carried fake contact details, and at least one in five was a hard case. No client data was involved.
• A blind second reader. Another instance labelled every message from the text alone, without seeing the writer’s labels. We dropped a message if the two disagreed on the request or the reader flagged doubt: 85 messages. 355 were kept, and on all 440 the two agreed on the request 98.6 % of the time.
• Split and seal. 120 messages for tuning and 235 for the test, split by language. Andrei read 30 test messages and their labels before the set was sealed, and rejected none.
• Tuning on the 120 only. Three revisions. The second sharpened the request descriptions and moved urgency and the next step from the model into code (next step on the tuning set: 81 % → 93 %). The third added the business-pitch question and set its cut at 0.6: at 0.5 it flagged five clients on the tuning set.
• One pass. The code was frozen. The test runner refuses any other version of it, and refuses a second pass.

Results

A field is scored only where the writer and the second reader agreed on it: property type on 234 messages, timeline on 224, urgency on 228, and mortgage and from abroad only where the message states them.

By language, on 26 to 60 messages each, the request was read right for Swedish 100 %, Spanish 97 %, German 90 %, French 89 %, English 87 %, Russian 87 % and Polish 77 % (20 of 26). Kev-4B’s model card says English only.

The three headline marks passed, so the pre-registered verdict is usable for triage, as a suggestion to an agent. Spam passed narrowly on both sides. The pre-registration would also have allowed one automatic action, discarding spam. After the three false spams we decided against it. The first two marks dropped from the tuning set to the test set (93 % → 89 % and 96 % → 90 %), as expected after tuning.

What went wrong

• The spam that wasn’t. All three false spams came from the business-pitch rule, at 0.60 to 0.63, just over the cut. They were a developer looking for building plots, a company booking a villa for a team retreat, and an owner listing a villa for sale. On the tuning set no client had scored above 0.55, so the margin was thin from the start. Auto-deleting spam would have thrown away a seller.
• Buying a holiday home, read as renting one. This was 10 of the 25 request errors. Holiday and short-stay cues (“over from Leeds for the week”, “flying in Thursday for four days”) pulled buyers toward holiday rental.
• Qualified and timeline are weak. The next step still scored 90 %, because the rules combine them with other answers. Shown alone, they would mislead.

What the test does not show

• These were synthetic messages. We removed the 85 that a second reader found ambiguous, so the test is easier than an inbox. A pass licenses a trial on real leads, not a claim about them.
• One model, one run. We did not measure whether the probabilities are calibrated.

After the test: a demo you can talk to

These are field notes, not measurements. To show agents, Kev now runs in our Telegram bot behind an admin command:

• Paste an invented enquiry (Telegram is a cloud service, so no real client messages go there), and the bot replies in about two seconds with what the agent would see: for example “Buy · Villa · From abroad · Cash · up to €1.5M → Send listings”.
• When the model is unsure (request confidence below 0.5), the reply names its top two readings: “Buy or investment? 30 %, 30 %. The agent decides.” These are the model’s raw probabilities, not calibrated confidences.
• Voice notes are transcribed on the Mac Mini (Whisper large-v3-turbo on MLX), the audio is deleted, and the transcript is triaged: about 3–4 s end to end in our rehearsals.
• A spoken reply reads the result aloud, built by code from the same answers and voiced offline by the operating system.

Two things surfaced while building it:

• The model gives about 0.6 to facts the message never states. A buyer who never mentioned financing got 0.6 for “needs a mortgage”. The test only scored stated facts, so this was invisible there. The demo now shows a yes/no fact only from 0.7.
• Our language detector read Spanish as Polish. It treated “ó” as a Polish letter, so “un millón” made a Spanish voice note Polish. Voice notes now use the language Whisper hears, and the detector no longer treats “ó” as Polish-only.

What we take from it

1. Keep the policy in code and ask the model only closed questions. Two of the model’s answers failed alone, yet the next step built from all of them scored 90 %. When an agency’s rules change, nothing has to be retrained.
2. Never delete on a model’s say-so. A 1.5 % false-spam rate passed our mark, and it still contained a seller. The cheap fix is a review folder.
3. Show when the model is unsure. This is a design choice, not a measured result: the demo names the two readings the model is torn between rather than one label, and says the numbers are raw probabilities.

So what

What changed because of this experiment:

• Spam goes to a review folder, never to the bin. This is now a rule in the code, not a setting.
• CasaSol now offers agencies a free assessment of 50–300 of their own enquiries, names removed, processed on our server under a data-processing agreement (casasol.ai/agencies). That is the real-lead test this record could not do.
• The demo above runs in CasaSol Guide for showing agents. It does not read any real inbound messages yet.

What it opened:

• A safer spam rule, for example requiring the business-pitch question and the model’s own spam choice to agree. That needs a fresh test, because this one is used.
• A direct “buying or renting?” question for the holiday-home confusion.
• Polish, the weakest language, as with our answerability gate.
• Budgets written as words (“un millón”) are not read yet, and a request for a call does not raise urgency.

Record, with every test message, label, prediction and the frozen code: exp_043.