Imagine asking Simba to compare two budget scenarios. You look at the response curves, realise there is a constraint missing, and say: “Actually, keep search spend fixed. Show me the assumptions before running it.”
That is the kind of interaction we want to support. A question develops as you see the evidence. You refine the brief, ask why something changed, and decide whether the next step is worth taking. Speech could make that easier, alongside typing and the charts, tables and diagnostics that an analytical product still needs.
But each of those sentences creates work for the system. It has to recognise the correction, update the intended constraints, understand whether anything has already started, and give you an accurate account of the result. A fluent answer is only one part of the job.
What the results below cover. These are development evaluations of a separate voice-agent system, using generated speech and real model providers. They inform our thinking about Simba. They are not measurements of a shipped Simba voice interface, analytical accuracy, or production reliability.
What we are actually testing
Our voice-agent work includes ordinary replies, tool requests, requests that should not trigger a tool, stopping speech, recovering on the next turn, and ending a conversation. Each needs an observable outcome. Did the intended action happen? Did something happen that should not have? How long did it take?
The development runs discussed here use two synthetic voices and four audio conditions: clean, quiet, added noise and reverberation. The audio goes through the real provider path. The harness records the resulting decisions, system effects and timing. A model saying “done” does not establish that an action happened.
We report correctness and timeliness separately. For the conversation-ending cases, the deadline was three seconds after the input finished. Negative cases had a full observation window, and missing decisions remained failures. We also checked that the test audio had been delivered completely and at the intended pace.
A small request that crosses the whole system
“End this conversation” sounds straightforward. The voice system may still be receiving audio, a model may be preparing a response, and a service request may be underway. There needs to be a specific point at which the conversation loses permission to continue.
We put that responsibility in the host application. Once it accepts an end request, it closes audio input to the model and revokes the conversation’s service access before waiting for connection cleanup. The surrounding host and wake-listening infrastructure can remain available for the next interaction.
That gives us two separate problems to investigate: how quickly we can recognise the request, and how reliably the application can carry it out. The pre-emption experiment focused on the first.
Starting the check before the user has finished
The initial path waited for final transcription before checking whether the user wanted to end the conversation. That puts the transcription delay and the decision delay one after the other.
Pre-emptive checking overlaps some of that work. When a partial transcript is stable, we can start a provisional intent check. If the final text still matches the current request, the result may be reusable. If the words change, that provisional decision must lose its authority.
In a matched development comparison, both paths saw the same 32 cases with the same model settings. The pre-emptive path produced more correct, timely outcomes and a much shorter median end latency.
| Original 32-case pack | Final-only check | Pre-emptive check |
|---|---|---|
| Correct outcomes | 31/32 | 32/32 |
| Correct and timely | 29/32 (90.6%) | 31/32 (96.9%) |
| Median valid end latency | 2.704 seconds | 0.796 seconds |
| Worst valid end latency | 3.415 seconds | 3.437 seconds |
| Unintended endings | 0 | 0 |
The median was about 71% lower in this comparison. The worst end was still late. This was a small, sequential development test, so it gives us a useful result to investigate rather than a general speed guarantee. The latency statistics describe the end cases; the 32-case totals also include requests where the conversation should stay open.
Then we let the user change their mind
We added a correction pack. One of the inputs was: “That’s all. Actually, one more thing. Keep listening, I have another question.”
That exposed the problem. The provider could finalise “That’s all” as a transcript segment before the correction arrived. The host could then end the conversation and cut off the rest of the input. In the 32-case correction pack, we observed four unintended endings.
The original pack would have missed this. A final transcript segment was not sufficient evidence that the person had finished their complete request. A faster decision made that distinction more consequential.
We refined the process so a preview remained cancellable when new text arrived, including after the final-check path had begun using it. We added a bounded stability hold before acting on a positive decision, and moved blocking response decoding off the audio-processing loop. These were several engineering changes, so the later result cannot be credited to pre-emption alone.
Here is the before-and-after comparison on the correction pack, with 32 attempts per version. The four unintended endings occurred in the early version; none were observed in the refined version.
| Correction pack — 32 attempts per version | Early pre-emptive version | Refined version |
|---|---|---|
| Correct outcomes | 27/32 | 32/32 |
| Correct and timely | 26/32 | 31/32 |
| Unintended endings | 4 | 0 |
We reran both the original pack and the correction pack using the refined version. The full results for that version were:
| Refined version only | Original pack | Correction pack | Combined |
|---|---|---|---|
| Correct outcomes | 32/32 | 32/32 | 64/64 |
| Correct and timely | 32/32 | 31/32 | 63/64 (98.44%) |
| Unintended endings | 0 | 0 | 0 |
| Ends within three seconds | 8/8 | 7/8 | 15/16 |
| Median end latency | 2.699 seconds | 2.792 seconds | — |
The guard used some of the latency we had saved. That trade-off belongs in the result: the refined candidate did not retain the earlier 0.796-second median. It handled every tested outcome correctly, with one end still outside the deadline. Zero unintended endings in this sample is encouraging; it does not prove that an unseen correction will always be handled safely.
Stopping speech is a different task
If someone says “stop talking”, they may still want to ask another question. Ending the whole conversation would be the wrong effect. We evaluated speech stopping and next-turn recovery separately.
A prompt candidate that explicitly treated silence as the acknowledgement was tested on a 32-case development matrix and an unchanged 32-case confirmation. Across those runs, all 16 interruption cases stopped within one second, and all 16 recovery cases responded within three seconds. The longest measured stop was 301 milliseconds.
Across the complete matrices, 64/64 outcomes were correct and 63/64 were timely. The remaining ordinary reply took about 4.14 seconds, beyond its deadline. That means the complete confirmation did not pass, even though the stop and recovery subsets did. This was measured model behaviour under the tested prompt and conditions, not a guarantee of deterministic playback cutting.
The evaluation itself needs checking
One useful finding had nothing to do with the model. An early version of our audio replay accumulated timing delays, stretching clips to roughly 1.45 times their intended duration. Another scoring path could turn a premature end into a zero-millisecond latency result.
We corrected the replay clock and required complete, correctly paced input before counting a timing success. The earlier results stayed in the record, but we excluded them from the latency comparison used here. Otherwise, we would have been optimising against a measurement defect.
These are development results. The packs were used to diagnose and tune behaviour. Broader held-out wording, more acoustic conditions, full conversation end-and-restart testing and physical-device acceptance remained outstanding for the cited end candidate. A high aggregate score does not replace those checks or a failed scenario deadline.
What this means for Simba
Simba already has an MCP integration that gives AI assistants access to modelling and analysis tools. We described that foundation in our MCP introduction. The next interaction questions are about how people guide work over several turns, inspect the evidence and revise their instructions.
The voice experiments give us concrete design lessons for that work. We can begin interpreting an instruction early, but we need to distinguish a provisional interpretation from permission to act. We need to cancel obsolete decisions when a user corrects themselves. We need the interface to reflect the actual state of a job.
Consider three instructions in a modelling session. “Stop explaining” should stop the explanation. “Cancel that scenario” should target the relevant scenario job and report whether cancellation succeeded. “I’m done for now” may end the conversation while a model fit continues. Those are different contracts, and they need different tests.
We would not copy the voice agent’s conversation lifecycle directly into Simba. A model fit can take time and produce a useful result after the user leaves. The transferable principle is to make ownership, cancellation and completion explicit, so the user knows what their instruction changed.
The experience we want is a shared analytical workspace. You can speak or type, see which dataset and model are in use, inspect uncertainty and diagnostics, change a constraint, and review the next action. An agent can help organise the workflow; the underlying statistical analysis still needs to support the conclusion.
The next tests should look like analytical work
For Simba, we would extend this approach to tasks such as comparing two named models, keeping a channel constraint intact through a correction, recovering from a failed scenario request, and distinguishing a submitted fit from a completed one. Those tests need to check the dataset, model, parameters and job state that were actually used.
They also need to preserve the scientific checks. Correctly calling a fitting tool does not establish convergence. Producing a budget allocation does not establish that its assumptions are appropriate. Conversational task success and model validity need their own evidence.
That is the development direction these experiments inform: make the interaction easier while keeping the analysis inspectable and the user in control. A correction halfway through a sentence should be an ordinary part of working with Simba. Our job is to make the system handle it properly.