Strands Decider: Why Not an Encoder?

# Strands Decider: Why Not an Encoder? Source: [https://brooker.co.za/blog/2026/10/04/encoders.html](https://brooker.co.za/blog/2026/10/04/encoders.html) Expertise level: exhausted\. *As I admitted in[my last post](https://brooker.co.za/blog/2026/09/28/engineering-system-one.html), I am very much not an expert model developer\. What follows is likely to be, at least partially, inaccurate\. I have tried to be careful and quantitative, to partially balance a lack of deep expertise\.* Since we launched[strands\-decider\-2B](https://github.com/strands-labs/strands-decider)last week, a couple of people have asked me why it’s a modified LLM and not an encoder \(like[BERT](https://arxiv.org/abs/1810.04805v2)\)\. Their instinct seems to be that*bidirectional*\(i\.e\. each token’s representation is built from tokens before and after it\) is likely to out\-perform*causal*\(i\.e\. only before tokens are used\)\. This is the same reason encoders are widely used as classifiers\. This is also a fair question, because if you squint at it right, we’re abusing a decoder as an encoder\. To test this hypothesis, I compared five designs: - **v19**is the final design for Qwen3\.5\-based strands\-decider \(check out the previous post and[strands\-decider README](https://github.com/strands-labs/strands-decider/blob/main/README.md)for the details\)\. - **b1**is an encoder\-decoder design \(like the original[Transformer](https://arxiv.org/pdf/1706.03762)\), based on[T5Gemma 2](https://blog.google/innovation-and-ai/technology/developers-tools/t5gemma-2/)with 1B encoder params and 1B decoder params, minus the LM head and plus our pointer head\. - **b2**is the same as b1, but 4B\+4B\. - **e1a**is just the encoder half of[T5Gemma](https://deepmind.google/models/gemma/t5gemma/?tab=1-t5gemma)\(not T5Gemma 2\), with 2\.6B parameters\. - **e1b**is a latency\-optimized version of e1a with a masked state cache, allowing for multiple questions to be evaluated in parallel without interference, and for the state cache to be reused[1](https://brooker.co.za/blog/2026/10/04/encoders.html#foot1)\. The`state`attends to the whole`state`bidirectionally, and each question attends to`state`and itself\. I also measured frozen[ModernBERT](https://huggingface.co/blog/modernbert)and[ettin\-encoder\-1b](https://huggingface.co/jhu-clsp/ettin-encoder-1b), but found that they under\-performed T5Gemma parameter\-for\-parameter, and so didn’t invest time in training them\. First, let’s look at accuracy and calibration\.  *b2*does very well here, beating*v19*on JevBench accuracy and Brier, and on JF100 Brier at the same accuracy\. This is good, but recall that*b2*has ~8B total parameters compared to*v19*’s 2B, and so is not an apples\-to\-apples comparison\. The really interesting comparison is*v19*and*e1b*\.*e1b*slightly beats*v19*on JevBench accuracy \(but within the run\-to\-run variance on*v19*\), with slightly worse calibration, and ties on calibration on JF100 with slightly weaker performance\. So far, not a slam\-dunk for the encoders\. Next, let’s turn to latency\.  The encoder\-only models win handily on latency, with a median of around 35ms compared to*v19*’s 65ms\. The encoder\-decoder models are slowest\. With half the latency and similar performance, this is looking like a big win for*e1b*\. But thing get a little more complicated when we dive into the data\.  Here, we see that*e1b*wins handily at lower prompt lengths, but scales worse as the prompt grows\. This is consistent with the great median performance\. Whether this matters depends on the workload mix: both JF100 and JevBench trend small, to*e1b*’s benefit\. The scaling difference here comes down to the compute costs of the two models:`e1b`has 26 full attention layers with the associated quadratic compute cost, while 75% of*v19*’s layers are[Gated DeltaNet](https://arxiv.org/abs/2412.06464)with linear compute cost\. These layers have a higher fixed cost, but better asymptotic cost, leading to the higher latency floor[2](https://brooker.co.za/blog/2026/10/04/encoders.html#foot2)\. Finally, let’s look at one interesting sub\-plot, going back to*e1a*\(our unmasked 2\.6B encoder\-only candidate\)\. On the eval sets, it beats*v19*on seven of nine tasks, and is very slightly behind on two\. But it does significantly worse at JevBench \(153 vs 168\), and a little worse at JF100 \(158 vs 162\)\. This suggests that the encoder generalizes worse, at least without the masked approach*e1b*takes\. Is being able to attend to the question when reading the state actually a weakness? Evaluation setv19e1aΔHeld\-out short tasks0\.6470\.695\+0\.048MuSiQue0\.8820\.948\+0\.066ContractNLI0\.8730\.864−0\.009BoardgameQA0\.8200\.884\+0\.064HotpotQA \(held out\)0\.7190\.748\+0\.029Generated, v16’s0\.8570\.851−0\.006Generated, v18’s0\.7770\.786\+0\.009Adequacy, HelpSteer20\.7220\.761\+0\.039Adequacy, generated, balanced0\.7880\.821\+0\.033Where does that leave us on the encoders vs decoders question? Not very much closer to the truth, I fear\. This experiment is hopelessly confounded, changing architectures, base models, instruction tuning vs base, and other variables\. But we have learned some interesting things\. Neither seems like a slam\-dunk winner for this kind of work at this size\. The code for this experiment is[in a branch for now](https://github.com/mbrooker/strands-decider/tree/bidi-b1)\. It doesn’t seem worth pulling into mainline strands\-decider yet, but may with some development\. **Footnotes** 1. Some decoder ideas sneaking back in\. Decoders do the same trick \([KV cache](https://arxiv.org/abs/2211.05102)and prefix cache\) for the same reasons\. Certain not for the first time memoizing incremental results has helped performance\. 2. Certainly not the first time an $O\(N^2\)$ algorithm has beaten an $O\(N\)$ one at small data sizes\.