Getting to know Jev part 1: Does option order matter?

There has been a lot of hype around Jev, TypeSafe’s new “System One” model. Its pitch is simple. Instead of generating text, Jev takes some state (a support ticket, a document, a chat log) plus a set of typed questions, and returns structured answers with a probability for each option. Oh, and it does so in ~100ms and at ~$0.042 per million input tokens. OpenAI quickly announced its own Decisions API after Jev launched, and I expect we’ll see a lot more of this category.

If you use an LLM as a classifier in production, you should ensure that its output does not depend on the order in which you list the options. This risk is well documented for LLMs on multiple-choice questions (see Zheng et al., 2023 and Pezeshkpour & Hruschka, 2023), and anyone who has used an LLM as a classifier has probably been bitten by it at some point (knowingly or unknowingly).

TypeSafe is upfront about this. Its own list of known Jev weaknesses says the model “leans toward the option that comes first”, and suggests reordering options to double check. But it gives no numbers. A few people have poked at it since launch; AI2SQL reordered six options across twelve tasks and saw no change, a small GitHub test found Jev kept the same label in 36 of 36 answers, and a pre-registered experiment found that when the input carries no evidence, Jev does lean toward options written earlier (and, oddly, toward names starting with “a” or “1”). What I couldn’t find was a test at scale, on real labeled data, against LLMs.

So I decided to find out.

1. Setup

Datasets

I used three public datasets with human labels:

• AG News (released 2015, built from news articles collected in 2004-05): news topics, 4 options per question (selected 300 items).
• Banking77 (released 2020): bank customer intents, 77 options per question (selected 300 items).
• CLINC150 (released 2019): virtual-assistant intents, 150 in total (I left out its separate “out of scope” class). For each item, I asked the question with 2, 4, 8, 16, 32, 64, 128 and all 150 options, always including the correct one (selected 200 items).

The options are plain label names without descriptions, i.e. how most of us would pass an enum.

Orderings and repeats

Each item was asked with its options in several different orders:

• AG News: all 24 possible orderings, for every item.
• Banking77: 8 rotations of the label list plus 8 random shuffles.
• CLINC150: 6 random orderings at each list size.

Every item was also sent 5 times with exactly the same request (same question, same option order), for every model. This gives us a noise floor, which turns out to matter. Models don’t always give the same answer to the same question, so some of what looks like an “order effect” is really just noise.

Models

Everything went through OpenRouter:

• typesafe/jev-1.13.
• OpenAI’s Decisions API (openai/gpt-6-luna-decisions), in public beta since October 6. It takes exactly the same request as Jev and returns the same answer types, so it ran the full Jev test suite. More on it in section 6.
• GPT-6 Luna, with reasoning off.
• Claude Haiku 4.5, at temperature 0.
• GLM-5.3-Flash, an open-weight model (weights on Hugging Face) served by Z.ai, at temperature 0 with reasoning set to “minimal” (it can’t be switched off for this model).

Why these models? OpenAI’s Decisions API is Jev’s most direct competitor: same request format, same question types, powered by a flavor of GPT-6 Luna. For the LLMs, I wanted models one could realistically consider for the same job as Jev, i.e. fast and cheap classification calls. GPT-6 Luna is OpenAI’s budget model (and the one powering the Decisions API), Haiku 4.5 is Anthropic’s small model and a common default for classification, and GLM-5.3-Flash is a recent open-weight model at roughly Luna’s price ($0.15 vs $0.10 per million input tokens, $0.50 for output). I left out frontier models like Claude Sonnet or Kimi K3; at 20x Luna’s price per token or more, they defeat the point of a System One model.

To keep things fair, the LLMs were prompted using TypeSafe’s own open-source adapter (system-one-adapter), which converts the same Choice question into an LLM prompt.¹ I ran the adapter in two modes: one asks the LLM for a probability per option (the documented setup), the other asks for a single label.

All in, that came to ~91K calls and $33.85 of OpenRouter credit, including pilot runs, a few test calls and the formatting test in section 6. By model, that was Jev $1.12 (~31K calls), OpenAI’s Decisions API $1.76 (~31K calls), Claude Haiku 4.5 $28.39, GLM-5.3-Flash $1.38 and GPT-6 Luna $1.12 (9,600 calls each). Most of Haiku’s bill came from writing out 77 probabilities per call on Banking77.

What does a question look like?

For reference, here is a sample AG News item exactly as Jev sees it. The only thing that changes between orderings is the order of the keys in criteria:

{
  "model": "typesafe/jev-1.13",
  "state": "Bush Faces Heavy Pre-RNC Travel Schedule (AP) AP - President Bush charges into the final runup to the Republican National Convention with a heavy campaign schedule in key states he needs to carry in November.",
  "questions": {
    "q0": {
      "type": "choice",
      "instructions": "Which topic does this news article belong to?",
      "criteria": {
        "World": null,
        "Sports": null,
        "Business": null,
        "Sci/Tech": null
      }
    }
  }
}

Jev returns a probability for each option, e.g. {"World": 0.83, "Business": 0.15, "Sports": 0.01, "Sci/Tech": 0.01} (so 83% World, 15% Business, and so on).

A Banking77 item looks the same, just with all 77 intents as options (shortened here; the human label is order_physical_card):

{
  "model": "typesafe/jev-1.13",
  "state": "I want a card. What is the procedure?",
  "questions": {
    "q0": {
      "type": "choice",
      "instructions": "Which intent best describes this customer message to a bank?",
      "criteria": {
        "Refund_not_showing_up": null,
        "activate_my_card": null,
        "age_limit": null,
        // ... 71 more intents, including "get_physical_card" and "order_physical_card" ...
        "why_verify_identity": null,
        "wrong_amount_of_cash_received": null,
        "wrong_exchange_rate_for_cash_withdrawal": null
      }
    }
  }
}

And here is the 8-option version of a CLINC150 item (human label: goodbye). The same item was also asked with 2, 4, 16, 32, 64, 128 and 150 options, always including goodbye plus randomly drawn other intents:

{
  "model": "typesafe/jev-1.13",
  "state": "it was great to speak with you",
  "questions": {
    "q0": {
      "type": "choice",
      "instructions": "Which intent best describes this request to a virtual assistant?",
      "criteria": {
        "credit_limit_change": null,
        "oil_change_how": null,
        "redeem_rewards": null,
        "roll_dice": null,
        "report_lost_card": null,
        "carry_on": null,
        "income": null,
        "goodbye": null
      }
    }
  }
}

Banking77 is hard on purpose. Many of its labels are near-synonyms (get_physical_card vs order_physical_card, card_arrival vs card_delivery_estimate) — this is difficult for humans too (not just LLMs!) and order starts playing a role here.

2. Experiment 1: How often does reordering change the answer?

Let us start with the news article above. Here is what each model said for each of the four rotations of the label list (each cell is a single call):

Jev says World every time, with its probability moving around a little (74-84% across these four calls). In fact, I asked Jev about this article 28 times (all 24 possible orderings, plus 4 identical repeats), and it said World every single time (74-91%). Haiku in probability mode gives four different answers for four orderings, each with 70-94% “confidence”. (On this article, it put most of its probability on whichever label came last.) Asked for a single label, Haiku mostly says World, but still slips once. The LLMs got 8 calls per item in each mode (the 4 rotations, plus 4 identical repeats).

That is one item. Overall, across all of them, Jev rarely changes its answer when you reorder the options.

• AG News (4 options): two orderings of the same four labels gave Jev different answers 0.4% of the time (95% CI 0.0-1.0%), about the same as identical repeats (0.3%). Across all 24 orderings, its accuracy stayed between 90.3% and 91.7%, which is within sampling error for 300 items.
• Banking77 (77 options): reordering changed 5.0% of answers, against 2.2% for identical repeats. So reordering adds roughly three points of instability on top of Jev’s run-to-run noise (which TypeSafe acknowledges too).

Here is what one of those Banking77 flips looks like. For “I want a card. What is the procedure?”, Jev’s two leading options across the four rotations were:

This is a real order effect, not noise; asked again with rotation 1, Jev said get_physical_card five times out of five (64-72%). It is also a question where I’m not sure I’d get the “right” answer (order_physical_card) myself.

How do the LLMs compare? That depends a lot on how you ask them. As a reminder, I asked each LLM in two ways (both through TypeSafe’s adapter).¹ In probability mode, the LLM writes out a probability for every option, in the same shape Jev returns. For the news article above, Haiku replied with something like this:

{"World": 0.15, "Sports": 0.05, "Business": 0.10, "Sci/Tech": 0.70}

In single-label mode, it just names one option ("World"). Probability mode is the closer like-for-like comparison with Jev (and it is the adapter’s documented setup), but single-label mode turned out to be much more stable.

(OpenAI’s Decisions API is the second row; I’ll come back to it in section 6.)

Two things stood out.

In the fairest comparison, Jev is still the most stable model. Against the LLMs picking a single label, Jev was several times steadier on AG News (6x or more), and somewhat steadier on Banking77 (about 1.5x). (The two datasets differ in more than just the number of options, so I wouldn’t read much into the difference between those two ratios; the CLINC150 test below changes only the length of the list.) It also gives you probabilities, which a single-label LLM call doesn’t.

Asking an LLM to write out probabilities makes it much more order-sensitive. Haiku is the starkest example. On AG News in probability mode, ask it the same question twice and it gives the same answer about 98% of the time. But reorder the four labels and it changes its answer 16.5% of the time. Its accuracy then ranges anywhere from 74% to 88%, depending purely on how the labels were sorted. Probability mode also cost Luna and Haiku accuracy; on Banking77 they scored 70.7% that way, against 78-79% in single-label mode (Jev: 78.9%).

GLM-5.3-Flash is the odd one out. Asking it for probabilities barely made it more order-sensitive, and (as we’ll see below) it showed little position bias either way. Its problem is plain noise; in probability mode, it changed its answer on 6% of identical repeats on AG News and 9% on Banking77, nearly as often as reordering did (7% and 11%). So for GLM, reordering adds almost nothing on top of the noise; the noise itself is the issue. Its accuracy was on par with the best of the other LLMs (89-91% on AG News and 77-78% on Banking77, against Jev’s 91% and 79%).

I’ll admit my first version of this post only used probability mode. That made Jev look about 40x more stable than Luna and Haiku on AG News (0.4% of answers changing under reordering, against roughly 16% for the two LLMs). Single-label mode shrinks that gap a lot, though Jev still comes out several times steadier. (I only caught it because a robustness check didn’t line up…) It is worth knowing if you’re using TypeSafe’s adapter to benchmark Jev against an LLM; its default mode puts the LLM at a noticeable disadvantage. Potentially a little sneaky here… 🤔

3. Experiment 2: Is there a first-option bias?

For Jev, I couldn’t find one.

• On AG News, every position won its fair share to within 1%, and accuracy was the same wherever the correct label happened to sit.
• On Banking77, every position got within about 8% of its fair share, with no trend from first to last (the first eighth of the list was actually picked slightly less often). The same held at every list size on CLINC150.

So despite what the docs say, I don’t see Jev leaning toward the first option, at least for plain label names. This doesn’t contradict the pre-registered experiment mentioned earlier, which found a small pull toward earlier-written options. That experiment deliberately gave Jev an input with no evidence for any option, and its author flags “whether the prior shrinks when the state carries real evidence” as the open question. On real classification items, at least, it seems to shrink to nothing.

Luna and Haiku are a different story in probability mode. On AG News, they picked whichever label came first 1.2x as often as chance; on Banking77, labels in the first quarter of the list were picked up to 1.3x as often as chance. When the correct label happened to be first, they got 93% of AG News items right, against 74-81% at other positions. GLM-5.3-Flash leaned only slightly toward the first option (1.07x on AG News, and not at all on Banking77). Interestingly, the bias mostly disappears for all three LLMs in single-label mode (every position stays within about 5% of its fair share, on both datasets).

There is one wrinkle. In a small probe I ran on a few hand-written support tickets with option descriptions, Jev’s probabilities moved more. The top probability for one ticket ranged from 82% to 97% depending on the order, though the top answer never changed. So options with descriptions might behave differently, and that is worth its own post.

4. Experiment 3: When does it matter?

This is the most practically useful result, I think. To measure how “close” a call is, I use the margin, i.e. the probability of Jev’s top option minus the probability of the runner-up (in the original ordering). If Jev gives 90% to card_arrival and 5% to card_delivery_estimate, the margin is 85 percentage points. A margin near 0 means a coin flip between two options, and a margin near 100 means Jev is sure. (Jev also returns its own confidence score, which is derived from the same probabilities. I find the margin easier to reason about.)

It turns out Jev’s flips are confined to close calls. On Banking77:

The same held on AG News, where none of the 260 items with a margin of 80+ points flipped. This means Jev’s margin works as a reliable guardrail. If you already route low-margin answers to a human or a bigger model (which TypeSafe recommends anyway), you’ve largely solved option order for free. The LLMs’ self-reported probabilities don’t behave this way; on Banking77, even when they claimed a margin above 80 points, 23-30% of Luna’s and Haiku’s answers changed under reordering. GLM did better (6%), but still not Jev’s zero.

This also fits the no-evidence experiment above. When the input doesn’t clearly favor one option, Jev seems to fall back on weak priors (like where an option sits in the list, or what it is called). A close call is exactly the situation where those priors get a say.

So what does a “close call” look like? Take “i want to track the card you sent”. Is that card_arrival or card_delivery_estimate? Jev essentially says “I can’t tell”; across rotations and repeats it put 42-74% on card_arrival and the rest on card_delivery_estimate, and its answer flipped even without any reordering. That is the kind of answer you’d want to route for review anyway. GPT-6 Luna, by contrast, said card_delivery_estimate with 88-100% probability every single time (8 calls). It was consistent, confident, and (according to the human label) wrong.

Sensitivity also grows with the number of options. On CLINC150, with the same 200 items at every list size:

• Reordering never changed Jev’s answer up to 8 options.
• At 16-64 options, it changed about 1% of answers.
• At 128-150 options, it changed about 3%, against 0.7-1.7% for identical repeats.

Accuracy dropped from 99.5% to 92.6% over the same range.

The examples suggest why. Take “it was great to speak with you” (label: goodbye). With anywhere from 2 to 64 options, Jev picked goodbye with 94-100% probability in every ordering. At 128 and 150 options, it split roughly 50/50 between goodbye and thank_you, and the winner depended on the order. The reason is simple; thank_you only made it into the randomly drawn option list at 128+ options. (Fair enough. I’m not sure I’d call that one confidently either.)

That pattern holds more generally. Of the 16 items whose answer flipped at 128-150 options, 13 had their main rival label appear only at those list sizes. So it is less that long lists confuse Jev, and more that long lists are more likely to contain a near-synonym, which creates exactly the kind of close call we saw above. (This lines up nicely with Pezeshkpour & Hruschka, who found that LLM position bias shows up mainly when the model is torn between its top two or three options.)

5. Experiment 4: Can we just average it away?

The standard fix for order bias (see e.g. Zheng et al.) is to ask several times with different orderings and average the probabilities. With an LLM, that means k separate calls, costing roughly k times as much (and taking k times as long if you run them one after another). With Jev, you can pack all k orderings into a single request as separate questions (TypeSafe calls this speculative fan-out), and they get evaluated in parallel against the same state.

To measure what is left after averaging, I compared the answers from two separate sets of k orderings. On Banking77, they disagreed:

• 6.7% of the time with one ordering each,
• 2.7% with two,
• 1.0% with four.

Packing 8 orderings into one request took 0.15s vs 0.14s for a single question, so basically no latency penalty. Input tokens went from ~1,000 to ~6,300, because each question repeats the 77-label list. At Jev’s prices, that is $0.00026 per decision instead of $0.00004.

Accuracy didn’t change (~78.5%), which makes sense; averaging makes the answer stable, not more correct. The card example from earlier shows this nicely. Averaged over the four rotations, Jev gives get_physical_card 57% and order_physical_card 43%. You now get the same answer no matter how the list is sorted, and the 14-point margin tells you it is still a close call (and, per the human label, still wrong). Packing questions together also didn’t change the individual answers. They agreed with standalone calls 98.8% of the time, which is within run-to-run noise.

6. What about OpenAI’s Decisions API?

OpenAI’s Decisions API went into public beta on October 6, a few days into writing this post. It is the closest thing Jev has to a direct competitor (the same state plus typed questions in, the same Choice, Noul and Score answers out, with GPT-6 Luna doing the work underneath). Because it accepts exactly the same request (and is still cheap!), I could run it through every test above (~28,000 calls for $1.54, plus another ~3,400 for the formatting test below).

Same request, same answer… until you add a space

The first thing I noticed was that the Decisions API never changed its answer when asked the same question twice. Not once, in thousands of repeats, with probabilities identical down to the last decimal. My first guess was caching, so I tested it. I took 100 random Banking77 messages (plus 100 borderline ones) and sent each one as the original request, and as several variants with exactly the same meaning but slightly different bytes.

It is not caching. Brand-new requests (with a never-seen request_id) came back just as fast as repeats (248ms vs 253ms median), OpenRouter billed every call in full, and even changed requests were perfectly repeatable. Reformatting the JSON or renaming the question key changed nothing at all. So the Decisions API does appear quite deterministic; it seems to compute its probabilities directly, rather than sampling them the way Jev appears to.

But it is also brittle. A single trailing space shifted its probabilities by 8 percentage points on average, about 5x Jev’s own run-to-run noise on the same messages (under 2 points), and changed its top answer for about 5% of messages (11% on the borderline ones). For Jev, whitespace made no difference beyond its usual noise. In practice, two requests that your code would treat as identical (say, with and without a trailing newline) can get different answers from the Decisions API, and it will give each of them that answer consistently.

It has the first-option bias

Remember the warning in Jev’s docs that it “leans toward the option that comes first”? On my data, that describes OpenAI’s product better than Jev. On Banking77, the Decisions API picked options in the first eighth of the list 1.17x as often as chance, sliding steadily to 0.87x for the last eighth (you can see the dashed line in the position chart). It was 83% accurate when the correct label sat near the top of the list, against 70-74% near the bottom. With 150 CLINC150 options, it was 93.5% accurate with the correct intent in the first quarter of the list, and 85.8% in the last.

It refuses some harmless questions

120 of its ~28,000 calls came back with an “OpenAI refused to answer question” error. The messages involved were not exactly edgy (“Do you support the EU?”, “be quiet”, “i am unable to say”, “Why did I get blocked?”). Stranger still, refusals depended on the request rather than the message. The same message would be answered under one option order and refused under another, consistently. “Is a copy of the police report necessary for completing the report process?” was refused every time as a plain message, but answered in 4 of 5 tries once I wrapped it with an irrelevant request_id field. Jev never refused anything. For a classifier sitting inside production code, a refusal is a hard failure you have to handle.

Averaging helps it even more

Packing several orderings into one request works for the Decisions API too, and matters more. On Banking77, disagreement between two sets of orderings fell from 15.8% with one ordering to 10.8% with two and 5.7% with four. Unlike Jev, its accuracy also improved, from 75.6% to 78.8%, which is what you’d expect when the order effect is a systematic bias rather than noise. (See the averaging chart.)

Its confidence is less useful for routing

If you only auto-handle answers above a confidence threshold that keeps you at 95% accuracy, Jev could handle 87% of AG News items against the Decisions API’s 54% (and 42% vs 38% on Banking77). Part of the reason is overconfidence; on AG News, the Decisions API put 100% probability on its answer for 76% of items, while being right 90% of the time.

To be fair to OpenAI, this is a public beta that was a few days old when I tested it, so its behavior (and price) may well change. But as of now, the more mature product is Jev.

7. So what should you do?

If you are building on Jev (or comparing it against an LLM), here is what I’d take away:

• With a handful of options, don’t worry about order. In the CLINC150 test (same items, only the list length changes), reordering never changed Jev’s answer with up to 8 options, and changed about 3% of answers with 128-150. On Banking77 (77 options), it added about 3 points on top of Jev’s run-to-run noise. (But… should you really be building a single-level classifier with dozens of options? Consider a layered classifier, i.e. a tree, if you have such needs. TypeSafe has a cookbook for exactly this.)
• Gate on the margin. When the top option was more than ~60 percentage points ahead of the runner-up, Jev’s answers essentially never changed under reordering (fewer than 1% of those Banking77 items flipped).
• If stability matters, pack 2-4 orderings into one request and average. It adds no latency and costs fractions of a cent.
• When comparing Jev against an LLM, use single-label mode for the LLM and fix the option order for both. Otherwise you’re partly benchmarking your enum’s sort order and the prompt format.
• Don’t rely on key order in Jev’s probabilities response. It comes back in random order. (Harmless, but it surprised me.)
• If you use OpenAI’s Decisions API, normalize your inputs and handle refusals. Strip whitespace and keep your request format fixed (since tiny formatting changes can change its answer), treat “refused” errors as a separate outcome, and average over a few orderings, which removes much of its first-option bias and improves accuracy.

Caveats and open questions

• LLM costs include prompt caching. OpenAI cached most of GPT-6 Luna’s repeated prompt (the long Banking77 label list), which cut its input cost by about 90%. Without caching, it would have cost several times more per call.
• LLM probabilities are self-reported. The LLMs wrote out their probabilities as text; I didn’t use token logprobs. Logprobs might behave differently.
• Two LLMs needed workarounds. Haiku’s structured-output mode rejected a 77-label schema (“compiled grammar is too large”), so it used the adapter’s plain JSON mode on Banking77. Z.ai’s endpoint for GLM-5.3-Flash ignores JSON schemas altogether, so GLM used plain JSON mode throughout (and about 1 in 150 of its Banking77 answers needed a re-run to produce valid JSON). GLM also can’t have reasoning switched off; at “minimal” it used only a handful of reasoning tokens per call (about 5-13 on average), so this shouldn’t change the picture much.
• Limited scope.
  • 200-300 items per dataset, English only.
  • Public datasets that are probably in every model’s training data.
  • Plain label names only.
  • Only Jev 1.13 and the October 2026 beta of OpenAI’s Decisions API, both called through OpenRouter.
• The formatting test is small (100 random and 100 borderline Banking77 messages). The probability shifts are clear-cut, but at the level of answers on typical messages, the confidence intervals for Jev and the Decisions API overlap (about 2-3% vs 4-8%).
• Option descriptions. Do longer option descriptions make Jev more order-sensitive? The probe hints yes; I’ll test this properly in a later post.

The code and raw results for this post are available upon request. (Thanks to Claude for helping me build the harness and run the experiments. All OpenRouter spend was my own.)