Jev Confidence Scores Explained: How to Gate and Size AI Trades

Related: Jev AI for Trading: What TypeSafe's New Model Can and Cannot Do

This article is general information, not financial advice. The thresholds and sizes below illustrate the method. They are not recommendations.

Most AI models give you an answer. Jev, the decision model from TypeSafe AI, gives you an answer, a probability for every possible answer, and a confidence score. In trading, those extra numbers matter more than the answer itself. This guide explains Jev confidence scores and probabilities, and how to use them safely. For the basics, see Jev AI for trading.

Jev probability versus Jev confidence

Ask Jev a choice question such as "buy, sell or hold?" and you get two kinds of number:

• Probabilities: one per option, for example buy 0.62, sell 0.08, hold 0.30. They show what the model thinks.
• Confidence: one number from 0 to 1 that summarises how concentrated those probabilities are. It shows how sure the model is.

When the probability piles onto one option, confidence is high. When it is spread out, confidence is low, even if one option is slightly ahead. Score questions return a confidence score too. Noul questions, the yes or no type, return a single probability, and a value near 0.5 means the model cannot tell.

TypeSafe's docs illustrate confidence with a simple approximation for three options: (3 × the top probability − 1) ÷ 2. Using it:

• Top probability 0.34, almost a three-way tie: confidence about 0.01.
• Top probability 0.62: confidence about 0.43.
• Top probability 0.90: confidence 0.85.

So a "buy" at 62 percent is a weaker signal than it first looks. Armin Ronacher, quoted by TechCrunch, made the same point about Jev: an answer that comes back at 50 percent may simply be a coin toss.

TypeSafe's suggested confidence bands

TypeSafe's documentation suggests three bands as a starting point:

It stresses that the right threshold depends on the cost of being wrong. One of its examples sets a floor of 0.6 for any action, and requires 0.85 before approving a money transfer automatically. In TypeSafe's words, "your code encodes the risk tolerance."

Mapping Jev confidence to trading actions

Here is one way a trading system could use those bands. It is an example of the method, not a recommendation:

Three rules keep this safe:

• Confidence can shrink a position, never push it past your limit. Maximum size comes from your risk rules, not from the model.
• Start with a few bands, not a formula. Steps are easier to test and explain than size as a smooth function of a probability. Formulas such as the Kelly criterion assume the probabilities are accurate, and unchecked model probabilities are exactly what you should not trust yet.
• Exits ignore confidence. A stop loss fires whatever the model thinks.

Much of the value is in the trades you skip

If Jev's confident calls really are right more often than its unsure ones, skipping the unsure ones raises your hit rate and cuts your trading costs at the same time. You trade less, and better. Whether that holds in your market is exactly what you need to test.

Is Jev calibrated? Check it yourself

A model is calibrated when its probabilities match reality: of all the times it says 70 percent, it should be right about 70 percent of the time. TypeSafe trains Jev specifically for calibration. But Jev was trained on synthetic data, and your market is not synthetic. Check it on your own data.

1. Log every answer with its probabilities, plus the outcome you care about, measured the same way every time. For example: did the price rise more than 0.2 percent in the next five minutes?
2. Group predictions into buckets by probability: 0.5 to 0.6, 0.6 to 0.7, and so on.
3. Compare each bucket's average prediction with how often the outcome actually happened.

A made-up example log:

Here the model is overconfident above 0.6. Its 75 percent calls come true only 58 percent of the time, so you would raise your thresholds, or recalibrate the probabilities before using them.

Two numbers help you track calibration over time:

• Brier score: the average of (probability − outcome)², where the outcome is 1 or 0. Lower is better. Always answering 0.5 scores 0.25, so a useful model must beat that.
• Hit rate by confidence band: simple, and easy to explain to anyone.

You need plenty of data. A few dozen trades tell you almost nothing. The open-source bot buberlo/jev-trader shows the right level of seriousness: it logs every state, decision and outcome, reports Brier score, log loss, expected calibration error and reliability curves, and can refit its thresholds from the results.

Jev confidence pitfalls TypeSafe warns about

• Do not reuse thresholds across question types. A good cut-off for a choice question does not transfer to a score or noul question.
• Do not assume related questions add up. "Is this bullish?" and "Is this bearish?" asked separately need not sum to 1.
• Do not read exact sizes from a score. A trend score of 2.4 does not mean the trend is 2.4 units strong.
• Retest after every change. A new Jev version, new state labels or reworded questions can all shift the numbers.

Write the bar down before you look

A developer on DEV Community who uses Jev to check a forex bot's decisions shows a good habit. They wrote the scoring rules into the code before the first result arrived, and committed in advance to dropping Jev if the data did not support it. That stops you moving the goalposts after seeing results, which is the easiest way to fool yourself in trading.

In that project Jev predicts nothing. It checks whether the direction another model chose is consistent with the reasoning that model wrote, and the test is whether inconsistent calls lose more money. It is a narrow, testable use of Jev, and a good template. More on combining models in Jev vs ChatGPT and other LLMs for trading.

Checklist

1. Use confidence, not just the top answer.
2. Start from TypeSafe's bands, then set your own from data.
3. Let confidence reduce size, never exceed your limits.
4. Keep exits independent of the model.
5. Log every prediction with its outcome.
6. Check calibration by bucket, and track the Brier score.
7. Write your success bar down before you test.
8. Retest after any model or question change.

To put this into a working system, see how to build a Jev AI trading bot. If you want help designing and testing one, talk to us.

Jev confidence score FAQ

What is a Jev confidence score?

A number from 0 to 1 returned with Jev's choice and score answers. It summarises how concentrated the model's probabilities are. High means the model strongly prefers one answer. Low means it is unsure.

What is the difference between Jev probability and confidence?

Probabilities show how likely the model thinks each option is. Confidence is a single number derived from the shape of those probabilities, showing how sure it is overall.

What confidence threshold should I use for trading?

There is no universal number. TypeSafe suggests acting automatically above 0.9 and not acting below 0.5, with caution in between. For trading, set thresholds from your own logged results and the cost of a wrong trade.

Are Jev's probabilities accurate?

TypeSafe trains Jev to give calibrated probabilities, but independent checks are only just starting. Test calibration on your own market data before trusting the numbers.

Does a Jev noul answer have a confidence score?

No. A noul answer is a single probability that the statement is true. Values near 0 or 1 are decisive, and values near 0.5 mean the model cannot tell.

Sources

• TypeSafe docs: Confidence
• TypeSafe docs: Confidence-gated routing
• TypeSafe docs: Jev 1.13 known weaknesses
• buberlo/jev-trader on GitHub
• DEV Community: JEV assisted LLM trading
• TechCrunch: A new kind of AI model from a ChatGPT inventor is thrilling developers