TypeSafe Jev: Decide Which Support Requests Need a Person

A clear billing question may go straight to billing. “The invoice export is wrong” needs more care: is the charge wrong, or is the download broken? Jev, an AI model from TypeSafe AI, can suggest a team by reading the message. Your support application still needs rules for deciding when a person should check that suggestion.

Keep requests for human help, unclear problems and actions requiring approval on a review path. For the remaining requests, test whether Jev's suggestions are reliable enough for the action you intend to allow. Its confidence number helps compare answers, but it cannot establish that an individual answer is correct.

I tested eight fictional support messages with Jev. For the invoice-export message, I had written no_match before the test, meaning a person should clarify the problem. Jev chose product_help with confidence 0.81. Applying two different review rules to that saved answer changed whether the example code suggested product support or human review.

Understand the two numbers in a team suggestion

A Jev Choice question asks the model to select one option from a list you supply, such as billing, login support or product support. It returns the chosen option, a probability for every option and a separate confidence value. Choice documentation

For “The invoice export is wrong,” Jev assigned probability 0.86 to product_help and returned confidence 0.81. The first number is the probability assigned to that particular option. Confidence summarizes how probability is spread across all the options. It is higher when probability is concentrated on one choice and lower when several choices have similar support. Confidence documentation

Neither number tells you what percentage of your customers will reach the correct team. To measure that, compare the suggestions with decisions checked by your support team on requests that represent its work.

TypeSafe describes its models as trained for calibration: probabilities are checked against outcomes across groups of predictions. Calibration concerns that overall relationship. TypeSafe explicitly says it does not guarantee that an individual answer is correct. System One documentation

Ask separately whether the customer wants a person

A message can clearly concern product support and also say, “Please let me speak to a human support agent.” Recognising the topic should not override that request.

Jev's Noul question returns the probability that the answer to a yes/no question is yes. For “Did the customer ask to speak to a person?”, a value near 0.50 means yes and no have similar probability. It does not describe how strongly the customer wants help. Noul has no separate confidence field. Noul documentation

In the test, “My CSV export keeps timing out. Please let me speak to a human support agent” received product_help with confidence 1.00. The separate probability that the customer requested a person was 0.99. Looking only at the first answer would miss a clearly expressed preference.

Agree which requests always need review

Sending a message to a team's queue, meaning its list of waiting requests, has different consequences from issuing a refund or granting account access. Set the rules for each action before using a confidence threshold, which is the minimum confidence your application will accept.

See how a higher threshold changes the workload

The fictional test used four options: billing, account_access, product_help and no_match. The code kept no_match results for human review. It also required review whenever the probability that the customer asked for a person exceeded 0.10.

I kept those rules fixed and applied two minimum confidence values to the saved team suggestions:

The invoice-export message explains the difference. Its confidence of 0.81 passed the first rule and failed the second. Raising the threshold therefore added one request to human review.

I chose 0.10, 0.80 and 0.90 after seeing the response, solely to show how the rules work. These eight cases cannot establish a suitable threshold for a real operation. Zero disagreements among four suggestions does not establish that future suggestions will be correct. A new set could contain wrong answers above either threshold.

The comparison required no further call to Jev and changed no help-desk record. The code applied different rules to the same saved answers. Developers call this a replay: it lets you examine what a rule would change without asking the model to answer again.

You can inspect the test response, answers written before the test and comparison results. The call used jev-1.13.0 on 20 September 2026, with 2,826 input tokens and 539 output tokens. I wrote all the fictional cases and expected answers before calling Jev; the labels were not independently assigned by another reviewer.

A separate source-check test showed why confidence cannot clear a claim

I also sent Jev eight short source-and-claim pairs. The task was only to classify whether each source supports, contradicts or is insufficient for its claim. My separate reading of the supplied evidence gave four supports, three contradictions and one insufficient. Jev returned those same labels.

This was a narrow consistency check on eight authored pairs, not a test of general factual accuracy. Two results show why the application still needs a review policy. For the false claim that a Noul value near 0.50 measures “medium intensity,” Jev chose contradicts with confidence 0.42; its probabilities were 0.62 contradicts and 0.38 insufficient. For the unsupported claim that 672 scheduled runs would cost exactly 134.4 times as much money as five scheduled runs, Jev chose insufficient with confidence 0.78; the source gave run counts but explicitly did not measure dollar cost.

The selected labels matched the expected labels in this small check. The confidence values did not turn the labels into automatically safe facts. A production claim-review workflow would still need representative material, independent expected decisions, a rule for low-confidence or high-consequence claims and a person who owns the final statement. You can inspect the test-case summary and raw response. The summary paraphrases the documentation excerpts and includes the original request hash; it is not the exact request that produced the recorded response.

Test requests that represent your customers' work

For each test request, record both the right team and whether a person needs to review it. A message can belong to billing while also needing someone to check account permissions. Recording only “billing” would hide that requirement.

Include ordinary requests, unclear wording, overlapping team responsibilities, requests for a person and tasks your service does not support. Include cases where a missing fact changes what the team should do. Use the languages and writing styles your customers use, with data your organisation is authorised to process.

Have a support reviewer decide the expected answers before comparing them with Jev's suggestions. When reviewers disagree, record the disagreement and clarify the rule. Confusing team boundaries can affect both people and the model.

Keep two sets of requests separate. Use the first to improve questions and choose candidate thresholds. Keep the second unused until you have selected the rule, then test it there. Reusing only the cases you already improved can make results look better than they will on new requests.

Measure mistakes and review time together

A support lead needs to know how much work the rule would automate, which mistakes remain and how much work people would receive.

Report counts with percentages. Zero errors in four assignments provides very different evidence from zero errors in thousands of representative cases. Record the consequences too: an extra handoff differs from exposing information, changing access or moving money.

If no threshold gives acceptable errors and a manageable review workload, change the workflow. Clarify team responsibilities, ask customers for a missing detail, reduce the actions allowed automatically, or keep people assigning the requests. A different threshold cannot fix every cause of error.

Translate the review rate into staffing

The following numbers are a fictional planning example, not results from Jev. Suppose a separately reviewed set has 1,000 tickets. Rule A would assign 800 automatically, including 16 wrong assignments, and send 200 for review. Rule B would assign 600 automatically, including three wrong assignments, and send 400 for review.

Rule B leaves 13 fewer wrong automatic assignments and requires 400 additional review minutes. A support lead can now compare the consequences of those 13 mistakes with the additional work. Neither rule is universally preferable. A team with only 500 minutes of review capacity would need more capacity, a narrower automatic-assignment scope or a better clarification step before choosing Rule B.

Use actual review times when available; a complex account-access ticket may take longer than a simple category check. Count corrections to wrong automatic assignments separately, so the calculation includes work that arrives after the initial routing decision.

Compare suggestions before changing live assignments

At first, let people assign requests normally while Jev suggests destinations in the background. Compare the two. Look for patterns by team, language and missing information, so a good overall average does not hide a recurring problem.

For a limited pilot, record the model version, question wording, team definitions, thresholds and rules requiring review. Name the owner who can stop automatic assignment. Review a sample of high-confidence suggestions as well as the unclear ones; mistakes can appear in both groups.

The support-system owner must approve live use and the queues it covers. Check whether assigning a queue also triggers a message or another action. Record each assignment with the rule version, and provide a way to turn off automatic assignment.

Repeat the evaluation when the model, rules or incoming requests change. TypeSafe warns that names such as jev-latest can point to newer model versions over time. A threshold tested with one version needs checking before it is used with another. Model version documentation

For the request format and code that applies these checks, see how to sort support requests with Jev. Follow Poorna Reddy on LinkedIn for practical AI examples that show the decision, the result and the checks a team needs.