Case study / Logit-level scoring, Oct 2026
Logit Scoring Ticket Router
A local LLM decision engine that routes support tickets from next-token logits.
The ticket router reads next-token logits on SGLang /v1/score (Llama 3.2) to route support tickets. Each decision is a single prefill with a restricted softmax over label tokens and a margin policy for automatic routing or human review.
Results
- 4.5x to 5.6xDecision latencyFaster decision latency for logit scoring.
- 0Unparseable outputsWith logit scoring, compared with 4 to 6 failures per 30 tickets in free generation.
- 3Lanes benchmarkedLogit scoring, constrained decode, and free generation.
Why logit scoring
Free generation is a slow and fragile way to make a routing decision. In the benchmark, free generation produced 4 to 6 unparseable outputs per 30 tickets.
The router scores the label tokens directly. One prefill produces the logits, a restricted softmax turns them into label probabilities, and a margin policy decides between an automatic route and human review.
Three lanes were benchmarked: logit scoring, constrained decode, and free generation. Logit scoring delivered 4.5x to 5.6x faster decision latency with zero unparseable outputs.
Request path
Each routing decision follows the same path, from ticket text to an automatic route or human review.
Receive the ticket
The support ticket and the label tokens go to a local Llama 3.2 model served by SGLang.
Output: Ticket and label tokens
Run a single prefill
The router calls SGLang /v1/score and reads next-token logits. Each decision is a single prefill.
Output: Next-token logits
Restrict the softmax
A softmax over the label tokens only turns the logits into label probabilities.
Output: Label probabilities
Apply the margin policy
The margin policy decides between an automatic route and human review.
Output: Automatic route or review flag
Hand off to a person
Tickets that do not clear the margin go to human review.
Output: Human routing decision
Benchmark three lanes
Logit scoring, constrained decode, and free generation were benchmarked for decision latency and unparseable outputs.
Output: 4.5x to 5.6x faster, zero unparseable outputs
Design decisions
Logit scoring instead of free generation
- Choice
- Read next-token logits on SGLang /v1/score instead of parsing generated text.
- Why
- Each decision is a single prefill. Logit scoring delivered 4.5x to 5.6x faster decision latency with zero unparseable outputs.
- Tradeoff
- The model chooses only among the defined label tokens.
A margin policy before automation
- Choice
- Route automatically only when a decision clears the margin policy.
- Why
- Uncertain decisions go to a person.
- Tradeoff
- Low-margin tickets add human review work.
Components
- Local model serving
- Llama 3.2 served locally with SGLang. Built in Python with CUDA and ROCm deployment support.
- Logit scoring endpoint
- SGLang /v1/score returns next-token logits for the label tokens.
- Restricted softmax
- Converts label-token logits into probabilities for a single decision.
- Margin policy
- Separates automatic routes from tickets that need human review.
Safeguards
- Low-margin decisions go to human review
- Every decision maps to a defined label token
- Zero unparseable outputs in the logit scoring lane
- Results describe this benchmark on Llama 3.2 with SGLang.
- The router runs on a local model.
Architecture walkthrough available on request.