Utterance Testing - NLP

Back to NLP Topics

Utterance Testing lets you enter user inputs and see how the NLP engine processes and matches them. Use it to validate intent identification and train the app directly from test results.

---

How to Test

1. Open the app and go to Testing > Utterance Testing.
2. (Optional) For a multiple-intent model, select the Intent Model to scope ML detection.
3. In the Type a user utterance field, enter an utterance. Example: Book a flight.
4. Review the result: single match, multiple matches, or no match.

---

Match Types

Match Type	Description	Behavior in Production
Definitive Match	High-confidence exact match.	Single: task executes directly. Multiple: user is prompted to choose.
Probable Match	Reasonably scored but not definitive.	Sent as “Did you mean?” suggestions to the user.

Possible outcomes:

• Single match: Correct — proceed or train further. Incorrect — mark as incorrect and select the right intent.
• Multiple matches: Select the correct radio button and train.
• Unidentified intent: Select an intent from the dropdown and add training.

---

Entity Match

Matched entities are displayed after intent detection. Processing order: NER and pattern entities first, then remaining entities.

Entity match details (v8.0+):

Field	Description
Identification Engine	ML, FM, or KG
Training Type	NER, pattern, entity name, system concept, etc. Click for pattern match details.
Confidence Score	ML NER score (only when Conditional Random Field is the NER model).

---

NLP Analysis

Each test result includes an NLP Analysis box showing shortlisted intents, which model found them, scores, and the winning intent.

NLP scores are absolute values. They can only be compared against other tasks for the same input, not across different utterances.

ML Model

Matches the input against task labels and training utterances. Multi-sentence inputs are tested sentence by sentence. More training utterances increase discovery chances.

FM Model

Scores every task using a custom NLP algorithm based on task names, synonyms, and patterns. Click Processed Utterance to see how the input was analyzed.

FM scoring factors:

Factor	Description
Words Matched	Score for the number of words matching the task name or utterances.
Word Coverage	Ratio of matched words to total task words (name, fields, utterances, synonyms).
Exact Words	Words matched without synonyms.
Sentence Structure	Bonus for sentence structure match.
Word Position	Score for words near the start of the sentence.
Order Bonus	Bonus for words in the same order as the task label.
Role Bonus	Bonus for matched primary/secondary roles (subject/verb/object).
Spread Bonus	Bonus based on distance between first and last matched words.
Penalty	Penalty for phrases before the task name or conjunctions in the task label.

FM scoring varies by language:

• German/French: Uses word roles, Universal Parts of Speech, and dependency relations.
• Other languages: Uses original word, role in sentence, and processed word (spell-corrected).

Knowledge Graph

Extracts terms from the utterance and maps them to the KG to fetch relevant paths. Paths covering more than a threshold number of terms are shortlisted. A path with 100% term coverage and a matching FAQ is a perfect match. See Knowledge Graph Training.

Ranking and Resolver

Determines the final winning intent across all engines.

• If ML or KG finds a perfect match, R&R presents it without rescoring (even multiple perfect matches are shown as options).
• All other good/unsure matches are rescored using the FM model. If the rescored intent exceeds the threshold, it is also considered a match.

R&R V1 intent elimination reasons:

• Intents matched only by entity values (date, number) from ML are eliminated.
• Possible matches are eliminated if a definitive match exists.
• A definitive match is eliminated if another definitive match appeared earlier in the utterance (multi-intent case).
• Intent patterns following a definitive match are eliminated.
• Intents below the minimum threshold are eliminated.
• Definitive matches against a negative pattern are eliminated.
• Intents with unmet pre-conditions are eliminated.
• KG “Search In Answer” definitive matches are eliminated if another match exists.

R&R V2 ranks only definitive matches from ML and KG (not FM) without rescoring. The R&R window shows the winning intent, ranking score, eliminated intents and their reasons, and the final result. See Ranking and Resolver.

---

Training from Utterance Testing

Utterance-Based Training

1. After testing, open training options:
   • Unmatched intent: Select an intent from the Select an Intent dropdown.
   • Multiple matches: Select the radio button for the correct intent.
   • Single match: Click the matched intent name.
2. The utterance appears in the ML Utterances field. Click Save to add it to the intent.
3. Under Intent Synonyms, add synonyms for each word in the task name.
4. Under Intent Patterns, add task patterns.
5. Click Re-Run Utterance to verify improved scores.

Matched Training Data

The Ranking and Resolver NLP Analysis window shows the training data (utterances, synonyms, questions, patterns, traits) that led to intent qualification.

NLP Engine	Supported Models	Matched Training Data Shown
Machine Learning	Standard, Few-shot	Sample utterance for every qualified intent.
Knowledge Graph	Ontology, Few-shot	Questions/alternate questions of qualified FAQs.
Fundamental Meaning	All models	Patterns for qualified intents.

For qualified and eliminated intents, the system also shows the elimination reason alongside the matched utterance, processed utterance, and winning intent.

Train with FAQ

To match utterances to FAQs:

• Set terms, term configuration, or classes from the FAQ page, train the KG, and retest.
• Add the utterance as an alternate question to the FAQ from the Knowledge Graph page, train, and retest.

See Knowledge Graph Training.

Mark an Incorrect Match

When a user input matches the wrong task, mark it as incorrect and select the right intent to retrain.