Health and Monitoring - NLP

Back to NLP Topics

The Health and Monitoring dashboard provides goal-driven insights into your app’s NLP model performance. It analyzes training data, test coverage, and test results to surface actionable recommendations.

Go to: Testing > Regression Testing > Health & Monitoring

---

Dashboard Sections

Section	Description
NLP	Aggregates batch test results. Shows performance metrics and test coverage for Dialog intents, FAQs, Small Talks, Traits, and Entities.
Flow	Summarizes conversation flow coverage. Shows transition coverage and intent summary from conversation test suites.

---

NLP Metrics

Displayed as aggregate values in the Bot Health summary and as individual scores in the intent-type panels (Dialog Intents, FAQs, Small Talk, Traits).

Metric	Description
Accuracy	Whether the identified intent is correct.
F1 Score	Weighted average of Precision and Recall.
Precision Score	Ratio of true positives to all predicted positives (TP / (TP + FP)).
Recall Score	Ratio of true positives to all actual positives (TP / (TP + FN)).
Total Test Coverage %	Average test coverage across Dialog Intents, FAQs, Small Talk, Traits, and Entities.

---

Test Cases Detailed Analysis

Click View Test Cases in the NLP section to open the detailed analysis window. It shows results for Intents, Entities, and Traits.

Intents Tab

Column	Description
Test Cases	Test case name.
Intent Type	Dialog intent, FAQ, or Small Talk.
Expected Intent	Intent expected from the utterance.
Matched Intent	Intent actually matched.
Result Type	True Positive, False Positive, or False Negative.
Tags	Follow-up labels assigned by the analyst.

Entities Tab

Column	Description
Utterances	The user utterance in the test case.
Entity Name	Entity name mapped to the test case.
Expected Value	Entity value expected.
Matched Value	Entity value actually matched.
Entity Result	True (matched) or False (not matched).
Tags	Follow-up labels.

Traits Tab

Column	Description
Test Cases	Trait test case name.
Intent Type	Displays “Trait”.
Trait Name	Name of the trait analyzed.
Expected Trait	Trait expected from the utterance.
Matched Trait	Trait actually matched.
Trait Result	True Positive, False Positive, or False Negative.
Tags	Follow-up labels.

Tags

Tags are labels for intent, entity, and trait test results that indicate follow-up actions.

Tag	Meaning
Add Negative Pattern	A negative pattern should be added.
NeedNLPHelp	Requires explicit NLP support.
Needs Negative Pattern	Needs a negative pattern to work as expected.
Needs Training	App needs training for this intent/entity/trait.
New Intent	A new intent was detected during execution.

---

NLP Analysis (per Test Case)

Click the NLP Analysis tab in the test case detail view to see the historic NLP analysis captured at the time of execution. Shows qualified (definitive and probable) and disqualified intents for:

• Traits (if applicable)
• ML engine
• FM engine
• KG engine
• Trait Rule (if applicable)
• Ranking and Resolver

This differs from Utterance Testing, which shows current analysis based on the latest training data. NLP Analysis here is the snapshot from when the test ran.

---

NLP Performance Metrics (per Intent Type)

Result Type	Description
True Positive (TP)	Utterances that correctly matched the expected intent.
False Positive (FP)	Utterances that matched an unexpected intent.
False Negative (FN)	Utterances that did not match the expected intent.

---

Performance Metrics Table

The details window provides a drill-down view for intents, entities, and traits:

Metric	Intent	Entity	Trait
Expected Intent/Value	Yes	Yes	Yes
Matched Intent/Value	Yes	Yes	Yes
Parent Intent	Yes	No	Yes
Task State (Configured/Published)	Yes	No	Yes
Result Type	Yes	No	Yes
ML / FM / RR Scores	Yes	No	Yes
Entity Name	No	Yes	No
Result (True/False)	No	Yes	No
Identified by (NLU engine)	No	Yes	No
Identified using (entity type reference)	No	Yes	No
Confidence Score	No	Yes	No

---

Dialog Intent Summary

Test Coverage

Shows the count and percentage of covered vs. uncovered intents. An intent is covered if it has at least one test case in the selected suite(s). Use View details to find uncovered intents and add test cases.

Recommendations

Training recommendations appear when errors or warnings are triggered during execution. Click View Recommendations to see the summary and corrective actions.

---

Intent Details Window

Click View Details in any summary panel (Dialog Intents, FAQs, Small Talks) to open a drill-down of performance metrics and recommendations.

Training Data Summary columns:

Metric	Dialog Intent	FAQ	Small Talk
Intent	Intent name	FAQ intent name	Small Talk intent name
Utterances	Count of training utterances	N/A	—
Test Cases	Count in selected suite(s)	—	—
True Positive (TP)	Count of TP results	—	—
False Negative (FN)	Count of FN results	—	—
False Positive (FP)	Count of FP results	—	—
Covered In	Test suite names	—	—
F1 / Accuracy / Precision / Recall	Recommendation scores	—	—
Recommendations	Count + corrective actions link	N/A	N/A
Group	N/A	N/A	Small Talk group
Path	N/A	KG node path	N/A
Alt Question	N/A	Count of alternate questions	N/A

View Intents Not Covered: Click the three-dot menu on the panel to list intents not covered in batch testing. Add them to your training data to improve coverage.

---

FAQ Summary

Shows recommendation scores for FAQs from the latest batch test. Click View Recommendations to review the last report. Click Knowledge Graph to navigate to KG Analysis.

Small Talk Summary

Shows recommendation scores for Small Talk interactions from the latest batch test. Click Small Talk to view group names, user utterances, and bot responses.

Trait and Entity Summary

Shows recommendation scores for traits and entities from the latest batch test. Use Test Coverage and Test Results Analysis in each panel for details.

---

Utterance Testing from Health and Monitoring

Click the go to utterance testing (magic wand) icon on the Test Cases Detailed Analysis page to open the Utterance Testing window. From there you can retrain the app based on test failures. See Utterance Testing.