What is Confusion Matrix?
A confusion matrix is the visualization of the performance of an AI model, in the field of machine learning. At ultimate.ai, Confusion Matrix is to help you understand if the AI model of your bot is performing well in terms of intent recognization. Or, is it "confused" by the similar expressions under two different intents?
Why is it important?
AI Technology is the heart of ultimate.ai. Not only do we ensure the performance of your virtual assistant with our technology, empowering you by sharing that technology is also something we strive for.
With the launch of this new feature, Confusion Matrix, we hope to bring you the same tools first class AI scientists are using, so you can easily monitor how well your bot is performing with just a few clicks - no coding required.
When to use it?
You should use it when you notice Bot Answered Rate in Analytics is decreasing (a good baseline is 80%), or when you notice that your bot constantly recognizes a message as an intent as opposed to the other.
How to read it?
The two dimensions, X and Y axis, represent the actual and predicted intents, respectively.
In the example above, column 1 and row 1 are the actual and predicted intent Affirmative, row 2 and column 2 are the intent Negative, so on and so forth.
The darker a cell is, the more a predicted intent overlap with an actual one.
A good model, like the example above, should have a dark line running across the table diagonally.
However, what you should pay attention to are the colored cells outside of the diagonal as they represent how often messages in an intent are predicted to another. Whenever you see cells with darker colors outside of the diagonal, this is an indicator of a problem in the accuracy of the AI model caused by potential overlaps of expressions between two intents. These can be tackled in a systematic manner in the List of Issues view where all overlapping issues are listed according to priority automatically.
To understand how to use the feature, check out keeping my bot smart using Confusion Matrix?