We want to know how we can analyze the failed facial match transactions. Do we have any report on the SAFR portal side that can show us the reason for the mismatch with the live feed?
Answer
First, I’ll define some terminology I’ll use:
- FMR: False Match Rate – SAFR got face image off camera but did not match to a person in dB.
- FMNR: False non-Match Rate – SAFR got face image off camera and matched to wrong person in dB.
- Match types:
- Known – Face image that was matched to person in dB
- Stranger – Face image for which we tried but failed to match to person in dB
- Unrecognizable – Face image for which there was insufficient quality in the image captured from camera. This means that the image did not meet the configured thresholds of face size, sharpness, contrast and center pose.
There is a way to get some insight into which images were probably false non-matches and which were false non-matches. But, there is no way to distinguish one from another without human intervention. More on why below.
Viewing/reporting false matches
As described above, SAFR reports either a stranger or a known match. A stranger does not carry with it any sort of designation that it may have matched someone close. But you can configure SAFR so that it is less likely to report someone as a stranger and instead report as a possible known match. Specifically you can adjust the recognition threshold and the proximity threshold so that instead of yes/no type designations, you get a yes/maybe/no designation. This manifests itself in the event as either “Known” face with 100% positive confirmation (“yes”), “Known” face with <100% possible match (“maybe”) or Stranger (no match / “no”). No match is technically still just a confidence lower than a threshold set (e.g. 75% match which in practice means there is no way it’s the same person).
The following settings in Preferences > Recognition are how you configure the necessary thresholds:
The “Identity recognition threshold” on the “Camera” row defines how “close” a face needs to be to the reference image in order to be considered a match. Anything outside of this threshold (more different than reference face) is considered a Stranger. The “Proximity threshold allowance” defines a range (additive to the Identity recognition threshold) that SAFR will use to consider something a close match. In this case since Proximit Threshold is 0, a face will be reported as either “Known” or “Stranger” – not in between.
To get probability match, set Proximity threshold allowance to greater than 0 (and it is recommended to lower the Identity recognition threshold as follows:
With above settings, you will see people who are close matches get reported btween 80 and 100%. If 100% then SAFR Event shows up as Known but if <100% then SAFR reports as “Probable match”. This information is included in the events as match confidence, but please understand that the user interface or existing reports do not provide a way to group or count based on the match confidence. If you were to export events from the REST APIs, this information is available and could be used in custom reports you or your customer build.
This group of “Probable match” could be a false match or false non-match.
The following helpdesk article explains the two thresholds in more detail:
I should put a caveat on the information above. SAFR can get very poor-quality images from a camera. From this it may not be able to make a match and thus a face that was sent to safr from the camera can still be a known person in the database for which SAFR did not match. Or it can even be a reasonably good quality face image and for whatever reason SAFR did not identify match above the configured threshold and thus reported as stranger. This is the nature of facial recognition algorithms – they are not based on exact science but rather more like probability models.
Viewing/reporting false non-matches
This would be the group of faces that we missed to match even though the face was indeed in the dB. Again, like with False matches, it will be possible to have this group show up in the “Possible Match” group defined by a non-100% match confidence. Obviously still some probability “N persons in 10,000” that a facial recognition algorithm does not match even with proximity threshold set. In NIST reporting this is known as “FNMR” (false non-match rate).
Quality Thresholds that determine IF recognition is attempted
I should also point out that SAFR has thresholds that determine if a given image from the camera will be sent to the recognition workflow. If an image has low quality (as measured by face size, sharpness, contrast, center pose/face gaze direction, and occlusion) then SAFR will not even try to recognize and will leave the face reported as Unrecognizable as defined above. There are two different thresholds that are set in SAFR Preferences:
- Threshold for SAFR to try to recognize a face. If lower than this, SAFR will not even send the face for recognition
- Threshold for SAFR to characterize a face as a stranger (by default higher than recognition threshold) – If lower than this SAFR will not change status from “Unrecognizable” to “Stranger” unless above this threshold, even if face was sent to the server for recognition.
So you will find cases where a face was above the recognition criterial but was not reported as stranger because they were not above the stranger threshold. BUT, if a face is matched during the recognition process, the face will always be reported as “Probable match” with a corresponding match confidence value.