In the context of facial recognition technology, "detection" refers to the process of finding objects in videos for subsequent processing. "Recognition" refers to the process of creating biometric signatures and matching faces. SAFR separates the detection and recognition processes into two distinct services. Although SAFR can run both processes on a single machine, it's helpful to discuss them separately in terms of load effect. This article describes the recognition process, which occurs on SAFR Servers. The detection process is discussed separately in Auto scaling and manual scaling of detection loads in VIRGO.
SAFR Server dynamically scales recognition loads by varying the frequency of recognition attempts on any given face. This is done based on client configurations coupled with server recognition response times.
Settings that Affect Recognition Scaling
To understand how this works, it's best to review SAFR settings that pertain to recognition frequency.
| VIRGO Feeds Attribute | Location in the SAFR Desktop Client | Default | Phase | Description |
|---|---|---|---|---|
| tracker.initial-recognition-attempts | Tracking Preferences menu > Initial recognition attempts | 3 (integer) | Initial recognition attempts | The number of initial recognition attempts to make on an unrecognized person as fast as possible. |
| tracker.failed-recognition-back-off-interval | Tracking Preferences menu > Failed recognition back-off interval | 0.34 sec (decimal) | After making the Initial recognition attempts as fast as possible back up this amount for each subsequent recognition to slow down. This goes on until the Retry failed recognitions after every is reached. SAFR delays this amount of time between each recognition attempt. | |
| tracker.minimum-number-identical-recognitions-lock | Tracking Preferences menu > Minimum recognitions to lock on to identity | 2 | The minimum number of consecutive recognition attempts that we must run and produce the same person identity before we lock onto this identity. | |
| tracker.failed-recognition-retry-interval | Tracking Preferences menu > Retry failed recognitions after every | 0 sec (decimal) | Ongoing recognition attempts | The interval in which to run recognition requests if the face has not been recognized (i.e. the person is a stranger). SAFR waits this amount of time from when the last recognition attempt returns and the next begins. |
| tracker.reconfirmation-interval | Tracking Preferences menu > Reconfirm identity after every | 1 sec (decimal) | The interval in which to run recognition requests if the face is recognized (i.e. Registered face). SAFR waits this amount of time from when the last recognition attempt returns and the next begins. |
SAFR recognition occurs in 2 phases - initial recognition attempts defined by the first three attributes and ongoing recognition attempts defined by the last two attributes.
Recognition Behavior
To better understand the discussion below, it helps to understand the recognition process.
- SAFR client scans every frame of video to find faces.
- As soon as a face is detected, the image quality metrics (center pose, sharpness, contrast, and occlusion) are evaluated to determine if face quality meets the recognition threshold.
- As soon as a face is obtained that meets recognition threshold criteria the face image is sent to the SAFR Server for recognition
- While awaiting a response, the client will continue to process frames and evaluate image quality for that face. When a better face image is found, the client will replace that face with its current for subsequent recognition attempts.
- During the Initial recognition attempts (3) the client will send a new face for recognition as soon as the server returns the previous result or until a positive match is confirmed.
- If a match is found, client will attempt enough recognitions to satisfy the Minimum recognitions to lock on to identity (2) at which point it waits for the Reconfirm identity after every (1 sec) between subsequent recognitions.
- If match is not found, client will first make Initial recognition attempts (3) with no delay, then increase delay by Failed recognition back-off interval (0.34 sec) and after that delay by Retry failed recognitions after every (0 sec) between each recognition attempt.
- It should be noted that 0 sec for Retry failed recognitions after every means that the back-off interval is effectively ignored.
Automatic Scaling
Recognition load is naturally scaled to lower the number of recognitions per second (RPS) as the response times from the server increase. This is because the SAFR client awaits a response from the SAFR Server before making new recognition requests. As the responses from the SAFR Server slow, so does the rate at which recognition attempts are made for each face found in the video. This delay applies both to known faces and strangers, although different delays are applied, as defined above.
Detection load is naturally scaled to lower the number of frames processed per second. We label this as "Detections per second", or DPS. As noted above, SAFR scans the frames for faces and other objects. It also perform other analytics such as tracking faces, badge detection, person detection, or direction of travel. If SAFR completes all processing for a frame before the next frame arrives, it will begin to process that frame. If SAFR is still busy when the next frame arrive, the incoming frame is discarded.
To improve performance, SAFR can perform detections on multiple threads.
SAFR reports if it is automatically scaling through the DPS reported in the Video Feeds window in the SAFR Desktop Client. Below is a screenshot of the Video Feeds window showing both FPS (input frame rate) and DPS (detections per second).
From SAFR Desktop Camera Analyzer or Video Feeds you can view the average recognition time for that feed. This is reported in the upper right of the video preview. Below are two screenshots.
Camera Feed Analyzer | Video Feeds Preview |
|---|---|
|  |
| dRt (recognition time) is 99 milliseconds (1/10th of a second) | Recognizer / Latency (recognition time) is 99 milliseconds (1/10th of a second) |
The display of performance metrics can be enabled at the following locations:
- In the Camera Analyzer window, enable "Performance Metrics" in the View drop-down menu.
- In Video Feeds Video Preview, right click in the video panel and choose Performance Metrics
Note that in both cases, recognition time is 99 milliseconds. This includes round trip time (RTT) to upload face images and receive responses from server. Anything below 50 ms (excluding RTT) suggests the server is performing as fast as expected. Some servers may perform slower even when unloaded due to disk or other I/O performance issues.
Nothing need be done to enable automatic scaling on SAFR Server. This is a natural behavior of the system.
Manual Scaling
When strangers are in the camera view, the default setting of 0 sec will result in an aggressive recognition loop. As soon as a response is received from the server, the client will send the next best image for recognition. This results in a very high recognitions per second (RPS) on the server for even a small number of faces. The only delay between recognitions on the server is the round trip time.
To reduce the load on the server, the value of Retry failed recognitions after every may be increased to 1 second or even more depending on your use case. SAFR will still attempt initial recognition attempts per other settings but will back off to the interval you specify in Retry failed recognitions after every once the initial recognition attempts are exhausted.
Note: Because the server will auto scale load, this is not necessary except in special situations where low server latency is desired and preferred over ensuring that strangers are recognized quickly even if not initially.
Measuring Load
Following are the key metrics for server load.
- CPU Load - System will become unstable if allowed to continuously run at 100% CPU load. Occasional spikes to 100% are ok but will most likely result in increased recognition latency and thus scaling down.
- GPU/CUDA Load - The CUDA processors on the GPU are the highest loaded and should be monitored. CUDA processor can be run at 100% without risk of instability but this will result in automatic scaling.
- RAM (CPU and GPU) - Both system and GPU RAM should never be allowed to run at 100%. Doing so will result in instability.
- Disk Performance - Disk performance should never be allowed to run at 100%. Allowing disk latency to increase significantly will result in memory accumulation and eventual instability.
Below is a graph of Windows Task Manager showing the important metrics to monitor. Note that the CUDA graph is not visible by default and has to be selected from the dropdown.
