This guide is intended to provide guidance on system configuration to ensure reliable operation and upgrades of SAFR Server.  This document does not cover system design.  If needed, discuss with your SAFR Support Engineer about deployment details such as:

• Combined architecture (Video Gateway and SAFR Server on same machine)
• Hardware sizing to determine if a single server or SAFR Cluster is needed
• Load Balancing strategies if multiple SAFR Servers or Video Gateways are needed

Hardware Configuration

Storage

The following table summarizes SAFR Storage configuration recommendation.  Storing images and database on separate drives as described here will improve performance and make it easier to perform backups and upgrades.

For more information, refer to the Hardware Selection Guidance article on SAFR Helpdesk.

System Specifications

Following table provides guidance for system specifications.  For more information Hardware Selection Guidance article on SAFR Helpdesk.  Ensure you are using sufficient RAM for both GPU and System RAM.  While SAFR is able to scale its effort based on available processing capacity, running out of RAM will result in unreliable operation.  GPUs should have at least 5 GB RAM for server to function at all and as the number of cameras increase, the number of GPUs and RAM capacity must also.

SAFR Clustering

If designing a SAFR Cluster with ore than 3 servers, consider designating only 3 of the servers as database and analytics only on others servers (using simple join).  Allocate limited HDD storage and RAM on analtyics machines and faster SSD and more system RAM on database nodes.

Installation

SAFR Server is surprisingly easy to install.  Even a clustered server environment requires very few interventions by the user.  SAFR handles many complex details such as upgrading system dependencies and preparing GPU models all without any human intervention.  But don't let this deceive you.  There is a significant amount of engineering that goes into the installation procedure and its always being refined.

The installer requires very little input but should you need, refer to the installation procedure described in Quickstart Guides for basic installation procedure.  Below is a short summary to the installation process.

Pre-installation proceedure

1. Prepare storage configuration as described above
2. Ensure you have your SAFR Account credentials.

Installation

1. Download and run the installer.  
2. At the end of the installation, SAFR will request you to sign in to acquire its license.
   1. On Windows SAFR will prompt you to sign into your SAFR account.  On Linux you will be given a URL to open in the browser.
   2. You will need your SAFR Account credentials for this step.
3. The license is acquired from the SAFR license server at instam.real.com
   1. If the server does not have Internet access, you must request your license to be configured as an offline license and follow the On-Premise Licensing licensing procedure.

Post Installation

Check Installation

1. Execute bin\check.bat or bin/check after installation
2. All services should be green - only ssl yellow if no SSL certs are used

Set up Object Storage for Scalability

For performance reasons, its a good idea to store images and database on separate high performance drives.  If you are have separate drives, you will need to create links to your Object Storage and Database drives as follows:

1. Stop SAFR Server (bin\stop.bat or bin\stop)
2. Move the cv-storage and mongo directories from ProgramData/RealNetworks/SAFR to the new locations
3. Create a link
   1. Windows: shortcut (right click in File menu)
   2. Linux symlink (ln -s <location of moved directory> cv-storage) and place it in the ProgramData/RealNetworks/SAFR directory
4. Start SAFR (bin\start.bat or bin/start)
5. Validate files are being created in the new object directory

Troubleshooting Installation

Should you run into any problems during installation, please copy the install.log file out of the SAFR Application Directory (C:\Program Files\RealNetworks\SAFR or /opt/RealNetworks/SAFR).  This file will help the SAFR Support team quickly identify the cause and resolution.

Quality Testing and Recognition Optimization

This topic is beyond the scope of this document.  This section will cover some of the basics of camera and SAFR configuration.

The Camera Analyzer window is the best tool for reviewing camera input quality and recognition results.  Open the feed in the Camera Analyzer.  See Connect Cameras to SAFR to get started.  The diagram below shows the Camera Analyzer and its key components.

Have a subject stand in the view of the camera and review the critical recognition parameters such as Face Size, Center Pose Quality, Sharpness, and Contrast.  These can tell you if the image from the camera is of low quality (Sharpness and Contrast) the camera location or angle of view is resulting in small face size.  Refer to SAFR documentation for information on how to use these parameters. 

Check also that the performance metrics in the upper right to ensure latency is not impacting recognition accuracy.  In particular check the dDt (Detection time) and dRt (recognition time).  If dDt is more than 100 ms or dRt is more than 300 ms, then recognition accuracy may be impacted.

Load Testing

Simulate Load

Its a good idea to load test your SAFR.  An easy way to simulate load in SAFR is to use files to simulate cameras.  You can process many files thru the Video Feeds Window.  Please refer to this article for information on how to add a file to the Video Feeds window and set it to looping.  You can add any number of files and start them.

As a convenience, FeedConfigv.xlsm (XLS Macro) can be used to add many video feeds with file source to virgo with a simple click on button.  Please check with your SAFR Support Engineer for information on how to use this tool if interested.

Check Performance

Once loaded, check that SAFR is performing well.  Check CPU and GPU processors are not overloaded and there is sufficient RAM.  The easiest way to do this is using the Windows Task Manager on Windows.  On Linux, nvtop and htop are excellent graphical tools to allow you to visual see load over time.   Below shows a screenshot of each.

Windows	Linux

On Windows, select the NVidia GPUs and change "Video Decode" in lower left of the 4 graphs to "CUDA".   NVIDIA cards are made up of different processors, CUDA being one of them.  As you can see above, except for 4k video streams, CUDA will be the bottleneck on the GPU.

Ensure that CPU does not spike to 100%.  Ensure you are running at highest load (most faces).  SAFR is able to scale its CUDA load so it is ok to allow CUDA to run high but its best if its below 100% for majority of the time.  Ensure that system and GPU RAM are not at their limits.

Monitoring

Its important to monitor your server to ensure that SAFR is functioning properly.  Following are best practices for monitoring.

Set up Monitoring on each processor (SAFR Video Gateway) and on the SAFR Server thru the Video Feeds Window.

In the Video Feeds window, choose "Configure Processor" for each processor (machine) listed.  You will see one processor for SAFR Server and one for any Video Gateways connected to your SAFR Server.  Under the "monitoring" section, add alarms by clicking the "+" button.

Add alarms for the following:

• delinquent - This reports if a processor (machine) is offline
• feed.error - This reports if any feed (camera) is offline
• lowRAM - Reports if system RAM is low
• lowGPUMemory - Reports if GPU RAM is low
• lowDisk - Reports if hard drive storage is low

You may or may not want to set up lowCPU and lowGPU alarms.  These may generate many false alarms due to the nature of variability in the inputs.

Backups

 Always backup SAFR before an upgrade or any significant change to your SAFR system.  (Murphy's law - whatever can happen will happen)

SAFR provides scripts to back up all configuration, data and images.   We recommend that you use SAFR scripts to backup configuration and data but use alternate solution for images backups such as such as https://www.backblaze.com or rsync to another machine.  Backup of images will increase the backup time significantly.  It may take up to 1 hour for each camera with medium traffic if including images.

SAFR backup scripts are located in the 'bin' directory of SAFR Server installation.  One script called 'backup.py'.  Run 'backup.py -h' for help.  Its a good idea to transfer the backup file to another server.

Restoring SAFR should be done in inverse using the restore.py script.

Backup and Restore for SAFR Clusters

The backup and restore process is only meant to be run on the Primary server of a SAFR Cluster.  Data restored on the primary will automatically be synchronized to the SAFR Secondary Servers after the SAFR Primary Server is restored.

Upgrading

SAFR Server

To upgrade a single SAFR Server, do the following

1. Make a backup of your system per previous section
2. Download and run the installer for the new version

Upgrade will take anywhere from 10 to 20 minutes depending on machine speed.  Services will be offline during this time.  There is no mechanism for zero-downtime release though this is something we would like to do in the future.

SAFR Clusters

To upgrade your SAFR Server cluster to the latest version of SAFR, do the following:

1. Prevent new faces from being added during the upgrade by performing one of the following:
   1. If you have no clients in a learning mode, then you only need to avoid adding people during this time
      1. Windows:  Find “SAFR Covi” in the Windows Services control panel and stop the service
      2. Linux:  systemctl stop safr-covi
   2. Stop all clients that are adding new faces by stopping the CoVi Service on all secondary SAFR Servers
2. Take a backup of your Primary SAFR Server.  See Server Backup and Restore
   1. Note: Backup or restore should only be performed on the primary server. Secondary servers will synchronize automatically with the primary
3. Run the installer for the new version on the primary SAFR Server.
   1. While the primary server is upgrading:
      1. If using SAFR Software Load Balancing, recognitions and events will not be generated during the installation
      2. There will be about a 2-20 second period in which newly generated events and video preview might be dropped while the database write instance is transferred to one of the secondaries during upgrade and similar outage at the end of installation while database write instance is transferred back to the primary
4. Run the installer for the new SAFR Server version on the secondary SAFR Servers
   1. Upon start up, the Secondary servers will resync with the primary
5. Start the SAFR Covi Service if you stopped it as part of Step 1.  You can resume submitting new faces on all clients