1 Introduction

This document provides guidance on the process to perform a SAFR Server upgrade to in a production environment. This document attempts to provide guidance, but administrators should review their specific requirements and ensure to augment this process to meet those needs.

1.1 General Upgrade Behavior

At a high level, the process to upgrade SAFR Server Software involves the over-install of SAFR on existing running system. For clustered environments, upgrade is rolling and no outage (assuming external load balance across clustered nodes) is observed. For non-clustered systems, the server is offline for the duration of the installer which can take up to 20 minutes on non-GPU enabled systems (typical for access control deployments) and up to 40 minutes for GPU enabled systems (typical for surveillance deployments).

During the software installation, SAFR hardware devices (SCAN and Camera) remain operation with their local database.

  • Identity updates from external access control system to SAFR Server
  • Configuration and identity updates from SAFR Server to devices are paused.
  • After installation completes, all updates are synchronized.

2 Upgrade Options

This guide is designed to provide an exhaustive list of options for SAFR Software upgrade. Not all deployments justify such diligence. The Quick Steps section of this guide is intended for environments where a loss of certain lower importance data may be acceptable. The trade-offs are explained in that section.

3 Quick Upgrade Process

This section describes an abbreviated upgrade process that provides a layer of protection against critical data loss with the assumption that data such as event history can be tolerated.

Preparation

  1. Notify parties such as SOC/NOC
  2. Download and stage SAFR Platform installer and Firmware from http://safr.com/portal > Downloads
    • On Products page, get SAFR Platform installer. For most deployments (no GPU), use CUDA 10 Edition
    • On Firmware page, get the firmware or your device model (use model selector at top of page)

Deployment steps

  1. Backup server configuration and data (no images)

Open Command Window as Administrator

cd c:\Program Files\RealNetworks\SAFR\bin
backup.exe --skip-objects

  • This backs up metadata (not images) and video/system configuration.
  1. Backup Person records with images
  1. Open SAFR Desktop
  2. Go to People Window
  3. Select "…" menu and chose "Export".
  4. Select SIFF Format
  1. Perform Firmware upgrade on each device

Perform firmware upgrade on a single device first. Validate functionality and then apply update to remaining devices.

See Update SAFR SCAN or SAFR Camera to latest versions at http://support.safr.com for instructions.

https://support.safr.com/en/support/solutions/articles/69000865632

  1. Run SAFR Platform Installer

Uncompress and run the SAFR Installer. The installer wizard will walk you through the steps. Don't select VMS Extensions on the Options page unless you integrating with one of the respective systems for video processing. These options are not required for the access control integrations.

  1. Verify installation

After installation:

  • If using access control integration
    • Validate that new or updated person records are received by SAFR.
    • Validate faces are authenticated and granted access at the readers
  • If using for watchlist use cases
    • Validate that faces are matched and appear in SAFR Events window
    • Verify alerts generated in your VMS if applicable

4 Comprehensive Upgrade Process

This section describes a comprehensive upgrade process that provides multiple layers of protection against data loss and down time. This process should be followed for enterprise class deployments.

4.1 Preparation

The following actions are intended to be carried out prior to the upgrade window and are recommended at a minimum in preparation for a SAFR Software Upgrade

4.1.1 Notify stakeholders

Notify relevant stakeholders about the maintenance window and expected downtime.

  • During upgrade, connected SAFR SCAN or SAFR Camera devices will continue to function normally with a local copy of the identities database.
  • During upgrade of non-redundant clusters, cameras processed in software will stop functioning for the duration of the server installation.
  • During upgrade of Video Gateway nodes, all cameras on that PC will stop functioning (no detections or recognitions) for the duration of the upgrade.
  • During upgrade, synchronization of persons and credentials will stop for the duration of the server installation. Updates that occurred during the installation will be synchronized when the services are resumed.

4.1.2 Plan Backup Strategy

Plan a backup of critical data and configurations. A data and configuration backup should be taken prior to the upgrade. The following items can be backed up:

4.1.2.1 Version-specific backups

  • Backup of database (Persons and Event metadata)
  • Backup of objects (Persons and Events images)
  • Backup of configuration (video feeds and server configuration)
4.1.2.1.1 Metadata-only or Metadata+Image Backups

This backup can be created using a simple command line. Including images in backup can result in very long backup times (in some cases days) for sites with very high volume of events (100,000 or more per day). In these cases, it may be advantageous to perform a backup of just metadata.

4.1.2.1.2 Mixed Strategy Backup

A mixed strategy can be employed also where a backup of just images is taken ahead of the release and a backups of metadata only is done within the release window. After the image backup, a freeze could be applied for all person add or image updates. A strategy such as this would result in only loss of event images that occurred since images backup was performed.

4.1.2.1.3 Test Backup Durations

Before the release window, a test of the SAFR Backup should be performed. Use this test to verify the following:

  • Duration of backup - Measure how long it takes to do a full backup. Plan release window around this.
  • Location of backup - Make sure location of the backup is known
  • Validity of backup - Ensure the backup is valid

Note: Backup is relevant only from the time the backup was started. Changes occurring after the backup was started are not guaranteed to be present in the backup.

4.1.2.2 Non-Version specific backups

  • Persons (metadata and images in the form of a SIFF file)
  • Events (metadata and images in the form of CSV and image files)
  • Video feed configuration (in the form of a JSON file)

It is also possible to take backups that can be restored across any version of SAFR Server (excluding older releases not supporting SIFF format).

4.1.2.2.1 Persons Backup

SAFR People window has option to export all person records in a SAFR Interchange File Format (SIFF) file. This file can then be restored to earlier or later versions of SAFR. For large databases, SIFF files are split into separate files and import must be performed separately on each.

4.1.2.2.2 Events Backup

A tool exists to export events to CSV and image files. This tool is highly parallelized so can create a backup of events with images significantly faster than the SAFR backup command but may result in performance impact on SAFR Server if configured so. Exported event data can be imported to any version of SAFR.

4.1.2.2.3 Video Feed Config Backup

SAFR video feed configuration is used to store all settings associated with software video processing or SAFR SCAN and SAFR Camera configuration. These settings can be exported to a JSON file that can be restored to any version of SAFR>

4.1.3 Verify Hardware

Verify that all essential hardware components are in good working condition. Because upgrades will result in possible read/write of disk and RAM that would not normally be accessed, It would be a good practice to run a hard disk inspection and test RAM prior to performing an upgrade.

4.1.4 Download and Stage Software Installer and Firmware

Before the release window, download the SAFR Platform installer and Firmware version. SAFR Platform installer is about 4 GB in size so can increase time of maintenance window if not pre-staged.

SAFR Software is available from http://safr.real.com/download.

See following article for where to obtain SAFR Firmware

https://support.safr.com/en/support/solutions/articles/69000865632

4.1.5 Backup Persistent Overrides if required

The persistent-overrides.properties file stores properties that modify the behavior of SAFR PACS synchronization. Backup this file in the event of a failure during upgrade.

4.1.6 Backup HEAP Memory if required

If the HEAP memory has been modified, take note of the HEAP settings in COVI as follows:

4.1.7 Estimate Maintenance Window Duration

Maintenance window duration will be the sum of the following:

  • Perform backup (time should be obtained from testing as described above)
  • Run SAFR Installer
    • On a Non-GPU enhanced system, install will take 10-20 minutes
    • On a GPU-enabled system, install may take up to 40 minutes
  • Test and monitoring after upgrade - 60 minutes

This estimate does not consider a clustered system. Clustering upgrades will take as long as a single server install (backup is done only once on the primary) but because clustering allows for dynamic failover, there will be no outage except for video processing per node.

?During server installation, SAFR SCAN and SAFR Camera will continue to function independent of the server as described above.

5 Pre-Maintenance Window Tasks

5.1 Perform Backups

Not all backup operations described below. It is strongly recommended that a metadata only server backup is performed. All other backups are optional and some are redundant.

5.1.1 Metadata Backup (strongly recommended)

Open Command Window as Administrator

cd c:\Program Files\RealNetworks\SAFR\bin
backup.exe --skip-objects

  • This backs up metadata (not images) and video/system configuration.

5.1.2 Images Backup (strongly recommended)

Open Command Window as Administrator

cd c:\Program Files\RealNetworks\SAFR\bin
backup.exe --objects-only

  • This backs up images (not metadata) and video/system configuration
  • May take a very long time (10s of hours) to run for high volume systems.
  • This can be run ahead of the release to provide some level of protection against image file loss.
  • Following backup, freeze new person or update images so only event images would be at risk.

Person

5.1.3 Backup People

While this is redundant to above backups, it provides another recovery option and thus extra insurance and is very easy to generate.

  1. Open SAFR Desktop
  2. Go to People Window
  3. Select "…" menu and chose "Export".
  4. Select SIFF Format

5.1.4 Backup Events

This option can provide extra insurance against loss of event data.

Event export tool is available from: https://docs.real.com/support/tools/migration_scripts

5.1.5 Backup Video Feed Configuration

A restorable backup of just the video feeds settings can be generated by following guidance in the following article:

https://support.safr.com/en/support/solutions/articles/69000144539

Backup Persistent Overrides File

all properties are located in the persistent overrides file as described below.

  1. Open Windows File Explorer and go to c:\Program Files\RealNetworks\SAFR\covi\app\config\covi
  2. Copy the file"'persistent.overrides.properties" if it exists and does not have the extension ".example"

5.1.6 Record HEAP Settings

Open Command Window as Administrator and run the following commands

cd c:\Program Files\RealNetworks\SAFR\bin
configuremem.bat -h

  • This will print out the current HEAP memory settings. Preserve this for later optional use.

5.2 Maintenance Window Tasks

5.2.1 Notify Impacted Teams

In preparation to start the maintenance window, impacted teams should be notified. This includes:

  • Potentially advising access control cardholders of the action (service is unlikely to be impacted . Advisory is a precautionary measure)
  • Advise SOC or NOC of upcoming maintenance
  • Disable any monitoring alarms that may be triggered upon SAFR Outage

5.2.2 Perform Firmware Upgrades

Perform firmware upgrade on a single device first. Validate functionality and then apply update to remaining devices.

See Update SAFR SCAN or SAFR Camera to latest versions at http://support.safr.com for instructions.

https://support.safr.com/en/support/solutions/articles/69000865632

5.2.3 Run SAFR Platform Installer

The SAFR Platform installer will take care of closing open SAFR applications and shutting down SAFR Services. Upon starting the installer, the outage will begin.

The installer will take anywhere from 10 minutes to 40 minutes. See above for installer duration.

5.3 Post-Upgrade Verification

  1. Check SAFR Face Matching
    1. Have someone present their face to the camera or SAFR SCAN reader
    2. Check event log and ensure face match
  2. Check access granted and denied at a reader
  3. Go to SAFR SCAN Web Console and check expected number of person records are at each reader
  4. Add a new identity on PACS and check sync with SAFR Server and SCAN.
  5. With newly added face, check face is matched and access granted.
  6. Update a new identity on PACS and check sync with SAFR Server and SCAN
  7. Delete a new identity on PACS and check sync with SAFR and SCAN

Questions or comments about the documentation? Email us at safr-doc-feedback@realnetworks.com .

1