Handling Video Feed Failover

Failover support for VIRGO video feeds isn't currently native to SAFR. However, you can easily create such functionality by using a simple watchdog script.

One way to implement this is as follows: 

1. For each feed that is to have redundant failover back-up create <feedName>_backup feed on hardware dedicated to be a failover (i.e. backup) system. These streams should be pointed to the same camera as the primary feed but be left in disabled state.
2. Write a watchdog script that monitors the status of processors using the GET /status API (https://virga.real.com/docs/index.html). The script needs to do the following:
   • Periodically (e.g. once every minute) obtain the status of all VIRGO processors via GET /status API call.
   • If any of the workers have a png-date more than 30 seconds ago, assume it is offline.  Normally, workers check in every 200ms. (This can be configured with the status-interval under root config, as shown below.)
   • If a worker is detected to be offline, for every feed under that worker, enable the corresponding <feedName>_backup if one is set up.
   • For any backup feed that's enabled, if the processor of the primary feed has a png-date more recent than 2 seconds ago, disable the backup feed.
   • Push all changes in one step via a PUT /config/worker API call. 

Instead of polling, you can also leverage the following worker config to be notified via URL when worker (VIRGO) outage occurs: