What settings affect performance the most?

The Minimum searched face size setting affects performance the most. It determines how small of face to search in the image. If your use case involves large faces in the camera frame you can dramatically speed up processing by increasing the Minimum searched face size setting; performance increases approximately by square law.