Adaptive fusion of infra-red and visible spectra camera data for particle filter tracking of moving targets

This paper presents a method for tracking a moving target by fusing bi-modal visual information from a deep infrared thermal imaging camera, and a conventional visible spectrum colour camera. The tracking method builds on well-known methods for colour-based tracking using particle filtering, but extends these to handle fusion of colour and thermal information when evaluating each particle. The key innovation is a method for continuously relearning local background models for each particle in each imaging modality, comparing these against a model of the foreground object being tracked, and thereby adaptively weighting the data fusion process in favour of whichever imaging modality is currently the most discriminating at each successive frame. The method is evaluated by testing on a variety of extremely challenging video sequences, in which people and other targets are tracked past occlusion, clutter and distracters causing severe and sustained camouflage conditions in one or both imaging modalities.