Vibration Extraction Using Rolling Shutter Cameras

Abstract

Measurements of vibrations, such as sound hitting an object or running a motor, are widely used in industry and research. Traditional methods need either direct contact with the object or a laser vibrometer. Although computer vision methods have been applied to solve this problem, high speed cameras are usually preferred. This study employs a consumer level rolling shutter camera for extracting main frequency components of small vibrations. A rolling shutter camera exposes continuously over time on the vertical direction of the sensor, and produces images with shifted rows of objects. We utilize the rolling shutter effect to boost our capability to extract vibration frequencies higher than the frame rate. Assuming the vibration amplitude of the target results in a horizontal fronto-parallel component in the image, we compute the displacement of each row from a reference frame by our novel phase matching approach in the complex-valued Shearlet transform domain. So far the only way to process rolling shutter video for vibration extraction is with the Steerable Pyramid in a motion magnification framework. However, the Shearlet transform is well localized in scale, location and orientation, and hence better suited to vibration extraction than the Steerable Pyramid used in the high speed video approach.

Using our rolling shutter approach, we manage to recover signals from 75Hz to 500Hz from videos of 30fps. We test our method by controlled experiments with a loudspeaker. We play sounds with certain frequency components and take videos of the loudspeaker's surface. Our approach recovers chirp signals as well as single frequency signals from rolling shutter videos. We also test with music and speech. Both experiments produce identifiable recovered audio.