DeforMe: Projection-based Nonrigid Deformation Visualization
We propose a projection-based mixed reality system that visualizes the tangential deformation of a nonrigid surface by superimposing graphics directly onto the surface by projected imagery. The superimposed graphics is deformed according to the surface deformation. To achieve this goal, we develop a computer vision technique that estimates the tangential deformation by measuring the frame-by-frame movement of an infrared (IR) texture on the surface. IR ink, which can be captured by an IR camera under IR light but is invisible to the human eye, is used to provide surface texture. Consequently, the textures do not degrade the image quality of the augmented graphics. The proposed technique measures surface motion between two successive frames individually. Therefore, it does not suffer from occlusions caused by interactions and allows various interactions (e.g., touching, pushing, pulling, and pinching). The moving least squares (MLS) technique interpolates the measured result to estimate denser surface deformation. The proposed method relies only on the apparent motion measurement; thus, it is not limited to a specific deformation characteristic, but is flexible for multiple deformable materials, such as viscoelastic and elastic materials. Experiments confirm that the proposed method can visualize surface deformation of various materials by projected illumination even when a user's hand occludes the surface from the camera.