HOME >> English >> SPAP: Simultaneous Projection and Positioning

SPAP: Simultaneous Projection and Positioning

This paper presents a novel projected pixel localization principle (SPAP: Simultaneous Projection And Positioning) for online geometric registration in dynamic projection mapping, or spatial augmented reality (SAR), applications. We propose applying a time measurement of a laser projector raster-scanning beam using a photosensor to estimate its position while the projector displays meaningful visual information to human observers. Based on this principle, we develop two types of position estimation techniques. One estimates the position of a projected beam when it directly illuminates a photosensor. The other localizes a beam by measuring the reflection from a retro-reflective marker with the photosensor placed in the optical path of the projector. We conduct system evaluations using prototypes to validate this method as well as to confirm the applicability of our principle. In addition, we discuss the technical limitations of the prototypes based on the evaluation results. Finally, we build several dynamic projection mapping applications to demonstrate the feasibility of our principle.

We propose the novel geometric registration approach of a laser projector for dynamic projection mapping. The image forming mechanism of the laser projector is based on raster scanning (Fig. 1). A MEMS (Micro-Electro-Mechanical Systems) mirror adjusts the direction of a projected beam, and the color of each projector pixel is controlled by modulating the laser diode intensities of different primary colors. Leveraging this mechanism at each frame, we measure the time when a projected beam scanning over a projection surface hits a photosensor that is, for example, embedded in the surface (Fig. 2). The time information is then used to estimate the position of the beam in the projector's screen coordinate system when it illuminates the sensor. Because the time information is invariant to the distance from the projector to the sensor, this method does not depend on the distance and therefore fits dynamic projection applications where a projection surface is non-planar and moving. This principle allows us to measure the position of a photosensor in each frame while projecting meaningful image content.

Fig. 1: Bidirectional raster scanning mechanism of a laser projector. Fig. 2: Prototype system and overview of the proposed principle

Based on the SPAP principle, we develop two types of geometric registration methods. One estimates the position of a projected beam when it directly illuminates the photosensor (Fig. 3). The other localizes a beam that is reflected from a retro-reflective marker by placing the photosensor in the optical path of the projector (Fig. 4). We implement prototype systems of these two methods and investigate their geometric registration performances by evaluating the estimation accuracies. The estimation errors are measured by varying the distance from the laser projector to the photosensor or a retro-reflective marker, the projected light intensity, and other critical factors.

Fig. 3: Projection mapping results on a surface in which photosensors are embedded.
Fig. 4: A visual marker that is a rectangular surface on which four retro-reflective markers are attached (left), and a projected result of a magic lens application in map viewing (right).

Finally, we implement various dynamic projection mapping applications to show the feasibility of our techniques. They include a projector-based texture mapping for a moving 3D surface (Fig. 3), a magic lens (Fig. 4), a pen drawing application (Fig. 5), a warehouse management application with a handheld projection system (Fig. 6), and a drone projection (Fig. 7).

Fig. 5: A pen-shaped device with a photosensor (left), and interactive drawing application (right).
Fig. 6: An inventory object with a photosensor (left), and a projected result of the inside of the boxes (right).
Fig. 7: Drone projection.

Publications

  • Yuki Kitajima, Daisuke Iwai, and Kosuke Sato, "Simultaneous Projection and Positioning of Laser Projector Pixels," IEEE Transactions on Visualization and Computer Graphics (Proceedings of IEEE International Symposium on Mixed and Augmented Reality), Vol. 23, No. 11, pp. 2419-2429, 2017. [pdf] [supplementary material]