Github Link:

Paper Link: Bard J., Bidgoli A., Chi W.W. (2019) Image Classification for Robotic Plastering with Convolutional Neural Network. In: Willmann J., Block P., Hutter M., Byrne K., Schork T. (eds) Robotic Fabrication in Architecture, Art and Design 2018. ROBARCH 2018. Springer, Cham


For this research project, I was tasked to create an automatic projection mapping algorithm for planar surface from an orthogonal viewpoint. The purpose of this research was to use CNN to classify different areas of the plaster created by the robot into 5 classes. The classes include different types of errors (rough, holes, bad finish, mark up) and pass. The capture image of the plaster was subdivided and classified into these 5 classes. The result of the classification was used to be mapped onto the plaster surface. In order to project the error onto the plaster surface, we needed to crop the image to the projected area before the classification.


Initially, I wanted to create a real time projection mapping where I can project markers onto the edges of the screen in order to know the corners of the cropped area. However, there were technical issues with the GoPro and OpenCV performance to get the live projection mapping since the GoPro was operating through the wireless protocol.

At the end, I projected a simple circle grid marker and captured the image to grab the edges and corners of the projected area. Then create a bounding box that occupied the corners in order to create the closest projection mapping area.

System Design


GoPro, RTS DLP Projector, ABB Robot


Python 3.6, OpenCv 3.4.0, GoProCam


Camera Calibration

Before the implementation of the Projection Mapping, I had to calibrate the GoPro camera using OpenCV CheckerBoard because I used GoPro Medium Angle to capture the image. The GoPro Medium Angle Setting still had a visible warp that needed to be corrected through Camera Intrinsic Calibration.

Projection Mapping

First the image of the circle grid marker was captured and then the image without the projection was captured. The circle grid image was used to find contour and corners using cv2.findContours. Then only the contours larger than 200x200 were considered in the bounding box. The bounding box was used to crop the clean image and send to the CNN model for classification. The result of the CNN is then projected onto the plaster surface.


While this is a limited projection mapping technique, the system was in a very controlled environment and did not need a complicated way to create the projection mapping. I also could have found the corners through contours and used homography to projection map onto the planar surface. Furthermore, it could be expanded to mapping 3D surface by first capturing 2 images for stereo 3D reconstruction and using a morphing and wrapping technique to map onto more complicated 3D surfaces.