SoftServe’s Augmented Reality App Reconstructs a Disappeared Wooden Medieval Ukrainian Castle
SoftServe’s R&D division was consulted to develop a mobile augmented reality (AR) application, which, with the help of added reality technology, would show the medieval Tustan fortress of the IX-XVI centuries.
The idea to reconstruct the castle was the dream of Ukrainian scientist and ethnographer, Mykhaylo Rozhko. Today his son Vasyl Rozhko is supporting the effort. Reconstruction started from manual measurements of about 4000 locations where wooden planks were placed on the stones between 1971 and 1977. The first graphic reconstruction was made with pencil and paper based on known architecture samples of the time and some wooden remains, with up to 90% accuracy.
Project Description
To begin developing this app, SoftServe needed to address several challenges, as all that remains of the fortress today are the sites where wooden blocks were situated. It’s an area that is not similar to other traditional historical sites. In addition, the rocks of the castle had crumbled over time which changed the original outline. SoftServe had to make sure the rocks on matched those in reality and the ones in the developed 3D model.
“Building the Tustan AR app was a unique challenge with new levels of complexity”, said Semen Frish, VR Engineer at SoftServe. “Working closely with Vasyl Rozhko, the founder of the reservation and ‘Tu Stan!’ festival, and the initiator of the project, we share his passion to help bring this historic castle back to life. The solution we developed can be applied to the virtual reconstruction of other historical sites in Ukraine and all over the World. We are ready to share our best practices for the preservation of historical heritage”.
Two specialists from SoftServe’s R&D team worked on the project over the course of a year. The reconstruction started with physical measurement of the location, examining the existing materials and plans. The use of drones aided in rea ching the less accessible areas. Photogrammetry reconstruction allowed the team to create digital graphic models of the rocks to within with millimeters of accuracy.
The task was to ensure that the rocks depicted in the photographs (with blue outline) coincided with the rocks in the real environment and the rocks on the developed 3D model (with orange contour). The 3D model had to match the actual landscape, taking into account the depth and clipping.
The camera and model had numerous parameters, such as focal length and physical size of the sensor, distortion of the lens, zoom and rotation. To pick up these parameters quickly and accurately, SoftServe used several algorithms for mathematical modeling of reconstruction.
- The BLAM plugin for Blender was used to determine the parameters of the camera. It worked flawlessly, but did not solve the problem, because there were no long straight lines on the territory of the fortress and around it. In order to avoid finding the actual location of dots for visual markers in manual mode, the team further analyzed technologies such as OpenCV for reconstructing camera parameters, a software package for laser scanning, LAStools, and photogrammetry techniques and MeshLab.
- SFM (structure from motion) and MVS (multi view stereo) algorithms were used based on the Photo Tourism project that Google Street View is based on to get a digital model that shows camera settings, photo positioning in space and 3D transformation models.
During the initial experiments, the idea was to abandon the rigid printed markers. With Google Tango, it was possible to build functionality to locate a castle in complemented reality in accordance with any flat surface in the room for those with a Tango-compatible phone or tablet. Today, Tango-compatible devices are not very common, but SoftServe is working to offer more interactive features for the Tustan AR application with the newer AR technology from Google ARCore in the near future.
Results
The Tustan AR mobile app is live in real time on location and is available in any weather conditions. Castle can be viewed remotely by downloading a free app in the App Store or the Play Market. Images to experience augmented reality are included on the apps pages. Code and models can be found in open access and anyone is welcome to make their own reconstructions for the outdoor landmarks on top of it. At the moment, our team is working on the next logical step of developing the application – Tustan Virtual Reality (VR).
SoftServe plans to continue to actively develop in both directions – AR and VR. Within the structure of the R&D department, a new "interactive" direction was created that deals with projects leveraging these technologies, as well as computer vision and potentially digital twins.