Photogrammetry - Will it simplify XR development?

Recently there has been an exponential increase in the adoption of immersive technologies such as augmented reality (AR) and virtual reality (VR) as enhanced customer experiences are no longer delightful additions but absolute must-haves. Businesses are now reaching out to customers through AR filters on social media, launching webinars/products through VR, and releasing AR product catalogs/user manuals to provide a completely immersive user experience. With the advent of Metaverse, this trend will continue to rise.

The creation of extended reality (XR) use cases requires the creation of assets that will serve as the building blocks for the applications. Assets such as 3D models, 2D assets (icons, backdrops, textures, and others), audio files, etc., form the foundation of an XR application. Among these, 3D models have been a prime differentiator in AR/VR applications as they enable users to visualize how the products look in real life. But creating 3D assets comes with its challenges.

3D/CAD modeling requires a lot of time, skills, and effort. Generally, 3D models are created using detailed sheet drawings representing the products in multiple views (top view, side view, and front view) using modeling software. Moreover, texturing and material application are needed to make them look realistic.

One of the industries that benefit from 3D modeling is manufacturing. Manufacturing firms use 3D models for various cases, including marketing and product development. These firms usually have an in-house 3D design team that handles these tasks. These 3D models can later be re-used in XR applications as well. But for businesses that don’t have the resources to create 3D models of their products, venturing into the XR domain becomes a big challenge.

This is why photogrammetry is extremely useful in XR applications creation.

Tech for building 3D assets quicker

As the name suggests, photogrammetry is a three-dimensional coordinate measuring technique that uses photographs as the fundamental medium for metrology (or measurement). The underlying principle of photogrammetry is triangulation, more specifically called aerial triangulation.

By taking photographs from at least two different angles, lines of sight can be developed from each camera to the points on the object. These lines of sight are mathematically intersected to produce the three-dimensional coordinates of the points of interest.

                                         Source: https://thehaskinssociety.wildapricot.org/photogrammetry

These points of interest are then converted into a 3D surface mesh model, shown below, which can be imported into any XR development engine like Unity3D.

      Source: https://www.3dflow.net/technology/documents/3df-zephyr-tutorials/convert-photos-3d-models-3df-zephyr/

To summarize, photogrammetry creates 3D data with the help of multiple, overlapping photographs.

Photogrammetry in XR development

Photogrammetry will be a significant differentiator in facilitating 3D model creation for XR apps. Here are a few advantages of using photogrammetry for creating 3D models over conventional 3D modeling techniques:

• Creation of organic 3D models
  Creating 3D models is a time-consuming task. The computer-aided design (CAD) modelers refer to multiple sheet drawings and views to understand how the object in focus looks from all angles. This becomes particularly difficult for non-industrial things like apparel, art pieces, grocery products, and the like, as there are no sheet drawings available for them. Photogrammetry will save a lot of time creating such organic 3D models. In domains like retail, this will be extremely useful to venture into the XR space, as 3D models creation will become easier.

• Cost
  Unless there’s a need for specialized pictures taken from drones and other hardware, photogrammetry may be less costly. Other alternatives like laser scans are expensive because of the hardware involved and cannot be used for larger objects such as buildings, heavy machinery, terrains, etc. The images required to build an accurate 3D model must be taken from high-quality (preferably 4K) cameras to perfectly capture all the details. The hardware investment (camera, drones) will be a one-time investment, and in the long run, photogrammetry will significantly bring down the costs. The beneficiaries will mainly be large equipment manufacturers, automotive industries, and realtors.

• Skillset
  3D modeling requires a very high skill level to create perfect 3D models. But since the photogrammetry technique can build the 3D model, the skill requirement is less than other 3D data creation techniques.

• Texture and color details
  Since photogrammetry creates meshes using images of the actual object, details like texture files and color properties are created and embedded along with it, unlike conventional 3D modeling software that requires the models to be textured separately.

Challenges with photogrammetry

• Surface mesh instead of volumetric 3D models
  The mesh generated from the point cloud is a surface mesh, so the technology does not capture the internal details of the object. We will only get a hollow shell of the object as the 3D model. This may work for products that do not have any internal details like clothes, shoes, groceries, furniture, and the like, but may become a challenge for industrial equipment and machines.

• Post-processing
  The 3D models created using photogrammetry may not have a smooth or finished surface, hence a need to refine the data. Also, based on the number of images, many meshes may be generated in the 3D model that must be reduced. For XR applications, low polygon 3D models are preferred as they are easier to render and are less performance-intensive. Hence, post-processing is a necessity for 3D models created using photogrammetry.

• Higher initial investment
  If the 3D models require aerial photography, or if the object focused is a large machinery, the cameras and equipment needed to capture the images will be very expensive. This may be concerning for businesses that don’t want to invest a significant amount initially.

Photogrammetry seems promising in resolving some bottlenecks in creating 3D assets for XR application development. Technological advancements will, in the future, result in the broader adoption of photogrammetry by making it more accurate.

At Zensar, we have explored commercially available tools to evaluate the benefits of using photogrammetry over 3D modeling software. Based on our exploration, some post-processing is still required on the output 3D models as they are not perfect and cannot be used directly for XR app development. For now, it won’t be easy to replace the conventional 3D modeling pipeline completely with photogrammetry. However, it can facilitate the 3D modeling process and significantly reduce development time. Photogrammetry makes XR app development easier, especially for businesses that don’t want to invest in a separate 3D design team but still want to venture into immersive technologies.