Canadian augmented reality tech takes CMMS to the next level

If you were visiting the fighter aircraft hangers of the past, you would see something quite different than you would today. Technicians and maintenance crews would be making copious notes on paper about surface damage and other maintenance aspects of active aircraft. Repair was, no doubt, a long, tedious and drawn-out process.

Today, you would see a different picture. Thanks to the advent of mobile devices, new trends like wearables and augmented reality are playing a larger role in aerospace manufacturing and maintenance. Fighter jet maintenance crews currently employ augmented reality (AR) software to view – and to interact with – 3D models of the equipment they are working on.

For example, Lockheed Martin currently uses an interactive 3D AR platform developed by Vancouver-based software firm NGRAIN, to inspect F-35 and F-22 fighter jets. The technology platform cuts both repair times and minimizes error, critical because even the tiniest bit of surface damage to a stealth fighter like the F-35 can make it detectable by enemy radar.

In addition, new data from each technician is shared on the platform, a system that has the ability to analyze and integrate information from each plane in order to do things like highlight where and when maintenance issues may occur in the future.

Beyond aerospace companies like Lockheed Martin, Raytheon and CAE, NGRAIN says its tech platform is being adopted in every sector, from energy, manufacturing and healthcare to U.S. and Canadian government agencies and universities.

“In short, our customers use our technology to equip their mobile workers with access to the right data at the right time on the right device,” says Director of Product Management Barry Po. “We provide real-time assistance and allow them to access all the information they need to do their jobs to the fullest extent possible.”

It works like this: NGRAIN software on a worker’s mobile device employs ‘computer vision’ techniques to track and render complex objects in the environment in real-time, digitally capturing the worker’s surroundings so they can be analyzed and augmented with real-time feedback.

Simulated real-time updates on factors like temperature, pressure and other indicators are available, including dynamically-changing parts colouring. The technology allows service crews to make better decisions on-the-spot, maximize uptime and eliminate costly mistakes. It all means that actions that used to take days can be finished in a matter of hours.

“NGRAIN has its origins as a company that provided interactive 3D software to support industrial training programs,” Po explains. “We spent almost fifteen years working with customers who face really tough industrial problems. As time passed, it was only natural for us to look at how new technologies – like tablets, wearables, and things like VR and the ‘Internet of Things’ – could transform the way people work and do business.”

But when the NGRAIN team members looked at how these new technologies could be integrated to solve thorny industry problems, they realized that they would need to start from scratch.

“The conventional historic way of visualizing things in a 3D environment is to use polygonal graphics,” Po says. “They’ve been used over the last few decades because they’ve worked well enough for the applications that existed and because they also don’t take much computing power. But polygons only allow you to visualize the surface of objects and can’t provide a true 3D experience.”

To visualize what’s inside assemblies in full detail and harness that information, NGRAIN had to create building blocks that were entirely new. Enter the voxel, a 3D pixel akin to digital grain of sand. Together, voxels permit direct access to attributes for every point in space.

“Just like a sand castle is built from countless grains of sand, we build 3D objects using billions of discrete points,” explains Po. “The voxels solve problems that could never be solved with polygons, and they can do this without the need for huge computing power. Any reasonable level of computing power will do. So it’s very exciting that we’ve developed technology that takes advantage of the convergence of immersive 3D, mobile hardware and big data to address industrial problems that have been around for a long time.”

Industrial Smart Glasses
Like other computerized maintenance management systems (CMMS), NGRAIN’s platforms enable workers in the field to access operational intelligence, predictive analytics and other kinds of enterprise ‘big data’. But NGRAIN goes beyond standard CMMS with the capacity to link 3D content to maintenance data. This is currently accomplished using tablets, but some NGRAIN customers are also actively experimenting with smart glasses.

“At this point, when you wear a pair of smart glasses driven by our software, you can get live information and 3D graphics precisely overlaid right on top of whatever it is you’re working on,” Po says. “So, data such as current state, stock levels, temperature, oil analysis and so on is all available at a glance. Step-by-step instructions and animated equipment procedures indicate what part you need to work on next, how to do it and even whether you’re doing it right.”

While Po acknowledges that tablets are great tools, he notes that as smart glasses develop, they’ll be able to solve problems that can’t be solved or solved easily with a different device. However, the best way to use smart glasses in an industrial setting is still being worked out.

“While you’re wearing them, you can make annotations on a tablet, but we’ve also been experimenting with some partners on voice input, although it becomes tricky in a noisy environment,” Po says. “I also see a lot of potential in using gesture-based interfaces in conjunction with smart glasses – it may become much more common than people would believe.”

He adds that what finally ‘wins out’ in industry may be a combination of several interfaces. For example, as workers look at assets through their smart glasses, they might select, move things around or make notes using a finger in the air front of them, and then use speech recognition to tell the system to save their work or pause things if interrupted.

Crunching the 3D Data
As primarily a software company, NGRAIN’s challenges have always been centered around creating a platform that can work with and adapt to new form factors and limited capability.

“Originally, the limitations on computing power made it very difficult to create a compelling augmented reality experience,” Po relates. “We needed to render complex 3D content in real-time while using computer vision to track and recognize objects in the surrounding environment while still enabling users to call up data on-demand. That’s a tall order using a great deal of computing power, but we needed to be able to do it on the limited capability of mobile tech. So we spent a lot of time optimizing our core 3D platform.”

The 3D model files of the Lockheed Martin F-35, for example, were enormous, impossible to fit on a tablet, but NGRAIN team members found a way to reduce the size but keep all the detail. The advent of smart glasses, however, has taken the challenges of mobile devices – limited computational power and poor battery life – to a new level for NGRAIN. The company first started working on integrating its platform with smart glasses about two years ago.

“The good news is that these devices are improving really fast,” says Po. “For example, we have a very strong working relationship with Epson, who manufactures the Moverio series of smart glasses. Within the span of a year, they released a second-generation product that is 40 per cent lighter.”

Po notes that it’s far from clear which wearable designs are best and what’s going to win out in different applications.

“But we’ve been able to take that leap and build some pretty compelling experiences with smart glasses because we were able to draw on our experiences building those for mobile devices,” he says. “We know that the hardware is only going to keep getting better over the next year, and we expect to be able to do even more incredible things – like automatically recognizing and surfacing relevant information about objects just by the wearer looking at them. We’re on the path to building an experience where a field specialist will be able to put on a pair of glasses, switch them on and get straight to work.”
www.ngrain.com