Augmentarium Alumnus Imagines an Immersive World

Imagine walking through New York City and seeing a restaurant's best-known dish digitally hovering above its overhead signage. Or, what if you could explore the National Gallery of Art in Washington, D.C., with a group of friends, all without ever having set foot in the nation’s capital?

Ruofei Du, who graduated from the University of Maryland in December with a doctorate in computer science, is intent on turning these ideas into reality, while also making the world a more fun and interactive place to explore.

For the past four years, Du has worked on projects that combine social media with powerful visualization tools based in virtual reality or augmented reality (VR and AR)—all with the goal of providing people with new ways to access information they can use in their daily lives.

Much of this work was done in the Augmentarium, a VR and AR testbed lab that launched in 2014 with funding from the National Science Foundation.

Working with his adviser, Amitabh Varshney, a professor of computer science and dean of the College of Computer, Mathematical and Natural Sciences, Du conceived new ideas, built several prototypes, and conducted user-studies to test the efficacy of his immersive tools.

“Ruofei’s research exemplifies how these new immersive technologies will change how we interact with each other and with our environment,” says Varshney. “We’re working on VR and AR applications that have many important uses—health care, education, public safety—but we also need to remember how these immersive tools will be essential for enriching our experience of everyday life around us.”

Two of Du’s projects have received praise from eternal computer visualization experts.

The first, called Social Street View, offers the first immersive social media navigation system where one can walk down the street and see geo-tagged images and comments overlaid on real-world locations. Du likens the software to the popular crowdsourcing platform Yelp, but in a 3–D setting.

“VR and AR devices are not so commonplace now, but we think they could be soon, leading to widescale use similar to how we currently rely on our smartphones,” Du says.

He explains that his own love of travel got him thinking that, “what if I could query local social media reviews and comments and digitally place them in the landscape viewed through my smartphone or some type of lightweight headwear?”

That’s what gave him the idea to create a totally new social media platform that allows users to view what kind of food they want to eat, what artwork they can see at museums, all in an interactive environment that allows you to access the information from almost anywhere in the world.

The platform also allows users the ability to virtually travel abroad—to the sun-drenched beaches of Hawaii, for example—while simultaneously accessing other people’s opinions, images and reviews of the landscape being viewed.

Social Street View can also be used for business advertising, Du adds.

“It has so many potential uses,” he says. “It could even be used to teach people about the local culture in a place they are visiting, even if it is not the right time of year. For example, letting a traveler experience a spring festival in a Chinese province even if they are there in the middle of the winter.”

The Social Street View platform won a Best Paper award at the 2016 ACM SIGGRAPH Web3D conference and Du says he is hoping that more people will come to embrace it.

The technology is also integrated into one of Du’s more recent projects, called Geollery. This platform presents a mixed-reality world for creating, sharing and exploring different places with friends and family—even if they are scattered across the globe. Users can virtually explore almost anywhere in the United States and Europe on Geollery, including Radio City Music Hall in New York City, the university’s McKeldin Mall, and Big Ben in London.

“We wanted to create a fun world for social good,” Du says. “Users can paint street art on walls, exchange gifts, and share photos on the digital street. It can be used for business purposes as well, such as hosting a virtual meeting and advertising with geospatial context.”

His paper on Geollery was recently accepted to the ACM CHI Conference on Human Factors in Computing Systems—the premier international conference of Human-Computer Interaction—scheduled for May 4–9 in Glasgow.

Du says none of his work would be possible without the technical support he received while at Maryland. The Augmentarium, which is now part of the Maryland Blended Reality Center, is supported by tech staff from the University of Maryland Institute for Advanced Computer Studies (UMIACS).

“They helped me set up the infrastructure for coding repositories, servers to host the websites, and setting up the demos on the immersive curved screens in the Augmentarium,” Du says. “I am really grateful for their help and expertise.”

Du, who is soon starting a new role as a research scientist at Google, also credits his mentor Varshney for making his virtual ideas become reality.

“I would like to say a special thank you to my adviser, Dr. Varshney," Du says. "I feel so lucky to have worked with him. He taught me not only how to conduct research, but also how to communicate science clearly and present my work—and that is invaluable.”

—Story by Melissa Brachfeld