All Projects
VR DevelopmentXR Interaction DesignUE Blueprint Scripting2022


Augmented
Tectonics VR

Taubman College of Architecture and Urban Planning at the University of Michigan
2022
XR Developer
12 months (2021 to 2022)

The Purpose

Architecture students spend years learning to read construction through flat drawings and photographs, yet those abstractions rarely click until they visit an actual construction site. Site visits, however, are logistically difficult to coordinate year over year due to liability constraints and limited access windows. Augmented Tectonics bridges that gap by placing students inside fully realized, 1:1 scale construction environments in VR, allowing them to walk through material assemblies, trigger process animations, and complete interactive knowledge challenges that reinforce lecture content before and after physical site visits.

Interactive Pedestals in the Metals Module arranged in a museum style environment
Interactive Pedestals in the Metals Module arranged in a museum style environment

Built for the A317-417 Construction Architecture course at the Taubman College of Architecture and Urban Planning at the University of Michigan, the experience serves approximately 100 undergraduate and graduate students each year across four core material modules: Metals, Concrete, Wood, and Masonry.

"Immersive VR learning environment, bringing architecture construction to life for 100+ students annually at the University of Michigan."

Metals, Concrete, Wood and Masonry from top left to bottom right

My Role and Contributions

I was one of two developers on the XR team at the Center for Academic Innovation responsible for the full technical development of this experience. My contributions spanned VR interaction architecture, Blueprint system design, reusable component development, custom material creation, quiz and challenge framework design, tutorial system development, and Virtual Production adaptation. I worked in close collaboration with the faculty content team and served as the bridge between pedagogical goals and technical implementation throughout the project lifecycle.

Design Philosophy and Approach

The guiding principle was that the technology should never upstage the learning. Every interaction, navigation choice, and UI element was designed to minimize cognitive friction so students could focus on the construction concepts rather than figuring out how to operate the experience. I advocated for reusable, configurable systems rather than bespoke one-off builds, which both accelerated development across modules and empowered the faculty to author content independently without writing code.

Technical Implementation

The experience was developed in Unreal Engine 4.27 targeting the Oculus Rift S and Meta Quest headsets (tethered via Oculus Link). The entire interaction layer was built using Unreal's Blueprint visual scripting system, leveraging a range of constructs including Blueprint Libraries, Interfaces, Components, Actors, Blueprint Widgets, Enums, and Structs.

Key systems I designed and built include:

Flyout Information Panel System

A fully configurable point-and-click panel component that faculty could drop into any scene. Configurable options included alignment, placement position, up to four image slots, and audio file assignment. This allowed the faculty to build and populate the museum-style pedestal areas without requiring developer assistance.

Instructions for configuring the flyout information panel
Instructions for configuring the flyout information panel
Examples of the flyout panels used in the Metals Module

Threshold Quiz Framework

A data-driven quiz system supporting multiple question types, all configurable through Blueprint data assets:

🧱 Multiple choice questions (single or multi-answer) with right and wrong feedback

🧱 3D object identification challenges where students physically pick up and carry the correct object to the threshold

🧱 True or false questions tied to labeled 3D objects in the scene

The entire quiz was structured as a Blueprint that faculty could configure by selecting question types and filling in the required data fields, including a scene browser for selecting 3D objects for grab challenges.

An early prototype of the Threshold Challenge system
Example of a Threshold Challenge in the Concrete Module

Matching Challenge System

A standardized end-of-module challenge where students physically pick up 3D models from a table and place them onto the pedestal of the corresponding 2D detail drawing. The system provided instructional UI, correct or incorrect feedback, and a portal trigger upon successful completion. The Blueprint framework was designed so faculty could set up each module's matching challenge by drag and drop, assigning which drawings correspond to which models.

An early prototype of the Matching Challenge
Final version of the Matching Challenge in the Concrete Module

Tutorial System

A fully sequential tutorial environment that walked students through every possible interaction: smooth locomotion, teleportation, grabbing objects, triggering audio at pedestals, answering quiz questions, and completing the matching challenge. The system presented each instruction, highlighted the relevant controller buttons, and waited for the student to correctly perform the action before advancing to the next step.

VR Tutorial System Walkthrough

Custom Animated Materials

I built several custom materials using Unreal Engine's Material Editor, including the invisible wall threshold effect used to gate entry between quiz rooms.

Desktop Accessibility Version

To ensure accessibility and accommodate students prone to motion sickness, I developed a full desktop version of the experience playable with standard mouse and keyboard controls without a headset.

Technologies Used

🧱 Unreal Engine 4.27

🧱 Unreal Blueprint Scripting (Libraries, Interfaces, Components, Widgets, Enums, Structs)

🧱 Oculus Rift S / Meta Quest via Oculus Link

🧱 Disguise Renderstream Plugin (Virtual Production adaptation)

🧱 Disguise Designer / Disguise Media Server

🧱 BlackTrax Real-Time Tracking System

🧱 TouchOSC Protocol

🧱 iPad as live performance controller

First Iteration

Module Specific Bespoke Challenges

The initial design approach for the Metals module included two fully custom challenges tailored specifically to steel construction:

Light Gauge Steel Framing Challenge

Students selected the correct stud for each step, progressively assembling a 3D steel structure in real time as correct answers were confirmed.

Light Gauge Steel Framing Challenge

Steel Framing Sizing and Spacing Challenge

Students manipulated sliders to adjust beam and column sizing and spacing, watching the 3D structural model update live. A button allowed them to zoom into the model at 1:1 scale to experience it at human height from the inside.

While compelling, these bespoke challenges posed two significant problems: development time per challenge was prohibitively long given the unique interactions required for each, and user testing revealed that students were struggling to understand how to interact with each new challenge type, adding cognitive load that detracted from learning.

Steel Framing Challenge

Second Iteration

Standardized Matching Challenge

Based on development learnings and user testing feedback, I designed the standardized Matching Challenge framework as a scalable replacement. A single reusable interaction pattern, placing 3D models onto the correct detail drawing pedestal, could be repeated across all four modules. Faculty could configure and deploy each instance independently, dramatically reducing per-module development time for subsequent modules.

Main Menu Iterations

The initial main menu, designed by the faculty, was modeled after the lobby of a modern museum with a grand arch and rich visual detail. While visually striking, user testing revealed that students were getting disoriented and could not easily identify how to enter a module and begin learning.

An early iteration of the main menu
An early iteration of the main menu

The redesigned main menu presented a semicircular arrangement of clearly labeled portals to each module and the tutorial, directly in front of the user on entry. Multiple navigation methods were provided simultaneously: point and click selection, teleportation walk-in, and a carousel menu panel. This reduced friction at the most critical moment of the experience, the moment students decide to start.

Final version of the main menu
Final version of the main menu

User Testing

Upon completing the Metals module, we conducted in-person user testing with 8 to 10 architecture students during COVID-19 protocols, staggering individual sessions for safety. Pre and post-experience interviews surfaced two primary issues: wayfinding confusion through the museum environment, and uncertainty about which pedestal audio triggers had already been activated. In response, I renumbered audio trigger buttons sequentially, made activated buttons visually distinct from the environment using a contrasting color, and added a clear visual indicator when audio was actively playing.

Some photos from the user testing session conducted at the UM Taubman School of Architecture

Virtual Production Adaptation

After the core VR experience was complete, we adapted the project for a second immersive format using the university's LED volume XR stage. The Wood module environment was loaded into the Disguise media server pipeline via the Disguise Renderstream plugin, enabling it to be rendered as a photorealistic real-time 3D background on the LED volume for live virtual production.

To support interactive teaching on the stage, we connected Disguise to an iPad application over TouchOSC, allowing faculty to trigger process animations in Unreal Engine in real time during a live demonstration. The stage's BlackTrax real-time tracking system was used to track the faculty member's physical position on stage, feeding that data through Disguise into Unreal Engine to enable the faculty to physically pick up and rotate virtual 3D artifacts as if they were tangible objects, while narrating details live to camera.

The resulting demo video was published to announce the university's new XR stage and was subsequently used to engage faculty across multiple schools and colleges in conversations about incorporating virtual production into online course delivery.

Some shots of the Wood Module that was used as a backdrop on the LED Volume

Project Challenges and Solutions

Challenge 1:

Bespoke challenges per module were unsustainable to develop and confusing to users

Solution: Designed a single reusable Matching Challenge framework configurable by faculty without developer involvement

Challenge 2:

Faculty needed to build content without relying on developers

Solution: Engineered all interactive components as configurable Blueprints with exposed parameters, turning complex VR systems into faculty-accessible tools

Challenge 3:

Wayfinding confusion in the museum environment

Solution: Redesigned layout and introduced sequential numbering, color contrast for interactive elements, and audio state indicators

Layout and wayfinding for the Metals Module
Layout and wayfinding for the Metals Module
Sequential numbering and audio indicators in the Concrete Module
Sequential numbering and audio indicators in the Concrete Module

Challenge 4:

COVID constraints limiting group user testing

Solution: Staggered individual user testing sessions to safely collect student feedback

Challenge 5:

Motion sickness and headset accessibility barriers

Solution: Built a full parallel desktop version playable with mouse and keyboard

Challenge 6:

Scaling from one module to four consistently

Solution: Completed deep iteration on the Metals module first, then built reusable systems so subsequent modules could be developed faster with faculty self-service content authoring

Skills Demonstrated

🧱 VR experience design and development for architectural education

🧱 Unreal Engine Blueprint scripting at a systems architecture level

🧱 Data-driven, configurable tool design for non-developer content authors

🧱 Iterative UX design driven by user testing and feedback

🧱 Custom material and shader creation in Unreal Engine

🧱 Interactive quiz and gamification system design

🧱 Accessible design (desktop parity version)

🧱 Virtual Production integration with Disguise, BlackTrax, and TouchOSC

🧱 Real-time 3D performance and live event scripting on LED volume stages

🧱 Technical documentation and developer handoff

Outcomes and Impact

🧱 Deployed as a required component of the A317-417 course curriculum, integrated week by week alongside corresponding lecture content

🧱 Used across multiple semesters and adopted by a second faculty member for a separate construction architecture course

🧱 Served approximately 100 undergraduate and graduate architecture students annually

🧱 Received strongly positive student feedback for enhancing conceptual understanding of construction materials and assemblies

🧱 Successfully adapted into a Virtual Production teaching format, generating institutional interest in LED volume-based course delivery across multiple schools and colleges at the University of Michigan

🧱 Full source code, developer documentation, and instruction manual handed off to faculty for continued independent use and extension

Recognition and Usage

The project was formally incorporated into the A317-417 course curriculum at the Taubman College of Architecture and Urban Planning. The Virtual Production adaptation was featured in a university news article announcing the Center for Academic Innovation's new XR stage and served as a proof-of-concept that drove faculty outreach across departments for virtual production in online learning.

Project Reflection

Augmented Tectonics reinforced a lesson that guides my approach to every XR project: the most sophisticated technology is only as valuable as the experience it creates for the person using it. The most impactful decisions on this project were not the most technically ambitious ones. They were the choices to standardize interactions, strip back the main menu, number the buttons, and build systems that faculty could use without asking for help.

This project also demonstrated that immersive environments built for one platform carry significant untapped potential. The same Unreal Engine assets that served as a VR headset experience were later transformed into a live virtual production stage performance with real-time interactivity. As the boundaries between XR, virtual production, and interactive media continue to converge, I see significant opportunity to build modular, multi-format immersive content pipelines that can serve learners, audiences, and storytellers across many simultaneous platforms.

Video walkthrough of the VR experience.
A Complete VR Walkthrough of the Concrete Module
Next Project
Ford Nuclear Reactor VR
→