VR and AR are just now becoming household terms. Unfortunately, both these technologies are limited by the social stigma of being nothing more than a gimmicky form of gaming and entertainment. Mixed Reality has an unrealized but undeniable potential to universally revolutionize HCI and change people's lives in the near future.
In order to facilitate this transition, we have to demonstrate its viability in technical fields as a practical tool. My group was passionate about exploring HoloLens' potential to revolutionize a tech-heavy field such as criminal investigation.
Through conducting interviews and contextual inquiries with the local UW police force, we gained valuable insight into the mindset and responsibilities of first responders to a crime scene. We learned about the tools used to document scenes, and the methods and protocol involved in passing along annotated evidence between relevant parties.
We interviewed two Law professors to learn about the usage of demonstrative evidence in court, and how prosecutors and defense attorneys create their cases.
Current demonstrative evidence includes sketched diagrams and graphs of crime scenes, used to convey a more holistic story to a plaintiff and jury. This, among many other areas in the criminal justice process, provided many opportunities for HoloLens to improve upon existing methods.
CSI teams currently go through incredibly tedious processes to document elements of a crime scene. Mapping out coordinates with tape measures, sketching scenes, and spending weeks on blood spatter analysis takes valuable time away from these teams.
HoloLens can automatically map out and analyze these elements at a crime scene, allowing CSI and forensics teams to devote more time to investigative areas that require more intuition.
Accessing physical evidence becomes a tedious and restricted process once it is removed from a crime scene. It can take weeks for prosecutors and defense attorneys to gain limited access to crucial pieces of evidence.
HoloLens can preserve holographic artifacts of physical evidence, reducing friction within the process of passing evidence between parties.
Illustrating a crime scene properly in court is a difficult task, due to the limited resources available at this stage.
HoloLens can revolutionize this process by giving prosecutors and defense attorneys a photoreal 3D canvas of the crime scene with which they can recreate scenarios and demonstrate various events.
This was an early 3D effect I created as we began to envision our design language and I started to learn 3D effects. I used a human model created in Blender and converted its surface into an editable particle mesh through After Effects. I was aiming for a more sci-fi, glitch art feel than what we would be including in our final video. We later changed our design language to to better match the precision and accuracy inherent in real-life crime scene analysis.
Part of our HoloLens blood spatter analysis included reconstructing a point of impact, which we wanted to represent with an abstracted orb. I created this model in Cinema 4D and After Effects, and later incorporated it into our final video. I was going for an egg yolk concept of blood being encased in a malleable glossy exterior, while reflecting the outside environment to appear more realistic.
In order to create a bunch of our 3D effects, I reconstructed a photoreal 3D model of the room we filmed in using Autodesk Remake. I later created a low-poly version of this model with Remesh, which we used for several effects in the final video.
Creating familiar interactions in a 3D space through light-based projections is no easy task. We made several variants of HUDs and spatially annotated markers before landing on the 2D/3D mixed design in our final video.
As the officer enters the crime scene, HoloLens automatically scans the entire room, creating a full 3D mesh of the space. This is used both on-site and off-site to annotate and replicate the physical space.
This eliminates the need to sketch the scene, take hundreds of photographs, and waste time manually documenting redundant or irrelevant pieces of information.
A combination of voice recognition and HUD tools are used to annotate individual items in the scene. Scanning a particular object records its x,y coordinates and dimensions, while also recording the time, date, and the name of the officer annotating the object. Voice memos, viewing tools, pen tools, and other options are available to the officer.
As detectives bag pieces of evidence, their holograms are preserved in the virtual space, allowing them to be viewed in context with annotations by other parties. This allows prosecutors and defense attorneys to examine a complete crime scene through HoloLens without needing to have the physical evidence with them at all times.
After selecting a surface with bloodstains on it, HoloLens can analyze the bloodstain patter and reconstruct an are of impact, complete with velocity, vector, and estimated size of impact object. This technology can be used in similar cases such as bullet hole trajectory analysis.This information is crucial to determining and proving the cause of injury at a crime scene.
Annotations are accessible through expandable orbs that hover over each documented element in the scene. The color of the orbs and annotations reflect the party that made them (forensics, defense, prosecution, etc.) Viewers can easily swap between parties' annotations by selecting colors of orbs.
This was one of the most cohesive, collaborative groups I've had the chance to work with. Each person had a uniquely and equally indispensable role throughout the entire process. My key individual roles during video production were: shot-by-shot video direction (including filming, editing, and and color correction), 3D room model reconstruction, creating various 3D effects in AE and C4D, masking layered 3D effects, and compiling 3D assets and animations into completed scenes.
Having never worked with 3D animation before, we're pretty happy with the final result! We reshot all our footage and created all our VFX in 5 sleepless days to win the UW chapter of this years' Microsoft Design Expo!
We will be heavily refining this project and continuing research and development through spring quarter. We're preparing to compete at the final round of the Microsoft Design Expo this July against other selected teams from top design schools from around the world. More details to come soon!