HoloLens: Project Serra

September 1, 2016 | Citizen Inc.

Citizen Inc.

In the 1950s and 60s, Popular Science magazine promised a glittering, optimistic future to Americans. Humans would lead lives of leisure, robots would do all our work, and we’d be flying in personal rocket ships to have picnics on the moon.

As it turned out, we got very little of that. We did get robots, but the life of leisure they’ve introduced has mostly been in the form of unemployment to a sub-set of factory workers. And the rocket ship picnics? Nope. New, exciting technology – cold fusion, consumer space travel, teleporters – has always been on our TV screens but never in our hands, pitched to an increasingly dubious public by a cavalcade of tech visionaries. Every new product is met with healthy suspicion.

Then came the HoloLens, a snazzy set of goggles produced by Microsoft promising to revolutionize just about every industry known to man. Video demos and early reviews got us excited, and when we got the opportunity to actually work with the HoloLens itself, we were completely blown away.

Microsoft HoloLens | Citizen, Inc. | Portland, Oregon

Image Source: Microsoft

After participating in Microsoft’s HoloLens Academy, Citizen was partnered directly with an existing team at Microsoft’s development lab. This team, consisting of some top Microsoft developers as well as two radiologists and a surgeon all from Stanford University, was working on a compelling new project to benefit breast cancer patients: Project Serra. The project would visualize MRI data, and in the HoloLens, overlay it on a patient being prepped for surgery to help surgeons precisely target tumors within the body.

The Problem

With as far as modern medicine has advanced, the commonly accepted procedure when surgeons prep to surgically remove a tumor requires the surgeon to view key 2D images or films taken from an MRI scan. The surgeon then must create a 3D model in their mind and plan how they are going to make their incisions, how they will remove the tumor, and how much additional tissue must be removed to ensure a successful outcome.

Consider also that the data captured from an MRI scan is a 3D dataset. Essentially, this commonly accepted procedure captures a 3D dataset, which is then translated into 2D images, which is then reconstructed into a 3D mental model. While we do have the capability to view MRI data represented as a 3D model, we still have a translation of 3D data being displayed on a 2D screen which then must still be translated into a mental model in the mind of the surgeon.

All of which lead to the question, what is lost in translation?

At the outset of Citizen’s engagement, the team from Microsoft and Stanford had a functional – if rough around the edges – prototype. MRI data visualized into actual 3D could be seen directly on a medical model, visually registered with a set of tags, in actual 3D. Once locked in, the model could be viewed and cross-sectioned at will, showing position, depth, and size of tumors beneath the patient’s skin. For all of the HoloLens’s futuristic potential, problems still existed with the design, functionality, and user experience.

The Solution

While surgeons are usually very accurate in their analysis, the aid of a computer could save patients from having too much or too little tissue removed, and benefit both clinical and aesthetic outcomes of the surgery.

The project would visualize MRI data into the real world, and in the HoloLens, overlay it on a patient being prepped for surgery, allowing for more precise targeting of tumors within the body.
Connected Health | Citizen, Inc. | Portland, Oregon

The Process

The project was ultimately beholden to two disparate audiences: first and foremost, surgeons who would actually be using the technology to guide their work, and second, other doctors and hospital administrators who needed to be convinced of its efficacy before investing. Citizen’s approach to every engagement is to service the essence of a client’s need with purposeful design, rather than create excessively complex design for its own sake. This fit well with the needs of the Stanford surgeon working on the project, who pushed for a minimal experience, wanting it to empower the clinical benefit and not draw attention to itself. This pushed the team to scale down some of its more grandiose brainstorms and contain complexity within an intricate but easily navigated interface.

The experimental nature of the project stretched all team members to work quickly and across disciplines. Initially convinced they would be able to work in 3D design software Maya to create basic prototypes, they quickly discovered that iterating within the HoloLens itself was their only path forward. Every idea needed to be tested “in the Lens” before committing, due to a set of challenges unique to the format. These included but were not at all limited to:

HoloLens Design Challenges | Citizen, Inc. | Portland, Oregon

Keeping iteration moving was a Herculean effort met by a supremely experienced team. New pivots and wrinkles surfaced every day, forcing the team to address them rapidly and keep moving forward. A small secretive section of Microsoft Studios that was centered around a body ‘phantom’ served as backdrop for a churning design and development mix. While sprints in a normal interactive process are no shorter than two weeks, the HoloLens team worked on sprints no longer than two hours.

As the project began to take more concrete shape, excitement among the team became infectious. A common feeling of purpose grew from seeing potential in the far-reaching potential of the product. The HoloLens project gave surgeons – for the first time ever – a look inside the human body in real time. No radiation, no surgical implements, merely a projection of light. Testing pleased the project surgeon so much that she continually spoke of how it would benefit her patients even in its current, early phase of development. Her enthusiasm only inspired the team to work harder.

The HoloLens project gave surgeons – for the first time ever – a look inside the human body in real time.
Surgeon using mixed reality in surgical preparation.

Surgeon using mixed reality in surgical preparation.

The Results

From a visual perspective, Project Serra is clear and easily navigable, comprised of one main menu and the core experience which allows a surgeon to view 3D MRI data superimposed on a patient from advantageous angles and perspectives.

Citizen took the working prototype it was handed, and through the guidance of the attached surgeon, molded it into a more usable and accessible state. Within this unobtrusive, streamlined tool is a kernel of profound promise. Built to demanding requirements with code that would need to be shared with future teams and other developers, this “simple” experience could well be the basis for an unlimited number of other medical applications. There is no ceiling for the product.

HoloLens | Citizen, Inc. | Portland, Oregon

Project Serra gave Citizen the chance to advance that notion with a product that contains the seed from which may grow a tool that improves the lives of millions. As rational dreamers and believers in the potential of technology, nothing could be more satisfying to us than to see a doctor use a product we helped shape and say that it could save a life.

This is the future we were promised.

Citizen Inc.