• Home
  • Healthcare
  • Virtual Reality Today and Tomorrow: Use Cases in Medical Treatment

Virtual Reality Today and Tomorrow: Use Cases in Medical Treatment

Main visual : Virtual Reality Today and Tomorrow: Use Cases in Medical Treatment

Today's virtual reality (VR) is not just for gaming and entertainment venues. This technology is also increasingly being used in business.
At Fujitsu Forum 2018 on May 17-18, 2018, we held a conference entitled "Fujitsu's Virtual Reality Presents New Possibilities for Business: Virtual Reality Extends Beyond Reality's Boundaries Science Fiction Technology Made Real."
For this, we invited a science fiction writer and VR application developers from the healthcare field (Heart Simulator); they presented some key points about adopting VR in business and VR's future possibilities.
[Fujitsu Forum 2018 Conference Report]

How Will VR Change Our Lives?

VR Estimated to Become a 13 Trillion Yen Market by 2025

First, Fujitsu Chief Evangelist Iwao Nakayama explained the VR market.

Iwao Nakayama
Chief Evangelist, Global Marketing Group Corporate Executive Officer
Fujitsu Limited

Virtual reality is defined as "technology that uses science and engineering to create something non-existent by stimulating the user's five senses." At present, various manufacturers sell VR hardware, such as headsets like Sony's PlayStation VR, HTC's Vive, and Samsung's Gear VR. VR applications are also growing in terms of types and numbers.
VR is forecasted to grow rapidly from 2018 to become a three trillion yen market (27 billion USD), five times or more the present level, by 2022. Some predict that by 2025 VR will surpass the 12 trillion yen TV market to reach the 13 trillion yen level (115 billion USD).

Allow me to introduce some of the latest VR case examples. The ShelfZone service enables users to shop from home in VR stores, providing a shopping experience with impressive realism without leaving your own room. I think the world depicted in the video below will become commonplace.

ShelfZone VR Shopping Video

Facebook acquired Oculus in 2014 and released a social VR application, Facebook Spaces, in 2017. Facebook Spaces enables users to socialize with real-world friends using avatars in a VR world. In addition to these examples, many very attractive, accessible VR services have already been launched.

Cutting-edge VR/Mixed Reality (MR) Use Cases in Medical Treatment

Next, Dr. Maki Sugimoto, a leading authority on VR use in healthcare, took the stage and spoke about the latest VR/MR use cases in the field of medical treatment.

Maki Sugimoto, M.D./Ph.D.
Founder, Managing Director, and COO, Holoeyes, Inc.; Founder and CEO, Mediaccel, Inc. Visiting Researcher at the Research Center for Advanced Science and Technology, Department of Information Physics and Computing, The University of Tokyo

As technology to make medical doctors' work more efficient, I am paying close attention to VR, augmented reality (AR), and MR. Recently, the term extended reality (XR) is sometimes used to encompass all three.

We are unable to perceive 3D space with conventional flat 2D monitors. For example, in the field of medical treatment, when surgeons use a flat 2D monitor to look at images of the inside of the abdomen during endoscopic surgery, they cannot perceive the actual depth and positions of the organs accurately.
Because of this, some statistics show that endoscopic surgeries cause more organ damage than open abdominal surgeries.

To solve such problems, we have developed technology to see things "in space." One example is OsiriX, which reconstructs two-dimensional CT and MRI images to produce three-dimensional images.
The software uses data pieces called polygons to transcribe the shapes of individual organs and express them via coordinates, thereby enabling users to grasp cancer and veins intuitively in space.
Making use of the binocular parallax of the human eyes, OsiriX synchronizes different data for each eye to create a stereoscopic view. By donning a head-mounted display, the user experiences depth and an improved feeling of immersion.
We apply this VR technology in surgery. For example, surgeons can conduct pre-op practice and simulation using actual patient data.

Another example is the use of VR in Heart Simulator (*), which we co-developed with Fujitsu. Heart Simulator simulates the heart's complex behavior and disease conditions as well as the heart's shape. It provides a stereoscopic view of heart movement, blood flow, pressure, and electrical signal flow in VR.
Since this technology enables us to visualize how a certain operation will change blood flow and pressure, we may be able to use it to pass on veteran heart surgeons' know-how and techniques to younger doctors.

VR/MR Provides More Realistic Healthcare Education (Japanese)

*: We ran a trial to display the output data of a simulation in VR.

An example of more advanced technology is mixed reality (MR).
We are promoting use of MR in surgery as well.

Microsoft's HoloLens can constantly scan the environment around the wearer. So, if both an experienced surgeon and a younger colleague wear a HoloLens during surgery, they can share in real time information like "We will make an incision here" or "Be careful of the artery there." This has realized benefits such as shorter operation times and reduced patient burdens.
These technologies can also be applied to healthy people. According to one survey, 60% of expectant mothers worry about their pregnancies, which sometimes leads to depression and other symptoms. New research suggests that using XR to view an unborn baby makes expectant mothers grow attached to their babies and creates a bond between the mother and child before birth, thus preventing depression and attachment disorder.

Will Technological Advancement Realize the Future Portrayed in Sci-Fi Novels?

Following Dr. Sugimoto, science fiction writer Mr. Taiyo Fujii predicted the future of VR from a sci-fi perspective.

Taiyo Fujii
Science Fiction Writer

First please allow me to introduce myself. I used to develop 3D software, but one day I felt the sudden urge to write a novel. I started writing on a train using an iPhone and debuted as a novelist in 2012.

My first novel, Gene Mapper, is set in 2038 and concerns genetically modified crops. In the book, I devoted many pages to describing future person-to-person communication using VR.
The characters in the story wear special contact lenses to always be in a state of MR/XR. Many businesspeople use the contact lenses. For example, at a business conference, the contact lenses project the avatars of other conference participants in front of them, and the avatars talk with each other. That is the future I wrote about.

I put extensive detail into the novel. For example, delays of about 0.2 seconds to 1 second are inevitable in Internet communications such as Skype today. That is why you cannot sing in chorus while teleconferencing. In Gene Mapper, I write about a world where avatars correct this latency to facilitate smooth communication, remove glitches in conversation (such as ums and ahs), and automatically delete any remarks that may violate non-disclosure agreements (NDA) during a business conference.

I put a lot of effort into describing this technology to make such a future believable to my readers. In doing so, I found Fujitsu's body area network (BAN) webpage very informative (http://pr.fujitsu.com/jp/news/2014/02/12.html).
BAN is a technology that uses the human body as a network transmission path. I wrote the story imagining that BAN might be able to send megabyte-sized images to the contact lenses.

Like the future I depict in my novel, technology continues to change the world we live in. As we all know, since the iPhone came out in 2012, our communication styles have completely changed. I think VR will transform our lives still further.

VR Is Transforming Our Lives

In the second half of the conference, we discussed the future that VR will bring about.

Will Virtual Reality Surpass Reality?

Nakayama: Will virtual reality surpass reality? I think many discussions today focus on this topic. Would you tell us about your specific efforts and perspective on this?

Sugimoto: Nowadays, we can do many things with VR, like currencies and communication. Moreover, technologies such as MR that combine VR with reality are becoming popular.
Ultimately, it is important to compartmentalize reality and VR and to find a suitable way for them to coexist.

Fujii: As soon as VR can work without requiring a computer connection and gains mobility, like the standalone VR headset Oculus Go, a paradigm shift will occur.
I think VR will approach the original meaning of the word "virtual," namely to be "capable of producing a certain effect or result," so the whole argument of virtual vs. real will become irrelevant.

Nakayama: I think use of Facebook Spaces will spread rapidly. Facebook Spaces is a world consisting solely of avatars that communicate. In response to the question, "Will virtual reality surpass reality?" I say I have high expectations for services like this one.

Sugimoto: The other day, I went to Palo Alto on business. There, I saw a conference carried out with YouTube avatars. I hear that using avatars makes it easier for people to say what they really think as well as to argue against others, therefore facilitating substantial discussion.

Nakayama: The second theme is the consumerization of medical treatment and health. What is your opinion on this?

Sugimoto: These days, ordinary people often create better content than professionals, and consequently professionals are losing their jobs. The border between the B2B domain and B2C domain is becoming unclear. Lately, I have become aware of the importance of the background in VR, which is usually black. When I was making content for practicing surgery, I tried using an image of an operating room for the background. This was well received among young doctors, who said, "I feel like I am actually practicing surgery in a real operating room." I think this is a kind of place where we can find clues for a new market.

VR Also Helps Pass on Tacit Knowledge in Medical Practice

Nakayama: Please share with us some hints for successfully applying VR/AI in business.

Masahiro Watanabe
Senior Manager,
Solutions Development Dept.I
Medical Solutions Div. III
Healthcare Solutions Unit II
Fujitsu Limited

Watanabe: On April 11, 2018, Fujitsu launched Heart Explorer, which utilizes VR. This heart simulator was created through joint research between Fujitsu and the University of Tokyo. The data output from the simulated heart, such as heart movement, blood flow, and coronary circulatory blood pressure, is used directly as a teaching material. The viewer (Heart Explorer) and the zSpace display recreate the complex structures of heart blood vessels and alterations in heart muscle over time, enabling students to have a virtual hands-on experience. Using the separate software zView for AR display, images can be shared with students who are not wearing VR glasses. With Heart Explorer, students can experience textbook knowledge and enhance their understanding.

Nakayama: What do you think about the integration of VR and AI?

Fujii: Suppose the price of the contact lens in my novel costs 30,000 yen (about 270 USD). If 300 million people use it, that creates a nine trillion yen market (81 billion USD). Once VR becomes as ubiquitous as mobile phones, it will have a huge business impact. Since everyone can make use of this business opportunity, I would love to see Japan-born innovations in must-have technologies like VR and AI.

Sugimoto: I think that VR and AI will be key to solving the tacit knowledge issue in medical practice. Describing seasoned doctors' techniques in words or by illustration is difficult. As a result, it is customary for younger doctors to learn by observation. I have been developing a system which uses VR to record expert doctors' actions and eye movements, enabling younger doctors to experience and practice with the VR data using a head-mounted display. The data also doubles as a surgery record. Moreover, if you create a database from the VR data and apply deep learning, the data can be used to treat other patients as well.

Consumers are also adopting VR. Today, people can easily convert their own CT or MRI images into VR data using free software. It would be nice if the popularization of such VR technology contributed to raising health awareness.


Maki Sugimoto, M.D./Ph.D.
Founder, Managing Director, and COO, Holoeyes, Inc.
Founder and CEO, Mediaccel, Inc.
Visiting Researcher at the Research Center for Advanced Science and Technology, Department of Information Physics and Computing, The University of Tokyo

Taiyo Fujii
Science Fiction Writer

Iwao Nakayama
Chief Evangelist, Global Marketing Group Corporate Executive Officer
Fujitsu Limited

Masahiro Watanabe
Senior Manager,
Solutions Development Dept.I
Medical Solutions Div. III
Healthcare Solutions Unit II
Fujitsu Limited