Research of Fetal Cardiac Ultrasound Screening Using AI

Today, various fields are utilizing AI technology. The medical field is making advancements to implement image diagnosis supporting technology, which aid in the examination of medical images. Fujitsu is collaborating with RIKEN and Showa University to study congenital heart disease, which is a major cause of infant mortality, by developing methods for early diagnosis during the fetal stage.

Detecting Congenital Heart Disease with Effective Screening Tests

One in a hundred newborn infants suffers from congenital heart disease. This disease is defined by defects found in the heart or vein connections. Among the various types, severe congenital heart disease is the cause of infant mortality for 20% of all cases. Meanwhile, medical treatment technology constantly advances. Particularly if the disease is detected before birth and appropriate measures are taken, the chance of successful treatment increases and future prospects improve. The key to achieving this is fetal cardiac ultrasound screening, which is a testing method conducted during pregnancy examinations.

The collaborative research group* formed by Fujitsu, RIKEN, and Showa University developed the novel Fetal Cardiac Ultrasound Screening Technology. It utilizes AI (Artificial Intelligence) to automatically detect heart defects in fetuses with high accuracy and shows the results in an easy-to-understand way.

Training data that contains information on healthy fetus hearts, such as the correct position that each part should appear in, was created and learned by the AI, after which the data is compared with the actual heart and great veins in order to detect any defects. The results are displayed in list format in real time.

*: Fujitsu and RIKEN established the RIKEN AIP-Fujitsu Collaboration Center in April 2017, which advances collaborative research with the aim of creating "AI technology that predicts the unpredictable." Moreover, Showa University Hospital operates four maternity departments that have the highest number of births among all of Japan's university hospitals.

A Research Project Started to Solve Social Issues

The Fetal Cardiac Ultrasound Screening Technology is being developed in cooperation with OB-GYN doctors. Many doctors strongly feel the need to detect congenital heart disease at an early stage. In treating congenital heart disease, establishing detailed plans before birth is thought to increase the chances of success (e.g., by making sufficient preparations during the fetal stage at a hospital where there are medical specialists and giving birth under a planned childbirth schedule).

However, although the rate of diagnosis through screening tests has increased in recent years, it remains insufficient. The heart of a fetus is small and has a complex structure and fast movements. Advanced technology is required to observe it through ultrasound examinations. The examination technique depends greatly on experience, which leads to disparities based on the skills of those conducting examinations. Moreover, a decrease in the number of OB-GYN doctors and the fact that such doctors are concentrated in major cities have led to a shortage of doctors, causing disparities in medical care levels among regions as well.

To fill these gaps, the ability to accurately detect heart disease in fetuses in order to pass such information to doctors has become a major issue. Fujitsu established a research project team together with Dr. Masaaki Komatsu, an OB-GYN doctor and research scientist in the Cancer Translational Research Team (Team Leader. Ryuji Hamamoto), the RIKEN Center for Advanced Intelligence Project, and Dr. Ryu Matsuoka, an expert in fetal cardiac ultrasound diagnosis and associate professor in the Department of Obstetrics and Gynecology (Prof. Akihiko Sekizawa), Showa University School of Medicine. In September 2018, they jointly announced that they had developed a technology to support diagnoses by OB-GYN doctors.

Ultrasound Imaging AI Research Requires Advanced Analysis Technology

As described above, ultrasound diagnoses of fetuses require advanced examination technology. In addition, AI research in ultrasound imaging is extremely complicated. AI technology has been already used in the medical field for AI imaging diagnosis support systems in radiography, CT scanning (computed tomography), and MRI (magnetic resonance imaging), all of which use images to detect pathological changes. To employ machine learning, a much-talked-about application of deep learning, one must collect a sufficient amount (over 100,000 pieces) of both normal and abnormal data for learning to enable a high recognition accuracy.

Meanwhile, congenital heart disease occurs very infrequently, which makes it difficult to collect the necessary amount of abnormal data. This is why the team used anomaly detection technology**, which learns only from normal data and recognizes any data that deviates from the learned pattern as an abnormality. However, ultrasound imaging is susceptible to noise (shadows), which corrupts the data and causes even normal data to be interpreted as abnormal. So, to detect abnormalities with high precision using conventional technology, a massive amount of normal data with a variety of noise patterns was needed.

**: Machine learning technology that learns only from normal data, and recognizes any data that deviates from the learned pattern as an abnormality.

Anomaly Detection Using Object Detection Technology

Faced with this challenge, the collaborative research group studied the use of robust machine learning technology***, which can make precise predictions even with small amounts of data or incomplete data. The team then developed automatic detection technology for abnormalities in fetal heart structures by using object detection technology, which is a form of AI technology.

There are very few individual differences in the heart structures of healthy fetuses. The heart is always in the same position with the same parts (e.g., valves and veins). Ultrasound screening detects abnormalities by checking in which position each part should appear under normal circumstances. The research group focused on this fact and used object detection technology, which can effectively distinguish the positions and types of multiple objects captured in an image, to perform anomaly detection.

In machine learning that applies to object detection technology, "training data" with the correct name and position added to each part of normal images is used as the normal information. The heart parts are detected from ultrasound images acquired as the target of examination, using the object detection technology developed by learning from the training data.

Figure 1: Example of anomaly detection in a fetal interventricular septum using object detection technology.
(a) How the interventricular septum should appear in a normal fetal heart is shown (outlined in red), then annotated as training data in the object detection technology. (b) The actual interventricular septum (outlined in white) is detected for a given medical case (ventricular septal defect). Based on the detected difference, this case is determined to be abnormal.

This technology can be used to detect specific parts with very high accuracy. As a result, by comparing the detection state with the positions of the parts of a normal heart, abnormalities can be detected even with an incomplete image that has some noise.

***: Conventional machine learning cannot make satisfactory predictions without a massive amount of data or high quality, complete data. Robust machine learning technology is an innovative base technology in machine learning that overcomes this problem by accurately predicting the future (i.e., robust predictions) based on a small amount of data or incomplete data.

Automatic Anomaly Detection in Real Time with the Fetal Cardiac Ultrasound Screening Technology

The collaborative research group applied this object detection technology to develop the Fetal Cardiac Ultrasound Screening technology. As training data, they used 2,000 ultrasound images of normal fetal hearts to achieve a high level of diagnostic accuracy.

In the current technology, the examiner must employ an ultrasound probe to sweep the pregnant woman's abdomen, and must scan the fetus in a fixed direction from the stomach to the heart. The 18 different parts of the fetal heart and peripheral organs are compared against the acquired moving image, and a rapid computation is performed to gauge the confidence level of each part and if it appears as it should. These results are then displayed on the control screen in real time. If any parts are not detected, or are detected with a low level of confidence, they are determined to be abnormalities.

For example, the tetralogy of Fallot, a frequent congenital heart disease, is a defect that is marked by a hole in the interventricular septum and a narrow great vessel on one side out of those protruding from the heart. These parts are often not captured in ultrasound scans, but this technology checks for their positions that should be there but are not, and can detect such cases as abnormalities.

Optimizing Diagnoses with the Novel Display Method of Detection Results

Detection results are displayed in the colored barcode-used diagram. The horizontal axis represents the sweep scanning timeline, while the vertical axis represents each part. If the AI detects something with a high level of confidence, it is colored blue, and if not, it is colored gray.

Figure 2: The output image of the developed Fetal Cardiac Ultrasound Screening Technology

Figure 3: The novel display method of the examination results.
The top diagram (a) shows a normal case, while the bottom diagram (b) shows an case of the tetralogy of Fallot. This can be used to easily understand and explain which part was the source for abnormality detection. In this case, the red outline shows the critical point.

Unlike the case of still images, when confirming the cross-sectional videos used in operation for ultrasound examinations, playback takes a long time, making it hard to judge at a glance. However, this technology enables doctors to see the detection state for each part, which greatly reduces the confirmation time.

Moreover, use of object detection technology ensures that the quality of examination results is consistent regardless of any skill gaps among the examiners. This also has the benefit of enabling examiners to use the displayed diagrams to grasp and explain which part was detected as an anomaly. Ultrasound specialists have praised this technology system, saying that it is easier than relying on videos to grasp and understand the entire structure.

Solidifying Plans and the Future Ahead

Use of this newly developed technology fills the gaps in radiographic image interpretation skills among examiners and makes possible supporting fetal ultrasound examinations. In addition, the technology is expected to more accurately detect severe and complex congenital heart disease, which requires early detection.

Going forward, there are plans to acquire an additional several hundred thousand ultrasound images of fetuses, which will be used to improve the screening accuracy and to expand the objects of verification examinations. In addition, by calling on relevant scientific societies both in Japan and overseas, the reliability will undergo more thorough verifications by collecting and verifying data on various types of congenital heart disease.

Moreover, by applying this to Fujitsu Human Centric AI Zinrai****, which is Fujitsu's AI technology platform, we aim to achieve early clinical applications in the field of fetal cardiac ultrasound screening by utilizing AI in cooperation with ultrasound equipment manufacturers.

****: A system of AI technology provided to society to achieve Fujitsu's goals of developing collaborative, human-centric AI and AI that achieves continuous growth.

As a Digital Co-creation Partner

This research project led to a breakthrough achieved by co-creation among people with completely different backgrounds, including AI researchers at Fujitsu and RIKEN as well as ultrasound specialists involved in OB-GYN medical care.

The collaborative research group's ultimate goal is to contribute to early diagnosis of congenital heart disease by having this technology become used in every hospital and pregnancy exam. Going forward, Fujitsu aims to provide medical specialists with this technology by combining accurate screenings and remote diagnosis using this technology for hospitals that are located in regions with few medical specialists and the latest examination devices; the goal is to spread use of the technology not just throughout Japan but globally.

At Fujitsu, we are pursuing co-creation with people who have digital technology and a wide range of know-how in order to advance this research project broadly, and we hope to achieve a prosperous future for people by continuing to provide innovative value to society.