How Will the Judging Support System International Gymnastics Federation Change the World of Sports?

Main visual : How Will the Judging Support System International Gymnastics Federation Change the World of Sports?

Gymnastics competitions require advanced judging technology, and the increasing complexity of elements has led to cases in which it is difficult to fairly judge competitions with the human eye alone. As a result, the development of judging support system that utilize AI to support judges is advancing. Here, we introduce a discussion that gives an overview of the judging support system and describes its future.
Fujitsu Forum 2019 Frontline Session Report

AI Technology for Supporting Fair Judgment

Before the panel discussion, Fujitsu Director Chairman Masami Yamamoto summarized the current state of the sports business in Japan as well as Fujitsu's efforts in sports-related ICT. As examples, he introduced case studies on B.LIVE, the next-generation live viewing system of B.LEAGUE, and the JAAF RunLink Project, which aims to grow the population of runners and invigorate the health market through collaboration with the Japan Association of Athletics Federations.

Photo : Masami Yamamoto Director Chairman, Fujitsu Limited

Masami Yamamoto
Director Chairman,
Fujitsu Limited

Next, Hidenori Fujiwara, who leads Fujitsu's Gymnastics Project, gave a detailed explanation of the judging support system, which supports judging in gymnastics competitions using AI.

Collating Data Obtained by 3D Laser Sensors with an Element Database

Photo : Hidenori Fujiwara Head of the Sports Business Development Division II, Sports & Cultural Event Business Development Unit, Fujitsu Limited

Hidenori Fujiwara
Head of the Sports Business Development Division II,
Sports & Cultural Event Business Development Unit,
Fujitsu Limited

The judging support system, which the International Gymnastics Federation has decided to officially adopt, employs 3D laser sensors to support judging in gymnastics competitions. About two million lasers are beamed at the gymnast every second, which enables the system to calculate distances based on reflection times, thereby capturing the human form in three dimensions. One major benefit of this system is its ability to take measurements without attaching markers to the gymnast's body. In addition, an algorithm has been developed that fits the 3D data taken from the gymnast to skeletal models, making it possible to obtain 3D data on joint positions. This is then collated with the element database.

The element database is a compilation of data on top athletes from Japan and around the world provided by the International Gymnastics Federation and Japan Gymnastics Association. This dictionary is based on data collected during various competitions and practice sessions. The judging support system helps achieve fairness in scoring by executing a series of processes, from capturing data using 3D laser sensors to skeletal recognition and matching against the element database.

Figure : Upper-left: Video captured by cameras attached to sensors; Lower-left: Time axis; Middle: Four-quadrant silhouette. For each element, the points of focus for judges are displayed on the right.

Upper-left: Video captured by cameras attached to sensors; Lower-left: Time axis; Middle: Four-quadrant silhouette. For each element, the points of focus for judges are displayed on the right.

Automated judging function matches the data obtained with the element database; when the element is identified, the element’s name, group, difficulty level, and score are automatically output. The current development goal is to provide multi-angle views and automated judging function for 5 apparutases by 2020, and all 10 apparutases by 2024.

Figure : The scoring support system is composed of three elements: 3D laser sensors, software for fitting skeletal models, and the element database.

The scoring support system is composed of three elements: 3D laser sensors, software for fitting skeletal models, and the element database.

Judging Process Visualization Prevents Misjudgment and Ensures Fairness

Next came a panel discussion between Koji Gushiken, president of the Nippon Sports Science University, and Airi Hatakeyama, a former member of Japan’s national rhythmic gymnastics team. They were joined by Masami Yamamoto and Hidenori Fujiwara.

Photo : Panel discussion with (from the left) Airi Hatakeyama, Koji Gushiken, Masami Yamamoto, and Hidenori Fujiwara

Panel discussion with (from the left) Airi Hatakeyama, Koji Gushiken, Masami Yamamoto, and Hidenori Fujiwara

Gushiken: The impetus behind the demand for an AI-based judging support system in competitive gymnastics were two misjudgments: one that occurred during the 2011 world championship in Tokyo, and another in an international competition in 2012. Both cases involved performances by the same gymnast, and after some dispute, the rankings were changed in both instances. This led to heightened demand to use technology to support proper scoring.

Photo : Koji Gushiken President, Nippon Sport Science University Vice Chairman, Japan Gymnastics Association

Koji Gushiken
President,
Nippon Sport Science University
Vice Chairman,
Japan Gymnastics Association

Hatakeyama: When I reflect on my time as a competitor, there were times when I felt a disparity between our sense as competitors and judges' scores. There were times when I felt there was a gap between spectators’ loud cheering and expectations versus the actual scores. As I was competing, I hoped that the scores would satisfy everyone. By implementing ICT, I believe that the reasoning behind scoring can become easier to understand for the spectators in the stadium as well as for the people watching on TV. I think that use of ICT can also fulfill gymnasts’ expectations for fair scoring.

Photo : Airi Hatakeyama Former member of Japan’s national rhythmic gymnastics team

Airi Hatakeyama
Former member of Japan’s national rhythmic gymnastics team

Creating an Element Database by Factorizing Movements, and Defining Scoring Standards

Fujiwara: Fujitsu is making efforts to utilize AI to score gymnastics competitions. Judging gymnastics elements is very complicated. The posture and where the elements are performed must be considered. For example, the pommel horse is divided into three sections by the handles, and the gymnast moves between them. Moreover, how that posture is maintained throughout the series of movements is crucial. These parts of elements are combined to complete the whole element, so all of these traits must be learned by deep learning. This is where it gets complicated. As shown in the following figure, the system observes the posture, positions, and series of movements, and then determines that it is the scissors to handstand element. To determine the element, the element database is very important. There are 819 elements for men, and 549 elements for women. After starting collaborative research with Japan Gymnastics Association, we visited Mr. Gushiken and talked to him extensively in order to collect information to create our database.

Figure : Element judgment mechanism for the pommel horse

Element judgment mechanism for the pommel horse

Gushiken: The engineers at Fujitsu determined that this vast number of elements could be factorized into 475 parts of elements and basic movements. When I heard that, I was quite surprised. Those of us involved in gymnastics would never have thought of such a thing; we have never systematized it like this. I felt as if this fusion of gymnastics and technology had sparked some kind of chemical reaction. However, the fact remains that much of the scoring rules in gymnastics are rather vague. I imagine you must have gone through great pains to develop such a system.

Fujiwara: It was definitely not easy. (Laughs) For example, with the lateral balance posture, the scoring textbook explains that if it is kept straight, you do not deduct points; if it is slightly bent, you deduct 0.1 points; and if it is very clearly bent; you deduct 0.3 points. Such a rule is too vague to implement in a system. To address this, we sent engineers to the International Gymnastics Federation in Lausanne, Switzerland to clear up the ambiguities one by one. In this instance, it was decided that "straightness" would be defined by a posture in which the angle of the knee and the waist is over 170 degrees.

Judging Support System Using AI Reduces Judges’ Burdens

Fujiwara: The more we learned, the more we understood how difficult it is to judge gymnastics elements. In gymnastics, there are two types of judges: the D judges, who judge element difficulty, and the E judges, who judge performance execution. There are also other judges with deep experience: line judges, timekeepers, and referees, as well as four assistant judges on standby. In addition, some high-level judges make final decisions. When I first learned from Mr. Gushiken that there are about 123 judges, I was very surprised.

Gushiken: In next year's international competition, a total of 196 gymnasts, both men and women, will compete. To judge the competition, there will be about 120 judges for the men, and about 80 judges for the women—so, a total of about 200 judges. In other words, there are more judges than gymnasts. I think this fact should help people understand how difficult the task of judging is, and how much we need judging support.

Hatakeyama: Judging rhythmic gymnastics is very difficult. The rules change frequently, and under the current set of rules, it is advantageous to land as many elements as possible, which makes it even more difficult. Group performances are two-and-a-half minutes, while individual performances are one-and-a-half minutes. When you try to land as many elements as you can within that time limit, you speed up. The elements become increasingly difficult, which also makes judging difficult.

Gushiken: Speaking of speed, how is the speed of the judging support system compared to that of human judges?

Fujiwara: During the World Cup, three high-level judges from the International Gymnastics Federation came to Japan, and we provided the judging support system to judge performances simultaneously alongside the top three judges. They found that the AI’s judging speed was faster than that of themselves.

Gushiken: Incredible. In that case, I think the important thing to focus on is how the system and the judges can coexist going forward.

Hatakeyama: In rhythmic gymnastics, the artistry is an important aspect of the performance. Can the judging support system make judgments on artistry?

Yamamoto: Recently, advancements have been made in research on AI’s ability to judge the quality of pictures and paintings, and it is now very close to human sensibilities. At Fujitsu, we are very interested in judging artistry. I think this is a very worthwhile challenge to tackle. In the end, we want to create a system that prioritizes human judgment.

Overcoming Unanticipated Barriers in the Course of Development, and Filing of 63 Patents

Fujiwara: The development of the judging support system was a series of hardships. However, the fact that we were exploring uncharted territory is precisely why we encountered so many barriers and could learn so much. For example, when we went to the arena with the engineering team to do operation checks, we found that there was a lot of noise entering the laser sensors. We could not determine the cause. The engineers did not know either, so we reluctantly decided to open up the sensors. The culprit was magnesium carbonate, which the gymnasts use as anti-slip powder. The fans on the sensors were vacuuming up this powder, which flies all around the stadium. We took care of the matter right away, but this was not something we had anticipated.

Also, when we were discussing with the engineers about which gymnastics apparatus out of the 10 we should start with, we decided that the pommel horse would be the easiest because it looked as if the gymnasts were just moving around on the stand. However, this was not the case. There are about 12 elements involved in a pommel horse performance, and which movements are actually considered to be elements are difficult to judge, even for the judges. Some people involved in gymnastics asked us why we would start with the most difficult apparatus. However, Fujitsu’s company culture encourages us to tackle the most difficult challenges, so I think it was all right in the end.

Also, when we were trying to use data to fit the skeletal model, we ended up drawing a silhouette that looked like a freshwater prawn. (See the figure below.)

Figure : In the initial stage of development, it was difficult to distinguish between the rings and the hands.

In the initial stage of development, it was difficult to distinguish between the rings and the hands.

Fujiwara: This shows how we accidentally identified the rings as the hands. Equipment such as the rings, the horizontal bar, and parallel bars move much more than we expected. This caused the AI to misidentify body parts and apparatus. To solve this, we had the AI thoroughly learn the equipment in isolation using a method called background removal and differentiation, which improved the algorithm.

With regard to movements, it is very difficult to fit skeletal models to the cannonball position, which is nearly spherical. Another difficulty is handling the choreography on the floor and on the balance beam. Choreography is not an element, so there are many variations. Because of this, judgment mistakes are made when recognizing elements. Thus, we must have the AI learn as many examples as possible.

Through the course of this project, we filed 63 patent applications. Some possible future developments include comparing good and bad performances during training, providing useful information and data related to elements during TV broadcasts, and producing shining trails when performances are executed beautifully. We have many ideas like these. Though there are many challenges, that is what makes this so interesting.

From Video Games, Animation, and Passing Down Traditions—Expanding the Scope Beyond Sports

Hatakeyama: Adding shining effects to performances will enable viewers who are not knowledgeable about the sport to clearly understand which performance are impressive and when they are executed correctly. I think that children and elderly people will also enjoy it. These days, entertainment elements are sought after in sports, so in that sense, I think it is a positive development. I hope to see this implemented in rhythmic gymnastics.

Yamamoto: Speaking of entertainment elements, we think that the technology used in the judging support system can also be used in e-sports, video games, and animation. In fact, we have already begun collaborating with a 3D animation company.

We also think the technology can be applied to passing down traditional culture and industrial techniques. The following is a photo of a metalworking site, where workers are performing metal spinning. They use a rod-like tool to stretch metal plates into cone shapes, which are then used for parts in rockets and aircraft. Subtle adjustments are made using the entire body, including the weight applied to the rod, the angle of the waist, and the positions of the hands. We believe that our technology can help pass down artisans’ techniques, which Japan prides itself on.

Photo : ICT can be used to pass down traditional culture and artisans’ techniques.

ICT can be used to pass down traditional culture and artisans’ techniques.

Athletes Who Make the Most of ICT Will Win Gold Medals

Hatakeyama: After listening to today's discussion, I am more excited than ever about the future of gymnastics. Scored competitions are often viewed as difficult to understand, but use of technology can put those concerns to rest. I am excited for the day when scoring results satisfy everyone.

Gushiken: For the world of gymnastics to develop further, I think it is important to incorporate more technology and to coexist with such technology. As people nurture technology, technology in turn nurtures us. I think this bidirectional flow will enable athletes to develop further. We are working with international organizations to change the rules globally. I think that this will lead to major transformations in the worlds of gymnastics and sports. Regarding the future of training athletes, I now believe that the athletes who can make the most of ICT will be the ones to win gold medals.

The session ended with Mr. Gushiken and Ms. Hatakeyama expressing their hopes for the future use of ICT in scoring competitions.

Photo : From the left: Airi Hatakeyama, Koji Gushiken, Masami Yamamoto, and Hidenori Fujiwara

From the left: Airi Hatakeyama, Koji Gushiken, Masami Yamamoto, and Hidenori Fujiwara

Presenters

Photo : Koji Gushiken President, Nippon Sport Science University Vice Chairman, Japan Gymnastics Association

Koji Gushiken
President,
Nippon Sport Science University
Vice Chairman,
Japan Gymnastics Association

Koji Gushiken was born on November 12, 1956 in Osaka. After being inspired by Sawao Kato's gymnastics performance at the 1968 Olympics in Mexico, he aspired to become a gymnast himself. In the 1984 Olympics in Los Angeles, he won the gold medal in the MAG All-Around Final and in the MAG Rings. After retiring in 1985, he studied abroad in Germany, and then became a coach for the Nippon Sports Science University Gymnastics Club, where he trained gymnasts such as Hisashi Mizutori (currently the Japan national coach). He became a professor at the Nippon Sport Science University in 2005, where he received the Medal of Honor with Purple Ribbon. He was Japan’s coach at the 2008 Beijing Olympics, where he led the Japanese team to second place in the MAG Team Final, and led Kohei Uchimura to second place in the MAG All-Around Final. He became president of Nippon Sport Science University in April 2017, where he is focusing his efforts on developing all sports.

Photo : Airi Hatakeyama Former member of Japan’s national rhythmic gymnastics team

Airi Hatakeyama
Former member of Japan’s national rhythmic gymnastics team

Former member of Japan’s national rhythmic gymnastics team Airi Hatakeyama started rhythmic gymnastics at the age of six, and she passed the Fairy Japan audition while in the ninth grade, becoming Japan’s representative. In 2012, at the age of 17, she competed with her team at the London Olympics, and the team won seventh place. After that, she was admitted to the Japan Women's College of Physical Education, and she competed in the 2015 Rhythmic Gymnastics World Championships, where her team won the bronze medal in the group Ribbon, Japan’s first in 40 years. She also competed in the 2016 Olympics in Rio de Janeiro with her team, which won eighth place. She subsequently retired. She currently works as a sportscaster and model.

Photo : Masami Yamamoto Director Chairman, Fujitsu Limited

Masami Yamamoto
Director Chairman,
Fujitsu Limited

Photo : Hidenori Fujiwara Head of the Sports Business Development Division II, Sports & Cultural Event Business Development Unit, Fujitsu Limited

Hidenori Fujiwara
Head of the Sports Business Development Division II,
Sports & Cultural Event Business Development Unit,
Fujitsu Limited