Cars are not the only targets for commercial use of self-driving technologies--these technologies are steadily advancing for ships as well. The Japanese government is working to launch self-navigating ships by 2025. Self-navigating is the maritime version of self-driving; it requires technological development to support autonomous operations that differ from those of cars. This article provides the latest updates on the self-navigation support technologies currently in development.
Aiming at Commercial Use of Self-driving Technologies for Ships by 2025
Development of self-driving technologies continues in the automobile industry. Anticipation for these technologies is growing particularly in the commercial vehicle industry due to serious driver shortages, traffic accidents, and the need for commercial vehicles in depopulated areas. In reality, installation of driving assistance technologies, such as automatic braking, is becoming mandatory starting with large trucks in consideration of the severity of damage due to traffic accidents.
Development of automation technologies is underway not just for cars but for other means of transport, including ships. The Japanese government commenced efforts toward commercial use of self-navigating ships that can be remotely controlled and operated autonomously. Under the Future Investment Strategy 2017 approved at a Cabinet meeting in June 2017, the government stated that it aims to commercialize self-navigating ships by 2025 to improve maritime logistics.
For self-navigating ships, research is underway to explore ways to use technologies such as satellite communication, the Internet of Things (IoT), and AI to achieve completely autonomous navigation, including avoidance of collisions with other ships and obstacles. As companies in various fields engage in R&D on advanced technologies to achieve commercial use of self-navigating ships, Fujitsu too is working on means to support autonomous ship operations. The following sections introduce Fujitsu's three solutions: ship collision avoidance, port service productivity enhancement technology, and automation of 24/7 surveillance currently handled by humans.
Avoiding Collisions Caused by Human Cognitive or Judgment Errors: Maritime Traffic Management
When the sea is crowded with ships, the distance between them decreases and the likelihood of accidents rises. Globally, 1,642 serious accidents occur at sea every year. Ship collisions account for 358 of these (2014 survey by the Maritime Bureau of the Ministry of Land, Infrastructure, Transport, and Tourism). According to this survey, human cognitive and judgment errors (e.g., misreading, assuming, and prediction errors) were the major causes of ship collisions in Japan.
Fujitsu's Maritime Traffic Management Solution uses ICT to address this issue. We harness AI and Big Data analysis to prepare two problem-solving approaches for actions to take immediately before a collision occurs: support for work leading to pre-collision actions, and prevention of the need for pre-collision actions.
Image 1: Fujitsu's two approaches
Near miss risk detection to support work leading to pre-collision actions
The first approach is to detect near miss risks in order to support work leading to pre-collision actions. Noticing the danger of a collision that may occur in the immediate future requires human experience and know-how. In this approach, risks are quantified as values to support detection of such dangers.
Conventionally, alerts go off when there is a danger of collision. However, in some cases, such alerts have not helped the crew's decision-making because alerts were issued too often. To address this issue, we developed a risk engine. This engine helps human decision-making by calculating risks based on accurate course prediction in order to decrease the number of unnecessary alerts and reduce the chance that risks are overlooked.
Prediction and reduction of dynamic hotspots to reduce the need for pre-collision actions
Another approach is to reduce the need for pre-collision actions. This solution supports the creation of a mechanism to predict and prevent generation of dynamic hotspots, which are areas with high collision risks due to the presence of multiple ships.
Dynamic hotspots keep changing over time and lead to chain reactions of ships trying to avoid collisions. Decisions by individual ships or communication among ships does not in and of itself eliminate dynamic hotspots. Therefore, it is important to predict and avoid the generation of dynamic hotspots in advance.
Image 2: Dynamic hotspots: definition and issues
To predict dynamic hotspots, this solution extracts the course patterns of each ship as well as the ship type from the Automatic Identification System (AIS) data. The extracted patterns are accumulated in a knowledge base and used to predict short-term courses and long-term traffic conditions. Combined use of this knowledge base and AI-based prediction enables accurate course and traffic prediction.
Fujitsu is now testing the Maritime Traffic Management Solution in Singapore. The Port of Singapore is one of the most crowded ports in the world, and maritime industries play an important role there, accounting for 7% of the country's GDP. Singapore is also a hub port for the ASEAN region.
In this test, we deployed an advanced solution using vessel traffic management technologies for the Port of Singapore to ensure safety and promote efficient traffic management.
Smart Port Operation to Optimize Port Services as a Whole
In addition, Fujitsu is developing the Smart Port Operation Solution for bulk terminals. A bulk terminal is a port where bulk carriers arrive. This solution is also undergoing joint testing at the Port of Singapore to examine improvements in port service efficiency and productivity.
At the Port of Singapore, ships sometimes must wait for a few days off the coast before they can dock. Berth congestion at the port entrance and gate congestion at the port exit are bottlenecks for port services, which may result in high costs or opportunity loss.
To minimize berth/gate congestion, Fujitsu is developing two solutions--the Intelligent Berth Planning and Gate Operation Optimization Solutions--for ports with many uncertainties, such as delayed ship arrivals and cargo unloading due to various factors (e.g., weather).
Minimizing berth/gate congestion
The Intelligent Berth Planning Solution uses AI technology (adaptive learning) to predict how long ships will stay at the port. Based on the prediction results, it reserves the most appropriate berth and prepares a port entry plan. This alleviates the situation in which ships must remain off the coast until a berth opens.
The Gate Operation Optimization Solution can be used as a decision-making tool to optimize the operation of gates where trucks pass through. It can handle increased transaction volumes and address issues such as limited land space and workforce aging.
We will apply the test results obtained in Singapore to other ports where transaction volumes are expected to increase, such as those in emerging countries.
AI-driven Surveillance Automation for Collision Avoidance
One way to support self-navigation is surveillance automation. Traditional 24/7 surveillance performed by crews imposes a heavy burden on crew members.
Fujitsu is developing a solution to reduce this burden. More specifically, our system monitors the course using a surveillance camera installed on the bow and sets off alarms as necessary. The solution's foundation is an image recognition system using AI technology (deep learning).
Development of an image recognition system that automatically recognizes other ships
This image recognition system monitors other ships on the course as well as floating objects such as signs, buoys, and fixed fishing nets. AI then analyzes the images from the surveillance camera to identify ships. Anything other than a ship will be identified as an obstacle. Ships are currently categorized into car carriers, bulk carriers, tankers, containerships, and miscellaneous small ships. However, this ship lineup changes between a port where many ships anchor and open ocean ship courses. Therefore, a future task is to consider how many and which ship categories are necessary.
Image 3: Result of ship recognition trial
To achieve course monitoring, we had to solve some issues. For example, there were cases in which waves were erroneously detected as ships in conventional image processing, and other cases in which radar failed to distinguish between waves and small ships. In addition, monitoring at night or through thick fog requires a night-vision camera or infrared camera instead of a regular optical camera.
In ship recognition testing in which AI analyzed course images taken by a camera, we used training data to create a Convolutional Neural Network (CNN) model, which is specifically for image analysis. We evaluated the test results and performed tuning, after which we continued to add training data until the test results reached a certain level. Note that training data is data that acts like a teacher who gives problems and answers.
Image 4: Method for learning ships
As a result, the feature points of waves were somewhat reduced in the learning process, and waves were no longer erroneously detected. We achieved this thanks to AI's learning effects, which image processing cannot match.
In the example of Image 3, the system is set to show objects that have been identified with an accuracy of 80% or higher. Although elements of the background cityscape were mistakenly detected as ships at the start of the learning process, repeated learning resulted in the extraction of ship characteristics, which in turn led to correct ship recognition.
In a system trial with ships cruising in the Uraga Channel, multiple ships in the distance, the type of ships that were crossing, and ships on the horizon were successfully recognized. This trial used images taken by a common video camera, and objects as far as four to five kilometers away were detected. Objects identified as ships by human eyes could also be identified by AI, and we confirmed that a high-spec camera is not necessary for this level of ship identification.
What are the issues for future practical application?
Evaluation of AI recognition results in hypothetical cruising situations revealed some points for improvement. For example, learning of ships using images taken from the front and back becomes necessary after the recognition rate improves by a certain amount. Fujitsu plans to install a camera on a ship, take photos of other ships, and import the image data. Since ships do not have many features to distinguish ship types on their fronts and backs, it is necessary to generate 3DCG models as needed to produce many pieces of training data.
Also, AI must be able to identify target objects under poor photographic conditions, such as backlighting or varied sunlight angles (e.g., in the early morning or evening).
Among the aforementioned issues, low visibility countermeasures for night and thick fog are necessary. To this end, Fujitsu is developing a system that uses an infrared camera. However, at present, images taken by infrared cameras have a low recognition rate due to the small volume of training data. Going forward, we plan to increase the amount of training data and improve accuracy. Another plan is to use learning data obtained through the current proof of concept (PoC).
Image 5: Ship recognition using images taken by infrared camera
When using the monitoring system, camera cost is an issue. Monitoring of small objects in the distance requires a high-resolution camera. However, use of a higher-spec camera makes the system more expensive, creating a hurdle for companies that wish to keep operating costs low. That said, image recognition systems that use AI are low cost so long as learning and modeling are performed. Therefore, this system can be deployed widely once the camera problem is resolved.
In addition to supporting self-navigation, Fujitsu also intends to use AI to realize smart port services. We will help improve the efficiency and productivity of port operations through safe cruising support by checking in-port or on-course inter-ship distances, automatic ship mooring capabilities, and self-driving trailer monitoring capabilities.
Steady Advancement for the Era of Self-navigating Ships in 2025
Establishing various technical elements is a prerequisite for realizing practical use of self-navigating ships. In the shipping industry, where development of ICT is said to be slow, realization of self-navigating ships will be key to recovering lowered global market share.
To this end, effective use of innovative technologies such as AI, IoT, and big data is essential. Fujitsu continues to carry out R&D to leverage these technologies in more comprehensive, integrated ways to meet diverse customer needs. By offering various solutions, we aim to realize safer marine areas and cruising.
Public Private Partnership Division
Co-Creation Business Group
Digital Transformation Business Unit