Pilot project 3 – Autonomous shipping technology supported by AI

CHALLENGE! To build a model for the automatic detection of small objects at sea and estimation of the height and direction of waves propagation, i.e. the sea state estimation.

HOW? Computer Vision utilization

WHY? To provide the necessary information in building the deep learning autonomous shipping and safe navigation model.

FINAL RESULT→ CV models for small objects detection and sea states description.

GOALS FOR INNO2MARE PROJECT: To assist in creating the autonomous ship DSS.


Denis Selimović, Franko Hržić, Jasna Prpić-Oršić, Jonatan Lerga, Estimation of sea state parameters from ship motion responses using attention-based neural networks, Ocean Engineering, Volume 281, 2023.
https://doi.org/10.1016/j.oceaneng.2023.114915. (https://www.sciencedirect.com/science/article/pii/S0029801823012994)

Abstract: On-site estimation of sea state parameters is crucial for ship navigation. Extensive research has been conducted on model-based estimation utilizing ship motion responses. Model-free approaches based on machine learning (ML) have recently gained popularity, and estimation from time-series of ship motion responses using deep learning (DL) methods has given promising results. In this study, we apply the novel, attention-based neural network (AT-NN) for estimating wave height, zero-crossing period, and relative wave direction from raw time-series data of ship pitch, heave, and roll. Despite reduced input data, it has been demonstrated that the proposed approaches by modified state-of-the-art techniques (based on convolutional neural networks (CNN) for regression, multivariate long short-term memory CNN, and sliding puzzle neural network) improved estimation MSE, MAE, and NSE by up to 86%, 66%, and 56%, respectively, compared to the best performing original methods for all sea state parameters. Furthermore, the proposed technique based on AT-NN outperformed all tested methods (original and enhanced), improving estimation MSE by 94%, MAE by 74%, and NSE by 80% when considering all sea state parameters. Finally, we proposed a novel approach for interpreting the uncertainty estimation of neural network outputs based on the Monte-Carlo dropout method to enhance the model’s trustworthiness.

Keywords: Ship motions; Sea state estimation; Deep learning; Attention neural network; Uncertainty estimation

Goran Paulin, Sasa Sambolek, Marina Ivasic-Kos, Application of raycast method for person geolocalization and distance determination using UAV images in Real-World land search and rescue scenarios, Expert Systems with Applications, Volume 237, Part A, 1 March 2024, 121495.

https://doi.org/10.1016/j.eswa.2023.121495. (https://www.sciencedirect.com/science/article/pii/S0957417423019978?via%3Dihub)

Abstract: People enjoy spending time in the wilderness for numerous reasons. However, they occasionally get lost or injured, and their survival depends on being efficiently found and rescued in the shortest possible time. A search and rescue operation (SAR) is launched after the accident is reported, and all possible resources are activated. The inclusion of drones in SAR operations has enabled the use of computer vision methods to detect persons in aerial imagery automatically. When searching by drone, preference is given to oblique photographs that cover a larger area within a single image, reducing the search time. Unlike vertical photographs, oblique photographs include a significant scale change, making it challenging to locate a person in the real world and determine their distance from the drone. In order to solve this problem, encouraged by our previous successful simulations, we explored the possibility of applying the raycast method for person geolocalization and distance determination for use in real-world scenarios. In this paper, we propose a system able to precisely geolocate persons automatically detected in offline processed images recorded during the SAR mission. After a series of experiments on terrains of different configurations and complexity, using a custom-made 3D terrain generator and raycaster, along with a deep neural network-based person detector trained on our custom dataset, we defined a method for geolocating detected person based on raycast, which allows using low-cost commercial drones with a monocular camera and no Real-Time Kinematic module while enabling laser rangefinder emulation during offline image analysis. Our person geolocating method overcomes the problems faced by previous methods and, using a single flight sequence with only 4 consecutive detections, significantly outperforms the previous best results, with reliability of 42,85% (geolocating error of 0.7 m on recording from a 30 m height). Also, a short time of only 247 s enables offline processing of data recorded during a 21-minute drone flight covering approximately an area of 10 ha, proving that the proposed method can be effectively used in actual SAR missions. We also proposed a new evaluation metric (ErrDist) for person geolocalization and provided recommendations for using the proposed system for person detection and geolocation in real-world scenarios.

Keywords: Raycasting; Drone imagery; Object detection; YOLOv4; Object geolocalization; Distance determination; Search and rescue missions

Upcoming publication

Skip to content