Railway experts at the core of the learning process - A case study on frog point detection

Context

The maintenance of large infrastructures poses the challenge of monitoring its components in a scalable way. The aim is to drastically increase the responsiveness of maintenance operations in the long term, with the possibility of integrating them coherently and as early as possible into an action schedule.

The use of inspection wagons allows and facilitates the acquisition of a large amount of heterogeneous data that is composed of sensors’ measurements and corresponding contextual metadata. Structuring this large mass of data by highlighting relevant information is a key point to cleverly schedule maintenance acts. In the case of the image data acquired by a wagon, it is crucial to position the different components of the infrastructure and assess their state.

At RAILwAI, our ambition is to propose predictive analysis solutions based on the most recent progress in Artificial Intelligence (AI) to index all these heterogeneous data and perform a wide range of downstream tasks on top of it. We rely on state-of-the-art generalist image segmentation models to extract the relative position of different identified components of interest (poles, rails, etc.).

We are working in a hybrid localisation system that may relies on different measurements sources to provide robust localisation estimations on different conditions. The system we describe in this article works in a general way and can be applied to a wide range of components in metro, tramway, and railway infrastructures such as poles, switches, catenary, etc. This hybrid system allows to have very accurate localisation even in case of missing or noisy conditions for odometer and GPS. It also keeps track of any component in the infrastructure and monitor its state from image analysis to have updated patrimonial data of the maintainer.

Figure 1. An illustration of a generic object segmentation model on images grabbed by an inspection wagon. Left panel: original image; right panel: segmented image.

The innovative part of our solution relies on the fact that it is an open and adaptive system. It may evolve and learn through time to get better to adapt to annotations provided by the users. It is possible for the users to annotate several specific components from the video stream coming from the camera. In this article, we demonstrate on an example

i) how to adapt the system to focus on frog point localization, and

ii) how to use this to consolidate train trajectory when dealing with noisy GPS measurements.

Use case: Frog point detection

In optimal conditions, GPS provides quite accurate localisation, especially in regions densely covered by satellites. The system uses those data to calibrate position of the inspection wagon with respect to reference element in the infrastructure by validating and updating geographical information.

In situations where GPS signal is weak or missing, like in tunnels or dense urban zones, the system may rely on odometer for localisation. However, some limitation may arise preventing accurate detection when the wagon switches track. In those situations, video recognition becomes a powerful tool to disambiguate the measures and get a robust estimate of the wagon localisation.

Embedded cameras continuously grab several pictures with different view on the wagon. Then, it is possible to use AI-based models to detect on which track the wagon is switching when nearby well identified and localised component of the rail infrastructure. In this article, we consider switch’s frog points detection.

As show in Figure 2, we tackle the problem of detecting frog points from images through a model pipeline composed of two main steps. The first model employs a state-of-the-art pre-trained segmentation model. Its main goal is to detect the rail track regions in front of the heading camera of the train and ignore any other detected tracks as shown in Figure 3.

Figure 2. Model Pipeline

The segmentation model can detect the masked track lines as an output to the next step. The second model of the pipeline is a supervised classification model that categorizes whether the output segmentation mask corresponds to a frog point or not.

Figure 4 demonstrates the full output of the pipeline. Based on the relative position of the train and the detected frog point, it is straightforward to monitor which track the wagon moves to.

Figure 3. An example of the detected rail track regions

Figure 4. An example of the full pipeline output

Integrating experts in the loop

Machine learning models could benefit from active collaboration and integrating Human-In-The-Loop (HITL) into the learning dynamics. Not only to ensure continuous improvement and excellence in predictive outcomes, but to create a convenient coupling between model precision and human cognitive capabilities as well.

Involving human expert knowledge in the learning process of intelligent systems could be presented in various stages. For instance, the most common strategy is building a Human Feedback Loop where expert feedback is collected after model development to create an iterative process that helps fine-tuning these models based on real-world usage and evolving requirements.

Another example of a more advanced paradigm is Active Learning (AL) where the predictive models are interactively engaged with experts during the learning process. Figure 5 presents one of the most common AL scenarios (pool-based) where a large pool of unlabelled samples is available for model evolution. The goal is to select the most informative and valuable data points for labelling which in turn reduces the amount of label data required for high predictive performance.

Figure 5. An Active Learning Pipeline

At RAILwAI, we pay close attention to integrating human expertise to enhance the overall performance, robustness, reliability, explainability, and security of our machine-learning models.

This is in line with the regulations settled by the Public Railway Safety Establishment - l’Établissement Public de Sécurité Ferroviaire (EPSF)- that defines the conditions under which machine learning and AI models could be authorized for deployment in railway applications. Particularly, these regulations are notably concerned with the role of feedback in maintaining and improving the level of safety of the rail system to enhance interpretability and explainability of AI models which are mostly deemed as black boxes. However, this raises the question of reproducibility of such models and tracking their evolution.

If you want to know more and get deeper insights into our cutting-edge solutions and services, reach out to us at RAILwAI to explore a world of possibilities to put your data on the smart track.