Hierarchical Visual SLAM based on Fiducial Markers

Scientific Conference (2023, June 02)

Fiducial markers can encode rich information about the environment and aid Visual SLAM (VSLAM) approaches in reconstructing maps with practical semantic information. Current marker-based VSLAM approaches ... [more ▼]

Fiducial markers can encode rich information about the environment and aid Visual SLAM (VSLAM) approaches in reconstructing maps with practical semantic information. Current marker-based VSLAM approaches mainly utilize markers for improving feature detection in low-feature environments and/or incorporating loop closure constraints, generating only low-level geometric maps of the environment prone to inaccuracies in complex environments. To bridge this gap, this paper presents a VSLAM approach utilizing a monocular camera and fiducial markers to generate hierarchical representations of the environment while improving the camera pose estimate. The proposed approach detects semantic entities from the surroundings, including walls, corridors, and rooms encoded within markers, and appropriately adds topological constraints among them. Experimental results on a real-world dataset demonstrate that the proposed approach outperforms a traditional marker-based VSLAM baseline in terms of accuracy, despite adding new constraints while creating enhanced map representations. Furthermore, it shows satisfactory results when comparing the reconstructed map quality to the one reconstructed using a LiDAR SLAM approach. [less ▲]

From SLAM to Situational Awareness: Challenges and Survey

in Sensors (2023), 23(10), 4849

The capability of a mobile robot to efficiently and safely perform complex missions is limited by its knowledge of the environment, namely the situation. Advanced reasoning, decision-making, and execution ... [more ▼]

The capability of a mobile robot to efficiently and safely perform complex missions is limited by its knowledge of the environment, namely the situation. Advanced reasoning, decision-making, and execution skills enable an intelligent agent to act autonomously in unknown environments. Situational Awareness (SA) is a fundamental capability of humans that has been deeply studied in various fields, such as psychology, military, aerospace, and education. Nevertheless, it has yet to be considered in robotics, which has focused on single compartmentalized concepts such as sensing, spatial perception, sensor fusion, state estimation, and Simultaneous Localization and Mapping (SLAM). Hence, the present research aims to connect the broad multidisciplinary existing knowledge to pave the way for a complete SA system for mobile robotics that we deem paramount for autonomy. To this aim, we define the principal components to structure a robotic SA and their area of competence. Accordingly, this paper investigates each aspect of SA, surveying the state-of-the-art robotics algorithms that cover them, and discusses their current limitations. Remarkably, essential aspects of SA are still immature since the current algorithmic development restricts their performance to only specific environments. Nevertheless, Artificial Intelligence (AI), particularly Deep Learning (DL), has brought new methods to bridge the gap that maintains these fields apart from the deployment to real-world scenarios. Furthermore, an opportunity has been discovered to interconnect the vastly fragmented space of robotic comprehension algorithms through the mechanism of Situational Graph (S-Graph), a generalization of the well-known scene graph. Therefore, we finally shape our vision for the future of robotic situational awareness by discussing interesting recent research directions. [less ▲]

S-Graphs+: Real-time Localization and Mapping leveraging Hierarchical Representations

E-print/Working paper (2022)

In this paper, we present an evolved version of the Situational Graphs, which jointly models in a single optimizable factor graph, a SLAM graph, as a set of robot keyframes, containing its associated ... [more ▼]

In this paper, we present an evolved version of the Situational Graphs, which jointly models in a single optimizable factor graph, a SLAM graph, as a set of robot keyframes, containing its associated measurements and robot poses, and a 3D scene graph, as a high-level representation of the environment that encodes its different geometric elements with semantic attributes and the relational information between those elements. Our proposed S-Graphs+ is a novel four-layered factor graph that includes: (1) a keyframes layer with robot pose estimates, (2) a walls layer representing wall surfaces, (3) a rooms layer encompassing sets of wall planes, and (4) a floors layer gathering the rooms within a given floor level. The above graph is optimized in real-time to obtain a robust and accurate estimate of the robot's pose and its map, simultaneously constructing and leveraging the high-level information of the environment. To extract such high-level information, we present novel room and floor segmentation algorithms utilizing the mapped wall planes and free-space clusters. We tested S-Graphs+ on multiple datasets including, simulations of distinct indoor environments, on real datasets captured over several construction sites and office environments, and on a real public dataset of indoor office environments. S-Graphs+ outperforms relevant baselines in the majority of the datasets while extending the robot situational awareness by a four-layered scene model. Moreover, we make the algorithm available as a docker file. [less ▲]

Unclonable human-invisible machine vision markers leveraging the omnidirectional chiral Bragg diffraction of cholesteric spherical reflectors

in Light: Science and Applications (2022), 11(309), 10103841377-022-01002-4

The seemingly simple step of molding a cholesteric liquid crystal into spherical shape, yielding a Cholesteric Spherical Reflector (CSR), has profound optical consequences that open a range of ... [more ▼]

The seemingly simple step of molding a cholesteric liquid crystal into spherical shape, yielding a Cholesteric Spherical Reflector (CSR), has profound optical consequences that open a range of opportunities for potentially transformative technologies. The chiral Bragg diffraction resulting from the helical self-assembly of cholesterics becomes omnidirectional in CSRs. This turns them into selective retroreflectors that are exceptionally easy to distinguish— regardless of background—by simple and low-cost machine vision, while at the same time they can be made largely imperceptible to human vision. This allows them to be distributed in human-populated environments, laid out in the form of QR-code-like markers that help robots and Augmented Reality (AR) devices to operate reliably, and to identify items in their surroundings. At the scale of individual CSRs, unpredictable features within each marker turn them into Physical Unclonable Functions (PUFs), of great value for secure authentication. Via the machines reading them, CSR markers can thus act as trustworthy yet unobtrusive links between the physical world (buildings, vehicles, packaging,...) and its digital twin computer representation. This opens opportunities to address pressing challenges in logistics and supply chain management, recycling and the circular economy, sustainable construction of the built environment, and many other fields of individual, societal and commercial importance. [less ▲]

Advanced Situational Graphs for Robot Navigation in Structured Indoor Environments

E-print/Working paper (2022)

Mobile robots extract information from its environment to understand their current situation to enable intelligent decision making and autonomous task execution. In our previous work, we introduced the ... [more ▼]

Mobile robots extract information from its environment to understand their current situation to enable intelligent decision making and autonomous task execution. In our previous work, we introduced the concept of Situation Graphs (S-Graphs) which combines in a single optimizable graph, the robot keyframes and the representation of the environment with geometric, semantic and topological abstractions. Although S-Graphs were built and optimized in real-time and demonstrated state-of-the-art results, they are limited to specific structured environments with specific hand-tuned dimensions of rooms and corridors. In this work, we present an advanced version of the Situational Graphs (S-Graphs+), consisting of the five layered optimizable graph that includes (1) metric layer along with the graph of free-space clusters (2) keyframe layer where the robot poses are registered (3) metric-semantic layer consisting of the extracted planar walls (4) novel rooms layer constraining the extracted planar walls (5) novel floors layer encompassing the rooms within a given floor level. S-Graphs+ demonstrates improved performance over S-Graphs efficiently extracting the room information while simultaneously improving the pose estimate of the robot, thus extending the robots situational awareness in the form of a five layered environmental model. [less ▲]

Situational Graphs for Robot Navigation in Structured Indoor Environments

E-print/Working paper (2022)

Semantic situation awareness of ellipse shapes via deep learning for multirotor aerial robots with a 2D LIDAR

in 2020 International Conference on Unmanned Aircraft Systems (ICUAS) (2020, September)

In this work, we present a semantic situation awareness system for multirotor aerial robots equipped with a 2D LIDAR sensor, focusing on the understanding of the environment, provided to have a drift-free ... [more ▼]

In this work, we present a semantic situation awareness system for multirotor aerial robots equipped with a 2D LIDAR sensor, focusing on the understanding of the environment, provided to have a drift-free precise localization of the robot (e.g. given by GNSS/INS or motion capture system). Our algorithm generates in real-time a semantic map of the objects of the environment as a list of ellipses represented by their radii, and their pose and velocity, both in world coordinates. Two different Convolutional Neural Network (CNN) architectures are proposed and trained using an artificially generated dataset and a custom loss function, to detect ellipses in a segmented (i.e. with one single object) LIDAR measurement. In cascade, a specifically designed indirect-EKF estimates the ellipses based semantic map in world coordinates, as well as their velocity. We have quantitative and qualitatively evaluated the performance of our proposed situation awareness system. Two sets of Software-In-The-Loop simulations using CoppeliaSim with one and multiple static and moving cylindrical objects are used to evaluate the accuracy and performance of our algorithm. In addition, we have demonstrated the robustness of our proposed algorithm when handling real environments thanks to real laboratory experiments with non-cylindrical static (i.e. a barrel) objects and moving persons. [less ▲]

Trajectory Tracking for Aerial Robots: an Optimization-Based Planning and Control Approach

in Journal of Intelligent and Robotic Systems (2020), 100

In this work, we present an optimization-based trajectory tracking solution for multirotor aerial robots given a geometrically feasible path. A trajectory planner generates a minimum-time kinematically ... [more ▼]

In this work, we present an optimization-based trajectory tracking solution for multirotor aerial robots given a geometrically feasible path. A trajectory planner generates a minimum-time kinematically and dynamically feasible trajectory that includes not only standard restrictions such as continuity and limits on the trajectory, constraints in the waypoints, and maximum distance between the planned trajectory and the given path, but also restrictions in the actuators of the aerial robot based on its dynamic model, guaranteeing that the planned trajectory is achievable. Our novel compact multi-phase trajectory definition, as a set of two different kinds of polynomials, provides a higher semantic encoding of the trajectory, which allows calculating an optimal solution but following a predefined simple profile. A Model Predictive Controller ensures that the planned trajectory is tracked by the aerial robot with the smallest deviation. Its novel formulation takes as inputs all the magnitudes of the planned trajectory (i.e. position and heading, velocity, and acceleration) to generate the control commands, demonstrating through in-lab real flights an improvement of the tracking performance when compared with a controller that only uses the planned position and heading. To support our optimization-based solution, we discuss the most commonly used representations of orientations, as well as both the difference as well as the scalar error between two rotations, in both tridimensional and bidimensional spaces $SO(3)$ and $SO(2)$. We demonstrate that quaternions and error-quaternions have some advantages when compared to other formulations. [less ▲]

Vision-Based Aircraft Pose Estimation for UAVs Autonomous Inspection without Fiducial Markers

in IECON 2019-45th Annual Conference of the IEEE Industrial Electronics Society (2019, October)

The reliability of aircraft inspection is of paramountimportance to safety of flights. Continuing airworthiness of air-craft structures is largely based upon the visual detection of smalldefects made by ... [more ▼]

The reliability of aircraft inspection is of paramountimportance to safety of flights. Continuing airworthiness of air-craft structures is largely based upon the visual detection of smalldefects made by trained inspection personnel with expensive,critical and time consuming tasks. At this aim, Unmanned AerialVehicles (UAVs) can be used for autonomous inspections, aslong as it is possible to localize the target while flying aroundit and correct the position. This work proposes a solution todetect the airplane pose with regards to the UAVs position whileflying autonomously around the airframe at close range forvisual inspection tasks. The system works by processing imagescoming from an RGB camera mounted on board, comparingincoming frames with a database of natural landmarks whoseposition on the airframe surface is known. The solution has beentested in real UAV flight scenarios, showing its effectiveness inlocalizing the pose with high precision. The advantages of theproposed methods are of industrial interest since we remove manyconstraint that are present in the state of the art solutions. [less ▲]

Deep learning based semantic situation awareness system for multirotor aerial robots using LIDAR

in 2019 International Conference on Unmanned Aircraft Systems (ICUAS) (2019, June)

In this work, we present a semantic situation awareness system for multirotor aerial robots, based on 2D LIDAR measurements, targeting the understanding of the environment and assuming to have a precise ... [more ▼]

In this work, we present a semantic situation awareness system for multirotor aerial robots, based on 2D LIDAR measurements, targeting the understanding of the environment and assuming to have a precise robot localization as an input of our algorithm. Our proposed situation awareness system calculates a semantic map of the objects of the environment as a list of circles represented by their radius, and the position and the velocity of their center in world coordinates. Our proposed algorithm includes three main parts. First, the LIDAR measurements are preprocessed and an object segmentation clusters the candidate objects present in the environment. Secondly, a Convolutional Neural Network (CNN) that has been designed and trained using an artificially generated dataset, computes the radius and the position of the center of individual circles in sensor coordinates. Finally, an indirect-EKF provides the estimate of the semantic map in world coordinates, including the velocity of the center of the circles in world coordinates.We have quantitative and qualitative evaluated the performance of our proposed situation awareness system by means of Software-In-The-Loop simulations using VRep with one and multiple static and moving cylindrical objects in the scene, obtaining results that support our proposed algorithm. In addition, we have demonstrated that our proposed algorithm is capable of handling real environments thanks to real laboratory experiments with non-cylindrical static (i.e. a barrel) and moving (i.e. a person) objects. [less ▲]