From Language to Logic: VLM-Grounded Safe Navigation in Novel Environments

ABSTRACT

Autonomous robots operating in unstructured outdoor environments must satisfy strict safety constraints while responding to high-level human rules, often using model checking software to provide run-time assurance. Existing model checking tools can verify system behavior over time using temporal logic, but they rarely reason over which parts of a novel environment are safe to enter or avoid, nor do they connect human language instructions to that visual understanding. Vision-language models (VLMs) are AI systems that jointly process visual input and natural language, and provide rich semantic information about the environment. Bridging this disconnect between visual perception, natural language, and formal verification motivates our approach.

We propose a navigation architecture in which human-provided rules are compiled into Signal Temporal Logic (STL) specifications and enforced at two stages. First, persistent environment-centric rules (e.g., obstacle clearance, terrain preferences) are grounded into a 2D cost map. Second, temporally dynamic rules (e.g., speed limits, time-bounded exposure to hazardous regions) are enforced by monitoring STL satisfaction along candidate trajectories during planning. This architecture supports reasoning about novel environments while enabling runtime assurance of safe robot behavior.

In this architecture, a Vision-Language Model (VLM) identifies key features in arbitrary environments based on human-generated natural language rules. Its semantic outputs encode both hard spatial constraints in the cost map and environmental properties that impact the robot’s state during navigation. The planner then uses this information to track the robot’s state and ensure temporal requirements and prohibitions are maintained as the robot progresses toward pre-defined goals.

We demonstrate the approach in a simulated outdoor scenario with two operational modes: a ”normal” mode with initial specifications and a ”degraded” low-battery mode that tightens specifications on speed limits, obstacle clearance, and time-bounded robot behavior. In this scenario, the robot experiences a mid-episode switch from the normal mode to the degraded mode, after which its behavior is governed by the stricter specifications. This switch triggers an online recomputation of the cost map and replanning of a new rule-compliant path using the updated version of the STL rules.

This integration of VLM-based perception with STL-based specification and runtime enforcement shows how AI can serve as an enabler for high confidence autonomy, fitting the ”AI as an Enabler: Software Development and Systems Engineering” theme by providing a workflow from informal human rules to rigorous, machine-checkable specifications and behavior for VLM-integrated autonomous robotic systems.

BIOS

Kristy Sakano received the B.S. degree in physics from the University of North Carolina, Chapel Hill in 2018, and the M.S. degree in computer science from the Georgia Institute of Technology in 2023. She is currently pursuing the Ph.D. degree in computer science at the University of Maryland, College Park.

From 2018 to 2024, she was an Operations Research Analyst with the Naval Air Warfare Center Aircraft Division. Her research interests include formal methods for autonomous systems, and safety certification of learning-based controllers.

Kalonji Harrington received the B.S. degree in computer science at the University of Maryland, College Park, MD, USA in 2024. He is pursuing the M.S. degree in computer science at the University of Maryland.

Since summer 2025, he has worked as a Computer Vision Intern at Precision Sustainable Agriculture, where he engages in research on vision systems for crop biomass estimation. His research interests include visual navigation for robotic systems and scene understanding in challenging environments.