Multi-Modal Autonomous Drone System for Real-Time Highway Incident Detection and Analysis

Highway accidents remain a critical public safety challenge, where delayed response times intensify injury severity and fatalities. Traditional systems relying on manual reporting suffer from inefficiencies and critical delays during emergencies. This paper presents an autonomous drone based highway incident management system that integrates advanced aerial technology with Vision-Language Models (VLMs) and Large Language Models (LLMs) to enable rapid detection, analysis, and response. The framework employs a DJI Matrice 30T drone equipped with a dock station for autonomous deployment for incident analysis by yielding incident coordinates provided by authorities. Upon arrival, real-time video is analyzed using an optimized fine-tuned YOLOv12n model selected after rigorous comparison of nine YOLO variants to detect incidents such as collisions and fires with high precision. To generate actionable intelligence, high-confidence video frames are processed using a hybrid vision-language pipeline comprising of LLaVA-OneVision-Qwen2 and GPT-4o API. This combination yields detailed natural language descriptions and structured summaries of incident scenes, offering enhanced contextual awareness of the incident site. By combining autonomous drone navigation, real-time object detection, VLM-based scene interpretation, and LLM-driven summarization, this end-to-end solution enhances situational awareness and decision-making for emergency responders. These results demonstrate the practical value of integrating multi-modal AI with UAVs in critical safety domains, laying the groundwork for scalable, intelligent transportation infrastructure and next-generation emergency response systems.