Publication Date
2025
Document Type
Dissertation/Thesis
First Advisor
Butail, Sachit
Degree Name
Ph.D. (Doctor of Philosophy)
Legacy Department
Department of Mechanical Engineering
Abstract
This research presents experimentally validated strategies for human-swarm robotic interaction through nonverbal communication. Applications of this work are in large-scale coverage problems that include search and rescue operations and environmental monitoring missions. The non-verbal interaction becomes critical in situations where wireless communication methods may be limited by bandwidth constraints, privacy concerns, or environmental clutter. The research objective of this dissertation is to design and evaluate methods of nonverbal communication between a human teleoperator and robotic swarms towards a bidirectional human-swarm interaction framework. Towards this, we first establish a reliable measure of real-time cognitive load that can serve to close the loop with a robotic swarm during complex tasks. In particular, we performed visual identification experiments to demonstrate that de-synchronization within alpha band frequencies of an electroencephalogram (EEG) signal (9.5-11.5 Hz) provides a robust correlate of mental workload, particularly during object classification. Next, we focus on human-to-swarm communication with the goal that an autonomous swarm be able to infer and respond to a human operator’s cognitive states from movement patterns. In search experiments conducted on a large scale virtual replica of a national park, we show that the teleoperated vehicle’s speed, turn rate, and “freezing” behavior encode information about the operator’s prior knowledge and situational awareness. These results motivated the design of a long-short-term memory (LSTM) network that achieves nearly 70% accuracy in classifying operator knowledge states based on movement features only. When integrated into adaptive search strategies, inference of such states enable human-swarm teams to locate missing persons up to 150 seconds faster than random search and 50 seconds faster than spiral search during a mission that took approximately 300 seconds, all while maintaining a high success rate. For swarm-to-human nonverbal interaction, we draw from literature to identify collective motion patterns that have been shown to elicit strong emotional cues in humans. We selected and designed distinct swarm behaviors to communicate complex environmental information to human teleoperators. In monitoring experiments inspired by the problem of sampling aquatic invasive fish species, participants interpret movement-based nonverbal cues to identify fish type and number and obstacle size on their path. Our results show that the performance with information transfer via nonverbal cues is comparable to text-based communication while registering a reduced cognitive load under certain conditions. Results from this work are expected to set the stage for new ways to integrate human and swarm intelligence for complex missions.
Recommended Citation
Bhattacharya, Arunim, "Human Swarm Interaction Using Non-Verbal Communication" (2025). Graduate Research Theses & Dissertations. 8105.
https://huskiecommons.lib.niu.edu/allgraduate-thesesdissertations/8105
Extent
165 pages
Language
en
Publisher
Northern Illinois University
Rights Statement
In Copyright
Rights Statement 2
NIU theses are protected by copyright. They may be viewed from Huskie Commons for any purpose, but reproduction or distribution in any format is prohibited without the written permission of the authors.
Media Type
Text
