Projects
RAAS Lab - Language Models
On-going
Built low cost arm, and mounted on a TurtleBot (Kobuki). Project involves integration
with Language models for user interaction, enabling it to navigate to
locations, identify, and deliver objects (π0), while interacting with humans and memorizing everything.
The left video displays the TurtleBot Kobuki robot with a pan-tilt camera joint.
The middle video shows the custom manipulator picking up an object for deivery.
The right video shows the (previous version) of manipulation
integrated onto the mobilerobot. When given a goal object
name, the TurtleBot navigates towards it using visual
servoing.
Multi-Modal Foundation Models
Presentation
•
Report
•
Video
This is a chatbot that can accurately answer questions from documents containing both text and images.
It uses FAISS for vector-based text indexing, ColPali for image retrival, and Qwen-VL for visual understanding, with Mistral LLM for response generation.
The dual-stream approach improves document understanding by extracting relevant information from both images and text
RAAS Lab - Imitation Learning, Reinforcement Learning
Code
Developed the
Gello arms for
UR5e, UR3e robots for teleoperation, and data collection for imitation learning,
and built a custom RL environment that reads observations (proprioception, camera images) and generate actions, enabling RL training and IL testing.
The first row shows the design of Gello arms for UR3e and UR5e robots
used for teleoperation. The second row demonstrates the teleoperation
of UR5e in mujoco simulation environment, real-world teleop of UR3e, UR5e robots respectively.
RAAS Lab - Imitation Learning
Code
A custom robot design replicating the UR3e configuration,
inspired by GELLO. Developed a passive
arm for teleop and data collection, implemented
joint space and Cartesian space controllers, and
created an RL environment for testing behavior cloning models.
The first video demonstrates teleoperation of
the custom arm in a simulation environment.
The second video shows real-world teleoperation,
while the third video presents behavior cloning (BC)
model testing on the custom manipulator, using camera images
as observations to generate actions.
Perception for Autonomous Robots
Presentation
•
Simulation Video
A perception-based challenge for both simulation and hardware, designed to complete four tasks:
detecting a STOP sign and stopping, Dynamic obstacle avoidance, Horizon line detection using projective geometry,
and Lane-based navigation along white paper.
The left image shows the robot in real-world environment setup, while
the GIF showcases its operation in the simulation world using Gazebo and ROS2 humble.
Planning for Autonomous Robots
Presentation
•
Report •
Video
A new method based on the RRT* algorithm for path planning on the same map.
RU-RRT* stores trees sampled from previous queries and reuses by importing subtrees to
speed up current query planning, allowing more exploration within less computation time while maintaining optimal path quality.
Planning for Autonomous Robots
Code •
Video
A competition to use any planning algorithm, (A-star), to find the optimal path and deploying it on the TurtleBot to navigate the maze in shortest time.
Robot Learning
Presentation •
Report
Developed a reinforcement learning environment for custom humanoid robot, and trained it
using PPO (Proximal Policy Optimization) and SAC (Soft Actor-Critic) algorithms,
comparing their performance in OpenAI Gymnasium and Google Brax environments.
Robot Modeling
Report •
Video
A transformable robot design that can operate as a
mobile robot, drone, Underwater Vehicle (UV), and
has an integrated 4-DOF manipulator, designed in SolidWorks.
The design is simulated in three environments in Gazebo-classic using ROS2 Galactic.
The first row shows the integrated manipulator design to perform object manipulation.
The second row shows the how design and can tranform into 3 different modes
The third row shows the robot in Gazebo environment
Robot Programming
Video •
Code
This project equips a TurtleBot with a camera to
navigate a maze using ArUco markers on the walls.
While navigating, it detects objects, records
their color and position, and displays the collected data upon stopping.
The first row shows the integrated manipulator design to perform object manipulation.
The second row shows the how design and can tranform into 3 different modes
The third row shows the robot in Gazebo environment
Project
Presentation •
Video
A system designed to assist in borewell rescue
operations, addressing accidents where a child
fall into uncovered borewells, a common issue in
some countries. It features a dexterous robotic
hand, teleoperated by a human for precise rescue operations.
The image showcases the concept and design of the RescueBOT and some snapshots while fabricating the design,
and the video shows the simple prototype of the concept.
MobileRobot Maze Navigation based on Color
Robot Modeling and Programming
Code •
Video
This project involves designing a mobile
robot in Fusion 360, exported and simulated
in Gazebo with ROS2 Galactic. Designed a
maze traversal environment and implemented
vision-based navigation using color blocks.
The first row shows the robot design (replica of Husky), in Fusion 360.
The robot is simulated with a LIDAR and camera sensor as shown and the video shows the navigation
Software Developement & Research
Presentation •
Report •
Video
Developed a software that takes a sheet metal product design,
especially free-form surfaces, flattens it, and backtracks to
determine the optimized the manufacturing steps required for production.
*: Zahir Mahammad
Mini-Projects
RRC, IIIT Hyderabad - Transmission Line Inspection
Designed and integrated the F550 hexacopter
model into the PX4 SITL package for simulation
in Gazebo with an integrated camera. Developed
a custom world for transmission line inspection
and implemented waypoint navigation using QGroundControl
The left image displays the F550 drone design,
while the video demonstrates its integration with
the PX4 SITL package and the implementation of
waypoint navigation in the Gazebo environment.
Dijkstra Algorithm - Pygame
Path Planning for Robots
Code •
Video
Object Tracking - OpenCV
Perception
Code •
Video