Task 2. Serving Food

Task 2: Serving food

The aim of this task is to 1) pour water to a designated level, 2) carry & balance a tray having a bowl and glass filled with water, 3) move to the second table on the side where there are no utensils, 4) and place the bowl and glass without touching the utensils.

It is critical to understand the scene and to provide proper commands to the pilot to complete the subtasks in task 2. To understand the scene, vision-based deep learning models were utilized and vibration signals were implmented to provide feedback to the pilot.

Pouring water

We utilized a scale to measure the desired amound of water poured into the cup and when there are enough water poured, the vibration motors attached to the pilot’s stopped vibratiing.

Deciding the target location

The objective of this subtask is to move to the target location where there are no utensils on the table. First, to determine the correct direction (right or left), YOLOv8 is utilized to extract the bounding box of the dining table. Then, I compared the center $x$ position of the dining table and the $x$ position of the averaged red pixels and decide the desired direction as the logic below:



1
2
3
4
5
6
7

 
if (tableX > redX){
  // Utensil is on left of the table
  // Move to right 
} else {
  // Utensil is on right of the table
  // Move to left
}

When the desired direction is determined, the red object detection function (Python) publishes an LCM message to the navigation code (C++). The following video indicates the desired direction on the top left corner in red text while the bounding box of the dining table is in blue and average position of the red objects are in green.

Balancing the tray

Task 1: Sidewalk. The aim is to reach the end line without bumping into any obstacles.

Placing bowl and glass

I finetuned YOLOv8 on our dataset collected in various viewpoints, light settings, and backgrounds. Based on the bounding boxes obtained during inference, the desired direction of placing the bowl was determined. This direction was then conveyed to the pilot via haptic feedback via vibration motors.

Thanks

Thankfully, the recruitment process was relatively smooth. We have two incredible pilots (with great atheletic abilities) and one guide dog trainer. We also have great faculty members joining. Finally, we have wonderful members for developing hardware, software, and conducting experiments. Always thankful for everyone. Especially, thankful for my advisor for the support.