Grounding

Views
From Spatial Language
Jump to: navigation, search

We developed a system that understands natural language commands involving movement and manipulation by defining a probabilistic graphical model that factors according to the compositional and hierarchical structure of the language. We collected a corpus of commands on Amazon Mechanical Turk by asking users to write a command they would give to an expert human operator to command them to carry out the action in the video. Commands for a corpus for a single video are shown right. Our system successfully follows many commands from the corpus fully automatically.


Commands from the corpus collected on Amazon Mechanical Turk for the same video
Place the lifted tyre pallet, next to another tyre pallet on the trolley.
Lift the tire pallet in the air, then proceed to deposit it to the right of the tire pallet already on the table right in front of you.
Place the pallet of tires on the left side of the trailer.
Please lift the set of six tires up and set them on the trailer, to the right of the set of tires already on it.
Place a second pallet of tires on the trailer.
lift the tire pallet you are carrying and set on the truck in front of you
Place the pallet of tires that is on the forklift next to the pallet of tires that is already loaded on the trailer.
Lift tire pallet. Move to unoccupied location on truck. Lower tire pallet. Reverse to starting location. Lower forks. End.
lift the tire pallet to the truck
Arrange tire pallet to the truck.
Place the pallet of tires on the right side of the truck.
Lift the tire pallet and proceed forward to set it on the platform directly ahead, to the right of the tire pallet already there.
Put the tire pallet on the trailer.
Load the skid right next to the other skid of tires on the trailer.
Put the tire pallet on the trailer to the right of the other tire pallet.
Lift pallet up and place the pallet beside the other pallet on the truck bed. Reverse the forklift slowly from the truck bed.
Raise tire pallet. Move forward to unoccupied location on truck. Lower tire pallet. Reverse to starting position. Lower forks.
Move the pallet on the ground to the platform; place it to the right of the pallet that is already on the platform.
Personal tools