Robot Ratatouille: Culinary Robot Project

Hardware

UR7e Arm + Gripper

6-DoF Universal Robots UR7e with a parallel gripper. The arm is driven through the scaled_joint_trajectory_controller's FollowJointTrajectory action.

• Gripper toggle: /toggle_gripper Triggers the gripper to open and close. It's assumed to be open at the startup.
• Joint feedback: /joint_states /joint_states is the topic that contains the current joint states in radians. It's used in the IK solver, armcircler, and the pick and place.

Hardware

Intel RealSense D-series

Wrist-mounted depth camera providing aligned RGB, depth, and intrinsics over the standard RealSense ROS 2 topics.

• /camera/.../color/image_raw — RGB stream used by YOLO to determine bounding boxes and centroids.
• /camera/.../depth/color/points — This is the pointcloud used detection_node to find the centroid and depths of objects detected with YOLO.
• /camera/.../color/camera_info — intrinsics used by pixel_to_world to calculate the position of objects based on where they appear on the image plane (camera).

Note — the camera launch is intentionally external to the bringup so we can restart it without tearing down the planner.

perception

Custom-Trained YOLO Model

A custom YOLO model was trained on images of 3D-printed food replicas for real-time object detection within the workspace. Done separately from the main design.

Note — food wasn't allowed in the lab; existing YOLO models trained on real food couldn't be utilized, requiring us to train one ourselves.

perception

DetectionNode

The main perception node. Runs YOLOv8 on every RGB frame, lifts each bbox into a 3D centroid through the depth pointcloud, and publishes the result in base_link.

1. YOLO inference. YOLODetector.detect() returns class IDs, confidence, bbox, and 2D center.
2. Depth lift. get_centroid_from_cloud_bbox() reprojects every cloud point into image coords, keeps the ones inside the bbox, and averages them.
3. TF lift. transform_point() resolves camera_link → base_link via the static TF + UR kinematic chain.
4. Plate gate. Any candidate within plate_radius = 0.14 m of the latest plate centroid is rejected.
5. Publish. Highest-confidence survivor on /detected_pick_point; plate always published on /detected_plate_point.

Topic ordering matters — /detected_class must arrive before /detected_pick_point, otherwise per-class offsets silently fall back to defaults.

perception

YOLODetector

Thin stateless wrapper around Ultralytics YOLOv8. detect(image) returns a list of {class_id, class_name, confidence, bbox, center} dicts.

• Weights: updated.pt (fine-tuned) is the default; best.pt and yolov8n.pt are kept as fallbacks.
• Threshold: conf_threshold parameter on DetectionNode (default 0.5).
• No state, no temporal smoothing — every frame is independent.

perception

pixel_to_world

Helper module that converts 2D detections into 3D centroids in base_link.

• get_centroid_from_cloud_bbox() — projects every depth point into camera image coords, averages the ones inside the YOLO bbox.
• transform_point() — TF lookup chained from camera_link through the static TF and the UR kinematic chain.
• Returns a PointStamped in base_link, ready for IK.

planning

ArmCircler — Orbit Inspection

The ArmCircler node is responsible for executing a sequence of joint trajectories using an action client and taking pictures at each waypoint. It uses the FollowJointTrajectory action type, which handles trajectory goals, feedback, and results.

1. Plate detection: Uses YOLO to find the plate and extracts the bounding box centerpoint as the reference point for trajectory planning.
2. Waypoint generation: Two nested for loops generate waypoints at two height levels—each tracing an arc around the plate—plus one point above facing down.
3. Pose storage: Stores the position and quaternion of each waypoint in poses.npy ([x,y,z,qx,qy,qz,qw]).
4. IK solving: Uses the IK planner from Lab 5 to compute joint angles for each waypoint and places them in the job queue.
5. Trajectory execution: Executes trajectories one-by-one, taking an RGB picture with the RealSense camera at each waypoint. Images stored in captured_images/.
6. Completion signal: Sends a True boolean value to the commander node to signify orbit completion.

• Look-at orientation: rot_z · rot_y · rot_x always points the camera at the plate.
• Heights: row 1 at 0.30 m, row 2 reversed at 0.20 m.
• Total waypoints: 2 rows × 20 waypoints + 1 overhead shot = 41 frames.

These frames are captured as a multi-view orbit around the plate. Drag the scrubber below to step through all 41 waypoints. The full set lives in captured_images/ for the downstream segmentation and triangulation pipeline.

Why orbit? Single-frame depth centroids degrade near the edge of the camera frame. Forty one views give the offline pipeline enough parallax for stable triangulation.

planning

Commander

1. The Commander node orchestrates the entire pipeline, stitching together all the individual components and running in parallel with main.py via the bringup file. It gives the user visibility into what stage the robot is currently in — whether it is orbiting, computing masks, or executing pick and place.

2. Commander also runs the filtering and planning logic before execution begins. It drops any triangulated position whose Z falls outside the range [-0.03, 0.15] m or whose confidence is at or below 0.6, then keeps only the single highest-confidence instance per class. From the surviving detections it computes the pick order and publishes the final task list to main.py.

3. Beyond orchestration, Commander was also a significant debugging aid throughout development. We could easily skip sections of the pipeline, such as the orbit phase, to jump directly to the parts we were actively testing. In early stages, before the automatic planning algorithm was complete, we could manually specify which object to pick, which let us tune the per-class grasp offsets independently of the rest of the system. Overall, Commander served as a quality-of-life node that made the project easier to iterate on and helped the final demo run smoothly.

planning

6D Poses — Offline Position Reconstruction

segment_batch.py — The commander node tells this node to run after the armcircler node is done. This node is responsible for detecting the foods in each image using YOLO and SAM2. It takes in the images folder and outputs a folder of images called "segmented" with the YOLO bounding boxes and SAM2 masks. → segmented/<stem>/<stem>_<class>_<i>_mask.png.

‹ ›

Background and Motivation

The original plan for localizing food items on the plate was to use COLMAP to reconstruct a 3D point cloud from the 41 images captured during the arm's orbital inspection pass. This was abandoned for two reasons:

• Too slow. COLMAP's feature matching and bundle adjustment pipeline is computationally expensive — waiting several minutes between the orbit phase and the pick phase was not acceptable at demo timescales.
• Too sparse. Food items are small, visually uniform objects on a reflective plate surface, which frustrates feature-based reconstruction. There are few distinctive keypoints, and the ones that exist don't triangulate reliably across views. The resulting point cloud had little to no geometry in the regions that actually mattered.

The solution was to abandon general-purpose reconstruction entirely and exploit the structure of the problem instead. Two facts made this possible:

• Because the robot arm is the image capture platform, every camera pose is already known precisely from forward kinematics — there is no need to solve the structure-from-motion problem at all.
• Because YOLO already identifies objects with bounding boxes and instance masks, 3D localization reduces to a much simpler sub-problem: given a set of known camera poses and 2D observations of an object, find its 3D position. This is classical multi-view triangulation, and it runs in milliseconds.

What the Pipeline Does

The offline pose estimation pipeline takes three inputs:

Poses

The 40 end-effector poses recorded during the orbit (poses.npy)

Images

The 40 corresponding images (captured_images/)

Masks

The per-image YOLO+SAM2 instance masks (segmented/)

It outputs results/object_poses.npy — a list of detected objects with their estimated 3D position in the robot's base_link frame and a confidence score.

The pipeline runs as a subprocess under the SAM2 virtual environment's Python interpreter rather than the ROS Python, because it depends on PyTorch, SAM2, trimesh, and pyrender — none of which are installed in the ROS workspace. This keeps the two dependency trees cleanly separated while still allowing the Commander node to trigger the pipeline and consume its output.

How Position-Only Mode Works

The pipeline runs in position_only=True mode, which skips the full 6DoF render-and-compare path and estimates only 3D position. It does this in three steps.

1. 01

   Step 1: Camera pose computation. For each waypoint, the known end-effector pose is converted into a camera-to-world transform via a chain of three operations:

   • End-effector pose (position + quaternion) → 4x4 homogeneous transform world_T_ee
   • Invert → ee_T_world
   • Compose with a fixed camera-mount offset T_CAM_FROM_EE (camera sits +3 cm forward and 13.5 cm above the wrist) → cam_T_world

   This transform chain is what makes the approach work: because the robot arm is the camera rig, known kinematics replace the bundle adjustment step entirely.

2. 02

   Step 2: Per-view 2D observation. For each image, the pipeline loads the SAM2 masks and processes each detected object:

   • Centroid: computed as the median pixel coordinate (median rather than mean, to be robust against edge clipping)
   • Apparent radius: estimated from mask area as r_px = sqrt(area / pi)
   • Depth: recovered from apparent size using z_m = fx * (diameter_m / 2) / r_px, where diameter_m is the known physical diameter of the object class stored in constants.py
   • X and Y in camera frame: unprojected from the centroid using the standard pinhole inverse

   The depth estimate is essentially monocular depth recovery using known object size as a scale reference — a simple but effective stand-in for a depth camera at this scale.

3. 03

   Step 3: RANSAC triangulation. Each object accumulates observations across all views in which it appears. These are passed to a RANSAC-based weighted Direct Linear Transform (DLT) triangulator:

   • The RANSAC outer loop randomly samples pairs of views, triangulates a candidate 3D position from each pair, and counts inliers by reprojection error (threshold: 8 pixels)
   • The best inlier set is then used for a final weighted DLT, where each view's contribution is weighted by its mask area — larger, more clearly visible instances get more influence
   • This robustly rejects views where the object was partially occluded, seen at a glancing angle, or near the image boundary

   Confidence is computed as the fraction of views in which the object was visible, capped at 1.0. The Commander uses this downstream to drop detections at or below 0.6 confidence and keep only the single highest-confidence instance per class.

Why Not Full 6DoF Orientation?

For a pick-and-place task on roughly symmetric food items, orientation is largely unnecessary anyway. The gripper approaches from above, and the contact geometry is forgiving enough that a position-only estimate is sufficient for a reliable grasp.

The orientation that does matter — the end-effector's approach angle relative to each object — is handled not by the pose estimator but by the per-class grasp offset table in main.py. This encodes empirically tuned x/y corrections for camera-to-gripper misalignment and z heights for the pre-grasp, grasp, and lift phases. This is simpler and more reliable than estimating object orientation from images, because it captures the actual systematic error of the specific robot and camera mount rather than inferring object geometry from appearance.

planning

UR7e_CubeGrasp — Pick-and-Place State Machine

Once segmentation and position estimation are complete, the commander node waits for user input before beginning pick and place. After the user presses Enter, the commander node sends a JSON-encoded list containing the sorted objects and their estimated positions to main.py (the pick-and-place node), and execution begins.

The pick-and-place pipeline is organized as a state machine that manages the busy semaphore, the job_queue, and the full grasp sequence. Every arm motion is executed asynchronously through trajectory actions, while gripper commands are handled synchronously through service calls. The node uses the same action design as the arm circler to fill a queue of joint trajectories and execute them sequentially.

All operations begin from the observation frame. After arm circler finishes and the user proceeds with pick and place, the arm moves to the observation frame before starting the first cycle. After every pick and place operation, the arm returns to this pose so the camera can view the entire workspace and recalculate object positions. Re-observing the workspace prevents error buildup that would occur if the arm relied only on predetermined coordinates.

1. Home — observation frame (idle pose); gripper open. Camera has a full view of the workspace for object detection and position recalculation.
2. Object selection — the pick-and-place node uses the sorted list of foods and estimated positions from the JSON payload to determine pick order. YOLO detections provide object classes and centroids for both foods and the plate.
3. Pre-pregrasp — IK solution generated at (cx, cy, cz + 0.5) so the camera can observe the object from a centered viewpoint, reducing fisheye distortion.
4. Refine — a second detection pass recalculates the centroid from the centered viewpoint. This improves consistency when objects are partially visible and rebuilds the queue using the full PICK_OFFSETS configuration for the detected object class.
5. Hover → descend → grip → lift — chained IK solutions reuse the previous joint solution as the seed to avoid discontinuities such as elbow flips. Calibrated hardcoded offsets compensate for object height and form factor differences between fruits while gripping.
6. Over plate → lower → release — translate relative to the plate centroid detected by YOLO using the placement positions from the JSON task list. The arm moves to (plate_x, plate_y, drop_z + 0.20), descends using hardcoded z and y offsets, releases the object, and retracts upward. The z-offset compensates for imperfect table-depth estimation and gripper height, while the y-offset improves lateral placement alignment.
7. Return home — _going_home clears busy once the arm settles at the observation frame, allowing the next pick cycle to begin safely. This repeats until no objects remain in the task list and the module declares the process complete.

State Management Flags

The pipeline uses boolean state flags to coordinate perception, trajectory execution, and grasp planning. These flags prevent overlapping commands and ensure the robot only reacts to detections when it is in a safe observation state.

Detection filtering: The detection node filters out objects within a specified radius of the plate so the pick-and-place node does not attempt to pick foods that have already been placed.