Grape Detection with Mask R-CNN

Harvesting is one of the most expensive and delicate jobs in a vineyard. It needs a lot of careful manual work, and it is still largely not automated. A harvesting robot first has to answer a simple question: where exactly are the grapes? A bounding box isn’t enough to decide where to cut. You need the bunch’s actual shape.

For my dissertation I trained, evaluated and deployed a custom Mask R-CNN model that does exactly that. It detects every grape bunch in an image or video frame and outputs a pixel-level mask for each one.

From photos to a trained model

CollectVineyard photos from Türkiye + Embrapa WGISD
LabelPolygon masks in LabelMe
ConvertCOCO JSON → TFRecords
Fine-tuneMask R-CNN Inception v2 (COCO)
DeployFrozen graph, Flask app, webcam

Building the dataset

I combined photos collected from vineyards in Türkiye with images from the public Embrapa Wine Grape Instance Segmentation Dataset. Every image was normalised to 480×640, with EXIF orientation fixed so phone photos weren’t silently rotated, and split randomly 80/20 into train and test sets:

for i in range(0, len(images)):
    im = Image.open(SOURCE + '/' + images[i])
    im = ImageOps.exif_transpose(im)
    im_resized = im.resize(resolution, Image.ANTIALIAS)
    im_resized.save(DESTINATION + '/' + str(i + 10) + ".png")

# Random 80/20 train/test split
train = random.sample(resized_images, int(len(resized_images) * 0.8))
test = list(set(resized_images).difference(set(train)))

Then came the slow part: drawing a polygon around every bunch, in all 520 images (416 train, 104 test), with LabelMe. Bunches overlap, hide behind leaves and change colour with ripeness, so labelling consistency mattered as much as the model.

A vineyard photo with several grape bunches partly hidden by leaves. — A training image and its hand-drawn LabelMe mask. Each red region is one 'grape' instance.

The same photo in greyscale with each grape bunch filled in red as a labelled mask. — A training image and its hand-drawn LabelMe mask. Each red region is one 'grape' instance.

Training

Instead of training from scratch, I fine-tuned a Mask R-CNN with an Inception v2 backbone pre-trained on COCO, using the TensorFlow Object Detection API. Transfer learning matters with only a few hundred labelled images: the network already knows edges, textures and objects, and only needs to learn what a grape bunch looks like. The key parts of the pipeline config:

model {
  faster_rcnn {
    num_classes: 1                  # just "grape"
    number_of_stages: 3             # RPN → box classifier → mask head
    first_stage_max_proposals: 300
    # ...
  }
}
train_config {
  batch_size: 4
  # Small, step-decayed learning rate: we're adapting, not relearning.
  # initial_learning_rate: 0.00002 → 0.000002
  fine_tune_checkpoint: ".../mask_rcnn_inception_v2_coco_2018_01_28/model.ckpt"
  num_steps: 200000
}

Training ran for tens of thousands of steps, with evaluation runs on the University of Sheffield’s ShARC HPC cluster. I tracked loss and mask quality in TensorBoard and evaluated against the held-out test set.

Results

The final model reaches an F1 score of 82% at IoU 0.50 on the test set, and predicts in about 0.3 seconds per image, fast enough for near-real-time video. On clearly visible bunches it is typically 97–99% confident, and it handles several overlapping bunches in one frame.

Close-up of a red grape bunch detected with 99% confidence and a tight mask. — Detections from the final model. Each bunch gets its own box, confidence score and segmentation mask. Click to enlarge.

Model output with grape bunches segmented on the vine. — Detections from the final model. Each bunch gets its own box, confidence score and segmentation mask. Click to enlarge.

The model works on video too. These clips were filmed walking along vineyard rows, and every frame was run through the model, so bunches are picked up as they come into view:

Frame-by-frame detection on video filmed in a vineyard. Click a clip to watch it larger.

Bunches mostly hidden behind leaves, and green grapes against green foliage, are naturally harder cases. More varied training data, rather than a bigger model, would be the next step.

Putting it to work

A model in a notebook isn’t much use to a vineyard, so I exported the best checkpoint as a frozen inference graph and built two front-ends for it:

A Flask web app. Upload a photo and get it back with masks drawn, plus a gallery of past results and processed videos.
A live camera script. It runs the model frame by frame on a webcam feed with OpenCV and can record the annotated video.

The core of inference is running the graph and then mapping the low-resolution masks back onto the full image:

if 'detection_masks' in tensor_dict:
    detection_boxes = tf.squeeze(tensor_dict['detection_boxes'], [0])
    detection_masks = tf.squeeze(tensor_dict['detection_masks'], [0])
    # Mask R-CNN predicts masks relative to each box; reframe them to image coordinates.
    real_num_detection = tf.cast(tensor_dict['num_detections'][0], tf.int32)
    detection_boxes = tf.slice(detection_boxes, [0, 0], [real_num_detection, -1])
    detection_masks = tf.slice(detection_masks, [0, 0, 0], [real_num_detection, -1, -1])
    detection_masks_reframed = utils_ops.reframe_box_masks_to_image_masks(
        detection_masks, detection_boxes, image.shape[0], image.shape[1])
    detection_masks_reframed = tf.cast(tf.greater(detection_masks_reframed, 0.5), tf.uint8)

Only detections above a high score threshold (0.9 on video, 0.95 for uploads) are drawn, so the output errs towards precision. For a cutting robot, a missed bunch is cheaper than a cut in the wrong place.

What I took away

This project taught me the full computer-vision lifecycle: data collection, careful labelling, transfer learning, evaluation with the right metric, and deployment behind a real interface. The most valuable lesson was how much model quality depends on the dataset. Every hour spent on consistent masks paid off more than any hyperparameter tweak.