Image Segmentation with Computer Vision

In this notebook, you’ll explore image segmentation — the most detailed form of computer vision. Instead of drawing a box around something, segmentation labels every single pixel in the frame, letting us recolor hair, replace backgrounds, and change clothing colors in real time.

What Is Image Segmentation?

You’ve seen how object detection draws bounding boxes — rectangles around objects. Segmentation goes much further: it assigns a class label to every pixel in the image.

This platform uses MediaPipe Selfie Multi-Class Segmenter, which classifies pixels into 6 categories:

Class	What it covers
`BACKGROUND`	Everything behind the person
`HAIR`	Hair on the head
`BODY_SKIN`	Exposed skin on arms and hands
`FACE_SKIN`	Skin on the face
`CLOTHES`	Clothing worn by the person
`OTHERS`	Accessories, glasses, etc.

Step 1: Explore Segment Classes

The cell below starts your camera and segmenter, then prints the segment classes it can see in the frame once per second. Make sure your face, hair, and shoulders are visible.

Click Allow for webcam access, then click Stop (■) when you’re ready to move on.

import cv
import graphics
import time

canvas = graphics.canvas()
camera = cv.start_camera(canvas)
segmenter = cv.start_segmenter(camera)

frame = 0
try:
  while True:
    if frame % 30 == 0:
      segments = segmenter.get_segments()
      print("Segments visible:", segments)
    frame += 1
    time.sleep(0.033)
finally:
  segmenter.stop()
  camera.stop()
  print("Camera stopped.")

Step 2: Recolor Your Hair

The cv.color_segment() function fills a segment class with a solid color at a given opacity. The original video still shows through — the color is an overlay blended on top.

Run the cell to color your hair pink in real time. Click Stop (■) when you’re done.

import cv
import graphics
import time

canvas = graphics.canvas()
camera = cv.start_camera(canvas)
segmenter = cv.start_segmenter(camera)

try:
  while True:
    cv.color_segment(canvas, segmenter, cv.SEGMENT.HAIR, "#ff69b4", opacity=0.7)
    time.sleep(0.033)
finally:
  segmenter.stop()
  camera.stop()
  print("Camera stopped.")

Step 3: Replace Your Background

The cv.apply_image_to_segment() function replaces a segment with an image file, clipped precisely to that segment’s shape.

Run the cell to replace your background with a sample image in real time. Click Stop (■) when you’re done.

import cv
import graphics
import time

canvas = graphics.canvas()
camera = cv.start_camera(canvas)
segmenter = cv.start_segmenter(camera)

try:
  while True:
    cv.apply_image_to_segment(canvas, segmenter, cv.SEGMENT.BACKGROUND, "/sample_files/cat.jpg", opacity=0.9)
    time.sleep(0.033)
finally:
  segmenter.stop()
  camera.stop()
  print("Camera stopped.")

Step 4: Experiment — Color Any Segment

Use the controls below to choose which body part to recolor, what color to apply, and how strong the effect is. Set your values, run the cell, and see the result live.

Stop the cell, adjust the controls, and run again to try something different. Click Stop (■) when you’re done.

import cv
import graphics
import time

SEGMENT = "HAIR"    #@param ["HAIR", "BODY_SKIN", "FACE_SKIN", "CLOTHES"]
COLOR   = "#ff69b4" #@param ["#ff69b4", "#00bfff", "#00ff88", "#ff4500", "#ffffff", "#ffff00"]
OPACITY = 0.6       #@param {type:"slider", min:0.1, max:1.0, step:0.1}

segment_map = {
  "HAIR":      cv.SEGMENT.HAIR,
  "BODY_SKIN": cv.SEGMENT.BODY_SKIN,
  "FACE_SKIN": cv.SEGMENT.FACE_SKIN,
  "CLOTHES":   cv.SEGMENT.CLOTHES,
}

canvas = graphics.canvas()
camera = cv.start_camera(canvas)
segmenter = cv.start_segmenter(camera)

try:
  while True:
    cv.color_segment(canvas, segmenter, segment_map[SEGMENT], COLOR, opacity=OPACITY)
    time.sleep(0.033)
finally:
  segmenter.stop()
  camera.stop()
  print("Camera stopped.")

How precise were the segment boundaries? Did you notice any areas where the model struggled — for example, at the edges of your hair or where your clothing meets your skin?

Virtual background replacement is now built into most video call apps. What are the benefits? Are there any situations where this technology could be misused or misleading?

Check for Understanding

{ “question_type”: “multiple_choice”, “question”: “What makes image segmentation more detailed than object detection?”, “options”: [ { “key”: “a”, “text”: “Segmentation is faster to run” }, { “key”: “b”, “text”: “Segmentation labels every pixel, not just bounding boxes” }, { “key”: “c”, “text”: “Segmentation only works on people” }, { “key”: “d”, “text”: “Segmentation doesn’t use neural networks” } ], “answer”: “b”, “submitted_answer”: “” }

{ “question_type”: “true_false”, “question”: “Image segmentation draws bounding boxes around each detected person.”, “answer”: “False”, “submitted_answer”: “” }

{ “question_type”: “multiple_choice”, “question”: “Which segment class would you use to recolor someone’s hair?”, “options”: [ { “key”: “a”, “text”: “cv.SEGMENT.FACE_SKIN” }, { “key”: “b”, “text”: “cv.SEGMENT.BODY_SKIN” }, { “key”: “c”, “text”: “cv.SEGMENT.HAIR” }, { “key”: “d”, “text”: “cv.SEGMENT.OTHERS” } ], “answer”: “c”, “submitted_answer”: “” }

{ “question_type”: “true_false”, “question”: “You can only apply color to one segment class per frame.”, “answer”: “False”, “submitted_answer”: “” }