3D Pose Avatar

In this miniapp, your body controls a 3D figure in real time. Pose landmark coordinates are mapped into a 3D scene — as you move, the avatar moves with you.

Open In Jupyter K-12

From 2D Landmarks to 3D Space

The pose detector gives us x and y coordinates for each joint in camera pixel space. To place them in a 3D scene we rescale and offset them:

scene_x = (landmark_x / canvas_width  - 0.5) * 5       
scene_y = -(landmark_y / canvas_height - 0.5) * 4 + 1.5

Each joint becomes a sphere in the scene, colored by body region. Joints with low visibility (occluded or out of frame) are hidden below the scene floor.

Introducing scene3d

The scene3d module creates an interactive 3D scene powered by BabylonJS. You get a free-orbit camera (drag to rotate, scroll to zoom), a ground plane, and simple shapes. The @scene.on_frame decorator registers a function called every render tick — that’s where we update the avatar positions each frame.

Step 1: Live Pose Preview

First confirm the pose detector is tracking you correctly. Step back so your full upper body is visible and check that the skeleton overlay looks right.

Click Stop (■) when you’re ready to launch the 3D scene.

import cv
import graphics
import time

canvas = graphics.canvas()
camera = cv.start_camera(canvas)
detector = cv.start_pose_detector(camera)

try:
  while True:
    poses = detector.get_detections()
    canvas.draw_poses(poses)
    time.sleep(0.033)
finally:
  detector.stop()
  camera.stop()
  print("Camera stopped.")

Step 2: 3D Avatar

The cell below shows a small camera preview (top) and a full 3D scene (below). Each sphere represents a joint — move your arms, tilt your head, and watch the avatar respond.

Drag the 3D scene to orbit the camera, scroll to zoom in or out.

Click Stop (■) when you’re done.

import cv
import scene3d

camera   = cv.start_camera()
detector = cv.start_pose_detector(camera)

scene = scene3d.Scene()
scene.set_sky('#0d1b2a')
scene.set_ground(length=12, width=12)

JOINTS = {
  'NOSE':           ('#ffffff', 1.25),
  'LEFT_SHOULDER':  ('#4488ff', 1.00),
  'RIGHT_SHOULDER': ('#4488ff', 1.00),
  'LEFT_ELBOW':     ('#44aaff', 0.5),
  'RIGHT_ELBOW':    ('#44aaff', 0.5),
  'LEFT_WRIST':     ('#88ccff', 1.0),
  'RIGHT_WRIST':    ('#88ccff', 1.0),
  'LEFT_HIP':       ('#ff8844', 1.0),
  'RIGHT_HIP':      ('#ff8844', 1.0),
}

spheres = {}
for name, (color, size) in JOINTS.items():
  s = scene3d.Shapes.Sphere(diameter=size, segments=8)
  s.set_color(color)
  s.set_position(0, -10, 0)
  scene.add(s)
  spheres[name] = s

def to_3d(lm):
  x = (lm['x'] / 320 - 0.5) * 5
  y = -(lm['y'] / 600 - 0.5) * 8 + 1.5
  return x, y, 0

@scene.on_frame
def update(dt):
  poses = detector.get_detections()
  if poses:
    pose = poses[0]
    for name, sphere in spheres.items():
      lm = pose[getattr(cv.POSE, name)]
      if lm['visibility'] > 0.5:
        sphere.set_position(*to_3d(lm))
      else:
        sphere.set_position(0, -10, 0)

scene.run()

What would make this avatar feel more alive or useful? What would you build if you could control a 3D character with your body?

Ideas to Extend This App

  • Bones — add a thin Box between each pair of joints (shoulder→elbow, elbow→wrist) and update its position each frame
  • Reactions — change a sphere’s color when a wrist rises above the shoulder (arm raised)
  • Squat counter — track when the hips drop below a threshold and count reps, displayed on the 3D scene’s HUD
  • More joints — add knees and ankles by stepping further back so the full body is visible
  • Particles — spawn a short-lived sphere at the wrist each frame for a motion trail effect