Back to Engineering Projects
Computer Vision👁 Computer Vision
Face & Hand Segmentation
A deep learning-based segmentation project for identifying facial and hand regions from images to support vision-based applications.
Benchmark Results & Metrics
Inference Speed
60 FPS
Mean IoU
0.89
Model Size
18MB
Latency
12ms
01 // Overview
This project implements an optimized pixel-level segmentation pipeline targeting human faces and hands. Designed for real-time applications, it operates efficiently on resource-constrained devices.
02 // The Problem
Existing segmentation models are either too large for real-time edge execution or lack accuracy under poor lighting conditions and extreme angles.
03 // System Architecture
Lightweight Dual-Stream Segmentation Architecture
A specialized CNN architecture leverages depthwise separable convolutions to independently predict binary masks for face and hand regions, maximizing throughput.
ARCHITECTURAL DATA FLOW:Image Frame -> MobileNetV3 -> Skip Connection Decoder -> Sigmoid Boundary Mask -> Face/Hand Regions
Architecture Components:
Video Capture Pipeline
MobileNetV3 Encoder
U-Net Decoder with Skip Connections
Post-processing Bilateral Filter
04 // Implementation
Built in Python using PyTorch and OpenCV. Pretrained weights were fine-tuned on custom datasets using dice-loss optimization, then exported to ONNX format for deployment.
05 // Execution Workflow
11. Input frames captured via webcam.
22. Images resized and normalized.
33. CNN generates probability masks.
44. OpenCV thresholds masks to binary regions.
55. Segmented boundaries overlaid onto visual output.
06 // Technology Stack
Computer Vision
OpenCVPyTorchAlbumentations
Deployment
ONNX RuntimePythonNumPy
07 // Technical Challenges
⚠️Differentiating overlapping hands and faces in low-resolution video inputs.
⚠️Reducing edge jitter during rapid gesture movements.
Lessons Learned
- •Using depthwise separable convolutions cuts inference latency in half with minimal accuracy loss.
- •Temporal smoothing filters eliminate mask flickering in consecutive frames.
Future Improvements
- •Adding 3D pose and joint landmark tracking alongside 2D segment masks.
- •Optimizing the pipeline for Apple CoreML and WebGL runtime environment.