CrowdKPI

CrowdKPI is a cross-platform audience analytics application that combines a React Native (Expo) mobile app with a FastAPI backend to measure real-time crowd engagement and emotion from video and images.

The mobile app, built with Expo Router for file-based navigation, uses expo-camera and expo-video to capture footage. Recorded frames and videos are sent to the FastAPI backend over HTTP. For image analysis, the backend runs a two-stage inference pipeline: DeepFace with a YOLOv12l detector identifies and crops each face, then a fine-tuned Vision Transformer (ViT) exported to ONNX and served via ONNX Runtime classifies each face into engagement categories. Emotion detection (7 classes) runs in parallel through DeepFace. Each face is returned with its bounding box, dominant emotion, emotion confidence, engagement label, and engagement confidence.

For video submissions, the backend enforces a validation pipeline before storing anything: frames are extracted with OpenCV and at least 3 must contain a detected face (confidence ≥ 0.5) before audio extraction is triggered. Audio is extracted with FFmpeg, transcribed, and the transcript must reach a minimum word count. A middle frame is then sent to Google Gemini Flash (via OpenRouter) for content moderation and a short scene description. Only videos that pass all three gates — face presence, transcript length, and content appropriateness — are uploaded to Cloudflare R2 and indexed in MongoDB alongside the computed valence/arousal/dominance scores submitted by the user.

A human-in-the-loop feedback endpoint lets reviewers submit corrected emotion and cognitive state labels per face, which are persisted back to MongoDB to support future model fine-tuning. The backend is containerized with Docker and deployed on Google Cloud via GitHub Actions CI/CD.