Real-Time Multi-Modal Audio, Video and Physiological Data Collection

Real-Time Multi-Modal Audio, Video and Physiological Data Collection#

by Paul Helstab & Danila Mamontov

Repository:
phelstab/sensoring

Documentation / Online Book:
https://phelstab.github.io/sensoring

INFO
This repository is a third party software project for real-time data acquisition and visualization of audio, video, and physiological signals with Polar H10, Empatica E4, and Azure Kinect DK sensors.

  • Empatica E4 (Real-time physiological signal sensoring)

  • Polar H10 (Real-time heart rate signal sensoring)

  • Azure Kinect Developer Kit (3D spatial information sensor device to create computer vision and speech model applications)

Sensor Information#

Device

Sensors

Data Stream

Frequency (set to)

Output

Empatica E4

Photoplethysmography sensor (PPG)

Blood Volume Pressure (BVP)

64Hz

.csv

Heart Rate (HR)

event-based (derived from BVP)

.csv

Interbeat Interval (IBI)

event-based (derived from BVP)

.csv

Electrodermal activity sensor (EDA)

Galvanic skin response (GSR)

4Hz

.csv

3-axis accelerometer

XYZ-axes per timestep (ACC)

32Hz

.csv

Optical thermometer (probably infrared)

Skin temperature
in celsius per timestep (TEMP)

4Hz

.csv

Polar H10

Electrocardiography sensor (ECG)

ECG data (ECG)
electric potential in microvolts (μV)

130Hz

.csv

Interbeat Interval (IBI)

event-based (derived from ECG)

.csv

Heart Rate (HR)

event-based (derived from ECG)

.csv

3-axis accelerometer (XYZ)

XYZ-axes per timestep (ACC)

200Hz

.csv

Azure Kinect DK

Depth camera: 1MP Time-of-flight

NFOV unbinned: 640x576
NFOV 2x2 binned: 320x288
WFOV 2x2 binned: 512x512
WFOV unbinned: 1024x1024
Passive IR: 1024x1024

0, 5, 15, or 30 FPS

-

RGB camera: 12MP CMOS sensor rolling shutter

3840x2160 (UHD)
2560x1440 (QHD)
1920x1080 (FHD)
1280x720 (HD)

0, 5, 15, or 30 FPS

.mkv

IMU (Inertial Measurement Unit): 3D accelerometer and gyroscope

XYZ-axes linear acceleration in (m/s²);
XYZ-axes rotational speed (rad/s)

Sampling rate: 1.6 kHz
Reporting rate to host: 208 Hz

-

Microphone: 7-microphone circular array

Channel Count: 7 channels
Sample Format: 16-bit integer (Int16)

Sample Rate: 16000 Hz (16 kHz)

.wav

[orbbec, 2024], [Empatica, 2024], [Polar, 2024], [Määttä, 2024]

Milestones#

  • Lightweight real time data acquisition of Polar H10 Sensors

  • Lightweight real time data acquisition of Empatica E4 Sensors

  • Flawless simultaneous recording of audio and video from the Azure Kinect DK

  • Parallel caputuring of sessions for experiments

  • Real time visual feedback of data aquisition

  • CI/CD Pipeline for versioning and fast deployment of features and fixes

  • Free online book for documentation and theorethical background about sensoring

Open To Do’s:

  • Audio input selection e.g. for alternative microphone inputs (not only Azure Kinect DK microphone)

  • Eye sensoring with HTC vive pro

Optional To Do’s:

  • Triggering data aquisition in the sensor software with the unreal engine endpoints

  • Real time Body Tracking with Azure Kinect DK

    • Could be done better afterwards on the video footage

References#

Contents