the most famous egocentric datasets are people cooking in their own kitchens. the robots we’re training on them are headed for warehouses, garages, and factory floors

apac egocentric stereo is 12 first-person recordings of people actually doing their jobs: an automotive garage, a construction site, an electronics factory, a bar, a shipment hub, a laundromat

head-mounted stereo rig, 1920x1080 per eye at 30 fps, plus a depth render, hand and head tracking, and a caption for what the wearer is doing at every moment. 248 segments spanning 62 distinct verbs

checkout the dataset parsed into fiftyone format. every stream scrubs on one shared timeline in fiftyone: both eyes, depth, tracking, and captions together, one line to load

checkout the dataset here: https://huggingface.co/datasets/Voxel51/APAC-Egocentric-Stereo…

or just jump right in with the hugging face space hosting the dataset: https://huggingface.co/spaces/harpreetsahota/APAC-Egocentric-Stereo-Explorer…

Voxel51/APAC-Egocentric-Stereo · Datasets at Hugging Face

Source: https://huggingface.co/datasets/Voxel51/APAC-Egocentric-Stereo

https://huggingface.co/datasets/Voxel51/APAC-Egocentric-Stereo#apac-egocentric-stereo-%E2%86%92-fiftyone-native-multimodal-mcapAPAC Egocentric Stereo → FiftyOne (Native Multimodal MCAP)

The labeled stereo release ofhumyn-labs/APAC-Egocentric-Stereo-Labeled, converted to native multimodal MCAP episodes.

Twelve people were filmed doing their jobs while wearing a head-mounted stereo rig. Each sequence runs about a minute and carries the rectified video from both eyes, a depth render, a hand and head tracking render, and a caption describing what the wearer is doing at every moment. The work spans industrial, hospitality, logistics and retail settings: an automotive garage, a construction site, an electronics factory, a restaurant, a kitchen, a bar, a shipment hub, a laundromat, a shopping mart, a clothes store and a book store.

https://huggingface.co/datasets/Voxel51/APAC-Egocentric-Stereo#installationInstallation

pip install fiftyone

https://huggingface.co/datasets/Voxel51/APAC-Egocentric-Stereo#usageUsage

import fiftyone as fo
import fiftyone.utils.huggingface as fouh

dataset = fouh.load_from_hub(
    "Voxel51/APAC-Egocentric-Stereo",
    name="APAC-Egocentric-Stereo",
    persistent=True,
)
fo.launch_app(dataset)

The industrial workplaces:

view = dataset.match({"environment": "Industrial"})

https://huggingface.co/datasets/Voxel51/APAC-Egocentric-Stereo#what-you-getWhat you get

Twelve episodes and 12.0 minutes of recording at 30 fps. Each episode carries:

• /stereo\-camera, the rectified pair at 3840x1080 with 1920x1080 per eye, asfoxglove\.CompressedVideo
• /depth\-mapand/hand\-tracking, both 1920x1080, asfoxglove\.CompressedVideo, on the same frame clock as the cameras
• /actionand/action\-caption, the verb, the object and the full caption for each segment, stamped where the segment opens
• /action\.plot, the index of the segment currently running
• /instruction, the task description for the sequence

Across the whole set that comes to 248 captioned segments and 62 distinct verbs.

Episodes carry the fieldssample\_id,environment,scene,task\_description,verbs,num\_segments,num\_frames,duration,fps,sbs\_width,sbs\_height,per\_eye\_width,depth\_widthanddepth\_height.

https://huggingface.co/datasets/Voxel51/APAC-Egocentric-Stereo#notes-on-the-conversionNotes on the conversion

The depth stream is a false-colour 8-bit render distributed as H.264 video. Its pixel values are a colormap. Nothing in the source carries per-pixel distance.

Hand and head tracking is an overlay drawn onto the camera frame, not a stream of joint coordinates.

Both eyes ride in one side-by-side stream. The source publishes the pair pre-rectified in a single file, and the halves are cut atper\_eye\_width.

All three streams are Annex-B H.264 without B-frames. The source stereo video is MPEG-4 Part 2, and the source depth and tracking videos are H.264 with B-frames, so each is re-encoded rather than carried through.

Segments tile each sequence end to end with no gaps. They run three seconds each, except the last in a sequence, which is whatever time remains. One sequence carries 17 segments rather than 21 because its clip runs 50.8 seconds rather than about 60.

The source’s verb vocabulary is inconsistently cased, listing bothApplyingandapplying. Theverbsfield is deduplicated case-insensitively. The/actionmessages carry the verb as written.

https://huggingface.co/datasets/Voxel51/APAC-Egocentric-Stereo#license–attributionLicense & attribution

The source dataset is released byhumyn labsunderCC BY 4.0, and this conversion is distributed under the same license.

Changes from the source: conversion to the FiftyOne MCAP flavor, encoding of the action segments as message streams, and re-encoding of the video.

A monocular counterpart is published athumyn-labs/APAC-Egocentric-Monocular-Labeled.