Hands
A box or outline around each hand in each frame, with left and right kept apart.
An egocentric data category, built on request
Egocentric household task video, filmed in real homes
First-person footage of the jobs people do at home: cooking, washing up, folding laundry, tidying a shelf. Contributors from our network film in their own homes on a head or chest mount, smart glasses or your own device, so a robot or assistant model sees the variety of real kitchens and living rooms, not one lab set.
✓ reviewed by a person kitchen_041 · ready to deliver
The short answer
What is egocentric household task video data?
Egocentric household task video data is first-person footage of people doing everyday jobs at home, filmed from a camera they wear, and labeled with the hands, objects and steps of each task. It's used to train robots and assistants that work in homes.
Homes vary more than any lab. Counter heights, cupboard handles, cluttered sinks and the way different people fold a towel all change what a model has to handle. Filming across many homes is how that variety gets into the data.
Household footage is one of the main sources for robot learning from human video: a model watches how people grasp, move and place things, then carries that over to a robot's own arm.
This category runs through our Data collection service, with hands, objects and steps labeled through our computer vision service. The egocentric video data page covers the devices and labels in full.
This category at a glance
- Data type
- Egocentric (first-person) video
- Typical tasks
- Cooking, washing up, laundry, tidying and cleaning.
- Devices
- Head or chest-mounted phones, action cameras, smart glasses, or devices you send us.
- Labels
- Hands, objects, tracking, and keysteps with a start and an end.
- Who films
- Contributors from our 30,000+ network, in their own homes.
- Service
- Data collection
- Your data
- NDA first, used only for the agreed scope, returned or deleted when you ask.
- First step
- A small free sample before you commit.
Before filming starts
Details worth planning for
A short brief that answers these saves sending people back to film again. The egocentric collection guide has a template that covers them all.
Tasks, and how scripted
A step list like fill the kettle, take a cup, pour gives clips you can compare. Free tasks give the variety of real homes. Many briefs mix the two.
How many homes
The number of homes, kitchen layouts and regions, and the most footage any one home can give, so one kitchen doesn't teach the whole model.
Mount and view
A head mount follows the eyes. A chest mount keeps the counter steadier in frame. Pick the one closest to your robot or device.
Privacy at home
Family members, photos, mail and screens turn up in home footage. Agree what stays out of frame and what gets removed before delivery.
What gets labeled
Labels for this footage
The usual label set for this category. The set you get is agreed with you before labeling starts.
Objects in use
The cups, plates, cloths and tools being handled, with the frames where a hand touches them.
Keysteps
The start and end of each step, like rinse the plate or stack it, from a label set you agree on.
Talk-through
What the wearer says as they work, transcribed with timestamps that match the footage.
Where it's used
What this footage trains
Common uses. They show the kind of work, and they aren't client names.
Household robots
Models that learn to load a dishwasher or wipe a counter by watching many people do it.
Assistants on glasses
Assistants that follow a recipe or a chore along with the wearer and know what comes next.
Step recognition
Models that tell which step of a task someone is on, and whether one was skipped.
What you get
What one delivered clip looks like
Each clip in this category comes with its labels, its keysteps, any transcribed speech and the details of who filmed it, where and on what device.
The example on the right is an illustration of the format, not client data. The fields, label names and file types are agreed with you before collection starts, so the delivery drops straight into your pipeline.
Files for each clip
- Video
- The first-person recording, as filmed on the head-mounted phone.
- Labels
- Boxes per frame for plate, sponge, plate stack and the hands.
- Keysteps
- The start and end of each step: rinse plate, scrub, stack.
- Speech
- What the wearer says, transcribed with timestamps, when the brief asks for it.
- Metadata
- Contributor id, place, device, task and date, so you can split training and test data by person and place.
kitchen_041.json illustration
{
"clip": "kitchen_041.mp4",
"device": "head-mounted phone",
"task": "wash and stack plates",
"length": "00:31.0",
"keysteps": [
{ "step": "rinse plate", "start": "00:01.2", "end": "00:10.5" },
{ "step": "scrub", "start": "00:10.5", "end": "00:20.5" },
{ "step": "stack", "start": "00:20.5", "end": "00:30.4" }
],
"labels": {
"boxes": ["plate", "sponge", "plate stack"],
"frames": "every frame"
},
"speech": "Rinsing this one first, then the pan.",
"contributor": "C-0192",
"home": "H-0077"
}
How it works
From your brief to reviewed footage
One named contact owns the project from scoping to the last file.
Brief
You tell us the tasks, the settings, the device and roughly how much you need.
NDA and scope
We sign an NDA before any data moves, then agree the scope with you.
Free sample
We film a small sample free, so you can check framing and quality before you commit.
Production
Contributors from our network film first-person, on the agreed device, to the brief you set.
Review
Footage and labels are reviewed before anything is delivered.
Delivery
The data comes to you, or moves on to labeling or evaluation with us.
Related
Where this footage fits
This category is part of our egocentric video data work. You can also browse every category on the datasets hub.
More egocentric categories: Egocentric walking and navigation video and Egocentric shopping and retail video.
FAQ
Questions about egocentric household task video
The egocentric video FAQ covers questions that apply to every first-person project.
Do you have household footage already recorded?
No. Footage is filmed to your brief: the tasks, the homes, the device and how much you need.
Can contributors film in their own homes?
Yes. Contributors film in their own homes, with their consent recorded before collection starts and the privacy rules from your brief applied to every clip.
Can this footage train a robot?
Yes, that's one of the main uses. Tell us where your robot's camera sits and which tasks it has to learn, and we'll match the mount and the label set to them.
Can we try a small sample first?
Yes, a small free sample before you commit to the full brief.
Can this run alongside other egocentric categories?
Yes. Household tasks often run with hand-object close-ups under one brief and one named contact, drawing on our 30,000+ person network and 150+ projects delivered to date.
Request this category
Pick what you need, say which tasks get filmed and roughly how much footage, and send it. We'll film a small sample free before you commit.
- Your named contact replies within one business day.
- We sign an NDA before any data changes hands.
- We film your free sample, then agree the full scope.
This opens your email app with the brief filled in. Nothing is sent until you press send. You can also use the contact page.