Hands
A box or outline around each hand in each frame, with left and right kept apart.
An egocentric data category, built on request
Hand-object interaction video, close up and first-person
Close-up first-person footage of hands reaching for, gripping, turning and putting down objects. It's filmed with both hands and the work surface in frame, then labeled hand by hand and object by object, so a grasping or manipulation model learns from many different hands.
✓ reviewed by a person closeup_118 · ready to deliver
The short answer
What is egocentric hand-object interaction video data?
Egocentric hand-object interaction video data is first-person footage centred on the hands as they reach for, grasp, use and release objects, labeled with where each hand and object is and when they touch.
Grasping looks simple until a model tries it. The grip changes with an object's size, weight and shape, and with the person: hand size, left- or right-handedness, how fast they move. Many contributors handling the same kinds of objects is how a model learns that range.
Contact is usually the label that matters most: the frame where a hand first touches an object, and the frame where it lets go. Those rules get agreed before labeling starts, along with how to treat a hand hidden behind the object it's holding.
This category runs through our Data collection service, with hands and objects labeled through our computer vision service. The egocentric video data page covers the devices and labels in full.
This category at a glance
- Data type
- Egocentric (first-person) video
- Focus
- Hands, grips and the objects they handle.
- Devices
- Action cameras or chest mounts for a wide, steady view, smart glasses, or your own rig.
- Labels
- Left and right hands, objects, contact and release frames, and tracking.
- Who films
- Contributors from our 30,000+ network, matched to the hand variety in your brief.
- Service
- Data collection
- Your data
- NDA first, used only for the agreed scope, returned or deleted when you ask.
- First step
- A small free sample before you commit.
Before filming starts
Details worth planning for
A short brief that answers these saves sending people back to film again. The egocentric collection guide has a template that covers them all.
The object set
Which objects, and how many of each: kitchen items, tools, packaging, or a fixed set you send.
Hand variety
Hand sizes, left- and right-handed contributors and skin tones, so the model doesn't learn one kind of hand.
Framing
A wide, steady view with both hands visible. Chest mounts and action cameras usually manage this better than a head strap.
Contact rules
What counts as touching, holding and released, and how a hand hidden behind an object gets labeled.
What gets labeled
Labels for this footage
The usual label set for this category. The set you get is agreed with you before labeling starts.
Objects
A box around each object in use, with its class from the set you agree on.
Contact and release
The frame where each hand touches an object, and the frame where it lets go.
Tracking
Each hand and object followed across the clip with one id.
Where it's used
What this footage trains
Common uses. They show the kind of work, and they aren't client names.
Robot grasping
Manipulation models that learn grips from how many different people pick things up.
Hand tracking for AR and VR
Models that follow the hands so a headset can respond to what they do.
Objects in hand
Recognition models that identify an object while fingers partly cover it.
What you get
What one delivered clip looks like
Each clip in this category comes with its labels, its keysteps, any transcribed speech and the details of who filmed it, where and on what device.
The example on the right is an illustration of the format, not client data. The fields, label names and file types are agreed with you before collection starts, so the delivery drops straight into your pipeline.
Files for each clip
- Video
- The first-person recording, as filmed on the chest-mounted action camera.
- Labels
- Boxes per frame for jar, lid and the hands.
- Keysteps
- The start and end of each step: grasp jar, twist lid, lift lid.
- Speech
- What the wearer says, transcribed with timestamps, when the brief asks for it.
- Metadata
- Contributor id, place, device, task and date, so you can split training and test data by person and place.
closeup_118.json illustration
{
"clip": "closeup_118.mp4",
"device": "chest-mounted action camera",
"task": "open a jar",
"length": "00:09.4",
"keysteps": [
{ "step": "grasp jar", "start": "00:00.4", "end": "00:03.2" },
{ "step": "twist lid", "start": "00:03.2", "end": "00:06.2" },
{ "step": "lift lid", "start": "00:06.2", "end": "00:09.2" }
],
"labels": {
"boxes": ["jar", "lid"],
"frames": "every frame",
"contact": { "hand": "R", "object": "lid", "from": "00:03.1" }
},
"speech": "The lid is tight, twisting it now.",
"contributor": "C-0192",
"home": "H-0214"
}
How it works
From your brief to reviewed footage
One named contact owns the project from scoping to the last file.
Brief
You tell us the tasks, the settings, the device and roughly how much you need.
NDA and scope
We sign an NDA before any data moves, then agree the scope with you.
Free sample
We film a small sample free, so you can check framing and quality before you commit.
Production
Contributors from our network film first-person, on the agreed device, to the brief you set.
Review
Footage and labels are reviewed before anything is delivered.
Delivery
The data comes to you, or moves on to labeling or evaluation with us.
Related
Where this footage fits
This category is part of our egocentric video data work. You can also browse every category on the datasets hub.
More egocentric categories: Egocentric walking and navigation video and Egocentric shopping and retail video.
FAQ
Questions about egocentric hand-object interaction video
The egocentric video FAQ covers questions that apply to every first-person project.
Do you have hand-object footage already recorded?
No. Footage is filmed to your brief: the objects, the hands, the device and how much you need.
Can you supply the objects, or should we?
Either works. Contributors can use everyday objects from their own homes, or you can send a fixed object set so every clip uses the same items.
Can you label hand outlines instead of boxes?
Yes. Outlines follow the hand's shape more closely than a box. They take longer per frame, so we agree with you where they're worth it.
Can we try a small sample first?
Yes, a small free sample before you commit to the full brief.
Can this run alongside other egocentric categories?
Yes. Hand-object close-ups often run with household task footage under one brief and one named contact, drawing on our 30,000+ person network and 150+ projects delivered to date.
Request this category
Pick what you need, say which tasks get filmed and roughly how much footage, and send it. We'll film a small sample free before you commit.
- Your named contact replies within one business day.
- We sign an NDA before any data changes hands.
- We film your free sample, then agree the full scope.
This opens your email app with the brief filled in. Nothing is sent until you press send. You can also use the contact page.