Four-stage pipeline: ingest and dead-space removal, camera-pose transition proposal with Depth Anything 3, clip segmentation, and grounded Qwen3.8 captioning
2026

Segmenting Egocentric Work Videos into Described Sub-Tasks Without Labels

Shivam Singh (intuition.dev)

Built a geometry-first pipeline that turns long body-camera recordings of manual work into sub-task clips, each with a grounded natural-language description, without training a segmentation model and without labels. Four cheap detectors remove dead footage (a parked camera, a blank wall, a black screen) and the remainder is stitched. The camera's metric trajectory, recovered with Depth Anything 3, is scanned with a station-change template: the wearer was still, turned, walked a distance scaled to the space, and stayed, with the cut placed at the walk onset. Each clip is described by a 27B Vision-Language Model (Qwen3.8 via vLLM) under strict grounding rules that separate what is visible from what is inferred, and a text-only pass over the ordered descriptions yields the video's global context. On a corpus of 20 videos (3.4 h) the pipeline removed 1,621 s of dead footage, proposed 37 station changes and produced 54 grounded clip descriptions with no failed requests, in under an hour on one GPU. Every boundary is explainable by three numbers; ablations of the pose models, the detectors, the cut anchor and a video-adaptive stillness window fixed the final configuration.

Spatial video understanding pipeline
2026

Spatial Video Understanding from Raw Monocular RGB Videos

Shivam Singh (AMD India)

Developed an end-to-end video reasoning framework that performed object tracking, spatial understanding, and long-horizon question answering directly from raw monocular RGB videos. Built a large-scale automatic data generation pipeline that combined Depth-Anything-3, YOLO-World, and Vision-Language Models to reconstruct the 3D structure of the environment, including object locations and camera trajectories, and generated high-quality QA supervision for object tracking, spatial reasoning, and goal-oriented planning. Trained the proposed SlotMem model with persistent cross-frame memory, achieving state-of-the-art performance on the OST-Bench benchmark (44.4%) while outperforming all previously reported baselines.

Pause and Think teaser figure
IEEE/RSJ IROS 2026

Pause and Think: A Dataset and Benchmark for Video-Grounded Assistive Action Suggestion

Shivam Singh, S Majumder, P Prabhanjan, Z Liu, E Barsoum

Recent Vision-Language Models struggle with grounded reasoning, temporal consistency, and context-aware planning in videos. We introduce Pause-and-Think-T, a reasoning-centric training dataset that encourages models to pause, reason over visual evidence, and produce concise, actionable responses. A compact 4B-parameter model fine-tuned on our dataset achieves 58.0% accuracy at 59× fewer parameters than Qwen3-VL-235B, matching GPT-5.2 on scene understanding while remaining suitable for real-time edge deployment on AMD Ryzen AI hardware.

Anticipate Adapt Act teaser figure
ECMR 2025

Anticipate, Adapt, Act: A Hybrid Framework for Task Planning

N Dash, A Kaura, Shivam Singh, R Singh, S Banerjee, M Sridharan, M Krishna

Anticipating and adapting to failures is a key capability robots need to collaborate effectively with humans in complex domains. We present a hybrid framework that integrates the generic prediction capabilities of an LLM with the probabilistic sequential decision-making capability of Relational Dynamic Influence Diagram Language. For any given task, the robot reasons about the task and the capabilities of the human attempting to complete it, predicts potential failures, and executes actions to prevent or recover from them.

AdaptBot teaser figure
ICRA 2025

AdaptBot: Combining LLM with Knowledge Graphs and Human Input for Generic-to-Specific Task Decomposition and Knowledge Refinement

Shivam Singh, K Swaminathan, N Dash, R Singh, S Banerjee, M Sridharan, M Krishna

We present AdaptBot, a framework combining Large Language Models, Knowledge Graphs, and Human-in-the-Loop approaches to enable embodied agents to handle unseen tasks and adapt to new scenarios. The system decomposes generic tasks into specific executable actions while refining domain knowledge through human feedback, bridging the gap between high-level reasoning and grounded task execution in household environments.

Anticipate and Act teaser figure
ICRA 2024

Anticipate & Act: Integrating LLMs and Classical Planning for Efficient Task Execution in Household Environments

R Arora*, Shivam Singh*, K Swaminathan, A Datta, S Banerjee, B Bhowmick, KM Jatavallabhula, M Sridharan, M Krishna

We present an intelligent household agent that learns task execution patterns by extracting user behavior and preferences using Large Language Models and designing task and action representations in Planning Domain Definition Language (PDDL). By integrating LLM-based task anticipation with classical planning, the system enables efficient task execution in household environments.

Workshops

ICRA 2024 Workshop

DaTAPlan: Data-driven Task Anticipation and Knowledge-driven Planning for Human-robot Collaboration

Shivam Singh*, K Swaminathan*, R Arora*, R Singh, A Datta, D Das, S Banerjee, M Sridharan, M Krishna — Cooking Robotics Workshop, Yokohama.

RSS 2024 Workshop

Anticipate & Collab: Data-driven Task Anticipation and Knowledge-driven Planning for Human-robot Collaboration

Shivam Singh*, K Swaminathan*, R Arora*, D Das, S Banerjee, M Sridharan, M Krishna — Task Specification for General-Purpose Intelligent Robots, Delft.

Patents

US Patent Application · 2026

Method for Automatic Knowledge Refinement and Task Execution

Shivam Singh, S Banerjee, K Swaminathan, N Dash, R Singh, M Sridharan, M Krishna — US20260216870A1, Tata Consultancy Services. Published Jul 2026 (pending).

US Patent Application · 2025

Method and System for Task Anticipation by Integrating Large Language Models and Classical Planning

Shivam Singh, S Banerjee, B Bhowmick, R Arora, A Datta, K Swaminathan, M Krishna, M Sridharan, KM Jatavallabhula — US20250342322A1, Tata Consultancy Services. Published Nov 2025 (pending).