Multimodal & Embodied Intelligence (Pt 1), Panel on Multimodal AI: Progress, Pitfalls, Possibilities

The session on Multimodal and Embodied Intelligence explored the evolving landscape of multimodal AI, its implications for physical AI, and the future of foundation models. It featured talks by Madhava Krishna (IIIT Hyderabad), Sriram Ganapathy (IISc Bangalore), and Somak Aditya (IIT Kharagpur), followed by a panel discussion.

Talk 1: Madhava Krishna (IIIT Hyderabad) – Hybrid Approaches in Robotics

Professor Madhava Krishna’s talk centered on balancing classical, modular robotics with modern end-to-end AI. He highlighted the interpretability and debuggability of classical systems (e.g., SLAM, localization, planning, collision avoidance) that solve problems via optimization and are scene-agnostic. In contrast, end-to-end AI, while promising direct input-to-action, faces challenges with interpretability, data dependency, difficulty with hard constraints, and the vast difference between input and robotic output spaces (e.g., quaternions, SE(3) matrices). He advocated a hybrid approach, combining “the best of both worlds.”

Key Hybrid Examples:

• Crowd Surfer: Integrates Vision-Language Models (VLMs) with optimal control.
• O3D Sim: Uses Large Language Models (LLMs) with classical planning for navigation.
• Smart Wheelchair: Leverages multimodal language and vision for goal identification and trajectory distribution (using VQVAE/flow matching), while retaining classical Model Predictive Path Integral (MPPI) control for navigation and LiDAR-to-camera extrinsics for 3D mapping. This allows natural language commands (e.g., “put some hardware away”) to be executed via classical planners.
• Open-Set Navigation: Self-driving cars interpret natural language queries (e.g., “take me to a roundabout”) using CLIP-aligned multimodal encodings, mapping concepts to 3D locations in SLAM maps, then navigate with classical planners.
• Sparse Log: Combines foundational models (LLaMA, Grounded DINO, SAM) for landmark identification (e.g., “tree trunk”, “traffic signals”) with classical Monte Carlo optimization for localization, drastically reducing map density (4M points to 4K centroids).
• Manipulation: Employs diffusion models for trajectory generation, guided by classical collision detection systems to ensure hard constraint satisfaction.

Professor Krishna emphasized that while navigation hardware is largely solved, dexterous manipulation and aggressive drone flight remain hardware-limited. He also noted that robotics, being application-driven, often requires tailored hardware, and deployment demands strong on-site troubleshooting and integration skills.

Talk 2: Sriram Ganapathy (IISc Bangalore) – Multimodal AI in Healthcare

Professor Sriram Ganapathy discussed the AI Center of Excellence in Health, a government-funded initiative at IISc Bangalore aimed at democratizing AI in healthcare, particularly focusing on Non-Communicable Diseases (NCDs). He highlighted the scarcity of major AI success stories in health, attributing this to: lack of deployment considerations in model development, the necessity for seamless human-AI integration (AI as assistance, not replacement), and issues with model calibration and uncertainty.

Key Initiatives and Challenges:

• Focus on NCDs: Aims to empower frontline health workers at the “first point of arrival” in the healthcare chain, where resource constraints are highest.
• Health Stack & India-Centric Foundation Models: Envisions a national “health stack” and is developing India-centric foundation models (e.g., for the Indian brain).
• Full Pathway Development: Emphasized the comprehensive effort required, from ethical approvals, data collection, AI model development, to rigorous validation (randomized control trials, regulatory approvals like FDA for “software as a medical device”), and integration into Digital Public Infrastructure (DPI).
• Oral Cancer Workflow: A mobile AI model detects suspicious oral lesions, trained on extensive real-world data, and is currently undergoing regulatory approval. It operates offline on edge devices, making it suitable for rural deployments.
• Uncertainty Modeling (SHORE): Advocated for models that provide confidence scores and can “abstain” or say “I don’t know” when uncertain, especially in cases of domain mismatch or unseen data. This “Semantic Uncertainty and Reliability Estimation” (SHORE) improves model reliability.
• Human-AI Interaction: Stressed that AI tools are to empower healthcare providers, not for direct patient interaction, ensuring human clinical judgment remains central.

Talk 3: Somak Aditya (IIT Kharagpur) – Multimodal Reasoning and Perception

Professor Somak Aditya presented his exploration into educational video question answering, which led to a deeper investigation into fundamental perception and reasoning challenges in multimodal AI.

Key Findings:

• Educational VQA: Initial efforts to use multimodal LLMs (MLMs) for generating pedagogically sound answers to student questions from lecture videos revealed low factual recall and precision, even with fine-tuning. Generating contextually appropriate answers remained difficult.
• Perception Bottleneck: Even for direct recall tasks from long videos, state-of-the-art GPT models showed poor factual recall (around 50%). This highlighted perception as a critical bottleneck for effective reasoning.
• Neuro-Symbolic Approach for Visual Puzzles: To isolate perception from reasoning, he used well-defined visual planning puzzles. He found that VLMs often failed due to incorrect perception (misinterpreting initial or goal states), not reasoning errors.
• Improving Perception and Program Synthesis:
  • Fine-grained Detection: Explored combining diffusion models (Stable Diffusion) and CLIP-based architectures to improve fine-grained detection of puzzle elements, noting diffusion models’ superior ability to capture fine details compared to CLIP’s high-level semantics.
  • Program Synthesis: Investigated evolutionary program synthesis to improve the generation of logic programs from visual inputs, focusing on sample efficiency and partial correctness fitness metrics.
• Four Fundamental Questions: He concluded with key challenges in multimodal reasoning: minimizing perception errors in VLMs/MLMs, increasing long-video recall, achieving high-accuracy action detection, and advancing reasoning and world models.
• Puzzle-Based Learning: Proposed using visual puzzles as an engaging, adaptive vehicle for education, with features like on-the-fly hint generation and vernacular support.

Panel Discussion: Multimodal AI – Progress, Pitfalls, and Possibilities

The panel, moderated by Tanuja Ganu, delved into the broader implications of multimodal AI.

1. Impact of Scale and Foundation Models:

• Multimodality as the New Standard: AI research is increasingly multimodal; single-modality work is becoming rare.
• Transformative Interaction: Multimodal models have changed how users interact with and consume media and information.
• Challenges: While proficiency has increased, factuality, grounded reasoning, and reliability (e.g., hallucinations, harmful responses) remain significant issues needing further research.

2. AGI vs. Human-like Perception and Reasoning:

• Vague AGI Definition: Panelists agreed that the term AGI is often vague. AI’s role should be to extend human capabilities, not just to replace them or match human performance.
• Digital vs. Physical AI: Current AGI discussions largely focus on the digital “web world,” with AI in the physical world (embodied intelligence) being far less mature.
• Defining AI’s Role: It’s crucial to define AI’s boundaries in collaboration with humans, rather than pursuing independent intelligence. AI should solve problems humans find undesirable or are incapable of.

3. Pitfalls and Gaps in Deployment:

• Human Factors: Deploying AI involves complex human interactions, political will, and convincing stakeholders who may not understand the technology but wish to implement it. “Humans are messier than models.”
• Efficiency and Cost: The significant gap between advanced proof-of-concept models and efficient, scalable, cost-effective deployment remains a major challenge. There’s a strong industry and academic push for efficient (e.g., 1-bit, 2-bit quantized) and task-specific models (e.g., newborn anthropometry on low-cost smartphones).
• Architectural Limitations: Current multimodal LLMs often treat other modalities as text-encoded inputs, limiting their ability to handle non-verbalizable, intuitive, or subconscious multimodal tasks effectively. The “one-size-fits-all” approach may not suit the diverse range of real-world multimodal problems.

4. Future Directions & Academia-Industry Collaboration:

• Small Language Models (SLMs) and Agents: A focus on SLMs and task-specific, goal-oriented agents is seen as a promising direction, reducing the reliance on massive, costly models.
• Rethinking “Foundation Model”: The term “foundation model” was critically assessed as often misused, applied to models covering only small regions of a modality (e.g., just segmentation in vision) or relying on external tools. A truly foundational vision model should seamlessly integrate classification, captioning, segmentation, 3D, and motion understanding. The need for AI to tackle unconventional problems (e.g., climbing a coconut tree) beyond human benchmarks was also raised.

• Tags:
• Foundation Models
• Robotics
• Feature
• Multimodal AI
• AI Ethics
• Healthcare AI
• End-to-End AI
• Embodied Intelligence
• Classical AI
• Hybrid AI
• Non-Communicable Diseases (NCDs)
• Uncertainty Modeling
• AI Deployment
• Perception in AI
• Multimodal Reasoning
• Neuro-Symbolic AI
• Small Language Models (SLMs)