Quick Buzz Feed
New features include Semantic Search with NVIDIA Cosmos, Comparison Mode, Remote Access and an Agent Sidebar.
Foxglove Semantic Search empowers robotics developers by allowing them to input natural language descriptions of desired behaviors or scenarios. The system then intelligently retrieves corresponding multimodal segments from vast, unlabeled datasets. A key advantage is that each search result maintains its connection to the synchronized sensor data, telemetry, logs, and system signals recorded at that specific moment. This allows developers to seamlessly transition from identifying an event on camera to gaining a comprehensive understanding of the robot's overall actions and state during that incident. This powerful semantic search capability is driven by NVIDIA Cosmos, an open video-text embedding model explicitly engineered for Physical AI. Unlike general-purpose models, NVIDIA Cosmos is trained on specialized robotics, driving, and ego-centric human action data, which ensures superior retrieval accuracy for complex physical AI benchmarks. The collaboration between Foxglove and NVIDIA has facilitated the integration of Cosmos, providing physical AI teams with a fully managed indexing and inference infrastructure, thereby eliminating operational overhead. Furthermore, by utilizing Foxglove’s Bring-Your-Own-Storage (BYOS) architecture, Semantic Search can directly index and search raw logs stored in a customer’s cloud storage, avoiding unnecessary data duplication.
As robotics operations expand, the volume of data generated by fleets—across various robots, environments, and model iterations—increases dramatically. The crucial events that lead to significant improvements are often rare and challenging to pinpoint. The value of this data hinges on the speed at which developers can locate these events, analyze the underlying causes, and translate them into effective fixes, robust evaluations, or enhanced training data. Foxglove’s newly introduced capabilities are designed to interconnect and accelerate each stage of this learning loop. The Agent Sidebar acts as a central coordinator, enabling developers to efficiently find specific data points, generate insightful visualizations, investigate system failures, perform comparative analyses between different operational runs, and curate high-quality datasets for further use. Semantic Search, leveraging NVIDIA Cosmos, plays a vital role in identifying pertinent behaviors and events hidden within extensive volumes of unlabeled robot data. Comparison Mode is a critical tool that synchronizes two or more robot runs, allowing teams to precisely pinpoint where system behavior diverged across different models, software versions, or operational scenarios. This functionality is indispensable for understanding 'drift' between model versions and for rigorously validating each new software release. Lastly, Remote Access revolutionizes field debugging by enabling engineers to stream every topic from a remote robot directly to their browser. This provides real-time observation and debugging of deployed systems from any location, with camera, lidar, telemetry, and logs arriving at low latency within the same familiar application used for visualizing recorded data. This eliminates the need for costly and time-consuming onsite visits for diagnosing field issues. All these features are built upon a foundational principle: the aggregation of time-synchronized data from diverse sources like cameras, lidar, radar, transforms, telemetry, and logs, forming a unified and reliable record of the robot's actual activities. This data-centric approach ensures the dependability of agentic workflows in Physical AI environments, where outcomes must align with real-world physical realities. As companies scale their robotic fleets into full production, Foxglove positions itself as the essential platform for transforming collected data into a powerful engine for continuous improvement and innovation.