ICCV Decoded: Explainability Through Model Representations

As someone deeply involved in deploying deep learning systems to production, I keep encountering the same challenge: models can be highly accurate and still fundamentally difficult to trust. When they fail, the reasons are often opaque. Internal representations are hard to reason about, hidden failure modes emerge late, and many post-hoc explainability tools stop at surface-level signals without answering a more fundamental question: what has the model actually learned?

This question was very much on my mind at ICCV, where I attended a talk by Thomas Fel. His presentation immediately stood out. The way he covered the space- both technically and philosophically- was precise, rigorous, and deeply thoughtful. He framed interpretability not as a visualization problem or an auxiliary debugging step, but as a way to reason about models at the level where decisions are formed: their internal representations. That framing resonated strongly with me.

Interpreting models through internal representations

In his NeurIPS 2023 paper, Thomas and his co-authors introduce Inverse Recognition (INVERT), a method for automatically associating hidden units with human-interpretable concepts. Unlike many prior approaches, INVERT does not rely on segmentation masks or heavy supervision. Instead, it identifies which concepts individual neurons discriminate and quantifies that alignment using a statistically grounded metric. This enables systematic auditing of representations, surfacing spurious correlations, and understanding how concepts are organized across layers- without distorting the model or making causal claims the method cannot support.

From research insight to applied practice

This line of work aligns closely with how we think about applied explainability at Tensorleap. For practitioners, interpretability only matters if it leads to action: better debugging, more reliable validation, and clearer decision-making around models in real systems. Representation-level analysis addresses a critical gap- cases where a model appears to perform well, but for the wrong reasons.

Beyond the research itself, Thomas is also a remarkably clear and insightful speaker. That was a key reason I invited him to join us for a Tensorleap webinar. Our goal was not to simplify the research, but to explore how these ideas translate into practical insight for engineers working with complex models in production.

If you’re grappling with understanding what your models have learned, or why they fail in unexpected ways, I strongly recommend watching the full webinar recording to dive deeper into representation analysis and applied interpretability.

Modern object detection models often achieve strong aggregate metrics while still failing in systematic, repeatable ways. These failures are rarely obvious from standard evaluation dashboards. Mean Average Precision (mAP), class-wise accuracy, and loss curves can suggest that a model is performing well- while masking where, how, and why it breaks down in practice.

In this post, we analyze a YOLOv11 object detection model trained on the COCO dataset and show how latent space analysis exposes failure patterns that standard evaluation workflows overlook. By examining how the model internally organizes data, we uncover performance issues related to overfitting, scale sensitivity, and inconsistent labeling.

Why aggregate metrics fail to explain model behavior

Object detection systems operate across diverse visual conditions: object scale, occlusion, density, and annotation ambiguity. Failures rarely occur uniformly; instead, they cluster around specific data regimes.

However, standard evaluation metrics average performance across the dataset, obscuring correlated error modes such as:

• Over-representation of visually simple samples during training
• Systematic degradation on small objects
• Conflicting supervision caused by inconsistent labeling

Understanding these behaviors requires analyzing how the model represents samples internally-not just how it scores them.This analysis was conducted using Tensorleap, a model debugging and explainability platform that enables systematic exploration of latent representations, performance patterns, and data-driven failure modes.

How latent space reveals structured model behavior

Deep neural networks learn internal representations that encode semantic and spatial structure. In object detection models, these latent embeddings determine how different images are perceived as similar or different by the model.

By projecting these embeddings into a lower-dimensional space and clustering them, we can observe how the model organizes the dataset according to its learned features.

Clusters in this space often correspond to shared performance characteristics. Regions with elevated mean loss or skewed data composition point to systematic failure modes rather than isolated errors.

Over-representation and overfitting in simple training samples

One prominent cluster is dominated by images containing train objects with minimal occlusion on training samples compared to validation samples. When the latent space is colored by the number of train objects per image, this region becomes immediately apparent.

This cluster exhibits low bounding box loss on the training set but significantly higher loss on validation data. The samples are strongly correlated with low occlusion, indicating that they represent visually “easy” cases.

The imbalance suggests that the model has overfit to these simple examples, memorizing their patterns rather than learning features that generalize. When validation samples deviate slightly, such as containing higher object overlap, performance degrades sharply.

Scale-dependent failure on small objects

A separate low-performance region emerges when the latent space is examined alongside bounding box size statistics. Samples dominated by small objects consistently show higher loss values.

Qualitative inspection of samples from this region confirms the pattern: the model reliably detects large objects while frequently missing smaller ones in the same scene. This behavior is reinforced by a clear trend in the data- loss increases as object size decreases.

Rather than sporadic errors, small-object failures appear as a structured limitation tied to how the model represents scale internally.

Labeling inconsistency and semantic ambiguity

Another low-performing cluster is dominated by images containing books. Coloring the latent space by the number of book instances reveals a strong concentration of such samples.

Inspection of these images exposes multiple labeling inconsistencies:

• Some visually identical books are annotated while others are not
• Books are sometimes labeled individually and other times as grouped objects
• Similar visual scenes receive conflicting supervision signals

Layer-wise attention analysis further reveals that earlier layers focus on individual books, while final layers often collapse attention onto entire shelves. This mismatch suggests that fine-grained representations exist internally but are not consistently reflected in final predictions.

As the number of books per image increases, loss rises accordingly, reinforcing the conclusion that labeling ambiguity directly degrades model performance.

Why these latent failure patterns matter

These findings highlight a fundamental limitation of aggregate evaluation metrics: they describe how well a model performs on average, but not where or why it fails.

Latent space analysis exposes:

• Overfitting driven by data imbalance
• Scale-sensitive performance degradation
• Supervision noise caused by inconsistent annotations

Crucially, these issues emerge as structured patterns across groups of samples, not as isolated mistakes.

Seeing model failures the way the model does

Object detection models can appear robust while harboring systematic weaknesses that only surface under specific conditions. By analyzing latent space structure and correlating it with performance and metadata, we can uncover hidden failure patterns and trace them back to data properties and representation gaps.

This shifts model debugging from reactive error inspection to evidence-based understanding, grounded in how the model actually perceives the dataset.

When pose estimation models fail, the reasons are rarely obvious. Aggregate metrics smooth over structure, and manual inspection of individual samples often leads to intuition-driven guesswork rather than understanding.

This is a recurring challenge in deep learning debugging: knowing not just where a model fails, but what it has learned, and how those learned concepts influence performance and behavior.

Tensorleap addresses this challenge by creating a visual representation of the dataset as seen by the model itself. By analyzing internal activations, Tensorleap reveals how the model interprets data, uncovers the concepts it has learned, and connects those concepts directly to performance patterns.

In this post, we analyze a pose estimation model trained on the COCO dataset, showing how latent space analysis exposes hidden failure patterns that systematically degrade model performance.

How the model sees the dataset

Before debugging failures, it helps to start with a more fundamental question: How does the model internally organize the data it learns from?

Tensorleap captures activations across the model’s computational graph and embeds each input sample into a high-dimensional latent space. This space is then projected into an interpretable visual representation, where proximity reflects similarity in how the model processes images.

In effect, this creates a map of the dataset from the model’s point of view- revealing how inputs are grouped based on learned representations rather than labels or predefined rules.

From latent space to learned concepts

Once the latent space is constructed, samples are automatically clustered based on their learned representations. These clusters reflect concepts learned by the neural network, not categories imposed by the dataset.

Overlaying dataset metadata on this space reveals strong semantic alignment. For example, indoor scenes containing household objects naturally group together, confirming that the latent space encodes meaningful semantic structure.

Identifying performance aggressors

Each latent cluster is linked to performance metrics, loss components, and dataset metadata. Clusters that contribute disproportionately to error are identified as performance aggressors- groups of samples that actively degrade model behavior during training and inference.

Below, we examine two such aggressors uncovered in the pose estimation model.

Aggressor #1: close-up images driving false positives

One cluster exhibits classification loss more than twice the dataset average.

This cluster is dominated by close-up images with heavy occlusion, including food on tables and pets such as cats. Most images contain no people, and when humans do appear, they are often partially visible.

Despite this, the model frequently predicts the presence of a person. Loss spikes most sharply in images with zero people, indicating a systematic bias toward positive human detection.

Why this matters:
‍This cluster does not represent isolated errors. These samples disproportionately influence classification loss and reshape the decision boundary, reinforcing false positives during training.
Isolating this cluster makes it possible to reason about how close-up, non-human imagery biases the classifier and to evaluate whether these samples should be treated differently during analysis or training.

Aggressor #2: crowded scenes with inconsistent pose labels

Another cluster shows degradation across all loss components: classification, bounding boxes, pose, and keypoint estimation.

These samples correlate strongly with sports and stadium scenes containing both players and spectators. Inspection of annotations reveals that many visible people are partially or entirely unlabeled for pose.

Comparing object-detection and pose annotations shows a median gap of 13 people per image, exposing inconsistency in what constitutes a labeled subject. Activation heatmaps confirm that the model attends to both labeled players and unlabeled spectators, leading to contradictory training signals.

Why this matters:
‍Performance degradation in these scenes is driven by label inconsistency, not pose complexity. Without distinguishing between the two, errors in crowded environments are easily misattributed to difficult motion or insufficient model capacity.

From insights to action

These failure patterns share three characteristics:

• They are systematic
• They are high-impact
• They are invisible to aggregate metrics

By visualizing how the model interprets the dataset and linking learned concepts to performance, teams gain a grounded understanding of why models fail and where deeper investigation is required.
This approach enables more informed decisions around data inspection, targeted analysis, and follow-up experimentation.

Bottom line

Pose estimation models do not fail randomly- they fail in patterns shaped by data, labels, and learned representations.

By creating a visual representation of how the model sees the data, Tensorleap uncovers hidden failure patterns and reveals the dataset concepts that most strongly affect performance.

When metrics stall and intuition runs out, the answer is often already there- embedded in the model’s latent space.

What’s most frustrating? The problem often isn’t your technical skills or the model’s potential. It’s the way the neural network operates that’s holding you back.
This is where Applied Explainability comes in. In this post, we’ll walk you through how to embed Applied Explanability across the deep learning lifecycle, transforming black-box models into transparent, reliable, and production-ready systems.

What is so hard to build and deploy deep learning models?

Customers, stakeholders and leaders all want AI and they want it now. But developing and deploying reliable and trustworthy neural networks, especially at scale can feel like fighting an uphill battle.

• The models themselves are opaque, making it hard to trace errors or understand why a prediction went wrong.
• Datasets are often bloated with irrelevant samples while critical edge cases are missing, hurting accuracy and generalization.
• Labeling is expensive and inconsistent
• Without proper safeguards, models can easily reinforce bias and produce unfair outcomes.
• Even when a model reaches production, the challenges don’t stop. Silent failures creep in through distribution shifts, or edge cases, and debugging becomes reactive and fragile.

What is applied explainability and how does it work?

Applied Explainability is a broad set of tools and techniques that make deep learning models more understandable, trustworthy, and ready for real-world deployment. By analyzing model behavior in depth, the reasons they fail and how to fix them, Applied Explainability allows data scientists to debug failures, improve generalization, and optimize labeling and dataset design. With Applied Explainability, teams can proactively detect failure points, reduce labeling costs, and evaluate models at scale with greater confidence.
Applied Explainability is woven throughout the deep learning pipeline, from data curation and model training to validation and production monitoring. When integrated into CI/CD workflows, it enables continuous testing, per-epoch dataset refinement, and real-time confidence monitoring at scale. The result? More structured, efficient, and reliable deep learning development.

Integrating applied explainability into deep learning dev pipelines

Applied Explainability isn't a post-production, monitoring analysis that is nice-to-have. Rather, it’s a mandatory active layer that should be integrated throughout the deep learning dev lifecycle:

Step 1: Data curation

Training datasets are often lacking: they can be repetitive, incomplete or include inaccurate information. Applied Explainability allows teams to prioritize which samples to add, re-label, or remove to construct more balanced and representative datasets, while saving significant time and resources on labeling, ultimately improving model performance and generalization.
This is done by analyzing the model’s internal representations, uncovering underrepresented or overfitted concepts in the dataset and calculating the most informative features, while finding the best samples for labeling to improve variance and representations.

Step 2: Model training

Training does not provide reliable generalization, which is required for ensuring production reliability. Applied Explainability enables automated adjustments to the dataset while training is in progress, such as removing redundant samples, exposing unseen populations, or targeting weak generalization areas. The result is faster convergence, better generalization, and fewer wasted cycles on uninformative data.
This is done by continuously refining the dataset and training process based on the model’s evolving understanding. It tracks activations and gradients during each training epoch to detect which features and data clusters the model is learning or overfitting. 

Step 3: Model validation

The model might pass validation metrics, but fail catastrophically on edge cases. Applied Explainability enables a deeper, more structured validation process. Teams can pinpoint not just if the model works, but where, why, and for whom it does or doesn’t.
It breaks down model performance across the concepts, identified in the latent space. Each subset is tracked using performance indicators and feature data, all indexed in a large database.
When a new model is introduced, the system compares its performance across these concepts, especially identifying where it performs better than older models. It helps detect clusters where the model consistently fails or overfits, tracks mutual information to assess how generalizable learned features are, and compares current model behavior against previous versions.

Step 4: Production monitoring

Failures happen in production and data scientists don’t know why or how to fix it. Applied Explainability gives teams visibility into the model’s performance holistically, providing an understanding of why the model failed.
When a failure occurs, it simulates where similar issues might arise in the future by grouping failing samples with identical root causes to evaluate the model’s quality and generalization. It also explores a model’s interpretation of each sample, identifies main reasons for the prediction and indexes and tracks performance across previously identified failure-prone populations, allowing teams to monitor whether new models fix past issues or reintroduce them. 

How applied explainability helps develop and deploy reliable deep learning models

The Applied Explainability impact:

• Faster development cycles- by identifying data issues, feature misbehaviors and edge-case failures early in the workflow, teams reduce back-and-forth debugging and costly retraining. This accelerates iteration loops, shortens time-to-deploy, and reduces the number of retraining cycles.
• Control and clarity- for data scientists under pressure to deliver accurate, fair, and defensible models, Applied Explainability offers reassurance. It makes development and deployment easier and helps build trust not just in the model, but in the data scientist’s work, making it easier to stand behind their decisions with confidence.
• Visibility, explainability, and transparency- data scientists get clear insights into why a model made a prediction, where it fails, and how it evolves across training and production. This transparency builds trust among stakeholders and speeds up incident investigation.
• Saving resources in engineering & labeling- Applied Explainability saves engineering time on unexplainable bugs. It reveals mislabeled data, overfit regions, and redundant or unnecessary features, helping teams optimize datasets and reduce manual labeling.
• Enhanced confidence in the deployed model- engineers, data scientists and business stakeholders can validate that the model is behaving reasonably, fairly and robustly, even under rare or critical conditions, and without surprise failures in production. This results in higher model adoption, especially for critical scenarios, fewer rollback risks, and better alignment with business and user expectations.
• Scalability and generalization- by analyzing behavior across diverse data clusters, Applied Explainability ensures the model generalizes well, not just on test data, but across real-world populations and unseen cohorts.

From guesswork to guarantees: make deep learning work at scale

Applied Explainability bridges the gap between deep learning models that break in production and robust systems that operate with confidence. If you’re tired of flying blind with black-box models, it’s time to bring Applied Explainability into the heart of your workflow. Learn how with Tensorleap.