Single Image Depth Estimation Based on CLIP Language Prior Injection

Single Image Depth Estimation (SIDE) has achieved notable progress on standard benchmarks [12]. Following the pioneering CNN-based work of Eigen et al. [13], subsequent research has evolved along three main avenues: (A) Architecture advancements, including the use of residual networks [14], multi-scale fusion strategies [15], Vision Transformers [16], and diffusion models [17]. (B) Improved optimization, such as classification regression paradigms [18] and geometric constraints [19]. and (C) Multi-task learning with auxiliary cues like surface normal [20] or semantics. While effective on indistribution data, these methods often lack the robustness required for real-world applications under diverse conditions. This has shifted the research focus towards generalizable SIDE.

Current efforts for generalization are primarily two-fold. The first, data-driven, relies on large-scale, often web-collected, image-depth pairs. Methods like MiDaS [4] predict affine-invariant relative depth, enabling training on mixed datasets. Omnidata [5] and Depth Anything [12] expanded this concept with massive datasets for zero-shot transfer. Others, like ZoeDepth [6], fine-tune such relative depth models for metric output. While powerful, this paradigm demands significant data and compute. The second, model-driven, leverages strong priors from models pre-trained on even larger datasets. Notably, vision-language models like CLIP-trained on 200 million image-text pairs offer rich, semantically-aligned cross-modal representations. Harnessing these priors for SIDE presents a promising path for improved generalization without exhaustive data collection.

The success of CLIP has spurred its application to various downstream vision tasks, including monocular depth estimation. For instance, Kim et al. [22] proposed a tuning-free method that realigns CLIP’s internal representations via engineered text prompts to elicit depth-aware priors. Chatterjee et al. [23] systematically studied the robustness of linguistic guidance in low-level vision, establishing an evaluation benchmark and a multi-level language dataset for depth estimation. However, these approaches primarily exploit CLIP’s global image-text matching capability, falling short of achieving pixel-level semantic understanding. This limits their potential in dense prediction tasks like depth estimation, where fine-grained spatial alignment is crucial. Bridging this gap-from global matching to pixel-wise adaptation-remains an open challenge.

To this end, we propose a novel text-to-pixel framework that leverages CLIP’s knowledge in a fine-grained manner. Our model consists of two core components: (1) the TRCA module, which establishes pixel-wise semantic correspondence between image features and text descriptions, enabling fine-grained semantic enhancement from text to pixels. and (2) the TBCP module, which discretizes continuous depth into learnable bins and uses gated attention to filter text-based structural cues, thereby mapping semantic priors to depth distributions adaptively. Together, these modules significantly improve the robustness of depth estimation in complex scenes.

Conventional SIDE methods rely solely on visual features, which often leads to significant errors in textureless or occluded regions. To address this, we enhance the Pixelformer architecture with cross-modal priors from CLIP, formulating depth estimation as a classification-regression task. As illustrated in Fig. 1, our framework operates as follows.

Fig. 1. Flowchart of the proposed network model.

Download Original Figure

The input image I is processed by a Swin Transformer backbone [24] to extract multi-scale visual features {E₁, E₂, E₃, E₄} at 1/4, 1/8, 1/16, and 1/32 resolutions, capturing global context via window-based self-attention.

In parallel, a human-written scene description T is tokenized and encoded by the CLIP Text Encoder (CLIP-ViT-L/14) to produce a 1024-dimensional global text embedding F_text. The core of our method consists of two novel modules that fuse these modalities. First, the TRCA module uses F_text to semantically enhance the multi-scale visual features. The enriched features are then passed to a pixel query initializer (PQI), which aggregates global context via multi-scale average pooling to generate initial pixel queries. Second, the TBCP module discretizes the continuous depth range into n_bins adaptive bins. It utilizes a gated attention mechanism to filter structural cues from F_text, producing a per-pixel probability distribution over the bins. The final depth d_i for pixel is obtained by a weighted sum of the bin center values using this distribution.

To effectively fuse linguistic and visual features while preserving visual details, we design the TRCA module, as illustrated in Fig. 2(A) as an interaction module between linguistic and visual features. Since E₄ and F_text differ in spatial dimensions and channel counts, a convolution operation reduces E₄ dimensions to match F_text channel count. This channel alignment preserves E₄ spatial dimensions, preventing information loss from reduced spatial resolution. Next, the query vector Q from F_text is dot-multiplied with the key vector K from E₄ to obtain the self-similarity score matrix S. The similarity score values determine the weights assigned to the value vector V from E₄ in the final output. The weighted sum of S values and vector V is then mapped back to the spatial dimension matching the original visual features via a multilayer perceptron (MLP).

Fig. 2. Module structure diagram.

Download Original Figure

where $\sqrt{d }$ denotes the dimension of the query and key vectors, and Q,K,V are all N×2C in size, with N=h×w representing the sequence length.

The proposed TBCP module is illustrated in Fig. 2(B). Structurally, it retains the core idea of adaptive prediction of depth interval centers from the Pixelformer method while functionally introducing a gated attention mechanism. This mechanism effectively filters scene structural information from textual features, thereby adaptively mapping semantic priors to the depth distribution. Similar to the TRCA module, Q₄ and text features F_text are first mapped to the same spatial dimension via convolution, as shown in Fig. 2(A). Global average pooling is then applied to pixel query Q4 to extract a global feature vector integrating visual-semantic information. Subsequently, adaptive gating is employed to fuse this with F_text:

where W_g denotes the gating weight matrix, F_gap represents the globally averaged pooling-processed visual feature vector, F_text is the global text embedding feature vector, σ is the Sigmoid activation function, and ⊙ denotes element-wise multiplication.

We generate fused features that simultaneously reflect scene visual characteristics and linguistic attributes. subsequently predict the width distribution b across n_bins of depth intervals after processing through a 3-layer MLP.

where Gap(Q₄, F_text) denotes the language-visual fused features processed by global average pooling.

The center of each depth interval is calculated within the dynamically adjusted depth range [d_min (T), d_max (T)] guided by linguistic context, ensuring the intervals align with both the image’s actual depth characteristics and the semantic meaning described in the text. Finally, the center of the i-th depth interval is computed as:

where n_bins denotes the number of depth bins, set to 256 in this paper following the configuration in [1]. d_min represents the minimum depth value, and d_max denotes the maximum depth value.

Finally, at each pixel location, the final depth valueb̃ is computed as a linear combination of the Softmax score at that pixel and the depth interval c(b_i), as shown in equation (6):

where p_k denotes the Softmax score of the pixel within the depth interval.

NYU v2 [26] is an indoor dataset comprising 120,000 RGB images and corresponding depth pairs of size 480×640. These depth pairs were extracted from video sequences captured using a Microsoft Kinect across 464 indoor scenes. This paper evaluates the algorithm using the official training and test set partition:249 scenes comprising 50,000 images for training and 654 images for testing. We adopt the centroid segmentation proposed by Eigen31, with a depth map upper bound of 10 meters. The network outputs depth predictions at 120×160 resolution, which we upsampled by a factor of 4 during training and testing to match ground truth resolution.

For quantitative assessment, we employ four metrics introduced by Eigen [13], which are most widely used in evaluating monocular depth estimation performance. Among these, Abs Rel denotes the absolute relative error at each pixel location, measuring the discrepancy between predicted and ground truth depths. RMSE represents the root mean square error between predicted and ground truth depths. log₁₀ is the average of the absolute values of the logarithmic differences between the predicted and true depths for each pixel. Acc is the proportion of pixels where the error bet-ween the predicted and true depths is below a certain threshold. They are defined as follows:

where N denotes the total number of pixels. d_i represents the predicted depth value for the i-th pixel. ${\text{d}}_{\text{i}}^{*}$ denotes the ground truth depth label for the i-th pixel. thr is the error threshold, typically set to 1.25, 1.25² or 1.25³. This paper introduces parameters (Parameters, Params) and floating-point operations (FLOPs) to evaluate the portability and computational complexity of the network model.

To enhance the credibility of the framework results, comparative experiments were conducted. Models were evaluated under identical experimental conditions on the NYU-Depth v2 dataset, including Eigen et al. [13], Yin et al. [19], BTS [27], Naderi [8], TransDepth [29], DPT [21], PackNet-SAN [30], Adabins [18], P3Depth [25], NeWCRFs [5], and PixelFormer [9]. Through comparative experiments, we analyze performance on the NYU-Depth v2 dataset to better evaluate the strengths and weaknesses of the proposed framework. Experimental results are shown in Table 1.

As shown in Table 1 compares our method with recent state-of-the-art approaches, including Eigen, BTS, AdaBins, NeWCRFs, and PixelFormer. Our method achieves the best performance on key metrics: an Abs Rel of 0.088 and an RMSE of 0.319. This represents a 3.5% reduction in Abs Rel and a 5% reduction in RMSE compared to the strong baseline PixelFormer. The improvement in accuracy thresholds (δ₁, δ₂) further confirms the effectiveness of integrating cross-modal semantic prior.

To assess generalization, we directly evaluate models rained on NYU Depth v2 on the unseen SUN RGB-D dataset without fine-tuning. As shown in Table 2, our method outperforms all compared methods, including PixelFormer. This demonstrates that the semantic priors from CLIP significantly enhance the model’s robustness to domain shifts. As shown in Fig. 3, compared to PixelFormer, the proposed method demonstrates greater robustness in complex indoor scenes. In the first row of high-glare scenes, depth estimation within the boxed area—covering reflective glass and surrounding environments—is more accurate with clearer hierarchical structure. In the second row of high-brightness metallic scenes, the prediction of highly reflective areas indicated by arrows is more stable. In the third row of large-scale indoor scenes, the edges of distant walls are rendered more clearly, with overall depth hierarchy appearing more distinct.

Fig. 3. Comparisons of the depth map estimation results of the PixelFormer method and the algorithm proposed in this paper on the NYU-Depth v2 dataset.

Download Original Figure

Fig. 4 further visualizes results on four distinct indoor test sets, showing that our framework consistently produces more geometrically coherent and robust depth estimates in complex, unseen environments, thanks to the semantically guided feature enhancement and depth discretization mechanisms.

Fig. 4. Comparison of the depth map estimation results of the PixelFormer method and the algorithm proposed in this paper on four different indoor test sets.

Download Original Figure

To validate the effectiveness of the proposed module, we conducted stepwise ablation experiments on the NYU Depth v2 dataset, with results shown in Table 3. It can be observed that: