Stroke-Aware Flow for License Plate Recognition

Deep learning-based License Plate Recognition (LPR) systems have achieved remarkable growth over the past few years. Laroca et al. [1] proposed a robust LPR system utilizing the YOLO object detector, achieving high accuracy under various shooting conditions, while Zherzdev and Gruzdev [2] demonstrated real-time recognition capabilities on embedded devices through the lightweight architecture of LPRNet. However, a limitation of these models is their reliance on high-resolution, clear images for training and evaluation data. Recently, Nascimento et al. [3] released the UFPR-SR-Plates dataset collected from real surveillance environments, quantita-tively demonstrating that the performance of existing models degrades rapidly under real-world degradation conditions such as low resolution and blur.

This issue of low resolution is not unique to license plates but is a common challenge in intelligent surveillance, including Person Re-Identification (Re-ID). For instance, Liu et al. [13] recently proposed a framework enhancing Person Re-ID performance by integrating Super-Resolution technology, proving that restoring visual details is crucial for downstream recognition tasks in surveillance scenarios.

To address this in the LPR domain, Nascimento et al. [4] attempted a Character-Driven approach utilizing OCR model feedback as a loss function, but issues regarding computational efficiency in complex restoration processes and resolution mismatch due to varying distances remain unresolved.

In contrast to these studies, our work specifically targets the severe performance drop of standard LPR systems [1,2] under real-world conditions, as highlighted by recent benchmarks [3]. Unlike previous restoration attempts [4] that are limited by computational complexity and fixed-scale upsampling, our proposed SAF-LPR provides a unified and efficient solution capable of handling complex, compound degradations and arbitrary resolution changes inherent in varying surveillance distances.

To enhance the readability of low-resolution text images, research utilizing structural characteristics of characters beyond simple pixel restoration has been actively conducted. Chen et al. [6] proposed the Text Gestalt model mimicking human cognitive processes, improving recognition performance by recovering incomplete stroke information. Furthermore, Ma et al. [7] presented the TPGSR framework generating semantically valid images by utilizing the probability sequence of text recognizers as a text prior, and Guan et al. [14] enhanced text consistency by combining OCR post-processing into the document restoration pipeline. In the license plate domain, Wang et al.[5]improved the sharpness of character contours through a neural network emphasizing edge information. However, these methods suffer from slow inference speeds due to being based on large-scale Transformers or involving multi-stage processing, and they face difficulties in balancing pixel-wise fidelity and semantic recognition rates.

Distinct from these existing semantic-aware approaches [5-7,14] which often rely on heavy computational structures like Transformers or complex multi-stage pipelines, our approach integrates stroke priors and semantic feedback into a highly efficient invertible framework. This unique integration allows SAF-LPR to achieve state-of-the-art restoration quality and recognition accuracy while maintaining fast inference speeds suitable for practical applications, effectively resolving the trade-off between pixel fidelity and semantic recoverability.

Invertible neural networks (INNs) are gaining attention in image generation and restoration fields due to their ability to minimize information loss by learning bijective mappings between input and output. Foundational work by Dinh et al. [15-16] with NICE and RealNVP, and Kingma and Dhariwal [17] with Glow, established the theoretical basis for invertible flow-based generative models. Applying this to image restoration, Liu et al. [10]'s InvDN achieved both lightweight architecture and high performance by effectively separating noise in the latent space, while Jing et al. [18]'s HiNet and Xie et al. [19]'s Inv-Compress demonstrated the superior information presservation capabilities of INNs in information hiding and compression. Recently, Xiao et al. [8] and Li et al. [11] improved performance in rescaling problems by intro-ducing invertible residual connections, yet most were limited to supporting fixed scaling factors. To overcome this, Pan et al. [9] proposed IARN capable of arbitrary rescaling, but it failed to incorporate structural information specific to domains like text or license plates. Additionally, Park et al. [12] proposed adaptive filters responding to complex degradations, but this tends to increase model complexity. Based on the efficiency of INNs and arbitrary scaling technology, our SAF-LPR presents a license plate restoration solution optimized for real-world environments by integrating stroke information and recognizer feedback.

Building upon the efficiency of INNs [10] and recent advancements in arbitrary rescaling [9], the core originality of our work lies in tailoring this architecture specifically for license plate recognition. Unlike generic INN-based restoration methods that lack domain-specific guidance [9,11], we propose a novel mechanism to inject explicit stroke structure priors and semantic acknowledgment feedback directly into the invertible flow. This unique combination enables SAF-LPR to handle complex real-world degradations and arbitrary scale changes efficiently, surpassing the limitations of previous generic approaches.

The goal of Stage 1 is to train an encoder E that takes I_HR as input and generates $I_{LR}^{gen}$ similar to the real I_LR, using a real-world HR-LR image pair dataset D = {(I_HR, I_LR)}. Unlike simple Bicubic downsampling, this encoder learns non-linear real-world degradations through Noise injection and residual blocks (ResBlock) within the network.

The encoder has a pyramid structure consisting of L levels to capture multi-scale information. Each level l consists of N residual blocks and one downsampling layer. To minimize information loss and ensure invertibility, we use the PixelUnshuffle operation instead of traditional pooling to move spatial resolution to the channel dimension.

To simulate sensor noise specific to CCTVs, we inject learnable Gaussian noise into the output of each residual block.

Here, σ is a learnable scalar parameter that self-learns the optimal noise intensity for each layer.

We apply Global Average Pooling (GAP) to the output F_L of the last level of the encoder to extract a Degradation Vector z_deg that summarizes the degradation characterristics of the entire image (e.g., degree of blur, lighting conditions).

This z_deg is used as a condition for Adaptive Modulation in the Stage 2 restoration process. Once Stage 1 training is complete, all weights W_E and noise parameters σ of the encoder are frozen and transferred to Stage 2.

Stage 2 is the process of restoring the low-resolution input I_LR to high-resolution I_HR based on the degradation information learned in Stage 1. This stage consists of decoder initialization via Weight Transfer and an Active Refinement process.

Instead of randomly initializing the restoration decoder D, we configure it as a ‘mirror image’ of the Stage 1 encoder E. The weights $W_{enc}^{\left(l\right)}$ of the convolution filters at the l-th level of the encoder are transposed to set the initial weights $W_{dec}^{\left(L-l\right)}$ of the corresponding level in the decoder.

Furthermore, the noise parameters σ learned in Stage 1 are also transferred to the decoder, serving as thresholds or regularization parameters for denoising during the restor-ation process. This transfer strategy enables the decoder to start training from a state close to the inverse operation of physical degradation, accelerating convergence.

The decoder progressively restores resolution through PixelShuffle operations. To accommodate various CCTV shooting distances, we apply Arbitrary Scale Rescaling techniques. The target scale and relative coordinate grid C_rel are injected as conditions to interpolate the image at continuous scales rather than fixed factors (e.g., x 4) [9].

Simultaneously, the degradation vector z_deg extracted in Stage 1 modulates each residual block of the decoder via spatial feature transform (SFT) layers. This adaptively alters the filter characteristics according to the degradation state (rain, blur, etc.) of the input image [12].

Physical restoration alone is often insufficient to perfectly recover severely damaged characters. Therefore, we add semantic guidance. The intermediately restored feature maps are fed into LPRNet to generate a character probability sequence P_ocr, which is then fed back into the decoder's bottleneck to assist contextual restoration [7]. Finally, the Stroke Attention Module calculates the correlation between the restored features and predefined stroke templates T_stroke to sharpen the character contours [6].

In Stage 2 training, only the decoder parameters are updated (while the encoder and LPRNet are frozen), minimizing the weighted sum of the following three loss functions.

To reduce pixel-level differences between the restored image Ĩ_HR and the ground truth I_HR, we use the Charbonnier Loss, which is robust to outliers.

To enhance high-frequency components that determine character readability, we calculate the L₁ distance between edge maps extracted using filters such as Sobel [5].

To directly optimize recognition performance, we use the frozen LPRNet as a discriminator. This comprises Perceptual Loss, which measures the distance between feature maps, and connectionist temporal classification (CTC). Loss, which improves the accuracy of the predicted text sequence [2,4].

Since the core objective of this study is to improve license plate recognition performance in real-world environments, we used the recently released UFPR-SR-Plates [3] as our benchmark dataset. This dataset consists of images collected from real surveillance camera environ-ments and includes realistic degradation factors such as resolution reduction due to distance, motion blur, and various lighting conditions. For the experiment, we classified the dataset into three groups (Easy, Medium, and Hard) according to difficulty. Additionally, to enhance the generalization performance of degradation patterns in the initial training phase (Stage 1), we utilized synthetic data created by adding artificial noise and blur to the high-resolution license plate dataset, CCPD, as an auxiliary resource [1].

The proposed SAF-LPR model was implemented using the PyTorch framework and trained on two NVIDIA A100 GPUs. For the Stage 1 (degradation encoder) training, the learning rate was set to 1 x 10⁻⁴ for 200 epochs. In Stage 2 (Restoration Decoder), fine-tuning was performed with a lower learning rate of 5 x 10⁻⁵ after weight transfer. We used the AdamW optimizer with a batch size of 32. For evaluation metrics, we used peak signal-to-noise ratio (PSNR) and structural similarity index map (SSIM) to measure image quality, and adopted recognition accuracy (RA) using a pre-trained LPRNet [2] as the primary metric to verify actual recognition performance.

To verify the performance of the proposed method, we conducted comparative experiments with representative image super-resolution models (Bicubic, SwinIR) and text/license plate-specific restoration models (Text Gestalt [6], TPGSR [7], EENN [5]). Table 1 presents the quantitative evaluation results for the ‘Hard’ difficulty level of the UFPR-SR-Plates dataset. The general-purpose SR model, SwinIR, recorded high PSNR but showed low recognition accuracy (RA). This implies that pixel-level restoration does not guarantee the structural shape of characters. On the other hand, text-specific models like Text Gestalt and TPGSR showed improvements in recognition rates, but the proposed SAF-LPR outperformed them by over 4.6% in recognition accuracy. In particular, we confirmed that the combination of stroke information and the information preservation capability of the invertible neural network enabled the restoration of clear contours even in severely blurred characters.

Qualitative evaluation also reveals the superiority of SAF-LPR. As shown in the comparison results in Fig. 2, SAF-LPR effectively suppressed noise based on the degradation information learned in Stage 1 and clearly distinguished fine stroke differences in easily confused characters such as ‘3’ and ‘8’. This demonstrates that the LPRNet-based semantic feedback successfully balanced visual quality and machine readability.

Fig. 2. Example of license plate recognition performance of the proposed SAF-LPR: It shows that the license plate text is accurately detected even for highly degraded LR inputs.

Download Original Figure

We conducted an ablation study to analyze the impact of each component of the proposed framework on performance.