GAN-based Vision Transformer for High-Quality Thermal Image Enhancement

Highlights

1. New Architecture: The paper proposes a fresh approach that combines vision transformers and GANs for enhancing thermal images. This unique architecture generates high-quality thermal images that closely resemble real ones.
2. Thermal Loss Feature: The paper introduces a specialized thermal loss feature that enhances the visual realism of the generated images. This feature plays a key role in producing high-quality results by focusing on the unique characteristics of thermal images.
3. Fine-tuning with Visible Images: By fine-tuning the enhancement process using visible images, the proposed method takes advantage of the additional details present in these images. This step improves the overall quality of thermal images and yields better performance compared to other methods.

Abstract

Generative Adversarial Networks (GANs) have shown an outstanding ability to generate high-quality images with visual realism and similarity to real images. This paper presents a new architecture for thermal image enhancement. Precisely, the strengths of architecture-based vision transformers and generative adversarial networks are exploited. The thermal loss feature introduced in our approach is specifically used to produce high-quality images. Thermal image enhancement also relies on finetuning based on visible images, resulting in an overall improvement in image quality. A visual quality metric was used to evaluate the performance of the proposed architecture. Significant improvements were found over the original thermal images and other enhancement methods established on a subset of the KAIST dataset. The performance of the proposed enhancement architecture is also verified on the detection results by obtaining better performance with a considerable margin regarding different versions of the YOLO detector