Auto-optimized parameter-efficient fine-tuning for skin lesion classification with vision transformers

Official URL: https://dx.doi.org/10.1109/RTSI64020.2025.11212626

Vision Transformers (ViTs) have achieved great performance on a variety of computer vision tasks, including medical image classification. Fine-tuning these models is computationally costly. However, Parameter-Efficient Fine-Tuning (PEFT) methods address this limitation by fine-tuning a minimal number of parameters. Despite their efficiency, the success of these methods depends greatly on the hyperparameter selection. In this paper, we propose an automatic hyperparameter search method for BEFT using Optuna.To validate the effectiveness and generality of our approach, we applied this Auto-Optimized PEFT strategy to three state-of-the-art pretrained Vision Transformers: ViT-Base, ViT-Medium, and Swin Transformer. After optimization using our method, ViT-Base achieved an accuracy improvement of +10.58% (87.17%), ViT-Medium achieved an accuracy improvement of +7.09% (83.23%), and Swin Transformer achieved an accuracy improvement of +18.62% (85.57%). The results highlight the effectiveness of PEFT methods in combination with automatic hyperparameter tuning to successfully fine-tune large vision models even in resource-scarce medical environments.