This paper proposes a meticulous fine-tuning strategy for the XLNet-Base model to achieve improved performance in Natural Language Understanding (NLU) tasks, particularly sentiment analysis on the SST-2 dataset. In the proposed method, the AdamW optimizer is adopted instead of classical configurations and is combined with a learning rate scheduling scheme that includes linear decay and a warm-up phase, which increases gradient stability during training. Correspondingly, implementing techniques such as half-precision floating-point training (FP16), utilizing dynamic padding, increasing the maximum input sequence length to 256 tokens, and leveraging updated GLUE dataset samples significantly contributed to improving the model's generalization capability and accelerating convergence. Evaluations demonstrate that the XLNet-Base model, configured with these settings, achieved an accuracy of 94.7% and an F1 score of 94.7% on the SST-2. These results signify an improvement of approximately 1.5 to 2 percentage points compared to the previously reported performance for XLNet-Base (around 93%) in the paper literature. Finally, these findings underscore that improving the fine-tuning process, even without structural changes to the model's core architecture, can play a pivotal role in elevating the efficiency of pre-trained language models.
