fix: use mtime by default in Trainer._rotate_checkpoints with automatic fallback

[Jerry-Terrasse]

What does this PR do?

This PR fixes an issue with checkpoint rotation in transformers.Trainer. When training with checkpoints saved every 100 steps and a maximum limit of 3 checkpoints, starting a new training session in the same output directory can cause a newly created checkpoint (e.g., checkpoint-100) to be mistakenly identified as the oldest and immediately deleted.

Detailed Problem Description

• Scenario: Training with Trainer using save_steps=100 and save_total_limit=3.
• Issue: After a training run with 700 steps, checkpoints checkpoint-500, checkpoint-600, and checkpoint-700 are produced. When starting a new training run in the same output directory, the new checkpoint (checkpoint-100) is mistakenly identified as the oldest checkpoint due to its lower numerical value and is immediately deleted.
• Cause: The current checkpoint rotation mechanism relies solely on the numerical ordering extracted from the checkpoint directory names, which ignores the actual creation times of the checkpoints.

Proposed Solution

• Default to mtime: Modify _rotate_checkpoints to use file modification time (mtime) by default for ordering checkpoints. This approach better reflects the actual creation order.
• Automatic Fallback: If the mtime values appear unreliable (for example, if they are identical or show insufficient differences on non-POSIX filesystems such as FUSE FS to HTTP blob storage), the system will automatically fall back to the original numerical ordering method.

Related Issue

This PR is related to #26961 and #28862 . In that issue, users observed that using use_mtime=True sometimes resulted in the unintended deletion of newer checkpoints. Although setting use_mtime=False could avoid these issues on certain filesystems, our solution defaults to use_mtime=True to accurately reflect checkpoint creation order, with an automatic fallback mechanism to ensure robustness when mtime is unreliable.

---

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline, Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the documentation guidelines, and here are tips on formatting docstrings.
• Did you write any new necessary tests?

[github-actions]

Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the Ready for review button (at the bottom of the PR page). This will assign reviewers and trigger CI.

[Rocketknight1]

cc @muellerzr @SunMarc