[megatron] feat: use flash as default attention_backend

[ISEEKYAN]

What does this PR do?

Add concise overview of what this PR aims to achieve or accomplish. Reference related GitHub issues and PRs that help with the review.

1. add mapping from string to megatron's enum for attention_backend choice.
2. use flash attention as default attention_backend, for consistency to FSDP

[gemini-code-assist]

Code Review

This pull request updates the default attention backend to 'flash' for consistency and introduces a utility function to map string values for attention_backend to the corresponding Megatron enum. The changes are applied across several configuration files and in the worker initialization logic. My review focuses on improving the implementation of the new utility function for better readability and conciseness. Overall, the changes align with the stated goals of the PR.

[ccclyu]

will switching to flash as default backend have potential break for some training recipes? also does it have any performance/efficiency regressions or boost?

[ISEEKYAN]

will switching to flash as default backend have potential break for some training recipes? also does it have any performance/efficiency regressions or boost?

@ccclyu There is no performance difference compared with the previous default TE.fused_attn. But in some cases (DAPO with 20k response with qwen3 14B/30B) I found the fused_attn will make the training crash after a long training (32H207days), while no crash found when flash_attn is used. with TE2.2.

I am still trying to find out what is the root cause and whether the crash is fixed with new versions of transformer engine(TE2.7).