why is the warmup_iters set 2000? #552

Open

Open

why is the warmup_iters set 2000?#552

The total batch size( in token ) is 0.5M in one iteration ,which follows the GPT3 paper. With the same logic, since the GPT3's warmup is 375M tokens, why is the warmup_iters not set to 375/0.5 = 750 but 2000? Is there any further consideration?

to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

Labels

No labels

No labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests