Tokenizer Bug: Floats with leading 0's wrong tokenized

[hsjobeki]

Describe the bug

As described in the Title, nix tokenizes floats with leading zeros differently.
The first part of the float is interpreted as an integer if it starts with a zero.

IEEE doesn't specify this, but usually, I would expect the implementation to take care of leading zeros. and interpret the number as just one number

• double checked for similar issues.

Steps To Reproduce

nix repl

nix-repl> [ 01.1e2 ]
[ 1 10 ]

Which comes due to a fault in the tokenizer

01.1e2

Is tokenized into

TOKEN_INTEGER, "01"
TOKEN_FLOAT, ".1e2"

Expected behavior

01.1e2

This should produce only one token

TOKEN_FLOAT, "01.1e2"

Priorities

Add 👍 to issues you find important.

[roberth]

This is technically a breaking change, similar to

• Support hexadecimal and octal syntax for integers #7578.

I'd be open to running an experiment where we intentionally and explicitly remove the offending syntax to experimentally verify that users don't actually make use of it.
However, on that issue the Nix team leaned towards implementing language versioning instead.