Support Hexadecimal, Octal and Binary syntax

[shikanime]

Motivation

Add support for integer literal syntax such as

• 0bXXXX for binary;
• 0xXXXX for hexadecimal;
• 0oXXXX for octal;

Context

         🌸＞ ' ' フ
         | . _  _l     < I'm just a regular person on the internet who doesn't have a
        ／` ミ＿xノ         lot of expertise in the Nix source code, but I saw a post about
       / .     |            someone wanting this certain feature and thought I'd chime in! 🌸
      /   ヽ . ﾉ
     │ . |  |  |
 ／￣| .  |  |  |
 | (￣ヽ＿_ヽ_)__)
 ＼二つ

fixes #7578

I don't know where to write the tests for the parser/lexer, feel free to comment on this!

Checklist for maintainers

Maintainers: tick if completed or explain if not relevant

• agreed on idea
• agreed on implementation strategy
• tests, as appropriate
  • functional tests - tests/**.sh
  • unit tests - src/*/tests
  • integration tests
• documentation in the manual
• code and comments are self-explanatory
• commit message explains why the change was made
• new feature or bug fix: updated release notes

[sivizius]

I suggest to change INT to [1-9][0-9]* and OCT to 0[0-7]*, so octal numbers start with 0 which is IMHO more common. On the other hand, this would break padding with leading zeros, but this looks better with spaces anyway IMHO. This would make 0 an octal 0, but it is still zero. The letter o is quite similar to 0, so 0o77 and 0077 look very similar and one might expect them to be both octal values, so at least add 0[0-7]+ as an alternative to 0o[0-7]+.

[roberth]

I don't think we should break 010 == 10, as Nix is meant to be a very stable language. It does seem like a good idea to warn about zero-padded integers, as the existence of octal support makes them quite ambiguous.
Aesthetics are of secondary importance, and explicit is good.

[sivizius]

We should definitely deprecate 010 == 10, as numbers starting with 0 are usually read as octals for historic reasons. Perhaps a warning like »Deprecated syntax: If you intended to zero-pad a decimal value, consider to use spaces instead. If you meant to write an octal value, write 0o<value>. Values starting with 0 are ambiguous.«.

Another issue: Binary values tend to be quite long, so like in python, allowing _ to group e.g. bytes might be beneficial for readability, for decimals (e.g. multiple of 1000) or hexadecimals (e.g. words/dwords) and perhaps for floating point literals as well.

[roberth]

I think we can potentially unblock this PR by running an experiment:

• Add a deprecation warning to the syntax we'd like to steal ([ <number><identifier> ] - a list with at least these two consecutive items, without a separating space)
• The warning refers to the issue, and asks users to report back if the change is a problem for them.

I think this should be discussed with the Nix team.
Question for the team: are we ok with running a deprecation experiment like this?

Click for possible future steps (not necessary to discuss yet; overly detailed)

Follow-up steps:

2. Change this PR to put the syntax change behind a feature flag

   • If the feature flag is enabled, use the new syntax, as simple as that
   • If the feature flag is not enabled, use the old syntax, but print a deprecation warning such as:
     Future ambiguous syntax found in file.nix:123, 0x100 We plan to reinterpret this syntax as a hexadecimal number in an upcoming release. We've estimated that this character sequence was highly unlikely to occur in practice, so if you would like for this file to be reproducible in future versions of Nix, please report back to us at .

3. Wait for reports. 1 year?

4. If nothing is reported, just steal the syntax. If something is reported, see if we can work around the issue by checking whether such a variable is in scope. Otherwise, this is blocked by a language version declaration feature, or we'd have to reconsider our backward compatibility policy

[fricklerhandwerk]

Triaged in the Nix team meeting:

• @thufschmitt: technically this is a breaking change
  • incidentally @yorickvP tried to address a very similar problem and stopped because of that, then started to work on an RFC to deal with breaking changes
  • @Ericson2314: if we had proper language versioning it wouldn't be so much of an issue
  • there would be some use cases for this syntax, seen this multiple times
• @edolstra: this is nice to have, but not sure if there is a huge demand for non-decimal notation
  • @fricklerhandwerk: depends on the audience you ask
• @Ericson2314: intermediate goals (figuring out the breaking change story, and gaining experience as a team) are probably more valuable than the feature itself
• agreement; postponed to after we deal with the overarching issue of breaking changes

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-02-24-nix-team-meeting-minutes-35/25757/1

[nyabinary]

Triaged in the Nix team meeting:

* @thufschmitt: technically this is a breaking change
  
  * incidentally @yorickvP tried to address a very similar problem and stopped because of that, then started to work on an RFC to deal with breaking changes
  * @Ericson2314: if we had proper language versioning it wouldn't be so much of an issue
  * there would be some use cases for this syntax, seen this multiple times

* @edolstra: this is nice to have, but not sure if there is a huge demand for non-decimal notation
  
  * @fricklerhandwerk: depends on the audience you ask

* @Ericson2314: intermediate goals (figuring out the breaking change story, and gaining experience as a team) are probably more valuable than the feature itself

* agreement; postponed to after we deal with the overarching issue of breaking changes

Any update about this?