Build a browsable site

[strogonoff]

The plan as I understand it:

• Each document has one repository elsewhere in this organization.
• Each document has an associated stage/status. It is captured in Metanorma metadata (I assume).
• Each document is built on GHA. Upon successful build completion, the deliverables (including XML, vanilla Metanorma HTML, etc.) of the document are committed into a single intermediate repository (this repository!) containing all CC documents.
• A website is built from the intermediate repository that has HTML for all documents and all stages for each document, and an index allowing to navigate between documents and stages.

Requirements and associated notes:

• This repository must preserve document artifacts for all stages. That is, not 1 entry per document, but one entry per document-stage combination.

  Example publication process

  1. New document A is created and has stage DRAFT. After build, the intermediate repository has A-DRAFT.
  2. Document A is edited. After build, the intermediate repository has A-DRAFT overwritten with the edited version.
  3. Document A is edited and the stage was changed to REVIEW. After build, the intermediate repository still has A-DRAFT as before, and it is not overwritten. However, the intermediate repository now also has A-REVIEW.

• There is additional metadata desirable about things like when the stage was transitioned.

  • This may or should be captured in Metanorma metadata (I assume).

    • Checking with @ronaldtse — is it captured in Metanorma metadata? Or, does Metanorma support this information (in case we want to add it there automatically during build)?

  • This may be captured by treating the document dataset as a registry, where every document is an item.

    Registry implementation details

    • Register item class would define document identifier and its stage as item data, which combined would obtain a path to document stage artifact within repository/built site.
    • Whenever a document is updated, the item must be clarified to reflect the new stage. This would ensure that there is revision history for each document. This can be done automatically.
    • Other metadata such as document title, etc., can be part of the register, but note that it would cause duplication. All of this data is already contained in Metanorma metadata. Instead of this duplication, I believe it is better if build tools retrieve document metadata for a stage from Metanorma data.

    Downsides to using 19135-1 registry for this purpose:

    • The goal is not to manage document stages by hand. Rather, the opposite: the goal is to automate this away. Proposal procedure is not used. However, the register model presupposes the manual process of proposal review. For it to be a proper register a lot of data would have to be filled in by a script for no actual use, including: updating registry item YAML, adding a directory for the proposal, creating proposal YAML, and creating clarified item YAML.
    • The timeline information about when document changed stages may be obtained from Git repository.
      • In fact, Anafero/Firelight versioned builds integrate with Git: if given the necessary CLI flag it could, for example, build a version of the document for each tag. The links to document versions, as well as their last change timestamps, would be exposed in Firelight GUI, allowing the user to navigate. (Future features may include diffing/comparison document versions.)
      • The information can be obtained from Git repository and made part of Metanorma metadata, which website generation flow (whether Firelight-based or initially not) can use in the index page HTML.

cc @ribose-jeffreylau

[strogonoff]

Initial thoughts on how to get to where we want:

• This repository should keep document artifacts in directories named per [persistent document identifier]-[stage]. If there is a document identifier format that contains a stage already, that should be used.

  • Such directory naming would naturally resolve the issue of not overwriting previous stage artifacts.

• I think build process should create/update appropriate Git tags when new document stages appear.

  • Tags could use format like [docid]/stage-[x]. This is a normal practice in a monorepo that wants to version individual libraries with tags. See also: https://stackoverflow.com/a/56558343/247441

• The metadata about stage transition, if not provided already manually, can be captured by the build process.

  • It can be captured as a separate YAML sidecar file, but preferably—if supported—injected into Metanorma metadata.

• We can make use of that information (whether when generating registry data structure, if we go with that approach, or in Firelight, etc.)

• Firelight can generate HTML for every doc individually, but as of now it won’t generate an index page. At first, a very simple Jekyll site can take care of that, for the immediate fix. Working in a branch I can implement a build that uses Firelight and provides a nice index page.

• We do not need to worry about a race condition when two documents finish building at the same time. Since this cannot happen with the same document/stage I assume we simply cannot run into a merge conflict.

  • That said, this is possible:
    1. workflow for doc A pulls this repo commit X
    2. workflow for doc B pulls this repo commit X
    3. doc A commits Y and pushes, and succeeds
    4. doc B commits Z and pushes, and fails
  • In this situation, it should be easy to just reset and pull again. Once doc B has commit Y, it can successfully commit Z and push.