Skip to content

Commit

Permalink
Grib1 loading defines a GRIB_PARAM attribute, like grib2 load. (#402)
Browse files Browse the repository at this point in the history 
* Grib1 loading defines a GRIB_PARAM attribute, like grib2 load.

* Shorten line.

* Update cube-load result snapshots.

* fix mock usage.

* Various documentation improvements.

* Further docs rework : doctests now working.

* Line-length fix.

* Review changes.

* GRIBcode to properly support GRIB1 and GRIB2 usage.

* Add repr test

* Update with comments

* Small notes.

* Revert to shorter names with no digits; add repr implementation

* Don't pep8 grib_phenom_translation as it now uses dataclasses.

* Fixes for rename.

* Fix result CMLs.

* Add grib1 testing.

* Replaced GRIBCode class with factory function.

* Move GRIBCode code into a separate submodule.

* Tidy and fix error case handling, and testing.

* Small docs fix, and whatsnews

* Fix doctests: don't use iris.tests

* Small review changes.

* Standardise optional type + ensure support for Python<3.10

* Review change.

---------

Co-authored-by: Martin Yeo <[email protected]>
  • Loading branch information
@pp-mo @trexfeathers
pp-mo and trexfeathers authored Apr 18, 2024
1 parent 590418b commit 569d5a1
Show file tree
Hide file tree
Showing 17 changed files with 643 additions and 196 deletions.
2 changes: 1 addition & 1 deletion docs/conf.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -296,5 +296,5 @@
# Example configuration for intersphinx: refer to the Python standard library.
intersphinx_mapping = {
"python": ("https://docs.python.org/", None),
"iris": ("https://scitools-iris.readthedocs.io/en/latest/", None),
"iris": ("https://scitools-iris.readthedocs.io/en/stable/", None),
}
242 changes: 183 additions & 59 deletions docs/index.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,92 +3,210 @@
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Iris-grib v0.19
===============
Iris-grib v0.20 (unreleased)
============================

The library ``iris-grib`` provides functionality for converting between weather and
climate datasets that are stored as GRIB files and :class:`Iris cubes <iris.cube.Cube>`.
climate datasets that are stored as GRIB files and Iris :class:`~iris.cube.Cube`\s.
GRIB files can be loaded as Iris cubes using ``iris-grib`` so that you can use Iris
for analysing and visualising the contents of the GRIB files. Iris cubes can be saved to
GRIB files using ``iris-grib``.
for analysing and visualising the contents of the GRIB files. Iris cubes can also be
saved to GRIB edition-2 files using ``iris-grib``.


Simple GRIB Loading and Saving with Iris
----------------------------------------
You can use the functionality provided by ``iris-grib`` directly within Iris
without having to explicitly import ``iris-grib``, as long as you have both Iris
and ``iris-grib`` installed in your Python environment.

**This is the preferred route if no special control is required.**

.. testsetup::

import iris
import iris_grib
import warnings
warnings.simplefilter('ignore')
cube = iris.load_cube(iris.sample_data_path("rotated_pole.nc"))
iris.save(cube, 'testfile.grib', saver='grib2')

For example, to load GRIB data :

>>> cube = iris.load_cube('testfile.grib')

Similarly, you can save cubes to a GRIB file directly from Iris :

>>> iris.save(cube, 'my_file.grib2')

.. note::
As the filename suggests, **only saving to GRIB2 is currently supported**.


Phenomenon translation
----------------------
``iris-grib`` attempts to translate between CF phenomenon identities
(i.e. 'standard_name' and possibly 'long_name' attributes), and GRIB parameter codes,
when converting cubes to or from the GRIB format.

A set of tables define known CF translations for GRIB1 and GRIB2 parameters, and can be
interrogated with the functions in :mod:`iris_grib.grib_phenom_translation`.


Parameter loading record
^^^^^^^^^^^^^^^^^^^^^^^^
All cubes loaded from GRIB have a ``GRIB_PARAM`` attribute, which records the parameter
encodings present in the original file message.

Examples :

* ``"GRIB2:d000c003n005"`` represents GRIB2, discipline=0 ("Meteorological products"),
category=3 ("Mass") and indicatorOfParameter=5 ("Geopotential height (gpm)").

The contents of ``iris-grib`` represent the former grib loading and saving capabilities
of :mod:`Iris <iris>` itself. These capabilities have been separated into a discrete library
so that Iris becomes less monolithic as a library.
* This translates to a standard_name and units of "geopotential_height / m"

* ``"GRIB1:t002c007n033"`` is GRIB1 with table2Version=2, centre=7
("US National Weather Service - NCEP (WMC)"), and indicatorOfParameter=33
("U-component of wind m s**-1").

Loading
-------
* This translates to a standard_name and units of "x_wind / m s-1".

To use ``iris-grib`` to load existing GRIB files we can make use of the
:func:`~iris_grib.load_cubes` function::
Parameter saving control
^^^^^^^^^^^^^^^^^^^^^^^^
When a cube has a ``GRIB_PARAM`` attribute, as described above, this controls what the
relevant message keys are set to on saving.
(N.B. at present applies only to GRIB2, since we don't support GRIB1 saving)

>>> import os
>>> import iris_sample_data
>>> import iris_grib
>>> cubes = iris_grib.load_cubes(os.path.join(iris_sample_data.path,
'polar_stereo.grib2'))
>>> print cubes
<generator object load_cubes at 0x7f69aba69d70>

Iris-grib Load and Save API
---------------------------
In addition to direct load and save with Iris, as described above,
it is also possible to load and save GRIB data using iris-grib functions.

Loading and saving Cubes
^^^^^^^^^^^^^^^^^^^^^^^^
Load
~~~~
To load from a GRIB file with ``iris-grib``, you can call the
:func:`~iris_grib.load_cubes` function :

>>> cubes_iter = iris_grib.load_cubes('testfile.grib')
>>> print(cubes_iter)
<generator object load_cubes at ...>

As we can see, this returns a generator object. The generator object may be iterated
over to access all the Iris cubes loaded from the GRIB file, or converted directly
to a list::

>>> cubes = list(cubes)
>>> print cubes
>>> cubes = list(cubes_iter)
>>> print(cubes)
[<iris 'Cube' of air_temperature / (K) (projection_y_coordinate: 200; projection_x_coordinate: 247)>]

.. note::
There is no functionality in iris-grib that directly replicates
``iris.load_cube`` (that is, load a single cube directly rather than returning
a length-one `CubeList`. Instead you could use the following, assuming that the
GRIB file you have loaded contains data that can be loaded to a single cube::
In effect, this is the same as using ``iris.load_raw(...)``.
So, in most cases, **that is preferable.**

>>> cube, = list(cubes)
>>> print cube
air_temperature / (K) (projection_y_coordinate: 200; projection_x_coordinate: 247)
Dimension coordinates:
projection_y_coordinate x -
projection_x_coordinate - x
Scalar coordinates:
forecast_period: 6 hours
forecast_reference_time: 2013-05-20 00:00:00
pressure: 101500.0 Pa
time: 2013-05-20 06:00:00
Save
~~~~
To use ``iris-grib`` to save Iris cubes to a GRIB file we can make use of the
:func:`~iris_grib.save_grib2` function :

This makes use of an idiom known as variable unpacking.
>>> iris_grib.save_grib2(cube, 'my_file.grib2')

In effect, this is the same as using ``iris.save(cube, ...)``.
So, in most cases, **that is preferable.**

Saving
------

To use ``iris-grib`` to save Iris cubes to a GRIB file we can make use of the
:func:`~iris_grib.save_grib2` function::
Working with GRIB messages
^^^^^^^^^^^^^^^^^^^^^^^^^^
Iris-grib also provides lower-level functions which allow the user to inspect and
adjust actual GRIB encoding details, for precise custom control of loading and saving.

These functions use intermediate objects which represent individual GRIB file
"messages", with all the GRIB metadata.

For example:

>>> iris_grib.save_grib2(my_cube, 'my_file.grib2')
* correct loading of some messages with incorrectly encoded parameter number
* save messages with adjusted parameter encodings
* load messages with an unsupported parameter definition template : adjust them to
mimic a similar type which *is* supported by cube translation, and post-modify the
resulting cubes to correct the Iris metadata

You can load and save messages to and from files, and convert them to and from Cubes.

.. note::
As the function name suggests, only saving to GRIB2 is supported.
at present this only works with GRIB2 data.

.. note::
Messages are not represented in the same way for loading and saving : the messages
generated by loading *from* files are represented by
:class:`iris_grib.message.GribMessage` objects, whereas messages generated from
cubes, for saving *to* files, are represented as message handles from the
`Python eccodes library <https://confluence.ecmwf.int/display/ECC/Python+3+interface+for+ecCodes>`_ .

Interconnectivity with Iris
---------------------------
Load
~~~~
The key functions are :func:`~iris_grib.load_pairs_from_fields` and
:func:`~iris_grib.message.GribMessage.messages_from_filename`.
See those for more detail.

You can use the functionality provided by ``iris-grib`` directly within Iris
without having to explicitly import ``iris-grib``, as long as you have both Iris
and ``iris-grib`` available to your Python interpreter.
You can load data to 'messages', and filter or modify them to enable or correct
how Iris converts them to 'raw' cubes (i.e. individual 2-dimensional fields).

For example:

>>> from iris_grib.message import GribMessage
>>> fields_iter = GribMessage.messages_from_filename('testfile.grib')
>>> # select only wanted data
>>> selected_fields = [
... field
... for field in fields_iter
... if field.sections[4]['parameterNumber'] == 33
... ]
>>> cube_field_pairs = iris_grib.load_pairs_from_fields(selected_fields)

For example::
Filtering fields can be very useful to speed up loading, since otherwise all data must
be converted to Iris *before* selection with constraints, which can be quite costly.

>>> import iris
>>> import iris_sample_data
>>> cube = iris.load_cube(iris.sample_data_path('polar_stereo.grib2'))

Similarly, you can save your cubes to a GRIB file directly from Iris
using ``iris-grib``::
Save
~~~~
The key functions are :func:`~iris_grib.save_pairs_from_cubes` and
:func:`~iris_grib.save_messages`.
See those for more detail.

>>> iris.save(my_cube, 'my_file.grib2')
You can convert Iris cubes to eccodes messages, and modify or filter them before saving.

.. note::
The messages here are eccodes message "ids", essentially integers, and *not*
:class:`~iris_grib.message.GribMessages`. Thus, they must be inspected and
manipulated using the eccodes library functions.

.. testsetup::

from iris.coords import DimCoord
import eccodes
cube_height_2m5 = iris.load_cube(iris.sample_data_path("rotated_pole.nc"))
cube_height_2m5.add_aux_coord(DimCoord([2.5], standard_name="height", units="m"), ())

For example:

>>> # translate data to grib2 fields
>>> cube_field_pairs = list(iris_grib.save_pairs_from_cube(cube_height_2m5))
>>> # adjust some of them
>>> for cube, field in cube_field_pairs:
... if cube.coords('height') and cube.coord('height').points[0] == 2.5:
... # we know this will have been rounded, badly, so needs re-scaling.
... assert eccodes.codes_get_long(field, 'scaleFactorOfFirstFixedSurface') == 0
... assert eccodes.codes_get_long(field, 'scaledValueOfFirstFixedSurface') == 2
... eccodes.codes_set_long(field, 'scaleFactorOfFirstFixedSurface', 1)
... eccodes.codes_set_long(field, 'scaledValueOfFirstFixedSurface', 25)
...
>>> # save to file
>>> messages = [msg for (cube, msg) in cube_field_pairs]
>>> iris_grib.save_messages(messages, 'temp.grib2')
>>> # check result
>>> print(iris.load_cube('temp.grib2').coord('height').points)
[2.5]


Getting Started
Expand All @@ -99,11 +217,17 @@ To ensure all ``iris-grib`` dependencies, it is sufficient to have installed
`ecCodes <https://software.ecmwf.int/wiki/display/ECC/ecCodes+Home>`_ .

The simplest way to install is with
`conda <https://conda.io/miniconda.html>`_ ,
using the `conda-forge channel <https://anaconda.org/conda-forge>`_ ,
`conda <https://conda.io/miniconda.html>`_ , using the
`package on conda-forge <https://anaconda.org/conda-forge/iris-grib>`_ ,
with the command

$ conda install -c conda-forge iris-grib

Pip can also be used, to install from the
`package on PyPI <https://pypi.org/project/iris-grib/>`_ ,
with the command

$ conda install -c conda-forge iris-grib
$ pip install iris-grib

Development sources are hosted at `<https://github.com/SciTools/iris-grib>`_ .

Expand Down
13 changes: 13 additions & 0 deletions docs/ref/release_notes.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,19 @@ Features
4.6, i.e. percentile forecasts.
`(PR#401) <https://github.com/SciTools/iris-grib/pull/401>`_

* `@pp-mo <https://github.com/pp-mo>`_ expanded the use of the "GRIB_PARAM"
attributes to GRIB1 loading, and document it more thoroughly.
`(ISSUE#330) <https://github.com/SciTools/iris-grib/issues/330>`_,
`(PR#402) <https://github.com/SciTools/iris-grib/pull/402>`_

Documentation
^^^^^^^^^^^^^
* `@pp-mo <https://github.com/pp-mo>`_ reworked the main docs page to :
headline basic load + save with Iris, rather than lower-level functions;
better explain load-pairs and save-pairs usage; make all usage examples into
doctests.
`(ISSUE#398) <https://github.com/SciTools/iris-grib/issues/398>`_

Dependencies
^^^^^^^^^^^^
* `@bjlittle <https://github.com/bjlittle>`_ migrated to ``pytest``.
Expand Down
25 changes: 19 additions & 6 deletions iris_grib/__init__.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -304,6 +304,7 @@ def _compute_extra_keys(self):
"_x_points": None,
"_y_points": None,
"_cf_data": None,
"_grib_code": None,
}

# cf phenomenon translation
Expand All @@ -317,6 +318,14 @@ def _compute_extra_keys(self):
)
self.extra_keys["_cf_data"] = cf_data

# Record the original parameter encoding
self.extra_keys["_grib_code"] = gptx.GRIBCode(
edition=1,
table_version=self.table2Version,
centre_number=centre_number,
number=self.indicatorOfParameter,
)

# reference date
self.extra_keys["_referenceDateTime"] = datetime.datetime(
int(self.year),
Expand Down Expand Up @@ -708,7 +717,7 @@ def _load_generate(filename):

def load_cubes(filenames, callback=None):
"""
Returns a generator of cubes from the given list of filenames.
Returns an iterator over cubes from the given list of filenames.
Args:
Expand All @@ -721,7 +730,7 @@ def load_cubes(filenames, callback=None):
Function which can be passed on to :func:`iris.io.run_callback`.
Returns:
A generator containing Iris cubes loaded from the GRIB files.
An iterator returning Iris cubes loaded from the GRIB files.
"""
import iris.fileformats.rules as iris_rules
Expand Down Expand Up @@ -809,17 +818,21 @@ def save_grib2(cube, target, append=False):

def save_pairs_from_cube(cube):
"""
Convert one or more cubes to (2D cube, GRIB message) pairs.
Returns an iterable of tuples each consisting of one 2D cube and
one GRIB message ID, the result of the 2D cube being processed by the GRIB
save rules.
Convert one or more cubes to (2D cube, GRIB-message-id) pairs.
Produces pairs of 2D cubes and GRIB messages, the result of the 2D cube
being processed by the GRIB save rules.
Args:
* cube:
A :class:`iris.cube.Cube`, :class:`iris.cube.CubeList` or
list of cubes.
Returns:
a iterator returning (cube, field) pairs, where each ``cube`` is a 2d
slice of the input and each``field`` is an eccodes message "id".
N.B. the message "id"s are integer handles.
"""
x_coords = cube.coords(axis="x", dim_coords=True)
y_coords = cube.coords(axis="y", dim_coords=True)
Expand Down
Loading

0 comments on commit 569d5a1

Please sign in to comment.