Retain file extension from input file instead of hardcoding .nc4

[frankinspace]

Currently, l2ss-py defaults the file extension of the subset file to .nc4

https://github.com/podaac/l2ss-py/blob/develop/podaac/subsetter/subset_harmony.py#L221

There is some question about if this is the appropriate file extension or if it should be .nc. Need to gather consensus about if this needs to change or not.

[frankinspace]

In terms of what is currently supported in l2ss-py, data is read using the NetCDF4DataStore backend for xarray which will produce NetCDF4 formatted data.

https://github.com/podaac/l2ss-py/blob/develop/podaac/subsetter/subset.py#L1282

This is also reflected in the SupportedOutputFormats in the UMM-S record for L2ss-py

https://github.com/podaac/l2ss-py/blob/develop/cmr/l2ss_cmr_umm_s.json#L39-L52

So, without further code modification, l2ss-py will only ever output NetCDF4 formatted data.

[frankinspace]

Quick google search came up with some context:

Filename extension
nc
The recommended file extension for netCDF-4, and the default supplied by the Unidata software library is .nc, the same as for netCDF-3. However, since the files are valid HDF5 files, the .h5 extension may be used in some contexts. See Why aren't different extensions used for the different [netCDF] formats, for example, .nc3 and .nc4?

https://www.loc.gov/preservation/digital/formats/fdd/fdd000332.shtml#sign

Unidata recommends using .nc but indicates it is just convention:

Why aren't different extensions used for the different formats, for example ".nc3" and ".nc4"?

The file extension used for netCDF files is purely a convention. The netCDF libraries don't use the file extension. A user can currently create a netCDF file with any extension, even one not consistent with the format of the file.

The ncgen utility uses ".nc" as a default extension for output, but this can be overridden using the "-o" option to specify the name for the output file. Recent versions of ncgen also have a "-k" option to specify what kind of output file is desired, selecting any of the 4 format variants, using either a numeric code or a text string. Most other netCDF client software pays no attention to the file extension, so using more explicit extensions by convention has no significant drawbacks, except possibly causing confusion about format differences that may not be important.

https://docs.unidata.ucar.edu/netcdf-c/current/faq.html#why-arent-different-extensions-used

Zarr library treats ['.nc','.nc4','.hdf','.h5'] all as the same:

zarr-developers/VirtualiZarr#142

[eni-awowale]

Hi @frankinspace!

Do you think it would be possible to have the file extension be the extension of the original file? Some of our collections at GES_DISC associated to l2ss-py are .HDF5. It doesn't seem like the .nc4 extension is currently breaking anything for those collections but it might be a good idea to have parity with the original granule.

I tested this in UAT with the GPM_2APR_07 collection. The granule has a .HDF5 extension. When I do a subset request through harmony it strips .HDF5 for a .nc4. The subset appears to be the same as when I do a local subset request with l2ss-py and directly specify the extension. I think it might be worth keeping it consistent with the original granule if this doesn't add too much work.

[frankinspace]

Hi @eni-awowale

Do you think it would be possible to have the file extension be the extension of the original file? Some of our collections at GES_DISC associated to l2ss-py are .HDF5. It doesn't seem like the .nc4 extension is currently breaking anything for those collections but it might be a good idea to have parity with the original granule.

That makes sense to me. I think the supported output formats are clear that the output is netcdf4 regardless of what file extension is used. Matching the input file extension seems reasonable.

[amscott-obdaac]

The solution of keeping the original file extension of the submitted file also works for OB.DAAC.