Expose CORSIKA photons in the SimulatedEventContainer

[vmharvey]

Please describe the use case that requires this feature.
I'm doing some studies that use the CORSIKA photon bunches from ctapipe, also with the IACTEXT compilation option enabled to store particle emitter information. In sim_telarray I enable the SAVE_PHOTONS option so that the MC photon bunches and emitter info are passed through to the simtel file. I can access them for each telescope event with source.file_.current_photons[tel_id - 1] and source.file_.current_emitter[tel_id - 1], but this feels very hackish.

Describe the solution you'd like
I would like them to be treated the same as other simulation properties and put under the SimulatedCameraContainer so they show up at say, event.simulation.tel[tel_id].shower_photons and event.simulation.tel[tel_id].shower_emitter.

Describe alternatives you've considered
As I said, I have a workaround but top-level support would be nice.

---

I'm happy to take on this issue and create a PR, if you want to support this use case.

[kosack]

Can you explain more about what data type is stored in event.simulation.tel[tel_id].shower_photons and shower_emitter? E.g. are they both lists of something? Does the length of the list change per event?

The reason I ask is that Containers are used to store information in the HDF5 output files (and read it back). However, they can only serialize fixed-length information, i.e. either a single scalar or a fixed-length n-dimensional array. If the size of the array changes per event, the Container will not work without adding a suitable transform that converts from variable to fixed-length. Think of a container field as representing one row of a column in a table, which is in fact what happens when they are written to disk.

[maxnoe]

Can you explain more about what data type is stored in event.simulation.tel[tel_id].shower_photons and shower_emitter? E.g. are they both lists of something? Does the length of the list change per event?

It's a structured numpy array, essentially the table of Cherenkov photons in the event (shower_photons) and their parent particle (shower_emitter).

So yes, this is variable length.

[vmharvey]

I see... Having access to the photon tables from the HDF5 output file isn't my use case though. I haven't made use of the HDF5 output format before, so I didn't know that the Containers mapped directly into it. For high energy events there could be tens or even hundreds of thousands of photon bunches, which would blow up the size of HDF5 files for any data simulated with the SAVE_PHOTONS setting turned on (it is off by default). That's probably undesirable :p

But still, it would be nice to have a transparent way to access the photon data in a ctapipe script like I showed, so that the telescope ID is already re-numbered and we can use event rather than source.file_... Maybe a type of Field in the SimulatedCameraContainer that is excluded from the serialisation? Then it wouldn't matter that it stores a numpy array that's variable length and possibly very large.