Hillas psi uncertainty ruo

[ruoyushang]

Adding estimation of image psi uncertainty and uncertainty of image cog along the rotated y-axis in hillas.py.
Also adding a test script to show the validation of psi uncertainty estimation using toy models.

[maxnoe]

This needs a better name. Also, for what is this additional uncertainty needed?

[ruoyushang]

The axis of the image is modeled with the function y = ax + b. After rotating the image (so the axis is horizontal), the uncertainty of a = psi uncertainty. However, the axis uncertainty along the transverse direction should be the combination of the psi uncertainty and the b uncertainty, i.e. transverse uncertainty sigma_t = pow( pow(dsigma_psi,2) + pow(sigma_b,2) , 0.5), where d is the longitudinal coordinate of the test point.

[ruoyushang]

How about "transverse_cog_uncertainty" ?

[kosack]

@ruoyushang to get the CI to work, you must add a file called docs/changes/2629.feature.rst with a human-readable description of the change that will go in the final changelog. After that we can merge this.

[ruoyushang]

@ruoyushang to get the CI to work, you must add a file called docs/changes/2629.feature.rst with a human-readable description of the change that will go in the final changelog. After that we can merge this.

@kosack Thank you for the information! I just added the file with a description. However, I also see this error when I pushed the commit:
CI / lint (pull_request) Failing after 25s. --> Is this critical?

[Tobychev]

@ruoyushang to get the CI to work, you must add a file called docs/changes/2629.feature.rst with a human-readable description of the change that will go in the final changelog. After that we can merge this.

@kosack Thank you for the information! I just added the file with a description. However, I also see this error when I pushed the commit: CI / lint (pull_request) Failing after 25s. --> Is this critical?

I guess you haven't set up the pre-commit hooks properly, typically they would catch that type of errors lint error before you even push the commit.

As for critical, well... It's very easy to fix so there's no reason to skip that check.

[ruoyushang]

@Tobychev Thank you for the information. Can you please show me how to find this bug? How do I use this pre-commit hook? I did pre-commit install following this link (https://ctapipe.readthedocs.io/en/latest/developer-guide/getting-started.html), but I don't know what to do next.

[maxnoe]

You can run it manually now to fix the past commits:

git fetch
pre-commit run --files $(git diff --name-only origin/main)

It will make changes, add and commit them. Some checks might also require manual changes, edit the files accordingly and then also add and commit.

[ctao-dpps-sonarqube]

• 6.56% Duplicated Lines (%) on New Code (is greater than 3.00%)

Analysis Details

0 Issues

• 0 Bugs
• 0 Vulnerabilities
• 0 Code Smells

Coverage and Duplications

• 100.00% Coverage (94.10% Estimated after merge)
• 6.56% Duplicated Code (0.80% Estimated after merge)

Project ID: cta-observatory_ctapipe_AY52EYhuvuGcMFidNyUs

View in SonarQube

[ruoyushang]

You can run it manually now to fix the past commits:

git fetch
pre-commit run --files $(git diff --name-only origin/main)

It will make changes, add and commit them. Some checks might also require manual changes, edit the files accordingly and then also add and commit.

Hi @maxnoe , thank you for the feedback. I was able to fix the error. Now there is one remaining failed check that complains about duplicated lines: 6.56% Duplicated Lines (%) on New Code (is greater than 3.00%)

Is this something that requires fixing, or we can proceed to merging?

[Tobychev]

I have a hard time understanding how "rotated centroid y coordinate uncertainty" and "uncertainty on the center of gravity along the transverse axis of the image" can mean the same thing, and I think the definition you give in the changelog is much clearer.

[Tobychev]

Hi @maxnoe , thank you for the feedback. I was able to fix the error. Now there is one remaining failed check that complains about duplicated lines: 6.56% Duplicated Lines (%) on New Code (is greater than 3.00%)

Is this something that requires fixing, or we can proceed to merging?

@ruoyushang Looking at what Sonarqube complains about, its actually an old sin that your four additional lines caused to be judged anew by the CI. It also looks a bit annoying to fix elegantly because the two different return Containers have different argument names.

So you could probably resolve the issue by stuffing all the arguments into a list and feed it into CameraHillasParametersContainer or CameraHillasParametersContainer depending on the condition, but it'll not be as pretty.

[maxnoe]

@Tobychev We can just ignore this duplication for now, we anyway want to remove the CameraFrame part and that wil then get rid of the duplication.

[maxnoe]

More critical here is to do some performance testing: how do these new computations affect the processing time?

[ruoyushang]

More critical here is to do some performance testing: how do these new computations affect the processing time?

Hi @Tobychev @maxnoe , thank you for the feedback! Do you have a suggestion on how the performance test should be done? Can I use time.perf_counter() and compare the difference in processing time?

[ruoyushang]

Hi @Tobychev @maxnoe ,

I calculated the time needed for the uncertainty estimation with the following code:

tic_psi_unc = time.perf_counter()
W = np.diag(image / pix_area)
X = np.column_stack([longi, np.ones_like(longi)])
lsq_cov = np.linalg.inv(X.T @ W @ X)
p = lsq_cov @ X.T @ W @ trans
psi_uncert = np.sqrt(lsq_cov[0, 0] + p[0] * p[0]) * (
1.0 + pow(np.tan(width / length * np.pi / 2.0), 2)
)
transverse_cog_uncert = np.sqrt(
lsq_cov[1, 1] * (1.0 + np.sin(p[0]) * np.sin(p[0]))
)
toc_psi_unc = time.perf_counter()
print(f"image size = {size:0.1f}, time spent: {toc_psi_unc-tic_psi_unc:0.6f} sec")

Here are several images and the time spent for the uncertainty calculation:

image size = 551.4, time spent: 0.000064 sec
image size = 500.3, time spent: 0.000051 sec
image size = 138.6, time spent: 0.000049 sec
image size = 2811.4, time spent: 0.000051 sec
image size = 2788.4, time spent: 0.000051 sec
image size = 2774.8, time spent: 0.000051 sec
image size = 45.7, time spent: 0.004733 sec