Add monitor mode, allowing a PHPA to be run without actually scaling anything to help with tuning

[akunafin]

Is there a way to run this solution in a kind of a monitor mode? I want to see what are the predictions going out from the operator, but not apply it. This will allow to tune it smoothly without affecting any real applications. In general I want to build the graphs like you have in the examples to see how good my smoothing parameters work before I'll start applying to the real deployment.
I was also looking for any metrics that I can scrape from the operator to monitor it via prometheus. Is there any endpoint for it?

[jthomperoo]

Hi @akunafin, there isn't currently any way to run this in monitor mode, but that's a good suggestion - I'll update this issue to track that.

The PHPA operator exposes standard metrics about the application at :8080/metrics for scraping with Prometheus, but there's two issues I can see with this:

1. This port isn't actually exposed from the container, and the Helm chart doesn't include a Service for this endpoint so I don't think it's properly set up for Prometheus yet. Adding this wouldn't be very difficult.

2. The metrics exposed are just standard application metrics, such as memory and CPU usage. I don't think these are metrics that will be particularly useful to you. We could add in some custom metrics to expose here, would there be any metrics in particular you'd want exposed? Perhaps the replica counts of the resources being managed?

[akunafin]

Hi @jthomperoo, thanks for your reply!
In general I would like to see the difference between the replica count calculated by the pure HPA and by the prediction mechanism. In examples I've seen a graph that shows how effective is the prediction:

So to build the same graph I need 2 metrics: HPA calculated replica count and predicted replica count.
So having PHPA running in monitor mode means that it doesn't affect real replica count, but exposes a metric with the value that it would apply as a replica count. Then I can use this metric to monitor how effective prediction is in real time, and tune smoothing parameters.
I see that for every PHPA object there is a configMap created. In this config map there is a JSON array with the replica count history ( .modelHistories.HoltWintersPrediction.replicaHistory[] ). What are these values in the array? is it a predicted value that should be applied?