Question: Help with adding a "State" metric

[FulcrumTeam]

Hello Amitasha!

I hope it is okay that I am opening an issue here to ask for some help. We are using your example to create a healthcheck for a web login process. I have posted a question to stackoverflow around adding an Enum metric with a "looping" selenium webscraper script here

Would you by chance have the time to lend some advice on how to approach this? Modifying your example flask app here has been the easy part, it is the communication between the webscraper status and the middleware that is the difficult part.

Any advice would be great, thanks for creating this project!

[amitsaha]

Why not use a gauge metric like, loging_healthcheck{state="starting"}, loging_healthcheck{state="stopped"} and so on? Then from the selenium script update that metric using the prometheus client methods?

[FulcrumTeam]

Thank you for the prompt reply Amit.

We have tried to update the metric from the selenium script using HEALTHCHECK_STATE.state('running'), HEALTHCHECK_STATE.state('stopped') but the metric never changed even though we know that code was being called. We are not Python experts by any means, so it is confusing how the 2 scripts (selenium loop and middleware) "share" the HEALTHCHECK_STATE object? Maybe this is a multi-process issue? Note the selenium script runs in a subprocess.

Appreciate the help!

[amitsaha]

Ah, good point about the different processes, that explains why you do not see it working.

Is your Flask server only for exporting the metrics? If so and even if not, I can suggest you to use https://github.com/prometheus/statsd_exporter in the following fashion:

• In your selenium test push a statsd metric to the locally running statsd exporter
• In prometheus, scrape the statsd exporter instead of your flask application

See https://rollout.io/blog/monitoring-your-synchronous-python-web-applications-using-prometheus/ for learning more about the statsd exporter approach.

[FulcrumTeam]

We switched over to using the statsd approach and have better results with that. We actually decided to switch to using a counter metric and will use a query to determine the state in grafana. The last issue we are facing seems to be a docker one..

We're researching it now, but perhaps you have some experience. Our flask app now has an endpoint, "raise_count" that is called from the selenium script when there is an exception. Problem seems to be in the Docker networking.

Selenium script calls the webhook using this:

r = requests.get(
            'http://localhost:5000/raise_count',
            verify=False)

docker-compose.yml (simplified):

webapp:
    expose:
      - 5000
    ports:
        - '5000:5000'
    command: python /application/sct_forgerock_login_app.py
  selenium-healthcheck:
    command: python /application/sct_selenium_login.py
  stastd:
    expose:
      - 9125
      - 9102

But we are getting a "connection refused" error.

[amitsaha]

We're researching it now, but perhaps you have some experience. Our flask app now has an endpoint, "raise_count" that is called from the selenium script when there is an exception.

Why not push the statsd metric directly from selenium script?