The custom part of this stack is the Prometheus exporter code which is developed separately here. The rest of the stack is a standard Prometheus container and a standard Grafana container. The additional configuration files in this repo will help to configure both Prometheus and Grafana so that it can be used out of the box.

Requirements

Docker host (I like using Ubuntu with docker.io and docker-compose installed)
docker-compose
internet connectivity (only for downloading the containers)
After initial deployment you can run this solution offline as long as it has access to the ZVM appliances that you want to get data from.

Deployment

git clone http://www.github.com/recklessop/zroc.git

Configuration

Edit docker-compose.yml, and provide values for the following variables for the zvmexporter container.

ZVM IP or hostname
ZVM client id (generate this in keycloak)
ZVM client secret (also generated in keycloak)
VCenter ip or hostname
VCenter username
VCenter password

  zertoexporter:
    image: recklessop/zerto-exporter:latest
    command: python python-node-exporter.py
    ports:
      - "9999:9999" # edit the port for each additional exporter, in this case it was changed to 9998
    volumes:
      - ./zvmexporter/:/usr/src/logs/
    environment:
      - VERIFY_SSL=False
      - ZVM_HOST=192.168.40.60 # ZVM specific info
      - ZVM_PORT=443
      - SCRAPE_SPEED=20 #how often should the exporter scrape the Zerto API in seconds
      - CLIENT_ID=api-script # ZVM specific info
      - CLIENT_SECRET=js51tDM8oappYUGRJBhF7bcsedNoHA5j # ZVM specific info
      - LOGLEVEL=DEBUG
      - VCENTER_HOST=192.168.40.50 # vcenter specific info
      - VCENTER_USER=administrator@vsphere.local # vcenter specific info
      - VCENTER_PASSWORD=password # vcenter specific info
    networks:
      - back-tier
    restart: always

Multi-ZVM Configuration (Optional)

If you want to monitor more than one ZVM/vCenter you can add additional zvmexporter containers.

For each site you want to monitor you need to have an exporter configured like the lines below. By default the docker-compose.yaml has two exporters for a production and DR site, but you can add more sites to achieve a single pain of glass for monitoring all the ZVM's that are reachable from your docker host.

  zertoexporter:
    container_name: zvmexporter1
    hostname: zvmexporter1 # this hostname will need to be set in the prometheus.yaml file as well
    image: recklessop/zerto-exporter:latest
    command: python python-node-exporter.py
    ports:
      - "9999:9999"
    volumes:
      - ./zvmexporter/:/usr/src/app/logs/
    environment:
      # Site 1 configuration settings
      - VERIFY_SSL=False
      - ZVM_HOST=192.168.50.60
      - ZVM_PORT=443
      - SCRAPE_SPEED=20 #how often should the exporter scrape the Zerto API
      - CLIENT_ID=api-script
      - CLIENT_SECRET=js51tDM8oappYUGRJBhF7bcsedNoHA5j
      - LOGLEVEL=DEBUG
      - VCENTER_HOST=192.168.50.50
      - VCENTER_USER=administrator@vsphere.local
      - VCENTER_PASSWORD=password
    networks:
      - back-tier
    restart: always

Next, modify the prometheus configuration and add additional targets for each scrape job. Notice the hostname is all that has been updated to reflect the 2nd exporter. (the hostname is whatever you set in the docker-compose.yaml file.

 global:
  scrape_interval:     60s
  evaluation_interval: 60s 
  scrape_timeout: 60s

scrape_configs:
  - job_name: 'vm-stats'
    scrape_interval: 30s
    scrape_timeout: 20s
    static_configs:
         - targets: ['zvmexporter1:9999']
         #- targets: ['zvmexporter2:9999'] # only needed if you have two ZVMs to monitor

  - job_name: 'encryption-stats'
    scrape_interval: 30s
    scrape_timeout: 20s
    metrics_path: /statsmetrics
    static_configs:
         - targets: ['zvmexporter1:9999']
         #- targets: ['zvmexporter2:9999'] # only needed if you have two ZVMs to monitor


  - job_name: 'thread-stats'
    scrape_interval: 30s
    scrape_timeout: 20s
    metrics_path: /threads
    static_configs:
         - targets: ['zvmexporter1:9999']
         #- targets: ['zvmexporter2:9999'] # only needed if you have two ZVMs to monitor

  - job_name: 'vra-stats'
    scrape_interval: 30s
    scrape_timeout: 20s
    metrics_path: /vrametrics
    static_configs:
         - targets: ['zvmexporter1:9999']
         #- targets: ['zvmexporter2:9999'] # only needed if you have two ZVMs to monitor

The default dashboards provisioned with this stack will show stats for all sites in the graphs.

If you want to create site specific dashboards, you can clone the existing dashboards, and then edit them to include filters for "siteName" labels.

Running

docker-compose up -d

Accessing Grafana

http://<IP_Address_of_Docker_Host>:3000

There will be several dashboards provisioned out of the box that will help monitor most metrics. Custom graphs and custom dashboards can be added too.

Troubleshooting

To make sure that the exporter is able to collect data you can browse to http://dockerip:9999 or :9998 if you have a second site and look in the logs directory for the exporter logs. You can also verify that the various metric files are being created.

Eye Candy

Here are screenshots from the dashboards which ship with zROC.

Main Zerto Metrics Dashboard

Every NOC needs some eye candy right? This dashboard is great for a heads-up display to see which VPGs and VMs may be falling behind in replication.

Protected VMs

This dashboard shows you how much journal storage each protected VM is using as well as how many journal disks are provisioned for each protected VM. This information is important as you approach the VRA maximums.

VRA Metrics

The goal of this dashboard is to help you understand the workload on each of your VRAs. How many outgoing VMs and disks, how many incoming VMs and disks, as well as simple things like how many vCPU and vRAM are assigned to each. (great to see if any VRAs are configured differently that the others.) It also shows you the total number of recovery volumes attached to a VRA (at recovery sites) so you can tell when your VRA is approaching the maximum number of protected volumes)

Encryption Detection

This feature is in private preview. If you would like to learn more contact your Zerto account team. With Zerto Encryption Detection enabled, Zerto looks at each write that is being protected to determine if it is encrypted data or unencrypted data. The goal is to help customers detect anomalies caused by ransomware attacks in near real time.

Languages

JavaScript 74.9%

Python 13.1%

Shell 5.1%

HCL 2.5%

CSS 1.9%

Other 2.5%