跳到主要内容

运维监控

此文档主要介绍 TuGraph 的可视化运维监控

1.设计思路​

可视化监控并不是TuGraph自身不可或缺的一部分，因此在设计时将可视化监控作为TuGraph周边生态中的一个应用，来减少和TuGraph数据库的耦合度，以及对于TuGraph自身的影响。TuGraph可视化监控采用目前最火热的开源解决方案，TuGraph Monitor + Prometheus + Grafana来实现。其中TuGraph Monitor作为TuGraph服务的客户端，通过TCP链接向TuGraph服务发起Procedure请求，TuGraph服务在接收到请求后收集自身所在机器的cpu，memory，disk，io，以及请求数量等指标的统计结果进行响应。TuGraph Monitor在接收到TuGraph响应的指标数据后，将数据包装成prometheus需要的格式，保存在内存中，等待Prometheus服务通过http请求获取。Prometheus服务会定期通过http请求从TuGraph Monitor获取封装好的请求数据，按照获取的时间保存在自己的时序数据库中。Grafana可以根据用户的配置，从Prometheus处获取某个时间段内的统计数据，并在web界面上绘制浅显易懂的图形来展示最终结果。整个请求链路中，都采用了主动获取，即PULL的模型，好处之一是它能最大限度的避免数据生产者和数据消费者之间的耦合度，使得开发更简单，好处之二是数据生产者不需要考虑数据消费者的数据处理能力，即使某个消费者的数据处理能力较弱，也不会因为生产者生产数据过快而压垮消费者。主动拉取模型的不足之处在于数据的实时性不够，但在这个场景中，数据并没有很高的实时性要求。

1.1.TuGraph​

TuGraph数据库提供了收集服务所在机器中磁盘，内存，网络IO，以及查询请求等多种数据信息的能力，并通过标准Procedure方式提供查询。收集数据这一动作仅在有用户通过接口查询时才会发生，避免了在用户不需要TuGraph监控服务所在机器的指标时对用户业务查询请求带来的影响。

1.2.TuGraph Monitor​

TuGraph Monitor是TuGraph周边生态中的一个工具，它作为TuGraph众多用户中的一个，通过C++ RPC Client与TuGraph进行通信，通过Procedure查询接口来查询TuGraph服务所在机器的性能指标，并将TuGraph返回的结果包装成Prometheus需要的数据模型，等待Prometheus获取。用户可以通过设置查询时间间隔来保证获取监控指标对于业务查询的影响最小化。

1.3.Prometheus​

Prometheus是一个开源的监控平台，并配备有专属的时序数据库，它会定期通过http请求从TuGraph Monitor服务获取统计指标，并保存在自己的时序数据库中。详细信息请参考官网: https://prometheus.io/docs/introduction/first_steps

1.4.Grafana​

Grafana是一个开源的可视化和分析软件，它可以从包含Prometheus在内的多个数据源中获取数据，并且可以将时序数据库中的数据转换为精美图形和可视化效果的工具。具体信息请参考官网: https://grafana.com/docs/grafana/v7.5/getting-started/

2.部署方案​

2.1.第一步​

启动TuGraph服务，详细方法请参考文档: https://github.com/TuGraph-db/tugraph-db/blob/master/doc/zh-CN/1.guide/3.quick-start.md

2.2.第二步​

启动TuGraph Monitor工具，启动命令如下：

./lgraph_monitor --server_host 127.0.0.1:9091 -u admin -p your_password \
			--monitor_host 127.0.0.1:9999  --sampling_interval_ms 1000

参数含义如下

Available command line options:
    --server_host       Host on which the tugraph rpc server runs.
                        Default=127.0.0.1:9091.
    -u, --user          DB username.
    -p, --password      DB password.
    --monitor_host      Host on which the monitor restful server runs.
                        Default=127.0.0.1:9999.
    --sampling_interval_ms
                        sampling interval in millisecond. Default=1.5e2.
    -h, --help          Print this help message. Default=0.

2.3.第三步​

tar -zxvf prometheus-2.37.5.linux-amd64.tar.gz
  • 修改配置文件prometheus.yml，新增如下配置，使其可以抓取TuGraph Monitor包装好的性能数据
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "tugraph"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["localhost:9111"]
  • 启动prometheus，具体的启动参数可以通过如下命令获取
./prometheus -h
  • 验证prometheus服务是否正常，可以通过web端登陆prometheus服务，查询监控指标resources_report是否已经获取到，能成功查询到数据则正确

2.4.第四步​

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": {
          "type": "grafana"
        },
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "target": {
          "limit": 100,
          "matchAny": false,
          "tags": [],
          "type": "dashboard"
        },
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 0,
  "id": 2,
  "links": [],
  "liveNow": false,
  "panels": [
    {
      "datasource": {
        "type": "prometheus"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            }
          },
          "mappings": [],
          "unit": "kbytes"
        },
        "overrides": [
          {
            "matcher": {
              "id": "byName",
              "options": "D {instance=\"localhost:7010\", job=\"TuGraph\", resouces_type=\"memory\", type=\"available\"}"
            },
            "properties": [
              {
                "id": "displayName",
                "value": "others"
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "D {__name__=\"resources_report\", instance=\"localhost:7010\", job=\"TuGraph\", resouces_type=\"memory\", type=\"available\"}"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "light-green",
                  "mode": "fixed"
                }
              },
              {
                "id": "displayName",
                "value": "others"
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "others"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "light-blue",
                  "mode": "fixed"
                }
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "graph_used"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "light-orange",
                  "mode": "fixed"
                }
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 16,
        "w": 6,
        "x": 0,
        "y": 0
      },
      "id": 14,
      "options": {
        "displayLabels": [
          "name",
          "value"
        ],
        "legend": {
          "displayMode": "table",
          "placement": "bottom",
          "values": [
            "percent",
            "value"
          ]
        },
        "pieType": "pie",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "tooltip": {
          "mode": "single",
          "sort": "none"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus"
          },
          "editorMode": "code",
          "expr": "resources_report{instance=\"localhost:7010\",job=\"TuGraph\",resouces_type=\"memory\",type=\"self\"}",
          "legendFormat": "{ {type} }",
          "range": true,
          "refId": "A"
        },
        {
          "datasource": {
            "type": "prometheus"
          },
          "editorMode": "code",
          "expr": "resources_report{instance=\"localhost:7010\",job=\"TuGraph\",resouces_type=\"memory\",type=\"available\"}",
          "hide": false,
          "legendFormat": "{ {type} }",
          "range": true,
          "refId": "B"
        },
        {
          "datasource": {
            "type": "prometheus"
          },
          "editorMode": "code",
          "expr": "resources_report{instance=\"localhost:7010\",job=\"TuGraph\",resouces_type=\"memory\",type=\"total\"}",
          "hide": true,
          "legendFormat": "{ {label_name} }",
          "range": true,
          "refId": "C"
        },
        {
          "datasource": {
            "type": "__expr__"
          },
          "expression": "$C -$A - $B",
          "hide": false,
          "refId": "D",
          "type": "math"
        }
      ],
      "title": "内存",
      "type": "piechart"
    },
    {
      "alert": {
        "alertRuleTags": {},
        "conditions": [
          {
            "evaluator": {
              "params": [
                1000
              ],
              "type": "gt"
            },
            "operator": {
              "type": "and"
            },
            "query": {
              "params": [
                "A",
                "5m",
                "now"
              ]
            },
            "reducer": {
              "params": [],
              "type": "avg"
            },
            "type": "query"
          }
        ],
        "executionErrorState": "alerting",
        "for": "5m",
        "frequency": "1m",
        "handler": 1,
        "message": "【生产图数据库Grafana】\n  QPS超过1000",
        "name": "请求统计 alert",
        "noDataState": "no_data",
        "notifications": []
      },
      "datasource": {
        "type": "prometheus"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 7,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "smooth",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": " "
        },
        "overrides": [
          {
            "matcher": {
              "id": "byName",
              "options": "write"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "light-blue",
                  "mode": "fixed"
                }
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 16,
        "w": 12,
        "x": 6,
        "y": 0
      },
      "id": 4,
      "options": {
        "legend": {
          "calcs": [
            "min",
            "max",
            "mean",
            "last"
          ],
          "displayMode": "table",
          "placement": "bottom"
        },
        "tooltip": {
          "mode": "single",
          "sort": "none"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus"
          },
          "editorMode": "code",
          "expr": "{instance=\"localhost:7010\",job=\"TuGraph\",resouces_type=\"request\",type=~\"total|write\"}",
          "legendFormat": "{ {type} }",
          "range": true,
          "refId": "A"
        }
      ],
      "thresholds": [
        {
          "colorMode": "critical",
          "op": "gt",
          "value": 1000,
          "visible": true
        }
      ],
      "title": "请求统计",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            }
          },
          "mappings": [],
          "unit": "decbits"
        },
        "overrides": [
          {
            "matcher": {
              "id": "byName",
              "options": "graph_used"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "light-red",
                  "mode": "fixed"
                }
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "available"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "light-orange",
                  "mode": "fixed"
                }
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "D"
            },
            "properties": [
              {
                "id": "displayName",
                "value": "other"
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "other"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "light-purple",
                  "mode": "fixed"
                }
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 16,
        "w": 6,
        "x": 18,
        "y": 0
      },
      "id": 12,
      "options": {
        "displayLabels": [
          "name",
          "value"
        ],
        "legend": {
          "displayMode": "table",
          "placement": "bottom",
          "sortBy": "Value",
          "sortDesc": true,
          "values": [
            "value",
            "percent"
          ]
        },
        "pieType": "pie",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "tooltip": {
          "mode": "single",
          "sort": "none"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus"
          },
          "editorMode": "code",
          "exemplar": false,
          "expr": "resources_report{instance=\"localhost:7010\",job=\"TuGraph\",resouces_type=\"disk\",type=\"available\"}",
          "format": "time_series",
          "instant": false,
          "interval": "",
          "legendFormat": "{ {type} }",
          "range": true,
          "refId": "A"
        },
        {
          "datasource": {
            "type": "prometheus"
          },
          "editorMode": "code",
          "expr": "resources_report{instance=\"localhost:7010\",job=\"TuGraph\",resouces_type=\"disk\",type=\"self\"}",
          "hide": false,
          "legendFormat": "{ {type} }",
          "range": true,
          "refId": "B"
        },
        {
          "datasource": {
            "type": "prometheus"
          },
          "editorMode": "code",
          "expr": "resources_report{instance=\"localhost:7010\",job=\"TuGraph\",resouces_type=\"disk\",type=\"total\"}",
          "hide": true,
          "legendFormat": "{ {type} }",
          "range": true,
          "refId": "C"
        },
        {
          "datasource": {
            "type": "__expr__"
          },
          "expression": "$C - $A - $B",
          "hide": false,
          "refId": "D",
          "type": "math"
        }
      ],
      "title": "磁盘",
      "transformations": [
        {
          "id": "configFromData",
          "options": {
            "applyTo": {
              "id": "byFrameRefID"
            },
            "configRefId": "config",
            "mappings": []
          }
        }
      ],
      "type": "piechart"
    },
    {
      "alert": {
        "alertRuleTags": {},
        "conditions": [
          {
            "evaluator": {
              "params": [
                90
              ],
              "type": "gt"
            },
            "operator": {
              "type": "and"
            },
            "query": {
              "params": [
                "A",
                "5m",
                "now"
              ]
            },
            "reducer": {
              "params": [],
              "type": "avg"
            },
            "type": "query"
          }
        ],
        "executionErrorState": "alerting",
        "for": "5m",
        "frequency": "1m",
        "handler": 1,
        "message": "【生产图数据库Grafana】\nCPU使用率超过90%",
        "name": "CPU使用率 alert",
        "noDataState": "no_data",
        "notifications": [
          {
          }
        ]
      },
      "datasource": {
        "type": "prometheus"
      },
      "description": "",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 4,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "percent"
        },
        "overrides": [
          {
            "matcher": {
              "id": "byName",
              "options": "graph_used"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "light-orange",
                  "mode": "fixed"
                }
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "total_used"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "light-purple",
                  "mode": "fixed"
                }
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "self"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "light-green",
                  "mode": "fixed"
                }
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "total"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "light-purple",
                  "mode": "fixed"
                }
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 14,
        "w": 12,
        "x": 0,
        "y": 16
      },
      "id": 6,
      "options": {
        "legend": {
          "calcs": [
            "min",
            "max",
            "mean",
            "last"
          ],
          "displayMode": "table",
          "placement": "bottom"
        },
        "tooltip": {
          "mode": "single",
          "sort": "none"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus"
          },
          "editorMode": "code",
          "expr": "resources_report{instance=\"localhost:7010\",job=\"TuGraph\",resouces_type=\"cpu\",type=~\"total|self\"}",
          "hide": false,
          "legendFormat": "{ {type} }",
          "range": true,
          "refId": "A"
        }
      ],
      "thresholds": [
        {
          "colorMode": "critical",
          "op": "gt",
          "value": 90,
          "visible": true
        }
      ],
      "title": "CPU使用率",
      "type": "timeseries"
    },
    {
      "alert": {
        "alertRuleTags": {},
        "conditions": [
          {
            "evaluator": {
              "params": [
                10000
              ],
              "type": "gt"
            },
            "operator": {
              "type": "and"
            },
            "query": {
              "params": [
                "A",
                "5m",
                "now"
              ]
            },
            "reducer": {
              "params": [],
              "type": "avg"
            },
            "type": "query"
          }
        ],
        "executionErrorState": "alerting",
        "for": "5m",
        "frequency": "1m",
        "handler": 1,
        "message": "【生产图数据库Grafana】\n  磁盘IO超过10MB/S",
        "name": "磁盘IO alert",
        "noDataState": "no_data",
        "notifications": []
      },
      "datasource": {
        "type": "prometheus"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 7,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "smooth",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "bps"
        },
        "overrides": [
          {
            "matcher": {
              "id": "byName",
              "options": "read"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "super-light-green",
                  "mode": "fixed"
                }
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "write"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "super-light-red",
                  "mode": "fixed"
                }
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 14,
        "w": 12,
        "x": 12,
        "y": 16
      },
      "id": 2,
      "options": {
        "legend": {
          "calcs": [
            "min",
            "max",
            "mean",
            "last"
          ],
          "displayMode": "table",
          "placement": "bottom"
        },
        "tooltip": {
          "mode": "single",
          "sort": "none"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus"
          },
          "editorMode": "builder",
          "expr": "resources_report{instance=\"localhost:7010\",job=\"TuGraph\",resouces_type=\"disk_rate\",type=~\"read|write\"}",
          "hide": false,
          "legendFormat": "{ {type} }",
          "range": true,
          "refId": "A"
        }
      ],
      "thresholds": [
        {
          "colorMode": "critical",
          "op": "gt",
          "value": 10000,
          "visible": true
        }
      ],
      "title": "磁盘IO",
      "type": "timeseries"
    }
  ],
  "refresh": "",
  "schemaVersion": 36,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": []
  },
  "time": {
    "from": "now-24h",
    "to": "now"
  },
  "timepicker": {
    "hidden": false,
    "refresh_intervals": [
      "10s"
    ]
  },
  "timezone": "",
  "title": "TuGraph监控页面",
  "version": 20,
  "weekStart": ""
}

验证效果，刷新浏览器页面。如果正确显示饼图和折线图，则配置完成。

3.未来计划​

目前可视化监控只支持单机监控，能监控服务所在机器的cpu，磁盘，网络io，请求qps等性能指标，未来将会实现监控ha集群的功能，也会将更多有意义的指标纳入监控范围