Harbor官方在Harbor v2.2开始提供对相关指标的采集和使用,即 Harbor v2.2 及更高版本中支持Prometheus监控Harbor。
1. 部署harbor
下载版本2.7.1
wget -c https://github.com/goharbor/harbor/releases/download/v2.7.1/harbor-offline-installer-v2.7.1.tgz
解压
tar -zxf harbor-online-installer-v2.7.1.tgz -C /opt/harbor
编辑 /opt/harbor/harbor.yml,将metric eneabled改为true
metric:
enabled: true
port: 9090
path: /metrics
执行安装并启动服务
./prepare
./install.sh --with-trivy --with-chartmuseum
2.Harbor暴露了哪些指标(metrics)
2.1 Harbor Exporter指标
exporter`组件指标与Harbor 实例配置相关,并从 Harbor 数据库中收集一些数据。
指标可在`<harbor_instance>:<metrics_port>/<metrics_path>`查看
① harbor_project_total
公共和私人项目总共数量
# curl 192.168.2.250:9090/metrics | grep harbor_project_total
harbor_project_total{public="false"} 1 # 私有项目有1个
harbor_project_total{public="true"} 1 # 公共项目数量为1
② harbor_project_repo_total
项目中的存储库总数
harbor_project_repo_total{project_name="library",public="true"} 0
harbor_project_repo_total{project_name="test-1",public="false"} 1
③ harbor_project_member_total
项目中的成员总数
harbor_project_member_total{project_name="library"} 1
harbor_project_member_total{project_name="test-1"} 2
④ harbor_project_quota_usage_byte
一个项目的总使用的存储空间
harbor_project_quota_usage_byte{project_name="library"} 0
harbor_project_quota_usage_byte{project_name="test-1"} 3.0347083e+07
⑤ harbor_project_quota_byte
项目中设置的配额
harbor_project_quota_byte{project_name="library"} -1
harbor_project_quota_byte{project_name="test-1"} -1
-1 表示不限制
⑥ harbor_artifact_pulled
项目中拉取的图像数量
harbor_artifact_pulled{project_name="library"} 0
harbor_artifact_pulled{project_name="test-1"} 1
library项目镜像下载次数为0 test-1项目下镜像下载次数为1 grafana仪表板设计:折线图
⑦ harbor_project_artifact_total
项目中的工件类型总数
artifact_type , project_name,public ( true, false)
harbor_project_artifact_total{artifact_type="IMAGE",project_name="test-1",public="false"} 1
⑧ harbor_health
Harbor状态
harbor_health 1 #Harbor在启动中
⑨ harbor_system_info
Harbor实例的信息
auth_mode (db_auth, ldap_auth, uaa_auth, http_auth, oidc_auth),harbor_version,self_registration( true, false)
harbor_system_info{auth_mode="db_auth",harbor_version="v2.3.1-1058f330",self_registration="false"} 1
⑩ harbor_up
Harbor组件运行状态,组件 ( chartmuseum, core, database,jobservice, portal, redis, registry, registryctl, trivy)
harbor_up{component="chartmuseum"} 1 # Chart插件状态
harbor_up{component="core"} 1
harbor_up{component="database"} 1
harbor_up{component="jobservice"} 1
harbor_up{component="portal"} 1
harbor_up{component="redis"} 1
harbor_up{component="registry"} 1
harbor_up{component="registryctl"} 1 # 镜像客户端运行状态
harbor_up{component="trivy"} 1 # Trivy扫描器运行状态
1为运行中,0为未运行
①① harbor_task_queue_size
队列中每种类型的任务总数
harbor_task_queue_size{type="DEMO"} 0
harbor_task_queue_size{type="GARBAGE_COLLECTION"} 0
harbor_task_queue_size{type="IMAGE_GC"} 0
harbor_task_queue_size{type="IMAGE_REPLICATE"} 0
harbor_task_queue_size{type="IMAGE_SCAN"} 0
harbor_task_queue_size{type="IMAGE_SCAN_ALL"} 0
harbor_task_queue_size{type="P2P_PREHEAT"} 0
harbor_task_queue_size{type="REPLICATION"} 0
harbor_task_queue_size{type="RETENTION"} 0
harbor_task_queue_size{type="SCHEDULER"} 0
harbor_task_queue_size{type="SLACK"} 0
harbor_task_queue_size{type="WEBHOOK"} 0
①② harbor_task_queue_latency
多久前要处理的下一个作业按类型排入队列
①③ harbor_task_scheduled_total
计划任务数
harbor_task_scheduled_total 0
①④ harbor_task_concurrency
Total上每种类型的并发任务总数
harbor_task_concurrency{pool="d4053262b74f0a7b83bc6add",type="GARBAGE_COLLECTION"}
0
2.2 Harbor核心指标
以下是从 Harbor 核心 pod 中提取的指标,可在
<harbor_instance>:<metrics_port>/<metrics_path>?comp=core.
# curl http://192.168.2.250:9090/metrics?comp=core.
harbor_core_http_inflight_requests
请求总数,操作(HarborAPI operationId中的值。一些遗留端点没有,因此标签值为)operationId“unknown
harbor_core_http_request_duration_seconds
请求的持续时间,方法 ( GET, POST, HEAD, PATCH, PUT), 操作 ( HarborAPIoperationId中的 值。一些遗留端点没有, 所以标签值为), 分位数operationId“unknown.
harbor_core_http_request_total
请求总数
方法(GET, POST, HEAD, PATCH, PUT),操作( Harbor
APIoperationId中的 值。一些遗留端点没有,因此标签值为)operationId“unknown
2.3 Registry Metrics(Registry组件指标)
注册表,以下是从 Docker 发行版中提取的指标,查看指标方式:
<harbor_instance>:<metrics_port>/<metrics_path>?comp=registry.
registry_http_in_flight_requests
进行中的 HTTP 请求,处理程序
registry_http_request_duration_seconds
HTTP 请求延迟(以秒为单位),处理程序、方法( ,,,, GET) POST,文件HEADPATCHPUT
registry_http_request_size_bytes
HTTP 请求大小(以字节为单位)。
2.4 Harbor Jobservice 指标
以下是从 Harbor Jobservice 提取的指标,查看:
<harbor_instance>:<metrics_port>/<metrics_path>?comp=jobservice.
harbor_jobservice_info
Jobservice的信息,
harbor_jobservice_task_total
每个作业类型处理的任务数
harbor_jobservice_task_process_time_seconds
任务处理时间的持续时间,即任务从开始执行到任务结束用了多少时间。
3.配置kube-prometheus抓取Harbor指标
编辑prometheus-additional.yaml,添加
- job_name: 'harbor-exporter'
scrape_interval: 20s
static_configs:
- targets: ['10.16.25.10:9090']
- targets: ['10.16.25.15:9090']
- job_name: 'harbor-core'
scrape_interval: 20s
params:
comp: ['core'] #从Harbor core组件中获取度量
static_configs:
- targets: ['10.16.25.10:9090']
- job_name: 'harbor-registry'
scrape_interval: 20s
params:
comp: ['registry'] #从Harbor registry组件中获取度量
static_configs:
- targets: ['10.16.25.10:9090']
- job_name: 'harbor-jobservice'
scrape_interval: 20s
params:
comp: ['jobservice'] # 从Harbor jobservice组件中获取度量
static_configs:
- targets: ['10.16.25.10:9090']
执行命令更新配置
kubectl delete secret additional-configs -n monitoring >/dev/null 2>&1kubectl create secret generic additional-configs --from-file=prometheus-additional.yaml -n monitoring
检查prometheus配置是否正常
##检查 prometheus-additional.yaml 是否有语法错误
kubectl exec -it prometheus-k8s-1 -n monitoring -- promtool check config /etc/prometheus/config_out/prometheus.env.yaml
重启kube-prometheus
4.在kube-prometheus中查看harbor Targets
5.grafana配置harbor监控图表
模板ID:14075
效果:
模板ID:16003
效果:
模板ID:16686
效果: