Prometheus metrics¶
Anbox Cloud gathers various performance metrics that you can access through API endpoints to create a monitoring solution. The following sections list all metrics returned to Prometheus by the Anbox Cloud services that support Prometheus metrics.
AMS¶
The Anbox Management Service provides metrics about the Anbox Cloud cluster (or the Anbox Cloud Application server) and the AMS API access.
AMS metrics endpoint¶
You can access the AMS metrics from any machine that is on the same network as your Anbox Cloud installation. Use the following endpoint:
http://<AMS_server>:<AMS_port>/internal/1.0/metrics
Replace <AMS_server>
with the IP address of your AMS server, which you can determine by running one of the following commands:
For a full Anbox Cloud deployment:
juju run --wait=5m --unit ams/0 -- unit-get private-address
For the Anbox Cloud Appliance:
juju run --wait=5m -m appliance:anbox-cloud --unit ams/0 -- unit-get private-address
Replace <AMS_port>
with the port for the API endpoint, which you can determine by running one of the following commands:
For a full Anbox Cloud deployment:
juju config ams prometheus_target_port
For the Anbox Cloud Appliance:
juju config -m appliance:anbox-cloud ams prometheus_target_port
You can then access the endpoint with curl
, for example:
curl http://192.0.2.55:20002/internal/1.0/metrics
AMS cluster metrics¶
Metrics prefixed with ams_cluster_
keep you informed about the status of your Anbox Cloud cluster.
Name |
Description |
Status |
---|---|---|
|
Number of nodes in the cluster |
Available since 1.0.0 |
|
Number of applications |
Available since 1.0.0 |
|
Number of containers currently in the cluster |
Deprecated since 1.20.0, use |
|
Number of container boot time measured |
Deprecated since 1.20.0, use |
|
Sum of all container boot times and can be used to compute the average boot time |
Deprecated since 1.20.0, use |
|
Container boot times bucket and can be used for alerting when above a threshold; see the Prometheus documentation for more information |
Deprecated since 1.20.0, use |
|
Number of containers per application |
Deprecated since 1.20.0, use |
|
Number of containers per container status |
Deprecated since 1.20.0, use |
|
Number of containers per worker node |
Deprecated since 1.20.0, use |
|
Number of instances currently in the cluster |
Available since 1.20.0 |
|
Number of instance boot time measured |
Available since 1.20.0 |
|
Sum of all instance boot times (can be used to compute the average boot time) |
Available since 1.20.0 |
|
Instance boot times bucket (can be used for alerting when above a threshold; see the Prometheus documentation for more information) |
Available since 1.20.0 |
|
Number of instances per application |
Available since 1.20.0 |
|
Number of instances per instance status |
Available since 1.20.0 |
|
Number of instances per worker node |
Available since 1.20.0 |
|
Total CPUs available in each worker node |
Available since 1.0.0 |
|
Used CPUs in each worker node |
Available since 1.0.0 |
|
Total memory available in each worker node |
Available since 1.0.0 |
|
Used memory in each worker node |
Available since 1.0.0 |
AMS API metrics¶
Metrics prefixed with ams_http_
allow to track access to the API.
These metrics are available since Anbox Cloud 1.10.0.
Name |
Description |
---|---|
|
Number of HTTP requests being processed at the moment |
|
The HTTP request latency in seconds |
|
The HTTP request size in bytes |
|
Total number of HTTP requests made |
|
The HTTP response sizes in bytes |
API handlers¶
To give a more granular approach to monitoring, the AMS API metrics contain handlers that identify the kind of API access.
The AMS API can be accessed through HTTP/HTTPS or a Unix domain socket. Therefore, the API metrics distinguish between http
, https
and unix
. For example:
ams_http_request_duration_seconds_bucket{handler="http_applications_GET",host="juju-2db13b-1",method="get",le="0.5"} 1
ams_http_request_duration_seconds_bucket{handler="https_applications_GET",host="juju-2db13b-1",method="get",le="0.5"} 1
ams_http_request_duration_seconds_bucket{handler="unix_applications_GET",host="juju-2db13b-1",method="get",le="0.5"} 1
All handler labels adopt the convention <transport method>_<object>_<http method>
, for example, unix_containers_POST
.
The following table contains all routes and their corresponding labels (ignoring the communication method prefix).
Method |
Route |
Label |
---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Anbox Stream Gateway¶
The Anbox Stream Gateway provides metrics about the streaming activities of your cluster or server and the Anbox Stream Gateway API access.
Access endpoint¶
You can access the Anbox Stream Gateway metrics from any machine that is on the same network as your Anbox Cloud installation. Use the following endpoint:
https://<gateway_server>:<gateway_port>/internal/1.0/metrics
Replace <gateway_server>
with the IP address of your Anbox Stream Gateway server, which you can determine by running one of the following commands:
For a full Anbox Cloud deployment:
juju run --unit anbox-stream-gateway/0 --wait=5m -- unit-get private-address
For the Anbox Cloud Appliance:
juju run --wait=5m -m appliance:anbox-cloud --unit anbox-stream-gateway/0 -- unit-get private-address
Replace <gateway_port>
with the port for the API endpoint, which you can determine by running one of the following commands:
For a full Anbox Cloud deployment:
juju config anbox-stream-gateway prometheus_port
For the Anbox Cloud Appliance:
juju config -m appliance:anbox-cloud anbox-stream-gateway prometheus_port
The Anbox Stream Gateway endpoint is on HTTPS, and therefore you must authenticate to access it. You can retrieve the credentials from the /var/snap/anbox-stream-gateway/common/service/config.yaml
file on the gateway server:
For a full Anbox Cloud deployment:
juju ssh anbox-stream-gateway/0 sudo cat /var/snap/anbox-stream-gateway/common/service/config.yaml
For the Anbox Cloud Appliance:
juju ssh -m appliance:anbox-cloud anbox-stream-gateway/0 sudo cat /var/snap/anbox-stream-gateway/common/service/config.yaml
You can then access the endpoint with curl
, for example:
curl -k -u prometheusadmin:thepassword https://192.0.2.55:9105/internal/1.0/metrics
Metrics¶
Metrics prefixed with anbox_stream_gateway_
give information about your cluster related to streaming, for example, the number of sessions and agents.
These metrics are available since Anbox Cloud 1.7.2.
Name |
Description |
---|---|
|
Total number of sessions, categorised by status |
|
Total number of accounts |
|
Number of active and unresponsive agents |
API metrics¶
Metrics prefixed with anbox_stream_gateway_http_
allow to track access to the streaming API.
These metrics are available since Anbox Cloud 1.9.0.
Name |
Description |
---|---|
|
Number of HTTP requests being processed at the moment |
|
The HTTP request latency in seconds |
|
The HTTP request size in bytes |
|
Total number of HTTP requests made |
|
The HTTP response sizes in bytes |
API handlers¶
To give a more granular approach to monitoring, the Anbox Stream Gateway API metrics contain handlers that identify the route that is accessed. The routes are indicated in the handler labels. For example:
anbox_stream_gateway_http_request_duration_seconds_bucket{handler="get_sessions",host="juju-2db13b-1",method="get",le="0.5"} 1
In this case, the label for the route is get_sessions
.
The following table contains all routes and their corresponding labels.
Method |
Route |
Label |
---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
LXD¶
LXD provides metrics about the LXD cluster that Anbox Cloud uses.
LXD metrics have been available since LXD version 4.19. You cannot access LXD metrics if you are running an older version of LXD.
LXD metrics endpoint¶
You can access the LXD metrics through the following endpoint:
https://<LXD_server>:8443/1.0/metrics
Replace <LXD_server>
with the IP address of your LXD server, which you can determine by running one of the following commands:
For a full Anbox Cloud deployment:
juju run --wait=5m --unit lxd/0 -- unit-get private-address
For the Anbox Cloud Appliance:
juju run --wait=5m -m appliance:anbox-cloud --unit lxd/0 -- unit-get private-address
The LXD metrics endpoint is on HTTPS, and therefore you must authenticate to access it. See Create metrics certificate in the LXD documentation for instructions on how to create a certificate.
Alternatively, if you are using the Anbox Cloud Appliance, you can also access the LXD metrics through the local Unix socket. In this case, you don’t need authentication. To use this method, enter the following command:
curl --unix-socket /var/snap/lxd/common/lxd/unix.socket s/1.0/metrics
LXD metrics¶
You can find the list of metrics that LXD provides in the LXD documentation.
WebRTC¶
The WebRTC metrics are collected by Telegraf on every cluster member.
WebRTC metrics endpoint¶
You can access the WebRTC metrics as part of the metrics that Telegraf collects and provides to Prometheus. You can access them from any machine that is on the same network as your Anbox Cloud installation. To find the endpoint, check the /var/snap/anbox-cloud-appliance/common/telegraf/main.conf
file on each cluster member and look for the listen
address and the path
in the [[outputs.prometheus_client]]
section.
You can then access the endpoint with curl
, for example:
curl http://192.0.2.1:9001/metrics
WebRTC metrics¶
Metrics prefixed with webrtc_
give you detailed insight about the WebRTC protocol for every streaming instance. See the official W3C reference for more information.
These metrics are available since Anbox Cloud 1.8.0.
Name |
Description |
---|---|
|
Total number of frames successfully encoded |
|
Total number of key frames, such as key frames in VP8 or IDR-frames in H.264 ( |
|
Total number of seconds that has been spent encoding the |
|
The current encoder target in bits per second |
|
Total number of bytes sent for a specific SSRC (a SSRC represents one resource - video, audio or binary data - sent over a WebRTC track) |
|
Total number of bytes that were re-transmitted for a specific SSRC, only including payload bytes |
|
Total number of packets that were re-transmitted for a specific SSRC |
|
Total number of seconds that packets have spent buffered locally before being transmitted onto the network |
|
Total number of RTP packets sent for this SSRC (includes re-transmissions) |
|
Total number of Negative Acknowledgement (NACK) packets received by this sender |
|
Total number of Full Intra Request (FIR) packets received by this sender (video only) |
|
Total number of Picture Loss Indication (PLI) packets received by this sender (video only) |
|
Total number of Slice Loss Indication (SLI) packets received by this sender (video only) |
|
Boolean value indicating if a relay ICE candidate type is in use for the stream |