Commit f45ed88f authored by threedytech's avatar threedytech
Browse files

release: trk_spotlight-15448

parent 6acdfef1
################################################
Guide: running instant3Dhub with Grafana
################################################
********************************************
Read First
********************************************
This section explains how to enable Grafana for instant3Dhub. As Grafana uses AGPL it is not directly packaged with a regular installation.
Grafana serves as a tracing and debugging tool to diagnose functional or performance issues.
instant3Dhub configuration
--------------------------------------------
The Grafana image is not hosted on the Threedy image repository. The official docker image hosted on docker.io is used by default. In order to copy the image to a local registry, the following script can transfer the image to your repository via docker:
.. code-block:: bash
docker pull docker.io/grafana/grafana:9.1.6
docker image docker.io/grafana/grafana:9.1.6 ${TARGET_IMAGE_REGISTRY_HOST}/grafana/grafana:9.1.6
docker push ${TARGET_IMAGE_REGISTRY_HOST}/grafana/grafana:9.1.6
Grafana is disabled by default. The following options must be set in ``values.yaml`` to enable Grafana support:
.. code-block:: yaml
# Controls third party system tracing and logging.
tracing:
# Options for deploying services required for integrated tracing and metrics
# dashboards.
withIntegrated:
# Whether this option is enabled. This does not include Grafana. Metrics
# must be manually explored via the Prometheus GUI. Traces must be manually
# explored via the Jaeger GUI.
enabled: true
# Settings for deploying Grafana.
grafana:
# Whether grafana should be enabled. This also deploys dashboards detailing
# system performance regarding transcoding, service runtimes, error rates
# and memory usage. Additionally, metrics are linked to traces to allow
# detailed inspection of internals for given metrics.
enabled: true
# Which Grafana image to use. The image is not hosted on the Threedy registry
# as it is AGPL licensed.
image: docker.io/grafana/grafana:9.1.6
Or, alternatively, via Helm
.. code-block:: bash
# helm chart parameter
helm install --set tracing.withIntegrated.enabled=true --set tracing.withIntegrated.grafana.enabled=true ...
......@@ -52,7 +52,7 @@ We provide three channels for helm packages:
* **stable**: This channel contains the stable release packages of instant3Dhub.
* **dev**: This channel contains all rc and dev packages.
* **trk_<track-name>**: Each track package has its own channel and is named using the suffix trk_ followed by the name of the track.
* **trk_<track-name>**: Each track package has its own channel and is named using the suffix ``trk_`` followed by the name of the track.
Now, add the instant3Dhub Helm repository from one of the channels:
......@@ -171,7 +171,7 @@ Using External Services
PostgreSQL
-------------------------------------------
The system can optionally work with a previously provisioned PostgreSQL instance. This instance must be initialized corrctly with our schema and tables. For that we provide ansible playbooks to install and initialize postgres manually. For more information please `see our guide <./INSTALL_EXT_POSTGRES_GUIDE>`_.
The system can optionally work with a previously provisioned PostgreSQL instance. This instance must be initialized corrctly with our schema and tables. For that we provide ansible playbooks to install and initialize postgres manually. For more information please :ref:`see our guide <Guide: running instant3Dhub with external PostgreSQL>`.
If you are using external PostgresSQL, we recommend using k8s secrets to store the connection string. The secrets can be referenced then using these values in our helm chart.
......
......@@ -28,8 +28,8 @@ To following resources have to be provisioned to enable the cluster to run insta
K8s Requirements
============================================
* Kubernetes: v1.18
* kubectl: v1.18
* Kubernetes: v1.21
* kubectl: v1.21
* helm (CLI): v3.4.0
PostgreSQL Requirements
......
......@@ -55,7 +55,7 @@ Clone the release repository
The script expects these env variables to be set in the setup file:
* RELEASE_NAME: Is the name of the release to be used when deploying instant3Dhub. This can be any name of your choice. Example: instant3dhub-test. Default is: i3dhub-singlenode.
* ENTRYPOINT: The external exposure hosts or proxies. Example: http://host-name.your-domain.com:30000. Default is: http://your-hostname:30042. For more information on how to set up a proxy please see our `guide <./INSTALL_PROXY_GUIDE.srt>`_.
* ENTRYPOINT: The external exposure hosts or proxies. Example: http://host-name.your-domain.com:30000. Default is: http://your-hostname:30042. For more information on how to set up a proxy please see our :ref:`guide <Guide: running instant3Dhub with external PostgreSQL>`.
* REGISTRY: Is the container registry where instant3Dhub images can be fetched. Example: hub.docker.com. Default is: images.threedy.io.
* LICENSE_SERVER: The address of the license server. Example: http://license.threedy.io:8200. Default is empty.
* ENABLE_GPU: Set to true if your system has a GPU and you want to enable it to be used by instant3Dhub visualization services. Default is false.
......
......@@ -190,6 +190,9 @@ will be used.
.. code-block:: yaml
# whether to use the BIOS ID of the current machine when performing host checks
use-bios-uuid: false
# interface and port to listen on
address: 0.0.0.0:8200
......
......@@ -34,4 +34,5 @@ For further help or to give much appreciated feedback contact your personal cont
LICENSE_SERVER
INSTALL_PROXY_GUIDE
INSTALL_EXT_POSTGRES_GUIDE
INSTALL_GRAFANA
############################################
install tools: Ansible
############################################
Install Ansible
============================================
For simplicity we provide an example with CentOS 7.
Other Linux LSB distributions should be handled similarly.
We assume that we are on a clean install as root.
.. code-block:: bash
yum -y update
yum -y install epel-release
yum -y update
(update cache)
.. code-block:: bash
yum -y install ansible
ansible --version
should show ansible 2.9.10
later versions should be fine aswell
\ No newline at end of file
############################################
install tools: Ansible
############################################
Install Ansible
============================================
For simplicity we provide an example with CentOS 7.
Other Linux LSB distributions should be handled similarly.
We assume that we are on a clean install as root.
.. code-block:: bash
yum -y update
yum -y install epel-release
yum -y update
(update cache)
.. code-block:: bash
yum -y install ansible
ansible --version
should show ansible 2.9.10
later versions should be fine aswell
......@@ -34,7 +34,7 @@ instant3Dhub issue. Possible sources of this behavior:
* Volume mounts not setup correctly.
* Special capability nodes are all claimed by other pods.
Refer to :ref:`debugging steps below<kubectl>`, especially ``kubecl describe``
Refer to :ref:`debugging steps below <kubectl>`, especially ``kubecl describe``
to determine exact failure causes.
********************************************************************************
......@@ -56,9 +56,10 @@ these will be in a ``CrashLoopBackOff``:
Once these have started succesfully the rest of the containers should start.
If any containers are not starting, it is worth trying the
:ref:`debugging steps below<kubectl>` to determine the root cause.
:ref:`debugging steps below <kubectl>` to determine the root cause.
.. _kubectl:
********************************************************************************
``kubectl`` troubleshooting
********************************************************************************
......
......@@ -20,3 +20,5 @@
.idea/
*.tmproj
.vscode/
templates/grafana/*.sh
......@@ -57,7 +57,30 @@ app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- define "i3dhub.config.generators" -}}
/i3dhub-config/generators/
{{- end -}}
{{- define "i3dhub.config.prometheus" -}}
/etc/prometheus/
{{- end -}}
{{- define "i3dhub.config.grafana.provision.datasource" -}}
/etc/grafana/provisioning/datasources/
{{- end -}}
{{- define "i3dhub.config.grafana.provision.dashboard" -}}
/etc/grafana/provisioning/dashboards/
{{- end -}}
{{- define "i3dhub.config.grafana.dashboards" -}}
/etc/dashboards/instant3Dhub
{{- end -}}
{{- define "i3dhub.db.version.store" -}}
store:2
{{- end -}}
{{- define "i3dhub.tracing.jaeger.settings" -}}
- name: I3DH_OTEL_ENABLED
value: "true"
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: http://i3dhub-jaeger:4318
- name: OTEL_TRACES_SAMPLER
value: parentbased_traceidratio
- name: OTEL_TRACES_SAMPLER_ARG
value: "{{ .Values.tracing.withIntegrated.samplingProbability }}"
{{- end -}}
......@@ -62,12 +62,7 @@ spec:
{{- end }}
{{- if not .Values.resourceLimits.ignore }}
resources:
requests:
memory: "75Mi"
cpu: "0.05"
limits:
memory: "75Mi"
cpu: "0.1"
{{ .Values.resourceLimits.adminsvc | toYaml | indent 12 }}
{{- end }}
ports:
- containerPort: 8080
......@@ -93,11 +88,14 @@ spec:
name: {{ .Values.credentials.admin.secretKeyRef.name }}
key: {{ .Values.credentials.admin.secretKeyRef.key }}
{{- end }}
{{- if .Values.tracing.withJaeger }}
- name: I3DH_JAEGER_ELK_ENABLED
{{- if .Values.tracing.withKibana}}
- name: I3DH_ELK_ENABLED
value: "true"
{{- else if .Values.tracing.withDatadog }}
- name: I3DH_DATADOG_ENABLED
{{- end }}
{{- if .Values.tracing.withIntegrated.enabled }}
{{ include "i3dhub.tracing.jaeger.settings" . | indent 12 }}
{{- else if .Values.tracing.withOTel }}
- name: I3DH_OTEL_ENABLED
value: "true"
{{- end }}
{{- if .Values.tracing.additionalContainerEnv }}
......@@ -154,3 +152,18 @@ spec:
- name: http
protocol: TCP
port: 8080
---
apiVersion: v1
kind: Service
metadata:
name: i3dhub-adminsvc-headless
spec:
selector:
app: i3dhub-adminsvc
clusterIP: None
ports:
- name: metrics
port: 8080
targetPort: 8080
......@@ -52,8 +52,10 @@ spec:
secretKeyRef:
name: {{ .Values.credentials.rabbitmq.secretKeyRef.name }}
key: {{ .Values.credentials.rabbitmq.secretKeyRef.key }}
{{- if .Values.auth.signing.enabled }}
- name: KEYSTORE_ADDRESS
value: http://i3dhub-keystore:8080
{{- end }}
- name: CONSUL_ADDRESS
value: i3dhub-consul:8500
containers:
......@@ -67,12 +69,7 @@ spec:
{{- end }}
{{- if not .Values.resourceLimits.ignore }}
resources:
requests:
memory: "100Mi"
cpu: "0.1"
limits:
memory: "100Mi"
cpu: "0.2"
{{ .Values.resourceLimits.apigw | toYaml | indent 12 }}
{{- end }}
ports:
- containerPort: 8080
......@@ -95,6 +92,10 @@ spec:
secretKeyRef:
name: {{ .Values.credentials.postgres.spaces.admin.secretKeyRef.name }}
key: {{ .Values.credentials.postgres.spaces.admin.secretKeyRef.key }}
{{- if and .Values.tracing.withIntegrated.enabled .Values.tracing.withIntegrated.grafana.enabled }}
- name: I3DH_INTEGRATED_METRICS
value: "true"
{{- end }}
volumeMounts:
- name: local-nginx-config
mountPath: "/opt/nginx.custom/"
......@@ -106,6 +107,9 @@ spec:
mountPath: "/var/cache/nginx"
- name: i3dhub-config-service
mountPath: {{ include "i3dhub.config.service" . }}
- name: i3dhub-nginx-ssl
mountPath: "/etc/nginx/ssl"
readOnly: true
- name: i3dhub-authsvc
image: {{ if .Values.registry }}{{ .Values.registry }}/{{ end }}i3dhub-authsvc:{{ .Values.imageTag }}
imagePullPolicy: {{ .Values.imagePullPolicy }}
......@@ -116,20 +120,13 @@ spec:
{{- end }}
{{- if not .Values.resourceLimits.ignore }}
resources:
requests:
memory: "100Mi"
cpu: "0.2"
limits:
memory: "100Mi"
cpu: "0.4"
{{ .Values.resourceLimits.authsvc | toYaml | indent 12 }}
{{- end }}
ports:
- containerPort: 6060
env:
- name: I3DH_CONSUL_HOST
value: i3dhub-consul:8500
- name: JAEGER_COLLECTOR_HOSTS
value: i3dhub-jaeger-collector:14250
- name: POSTGRES_CONNECTION_STRING
valueFrom:
secretKeyRef:
......@@ -140,11 +137,14 @@ spec:
secretKeyRef:
name: {{ .Values.credentials.rabbitmq.secretKeyRef.name }}
key: {{ .Values.credentials.rabbitmq.secretKeyRef.key }}
{{- if .Values.tracing.withJaeger }}
- name: I3DH_JAEGER_ELK_ENABLED
{{- if .Values.tracing.withKibana}}
- name: I3DH_ELK_ENABLED
value: "true"
{{- else if .Values.tracing.withDatadog }}
- name: I3DH_DATADOG_ENABLED
{{- end }}
{{- if .Values.tracing.withIntegrated.enabled }}
{{ include "i3dhub.tracing.jaeger.settings" . | indent 12 }}
{{- else if .Values.tracing.withOTel }}
- name: I3DH_OTEL_ENABLED
value: "true"
{{- end }}
{{- if .Values.tracing.additionalContainerEnv }}
......@@ -166,6 +166,10 @@ spec:
- name: i3dhub-config-generators
configMap:
name: i3dhub-config-generators
- name: i3dhub-nginx-ssl
secret:
secretName: apigw-tls-secret
optional: true
- name: local-nginx-config
emptyDir: {}
- name: local-nginx-log
......@@ -226,3 +230,18 @@ spec:
type: NodePort
{{- end }}
{{- end }}
---
apiVersion: v1
kind: Service
metadata:
name: i3dhub-apigw-authsvc-headless
spec:
selector:
app: i3dhub-apigw
clusterIP: None
ports:
- name: metrics
port: 6060
targetPort: 6060
......@@ -8,7 +8,7 @@ TODO: Find an automated way to list these with helm, .Files does not
work as we cannot access files in the templates/ folder. We need these to
be templated though as helm needs to be able modify them.
*/}}
{{ $services := tuple "AdminService.yml" "APIGateway.yml" "AuthService.yml" "BaseConfig.yml" "DeliveryService.yml" "EntryGateway.yml" "GeometricService.yml" "KeyStore.yml" "MeasurementService.yml" "QueryService.yml" "ResourceGateway.yml" "SharedSessionService.yml" "SpaceService.yml" "SpaceStore.yml" "SpaceStoreService.yml" "TrackerService.yml" "TranscoderService.yml" "UploadService.yml" "VisService.yml" "MigrateDatabases.yml" }}
{{ $services := tuple "AdminService.yml" "APIGateway.yml" "APIGatewayHTTP.yml" "APIGatewayHTTPS.yml" "AuthService.yml" "BaseConfig.yml" "DeliveryService.yml" "EntryGateway.yml" "GeometricService.yml" "KeyStore.yml" "MeasurementService.yml" "QueryService.yml" "ResourceGateway.yml" "SharedSessionService.yml" "SpaceService.yml" "SpaceStore.yml" "SpaceStoreService.yml" "TrackerServiceResGw.yml" "TrackerServiceGPU.yml" "TrackerServiceCPU.yml" "TranscoderService.yml" "UploadService.yml" "VisServiceResGw.yml" "VisServiceGPU.yml" "VisServiceCPU.yml" "MigrateDatabases.yml" }}
{{- $scope := . -}}
{{- range $_, $t := $services }}
{{- with $scope }}
......@@ -58,3 +58,94 @@ TODO: Find an automated way to list these with helm, .Files does not
{{ include $t . | indent 4 }}
{{- end }}
{{- end }}
{{- if .Values.tracing.withIntegrated.enabled }}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: i3dhub-config-prometheus
data:
{{- /*
TODO: Find an automated way to list these with helm, .Files does not
work as we cannot access files in the templates/ folder. We need these to
be templated though as helm needs to be able modify them.
*/}}
{{ $cfgs := tuple "prometheus.yml" }}
{{- $scope := . -}}
{{- range $_, $t := $cfgs }}
{{- with $scope }}
{{ $t }}: |-
{{ include $t . | indent 4 }}
{{- end }}
{{- end }}
{{- if .Values.tracing.withIntegrated.grafana.enabled }}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: i3dhub-config-grafana-provision-dashboard
data:
{{- /*
TODO: Find an automated way to list these with helm, .Files does not
work as we cannot access files in the templates/ folder. We need these to
be templated though as helm needs to be able modify them.
*/}}
{{ $cfgs := tuple "dashboards.yml" }}
{{- $scope := . -}}
{{- range $_, $t := $cfgs }}
{{- with $scope }}
{{ $t }}: |-
{{ include $t . | indent 4 }}
{{- end }}
{{- end }}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: i3dhub-config-grafana-provision-datasource
data:
{{- /*
TODO: Find an automated way to list these with helm, .Files does not
work as we cannot access files in the templates/ folder. We need these to
be templated though as helm needs to be able modify them.
*/}}
{{ $cfgs := tuple "datasources.yml" }}
{{- $scope := . -}}
{{- range $_, $t := $cfgs }}
{{- with $scope }}
{{ $t }}: |-
{{ include $t . | indent 4 }}
{{- end }}
{{- end }}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: i3dhub-config-grafana-dashboards
data:
{{- /*
TODO: Find an automated way to list these with helm, .Files does not
work as we cannot access files in the templates/ folder. We need these to
be templated though as helm needs to be able modify them.
*/}}
{{ $cfgs := tuple "dashboard_main.json" "dashboard_transcoder.json" }}
{{- $scope := . -}}
{{- range $_, $t := $cfgs }}
{{- with $scope }}
{{ $t }}: |-
{{ include $t . | indent 4 }}
{{- end }}
{{- end }}
{{- end }} # .Values.tracing.withIntegrated.grafana.enabled
{{- end }} # .Values.tracing.withIntegrated.enabled
......@@ -36,12 +36,7 @@ spec:
{{- end }}
{{- if not .Values.resourceLimits.ignore }}
resources:
requests:
memory: "100Mi"
cpu: "0.05"
limits:
memory: "100Mi"
cpu: "0.1"
{{ .Values.resourceLimits.consul | toYaml | indent 12 }}
{{- end }}
ports:
- containerPort: 8500
......
......@@ -52,8 +52,10 @@ spec:
secretKeyRef:
name: {{ .Values.credentials.rabbitmq.secretKeyRef.name }}
key: {{ .Values.credentials.rabbitmq.secretKeyRef.key }}
{{- if .Values.auth.signing.enabled }}
- name: KEYSTORE_ADDRESS
value: http://i3dhub-keystore:8080
{{- end }}
- name: CONSUL_ADDRESS
value: i3dhub-consul:8500
containers:
......@@ -67,12 +69,7 @@ spec:
{{- end }}
{{- if not .Values.resourceLimits.ignore }}
resources:
requests:
memory: "100Mi"
cpu: "0.1"
limits:
memory: "100Mi"
cpu: "0.2"
{{ .Values.resourceLimits.deliverysvc | toYaml | indent 12 }}
{{- end }}
ports:
- containerPort: 8080
......@@ -81,13 +78,14 @@ spec:
value: i3dhub-elastic:9200
- name: I3DH_CONSUL_HOST
value: i3dhub-consul:8500
- name: JAEGER_COLLECTOR_HOSTS
value: i3dhub-jaeger-collector:14250
{{- if .Values.tracing.withJaeger }}
- name: I3DH_JAEGER_ELK_ENABLED
{{- if .Values.tracing.withKibana}}
- name: I3DH_ELK_ENABLED
value: "true"
{{- else if .Values.tracing.withDatadog }}
- name: I3DH_DATADOG_ENABLED
{{- end }}
{{- if .Values.tracing.withIntegrated.enabled }}
{{ include "i3dhub.tracing.jaeger.settings" . | indent 12 }}
{{- else if .Values.tracing.withOTel }}
- name: I3DH_OTEL_ENABLED
value: "true"
{{- end }}
{{- if .Values.tracing.additionalContainerEnv }}
......@@ -115,20 +113,13 @@ spec:
{{- end }}
{{- if not .Values.resourceLimits.ignore }}
resources:
requests:
memory: "100Mi"
cpu: "0.2"
limits:
memory: "100Mi"
cpu: "0.4"
{{ .Values.resourceLimits.authsvc | toYaml | indent 12 }}
{{- end }}
ports:
- containerPort: 6060
env:
- name: I3DH_CONSUL_HOST
value: i3dhub-consul:8500
- name: JAEGER_COLLECTOR_HOSTS
value: i3dhub-jaeger-collector:14250
- name: POSTGRES_CONNECTION_STRING
valueFrom:
secretKeyRef:
......@@ -139,11 +130,14 @@ spec:
secretKeyRef:
name: {{ .Values.credentials.rabbitmq.secretKeyRef.name }}
key: {{ .Values.credentials.rabbitmq.secretKeyRef.key }}
{{- if .Values.tracing.withJaeger }}
- name: I3DH_JAEGER_ELK_ENABLED
{{- if .Values.tracing.withKibana}}
- name: I3DH_ELK_ENABLED
value: "true"
{{- else if .Values.tracing.withDatadog }}
- name: I3DH_DATADOG_ENABLED
{{- end }}
{{- if .Values.tracing.withIntegrated.enabled }}
{{ include "i3dhub.tracing.jaeger.settings" . | indent 12 }}
{{- else if .Values.tracing.withOTel }}
- name: I3DH_OTEL_ENABLED
value: "true"
{{- end }}
{{- if .Values.tracing.additionalContainerEnv }}
......@@ -190,3 +184,33 @@ spec:
ports:
- port: 8080
targetPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: i3dhub-deliverysvc-headless
spec:
selector:
app: i3dhub-deliverysvc
clusterIP: None
ports:
- name: metrics
port: 8080
targetPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: i3dhub-deliverysvc-authsvc-headless
spec:
selector:
app: i3dhub-deliverysvc
clusterIP: None
ports:
- name: metrics
port: 6060
targetPort: 6060
{{- if .Values.tracing.withJaeger }}
{{- if .Values.tracing.withKibana }}
# ELASTIC BEGIN
apiVersion: apps/v1
......@@ -40,12 +40,7 @@ spec: