Commit cc231169 authored by threedytech's avatar threedytech
Browse files

release: 3.4.0

parent 71203be7
################################################################################
User Authentication Passing
################################################################################
********************************************************************************
Introduction
********************************************************************************
The system does not understand the concept of users directly, it is only
possible to pass existing credentials along to external systems. Credentials
must exist in the form of HTTP headers on any requests the system receives.
External systems are then required to perform authorization checks with these
headers when 3D data is accessed or downloaded.
.. warning::
Configuration of authentication forwarding rules is required in order to pass
any authentication tokens, no matter the method described here. No headers
ever leave the system in the default configuration.
Forwarding Configuration
################################################################################
All configuration takes place in ``values.yaml``.
Rules must be configured which define which headers are allowed to be sent to
which backend addresses. Preferably this is done directly via URNs:
.. code-block:: yaml
# Set the URN mapping rules per data gateway. See the integration documentation for
# motivation and concepts behind data gateways and URN mappings.
#
# For templates, $(n) is the n-th value in the URN separated after urn:namespace:specifier:xxx
# e.g.: urn:x-i3d:shape:sphere ($(1) == sphere)
dataGateways:
exampleGateway:
- namespace: customer
specifier: document-uuid
urlContentType: [ "openjt" ]
urlTemplate: https://download.example.com/documents/$(1).jt
authUrlTemplate: https://auth.example.com/documents/$(1).jt
forwardCookies: [ "Cookie1" ]
forwardHeaders: [ "Header1" ]
The fields ``forwardCookies`` and ``forwardHeaders`` can be set in order to
specify which cookies (by name) or which headers (by name) will be forwarded
(if present) when accessing any resource to which the URN points.
If all cookies should be forwarded, regardless of name, ``Cookies`` can be added
to ``forwardHeaders``.
Alternatively, it is also possible to define a regular expression which defines
a set of backends to forward headers to:
.. code-block:: yaml
# Configure options relating to required authorization headers and caching.
auth:
# Set any authorization header names which must be passed to data backends.
# These will be copied from incoming client requests and used while downloading
# data or performing authorization requests against the backends.
# No client headers or cookies will be forwarded to any backends by default as
# this could leak client credentials to arbitrary URLs.
# If dataGateways are defined, forwardHeaders or forwardCookies fields can also be
# defined there on a per-rule basis. This should be preferred as it is more robust
# than regex matching URLs.
# In either case, rules from URN definitions are applied first, after which the
# regex rules defined here are applied on the URL generated by resolving the URN.
forwardHeaders:
- match: "https://server1.*"
headers: [ "Header1", "Header2" ]
cookies: [ "Cookie1", "Cookie2" ]
- match: "https://server2.*"
headers: [ "Header1", "Header3" ]
cookies: [ "Cookie1", "Cookie3" ]
Caching Configuration
################################################################################
External backends can be exposed to a high load if caching is not configured. As
the system does not understand users directly, caching can only work effectively
if it is possible to derive a user ID from an authentication token.
OpenID-Connect provides this functionality and can be configured if available.
If OpenID-Connect is not available, it is possible to also cache directly by
header value, but this may be unstable if values frequently change on a request
by request basis.
.. code-block:: yaml
# Configure options relating to required authorization headers and caching.
auth:
caching:
# Address of the used openid provider. For google this would be
# https://accounts.google.com
# .well-known/openid-configuration is appended to this URL in order to
# discover relevant endpoints.
oidcProviderEndpoint:
# Name of the header which contains the OIDC token. If a provider endpoint
# is given, the service will attempt to validate this token as an
# OpenID-Connect JWT token and use the contained user ID to cache
# authorization responses. Without a provider endpoint, the token
# value will be used to cache authorization responses.
oidcHeader:
# Duration for which to cache an authorization result. Only successful
# authorizations are cached. Must be set to 0s if no header is given.
# Otherwise responses will be cached for the given duration and used for
# every user.
duration: 0s
Custom Client Header Configuration
################################################################################
There are two use-cases based on how authentication tokens are aquired:
* Applications always have SSO authentication tokens set by cookies
* Applications must perform authentication flow and set headers themselves
For the second case, the backend must be configured to allow these to be set in
CORS scenarios:
.. code-block:: yaml
# Configure options relating to required authorization headers and caching.
auth:
# Sets a list of headers to include in any Access-Control-Allow-Headers preflight
# responses. This is required when setting headers manually via the webvis-API.
# For example, if clients are required to attach credentials not in cookie form,
# the specfic headers must be allowed here.
clientHeaders:
- "Custom-Authorization"
- "X-Requested-With"
############################################
instant3Dhub on Docker Compose
############################################
********************************************
Read First
********************************************
.. warning::
We value Docker-Compose as a lightweight, rapid development alternative to our
Helm deploy format. Aspects like Scaling, Security and Integration into the
latest orchestration tools are not supported with this method. We recommend
Kubernetes for production systems.
* **Services**: Currently only the SharedSession, SpaceStore, Measurement and Query services are enabled. This means a range of functionalities will not be available in webvis and other API-libraries.
* **Security/Signatues**: The third main version instant3Dhub is designed to include security on all layers of the system like controlling the access to the management APIs, resource APIs, the services or the data that goes through the system. It is not yet possible to configure custom keys.
* **Volumes** : Currently our Docker-Compose deploy uses local volumes.
********************************************
Installation
********************************************
The instant3Dhub Docker-Compose deployment is based on `Docker-Compose v2 <https://docs.docker.com/compose/compose-file/compose-file-v2/>`_ files.
Overview
--------------------------------------------
The installation of instant3Dhub consists of three phases:
#. **Provisioning**: First the server and required resources need to be set up. For more information refer `here <./INSTALL_REQUIREMENTS.rst>`_
#. **Configuration**: instant3Dhub needs to be configured to integrate correctly with the infrastructure.
#. **Startup**: Finally the system is applied to the server or passed to a GitOps pipeline.
Configuration
--------------------------------------------
The instant3Dhub Docker-Compose deployment is arranged to be as similar to our helm deploy as possible.
A set of variables need to be set in your environment before startup.
Please take reference in ``./reference/compose/``
The following required to be set and dependent on the environment:
* ``entrypoints``: The list of URLs from which the system is reachable from outside the cluster.
* ``registry``: The URI to the container image registry. When you use pre-pulled images remove this entry.
* ``licenseServer``: The cluster internal URL to the instant3Dhub `License Server <./LICENSE_SERVER.rst>`_.
Startup
--------------------------------------------
Utilize the scripts in ``./reference/compose/`` to install or uninstall the
system. After installation, the endpoint URL to reach webVis will be printed.
The system may take a few minutes to initialize and start correctly.
...@@ -48,11 +48,18 @@ Therefore, helm should be installed first: ...@@ -48,11 +48,18 @@ Therefore, helm should be installed first:
&& chmod 700 get_helm.sh \ && chmod 700 get_helm.sh \
&& ./get_helm.sh && ./get_helm.sh
Now, add the instant3Dhub Helm repository: We provide three channels for helm packages:
* **stable**: This channel contains the stable release packages of instant3Dhub.
* **dev**: This channel contains all rc and dev packages.
* **trk_<track-name>**: Each track package has its own channel and is named using the suffix trk_ followed by the name of the track.
Now, add the instant3Dhub Helm repository from one of the channels:
.. code-block:: bash .. code-block:: bash
helm repo add instant3Dhub https://repo.threedy.io/api/v4/projects/2/packages/helm/stable \ helm repo add instant3Dhub https://repo.threedy.io/api/v4/projects/2/packages/helm/<channel> \
&& helm repo update && helm repo update
You can now deploy using helm install but please continue reading to know which configuration you need to set. There are four main configuration that are required and need to be adjusted before you deploy. You can now deploy using helm install but please continue reading to know which configuration you need to set. There are four main configuration that are required and need to be adjusted before you deploy.
......
...@@ -6,7 +6,8 @@ instant3Dhub on single node ...@@ -6,7 +6,8 @@ instant3Dhub on single node
Read First Read First
******************************************** ********************************************
This guide targets those who have neither an existing k8s cluster nor the resources to set up and manage one. This guide targets those who have neither an existing k8s cluster nor the resources to set up and manage one.
While having a k8s cluster to run instant3Dhub is still a requirement, we provide tools to get all the requirments ready for you on a single CentOS 8 machine. While having a k8s cluster to run instant3Dhub is still a requirement, we provide tools to get all the requirements ready for you on a single CentOS 8 machine.
For installing on WINDOWS, please follow the instructions `here <INSTALL_WINDOWS.rst>`_
******************************************* *******************************************
...@@ -66,3 +67,9 @@ Now you can install instant3Dhub ...@@ -66,3 +67,9 @@ Now you can install instant3Dhub
.. code-block:: bash .. code-block:: bash
./install.sh ./install.sh
.. toctree::
:hidden:
INSTALL_WINDOWS
\ No newline at end of file
#####################################
instant3Dhub on single node - WINDOWS
#####################################
********************************************
Read First
********************************************
This guide targets those who wants to run instant3Dhub for local testing on a WINDOWS machine. Installing the
`license server <LICENSE_SERVER.rst>`_ is not part of this guide as it requires Linux machine and we do not support WINDOWS.
*******************************************
Prerequisites
*******************************************
The machine where you want to run instant3Dhub should have the following requirements in order to run the system properly:
These minimal resources are required to run instant3Dhub
* CPU-only: CPU=4 Cores, Mem=16GB
* with GPU: CPU=8 Cores, Mem=32GB, GPU=1 NVIDIA GPU
*************************************************
Installation
*************************************************
#. Install Docker Desktop: Docker Desktop is freely available in a community edition, for WINDOWS and Mac. Start by downloading and installing docker desktop: https://hub.docker.com/editions/community/docker-ce-desktop-windows.
#. Enable Kubernetes
#. Make sure you have Docker Desktop running - in the taskbar in WINDOWS you’ll see Docker’s whale logo. Click the whale and select Settings.
#. A new screen opens with all of Docker Desktop’s configuration options. Click on Kubernetes and check the Enable Kubernetes checkbox.
#. Docker Desktop will download all the Kubernetes images in the background and get everything started up. When it’s ready you’ll see two green lights in the bottom of the settings screen saying Docker running and Kubernetes running.
#. Verify your Kubernetes cluster by running the following command
.. code-block:: bash
kubectl get nodes
You should see a single node in the output called docker-desktop. That’s a full Kubernetes cluster, with a single node that runs the Kubernetes API and your own applications.
#. Install helm:
We provide a Helm Chart repository for the deployment of instant3Dhub on Kubernetes.
Therefore, helm should be installed first:
Please follow the instructions from the official helm documentation to get helm on WINDOWS: https://helm.sh/docs/intro/install/
#. Now, open a command prompt and run the following command to add the instant3Dhub Helm repository:
.. code-block:: bash
helm repo add instant3Dhub https://repo.threedy.io/api/v4/projects/2/packages/helm/stable \
&& helm repo update
#. Run helm install to deploy instant3Dhub
.. code-block:: bash
helm install -n i3dhub i3dhub-windows instant3Dhub/instant3Dhub \
--set licenseServer=license.yourdomain.com:8200 \
--set storage.class=hostpath \
--set entrypoints={http://localhost:30042} \
--set registry=images.threedy.io
#. The startup and initialization of all containers can take a few minutes, depending on your cluster and registry connection speed. To check the status of the containers run:
.. code-block:: bash
kubectl get pods -n i3dhub
#. Once all containers are running, you should be able to access the splash page at
http://localhost:30042
############################################ ################################################################################
instant3Dhub License Server instant3Dhub License Server
############################################ ################################################################################
******************************************** ********************************************************************************
Read First Read First
******************************************** ********************************************************************************
.. important:: The instant3Dhub License Server needs to be accessible by the servers
Previous installations of the LicenseServer must be uninstalled before installing instant3Dhub 3.0. instant3Dhub is running on. The license server cannot be run in container
environments as it is bound to a host machine. Although this might look similar
to the external PostgreSQL install, it uses a different set of Ansible
playbooks.
The instant3Dhub License Server needs to be accessible by the servers instant3Dhub is running on. .. warning::
The license server cannot be run in container environments as it is bound to a host machine. If upgrading the license server from version 21.4, see the upgrading section
Although this might look similar to the external PostgreSQL install, it uses a different set of Ansible playbooks. below for instructions. Upgrading from older versions requires a full
reinstall. Upgrading from newer versions is done automatically.
Installation Installation
*********************************************** ********************************************************************************
The instant3Dhub License Server is installed from rpm files. The instant3Dhub License Server is installed from RPM files. Configuration is
Configuration is done via Ansible. If you need guidance on how to install Ansible please refer to our Ansible install `guide <./TOOLS_ANSIBLE.rst>`_. done via Ansible. If you need guidance on how to install Ansible please refer to
The package can be downloaded here: our Ansible install `guide <./TOOLS_ANSIBLE.rst>`_.
Installation Package Installation Package
============================================ ================================================================================
The installation requires two packages, the OS dependent The installation requires two packages, the OS dependent license server package:
license server package:
* **CentOS 7** : `instant3Dhub-licenseServer-CentOS-Linux-7-x64\*.rpm <https://repo.threedy.io/repository/instant3Dhub_raw/licenseServer/instant3Dhub-licenseServer-CentOS-Linux-7-x64.21.4.rpm>`_ * **CentOS 7** : instant3Dhub-licenseServer-CentOS-Linux-7-x64\*.rpm
* **SLES-12.3** : `instant3Dhub-licenseServer-SLES-12.3-x64\*.rpm <https://repo.threedy.io/repository/instant3Dhub_raw/licenseServer/instant3Dhub-licenseServer-SLES-12.3-x64.21.4.rpm>`_ * **SLES-12.3** : instant3Dhub-licenseServer-SLES-12.3-x64\*.rpm
* **Fedora 23** : `instant3Dhub-licenseServer-Fedora-23-x64\*.rpm <https://repo.threedy.io/repository/instant3Dhub_raw/licenseServer/instant3Dhub-licenseServer-Fedora-23-x64.21.4.rpm>`_ * **Fedora 23** : instant3Dhub-licenseServer-Fedora-23-x64\*.rpm
* **Fedora 29** : `instant3Dhub-licenseServer-Fedora-29-x64\*.rpm <https://repo.threedy.io/repository/instant3Dhub_raw/licenseServer/instant3Dhub-licenseServer-Fedora-29-x64.21.4.rpm>`_ * **Fedora 29** : instant3Dhub-licenseServer-Fedora-29-x64\*.rpm
and the ansible roles to install and initialize the database: and the ansible roles to install and initialize the database:
* `instant3Dhub-ansible-noarch-\*.tgz <https://repo.threedy.io/repository/instant3Dhub_raw/licenseServer/instant3Dhub-ansibledb-noarch-21.4.tgz>`_ * instant3Dhub-ansible-noarch-\*.tgz
Install License Server DB: (utilizes ansible) The latest installation files and a changelog can be found `here <https://repo.threedy.io/licenseserver/>`_
=============================================
Fresh Installation with Ansible
================================================================================
We assume that no instant3Dhub version 2 PostgreSQL installation is running on We assume that no instant3Dhub version 2 PostgreSQL installation is running on
the same machine. If this is the case the port and socket directory used by the same machine. If this is the case the port and socket directory used by
the license database postrgres cluster must be changed from 5433 to another the license database postrgres cluster must be changed from 5433 to another
port, for example to 5435 and ``/var/run/instant3DhubLicensePgsql``. port, for example to 5435 and ``/var/run/instant3DhubLicensePgsql``.
In the following documentation we assume that the License Server is installed on the same host used as the ansible control host. We recommend PostgreSQL version 14, but require a minmum version of 9.5. The
For that reason the following inventory is a localhost inventory. minmum version may change in the future, so we recommend installing the highest
version available in order to avoid time consuming upgrades.
For RedHat based systems, we recommend using the
`official PostgreSQL repository <https://www.postgresql.org/download/linux/redhat/>`_
to download packages. This guide will exemplify the installation using CentOS 7.
First unzip the provided ``instant3Dhub-ansible-noarch-\*.tgz``: First, add the official PostgreSQL repository to the package manager:
.. code-block:: bash .. code-block:: bash
tar zxvf ./instant3Dhub-ansible-*.tgz sudo yum install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm
The folder where this is extracted should now contain a ``./PostgreSQL/`` folder. Next, install PostgreSQL 14:
Create the following ``hosts.yml``: .. code-block:: bash
sudo yum install -y postgresql14-server
sudo yum install -y postgresql14-contrib
Now we will use Ansible to configure PostgreSQL for our needs. Unzip the
provided ``instant3Dhub-ansibledb-noarch-\*.tgz`` from above:
.. code-block:: bash
tar zxvf ./instant3Dhub-ansibledb-*.tgz
The current folder should now contain an ``./ansible/PostgreSQL/`` folder.
In the following documentation we assume that the License Server is installed on
the same host used as the ansible control host. For that reason the following
inventory is a localhost inventory. Create the following ``hosts.yml``:
.. code-block:: yaml .. code-block:: yaml
...@@ -63,36 +89,37 @@ Create the following ``hosts.yml``: ...@@ -63,36 +89,37 @@ Create the following ``hosts.yml``:
localhost: localhost:
vars: vars:
ansible_connection: local ansible_connection: local
pgsql_version: 14
pgsql_do_install: false
pgsql_bin_directory: /usr/pgsql-14/bin
#pgsql_configure_selinux: true
SELinux: .. note::
**SELinux**: if you are required to run SELinux please uncomment the
if you are required to run SELinux please add the following var with the correct identation to your hosts.yml. ``pgsql_configure_selinux`` part of the inventory.
.. code-block:: yaml
pgsql_configure_selinux: true
Invoke the following playbooks: Invoke the following playbooks:
.. code-block:: bash .. code-block:: bash
ansible-playbook -i hosts.yml ./PostgreSQL/pgsql.licensedb.install.yml ansible-playbook -i hosts.yml ./ansible/PostgreSQL/pgsql.licensedb.install.yml
ansible-playbook -i hosts.yml ./PostgreSQL/pgsql.licensedb.init.yml ansible-playbook -i hosts.yml ./ansible/PostgreSQL/pgsql.licensedb.init.yml
This installs and initializes a PostgreSQL instance.
This installs and initializes a PostgreSQL instance.
Install License Server Install License Server
============================================ ================================================================================
.. code-block:: bash .. code-block:: bash
yum install ./instant3Dhub-licenseServer-*.rpm sudo yum install -y ./instant3Dhub-licenseServer-*.rpm
HostID generation HostID generation
***************** ********************************************************************************
please invoke Licenses are bound to a single host. We require a HostID to provide a license.
Please invoke
.. code-block:: bash .. code-block:: bash
...@@ -104,19 +131,20 @@ The output should look similar to the following: ...@@ -104,19 +131,20 @@ The output should look similar to the following:
Current Host ID: 7bf3b23f5c6ff5a444a37f2dfc42ed34f5c470df Current Host ID: 7bf3b23f5c6ff5a444a37f2dfc42ed34f5c470df
Please provide your threedy contact or if you have no valid license agreement yet ``sales@threedy.io`` with this output to start the license key aquisition process. Please provide your threedy contact or if you have no valid license agreement
In return a license file should be provided to you. yet ``sales@threedy.io`` with this output to start the license key aquisition
process. In return a license file will be provided to you.
LicenseFile placement License File placement
********************* ********************************************************************************
please put the license file you received here Please put the license file you received to:
.. code-block:: bash .. code-block:: bash
/opt/instant3Dhub.custom/license.xml /opt/instant3Dhub.custom/license.xml
restart the License Server Restart the License Server:
.. code-block:: bash .. code-block:: bash
...@@ -125,6 +153,248 @@ restart the License Server ...@@ -125,6 +153,248 @@ restart the License Server
The status should now show be active. The status should now show be active.
In default configuration this service exposes 8200 and records license data in a local PostgreSQL instance. In default configuration this service exposes 8200 and records license data in a
local PostgreSQL instance.
An address to this License Server must be placed inside the setup file of your
instant3Dhub deployment.
********************************************************************************
License Server Upgrades
********************************************************************************
Upgrades are normally painless and only require the installation of a new RPM
package. The database will automatically be updated when the license server
starts up the next time. Below is a list of upgrades which require manual
intervention.
Upgrading from 21.4 to 22.1
********************************************************************************
This process is fairly involved as any version before 22.1 used PostgreSQL 9.2.
Version 22.1 however requires a minmum PostgreSQL version of 9.5. In order to
avoid more upgrades in the future, we recommend upgrading to the latest
PostgreSQL version available. Below is the process for upgrading from PostgreSQL
9.2 to PostgreSQL 14 on CentOS 7.
First, shut down the currently running license server:
.. code-block:: bash
systemctl stop instant3DhubLicenseServer
systemctl stop instantHubPGDB
Next, move the internal PostgreSQL data to a new folder as the newer version
will use the same directory as the old installation:
.. code-block:: bash
mv /var/cache/instant3Dhub3/pgsql/data /var/cache/instant3Dhub3/pgsql/data.bak
Now we are ready to install a new PostgreSQL version. First, add the official
PostgreSQL repository to the package manager:
.. code-block:: bash
sudo yum install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm
Next, install PostgreSQL 14:
.. code-block:: bash
sudo yum install -y postgresql14-server
sudo yum install -y postgresql14-contrib
Now we will use Ansible to configure the new version for our needs. Unzip the
provided ``instant3Dhub-ansibledb-noarch-\*.tgz`` from above:
.. code-block:: bash
tar zxvf ./instant3Dhub-ansibledb-*.tgz
The current folder should now contain an ``./ansible/PostgreSQL/`` folder.
Create the following ``hosts.yml``:
.. code-block:: yaml
hubs:
hosts:
localhost:
vars:
ansible_connection: local
pgsql_version: 14
pgsql_do_install: false
pgsql_bin_directory: /usr/pgsql-14/bin
#pgsql_configure_selinux: true
.. note::
**SELinux**: if you are required to run SELinux please uncomment the
``pgsql_configure_selinux`` part of the inventory.
Invoke the following playbook to initializes a PostgreSQL instance ready to be
upgraded:
.. code-block:: bash
ansible-playbook -i hosts.yml ./ansible/PostgreSQL/pgsql.licensedb.install.yml
Next, stop the new instance again:
.. code-block:: bash
systemctl stop instantHubPGDB
There is an issue with ``pg_upgrade`` starting the old and new PostgreSQL
instances during the upgrade process where the wrong command line parameters are
passed, so we need to fix this by rerouting ``pg_ctl`` through a shell script
first. We do this with the following commands:
.. code-block:: bash
mv /usr/bin/pg_ctl{,-orig}