fix project check in secret_sync_gitlab_token (#378)

* feat: add Data Quality Agent configuration and setup

* fix: correct environment variable name in Data Quality Agent configuration

* feat: update Data Quality Agent configuration and setup instructions

* feat: update Data Quality Agent setup and documentation

* feat: add volume configuration for agent data in Data Quality Agent compose file

* feat: update volume configuration for Data Quality Agent in compose file

* Update README.md

Co-authored-by: Tobias Kussel <TKussel@users.noreply.github.com>

---------

Co-authored-by: Tobias Kussel <TKussel@users.noreply.github.com>

README.md

@@ -27,6 +27,7 @@ This repository is the starting point for any information and tools you will nee
     - [Teiler (Frontend)](#teiler-frontend)
     - [Data Exporter Service](#data-exporter-service)
       - [Data Quality Report](#data-quality-report)
+    - [Data Quality Agent](#data-quality-agent)
 4. [Things you should know](#things-you-should-know)
     - [Auto-Updates](#auto-updates)
     - [Auto-Backups](#auto-backups)
@@ -73,7 +74,7 @@ We recommend to install Docker(-compose) from its official sources as described
 
 A Bridgehead communicates to all central components via outgoing HTTPS connections.
 
-Your site might require an outgoing proxy (i.e. HTTPS forward proxy) to connect to external servers; you should discuss this with your local systems administration. In that case, you will need to note down the URL of the proxy. If the proxy requires authentication, you will also need to make a note of its username and password. This information will be used later on during the installation process. Special characters in the proxy values, e.g. in the access credentials, must be [URL-encoded](https://en.wikipedia.org/wiki/Percent-encoding), e.g. by replacing `@` with `%40`, `/` with `%2F` and so on. TLS terminating proxies are also supported, see [here](#tls-terminating-proxies). Apart from the Bridgehead itself, you may also need to configure the proxy server in [git](https://gist.github.com/evantoli/f8c23a37eb3558ab8765) and [docker](https://docs.docker.com/network/proxy/).
+Your site might require an outgoing proxy (i.e. HTTPS forward proxy) to connect to external servers; you should discuss this with your local systems administration. In that case, you will need to note down the URL of the proxy. If the proxy requires authentication, you will also need to make a note of its username and password. This information will be used later on during the installation process. TLS terminating proxies are also supported, see [here](#tls-terminating-proxies). Apart from the Bridgehead itself, you may also need to configure the proxy server in [git](https://gist.github.com/evantoli/f8c23a37eb3558ab8765) and [docker](https://docs.docker.com/network/proxy/).
 
 The following URLs need to be accessible (prefix with `https://`):
 * To fetch code and configuration from git repositories
@@ -424,6 +425,32 @@ ENABLE_EXPORTER=true
 ```
 [For further information](docs/exporter.md)  
 
+### Data Quality Agent
+
+The Data Quality Agent is an optional module that periodically evaluates the quality of FHIR data stored in Blaze. It generates local data quality reports accessible via the Bridgehead web interface.
+
+To enable the service, set the following variable in your `<PROJECT>.conf` file:
+
+```bash
+ENABLE_DATA_QUALITY_AGENT=true
+```
+
+#### Sharing Data Quality Reports (recommended)
+
+We encourage sharing your data quality reports with the central BBMRI-ERIC quality dashboard. The reports contain only aggregated, non-patient-identifiable statistics and help the network to monitor and improve overall data quality. However, quality reporting is completely optional and opt-in.
+
+To opt in, additionally set the following variables in your `<PROJECT>.conf` file:
+
+```bash
+DATA_QUALITY_SERVER_URL=https://quality-dashboard.bbmri-eric.eu
+DATA_QUALITY_SERVER_NAME=Central Data Quality Server of BBMRI
+```
+
+If these variables are not set, the Data Quality Agent will still run and generate local reports, but no data will be shared externally.
+
+Reports are accessible at `https://<your-host>/bbmri-data-quality-agent` (default credentials are admin:admin, please change it after first login!!).
+
+[Official documentation](https://fdqf.bbmri-eric.eu/user/deployment.html)
 ## Things you should know
 
 ### Auto-Updates
@@ -536,8 +563,6 @@ and restart the docker daemon:
 sudo systemctl restart docker
 ```
 
-Please note that special characters in the proxy value, such as `#?!()[]{}`, must be double escaped using `%%`.
-
 For more information, please consult the [official documentation](https://docs.docker.com/config/daemon/systemd/#httphttps-proxy).
 
 ### Monitoring

bbmri/modules/data-quality-agent-compose.yml

@@ -0,0 +1,23 @@
+version: "3.7"
+
+services:
+  data-quality-agent:
+    image: ghcr.io/bbmri-cz/data-quality-server:${DATA_QUALITY_AGENT_TAG}
+    container_name: bridgehead-bbmri-data-quality-agent
+    environment:
+      APP_SETTING_FHIR_URL: http://bridgehead-bbmri-blaze:8080/fhir
+      REPORTING_SERVER_URL: ${DATA_QUALITY_SERVER_URL}
+      REPORTING_SERVER_NAME: ${DATA_QUALITY_SERVER_NAME}
+    labels:
+      - "traefik.enable=true"
+      - "traefik.http.routers.data_quality_agent_bbmri.rule=PathPrefix(`/bbmri-data-quality-agent`)"
+      - "traefik.http.services.data_quality_agent_bbmri.loadbalancer.server.port=8082"
+      - "traefik.http.routers.data_quality_agent_bbmri.tls=true"
+      - "traefik.http.middlewares.data_quality_agent_bbmri_strip.stripprefix.prefixes=/bbmri-data-quality-agent"
+      - "traefik.http.routers.data_quality_agent_bbmri.middlewares=data_quality_agent_bbmri_strip,auth"
+    depends_on:
+      - "blaze"
+    volumes:
+      - /var/cache/bridgehead/bbmri/agent-db:/app/data
+      - /etc/localtime:/etc/localtime:ro
+      - /etc/timezone:/etc/timezone:ro

bbmri/modules/data-quality-agent-setup.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+
+if [ "$ENABLE_DATA_QUALITY_AGENT" == "true" ]; then
+    log INFO "Data Quality Agent setup detected -- will start data-quality-agent service."
+    OVERRIDE+=" -f ./$PROJECT/modules/data-quality-agent-compose.yml"
+fi
+

bbmri/modules/directory-sync.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-if [ -n "${DS_DIRECTORY_USER_NAME}" ]; then
+if [ -n "${DS_DIRECTORY_USER_NAME}" ] || [ -n "${DS_DIRECTORY_USER_TOKEN}" ]; then
 	log INFO "Directory sync setup detected -- will start directory sync service."
 	OVERRIDE+=" -f ./$PROJECT/modules/directory-sync-compose.yml"
 fi

cce/modules/lens-compose.yml

@@ -3,6 +3,8 @@ services:
   lens:
     container_name: lens_federated-search
     image: samply/cce-explorer:main
+    environment:
+      PUBLIC_SPOT_URL: https://${HOST}/prod
     labels:
       - "traefik.http.services.lens.loadbalancer.server.port=3000"
       - "traefik.enable=true"

cce/modules/osiris2fhir-setup.sh

@@ -0,0 +1,6 @@
+#!/bin/bash
+if [ -n "$ENABLE_OSIRIS2FHIR" ]; then
+  log INFO "OSIRIS2FHIR-REST setup detected -- will start osiris2fhir module."
+  OVERRIDE+=" -f ./pscc/modules/osiris2fhir-compose.yml"
+  LOCAL_SALT="$(echo \"local-random-salt\" | openssl pkeyutl -sign -inkey /etc/bridgehead/pki/${SITE_ID}.priv.pem | base64 | head -c 30)"
+fi

itcc/docker-compose.yml

@@ -15,7 +15,7 @@ services:
       - "blaze-data:/app/data"
     labels:
       - "traefik.enable=true"
-      - "traefik.http.routers.blaze_itcc.rule=PathPrefix(`/itcc-localdatamanagement`)"
+      - "traefik.http.routers.blaze_itcc.rule=Host(`${HOST}`) && PathPrefix(`/itcc-localdatamanagement`)"
       - "traefik.http.middlewares.itcc_b_strip.stripprefix.prefixes=/itcc-localdatamanagement"
       - "traefik.http.services.blaze_itcc.loadbalancer.server.port=8080"
       - "traefik.http.routers.blaze_itcc.middlewares=itcc_b_strip,auth"
@@ -34,7 +34,6 @@ services:
       EPSILON: 0.28
       QUERIES_TO_CACHE: '/queries_to_cache.conf'
       ENDPOINT_TYPE: ${FOCUS_ENDPOINT_TYPE:-blaze}
-      CQL_PROJECTS_ENABLED: "itcc"
     volumes:
       - /srv/docker/bridgehead/itcc/queries_to_cache.conf:/queries_to_cache.conf:ro
     depends_on:

itcc/modules/itcc-omics-ingest.sh

@@ -0,0 +1,6 @@
+#!/bin/bash
+
+if [ -n "$ENABLE_OMICS" ];then
+  OVERRIDE+=" -f ./$PROJECT/modules/itcc-omics-ingest.yaml"
+  GENERATE_API_KEY="$(generate_simple_password 'omics')"
+fi

itcc/modules/itcc-omics-ingest.yaml

@@ -0,0 +1,14 @@
+services:
+  omics-endpoint:
+    image: ghcr.io/samply/itcc-omics-ingest:main
+    environment:
+      - API_KEY=${GENERATE_API_KEY}
+    volumes:
+      - /var/cache/bridgehead/omics/data:/data/uploads
+    labels:
+      - "traefik.http.routers.omics.rule=Host(`${HOST}`) && PathPrefix(`/api/omics`)"
+      - "traefik.enable=true"
+      - "traefik.http.services.omics.loadbalancer.server.port=6080"
+      - "traefik.http.routers.omics.tls=true"
+      - "traefik.http.middlewares.omics-stripprefix.stripprefix.prefixes=/api"
+      - "traefik.http.routers.omics.middlewares=omics-stripprefix"

itcc/modules/lens-compose.yml

@@ -1,33 +1,47 @@
 version: "3.7"
 services:
-  landing:
-    container_name: lens_federated-search
-    image: docker.verbis.dkfz.de/ccp/lens:${SITE_ID}
+  itcc-explorer:
+    container_name: lens_itcc_explorer
+    image: samply/itcc-explorer:main
+    environment:
+      HOST: "0.0.0.0"
+      BIND_ADDR: "0.0.0.0:3000"
+      PUBLIC_ENVIRONMENT: ${PUBLIC_ENVIRONMENT}
     labels:
       - "traefik.enable=true"
-      - "traefik.http.routers.landing.rule=PathPrefix(`/`)"
-      - "traefik.http.services.landing.loadbalancer.server.port=80"
-      - "traefik.http.routers.landing.tls=true"
+      - "traefik.http.routers.itcc.rule=Host(`${HOST}`) && PathPrefix(`/`)"
+      - "traefik.http.routers.itcc.entrypoints=websecure"
+      - "traefik.http.services.itcc.loadbalancer.server.port=3000"
+      - "traefik.http.routers.itcc.tls=true"
 
   spot:
-    image: docker.verbis.dkfz.de/ccp-private/central-spot
+    image: samply/rustyspot:latest
     environment:
       BEAM_SECRET: "${FOCUS_BEAM_SECRET_SHORT}"
-      BEAM_URL: http://beam-proxy:8081
+      BEAM_PROXY_URL: http://beam-proxy:8081
       BEAM_PROXY_ID: ${SITE_ID}
       BEAM_BROKER_ID: ${BROKER_ID}
-      BEAM_APP_ID: "focus"
-      PROJECT_METADATA: "itcc"
+      BEAM_APP_ID: "spot.${SITE_ID}.${BROKER_ID}"
+      CORS_ORIGIN: "https://${HOST}"
+      SITES: ${SITES}
+      TRANSFORM: LENS
+      PROJECT: "itcc"
+      BIND_ADDR: 0.0.0.0:8055
     depends_on:
       - "beam-proxy"
     labels:
       - "traefik.enable=true"
-      - "traefik.http.services.spot.loadbalancer.server.port=8080"
+      - "traefik.http.services.spot.loadbalancer.server.port=8055"
       - "traefik.http.middlewares.corsheaders2.headers.accesscontrolallowmethods=GET,OPTIONS,POST"
+      - "traefik.http.middlewares.corsheaders2.headers.accesscontrolallowheaders=content-type"
       - "traefik.http.middlewares.corsheaders2.headers.accesscontrolalloworiginlist=https://${HOST}"
       - "traefik.http.middlewares.corsheaders2.headers.accesscontrolallowcredentials=true"
       - "traefik.http.middlewares.corsheaders2.headers.accesscontrolmaxage=-1"
-      - "traefik.http.routers.spot.rule=Host(`${HOST}`) && PathPrefix(`/backend`)"
-      - "traefik.http.middlewares.stripprefix_spot.stripprefix.prefixes=/backend"
+      - "traefik.http.routers.spot.rule=Host(`${HOST}`) && PathPrefix(`/prod`)"
+      - "traefik.http.middlewares.stripprefix_spot.stripprefix.prefixes=/prod"
       - "traefik.http.routers.spot.tls=true"
       - "traefik.http.routers.spot.middlewares=corsheaders2,stripprefix_spot,auth"
+
+  beam-proxy:
+    environment:
+      APP_spot_KEY: ${FOCUS_BEAM_SECRET_SHORT}

itcc/vars

@@ -6,6 +6,7 @@ FOCUS_RETRY_COUNT=${FOCUS_RETRY_COUNT:-64}
 SUPPORT_EMAIL=arturo.macias@dkfz-heidelberg.de
 PRIVATEKEYFILENAME=/etc/bridgehead/pki/${SITE_ID}.priv.pem
 BROKER_URL_FOR_PREREQ=$BROKER_URL
+PUBLIC_ENVIRONMENT=prod
 
 for module in $PROJECT/modules/*.sh
 do

kr/docker-compose.yml

@@ -12,7 +12,8 @@ services:
       BASE_URL: "http://bridgehead-kr-blaze:8080"
       JAVA_TOOL_OPTIONS: "-Xmx${BLAZE_MEMORY_CAP:-4096}m"
       DB_RESOURCE_CACHE_SIZE: ${BLAZE_RESOURCE_CACHE_CAP:-2500000}
-      DB_BLOCK_CACHE_SIZE: $BLAZE_MEMORY_CAP
+      DB_BLOCK_CACHE_SIZE: ${BLAZE_MEMORY_CAP}
+      CQL_EXPR_CACHE_SIZE: ${BLAZE_CQL_CACHE_CAP:-32}
       ENFORCE_REFERENTIAL_INTEGRITY: "false"
     volumes:
       - "blaze-data:/app/data"

kr/modules/export-and-qb.curl-templates

@@ -1,6 +0,0 @@
-# Full Excel Export
-curl --location --request POST 'https://${HOST}/ccp-exporter/request?query=Patient&query-format=FHIR_PATH&template-id=ccp&output-format=EXCEL' \
---header 'x-api-key: ${EXPORT_API_KEY}'
-
-# QB
-curl --location --request POST 'https://${HOST}/ccp-reporter/generate?template-id=ccp'

kr/modules/lens-compose.yml

@@ -4,32 +4,41 @@ services:
     deploy:
       replicas: 1 #reactivate if lens is in use
     container_name: lens_federated-search
-    image: docker.verbis.dkfz.de/ccp/lens:${SITE_ID}
+    image: docker.verbis.dkfz.de/ccp/kr-explorer:main
+    environment:
+      PUBLIC_SPOT_URL: https://${HOST}/prod
     labels:
+      - "traefik.http.services.lens.loadbalancer.server.port=3000"
       - "traefik.enable=true"
-      - "traefik.http.routers.landing.rule=PathPrefix(`/`)"
-      - "traefik.http.services.landing.loadbalancer.server.port=80"
-      - "traefik.http.routers.landing.tls=true"
+      - "traefik.http.routers.lens.rule=Host(`${HOST}`)"
+      - "traefik.http.routers.lens.tls=true"
 
   spot:
-    image: docker.verbis.dkfz.de/ccp-private/central-spot
+    image: samply/rustyspot:latest
     environment:
       BEAM_SECRET: "${FOCUS_BEAM_SECRET_SHORT}"
-      BEAM_URL: http://beam-proxy:8081
-      BEAM_PROXY_ID: ${SITE_ID}
-      BEAM_BROKER_ID: ${BROKER_ID}
-      BEAM_APP_ID: "focus"
-      PROJECT_METADATA: "kr_supervisors"
+      BEAM_PROXY_URL: http://beam-proxy:8081
+      BEAM_APP_ID: "spot.${SITE_ID}.${BROKER_ID}"
+      CORS_ORIGIN: "https://${HOST}"
+      SITES: ${SITES}
+      TRANSFORM: LENS
+      PROJECT: kr
+      BIND_ADDR: 0.0.0.0:8055
     depends_on:
       - "beam-proxy"
     labels:
       - "traefik.enable=true"
-      - "traefik.http.services.spot.loadbalancer.server.port=8080"
+      - "traefik.http.services.spot.loadbalancer.server.port=8055"
       - "traefik.http.middlewares.corsheaders2.headers.accesscontrolallowmethods=GET,OPTIONS,POST"
+      - "traefik.http.middlewares.corsheaders2.headers.accesscontrolallowheaders=content-type"
       - "traefik.http.middlewares.corsheaders2.headers.accesscontrolalloworiginlist=https://${HOST}"
       - "traefik.http.middlewares.corsheaders2.headers.accesscontrolallowcredentials=true"
       - "traefik.http.middlewares.corsheaders2.headers.accesscontrolmaxage=-1"
-      - "traefik.http.routers.spot.rule=Host(`${HOST}`) && PathPrefix(`/backend`)"
-      - "traefik.http.middlewares.stripprefix_spot.stripprefix.prefixes=/backend"
+      - "traefik.http.routers.spot.rule=Host(`${HOST}`) && PathPrefix(`/prod`)"
+      - "traefik.http.middlewares.stripprefix_spot.stripprefix.prefixes=/prod"
       - "traefik.http.routers.spot.tls=true"
-      - "traefik.http.routers.spot.middlewares=corsheaders2,stripprefix_spot"
+      - "traefik.http.routers.spot.middlewares=corsheaders2,stripprefix_spot,auth"
+
+  beam-proxy:
+    environment:
+      APP_spot_KEY: ${FOCUS_BEAM_SECRET_SHORT}

kr/modules/obds2fhir-rest-compose.yml

@@ -3,7 +3,7 @@ version: "3.7"
 services:
   obds2fhir-rest:
     container_name: bridgehead-obds2fhir-rest
-    image: docker.verbis.dkfz.de/ccp/obds2fhir-rest:main
+    image: docker.verbis.dkfz.de/samply/obds2fhir-rest:main
     environment:
       IDTYPE: BK_${IDMANAGEMENT_FRIENDLY_ID}_L-ID
       MAINZELLISTE_APIKEY: ${IDMANAGER_LOCAL_PATIENTLIST_APIKEY}

kr/vars

@@ -3,7 +3,7 @@ BROKER_URL=https://${BROKER_ID}
 PROXY_ID=${SITE_ID}.${BROKER_ID}
 FOCUS_BEAM_SECRET_SHORT="$(cat /proc/sys/kernel/random/uuid | sed 's/[-]//g' | head -c 20)"
 FOCUS_RETRY_COUNT=${FOCUS_RETRY_COUNT:-64}
-SUPPORT_EMAIL=arturo.macias@dkfz-heidelberg.de
+SUPPORT_EMAIL=p.delpy@dkfz-heidelberg.de
 PRIVATEKEYFILENAME=/etc/bridgehead/pki/${SITE_ID}.priv.pem
 BROKER_URL_FOR_PREREQ=$BROKER_URL
 

lib/functions.sh

@@ -337,7 +337,7 @@ function sync_secrets() {
 }
 
 function secret_sync_gitlab_token() {
-    if [[ "$PROJECT" != "dktk" && "$PROJECT" != "bbmri" ]]; then
+    if [[ "$PROJECT" != "ccp" && "$PROJECT" != "bbmri" ]]; then
         log "INFO" "Not running Secret Sync for project minimal"
         return
     fi

pscc/modules/osiris2fhir-compose.yml

@@ -0,0 +1,15 @@
+services:
+  osiris2fhir:
+    container_name: bridgehead-osiris2fhir
+    image: docker.verbis.dkfz.de/ccp/osiris2fhir
+    environment:
+      FHIR_PROFILE: ${PROJECT:-pscc}
+      LOG_LEVEL: ${LOG_LEVEL:-INFO}
+      SALT: ${LOCAL_SALT}
+    labels:
+      - "traefik.enable=true"
+      - "traefik.http.routers.osiris2fhir.rule=PathPrefix(`/osiris2fhir`)"
+      - "traefik.http.middlewares.osiris2fhir_strip.stripprefix.prefixes=/osiris2fhir"
+      - "traefik.http.services.osiris2fhir.loadbalancer.server.port=8080"
+      - "traefik.http.routers.osiris2fhir.tls=true"
+      - "traefik.http.routers.osiris2fhir.middlewares=osiris2fhir_strip,auth"

pscc/modules/osiris2fhir-setup.sh

@@ -0,0 +1,6 @@
+#!/bin/bash
+if [ -n "$ENABLE_OSIRIS2FHIR" ]; then
+  log INFO "OSIRIS2FHIR-REST setup detected -- will start osiris2fhir module."
+  OVERRIDE+=" -f ./pscc/modules/osiris2fhir-compose.yml"
+  LOCAL_SALT="$(echo \"local-random-salt\" | openssl pkeyutl -sign -inkey /etc/bridgehead/pki/${SITE_ID}.priv.pem | base64 | head -c 30)"
+fi

versions/acceptance

@@ -4,3 +4,4 @@ BLAZE_TAG=0.32
 POSTGRES_TAG=15.13-alpine
 TEILER_DASHBOARD_TAG=develop
 MTBA_TAG=develop
+DATA_QUALITY_AGENT_TAG=latest

versions/prod

@@ -4,3 +4,4 @@ BLAZE_TAG=0.32
 POSTGRES_TAG=15.13-alpine
 TEILER_DASHBOARD_TAG=main
 MTBA_TAG=main
+DATA_QUALITY_AGENT_TAG=0.1

versions/test

@@ -4,3 +4,4 @@ BLAZE_TAG=0.32
 POSTGRES_TAG=15.13-alpine
 TEILER_DASHBOARD_TAG=develop
 MTBA_TAG=develop
+DATA_QUALITY_AGENT_TAG=latest