docs: close Exporter code block in readme (#318)

Co-authored-by: Tobias Kussel <TKussel@users.noreply.github.com>

README.md

@@ -24,6 +24,9 @@ This repository is the starting point for any information and tools you will nee
     - [BBMRI-ERIC Directory entry needed](#bbmri-eric-directory-entry-needed)
     - [Directory sync tool](#directory-sync-tool)
     - [Loading data](#loading-data)
+    - [Teiler (Frontend)](#teiler-frontend)
+    - [Data Exporter Service](#data-exporter-service)
+      - [Data Quality Report](#data-quality-report)
 4. [Things you should know](#things-you-should-know)
     - [Auto-Updates](#auto-updates)
     - [Auto-Backups](#auto-backups)
@@ -379,6 +382,37 @@ Normally, you will need to build your own ETL to feed the Bridgehead. However, t
 
 You can find the profiles for generating FHIR in [Simplifier](https://simplifier.net/bbmri.de/~resources?category=Profile).
 
+### Teiler (Frontend)
+
+Teiler is the web-based frontend of the Bridgehead, providing access to its various internal, and external services and components.   
+To learn how to integrate your custom module into Teiler, please refer to https://github.com/samply/teiler-dashboard.
+- To activate Teiler, set the following environment variable in your `<PROJECT>.conf` file:  
+
+```bash
+ENABLE_TEILER=true
+```
+
+### Data Exporter Service
+
+The Exporter is a dedicated service for extracting and exporting Bridgehead data in (tabular) formats such as Excel, CSV, Opal, JSON, XML, ...  
+- To enable the Exporter service, set the following environment variable in your `<PROJECT>.conf` file:  
+
+```bash
+ENABLE_EXPORTER=true
+```
+
+#### Data Quality Report
+To assess the quality and plausibility of your imported data, the Reporter component is pre-configured to generate Excel reports with data quality metrics and statistical analyses. Reporter is part of the Exporter and can be enabled by setting the same environment variable in your `<PROJECT>.conf` file:
+```bash
+ENABLE_EXPORTER=true
+```
+
+For convenience, it's recommended to enable the Teiler web frontend alongside the Exporter to access export and quality control features via a web interface: set the following environment varibles in your `<PROJECT>.conf` file:
+```bash
+ENABLE_TEILER=true
+ENABLE_EXPORTER=true
+```
+
 ## Things you should know
 
 ### Auto-Updates

bbmri/modules/eric.acc.root.crt.pem

@@ -1,20 +1,20 @@
 -----BEGIN CERTIFICATE-----
-MIIDNTCCAh2gAwIBAgIUE/wu6FmI+KSMOalI65b+lI3HI4cwDQYJKoZIhvcNAQEL
+MIIDNTCCAh2gAwIBAgIUFzdpDi1OLdXyogtCsktHFhCILtMwDQYJKoZIhvcNAQEL
-BQAwFjEUMBIGA1UEAxMLQnJva2VyLVJvb3QwHhcNMjQwOTE2MTUyMzU0WhcNMzQw
+BQAwFjEUMBIGA1UEAxMLQnJva2VyLVJvb3QwHhcNMjUwNjEwMTQzNjE1WhcNMzUw
-OTE0MTUyNDI0WjAWMRQwEgYDVQQDEwtCcm9rZXItUm9vdDCCASIwDQYJKoZIhvcN
+NjA4MTQzNjQ1WjAWMRQwEgYDVQQDEwtCcm9rZXItUm9vdDCCASIwDQYJKoZIhvcN
-AQEBBQADggEPADCCAQoCggEBAOt1I1FQt2bI4Nnjtg8JBYid29cBIkDT4MMb45Jr
+AQEBBQADggEPADCCAQoCggEBALpJCWE9Qe19R9DqotdkPV6jfiuJSKI3UYkCWdWG
-ays24y4R3WO7VJK9UjNduSq/A1jlA0W0A/szDf8Ojq6bBtg+uL92PTDjYH1QXwX0
+nRfkKB6OaY5t3JCHDqaEME9FwSd2nFXhTp5F6snG/K7g8MCLIEzGzuSnrdjGqINq
-c7eMo2tvvyyrs/cb2/ovDBQ1lpibcxVmVAv042ASmil3SdqKKXpv3ATnF9I7V4cv
+zXLfgqnxvQpPR4ARLNNgnKxZaq7m4Q3T/l+QAshK6CnCUWFQ6q5x3g/pZHFP2USd
-fwB56FChaGIov5EK+9JOMjTx6oMlBEgUFR6qq/lSqM9my0HYwUFbX2W+nT9EKEIP
+/G2FtDHX6YK4bHbbnigIPG6PdY2RYy60i30XGdIPBNf82XGkAtPUBz731gHOV5Vg
-9UP1eyfRZR3E/+oticnm/cS20BGCbjoYrNgLthXKyaASuhGoElKs8EZ3h9MiI+u0
+d+jfAqTwZAhYC2CcNmswFw1H9GrvTI/9KZWKcZNUIqemc0A/FyEyONUM18/vjQ7D
-DpR0KpePhAkMLugBrgYWqkMwwD1684LfC4YVQrsLwzo5OW8CAwEAAaN7MHkwDgYD
+lUwOcQsgAg44QTOUPgqXv3sJPQM5EnGuv3yYV9u6Y2i78M8CAwEAAaN7MHkwDgYD
-VR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFPbXs3g3lMjH
+VR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFPrDeNWgtEyZ
-1JMe0a5aVbN7lB92MB8GA1UdIwQYMBaAFPbXs3g3lMjH1JMe0a5aVbN7lB92MBYG
+VM0yeoRZdK2QGjyvMB8GA1UdIwQYMBaAFPrDeNWgtEyZVM0yeoRZdK2QGjyvMBYG
-A1UdEQQPMA2CC0Jyb2tlci1Sb290MA0GCSqGSIb3DQEBCwUAA4IBAQBM5RsXb2HN
+A1UdEQQPMA2CC0Jyb2tlci1Sb290MA0GCSqGSIb3DQEBCwUAA4IBAQAD2S0kqL18
-FpC1mYfocXAn20Zu4d603qmc/IqkiOWbp36pWo+jk1AxejyRS9hEpQalgSnvcRPQ
+laewh+qnyZ0WMq12mLV/Rwll6ZuShCx2uAu3UZuIGWk3l7gG5zlws+i+zbaNcn4o
-1hPEhGU+wvI0WWVi/01iNjVbXmJNPQEouXQWAT17dyp9vqQkPw8LNzpSV/qdPgbT
+HsS3WG9kiNLOMKp8LXGkjErl6RaQr+kb8qgYFTPjOr6v0OdVn6ve9RDNYB5Hd+zE
-Z9o3sZrjUsSLsK7A7Q5ky4ePkiJBaMsHeAD+wqGwpiJ4D2Xhp8e1v36TWM0qt2EA
+9jAWmS8PfS2AldE4VAd0C4pWTAinhnKGrKdn1YAX5x+LMq1y0lc1Pd4CDgsjD6SS
-gySx9isx/jeGGPBmDqYB9BCal5lrihPN56jd+5pCkyXeZqKWiiXFJKXwcwxctYZc
+3td7JtenXqCX0mN0XSeck7vvFGa6QpcQoVcN9tRENctHZTwyeGA21IkXylpFPUkE
-ADHIiTLLPXE8LHTUJAO51it1NAZ1S24aMzax4eWDXcWO7/ybbx5pkYkMd6EqlKHd
+LT60k48fNC8TZkBlfvtVGRebpm5krXIKEaVy5LniEpSuOR4hTqsgoQDntBjW4zHA
-8riQJIhY4huX
+GeWQ1wQNTEBX
 -----END CERTIFICATE-----

bbmri/modules/exporter-compose.yml

@@ -23,6 +23,18 @@ services:
       - "traefik.http.routers.exporter_bbmri.tls=true"
       - "traefik.http.middlewares.exporter_bbmri_strip.stripprefix.prefixes=/bbmri-exporter"
       - "traefik.http.routers.exporter_bbmri.middlewares=exporter_bbmri_strip"
+      # Main router
+      - "traefik.http.routers.exporter_bbmri.priority=20"
+
+      # API router
+      - "traefik.http.routers.exporter_bbmri_api.middlewares=exporter_bbmri_strip,exporter_auth"
+      - "traefik.http.routers.exporter_bbmri_api.rule=PathRegexp(`/bbmri-exporter/.+`)"
+      - "traefik.http.routers.exporter_bbmri_api.tls=true"
+      - "traefik.http.routers.exporter_bbmri_api.priority=25"
+
+      # Shared middlewares
+      - "traefik.http.middlewares.exporter_auth.basicauth.users=${EXPORTER_USER}"
+
     volumes:
       - "/var/cache/bridgehead/bbmri/exporter-files:/app/exporter-files/output"
 
@@ -65,3 +77,10 @@ services:
       - "traefik.http.routers.reporter_bbmri.tls=true"
       - "traefik.http.middlewares.reporter_bbmri_strip.stripprefix.prefixes=/bbmri-reporter"
       - "traefik.http.routers.reporter_bbmri.middlewares=reporter_bbmri_strip"
+      - "traefik.http.routers.reporter_bbmri.priority=20"
+
+      - "traefik.http.routers.reporter_bbmri_api.middlewares=reporter_bbmri_strip,exporter_auth"
+      - "traefik.http.routers.reporter_bbmri_api.rule=PathRegexp(`/bbmri-reporter/.+`)"
+      - "traefik.http.routers.reporter_bbmri_api.tls=true"
+      - "traefik.http.routers.reporter_bbmri_api.priority=25"
+

bbmri/modules/teiler-compose.yml

@@ -68,6 +68,3 @@ services:
       TEILER_DASHBOARD_DE_URL: "https://${HOST}/bbmri-teiler-dashboard/de"
       TEILER_DASHBOARD_EN_URL: "https://${HOST}/bbmri-teiler-dashboard/en"
       HTTP_PROXY: "http://forward_proxy:3128"
-      ENABLE_MTBA: "${ENABLE_MTBA}"
-      ENABLE_DATASHIELD: "${ENABLE_DATASHIELD}"
-      IDMANAGER_UPLOAD_APIKEY: "${IDMANAGER_UPLOAD_APIKEY}" # Only used to check if the ID Manager is active

bbmri/modules/teiler-setup.sh

@@ -5,5 +5,4 @@ if [ "$ENABLE_TEILER" == true ];then
   OVERRIDE+=" -f ./$PROJECT/modules/teiler-compose.yml"
   TEILER_DEFAULT_LANGUAGE=EN
   TEILER_DEFAULT_LANGUAGE_LOWER_CASE=${TEILER_DEFAULT_LANGUAGE,,}
-  # add_public_oidc_redirect_url "/ccp-teiler/*"
 fi

bbmri/vars

@@ -3,7 +3,6 @@ BROKER_URL=https://${BROKER_ID}
 PROXY_ID=${SITE_ID}.${BROKER_ID}
 PRIVATEKEYFILENAME=/etc/bridgehead/pki/${SITE_ID}.priv.pem
 BROKER_URL_FOR_PREREQ=$BROKER_URL
-echo $PRIVATEKEYFILENAME
 
 # Makes sense for all European Biobanks
 : ${ENABLE_ERIC:=true}
@@ -13,8 +12,6 @@ echo $PRIVATEKEYFILENAME
 
 FOCUS_RETRY_COUNT=${FOCUS_RETRY_COUNT:-64}
 
-POSTGRES_TAG=15.6-alpine
-
 for module in $PROJECT/modules/*.sh
 do
     log DEBUG "sourcing $module"

ccp/modules/datashield-compose.yml

@@ -22,7 +22,7 @@ services:
 
   opal:
     container_name: bridgehead-opal
-    image: docker.verbis.dkfz.de/ccp/dktk-opal:test
+    image: docker.verbis.dkfz.de/ccp/dktk-opal:latest
     labels:
       - "traefik.enable=true"
       - "traefik.http.routers.opal_ccp.rule=PathPrefix(`/opal`)"

ccp/modules/teiler-compose.yml

@@ -19,8 +19,7 @@ services:
       HTTP_RELATIVE_PATH: "/ccp-teiler"
 
   teiler-dashboard:
-    #image: docker.verbis.dkfz.de/cache/samply/teiler-dashboard:develop
+    image: docker.verbis.dkfz.de/cache/samply/teiler-dashboard:develop
-    image: samply/teiler-dashboard:develop
     container_name: bridgehead-teiler-dashboard
     labels:
       - "traefik.enable=true"

ccp/vars

@@ -15,8 +15,6 @@ OIDC_PUBLIC_CLIENT_ID=${SITE_ID}-public
 OIDC_URL="https://login.verbis.dkfz.de/realms/test-realm-01"
 OIDC_GROUP_CLAIM="groups"
 
-POSTGRES_TAG=15.6-alpine
-
 for module in $PROJECT/modules/*.sh
 do
     log DEBUG "sourcing $module"

dhki/vars

@@ -8,8 +8,6 @@ PRIVATEKEYFILENAME=/etc/bridgehead/pki/${SITE_ID}.priv.pem
 
 BROKER_URL_FOR_PREREQ=$BROKER_URL
 
-POSTGRES_TAG=15.6-alpine
-
 for module in ccp/modules/*.sh
 do
     log DEBUG "sourcing $module"

lib/functions.sh

@@ -301,34 +301,19 @@ function sync_secrets() {
     if [[ $secret_sync_args == "" ]]; then
         return
     fi
-
-    if [ "$PROJECT" == "bbmri" ]; then
-        # If the project is BBMRI, use the BBMRI-ERIC broker and not the GBN broker
-        proxy_id=$ERIC_PROXY_ID
-        broker_url=$ERIC_BROKER_URL
-        broker_id=$ERIC_BROKER_ID
-        root_crt_file="/srv/docker/bridgehead/bbmri/modules/${ERIC_ROOT_CERT}.root.crt.pem"
-    else
-        proxy_id=$PROXY_ID
-        broker_url=$BROKER_URL
-        broker_id=$BROKER_ID
-        root_crt_file="/srv/docker/bridgehead/$PROJECT/root.crt.pem"
-    fi
-
     mkdir -p /var/cache/bridgehead/secrets/ || fail_and_report 1 "Failed to create '/var/cache/bridgehead/secrets/'. Please run sudo './bridgehead install $PROJECT' again."
     touch /var/cache/bridgehead/secrets/oidc
-    echo $PRIVATEKEYFILENAME
     docker run --rm \
         -v /var/cache/bridgehead/secrets/oidc:/usr/local/cache \
         -v $PRIVATEKEYFILENAME:/run/secrets/privkey.pem:ro \
-        -v $root_crt_file:/run/secrets/root.crt.pem:ro \
+        -v /srv/docker/bridgehead/$PROJECT/root.crt.pem:/run/secrets/root.crt.pem:ro \
         -v /etc/bridgehead/trusted-ca-certs:/conf/trusted-ca-certs:ro \
         -e TLS_CA_CERTIFICATES_DIR=/conf/trusted-ca-certs \
         -e NO_PROXY=localhost,127.0.0.1 \
         -e ALL_PROXY=$HTTPS_PROXY_FULL_URL \
-        -e PROXY_ID=$proxy_id \
+        -e PROXY_ID=$PROXY_ID \
-        -e BROKER_URL=$broker_url \
+        -e BROKER_URL=$BROKER_URL \
-        -e OIDC_PROVIDER=secret-sync-central.central-secret-sync.$broker_id \
+        -e OIDC_PROVIDER=secret-sync-central.central-secret-sync.$BROKER_ID \
         -e SECRET_DEFINITIONS=$secret_sync_args \
         docker.verbis.dkfz.de/cache/samply/secret-sync-local:latest
 

lib/install-bridgehead.sh

@@ -49,6 +49,12 @@ if [ -z "$TRANSFAIR_AUTH" ]; then
   fi
 fi
 
+if [ "$ENABLE_EXPORTER" == "true" ] && [ -z "$EXPORTER_USER" ]; then
+  log "INFO" "Now generating basic auth for the exporter and reporter (see adduser in bridgehead for more information)."
+  generated_passwd="$(cat /proc/sys/kernel/random/uuid | sed 's/[-]//g' | head -c 32)"
+  add_basic_auth_user $PROJECT $generated_passwd "EXPORTER_USER" $PROJECT
+fi
+
 log "INFO" "Registering system units for bridgehead and bridgehead-update"
 cp -v \
     lib/systemd/bridgehead\@.service \

versions/acceptance

@@ -1,3 +1,4 @@
 FOCUS_TAG=develop
 BEAM_TAG=develop
 BLAZE_TAG=main
+POSTGRES_TAG=15.13-alpine

versions/prod

@@ -1,3 +1,4 @@
 FOCUS_TAG=main
 BEAM_TAG=main
 BLAZE_TAG=0.32
+POSTGRES_TAG=15.13-alpine

versions/test

@@ -1,3 +1,4 @@
 FOCUS_TAG=develop
 BEAM_TAG=develop
 BLAZE_TAG=main
+POSTGRES_TAG=15.13-alpine