9.2 KiB
Exporter and Reporter
Exporter
GitHub: https://github.com/samply/exporter
The Exporter is a REST API that enables the export of data from various bridgehead databases as structured tables. It currently supports only FHIR sources such as Blaze, but it is designed to be extended to other types of data sources. The Exporter provides multiple output formats, including CSV, Excel, JSON, and XML, and can also export data directly into Opal (DataSHIELD).
How it works
The user submits a query and specifies the desired export template and output format. The query acts like the WHERE clause in SQL, filtering data, while the template defines what data to select and how to format it, similar to the SELECT clause. The Exporter then processes this to generate the export files.
Exporter Templates
Environment Variables
Below is a list of configurable environment variables used by the Exporter:
| Variable | Default | Description |
|---|---|---|
| APPLICATION_PORT | 8092 | Port on which the application runs. |
| ARCHIVE_EXPIRED_QUERIES_CRON_EXPRESSION | 0 0 2 * * * |
Cron expression for archiving expired queries. |
| CLEAN_TEMP_FILES_CRON_EXPRESSION | 0 0 1 * * * |
Cron expression for cleaning temporary files. |
| CLEAN_WRITE_FILES_CRON_EXPRESSION | 0 0 2 * * * |
Cron expression for cleaning written files. |
| CONVERTER_TEMPLATE_DIRECTORY | Directory containing conversion templates. | |
| CONVERTER_XML_APPLICATION_CONTEXT_PATH | Path to the XML application context used by the converter. | |
| CROSS_ORIGINS | Allowed CORS origins (comma-separated). | |
| CSV_SEPARATOR_REPLACEMENT | Character to replace CSV separators within values. | |
| EXCEL_WORKBOOK_WINDOW | 30000000 | Memory window size for Excel workbook processing. |
| EXPORTER_API_KEY | API key for authenticating access to the exporter. | |
| EXPORTER_DB_FLYWAY_MIGRATION_ENABLED | true | Enable Flyway DB migrations on startup. |
| EXPORTER_DB_PASSWORD | Password for exporter database. | |
| EXPORTER_DB_URL | jdbc:postgresql://localhost:5432/exporter |
JDBC URL for exporter DB. |
| EXPORTER_DB_USER | Username for exporter DB. | |
| FHIR_PACKAGES_DIRECTORY | Directory where FHIR packages are stored. | |
| HAPI_FHIR_CLIENT_LOG_LEVEL | OFF | Log level for HAPI FHIR client. |
| HIBERNATE_LOG | false | Enable Hibernate SQL logging. |
| HTTP_RELATIVE_PATH | Relative base path for HTTP endpoints. | |
| HTTP_SERVLET_REQUEST_SCHEME | http | Default HTTP scheme. |
| LOG_FHIR_VALIDATION | Enable logging of FHIR validation results. | |
| LOG_LEVEL | INFO | Application log level. |
| MAX_NUMBER_OF_EXCEL_ROWS_IN_A_SHEET | 100000 | Max rows per Excel sheet. |
| MAX_NUMBER_OF_RETRIES | 10 | Max retry attempts. |
| MERGE_FILENAME | Name of merged output file. | |
| SITE | Site identifier for filenames/logs. | |
| TEMP_FILES_LIFETIME_IN_DAYS | 1 | Lifetime of temporary files (days). |
| TEMPORAL_FILE_DIRECTORY | Directory for temporary files. | |
| TIMEOUT_IN_SECONDS | 10 | Default timeout (seconds). |
| TIMESTAMP_FORMAT | Timestamp format string. | |
| WEBCLIENT_BUFFER_SIZE_IN_BYTES | 8192 | Buffer size for web client. |
| WEBCLIENT_CONNECTION_TIMEOUT_IN_SECONDS | 5 | Connection timeout (seconds). |
| WEBCLIENT_MAX_NUMBER_OF_RETRIES | 10 | Max retries for web client. |
| WEBCLIENT_REQUEST_TIMEOUT_IN_SECONDS | 10 | Request timeout (seconds). |
| WEBCLIENT_TCP_KEEP_CONNECTION_NUMBER_OF_TRIES | 3 | TCP keepalive retry attempts. |
| WEBCLIENT_TCP_KEEP_IDLE_IN_SECONDS | 30 | TCP keepalive idle time (seconds). |
| WEBCLIENT_TCP_KEEP_INTERVAL_IN_SECONDS | 10 | TCP keepalive probe interval (seconds). |
| WEBCLIENT_TIME_IN_SECONDS_AFTER_RETRY_WITH_FAILURE | 1 | Wait time after failed retry (seconds). |
| WRITE_FILE_DIRECTORY | Directory for final output files. | |
| WRITE_FILES_LIFETIME_IN_DAYS | 30 | Lifetime of written files (days). |
| XML_FILE_MERGER_ROOT_ELEMENT | Containers | Root element for XML file merging. |
| ZIP_FILENAME | exporter-files-${SITE}-${TIMESTAMP}.zip |
Pattern for ZIP archive naming. |
About Cron Expressions in Spring
Cron expressions configure scheduled tasks and consist of six space-separated fields representing second, minute, hour, day of month, month, and day of week. For example, the default 0 0 2 * * * means “at 2:00 AM every day.” These expressions allow precise scheduling for maintenance tasks such as cleaning files or archiving data.
Exporter-DB
GitHub: https://github.com/samply/exporter-db (If exists; if not, just remove or adjust accordingly)
The Exporter-DB stores queries for execution by the Exporter and tracks multiple executions of the same query, managing versioning and scheduling.
Reporter
GitHub: https://github.com/samply/reporter
The Reporter is a plugin for the Exporter designed for generating complex Excel reports based on customizable templates. It supports various template engines like Groovy and Thymeleaf, making it ideal for producing detailed documents such as the traditional CCP data quality report.