OMERO connector for the Huygens Remote Manager (HRM).

The HRM-OMERO connector

This project provides a connector to allow for communication between an HRM (Huygens Remote Manager) and an OMERO server.

Its purpose is to simplify the data transfer by allowing raw images to be downloaded from OMERO as well as uploading deconvolution results back to OMERO directly from within the HRM web interface.

Setup

Installing requirements

NOTE: strictly speaking, Java is only required for uploading data from the HRM to OMERO, so in case for whatever reason you are planning to use the connector in a unidirectional way only you might skip installing the Java packages below. Keep in mind this scenario won't be tested by us though.

CentOS / RHEL 7 and 8

# install the build-time requirements for Python 3.6 and Java 1.8 for Bio-Formats
sudo yum install \
    python36 \
    python36-devel \
    openssl-devel \
    bzip2-devel \
    readline-devel \
    gcc-c++ \
    java-1.8.0-openjdk

# define the target path for the virtual environment:
HRM_OMERO_VENV="/opt/venvs/hrm-omero"

# create a Python 3.6 virtual environment:
python3 -m venv $HRM_OMERO_VENV

# upgrade pip, install wheel:
$HRM_OMERO_VENV/bin/pip install --upgrade pip wheel

Ubuntu 20.04

apt install -y \
    python3-venv \
    openjdk-11-jre-headless

# define the target path for the virtual environment:
HRM_OMERO_VENV="/opt/venvs/hrm-omero"

# create a Python virtual environment:
python3 -m venv $HRM_OMERO_VENV

# upgrade pip, install wheel:
$HRM_OMERO_VENV/bin/pip install --upgrade pip wheel

# install the pre-built Ice wheel from the OME project:
ICE_WHEEL="zeroc_ice-3.6.5-cp38-cp38-linux_x86_64.whl"
wget "https://github.com/ome/zeroc-ice-ubuntu2004/releases/download/0.2.0/$ICE_WHEEL"
$HRM_OMERO_VENV/bin/pip install $ICE_WHEEL

Ubuntu 22.04

apt install -y \
    python3-venv \
    openjdk-11-jre-headless

# define the target path for the virtual environment:
HRM_OMERO_VENV="/opt/venvs/hrm-omero"

# create a Python virtual environment:
python3 -m venv $HRM_OMERO_VENV

# upgrade pip, install wheel:
$HRM_OMERO_VENV/bin/pip install --upgrade pip wheel

# install the pre-built Ice wheel from the OME project:
ICE_WHEEL="zeroc_ice-3.6.5-cp310-cp310-linux_x86_64.whl"
wget "https://github.com/ome/zeroc-ice-py-github-ci/releases/download/0.2.0/$ICE_WHEEL"
$HRM_OMERO_VENV/bin/pip install $ICE_WHEEL

Installing the HRM-OMERO package

# install the connector - please note that it takes quite a while (~15min) as it needs
# to build (compile) the ZeroC Ice bindings:
$HRM_OMERO_VENV/bin/pip install hrm-omero

# from now on you can simply call the connector using its full path, there is no need
# to pre-activate the virtual environment - you could even drop your pyenv completely:
$HRM_OMERO_VENV/bin/ome-hrm --help

# this is even usable as a drop-in replacement for the legacy `ome_hrm.py` script:
cd $PATH_TO_YOUR_HRM_INSTALLATION/bin
mv "ome_hrm.py" "__old__ome_hrm.py"
ln -s "$HRM_OMERO_VENV/bin/ome-hrm" "ome_hrm.py"

Configuration

Add the following lines to /etc/hrm.conf and fill in the desired values:

# Interaction with OMERO (if switched on in hrm/config).
OMERO_HOSTNAME="omero.example.xy"
# OMERO_PORT="4064"
OMERO_CONNECTOR_LOGLEVEL="DEBUG"
# OMERO_CONNECTOR_LOGFILE_DISABLED="true"

On top of that it is necessary to explicitly set two environment variables for the Apache process. By default (at least on recent Ubuntu and CentOS / RHEL versions) the system user running Apache is not allowed to write to its $HOME directory for security reasons. Therefore it is required to specify where the OMERO Python bindings and also Java may store cache files and preferences. This can be done by running the following command:

systemctl edit apache2.service  # Debian / Ubuntu
systemctl edit httpd.service  # CentOS / RHEL / AlmaLinux

There, add the following section, adjusting the path if desired:

[Service]
Environment=OMERO_USERDIR=/var/cache/omero
Environment=JAVA_OPTS="-Djava.util.prefs.userRoot=/var/cache/omero/javaUserRoot"

Now make sure the specified directory exists and is writable by the Apache system user:

mkdir -v /var/cache/omero
chown www-data:www-data /var/cache/omero  # Debian / Ubuntu
chown apache:apache /var/cache/omero  # CentOS / RHEL / AlmaLinux

Finally, restart Apache by running the respective systemctl command from above while replacing edit for restart.

Debugging

The connector will try to place log messages in a file in the directory specified as $HRM_LOG in the HRM configuration file unless a configuration option named $OMERO_CONNECTOR_LOGFILE_DISABLED is present and non-empty. In a standard setup this will result in the log file being /var/log/hrm/omero-connector.log.

In addtion, log messages produced by the connector when called by HRM will be sent to stderr, which usually means they will end up in the web server's error log.

By default the connector will be rather silent as otherwise the log files will be cluttered up quite a bit on a production system. However, it is possible to increase the log level by specifying -v, -vv and so on.

Since this is not useful when being operated through the HRM web interface (which is the default) it's also possible to set the verbosity level by adjusting the OMERO_CONNECTOR_LOGLEVEL in /etc/hrm.conf.

Valid settings are "SUCCESS", "INFO", "DEBUG" and "TRACE". If the option is commented out in the configuration file, the level will be set to WARNING.

Example Usage

Store username and password in variables, export the OMERO_PASSWORD variable:

read OMERO_USER
read -s OMERO_PASSWORD
export OMERO_PASSWORD   # use 'set --export OMERO_PASSWORD $OMERO_PASSWORD' for fish

Verifying Credentials

ome-hrm \
    --user $OMERO_USER \
    checkCredentials

Fetching OMERO tree information

Set the --id parameter according to what part of the tree should be retrieved:

OMERO_ID="ROOT"                # fetches the base tree view for the current user
OMERO_ID="G:4:Experimenter:9"  # fetches the projects of user '9' in group '4'
OMERO_ID="G:4:Project:12345"   # fetches the datasets of project '12345'
OMERO_ID="G:4:Dataset:65432"   # lists the images of dataset '65432'

Then run the actual command to fetch the information, the result will be a JSON tree:

ome-hrm \
    --user $OMERO_USER \
    retrieveChildren \
    --id "$OMERO_ID"

For example this could be the output when requesting "G:4:Dataset:65432":

[
    {
        "children": [],
        "class": "Image",
        "id": "G:4:Image:1311448",
        "label": "4321_mko_ctx_77.tif",
        "owner": "somebody"
    },
    {
        "children": [],
        "class": "Image",
        "id": "G:4:Image:1566150",
        "label": "test-image.tif",
        "owner": "somebody"
    }
]

Downloading an image from OMERO

This will fetch the second image from the example tree above and store it in /tmp/:

ome-hrm \
    --user $OMERO_USER \
    OMEROtoHRM \
    --imageid "G:4:Image:1566150" \
    --dest /tmp/

Uploading an image from the local file system to OMERO

The command below will import a local image file into the example dataset from above:

ome-hrm \
    --user $OMERO_USER \
    HRMtoOMERO \
    --dset "G:4:Dataset:65432" \
    --file test-image.tif

Developing the HRM-OMERO connector

Using poetry

The project is using poetry for packaging and dependency management. To set up a development environment use this command, it will set up a fresh virtual environment with the correct dependencies and install the project in editable mode:

git clone https://github.com/imcf/hrm-omero
cd hrm-omero
poetry install

Installing a pre-release from TestPyPI

To make dependency resolution work with the test repository a command like this can be used:

pip install \
    -i https://test.pypi.org/simple/ \
    --extra-index-url https://pypi.org/simple \
    hrm-omero==0.4.0.dev1

Updating local dev wheels

One common scenario is to work on the source tree using your local Desktop and manually testing interaction with the HRM's web interface in a container by (repeatedly) creating wheel packages through poetry and pip-installing them inside the container.

However, when installing a wheel with the same version (e.g. 0.4.0-dev7) multiple times subsequently, pip's --force option has to be used. Unfortunately this will also try to re-build the Ice wheel from scratch which is quite a time-consuming process. To avoid this, you can simply use the locally cached Ice wheel and supply it along the command to "force-reinstall" the hrm-omero wheel, e.g. like this:

# find the previously built and cached Ice wheel:
ICE_WHEEL=$(find .cache/pip/wheels/ | grep -i ice)
# move it out of the cache to preserve it and have a shorter path:
mv -v "$ICE_WHEEL" .

# use 'pip' to install the new hrm-omero wheel while using the previous Ice:
/opt/venvs/hrm-omero/bin/pip install \
    --upgrade \
    --force \
    zeroc_ice-3.6.5-cp36-cp36m-linux_x86_64.whl \
    hrm_omero-0.4.0.dev7-py3-none-any.whl

Testing

Testing is done through pytest and can be triggered by running this command:

poetry run pytest

By default only local tests will be performed, all tests that require the connection to an actual OMERO instance are disabled. See the next section for instructions on testing with an OMERO server.

Testing against an OMERO instance

Please see the following sub-sections on how to set up and prepare an OMERO installation that can serve for running the full suite of hrm-omero tests.

To activate the online tests interacting with an OMERO instance you need to pass the --online flag to the pytest call and a corresponding file site_specific.py needs to be present in tests/online/settings/ that contains the settings on how to connect to your OMERO, plus the mapping of object and user IDs. See tests/resources/settings/ for a template file, copy it to the repository root and make your adjustments there. Then run this command:

poetry run pytest --online

NOTE: each online test does a connectivity check first, if the OMERO server specified in the site-specific settings can't be reached they will be skipped (instead of generating tons of failed test results).

Setting up OMERO

It is recommended to test against an isolated OMERO instance to avoid messing up any production data. The script in resources/omero-recipes/setup-ubuntu-20-04.sh can be used to install OMERO quickly on Ubuntu 20.04. A reasonable approach would be this:

• Create a VM / container with a basic Ubuntu 20.04 installation.
• Copy the OMERO-setup-script into the VM / container.
• Run the setup script from within the VM / container.

⚠️ It is strongly recommended to snapshot your container / VM at this point, so you can easily go back to the initial ("fresh") OMERO state later on. ⚠️

Preparing OMERO for testing

Once OMERO is running, you can use the preparation script to create the required groups, users, projects, ... in that OMERO instance. Please note that the preparation script is meant to be run from the HRM-OMERO development environment (i.e. not from within your shiny new OMERO-VM)!

Please note that values for the OMERO server address, passwords for the omero root user and the additional users created during the preparation can be pre-seeded by providing an input file, see resources/scripts/omero-seeds.inc.sh for an example.

Assuming you copied the seeds file to my-omero-seeds.inc.sh, you can simply run the preparation script like this:

# get yourself a poetry shell that has access to the `omero` command:
poetry shell

# run the OMERO preparation script:
bash resources/scripts/prepare-omero-for-testing.sh \
    resources/scripts/my-omero-seeds.inc.sh \
    tests/resources/settings/site_specific.yml

Generating Documentation

The project is using pdoc for generating API documentation. To update or (re-) generate the HTML documentation use this poetry command:

poetry run pdoc --docformat numpy --output-directory docs/ src/hrm_omero/

ToDo list

• sort tree levels (users, projects, ...) alphabetically, by default they are returned in the same order as OMERO is iterating them
• use "key-value pairs" for the HRM job parameter summaries
• trees for different groups
  • (REJECTED) requesting groups through a commandline option
• logging
  • proper logging (loguru)
  • separate logfile for the connector
  • adjust log verbosity through a parameter
  • adjust log verbosity through the configuration file
• allow debug logging of the "omero import" call
  • (REJECTED) requested through a command line argument
  • through the configuration file
• (REJECTED) offer download of OME-TIFFs
• don't use a command line parameter for the OMERO password
• lift assumption that datasets are always members of a project in OMERO
• create target directory for thumbnails in case it doesn't exist
• create documentation on the OMERO_USERDIR configuration
• make log verbosity setting optional in the configuration file, otherwise Adding a file sink for logging failed: Level '' does not exist is issued from hrm_omero.cli:logger_add_file_sink if nothing is present
• retain structure of complex multi-file datasets like Olympus .vsi that expects files to be in specific sub-directories etc.
• add group-write permissions to files and directories created by the connector

Planned features

• add command line action to create new projects and datasets in OMERO
• speed up tree-loading by avoiding the necessary repeated calls to the connector
• catch "assert failed" errors when uploading data to OMERO, this usually indicates that the file / fileset was refused by Bio-Formats

Changelog

Planned for 1.0.0

New in 1.0.0

Changes in 1.0.0

• The previously deprecated option of providing the password through a command line argument has been removed.

Fixes in 1.0.0

0.4.0

New in 0.4.0

• An environment variable OMERO_PASSWORD can (and should!) now be used to supply the sensitive part of the user credentials that are necessary to connect to OMERO. This avoids having the password as plain-text in the system's process list (e.g. when calling ps fu -e or similar) and also prevents it from showing up in an annotated stack trace in case an uncaught exception is raised.
• Log level of the HRM-OMERO connector itself can now be set through the configuration option OMERO_CONNECTOR_LOGLEVEL in the HRM config file.
• Log messages of the connector will also be placed in a separate log file in the default location defined via HRM_LOG in the HRM config file. This can be disabled by setting OMERO_CONNECTOR_LOGFILE_DISABLED=true.
• Debug logging for the OMERO import call can now be requested be setting the configuration option OMERO_DEBUG_LOG in the HRM config file.
• hrm_omero.hrm.parse_summary() has been added to provide a function for parsing the parameter summary from an HRM job into a (nested) dict.
• hrm_omero.hrm.parse_job_basename() has been added to identify the base name of all files that belong to the set of results from a certain HRM job.
• hrm_omero.omero.add_annotation_keyvalue() can be used to add so-called Map Annotations (key-value pairs) to objects in OMERO.
• hrm_omero.omero.find_recently_imported() tries to identify an image in OMERO using the import timestamp and the object label as criteria.
• A decorator hrm_omero.decorators.connect_and_set_group() is now available that can be used with functions that require a valid connection plus an OMERO object identifier pair (submitted on the command line as "G:4:Image:12345" or similar, passed on to the decorated function as obj_type and obj_id).
• hrm_omero.misc.printlog() can be used to push a message to the log and stdout.
• A class hrm_omero.misc.OmeroId provides parsing, validation and access to all group-qualified OMERO object IDs (commonly passed as function parameter id_str throughout the existing code).
• The CLI has a new optional parameter --dry-run that prevents any action from being performed, instead the name of the function and the corresponding parameters that would be called are printed to stdout.
• Unit tests using pytest and pytest-cov.

Changes in 0.4.0

• Uploading images to OMERO into another group than the user's default one is now properly supported.
• HRM job parameter summaries are now being added as OMERO "Map Annotations" instead of simply being a comment string.
• Formats like e.g. Olympus .vsi requiring their files to be placed in a specific directory structure are now properly supported.
• Tree levels (users, projects, ...) are now sorted alphabetically instead of the (seemingly) random order as they are returned by OMERO with the exception of the current user always being listed first on the group level, followed by their colleagues.
• Datasets in OMERO that are not associated to a Project will no longer be ignored but rather be shown on the same level than the user's projects, just as OMERO.web and OMERO.insight are doing it.
• The target directory for downloading preview thumbnails from OMERO will be created automatically in case it doesn't exist yet.
• The command line parameter --password (or -w) is now deprecated in favor of using the environment variable described above.
• The following functions are now deprecated and will be removed in a future release.
  • hrm_omero.omero.connect()
  • hrm_omero.cli.parse_arguments()
• Functions hrm_omero.transfer.to_omero() and hrm_omero.transfer.from_omero() are now raising a ValueError in case the provided id_str is malformed.
• Various improvements on log messages.
• Unit string literal µm is no longer converted to um in parameter summaries.
• New dependencies: Pillow

Fixes in 0.4.0

• Thumbnail download has been adapted to recent code changes in OMERO.

API Changes

The tables below describe the API changes compared to the "integrated" HRM-OMERO connector that used to be shipped with the HRM itself previously.

hrm_config

ome_hrm