hrm_omero

OMERO connector for the Huygens Remote Manager (HRM).

The HRM-OMERO connector

This project provides a connector to allow for communication between an HRM (Huygens Remote Manager) and an OMERO server.

Its purpose is to simplify the data transfer by allowing raw images to be downloaded from OMERO as well as uploading deconvolution results back to OMERO directly from within the HRM web interface.

Installation

Distributions that include Python 3.6 or newer (e.g. Ubuntu 20.04)

pip install hrm-omero

CentOS 6

CentOS 6 is EOL since 2020-11-30, so you should really consider upgrading to a newer release. However, we know that sometimes this is not easily doable due to dependencies, hardware support or whatever reason - so here are instructions to make the connector work on that old distribution.

It's strongly recommended to use pyenv for installing Python 3.6, which is the absolute minimum for using the HRM-OMERO connector (or its actual dependencies, to be fully correct). In case you don't want pyenv to mess with your system setup, you can simply ask it to install that version somewhere and then only create a virtual environment from it using the --copies flag - this will result in a standalone setup that won't affect anything else on the system.

# install the build-time requirements for Python 3.6 and Java 1.8 for Bio-Formats
sudo yum install openssl-devel bzip2-devel readline-devel gcc-c++ java-1.8.0-openjdk

# get pyenv and put it into your home directory or wherever you prefer it to be
git clone https://github.com/pyenv/pyenv.git ~/.pyenv

# activate pyenv *FOR THIS SHELL ONLY* (needs to be done whenever you want to use it)
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init --path)"
eval "$(pyenv init -)"

# ask pyenv to install Python 3.6.15 (will end up in "~/.pyenv/versions/3.6.15/")
pyenv install 3.6.15  # takes a bit (compiling...)

# define the target path for the virtual environment:
$HRM_OMERO_VENV="/opt/venvs/hrm-omero"

# create a bare, stand-alone Python 3.6 virtual environment
~/.pyenv/versions/3.6.15/bin/python -m venv --copies $HRM_OMERO_VENV

# now you can install the connector into this virtual environment - please note that the
# installation takes quite a while (~15min) as it needs to build the ZeroC Ice bindings
$HRM_OMERO_VENV/bin/pip install hrm-omero

# from now on you can simply call the connector using its full path, there is no need
# to pre-activate the virtual environment - you could even drop your pyenv completely:
$HRM_OMERO_VENV/bin/ome-hrm --help

# this is even usable as a drop-in replacement for the legacy `ome_hrm.py` script:
cd $PATH_TO_YOUR_HRM_INSTALLATION/bin
mv "ome_hrm.py" "__old__ome_hrm.py"
ln -s "$HRM_OMERO_VENV/bin/ome-hrm" "ome_hrm.py"

Debugging

By default the connector will be rather silent as otherwise the log files will be cluttered up quite a bit on a production system. However, it is possible to increase the log level by specifying -v, -vv and so on.

Since this is not useful when being operated through the HRM web interface (which is the default) it's also possible to set the verbosity level by adjusting the OMERO_CONNECTOR_LOGLEVEL in /etc/hrm.conf.

Valid settings are "SUCCESS", "INFO", "DEBUG" and "TRACE". If the option is commented out in the configuration file, the level will be set to WARNING. Log messages produced by the connector when called by HRM will usually end up in the web server's error log (as they go to stderr).

Example Usage

Store username and password in variables, export the OMERO_PASSWORD variable:

read OMERO_USER
read -s OMERO_PASSWORD
export OMERO_PASSWORD   # use 'set --export OMERO_PASSWORD $OMERO_PASSWORD' for fish

Verifying Credentials

ome-hrm \
    --user $OMERO_USER \
    checkCredentials

Fetching OMERO tree information

Set the --id parameter according to what part of the tree should be retrieved:

OMERO_ID="ROOT"                # fetches the base tree view for the current user
OMERO_ID="G:4:Experimenter:9"  # fetches the projects of user '9' in group '4'
OMERO_ID="G:4:Project:12345"   # fetches the datasets of project '12345'
OMERO_ID="G:4:Dataset:65432"   # lists the images of dataset '65432'

Then run the actual command to fetch the information, the result will be a JSON tree:

ome-hrm \
    --user $OMERO_USER \
    retrieveChildren \
    --id "$OMERO_ID"

For example this could be the output when requesting "G:4:Dataset:65432":

[
    {
        "children": [],
        "class": "Image",
        "id": "G:4:Image:1311448",
        "label": "4321_mko_ctx_77.tif",
        "owner": "somebody"
    },
    {
        "children": [],
        "class": "Image",
        "id": "G:4:Image:1566150",
        "label": "test-image.tif",
        "owner": "somebody"
    }
]

Downloading an image from OMERO

This will fetch the second image from the example tree above and store it in /tmp/:

ome-hrm \
    --user $OMERO_USER \
    OMEROtoHRM \
    --imageid "G:4:Image:1566150" \
    --dest /tmp/

Uploading an image from the local file system to OMERO

The command below will import a local image file into the example dataset from above:

ome-hrm \
    --user $OMERO_USER \
    HRMtoOMERO \
    --dset "G:4:Dataset:65432" \
    --file test-image.tif

Developing the HRM-OMERO connector

Using poetry

The project is using poetry for packaging and dependency management. To set up a development environment use this command, it will set up a fresh virtual environment with the correct dependencies and install the project in editable mode:

git clone https://github.com/imcf/hrm-omero
cd hrm-omero
poetry install

Installing a pre-release from TestPyPI

To make dependency resolution work with the test repository a command like this can be used:

pip install \
    -i https://test.pypi.org/simple/ \
    --extra-index-url https://pypi.org/simple \
    hrm-omero==0.4.0.dev1

Testing

Testing is done through pytest and can be triggered by running this command:

poetry run pytest

By default only local tests will be performed, all tests that require the connection to an actual OMERO instance are disabled. To activate them you need to pass the --online flag to the pytest call and a corresponding file omero_test_settings.py needs to be present in the repository root that contains the settings on how to connect to your OMERO, plus the information on what to test. See tests/resources/settings/ for an example file, copy it to the repository root and make your adjustments there. Then run this command:

poetry run pytest --online

Prepare an OMERO instance for tests

See the example script in resources/scripts/prepare-omero-for-testing.sh on how to set up an OMERO instance so it can be used with the --online tests. Note that currently the settings file will still need to be adjusted for user and object IDs!

Generating Documentation

The project is using pdoc for generating API documentation. To update or (re-) generate the HTML documentation use this poetry command:

poetry run pdoc --docformat numpy --output-directory docs/ src/hrm_omero/

ToDo list

  • use "key-value pairs" for the HRM job parameter summaries
  • trees for different groups
    • (REJECTED) requesting groups through a commandline option
  • logging
    • proper logging (loguru)
    • separate logfile for the connector
    • adjust log verbosity through a parameter
    • adjust log verbosity through the configuration file
  • allow debug logging of the "omero import" call
    • (REJECTED) requested through a command line argument
    • through the configuration file
  • (REJECTED) offer download of OME-TIFFs
  • don't use a command line parameter for the OMERO password
  • add command line action to create new projects and datasets in OMERO
  • lift assumption that datasets are always members of a project in OMERO

Changelog

0.4.0

New

  • An environment variable OMERO_PASSWORD can (and should!) now be used to supply the sensitive part of the user credentials that are necessary to connect to OMERO. This avoids having the password as plain-text in the system's process list (e.g. when calling ps fu -e or similar) and also prevents it from showing up in an annotated stack trace in case an uncaught exception is raised.
  • Log level of the HRM-OMERO connector itself can now be set through the configuration option OMERO_CONNECTOR_LOGLEVEL in the HRM config file.
  • Debug logging for the OMERO import call can now be requested be setting the configuration option OMERO_DEBUG_LOG in the HRM config file.
  • hrm_omero.hrm.parse_summary() has been added to provide a function for parsing the parameter summary from an HRM job into a (nested) dict.
  • hrm_omero.hrm.parse_job_basename() has been added to identify the base name of all files that belong to the set of results from a certain HRM job.
  • hrm_omero.omero.add_annotation_keyvalue() can be used to add so-called Map Annotations (key-value pairs) to objects in OMERO.
  • A decorator hrm_omero.decorators.connect_and_set_group() is now available that can be used with functions that require a valid connection plus an OMERO object identifier pair (submitted on the command line as "G:4:Image:12345" or similar, passed on to the decorated function as obj_type and obj_id).
  • hrm_omero.misc.printlog() can be used to push a message to the log and stdout.
  • A class hrm_omero.misc.OmeroId provides parsing, validation and access to all group-qualified OMERO object IDs (commonly passed as function parameter id_str throughout the existing code).
  • The CLI has a new optional parameter --dry-run that prevents any action from being performed, instead the name of the function and the corresponding parameters that would be called are printed to stdout.
  • Unit tests using pytest and pytest-cov (incomplete).

Changes

  • Uploading images to OMERO into another group than the user's default one is now properly supported.
  • HRM job parameter summaries are now being added as OMERO "Map Annotations" instead of simply being a comment string.
  • The command line parameter --password (or -w) is now deprecated in favor of using the environment variable described above.
  • The following functions are now deprecated and will be removed in a future release.
  • Functions hrm_omero.transfer.to_omero() and hrm_omero.transfer.from_omero() are now raising a ValueError in case the provided id_str is malformed.
  • Various improvements on log messages.
  • Unit string literal ┬Ám is no longer converted to um in parameter summaries.
  • New dependencies: Pillow

Fixes

  • Thumbnail download has been adapted to recent code changes in OMERO.

API Changes

The tables below describe the API changes compared to the "integrated" HRM-OMERO connector that used to be shipped with the HRM itself previously.

hrm_config

hrm_config hrm_omero
parse_hrm_conf() hrm.parse_config()
check_hrm_conf() hrm.check_config()

ome_hrm

ome_hrm hrm_omero
omero_login() omero.connect()
check_credentials() omero.check_credentials()
gen_obj_dict() tree.gen_obj_dict()
gen_children() tree.gen_children()
gen_base_tree() tree.gen_base_tree()
gen_group_tree() tree.gen_group_tree()
tree_to_json() formatting.tree_to_json()
print_children_json() formatting.print_children_json()
gen_parameter_summary() hrm.job_parameter_summary()
omero_to_hrm() transfer.from_omero()
hrm_to_omero() transfer.to_omero()
download_thumb() transfer.fetch_thumbnail()
bool_to_exitstatus() cli.bool_to_exitstatus()
parse_arguments() cli.parse_arguments()
main() cli.run_task()
View Source
"""OMERO connector for the Huygens Remote Manager (HRM).

.. include:: ../../README.md

.. include:: ../../DEVELOPMENT.md

.. include:: ../../CHANGELOG.md

.. include:: ../../API-CHANGES.md
"""

__version__ = "0.4.0-dev2"