pyppms

The pyppms package provides a Python interface to Stratocore's PUMAPI.

PyPPMS

PUMAPI - Python Interface

Stratocore's PPMS booking system offers an API (the so-called PUMAPI, short for PPMS Utility Management API) for fetching information from the booking system as well as changing its state and properties.

This is a Python 3 package for talking to the PUMAPI.

Usage Example

Fetch email addresses of all active users:

from pyppms import ppms
from credentials_ppms import PPMS_URL, PPMS_API_KEY

conn = ppms.PpmsConnection(PPMS_URL, PPMS_API_KEY)

print("Querying PPMS for emails of active users, can take minutes...")
emails = ppms.get_users_emails(active=True)
print(f"Got {len(emails)} email addresses from PPMS:")
print("\n".join(emails))

Testing

Automated testing is described in the TESTING document on github.

Note

The PPMS API sometimes exposes a bit of a surprising behavior. During development of the package, we came across several issues (this list is certainly incomplete):

  • HTTP status return code is always 200, even on failed authentication.
  • Results of queries are a mixture of CSV (with headers) and and text with newlines (with no headers and therefore without structural information on the data). JSON is implemented in some cases only.
  • The CSV headers sometimes do contain spaces between the colons, sometimes they don't.
  • Some fields are quoted in the CSV output, some are not. Difficult to separate the values since there are colons in the values too.
  • Semantics of keys is not consistent. Sometimes user is the user id, sometimes it refers to the user's full name.
  • Using an invalid permission level (e.g. Z) with the setright action is silently ignored by PUMAPI, the response is still done even though this doesn't make any sense.
  • There is no (obvious) robust way to derive the user id from the user's full name that is returned e.g. by getrunningsheet, making it very hard to cross-reference it with data from getuser.
  • The result of the getrunningsheet query in general is not suited very well for automated processing, it seems to be rather tailored for humans and subject to (mis-) interpretation.
  • Unfortunately Username and Systemname are not the unique id, they are rather the full description. Therefore sometimes looping over all users and systems is necessary.
  • Some results have a very strange format - for example, the starting time of the next booking is given as minutes from now instead of an absolute time.
  • Official documentation is rather rudimentary, i.e. it contains almost no information on what is returned in case wrong / invalid parameters are supplied and similar situations.

References

Testing PyPPMS

Automated testing has been a core design goal for pyppms, aiming for a coverage of 100%. Testing of the project is performed through pytest.

Concept of PyPPMS Unit Tests

As proper testing of an HTTP-based API will require interaction with a real instance of that given API the complete suite of tests will only be able to run if you're having access to PPMS somewhere. Obviously, this should not be done on a production instance but a separate test setup (contact Stratocore to get one).

To speed up testing, make it more convenient and provide a certain level of robustness against silent changes of the PPMS API, tests are split into more or less three categories:

  • local unit tests - they don't need a PPMS / PUMAPI instance
  • tests using cached responses from a real PPMS / PUMAPI
  • tests using mocked responses to simulate specific behavior of PPMS that cannot be triggered otherwise

Development installation through poetry

The project is using poetry for packaging and dependency management. To set up a development environment and prepare for testing use the command below, it will set up a fresh virtual environment with the correct dependencies and install the project in editable mode:

git clone https://github.com/imcf/pyppms
cd pyppms
poetry install

Cached Testing

Almost all of the request-response tests, which is basically anything in the PpmsConnection class, do NOT require a valid API-key or a connection to a PUMAPI instance. Instead, they can be performed using the built-in response-caching mechanism combined with the mocks and cached responses provided with the repository.

Using the cache

Working with the provided cached responses is the default when running the tests. The only exception are those tests that do not make sense in such a scenario (i.e. that do test if interaction with an actual PUMAPI instance is effectively working). Those tests have to be requested explicitly by adding the "--online" flag to the pytest-call.

Validating the cache

Validating or re-building the cache requires access to an actual PUMAPI, see the section on running online tests below for details.

Configuration and API Key

To run the tests, copy the example pyppmsconf.py file to the /tests/ directory. For the online tests, please edit it according to your instance and key - the offline tests will work without modifying the config.

cp -v resources/examples/pyppmsconf.py tests/

To generate an API key a so-called "Super-Admin" needs to log on to your PPMS instance, navigate to My PPMS using the drop-down menu on the top-right, select API from the top bar and finally hit the Create PUMAPI key button.

Running Tests

Once everything is set up, you should be good to simply type poetry run pytest on the command line, the output should look something like this:

poetry run pytest
============================ test session starts =============================
platform linux -- Python 3.8.10, pytest-7.1.1, pluggy-1.0.0
cachedir: .pytest_cache
rootdir: /tmp/imcf/pyppms, configfile: pyproject.toml
plugins: cov-3.0.0
collected 43 items

tests/test_booking.py .........                                        [ 20%]
tests/test_common.py ....                                              [ 30%]
tests/test_ppms.py s.s.......................                          [ 90%]
tests/test_system.py ..                                                [ 95%]
tests/test_user.py ..                                                  [100%]

========================== short test summary info ===========================
SKIPPED [1] tests/test_ppms.py:95: need --online option to run
SKIPPED [1] tests/test_ppms.py:108: need --online option to run
======================= 41 passed, 2 skipped in 0.14s ========================

Running Online Tests

To run those tests requiring access to a real PUMAPI instance in addition to the default ones, simply add the --online flag to the pytest command above. Obviously you will need to have valid settings for PUMAPI_URL and PPMS_API_KEY in the config file used for testing.

Please note that this will still run the majority of tests using the cached / mocked responses!

However, having a cache of the expected responses from PUMAPI for a given query (or a series of queries) allows for checking if the behavior of the API has silently changed by simply deleting the cache and re-building it afterwards. To do so, the following steps are required:

  • preparing your test instance of PPMS - unfortunately this is a manual operation, but it has to be done only once (unless Stratocore resets your test instance)
  • removing the cache
  • running the tests in online mode to re-populate the cache
  • filtering / checking / validating the results

Those steps are described in details in the following sections.

PPMS Preparations

As the tests assume certain users and systems to exist in the PUMAPI instance used for testing, your test instance needs to be prepared accordingly. Currently there is not yet a mechanism to automatically create those items unfortunately - sorry, might come at some point...

For now, simply run the following command to see what needs to be configured in your PPMS, then log into the web interface with your browser and manually create the required items:

poetry run python tests/show_required_ppms_values.py

Remarks / additional details:

  • All created users should be members of the previously created group.
  • After creating the admin user, go to the Admins page in PPMS, hit the Create a new administrator button, then select the correct facility and pick the user account. In options, simply check the System management box, then click Create administrator.
  • After creating all users, navigate to the Rights page, select the newly created system and pick the regular user, then hit the Create button to assign booking permissions to the user account. Then repeat this for the admin user and the inactive user (account needs to be set to active for adding the permissions).
  • In addition to the above, four bookings for the regular user on the created system need to be made on 2028-12-24:
    • from 09:00 to 10:00
    • from 11:00 to 12:00
    • from 13:00 to 14:00
    • from 15:00 to 16:00

Removing the Cache

As easy as running:

rm -r tests/cached_responses

Re-populating and validating the Cache

NOTE: As a test-instance of PPMS usually is a clone of a real one it will contain many more but the previously created objects. Therefore when re-populating the cache from a real PPMS instance a few filtering steps have to be done to validate the new cache files and ignore those "unpredictable" ("instance-specific") elements.

First run the most time-consuming tests that will fetch all users from your PPMS (this can easily take several minutes, depending on your PPMS instance):

poetry run pytest --online tests/test_ppms.py::test_get_users
poetry run pytest --online tests/test_ppms.py::test_get_admins

As a result, the tests/cached_responses/stage_0/getuser/ directory will be cluttered up with plenty of files from users in your PPMS instance that the cache doesn't know about (and also shouldn't). To clean this, simply remove all corresponding response-cache files untracked by git:

git clean -f tests/cached_responses/stage_0/getuser/

Now the freshly re-created response files need to be checked if they contain all the expected values while discarding / ignoring the additional ones introduced by your specific PPMS instance. To simplify this task use this shortcut function (bash) to show the git diff of a file while discarding all lines that were added to it (as they are specific to your instance):

filternew() {
    git diff --no-color "$1" | grep -v '^+' | tail -n +5
}

First, this needs to be done for the files created by the two tests from above (active users and admins). Run the command and compare the output that is expected to look as shown here:

filternew "tests/cached_responses/stage_0/getusers/active--true.txt"
 pyppms
 pyppms-adm

filternew "tests/cached_responses/stage_0/getadmins/response.txt"
 pyppms-adm

If the output matches, discard the changes to those files:

git restore \
  "tests/cached_responses/stage_0/getusers/active--true.txt" \
  "tests/cached_responses/stage_0/getadmins/response.txt"

Now run all --online tests - with the just (re-)created cache files for the users and admins, this should only take a few seconds:

poetry run pytest --online

Then, check the remaining re-created cache files for their content:

filternew "tests/cached_responses/stage_0/getusers/response.txt"
 pyppms
 pyppms-adm
 pyppms-deact

filternew "tests/cached_responses/stage_0/getgroups/response.txt"
 pyppms_group

filternew "tests/cached_responses/stage_0/getsysrights/id--*"
 A:pyppms
 A:pyppms-adm
 D:pyppms-deact
 S:pyppms-adm

filternew "tests/cached_responses/stage_1/getsysrights/id--*"
 D:pyppms
 A:pyppms-adm
 D:pyppms-deact
 S:pyppms-adm


filternew "tests/cached_responses/stage_2/getsysrights/id--*"
 D:pyppms
 A:pyppms-adm
 D:pyppms-deact
 S:pyppms-adm

Do the same for the systems and user experience responses, taking into account that the system ID will differ in your case, those lines will then show as missing in the diff:

filternew "tests/cached_responses/stage_0/getsystems/response.txt"
 Core facility ref,System id,Type,Name,Localisation,Active,Schedules,Stats,Bookable,Autonomy Required,Autonomy Required After Hours
 2,69,"Virtualized Workstation","Python Development System","VDI (Development)",True,True,True,True,True,False

filternew "tests/cached_responses/stage_0/getuserexp/response.txt"
 login,id,booked_hours,used_hours,last_res,last_train
 "pyppms",69,0,0,n/a,n/a
 "pyppms-adm",69,0,0,n/a,n/a
 "pyppms-deact",69,0,0,n/a,n/a

The last one to check is the response for the nextbooking query, which will differ in the two additional lines for the remaining time and the session, so the result should look something like this:

filternew "tests/cached_responses/stage_0/nextbooking/id--*"
 pyppms
-303520
-31432

If all output matches, discard the changes to those files:

git restore \
  "tests/cached_responses/stage_0/getusers/response.txt" \
  "tests/cached_responses/stage_0/getgroups/response.txt" \
  "tests/cached_responses/stage_0/getsysrights/id--*" \
  "tests/cached_responses/stage_1/getsysrights/id--*" \
  "tests/cached_responses/stage_2/getsysrights/id--*" \
  "tests/cached_responses/stage_0/getsystems/response.txt" \
  "tests/cached_responses/stage_0/getuserexp/response.txt" \
  "tests/cached_responses/stage_0/nextbooking/id--*"

PyPPMS Changelog

NOTE: potentially breaking changes are flagged with a 🧨 symbol.

3.3.0

Added

3.2.1

Fixed

  • 🕛🌃 end time: pyppms.booking.PpmsBooking.endtime_fromstr() contained a bug where the end time of a booking finishing at midnight got wrongly assigned to the start of the given day (instead of the end). This is now fixed by setting the end time to the start of the following day.

3.2.0

Added

Changed

  • Several log messages have been demoted to lower levels for further reducing logging clutter.

3.1.0

Added

Changed

  • Several log messages have been demoted from debug to trace level and might have been shortened / combined to reduce logging clutter.

3.0.0

Changed

  • 🧨 Minimum required Python version is now 3.9.
  • Dependencies have been updated to their latest (compatible) versions.
  • Logging is now done through Loguru.

2.3.0

Added

  • pyppms.ppms.PpmsConnection() now takes an optional parameter cache_users_only that will prevent any request but getuser from being stored in the local cache. This is useful in scenarios where frequent requests to PPMS are being done to fetch booking states and such that would be slowed down enormously if no user caching was present. Obviously the cached users need to be refreshed explicitly on a regular basis then. Defaults to False which will result in the same behavior as before. Please note that several things are implicitly being cached (in memory) during the lifetime of the PpmsConnection object (e.g. the PPMS systems) unless their corresponding method is being called with force_refresh=True.
  • pyppms.ppms.PpmsConnection.update_users() and pyppms.ppms.PpmsConnection.get_users() now both have an optional parameter active_only (defaulting to True) that can be used to also request users that are marked as inactive in PPMS.

Changed

  • pyppms.ppms.PpmsConnection.get_user() is only logging a DEBUG level message (before: ERROR) in case the requested user can't be found since it also raises a KeyError. This is done to prevent cluttering up the logs of calling code that might use this method to figure out if an account exists in PPMS and properly deals with the exception raised.

2.2.0

Added

  • pyppms.ppms.PpmsConnection.flush_cache() to flush the on-disk cache with an optional argument keep_users (defaulting to False) that allows for flushing the entire cache except for the user details. This provides the opportunity of refreshing the cache on everything but existing users. Note that this will not affect new users, they will still be recognized and fetched from PUMAPI (and stored in the cache).

Changed

  • pyppms.ppms.PpmsConnection.get_systems_matching() now raises a TypeError in case the parameter name_contains is accidentially as str instead of a list.
  • pyppms.ppms.PpmsConnection.get_running_sheet() now has an optional parameter ignore_uncached_users (defaulting to False) that allows to process the running sheet even if it contains users that are not in the fullname_mapping attribute.
  • If the cache_path attribute is set for an pyppms.ppms.PpmsConnection instance but creating the actual subdir for an intercepted response fails (e.g. due to permission problems) the response-cache will not be updated. Before, the exception raised by the underlying code (e.g. a PermissionError) was passed on.
  • Methods of pyppms.ppms.PpmsConnection are now sorted in alphabetical order, making it easier to locate them e.g. in the API documentation.

Removed

2.1.0

Changed

2.0.0

Changed

  • [API] 🧨 the signature for pyppms.user.PpmsUser has been changed and now expects a single argument (the PUMAPI response text)
  • [API] 🧨 the constructor signature for pyppms.system.PpmsSystem() has been changed and now expects a single argument (a dict as generated by pyppms.common.parse_multiline_response())
  • [API] 🧨 the constructor signature for pyppms.booking.PpmsBooking() has been changed and now expects the PUMAPI response text, the booking type (if the booking is currently running or upcoming) and the system ID
  • [API] 🧨 the following methods have been removed as their behavior is now achieved by the corresponding default constructor of the respective class:
    • pyppms.user.PpmsUser.from_response()
    • pyppms.system.PpmsSystem.from_parsed_response()
    • pyppms.booking.PpmsBooking.from_booking_request()
 1"""The `pyppms` package provides a Python interface to Stratocore's PUMAPI.
 2
 3.. include:: ../../README.md
 4
 5.. include:: ../../TESTING.md
 6
 7.. include:: ../../CHANGELOG.md
 8"""
 9
10__version__ = "0.0.0"
11
12from .ppms import PpmsConnection