pyppms
The pyppms
package provides a Python interface to Stratocore's PUMAPI.
PyPPMS
PUMAPI - Python Interface
Stratocore's PPMS booking system offers an API (the so-called PUMAPI, short for PPMS Utility Management API) for fetching information from the booking system as well as changing its state and properties.
This is a Python 3 package for talking to the PUMAPI.
Usage Example
Fetch email addresses of all active users:
from pyppms import ppms
from credentials_ppms import PPMS_URL, PPMS_API_KEY
conn = ppms.PpmsConnection(PPMS_URL, PPMS_API_KEY)
print("Querying PPMS for emails of active users, can take minutes...")
emails = ppms.get_users_emails(active=True)
print(f"Got {len(emails)} email addresses from PPMS:")
print("\n".join(emails))
Testing
Automated testing is described in the TESTING
document on github.
Note
The PPMS API sometimes exposes a bit of a surprising behavior. During development of the package, we came across several issues (this list is certainly incomplete):
- HTTP status return code is always
200
, even on failed authentication. - Results of queries are a mixture of CSV (with headers) and and text with newlines (with no headers and therefore without structural information on the data). JSON is implemented in some cases only.
- The CSV headers sometimes do contain spaces between the colons, sometimes they don't.
- Some fields are quoted in the CSV output, some are not. Difficult to separate the values since there are colons in the values too.
- Semantics of keys is not consistent. Sometimes
user
is the user id, sometimes it refers to the user's full name. - Using an invalid permission level (e.g.
Z
) with thesetright
action is silently ignored by PUMAPI, the response is stilldone
even though this doesn't make any sense. - There is no (obvious) robust way to derive the user id from the user's full
name that is returned e.g. by
getrunningsheet
, making it very hard to cross-reference it with data fromgetuser
. - The result of the
getrunningsheet
query in general is not suited very well for automated processing, it seems to be rather tailored for humans and subject to (mis-) interpretation. - Unfortunately
Username
andSystemname
are not the unique id, they are rather the full description. Therefore sometimes looping over all users and systems is necessary. - Some results have a very strange format - for example, the starting time of the next booking is given as minutes from now instead of an absolute time.
- Official documentation is rather rudimentary, i.e. it contains almost no information on what is returned in case wrong / invalid parameters are supplied and similar situations.
References
Testing PyPPMS
Automated testing has been a core design goal for pyppms
, aiming for a
coverage of 100%. Testing of the project is performed through pytest.
Concept of PyPPMS Unit Tests
As proper testing of an HTTP-based API will require interaction with a real instance of that given API the complete suite of tests will only be able to run if you're having access to PPMS somewhere. Obviously, this should not be done on a production instance but a separate test setup (contact Stratocore to get one).
To speed up testing, make it more convenient and provide a certain level of robustness against silent changes of the PPMS API, tests are split into more or less three categories:
- local unit tests - they don't need a PPMS / PUMAPI instance
- tests using cached responses from a real PPMS / PUMAPI
- tests using mocked responses to simulate specific behavior of PPMS that cannot be triggered otherwise
Development installation through poetry
The project is using poetry for packaging and dependency management. To set up a development environment and prepare for testing use the command below, it will set up a fresh virtual environment with the correct dependencies and install the project in editable mode:
git clone https://github.com/imcf/pyppms
cd pyppms
poetry install
Cached Testing
Almost all of the request-response tests, which is basically anything in the PpmsConnection class, do NOT require a valid API-key or a connection to a PUMAPI instance. Instead, they can be performed using the built-in response-caching mechanism combined with the mocks and cached responses provided with the repository.
Using the cache
Working with the provided cached responses is the default when running the tests. The
only exception are those tests that do not make sense in such a scenario (i.e. that do
test if interaction with an actual PUMAPI instance is effectively working). Those tests
have to be requested explicitly by adding the "--online
" flag to the pytest-call.
Validating the cache
Validating or re-building the cache requires access to an actual PUMAPI, see the section on running online tests below for details.
Configuration and API Key
To run the tests, copy the example pyppmsconf.py
file to the /tests/
directory. For the online tests, please edit it according to your
instance and key - the offline tests will work without modifying the config.
cp -v resources/examples/pyppmsconf.py tests/
To generate an API key a so-called "Super-Admin" needs to log on to your PPMS instance,
navigate to My PPMS
using the drop-down menu on the top-right, select API
from the
top bar and finally hit the Create PUMAPI key
button.
Running Tests
Once everything is set up, you should be good to simply type poetry run pytest
on the
command line, the output should look something like this:
poetry run pytest
============================ test session starts =============================
platform linux -- Python 3.8.10, pytest-7.1.1, pluggy-1.0.0
cachedir: .pytest_cache
rootdir: /tmp/imcf/pyppms, configfile: pyproject.toml
plugins: cov-3.0.0
collected 43 items
tests/test_booking.py ......... [ 20%]
tests/test_common.py .... [ 30%]
tests/test_ppms.py s.s....................... [ 90%]
tests/test_system.py .. [ 95%]
tests/test_user.py .. [100%]
========================== short test summary info ===========================
SKIPPED [1] tests/test_ppms.py:95: need --online option to run
SKIPPED [1] tests/test_ppms.py:108: need --online option to run
======================= 41 passed, 2 skipped in 0.14s ========================
Running Online Tests
To run those tests requiring access to a real PUMAPI instance in addition to the default
ones, simply add the --online
flag to the pytest
command above. Obviously you will
need to have valid settings for PUMAPI_URL
and PPMS_API_KEY
in the config file used
for testing.
Please note that this will still run the majority of tests using the cached / mocked responses!
However, having a cache of the expected responses from PUMAPI for a given query (or a series of queries) allows for checking if the behavior of the API has silently changed by simply deleting the cache and re-building it afterwards. To do so, the following steps are required:
- preparing your test instance of PPMS - unfortunately this is a manual operation, but it has to be done only once (unless Stratocore resets your test instance)
- removing the cache
- running the tests in online mode to re-populate the cache
- filtering / checking / validating the results
Those steps are described in details in the following sections.
PPMS Preparations
As the tests assume certain users and systems to exist in the PUMAPI instance used for testing, your test instance needs to be prepared accordingly. Currently there is not yet a mechanism to automatically create those items unfortunately - sorry, might come at some point...
For now, simply run the following command to see what needs to be configured in your PPMS, then log into the web interface with your browser and manually create the required items:
poetry run python tests/show_required_ppms_values.py
Remarks / additional details:
- All created users should be members of the previously created group.
- After creating the admin user, go to the Admins page in PPMS, hit the Create a new administrator button, then select the correct facility and pick the user account. In options, simply check the System management box, then click Create administrator.
- After creating all users, navigate to the Rights page, select the newly created system and pick the regular user, then hit the Create button to assign booking permissions to the user account. Then repeat this for the admin user and the inactive user (account needs to be set to active for adding the permissions).
- In addition to the above, four bookings for the regular user on the created
system need to be made on 2028-12-24:
- from 09:00 to 10:00
- from 11:00 to 12:00
- from 13:00 to 14:00
- from 15:00 to 16:00
Removing the Cache
As easy as running:
rm -r tests/cached_responses
Re-populating and validating the Cache
NOTE: As a test-instance of PPMS usually is a clone of a real one it will contain many more but the previously created objects. Therefore when re-populating the cache from a real PPMS instance a few filtering steps have to be done to validate the new cache files and ignore those "unpredictable" ("instance-specific") elements.
First run the most time-consuming tests that will fetch all users from your PPMS (this can easily take several minutes, depending on your PPMS instance):
poetry run pytest --online tests/test_ppms.py::test_get_users
poetry run pytest --online tests/test_ppms.py::test_get_admins
As a result, the tests/cached_responses/stage_0/getuser/
directory will be cluttered
up with plenty of files from users in your PPMS instance that the cache doesn't know
about (and also shouldn't). To clean this, simply remove all corresponding
response-cache files untracked by git:
git clean -f tests/cached_responses/stage_0/getuser/
Now the freshly re-created response files need to be checked if they contain all the
expected values while discarding / ignoring the additional ones introduced by your
specific PPMS instance. To simplify this task use this shortcut function (bash) to show
the git diff
of a file while discarding all lines that were added to it (as they are
specific to your instance):
filternew() {
git diff --no-color "$1" | grep -v '^+' | tail -n +5
}
First, this needs to be done for the files created by the two tests from above (active users and admins). Run the command and compare the output that is expected to look as shown here:
filternew "tests/cached_responses/stage_0/getusers/active--true.txt"
pyppms
pyppms-adm
filternew "tests/cached_responses/stage_0/getadmins/response.txt"
pyppms-adm
If the output matches, discard the changes to those files:
git restore \
"tests/cached_responses/stage_0/getusers/active--true.txt" \
"tests/cached_responses/stage_0/getadmins/response.txt"
Now run all --online
tests - with the just (re-)created cache files for the users and
admins, this should only take a few seconds:
poetry run pytest --online
Then, check the remaining re-created cache files for their content:
filternew "tests/cached_responses/stage_0/getusers/response.txt"
pyppms
pyppms-adm
pyppms-deact
filternew "tests/cached_responses/stage_0/getgroups/response.txt"
pyppms_group
filternew "tests/cached_responses/stage_0/getsysrights/id--*"
A:pyppms
A:pyppms-adm
D:pyppms-deact
S:pyppms-adm
filternew "tests/cached_responses/stage_1/getsysrights/id--*"
D:pyppms
A:pyppms-adm
D:pyppms-deact
S:pyppms-adm
filternew "tests/cached_responses/stage_2/getsysrights/id--*"
D:pyppms
A:pyppms-adm
D:pyppms-deact
S:pyppms-adm
Do the same for the systems and user experience responses, taking into account that the system ID will differ in your case, those lines will then show as missing in the diff:
filternew "tests/cached_responses/stage_0/getsystems/response.txt"
Core facility ref,System id,Type,Name,Localisation,Active,Schedules,Stats,Bookable,Autonomy Required,Autonomy Required After Hours
2,69,"Virtualized Workstation","Python Development System","VDI (Development)",True,True,True,True,True,False
filternew "tests/cached_responses/stage_0/getuserexp/response.txt"
login,id,booked_hours,used_hours,last_res,last_train
"pyppms",69,0,0,n/a,n/a
"pyppms-adm",69,0,0,n/a,n/a
"pyppms-deact",69,0,0,n/a,n/a
The last one to check is the response for the nextbooking
query, which will differ in
the two additional lines for the remaining time and the session, so the result should
look something like this:
filternew "tests/cached_responses/stage_0/nextbooking/id--*"
pyppms
-303520
-31432
If all output matches, discard the changes to those files:
git restore \
"tests/cached_responses/stage_0/getusers/response.txt" \
"tests/cached_responses/stage_0/getgroups/response.txt" \
"tests/cached_responses/stage_0/getsysrights/id--*" \
"tests/cached_responses/stage_1/getsysrights/id--*" \
"tests/cached_responses/stage_2/getsysrights/id--*" \
"tests/cached_responses/stage_0/getsystems/response.txt" \
"tests/cached_responses/stage_0/getuserexp/response.txt" \
"tests/cached_responses/stage_0/nextbooking/id--*"
PyPPMS Changelog
NOTE: potentially breaking changes are flagged with a 🧨 symbol.
3.3.0
Added
pyppms.ppms.PpmsConnection.get_running_sheet()
now has an optional parameterlocalisation
(defaulting to an emptystr
) that will be passed to the call topyppms.ppms.PpmsConnection.get_systems_matching()
, allowing to restrict the runningsheet to systems of a given "room".
3.2.1
Fixed
- 🕛🌃 end time:
pyppms.booking.PpmsBooking.endtime_fromstr()
contained a bug where the end time of a booking finishing at midnight got wrongly assigned to the start of the given day (instead of the end). This is now fixed by setting the end time to the start of the following day.
3.2.0
Added
pyppms.ppms.PpmsConnection.last_served_from_cache
has been added to indicate if the last request was served from the cache or on-line.
Changed
- Several log messages have been demoted to lower levels for further reducing logging clutter.
3.1.0
Added
pyppms.common.fmt_time()
to string-format a datetime object that might also be None (in which case a fixed string is returned).pyppms.booking.PpmsBooking.desc
has been added as a property to retrieve a shorter description of the object than callingstr()
on it.pyppms.exceptions.NoDataError
has been added to indicate a PUMAPI response did not contain any useful data.pyppms.common.parse_multiline_response()
will now raise the newly addedNoDataError
in case the requested runningsheet for a day doesn't contain any bookings to allow for properly dealing with "empty" days.
Changed
- Several log messages have been demoted from
debug
totrace
level and might have been shortened / combined to reduce logging clutter.
3.0.0
Changed
- 🧨 Minimum required Python version is now
3.9
. - Dependencies have been updated to their latest (compatible) versions.
- Logging is now done through Loguru.
2.3.0
Added
pyppms.ppms.PpmsConnection()
now takes an optional parametercache_users_only
that will prevent any request butgetuser
from being stored in the local cache. This is useful in scenarios where frequent requests to PPMS are being done to fetch booking states and such that would be slowed down enormously if no user caching was present. Obviously the cached users need to be refreshed explicitly on a regular basis then. Defaults toFalse
which will result in the same behavior as before. Please note that several things are implicitly being cached (in memory) during the lifetime of thePpmsConnection
object (e.g. the PPMS systems) unless their corresponding method is being called withforce_refresh=True
.pyppms.ppms.PpmsConnection.update_users()
andpyppms.ppms.PpmsConnection.get_users()
now both have an optional parameteractive_only
(defaulting toTrue
) that can be used to also request users that are marked as inactive in PPMS.
Changed
pyppms.ppms.PpmsConnection.get_user()
is only logging aDEBUG
level message (before:ERROR
) in case the requested user can't be found since it also raises aKeyError
. This is done to prevent cluttering up the logs of calling code that might use this method to figure out if an account exists in PPMS and properly deals with the exception raised.
2.2.0
Added
pyppms.ppms.PpmsConnection.flush_cache()
to flush the on-disk cache with an optional argumentkeep_users
(defaulting toFalse
) that allows for flushing the entire cache except for the user details. This provides the opportunity of refreshing the cache on everything but existing users. Note that this will not affect new users, they will still be recognized and fetched from PUMAPI (and stored in the cache).
Changed
pyppms.ppms.PpmsConnection.get_systems_matching()
now raises aTypeError
in case the parametername_contains
is accidentially asstr
instead of a list.pyppms.ppms.PpmsConnection.get_running_sheet()
now has an optional parameterignore_uncached_users
(defaulting toFalse
) that allows to process the running sheet even if it contains users that are not in thefullname_mapping
attribute.- If the
cache_path
attribute is set for anpyppms.ppms.PpmsConnection
instance but creating the actual subdir for an intercepted response fails (e.g. due to permission problems) the response-cache will not be updated. Before, the exception raised by the underlying code (e.g. aPermissionError
) was passed on. - Methods of
pyppms.ppms.PpmsConnection
are now sorted in alphabetical order, making it easier to locate them e.g. in the API documentation.
Removed
- The following previously deprecated (or not even implemented) methods of
pyppms.ppms.PpmsConnection
have been removed in favor ofpyppms.ppms.PpmsConnection.get_systems_matching()
:_get_system_with_name()
_get_machine_catalogue_from_system()
get_bookable_ids()
- Removed the stub
pyppms.ppms.PpmsConnection.get_system()
that was only raising aNotImplementedError
.
2.1.0
Changed
- [API]
pyppms.ppms.PpmsConnection.get_user()
andpyppms.ppms.PpmsConnection.get_user_dict()
now both accept an optional parameterskip_cache
that is passed on to thepyppms.ppms.PpmsConnection.request()
call - [FIX]
pyppms.ppms.PpmsConnection.update_users()
now explicitly asks for the cache to be skipped
2.0.0
Changed
- [API] 🧨 the signature for
pyppms.user.PpmsUser
has been changed and now expects a single argument (the PUMAPI response text) - [API] 🧨 the constructor signature for
pyppms.system.PpmsSystem()
has been changed and now expects a single argument (a dict as generated bypyppms.common.parse_multiline_response()
) - [API] 🧨 the constructor signature for
pyppms.booking.PpmsBooking()
has been changed and now expects the PUMAPI response text, the booking type (if the booking is currently running or upcoming) and the system ID - [API] 🧨 the following methods have been removed as their behavior is now
achieved by the corresponding default constructor of the respective class:
pyppms.user.PpmsUser.from_response()
pyppms.system.PpmsSystem.from_parsed_response()
pyppms.booking.PpmsBooking.from_booking_request()