Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
repos:
- repo: https://github.com/psf/black
rev: 23.1.0
rev: 26.10.1
hooks:
- id: black
language_version: "3.11"
- repo: https://github.com/pycqa/isort
rev: 5.12.0
rev: 9.0.2
hooks:
- id: isort
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.4.0
rev: v6.0.0
hooks:
- id: check-yaml
args: ["--allow-multiple-documents"]
12 changes: 9 additions & 3 deletions Dockerfile.test
Original file line number Diff line number Diff line change
@@ -1,12 +1,18 @@
FROM python:3.11
RUN pip install pipenv

ENV POETRY_VERSION=2.4.1 \
POETRY_VIRTUALENVS_CREATE=false \
POETRY_NO_INTERACTION=1
RUN pip install "poetry==${POETRY_VERSION}"

RUN mkdir /app
WORKDIR /app

COPY Pipfile Pipfile.lock ./
RUN pipenv install --dev
COPY pyproject.toml poetry.lock ./
RUN poetry install --no-root

COPY . .
# The package itself, for its pytest11 entry point (editable: the compose volume replaces /app)
RUN poetry install --only-root

EXPOSE 8081
25 changes: 0 additions & 25 deletions Pipfile

This file was deleted.

1,008 changes: 0 additions & 1,008 deletions Pipfile.lock

This file was deleted.

98 changes: 97 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,103 @@ docker run -v "$(pwd)/resources:/app" --env-file ./env/testscript bedasoftware/t
```
### Restrictions
1. TestScript files should be stored in the TestScript folder;
2. TestScript files should be in YAML format.
2. TestScript files should be in YAML (`.yaml`) or JSON (`.json`) format.

A failed `warningOnly` assert does not fail the test; it is reported as a `TestScriptWarning`.

### As a library
```python
import asyncio

import testscript_eval

definition = testscript_eval.load_file("TestScript/testscript-example.yaml")
warnings = asyncio.run(testscript_eval.eval(definition, {
"baseUrl": "http://localhost:8080/fhir",
"authorization": "Basic cm9vdDpzZWNyZXQ=",
}))
```

`testscript_eval.eval` loads fixtures, autocreates them, then runs `setup`, every `test` and
`teardown` (always), and finally autodeletes fixtures. Variables are evaluated where they are used,
so they can read the responses of earlier operations.

- A failed assert raises `testscript_eval.AssertionFailed` (an `AssertionError`). So does a fixture
the server rejects or does not have (`FixtureRejected`), as that is the server's answer.
- A failed `warningOnly` assert does not stop the script; its message is in the returned list.
- Anything else (an unsupported assert or operator, a missing variable, a `$validate` the server
could not run) raises `testscript_eval.TestScriptError` or one of its subclasses, `warningOnly` or not.

The `env` dict holds `baseUrl`, `authorization` (sent as is in the `Authorization` header) and
values for TestScript variables. Without one, it is read from `FHIR_SERVER_BASE_URL` and
`FHIR_SERVER_AUTHORIZATION`.

Options:

- `root_dir` (default: the current directory) resolves file fixture references (`.json`, `.yaml`),
as a `file://` URL or a path from the root: `file://tests/resources/patient.yaml`,
`/tests/resources/patient.yaml`, `../Fixtures/patient.json`.
- `include_dir` (default: `<root_dir>/tests/resources`) resolves `!include path` tags in YAML fixtures.
- `client`: a `testscript_eval.Client` to send requests with (default: `Client(baseUrl, authorization=...)`).
`Client(base_url, headers={...}, on_response=callback, session=aiohttp_session)` adds headers to
every request and passes every `Response` (with its `Request`) to `callback`, e.g. to record requests.
- `validator`: what `validateProfileId` asserts call: any callable (sync or async) taking
`(resource, profile_url)` and returning the validation error messages (default: `ServerValidator`,
`$validate` on the server under test).

YAML files are loaded safely, with dates and datetimes kept as strings.

#### Running a script step by step

`eval` is `Evaluator.run`, which is made of three phases that can also be called one at a time,
e.g. to report every test separately:

```python
evaluator = testscript_eval.Evaluator(definition, env)
try:
await evaluator.start() # fixtures, autocreate, setup
for test in evaluator.tests:
warnings = await evaluator.run_test(test) # raises on a failed assert
finally:
await evaluator.finish() # teardown, autodelete; also after a failed start
```

`evaluator.client` and `evaluator.validator` can be swapped between phases.

### Supported features

- Fixtures: contained (`#Type/id`), files (`file://`, `/path`, relative paths; JSON or YAML),
server references (`Type/id` or a URL), fixtures without a resource (placeholders) or an id.
`autocreate` puts fixtures with ids in one transaction (so references between them resolve, even in
cycles; one by one when the server refuses it), posts transaction/batch Bundles to the server root;
`autodelete` deletes them, including what a transaction Bundle created.
- Operations: every `testscript-operation-codes` type (`read`, `vread`, `search`, `create`, `update`,
`updateCreate`, `patch`, `delete`, `history`, `transaction`, `batch`, `capabilities`, `validate`,
`operation` with `params: $name`, `populate`, `extract`, …) or an explicit `method`, with `url`,
`params` (path, query or `$operation`), `resource`, `targetId`, `sourceId` (a Binary goes as its
content), `responseId`, `requestHeader`, `accept` and `contentType`
- Variables: `defaultValue`, env values, `expression` (FHIRPath), `path` (JSONPath, or an XPath
evaluated on the JSON form), `headerField`, against a `sourceId` fixture or response or the last response
- Asserts: `response` (all response codes), `responseCode`, `resource`, `headerField`, `contentType`,
`path`, `expression`, `value`, `compareToSourceExpression`, `compareToSourcePath`, `minimumId`,
`navigationLinks`, `requestURL`, `requestMethod`, `validateProfileId`, `sourceId`, and
`direction: request`; `${variables}` in values, paths and expressions
- Operators: `equals`, `notEquals`, `in`, `notIn`, `contains`, `notContains`, `greaterThan`, `lessThan`,
`empty`, `notEmpty`, `exists`, `eval`
- R5: `common` actions with parameters, `profile` as canonical URLs with ids in `_profile`
- Not supported: XML (requests and responses are always JSON; `xml` formats are sent as JSON, XML
fixture files cannot be loaded), `origin`/`destination` (every operation goes to the one server),
`stopTestOnFail` (a failed assert always ends its test)

## Development
Requires Python 3.11+ and [Poetry](https://python-poetry.org/) 2. Unit tests run offline against a
local stub server:
```bash
poetry install
poetry run pytest tests/unit
```
`make up` runs the TestScripts in `tests/TestScript` against Aidbox in Docker.

## Recommendations
Highly recommend using testscript-eval-py alongside [Kaitenzushi](https://github.com/beda-software/kaitenzushi/) to enhance the TestScript development experience.
## References
Expand Down
2 changes: 1 addition & 1 deletion compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -79,4 +79,4 @@ services:
FHIR_SERVER_BASE_URL: http://devbox:8080
volumes:
- .:/app
command: ["pipenv", "run", "pytest"]
command: ["pytest"]
Loading
Loading