Repository navigation
Adding GUI System Tests #249
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
MikeSullivan7
wants to merge
29
commits into
RascalSoftware:main
Choose a base branch
from
MikeSullivan7:gui_system_tests
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
29 commits
Select commit
Hold shift + click to select a range
738a0a5
added gui system test base class GuiSystemBase
MikeSullivan7 6ec42ba
pytests differs between system and unit tests using flags --run-syste…
MikeSullivan7 f30421a
ruff fix
MikeSullivan7 26e18fb
add exception handling to GuiSystemBase
MikeSullivan7 72e0f4d
added TestGuiSystemMainWindow test class
MikeSullivan7 f59b8e8
ruff fix
MikeSullivan7 c99d9e7
added TestGuiSystemLoading::test_load
MikeSullivan7 a3dff02
ruff fix
MikeSullivan7 b4f741e
added TestGuiSystemLoading::test_run and automatic closing of unsaved…
MikeSullivan7 1ff717f
fix issue with START_PROCESSES environment variable and tests hanging…
MikeSullivan7 d821341
unit and system tests run by default, can skip tests with --skip_syst…
MikeSullivan7 9ba39a9
fix problem with logging not printing to terminal and Matlab Engine n…
MikeSullivan7 ad570cc
fix issue with terminal being written to after deletion due to Logging
MikeSullivan7 2791bf4
ruff fix
MikeSullivan7 c6d6255
allow mulitple possible values for _click_messagebox
MikeSullivan7 ac1dac7
change expected chi_squared to 142.346 in test_run
MikeSullivan7 58422e2
test_run now runs each procedure and compares against expected values
MikeSullivan7 93b6397
ruff fixes
MikeSullivan7 b3c0241
added parameterized to requirements-dev.txt
MikeSullivan7 ce241d0
increase timing for test_run ns and dream procedures
MikeSullivan7 b7166b5
add --run_slow_tests flag to dynamically skip tests with 'slow' in name
MikeSullivan7 cd8f71c
mark Nested Sampler and DREAM runs as slow
MikeSullivan7 8787984
set random seed to fix stochastic tests
MikeSullivan7 13d7329
add system test information to CONTRIBUTING.md
MikeSullivan7 b414c8c
suggested changes
MikeSullivan7 8e83f2f
cleanup and make test_load more reliable
MikeSullivan7 04eb4d3
test_run_slow fix for linux
MikeSullivan7 7d749c3
remove nested sampler from testing
MikeSullivan7 8f35beb
ruff fix
MikeSullivan7 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -3,3 +3,4 @@ pytest-cov | |
| ruff | ||
| Sphinx | ||
| pydata-sphinx-theme | ||
| parameterized | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,78 @@ | ||
| import logging | ||
| import sys | ||
| import unittest | ||
| from collections.abc import Callable | ||
|
|
||
| from PyQt6.QtCore import Qt, QTimer | ||
| from PyQt6.QtTest import QTest | ||
| from PyQt6.QtWidgets import QApplication, QMessageBox | ||
|
|
||
| from rascal2.config import setup_logging | ||
| from rascal2.ui.view import MainWindowView | ||
|
|
||
| SHOW_DELAY = 10 # Can be increased to watch tests | ||
| SHORT_DELAY = 100 | ||
|
|
||
|
|
||
| def wait_until( | ||
| test_func: Callable[[], bool], delay=0.1, max_retry=100, message: str = "wait_until reached max retries" | ||
| ): | ||
| """Repeat test_func every delay seconds until it becomes true. Raises RuntimeError if max_retry is reached.""" | ||
| for _ in range(max_retry): | ||
| if test_func(): | ||
| return True | ||
| QTest.qWait(int(delay * 1000)) | ||
| raise RuntimeError(message) | ||
|
|
||
|
|
||
| class GuiSystemBase(unittest.TestCase): | ||
| app: QApplication = QApplication.instance() | ||
|
|
||
| def setUp(self) -> None: | ||
| setup_logging() | ||
| self.no_exceptions = True | ||
|
|
||
| sys.excepthook = self.exception_hook | ||
| self.main_window = MainWindowView() | ||
| self.main_window.show() | ||
| QTest.qWait(SHORT_DELAY) | ||
|
|
||
| def tearDown(self) -> None: | ||
| if not self.no_exceptions: | ||
| raise Exception("An exception occurred in a PyQt slot") | ||
| sys.excepthook = sys.__excepthook__ | ||
| logger = logging.getLogger("rascal2") | ||
| logger.handlers.clear() | ||
| if not self.main_window.presenter.runner.matlab_helper.close_event.is_set(): | ||
| self.main_window.presenter.runner.matlab_helper.close_event.set() | ||
| QTimer.singleShot(SHORT_DELAY, lambda: self._click_messagebox()) | ||
| self.main_window.close() | ||
| wait_until( | ||
| lambda: not self.main_window.isVisible(), | ||
| delay=0.05, | ||
| max_retry=200, | ||
| message="Main window did not close within 10 seconds", | ||
| ) | ||
| del self.main_window | ||
|
|
||
| def exception_hook(self, exc_type, exc_value, exc_traceback): | ||
| self.no_exceptions = False | ||
| sys.__excepthook__(exc_type, exc_value, exc_traceback) | ||
|
|
||
| @classmethod | ||
| def _click_messagebox(cls): | ||
| """Needs to be queued with QTimer.singleShot before triggering the message box.""" | ||
| no_buttons_found = True | ||
| button_texts = [] | ||
| for widget in cls.app.topLevelWidgets(): | ||
| if isinstance(widget, QMessageBox) and widget.isVisible(): | ||
| for button in widget.buttons(): | ||
| if widget.buttonRole(button) == widget.ButtonRole.DestructiveRole: | ||
| QTest.mouseClick(button, Qt.MouseButton.LeftButton) | ||
| no_buttons_found = False | ||
| button_texts = [button.text() for button in widget.buttons()] | ||
| if no_buttons_found: | ||
| raise ValueError( | ||
| f"Could not find buttons in {button_texts}'.\n" | ||
| f"Message box: {widget.windowTitle()} {widget.text()}" | ||
| ) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,81 @@ | ||
| import random | ||
| from pathlib import Path | ||
|
|
||
| from parameterized import parameterized | ||
| from PyQt6.QtTest import QTest | ||
| from PyQt6.QtWidgets import QApplication | ||
|
|
||
| from rascal2.dialogs.startup_dialog import LoadDialog | ||
| from tests.system.gui_system_base import SHORT_DELAY, GuiSystemBase, wait_until | ||
|
|
||
| EXAMPLES_PATH = Path(__file__, "../../../examples").resolve().as_posix() | ||
|
|
||
|
|
||
| class TestGuiSystemLoading(GuiSystemBase): | ||
| def setUp(self) -> None: | ||
| random.seed(20) | ||
| super().setUp() | ||
|
|
||
| def tearDown(self) -> None: | ||
| super().tearDown() | ||
|
|
||
| def watch_output(self, test_duration=100) -> None: | ||
| while "Finished RAT" not in ( | ||
| most_recent_line := self.main_window.terminal_widget.text_area.toPlainText().split("\n")[-1] | ||
| ): | ||
| wait_until( | ||
| lambda: self.main_window.terminal_widget.text_area.toPlainText().split("\n")[-1] != most_recent_line, | ||
| max_retry=test_duration, | ||
| ) | ||
|
|
||
| def test_load(self): | ||
| QTest.qWait(SHORT_DELAY) | ||
| self.main_window.startup_dlg.import_project_button.click() | ||
| load_dialog = self.main_window.findChild(LoadDialog) | ||
| load_dialog.tabs.setCurrentIndex(2) | ||
| load_dialog.example_list_widget.itemClicked.emit(load_dialog.example_list_widget.item(0)) | ||
| wait_until(lambda: self.main_window.controls_widget.chi_squared.text() != "") | ||
| assert self.main_window.presenter.model.project.name == "DSPC Standard Layers" | ||
|
|
||
| @parameterized.expand( | ||
| [ | ||
| ("calculate", 142.346), | ||
| ("simplex", 13.1816), | ||
| ("de", 10.4746), | ||
| ] | ||
| ) | ||
| def test_run(self, procedure_name, expected_chi): | ||
| QTest.qWait(SHORT_DELAY) | ||
| self.main_window.startup_dlg.import_project_button.click() | ||
| load_dialog = self.main_window.findChild(LoadDialog) | ||
| load_dialog.tabs.setCurrentIndex(2) | ||
| load_dialog.example_list_widget.itemClicked.emit(load_dialog.example_list_widget.item(0)) | ||
| wait_until(lambda: self.main_window.controls_widget.chi_squared.text() != "") | ||
| self.main_window.controls_widget.update_chi_squared("") | ||
| self.main_window.controls_widget.procedure_dropdown.setCurrentText(procedure_name) | ||
| QApplication.processEvents() | ||
| self.main_window.controls_widget.run_button.click() | ||
| self.watch_output() | ||
| QTest.qWait(SHORT_DELAY) | ||
| assert self.main_window.controls_widget.chi_squared.text() == str(expected_chi) | ||
| assert self.main_window.presenter.runner.error is None | ||
|
|
||
| @parameterized.expand( | ||
| [ | ||
| ("dream", 9.67234, 2000), | ||
| ] | ||
| ) | ||
| def test_run_slow(self, procedure_name, expected_chi, test_duration): | ||
| self.main_window.startup_dlg.import_project_button.click() | ||
| load_dialog = self.main_window.findChild(LoadDialog) | ||
| load_dialog.tabs.setCurrentIndex(2) | ||
| load_dialog.example_list_widget.itemClicked.emit(load_dialog.example_list_widget.item(0)) | ||
| wait_until(lambda: self.main_window.controls_widget.chi_squared.text() != "") | ||
| self.main_window.controls_widget.update_chi_squared("") | ||
| self.main_window.controls_widget.procedure_dropdown.setCurrentText(procedure_name) | ||
| QApplication.processEvents() | ||
| self.main_window.controls_widget.run_button.click() | ||
| self.watch_output(test_duration) | ||
| QTest.qWait(SHORT_DELAY) | ||
| assert self.main_window.controls_widget.chi_squared.text() == str(expected_chi) | ||
| assert self.main_window.presenter.runner.error is None |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| from rascal2 import RASCAL2_VERSION | ||
| from tests.system.gui_system_base import GuiSystemBase | ||
|
|
||
|
|
||
| class TestGuiSystemMainWindow(GuiSystemBase): | ||
| def setUp(self) -> None: | ||
| super().setUp() | ||
|
|
||
| def tearDown(self) -> None: | ||
| super().tearDown() | ||
|
|
||
| def test_main_window(self): | ||
|
MikeSullivan7 marked this conversation as resolved.
Resolved
|
||
| self.main_window.presenter.create_project("project", ".") | ||
| names = [win.windowTitle() for win in self.main_window.mdi.subWindowList()] | ||
| assert names == ["Fitting Controls", "Terminal", "Project", "Plots"] | ||
| assert ( | ||
| self.main_window.terminal_widget.text_area.toPlainText() | ||
| == f""" | ||
| ███████████ █████████ █████████ █████ | ||
| ░░███░░░░░███ ███░░░░░███ ███░░░░░███ ░░███ | ||
| ░███ ░███ ██████ █████ ███ ░░░ ░███ ░███ ░███ | ||
| ░██████████ ░░░░░███ ███░░ ░███ ░███████████ ░███ | ||
| ░███░░░░░███ ███████ ░░█████ ░███ ░███░░░░░███ ░███ | ||
| ░███ ░███ ███░░███ ░░░░███░░███ ███ ░███ ░███ ░███ █ | ||
| █████ █████░░████████ ██████ ░░█████████ █████ █████ ███████████ | ||
| ░░░░░ ░░░░░ ░░░░░░░░ ░░░░░░ ░░░░░░░░░ ░░░░░ ░░░░░ ░░░░░░░░░░░ | ||
| RasCAL-2: software for neutron reflectivity calculations v{RASCAL2_VERSION}""" | ||
| ) | ||
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please revert this since the close event set call is now gone