Developer Test Suite

This page is a developer-focused guide to OpenTestability testing strategy, execution, and debugging.

1. Test suite goals

protect metric correctness across COP, SCOAP, reconvergence, and TPI paths
prevent regressions in the Yosys JSON analysis and TPI flows
keep quick confidence checks available for daily development
provide deeper end-to-end validation before major merges

2. Test directory layout

tests/
|-- conftest.py
|-- fixtures/
|-- unit/
|   |-- core/
|   |-- parsers/
|   |-- testpoint/
|   |-- utils/
|   `-- yosys/
|-- integration/
|   `-- yosys_mode/
|-- system/
`-- benchmark/

Typical intent:

unit/: fast isolated checks for functions and small modules
integration/: behavior across module boundaries and mode workflows
system/: full end-to-end flows against known benchmark designs
benchmark/: pytest-benchmark performance harnesses (engines, TPI), run separately from the correctness suite

3. Marker map

Marker	Purpose	Typical command
`smoke`	fastest confidence checks	`pytest tests/ -m smoke -v`
`slow`	large-design tests (>30s); excluded by default via `addopts` in `pytest.ini`	`pytest tests/ -m slow -v`
`integration`	cross-module behavior	`pytest tests/ -m integration -v`
`system`	end-to-end flow validation	`pytest tests/ -m system -v`
`yosys`	yosys mode specific tests	`pytest tests/ -m yosys -v`

There is no unit marker: pytest tests/ -m unit -v silently selects zero tests. Unit tests are selected by path instead: pytest tests/unit/ -v.

4. Running tests locally

Linux/macOS workflow (on Windows, use WSL; or nix develop for a one-command shell):

cd <path-to>/OpenTestability
source venv/bin/activate

# All tests
pytest tests/ -v

# Fast loop during development
pytest tests/ -m smoke -v
pytest tests/unit/ -v

# Yosys-mode validation
pytest tests/ -m yosys -v

Useful targeted runs:

pytest tests/unit/utils/test_logging_config.py -v
pytest tests/unit/utils/test_session_manager.py -v
pytest tests/unit/testpoint/ -v
pytest tests/integration/test_verbose_logging.py -v
pytest tests/system/test_s27.py -v
pytest tests/system/test_tpi_quality.py -v

5. Recommended workflow for new changes

add or update unit tests close to changed logic
add at least one integration test for the affected flow
run the smoke marker and tests/unit/ first
run full affected marker class (integration or system) before merge
run the full integration suite when touching shared code

6. Fixtures and shared setup

tests/conftest.py holds shared fixtures and common setup hooks
tests/fixtures/ stores reusable test data and helpers
keep fixtures deterministic and small so CI and local runs stay stable

7. Reports and artifacts during testing

Depending on test type and verbosity:

pytest summary is printed to terminal
.pytest_cache/ stores local test cache metadata
verbose OpenTest runs can generate logs in results/log/
verbose OpenTest runs can generate reports in results/reports/
metrics or generated netlists may appear in data/results/, data/TPI/, or output/

When writing tests, prefer temporary paths/fixtures to avoid polluting shared output directories.

8. Common failure patterns

invalid input: the netlist is not valid Yosys JSON
missing environment setup: virtualenv not activated or dependency missing
path assumptions: hardcoded paths instead of project utilities
flaky output comparisons: assertions tied to non-deterministic ordering or timestamps

9. Developer PR checklist

tests added/updated for changed behavior
relevant marker subsets pass locally
no regressions in both modes for shared-code changes
docs updated for user-visible behavior changes
generated artifacts are either ignored or intentionally tracked