RIGOR
Implementation
Performance and Testing Strategy (Normative – v0.1)
Status: Normative Scope: Defines performance guarantees, benchmarking expectations, determinism validation, and mandatory testing strategy across all RIGOR engines.
This document formalizes quality constraints required for a compliant implementation.
1. Purpose
This document establishes:
- Performance complexity expectations
- Determinism validation requirements
- Test coverage obligations
- Regression guarantees
- Stability verification strategy
It applies to all engines and the CLI execution layer.
2. Performance Model
RIGOR is designed for deterministic, structural analysis of specifications.
Implementations MUST optimize for:
- Predictable complexity
- Linear or near-linear scaling
- Deterministic execution time patterns
3. Expected Complexity per Engine
3.1 Canonicalization Engine
Expected complexity:
O(N + E)
Where:
- N = number of nodes
- E = number of edges
Must avoid nested scans.
3.2 Validation Engine
Expected complexity:
O(R × N)
Where:
- R = number of rules
- N = nodes in scope
Rules MUST avoid full-graph nested traversal where possible.
3.3 Constraint Engine
Expected complexity:
O(C × S)
Where:
- C = number of constraints
- S = scope size
Aggregation functions MUST operate on sorted canonical collections.
3.4 Diff Engine
Expected complexity:
O(N + E)
Attribute comparison MUST avoid quadratic behavior.
3.5 Versioning Engine
Expected complexity:
O(C)
Where:
- C = number of changes in ChangeSet
3.6 Migration Engine
Expected complexity:
O(O + N)
Where:
- O = number of operations
- N = graph size
3.7 Event Resolution Engine
Expected complexity:
O(Ev + D)
Where:
- Ev = number of events
- D = number of dependencies
Cycle detection MUST be linear.
4. Memory Expectations
Memory usage SHOULD scale linearly with graph size.
Implementations MUST NOT:
- Duplicate full graph unnecessarily
- Maintain redundant copies of large collections
- Leak memory across CLI invocations
5. Determinism Testing
Determinism is a mandatory property.
Implementations MUST include:
5.1 Repeatability Tests
Run identical input multiple times → identical output (byte-level equality in JSON mode).
5.2 Order Randomization Tests
If internal data structures are unordered (e.g., hash maps), tests MUST:
- Shuffle insertion order
- Verify identical canonical output
5.3 Concurrency Stability Tests
If parallelization is used:
- Parallel and sequential runs MUST produce identical results.
6. Test Categories
The following test layers are mandatory.
6.1 Unit Tests
Each engine component MUST have:
- Rule-level tests
- Constraint-level tests
- Operation-level tests
- Expression parser tests
Coverage MUST include success and failure cases.
6.2 Integration Tests
End-to-end scenarios MUST test:
- Full CLI pipeline
- Diff + Version interaction
- Migration + Validation chain
- Event resolution after migration
6.3 Golden Tests
Golden tests MUST:
- Serialize JSON output
- Compare against committed reference files
- Fail on any structural change
Golden tests MUST validate determinism.
6.4 Regression Tests
Every discovered defect MUST produce:
- Reproducible test case
- Permanent regression test
- Clear mapping to error code
Regression suite MUST grow monotonically.
6.5 Property-Based Tests
Implementations SHOULD include property tests for:
- Canonical ordering invariants
- Diff symmetry properties
- Version bump monotonicity
- Constraint aggregation consistency
Example properties:
- diff(A, A) → no-change
- canonicalize(canonicalize(G)) = canonicalize(G)
6.6 Stress Tests
Large synthetic graphs MUST test:
- Scaling behavior
- Memory stability
- Performance degradation patterns
Stress tests MUST not alter determinism.
7. Benchmarking Requirements
Implementations SHOULD provide:
- Benchmark suite
- Stable dataset set
- Time measurement without logging noise
Benchmarks MUST report:
- Node count
- Edge count
- Execution time per engine
- Memory usage
Results MUST be reproducible under same environment.
8. CI Requirements
Continuous Integration MUST:
- Run full test suite
- Run determinism tests
- Validate golden outputs
- Fail on nondeterministic ordering
- Fail on unregistered error codes
CI MUST block merge on test failure.
9. Error Code Stability Tests
A registry of error codes MUST exist.
Tests MUST verify:
- No duplicate codes
- No deleted codes without deprecation
- No changed semantics without version bump
Error model stability is mandatory.
10. Performance Regression Detection
CI SHOULD include:
- Performance baseline
- Threshold alerts
- Historical comparison
Hard failures MAY be configured if performance drops exceed defined threshold.
11. Compatibility Testing
Backward compatibility MUST be tested across versions:
- Load older specs
- Validate against new engine
- Verify expected error evolution
Breaking behavior MUST require major version increment.
12. Migration Safety Testing
Migration tests MUST verify:
- Atomic rollback on failure
- No partial mutation
- Correct application ordering
- Post-migration validation success
13. Security and Robustness Testing
Even though RIGOR is not runtime-executed, implementation MUST test:
- Malformed input resilience
- Deep nesting handling
- Cycle detection robustness
- Large expression handling
Engines MUST fail deterministically, not crash unpredictably.
14. Non-Goals
The performance strategy does NOT require:
- Real-time guarantees
- Sub-linear diff
- Distributed execution
- Runtime data validation
Focus remains structural determinism.
15. Compliance Criteria
An implementation is compliant if:
- It meets expected complexity class
- It guarantees deterministic output
- It passes golden and regression tests
- It enforces error code stability
- It maintains reproducible benchmarks
16. Summary
The Performance and Testing Strategy ensures:
- Deterministic behavior
- Scalable architecture
- Stable error semantics
- Long-term maintainability
- Strict regression control