Testing Strategy

Committed strategy, automation backlog

Testing must prove creation, useful behavior, failure recovery, and cleanup. A deployment that appears successful but leaks resources is a failed lab run.

Segment quality gates

Gate	Minimum evidence
Contract	Versioned schemas and interfaces, deterministic offline fixtures, no unexpected cloud calls, secret references only, explicit security, cost, ownership, and teardown intent
Live	Contract evidence plus isolated deployment, positive and negative probes, security and vulnerability review, cost and TTL controls, failure handling, teardown, orphan query, and still-billable-resource report
Release	Live evidence plus three consecutive clean lifecycles, interrupted resume, drift, duplicate-trigger locking and fencing, zero unresolved critical or high findings, signed artifacts, and current docs

Each S0-S22 segment names the gate it must pass. A later target receives no credit merely because an earlier provider or engine passed.

Plan and WhatIf are different tests

Mode	Azure access	Test assertion
-Plan	None	The same YAML and catalog versions produce the same normalized actions, derived values, warnings, policy decisions, and report data
-WhatIf	Required	The resolved plan is compared with permitted live Azure resources and reports create, reuse, update, replace, conflict, drift, or no-change without mutation

Plan tests use fixtures and snapshots. WhatIf tests use a protected identity with read and validation permissions appropriate to the provider. A WhatIf test must fail if it mutates Azure or silently treats an unreadable resource as absent.

Test layers

Layer	Typical run	Focus
Static and schema	Every PR	PowerShell parsing and analysis, YAML schemas, examples, docs
Unit	Every PR	Precedence, naming, redaction, policy, plan, cost, ownership
Contract	Every PR	Provider, engine, catalog, probe, state, and report interfaces
Fixture and snapshot	Every PR	Deterministic plans and output schemas without cloud access
WhatIf	Approved PR or schedule	Azure validation, reuse, drift, and denied-access behavior
Live smoke	Protected and narrow	Deploy, configure, probe, report, teardown, cleanup query
Scheduled matrix	Budgeted schedule	Providers, regions, images, versions, engines, recovery
Release	Candidate tag	Packaging, install, upgrade, compatibility, security evidence

SQL VM reference path

YAML
  -> validate and normalize
  -> Plan and cost review
  -> Azure-aware WhatIf
  -> approved SQL VM deployment
  -> data and tool setup
  -> service, data, security, and software probes
  -> teardown preview and confirmation
  -> ownership-aware deletion
  -> Azure cleanup proof

Provider probes will verify reachable services, expected configuration, installed tools, loaded data, identity access, and report evidence. A successful deployment command is not enough.

SQL VM planning tests also assert an explicit secretStoreDecision and administrativeAccessDecision. The secure default is a Key Vault decision plus Azure Bastion with no public IP. Reuse or opt-out must carry rationale, approval, cost, and security evidence.

Failure injection

Tests will inject failure after Azure resource creation, identity assignment, guest setup, artifact transfer, data loading, probe execution, state persistence, and partial teardown. Recovery must be idempotent, preserve evidence, avoid duplicate actions, and stop when ownership or state cannot be proven.

Concurrency tests will cover two sessions targeting the same lab, stale locks, interrupted runs, expired credentials, throttling, delayed role assignments, and drift introduced between Plan, WhatIf, and execution.

Interactive local-state tests cover owner-only storage and a per-run lock. CI, webhook, or unattended mutation tests require remote leases, monotonically increasing fencing tokens, OIDC or managed identity, idempotency keys, and duplicate-trigger rejection.

Cost and cleanup protection

Live tests require approved subscriptions and regions, constrained SKUs, unique tags, concurrency limits, a short TTL, and Azure budget alerts. Budgets alert; they do not impose a hard spending cap.

Cleanup runs even after test failure, but it can delete only owned resources or explicitly eligible adopted resources in the approved teardown scope. A scheduled sweeper may report expired, untagged, drifted, or unknown resources. It must not auto-delete unknown resources.

Test data and evidence

Pull requests use synthetic or redistributable fixtures. Real or sensitive databases run only in explicitly approved environments with ContainsSensitiveData, restricted reports, verified retention, and deletion evidence.

Each live run will retain tool versions, normalized plan hash, WhatIf result, cost assumptions, approvals, probes, teardown summary, and cleanup query. Secrets and sensitive values remain redacted.

For every target and solution pack, the retained evidence includes its secretStoreDecision. For every Azure VM, it also includes the administrativeAccessDecision, Bastion tier or approved alternative, and public IP outcome.