Implement explicit restart verification scenario
This commit is contained in:
@@ -134,7 +134,7 @@ Each row is a delivery milestone, not a completion claim. Expand its record befo
|
||||
| 22 | Transactional draft editing and client creation vertical slice | 21 | accepted |
|
||||
| 23 | Readable saved-state Commit and restore scenarios | 22 | accepted |
|
||||
| 24 | Session-owned first delivery and password reissue scenarios | 23 | accepted |
|
||||
| 25 | Explicit restart verification scenario | 24 | pending |
|
||||
| 25 | Explicit restart verification scenario | 24 | accepted |
|
||||
| 26 | Application read models and thin TUI command adapter | 25 | pending |
|
||||
| 27 | Integrated refactoring acceptance and architecture handoff | 26 | pending |
|
||||
|
||||
@@ -4071,7 +4071,7 @@ Focused commands `cargo test --locked app::` (67), `delivery::` (22), `vault::`
|
||||
|
||||
### Milestone 25 — Explicit restart verification scenario
|
||||
|
||||
Status: pending. Prerequisite: M24 accepted.
|
||||
Status: accepted (2026-10-30T21:35:56+02:00; independent source review, M25-R1 evidence re-review and integrated Rust gates passed). Prerequisite: M24 accepted. M1–8 and M10–24 are accepted; M9 is retired and not a prerequisite. This session's coordinator GPT-6-Sol/medium was user-confirmed; `.codex/config.toml` and explicit worker dispatch use GPT-6-Luna/high. Initial tree clean at `7cef400`; Rust/Cargo 1.97.1 verified.
|
||||
|
||||
#### Outcome and scope
|
||||
|
||||
@@ -4079,9 +4079,9 @@ Make restart orchestration independently readable while preserving complete tech
|
||||
|
||||
| Task | Owner | State | Bounded deliverable |
|
||||
| --- | --- | --- | --- |
|
||||
| M25.1 | implementer | pending | Restart preview, bound attempt and preparation extraction |
|
||||
| M25.2 | implementer | pending | Ordered job/observation/verification/durable publication scenario |
|
||||
| M25.3 | reviewer; repairer; coordinator | pending | Failure/deadline/legacy-peer review, repair and RUST acceptance |
|
||||
| M25.1 | implementer `/root/m25_prepare` | complete | Preview/types and confirmed preparation through backup, attempt binding, optional discard and shared deadline moved to private restart owner; app/runtime-restart tests, fmt and Clippy passed |
|
||||
| M25.2 | implementer `/root/m25_sequence` | complete | Service/job, bounded observation, complete comparison and durable publication moved into restart owner; all five focused commands, fmt, Clippy and diff check passed |
|
||||
| M25.3 | reviewer `/root/m25_review`; coordinator | complete | No source findings; M25-R1 evidence gap closed on independent re-review; integrated RUST passed |
|
||||
|
||||
1. Move preview/report/choice/diagnostic types and restart implementation coherently. Preserve current public imports with re-exports until callers migrate. Preview validates unsafe SaveConfig, structural ambiguity and complete SavedPeerProjection; it must not require every saved peer to fit Client/Snapshot.
|
||||
2. Read the scenario in this order: validate preview against saved state -> honor cancel/draft choice -> complete backup -> discard draft at existing safe point -> submit configured service restart -> successful job completion -> bounded interface observation -> compare complete saved peer set -> durable verification bookkeeping -> publish applied knowledge/release reservations. Preserve actual existing ordering where this outline omits preparation; document it instead of introducing behavior changes.
|
||||
@@ -4091,11 +4091,11 @@ Make restart orchestration independently readable while preserving complete tech
|
||||
|
||||
#### Acceptance and validation
|
||||
|
||||
- [ ] M25-A1: backup/job/read/verify/persist/publication sequence and cancellation semantics unchanged; current/saved generations and attempt identity remain bound.
|
||||
- [ ] M25-A2: full-peer legacy cases, shared deadlines, service/transport distinctions, mismatch and post-verification persistence failure tests pass; reservations release only after durable verified success.
|
||||
- [ ] M25-A3: no production fixture bypass, rollback or transport change; independent review and RUST pass. Real-system/IPv6 limits remain explicit.
|
||||
- [x] M25-A1: backup/job/read/verify/persist/publication sequence and cancellation semantics unchanged; current/saved generations and attempt identity remain bound.
|
||||
- [x] M25-A2: full-peer legacy cases, shared deadlines, service/transport distinctions, mismatch and post-verification persistence failure tests pass; reservations release only after durable verified success.
|
||||
- [x] M25-A3: no production fixture bypass, rollback or transport change; independent review and RUST pass. Real-system/IPv6 limits remain explicit.
|
||||
|
||||
Focused commands: `cargo test --locked app::`, `cargo test --locked runtime::`, `cargo test --locked service::`, `cargo test --locked --test commit_restore`, `cargo test --locked --test runtime_restart`. Run only synthetic non-ignored container-driver tests in ordinary gates. Evidence: docs/evidence/milestone-25.md. Outcome: not accepted. Handoff: M26 after acceptance.
|
||||
Focused commands `cargo test --locked app::` (67), `runtime::` (25), `service::` (26), `--test commit_restore` (9) and `--test runtime_restart` (3) exited 0 on the integrated tree. Final coordinator `cargo fmt --all -- --check`, `cargo clippy --locked --all-targets -- -D warnings`, `cargo test --locked` (library 337 passed, 2 ignored; integration/doc tests passed), `cargo build --locked --release`, and `git diff --check` exited 0. Fresh reviewer found no source defect. M25-R1: specified evidence file absent; coordinator created it and reviewer independently verified source/artifact hashes, check records and limits, closing the finding. [M25 evidence](docs/evidence/milestone-25.md) identifies inputs and host/fixture scope. No fresh Alpine or VM/systemd run is claimed. Outcome: accepted for M25 host/fixture scope. Stop after M25; M26 starts only on a separate request.
|
||||
|
||||
### Milestone 26 — Application read models and thin TUI command adapter
|
||||
|
||||
@@ -4159,7 +4159,7 @@ Evidence: docs/evidence/milestone-27.md. Outcome: not accepted. Handoff: stop af
|
||||
|
||||
## Current progress and resume point
|
||||
|
||||
- M24 is accepted for host/synthetic scope: session-owned first-delivery transitions, first-file/QR and per-action password-reissue scenarios passed independent security-path review and required Rust gates. M24-R1 evidence gap was closed on re-review. [M24 evidence](docs/evidence/milestone-24.md) records final inputs and limits. M25–M27 remain pending; M9 remains retired/unaccepted and its real-system limitations remain unchanged.
|
||||
- M25 is accepted for host/fixture scope: bound preview/preparation and ordered restart verification moved into one scenario; independent review and required Rust gates passed. M25-R1 evidence gap was closed on re-review. [M25 evidence](docs/evidence/milestone-25.md) records final inputs and limits. M26–M27 remain pending; M9 remains retired/unaccepted and its real-system limitations remain unchanged.
|
||||
|
||||
- M18 was accepted on 2026-10-29: structured, accurate in-app Help, responsive wrapping and bounded scrolling passed independent R1/R2 closure, synthetic/PTY checks and pinned Rust completion gates. [M18 evidence](docs/evidence/milestone-18.md) records final inputs and limits. At that handoff, M19 had not started.
|
||||
|
||||
@@ -4188,7 +4188,7 @@ Evidence: docs/evidence/milestone-27.md. Outcome: not accepted. Handoff: stop af
|
||||
|
||||
- Milestone 8 delivered reviewed opt-in Alpine Testcontainers coverage for real IPv4 WireGuard peers, application lifecycle, saved/runtime separation, restoration, import/reissue, backup failures and the real Rustix reader. Host gates and three fresh real-container runs passed on one stable source fingerprint; M8-A1–A10 accepted. Real IPv6 behavior is unverified by user-authorized scope.
|
||||
- Milestone 9 is retired and unaccepted: the reviewed disposable VM preparation and guarded first start did not establish Debian/systemd acceptance. The exact VM was powered off after the user stopped work. M9.2–M9.3 were not started; M9-A1–A9 are unchecked.
|
||||
- M18–M24 handoffs remain complete. Stop after M24; begin M25.1 only on a separate request. Keep M16 validation limitations separate from TUI evidence. Do not access production, commit, deploy or resume VM work.
|
||||
- M18–M25 handoffs remain complete. Stop after M25; begin M26.1 only on a separate request. Keep M16 validation limitations separate from TUI evidence. Do not access production, commit, deploy or resume VM work.
|
||||
|
||||
## Plan maintenance
|
||||
|
||||
|
||||
@@ -38,6 +38,7 @@ src/
|
||||
commit.rs # Prepare, backup, install, and publish saved-state Commit
|
||||
restore.rs # Bound preview and confirmed saved-state replacement
|
||||
backups.rs # Manual/pre-operation backup and archive-list scenarios
|
||||
restart.rs # Bound restart preview, backup, job, verification, and publication scenario
|
||||
wireguard.rs # Server configuration module boundary
|
||||
wireguard/
|
||||
document.rs # Parsing and edits preserving original text
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
# Milestone 25 evidence — explicit restart verification scenario
|
||||
|
||||
Run date: 2026-10-30T21:35:10+02:00. Baseline commit: `7cef400`; the working tree has uncommitted M25 changes in `src/app.rs`, new `src/app/restart.rs`, and `docs/PROJECT_LAYOUT.md`, plus coordinator-owned `PLANS.md`. Rust/Cargo: 1.97.1, native host target. Tests used synthetic fixtures/adapters only. No production configuration, keys, VPN, host service, Docker environment, or VM was accessed.
|
||||
|
||||
## Integrated inputs
|
||||
|
||||
| Input | SHA-256 |
|
||||
| --- | --- |
|
||||
| `src/app.rs` | `784001013f90d9bbdb1579cf3c6672a27dd34dc0e295d3e6ea3f70d53d05c586` |
|
||||
| `src/app/restart.rs` | `fb724054faec6adf1b58a31800a2a636307b2620ac3d980309638ad508a00fa3` |
|
||||
| `docs/PROJECT_LAYOUT.md` | `8f156088800e469e6bbf7ac0f012afdb4bf1f190ab4c9161fb3c41d871e04df2` |
|
||||
| `target/release/wireguard-admin` | `b383a1f30e776dcce983bb123af6f4e474e71d6b3d28866f3a06ba3c423228b7` |
|
||||
|
||||
No dependency, manifest, lockfile, service-adapter or runtime-transport files changed. The release artifact hash identifies the binary built from this integrated source; no deployment occurred.
|
||||
|
||||
## Checks on integrated tree
|
||||
|
||||
| Command | Result |
|
||||
| --- | --- |
|
||||
| `cargo test --locked app::` | Exit 0; 67 app tests passed. |
|
||||
| `cargo test --locked runtime::` | Exit 0; 25 runtime tests passed. |
|
||||
| `cargo test --locked service::` | Exit 0; 26 service tests passed. |
|
||||
| `cargo test --locked --test commit_restore` | Exit 0; 9 tests passed. |
|
||||
| `cargo test --locked --test runtime_restart` | Exit 0; 3 tests passed. |
|
||||
| `cargo fmt --all -- --check` | Exit 0. |
|
||||
| `cargo clippy --locked --all-targets -- -D warnings` | Exit 0. |
|
||||
| `cargo test --locked` | Exit 0; library 337 passed, 2 ignored; integration and doc tests passed. The selected Alpine tests stayed ignored. |
|
||||
| `cargo build --locked --release` | Exit 0. |
|
||||
| `git diff --check` | Exit 0. |
|
||||
|
||||
## Acceptance mapping and review
|
||||
|
||||
- **M25-A1:** `restart.rs` contains the preview, validated preparation, completed pre-restart backup, one-use attempt, optional draft discard, service job, bounded runtime read, full comparison and durable publication in the original order. Cancel preserves draft and performs no backup/job/read. The service deadline is created after backup and shared with observation; both saved generation and attempt ID are checked before persistence.
|
||||
- **M25-A2:** Existing app/runtime/service tests cover legacy metadata-free, IPv4-only, subnet-route and missing-IPv4 valid peers; malformed or duplicate identities, incomplete observations, mismatches, service and transport distinctions, shared deadline exhaustion, and post-match persistence failure. `SavedPeerProjection` remains the complete technical target while a management `Snapshot` is optional. Reservations release only after durable verified success.
|
||||
- **M25-A3:** `Application::restart` still calls the existing `service::restart_systemd_unit_report` and `runtime::transport::observe_interface` adapters. The fixture apply method and container driver remain `#[cfg(test)]`; no rollback or second restart was introduced. `app` public restart types remain re-exported.
|
||||
|
||||
Fresh independent reviewer `/root/m25_review` examined the integrated source and original behavior, verified all five focused suites and Rust gates, and found no actionable source defect. **M25-R1** was the missing specified evidence artifact; the coordinator created this report after independently running all required commands. The reviewer independently rechecked the record and all four hashes and closed M25-R1; no finding remains.
|
||||
|
||||
This is host/fixture evidence only. No fresh Alpine Testcontainers scenario or disposable systemd VM test was run, so real systemd/D-Bus and live IPv6 behavior remain unverified. Alpine would not establish systemd behavior. The retired M9 real-system acceptance limitation remains outside M25.
|
||||
+20
-625
@@ -10,8 +10,14 @@ mod draft;
|
||||
mod first_delivery;
|
||||
mod first_file;
|
||||
mod reissue;
|
||||
mod restart;
|
||||
mod restore;
|
||||
|
||||
pub use restart::{
|
||||
RestartChoice, RestartDiagnostic, RestartFailureKind, RestartPreview, RestartReport,
|
||||
RestartStage,
|
||||
};
|
||||
|
||||
use draft::DraftChanges;
|
||||
use first_delivery::{FirstDeliverySession, FirstIssueAuthorization};
|
||||
|
||||
@@ -22,8 +28,6 @@ use std::net::IpAddr;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use thiserror::Error;
|
||||
use time::OffsetDateTime;
|
||||
use time::format_description::well_known::Rfc3339;
|
||||
|
||||
use crate::backup::{self, BackupError, BackupReason, HistoricalApplyStatus, SavedBytes};
|
||||
use crate::delivery::files::FileOutputError;
|
||||
@@ -34,8 +38,8 @@ use crate::domain::{
|
||||
};
|
||||
use crate::persistence::{BackupComplete, PersistedState, PersistenceError, SavedStore};
|
||||
use crate::runtime::transport::TransportError;
|
||||
use crate::runtime::{self, RuntimeObservation, SavedPeerProjection, TechnicalMatch};
|
||||
use crate::service::{self, RestartDeadline, RestartExecution, RestartOutcome};
|
||||
use crate::runtime::{self, RuntimeObservation, SavedPeerProjection};
|
||||
use crate::service::{self, RestartDeadline, RestartExecution};
|
||||
use crate::vault::source::{
|
||||
PreparedImportSetup, SelectedFileImportReport, SourceImportError, SourceImportReport,
|
||||
};
|
||||
@@ -210,96 +214,6 @@ impl fmt::Debug for ReissueDelivery {
|
||||
}
|
||||
}
|
||||
|
||||
/// Read-only, saved-generation-bound restart preview for a future confirmation UI.
|
||||
pub struct RestartPreview {
|
||||
saved: SavedBytes,
|
||||
interface: String,
|
||||
unit: String,
|
||||
timeout: Duration,
|
||||
projection: SavedPeerProjection,
|
||||
expected_clients: Option<Snapshot>,
|
||||
has_draft: bool,
|
||||
}
|
||||
|
||||
impl RestartPreview {
|
||||
pub fn interface(&self) -> &str {
|
||||
&self.interface
|
||||
}
|
||||
|
||||
pub fn systemd_unit(&self) -> &str {
|
||||
&self.unit
|
||||
}
|
||||
|
||||
pub const fn has_draft(&self) -> bool {
|
||||
self.has_draft
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||
pub enum RestartChoice {
|
||||
Cancel,
|
||||
RestartSavedState,
|
||||
DiscardDraftAndRestart,
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||
pub enum RestartStage {
|
||||
Cancelled,
|
||||
Preparation,
|
||||
Backup,
|
||||
Service,
|
||||
RuntimeRead,
|
||||
Verification,
|
||||
Persistence,
|
||||
Complete,
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||
pub enum RestartFailureKind {
|
||||
InvalidOrStaleSavedGeneration,
|
||||
UnsafeSaveConfig,
|
||||
UnsupportedSavedShape,
|
||||
BackupFailed,
|
||||
DraftDiscardFailed,
|
||||
ServiceFailed,
|
||||
RuntimeReadFailed,
|
||||
RuntimeNoDevice,
|
||||
RuntimePermissionDenied,
|
||||
RuntimeMalformed,
|
||||
RuntimeTransport,
|
||||
RuntimeTimeout,
|
||||
RuntimeLimitExceeded,
|
||||
RuntimeMismatch,
|
||||
VerificationPersistenceFailed,
|
||||
}
|
||||
|
||||
/// Secret-free explanation for saved configuration that cannot be used as a
|
||||
/// restart verification target. The identity is a public WireGuard key.
|
||||
#[derive(Clone, Debug, Eq, PartialEq)]
|
||||
pub struct RestartDiagnostic {
|
||||
pub identity: Option<PublicKey>,
|
||||
pub section_line: Option<usize>,
|
||||
pub reason: &'static str,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Eq, PartialEq)]
|
||||
pub struct RestartReport {
|
||||
pub stage: RestartStage,
|
||||
pub failure: Option<RestartFailureKind>,
|
||||
pub service_failure: Option<service::RestartFailure>,
|
||||
pub service_phase: Option<service::RestartPhase>,
|
||||
pub systemd_unit: Option<String>,
|
||||
pub backup_path: Option<std::path::PathBuf>,
|
||||
pub restart_submitted: bool,
|
||||
pub restart_may_still_complete: bool,
|
||||
pub manual_recovery_required: bool,
|
||||
}
|
||||
|
||||
struct RestartAttempt {
|
||||
id: u64,
|
||||
saved: SavedBytes,
|
||||
}
|
||||
|
||||
/// Read-only restore details and an opaque, byte-bound confirmation value.
|
||||
/// The value contains the validated archive payload; confirmation never rereads
|
||||
/// a path whose contents may have changed since preview.
|
||||
@@ -690,7 +604,7 @@ impl Application {
|
||||
if !same_saved_bytes(&self.saved, installed) || self.verified_saved_projection.is_none() {
|
||||
return false;
|
||||
}
|
||||
let Ok((projection, _)) = restart_projection_from_document(document) else {
|
||||
let Ok((projection, _)) = restart::restart_projection_from_document(document) else {
|
||||
return false;
|
||||
};
|
||||
self.draft.states.applied() == &ApplyStatus::Verified
|
||||
@@ -1030,37 +944,7 @@ impl Application {
|
||||
|
||||
/// Prepare a read-only preview bound to the exact currently installed generation.
|
||||
pub fn preview_restart(&self) -> Result<RestartPreview, AppError> {
|
||||
let installed = self.store.read_saved()?;
|
||||
if !same_saved_bytes(&installed, &self.saved) {
|
||||
return Err(AppError::SavedGenerationChanged);
|
||||
}
|
||||
let settings = self.store.resolved_settings_for(&installed)?;
|
||||
let document = ConfigDocument::parse(&installed.server_config)?;
|
||||
if document
|
||||
.sections()
|
||||
.iter()
|
||||
.filter(|section| section.kind == SectionKind::Interface)
|
||||
.count()
|
||||
!= 1
|
||||
{
|
||||
return Err(AppError::InvalidSavedState);
|
||||
}
|
||||
if document.has_unsafe_save_config() {
|
||||
return Err(AppError::UnsafeSaveConfig);
|
||||
}
|
||||
let (projection, expected_clients) =
|
||||
restart_projection_from_document(&document).map_err(AppError::UnsafeSavedPeers)?;
|
||||
let expected_clients =
|
||||
expected_clients.filter(|snapshot| snapshot == self.draft.states.saved());
|
||||
Ok(RestartPreview {
|
||||
saved: clone_saved(&installed),
|
||||
interface: settings.interface_name,
|
||||
unit: settings.systemd_unit,
|
||||
timeout: settings.restart_timeout,
|
||||
projection,
|
||||
expected_clients,
|
||||
has_draft: self.has_draft(),
|
||||
})
|
||||
restart::preview_restart(self)
|
||||
}
|
||||
|
||||
fn has_draft(&self) -> bool {
|
||||
@@ -1152,317 +1036,11 @@ impl Application {
|
||||
install: impl FnOnce(&SavedStore, &SavedBytes, BackupComplete) -> Result<(), PersistenceError>,
|
||||
now: impl Fn() -> Instant,
|
||||
) -> RestartReport {
|
||||
if choice == RestartChoice::Cancel {
|
||||
return restart_report(RestartStage::Cancelled, None, None, false, false, false);
|
||||
}
|
||||
self.active_restart_attempt = None;
|
||||
if self.has_draft() != preview.has_draft
|
||||
|| (preview.has_draft && choice != RestartChoice::DiscardDraftAndRestart)
|
||||
|| (!preview.has_draft && choice == RestartChoice::DiscardDraftAndRestart)
|
||||
{
|
||||
return restart_report(
|
||||
RestartStage::Preparation,
|
||||
Some(RestartFailureKind::InvalidOrStaleSavedGeneration),
|
||||
None,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
);
|
||||
}
|
||||
let installed = match self.store.read_saved() {
|
||||
Ok(saved)
|
||||
if same_saved_bytes(&saved, &preview.saved)
|
||||
&& same_saved_bytes(&saved, &self.saved) =>
|
||||
{
|
||||
saved
|
||||
}
|
||||
_ => {
|
||||
self.active_restart_attempt = None;
|
||||
return restart_report(
|
||||
RestartStage::Preparation,
|
||||
Some(RestartFailureKind::InvalidOrStaleSavedGeneration),
|
||||
None,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
);
|
||||
}
|
||||
let prepared = match restart::prepare_confirmed_restart(self, preview, choice, &now) {
|
||||
Ok(prepared) => prepared,
|
||||
Err(report) => return report,
|
||||
};
|
||||
let settings = match self.store.resolved_settings_for(&installed) {
|
||||
Ok(settings) => settings,
|
||||
Err(_) => {
|
||||
return restart_report(
|
||||
RestartStage::Preparation,
|
||||
Some(RestartFailureKind::InvalidOrStaleSavedGeneration),
|
||||
None,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
);
|
||||
}
|
||||
};
|
||||
let document = match ConfigDocument::parse(&installed.server_config) {
|
||||
Ok(document) => document,
|
||||
Err(_) => {
|
||||
return restart_report(
|
||||
RestartStage::Preparation,
|
||||
Some(RestartFailureKind::InvalidOrStaleSavedGeneration),
|
||||
None,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
);
|
||||
}
|
||||
};
|
||||
if document
|
||||
.sections()
|
||||
.iter()
|
||||
.filter(|section| section.kind == SectionKind::Interface)
|
||||
.count()
|
||||
!= 1
|
||||
{
|
||||
return restart_report(
|
||||
RestartStage::Preparation,
|
||||
Some(RestartFailureKind::InvalidOrStaleSavedGeneration),
|
||||
None,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
);
|
||||
}
|
||||
if document.has_unsafe_save_config() {
|
||||
return restart_report(
|
||||
RestartStage::Preparation,
|
||||
Some(RestartFailureKind::UnsafeSaveConfig),
|
||||
None,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
);
|
||||
}
|
||||
let (projection, expected_clients) = match restart_projection_from_document(&document) {
|
||||
Ok(projection) => projection,
|
||||
Err(_) => {
|
||||
return restart_report(
|
||||
RestartStage::Preparation,
|
||||
Some(RestartFailureKind::UnsupportedSavedShape),
|
||||
None,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
);
|
||||
}
|
||||
};
|
||||
let expected_clients =
|
||||
expected_clients.filter(|snapshot| snapshot == self.draft.states.saved());
|
||||
if settings.interface_name != preview.interface
|
||||
|| settings.systemd_unit != preview.unit
|
||||
|| settings.restart_timeout != preview.timeout
|
||||
|| projection != preview.projection
|
||||
|| expected_clients != preview.expected_clients
|
||||
{
|
||||
return restart_report(
|
||||
RestartStage::Preparation,
|
||||
Some(RestartFailureKind::InvalidOrStaleSavedGeneration),
|
||||
None,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
);
|
||||
}
|
||||
let historical_apply = match historical_apply_from_saved(&installed) {
|
||||
Ok(status) => status,
|
||||
Err(_) => {
|
||||
return restart_report(
|
||||
RestartStage::Preparation,
|
||||
Some(RestartFailureKind::InvalidOrStaleSavedGeneration),
|
||||
None,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
);
|
||||
}
|
||||
};
|
||||
let backup_path = match backup::create_archive(
|
||||
self.store.archives_directory(),
|
||||
BackupReason::PreRestart,
|
||||
historical_apply,
|
||||
&installed,
|
||||
) {
|
||||
Ok(path) => path,
|
||||
Err(_) => {
|
||||
self.active_restart_attempt = None;
|
||||
return restart_report(
|
||||
RestartStage::Backup,
|
||||
Some(RestartFailureKind::BackupFailed),
|
||||
None,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
);
|
||||
}
|
||||
};
|
||||
|
||||
let id = self.next_restart_attempt;
|
||||
let Some(next_id) = id.checked_add(1) else {
|
||||
return restart_report(
|
||||
RestartStage::Preparation,
|
||||
Some(RestartFailureKind::InvalidOrStaleSavedGeneration),
|
||||
Some(backup_path),
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
);
|
||||
};
|
||||
self.next_restart_attempt = next_id;
|
||||
self.active_restart_attempt = Some(id);
|
||||
if choice == RestartChoice::DiscardDraftAndRestart && self.discard_draft().is_err() {
|
||||
self.active_restart_attempt = None;
|
||||
return restart_report(
|
||||
RestartStage::Preparation,
|
||||
Some(RestartFailureKind::DraftDiscardFailed),
|
||||
Some(backup_path),
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
);
|
||||
}
|
||||
|
||||
self.draft.states.runtime_knowledge_unknown();
|
||||
let deadline = RestartDeadline::new(now(), settings.restart_timeout);
|
||||
let execution = service(&settings.systemd_unit, deadline);
|
||||
let job = execution.outcome;
|
||||
let submitted = execution.submitted;
|
||||
let service_phase = execution.phase;
|
||||
if job != RestartOutcome::Succeeded {
|
||||
self.active_restart_attempt = None;
|
||||
let may_complete = submitted
|
||||
&& !matches!(
|
||||
job,
|
||||
RestartOutcome::Failed(
|
||||
service::RestartFailure::JobFailed
|
||||
| service::RestartFailure::JobDependency
|
||||
| service::RestartFailure::JobTimedOut
|
||||
| service::RestartFailure::JobCanceled
|
||||
| service::RestartFailure::JobSkipped
|
||||
| service::RestartFailure::JobResultUnknown
|
||||
)
|
||||
);
|
||||
let mut report = restart_report(
|
||||
RestartStage::Service,
|
||||
Some(RestartFailureKind::ServiceFailed),
|
||||
Some(backup_path),
|
||||
submitted,
|
||||
may_complete,
|
||||
submitted,
|
||||
);
|
||||
report.service_failure = match job {
|
||||
RestartOutcome::Failed(failure) => Some(failure),
|
||||
RestartOutcome::Waiting | RestartOutcome::Succeeded => None,
|
||||
};
|
||||
report.service_phase = Some(service_phase);
|
||||
report.systemd_unit = Some(settings.systemd_unit);
|
||||
return report;
|
||||
}
|
||||
let remaining = deadline.remaining_at(now());
|
||||
if remaining.is_zero() {
|
||||
self.active_restart_attempt = None;
|
||||
return restart_report(
|
||||
RestartStage::RuntimeRead,
|
||||
Some(RestartFailureKind::RuntimeTimeout),
|
||||
Some(backup_path),
|
||||
true,
|
||||
false,
|
||||
true,
|
||||
);
|
||||
}
|
||||
let observation = match runtime_read(&settings.interface_name, remaining) {
|
||||
Ok(observation) => observation,
|
||||
Err(error) => {
|
||||
self.active_restart_attempt = None;
|
||||
return restart_report(
|
||||
RestartStage::RuntimeRead,
|
||||
Some(match error {
|
||||
TransportError::NoDevice => RestartFailureKind::RuntimeNoDevice,
|
||||
TransportError::PermissionDenied => {
|
||||
RestartFailureKind::RuntimePermissionDenied
|
||||
}
|
||||
TransportError::Malformed => RestartFailureKind::RuntimeMalformed,
|
||||
TransportError::Transport => RestartFailureKind::RuntimeTransport,
|
||||
TransportError::Timeout => RestartFailureKind::RuntimeTimeout,
|
||||
TransportError::LimitExceeded => RestartFailureKind::RuntimeLimitExceeded,
|
||||
}),
|
||||
Some(backup_path),
|
||||
true,
|
||||
false,
|
||||
true,
|
||||
);
|
||||
}
|
||||
};
|
||||
if deadline.is_expired_at(now()) {
|
||||
self.active_restart_attempt = None;
|
||||
return restart_report(
|
||||
RestartStage::Verification,
|
||||
Some(RestartFailureKind::RuntimeTimeout),
|
||||
Some(backup_path),
|
||||
true,
|
||||
false,
|
||||
true,
|
||||
);
|
||||
}
|
||||
if runtime::compare_peers(&projection, &settings.interface_name, &observation)
|
||||
!= TechnicalMatch::Match
|
||||
{
|
||||
self.active_restart_attempt = None;
|
||||
return restart_report(
|
||||
RestartStage::Verification,
|
||||
Some(RestartFailureKind::RuntimeMismatch),
|
||||
Some(backup_path),
|
||||
true,
|
||||
false,
|
||||
true,
|
||||
);
|
||||
}
|
||||
let Some(verified_at) = OffsetDateTime::now_utc()
|
||||
.replace_nanosecond(0)
|
||||
.ok()
|
||||
.and_then(|timestamp| timestamp.format(&Rfc3339).ok())
|
||||
else {
|
||||
self.active_restart_attempt = None;
|
||||
return restart_report(
|
||||
RestartStage::Persistence,
|
||||
Some(RestartFailureKind::VerificationPersistenceFailed),
|
||||
Some(backup_path),
|
||||
true,
|
||||
false,
|
||||
true,
|
||||
);
|
||||
};
|
||||
let attempt = RestartAttempt {
|
||||
id,
|
||||
saved: clone_saved(&installed),
|
||||
};
|
||||
let marked =
|
||||
self.mark_verified_with_attempt(attempt, expected_clients, verified_at, install);
|
||||
if marked.is_err() {
|
||||
self.active_restart_attempt = None;
|
||||
return restart_report(
|
||||
RestartStage::Persistence,
|
||||
Some(RestartFailureKind::VerificationPersistenceFailed),
|
||||
Some(backup_path),
|
||||
true,
|
||||
false,
|
||||
true,
|
||||
);
|
||||
}
|
||||
restart_report(
|
||||
RestartStage::Complete,
|
||||
None,
|
||||
Some(backup_path),
|
||||
true,
|
||||
false,
|
||||
false,
|
||||
)
|
||||
restart::execute_confirmed_restart(self, prepared, service, runtime_read, install, now)
|
||||
}
|
||||
|
||||
/// Validate a selected archive without changing saved files or the draft.
|
||||
@@ -1491,78 +1069,6 @@ impl Application {
|
||||
restore::confirm_restore(self, preview, install)
|
||||
}
|
||||
|
||||
/// Persist applied state only with the one-use authorization produced by
|
||||
/// the active confirmed restart after full technical verification.
|
||||
fn mark_verified_with_attempt(
|
||||
&mut self,
|
||||
attempt: RestartAttempt,
|
||||
observed_management_snapshot: Option<Snapshot>,
|
||||
verified_at_utc: String,
|
||||
install: impl FnOnce(&SavedStore, &SavedBytes, BackupComplete) -> Result<(), PersistenceError>,
|
||||
) -> Result<(), AppError> {
|
||||
if self.active_restart_attempt != Some(attempt.id) {
|
||||
return Err(AppError::PreRestartBackupRequired);
|
||||
}
|
||||
let installed = self.store.read_saved()?;
|
||||
if !same_saved_bytes(&installed, &self.saved)
|
||||
|| !same_saved_bytes(&installed, &attempt.saved)
|
||||
{
|
||||
return Err(AppError::SavedGenerationChanged);
|
||||
}
|
||||
let mut next_states = self.draft.states.clone();
|
||||
next_states.prepare_technically_verified_result(observed_management_snapshot)?;
|
||||
let mut persisted = match &installed.internal_state {
|
||||
Some(bytes) => PersistedState::from_toml(bytes)?,
|
||||
None => PersistedState::new(
|
||||
next_states
|
||||
.reserved_addresses()
|
||||
.iter()
|
||||
.map(ToString::to_string),
|
||||
None,
|
||||
)?,
|
||||
};
|
||||
persisted.reserved_addresses = next_states
|
||||
.reserved_addresses()
|
||||
.iter()
|
||||
.map(ToString::to_string)
|
||||
.collect();
|
||||
let historical_apply = HistoricalApplyStatus::Verified {
|
||||
verified_at_utc: verified_at_utc.clone(),
|
||||
};
|
||||
persisted.last_verified_at_utc = Some(verified_at_utc);
|
||||
let mut proposed = installed;
|
||||
proposed.internal_state = Some(persisted.to_toml()?);
|
||||
self.store.validate_saved(&proposed)?;
|
||||
if let Err(error) = install(&self.store, &proposed, BackupComplete(())) {
|
||||
self.first_delivery.invalidate_active_authorizations();
|
||||
match self.store.read_saved() {
|
||||
Ok(current) => {
|
||||
let document = ConfigDocument::parse(¤t.server_config).ok();
|
||||
let vault = current
|
||||
.vault
|
||||
.as_deref()
|
||||
.and_then(|bytes| std::str::from_utf8(bytes).ok())
|
||||
.and_then(|text| VaultBundle::from_toml(text).ok());
|
||||
if let Some(document) = document {
|
||||
self.first_delivery
|
||||
.retain_matching(&document, vault.as_ref());
|
||||
} else {
|
||||
self.first_delivery.clear();
|
||||
}
|
||||
}
|
||||
Err(_) => self.first_delivery.clear(),
|
||||
}
|
||||
return Err(error.into());
|
||||
}
|
||||
self.saved = proposed;
|
||||
self.draft.states = next_states;
|
||||
self.verified_saved_projection =
|
||||
Some(SavedPeerProjection::from_document(&self.draft.document));
|
||||
self.historical_apply = historical_apply;
|
||||
self.active_restart_attempt = None;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Commit current configuration, vault, and reservations. The archive is
|
||||
/// complete before `SavedStore::install` receives its capability. In-memory
|
||||
/// saved state advances only after the adapter reports full success.
|
||||
@@ -1685,119 +1191,6 @@ fn document_host_addresses(document: &ConfigDocument) -> Vec<IpAddr> {
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn restart_projection_from_document(
|
||||
document: &ConfigDocument,
|
||||
) -> Result<(SavedPeerProjection, Option<Snapshot>), Vec<RestartDiagnostic>> {
|
||||
use crate::wireguard::{DocumentIssue, PeerIssue};
|
||||
|
||||
let mut diagnostics = Vec::new();
|
||||
for issue in document.issues() {
|
||||
let (line, reason) = match issue {
|
||||
DocumentIssue::MalformedSectionHeader { line } => (*line, "malformed section header"),
|
||||
DocumentIssue::DirectiveOutsideSection { line } => {
|
||||
(*line, "directive appears outside a section")
|
||||
}
|
||||
DocumentIssue::MalformedDirective { line } => (*line, "malformed directive"),
|
||||
// Managed metadata is descriptive and does not affect the technical
|
||||
// peer set used for runtime verification.
|
||||
DocumentIssue::ManagedMetadata(_) => continue,
|
||||
};
|
||||
diagnostics.push(RestartDiagnostic {
|
||||
identity: None,
|
||||
section_line: Some(line),
|
||||
reason,
|
||||
});
|
||||
}
|
||||
|
||||
for peer in document.peers() {
|
||||
let identity = peer
|
||||
.identity
|
||||
.or_else(|| peer.public_key_candidates.first().copied());
|
||||
for issue in &peer.issues {
|
||||
let reason = match issue {
|
||||
PeerIssue::MissingPublicKey => Some("peer is missing a public key"),
|
||||
PeerIssue::MalformedPublicKey { .. } => Some("peer has a malformed public key"),
|
||||
PeerIssue::DuplicatePublicKeyField { .. } => {
|
||||
Some("peer has duplicate public key fields")
|
||||
}
|
||||
PeerIssue::DuplicatePeerIdentity { .. } => {
|
||||
Some("public key is used by multiple peer sections")
|
||||
}
|
||||
PeerIssue::MalformedAllowedIp { .. } => {
|
||||
Some("peer has a malformed AllowedIPs item")
|
||||
}
|
||||
PeerIssue::UnsupportedAllowedIpShape { .. } | PeerIssue::MissingIpv4HostRoute => {
|
||||
None
|
||||
}
|
||||
};
|
||||
if let Some(reason) = reason {
|
||||
diagnostics.push(RestartDiagnostic {
|
||||
identity,
|
||||
section_line: Some(peer.section_line),
|
||||
reason,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let projection = SavedPeerProjection::from_document(document);
|
||||
if (projection.document_unverifiable
|
||||
|| !projection.unverifiable_peer_indices.is_empty()
|
||||
|| !projection.duplicate_public_keys.is_empty())
|
||||
&& diagnostics.is_empty()
|
||||
{
|
||||
diagnostics.push(RestartDiagnostic {
|
||||
identity: None,
|
||||
section_line: None,
|
||||
reason: "saved peer set is not uniquely verifiable",
|
||||
});
|
||||
}
|
||||
if !diagnostics.is_empty() {
|
||||
return Err(diagnostics);
|
||||
}
|
||||
|
||||
let mut clients = Vec::with_capacity(document.peers().len());
|
||||
for peer in document.peers() {
|
||||
let Some(identity) = peer.identity else {
|
||||
return Err(vec![RestartDiagnostic {
|
||||
identity: None,
|
||||
section_line: Some(peer.section_line),
|
||||
reason: "peer identity is unavailable",
|
||||
}]);
|
||||
};
|
||||
let Ok(client) = document.project_client(&identity) else {
|
||||
clients.clear();
|
||||
break;
|
||||
};
|
||||
clients.push(client);
|
||||
}
|
||||
let management_snapshot = (clients.len() == document.peers().len())
|
||||
.then(|| Snapshot::new(clients).ok())
|
||||
.flatten();
|
||||
Ok((projection, management_snapshot))
|
||||
}
|
||||
|
||||
fn restart_report(
|
||||
stage: RestartStage,
|
||||
failure: Option<RestartFailureKind>,
|
||||
backup_path: Option<std::path::PathBuf>,
|
||||
restart_submitted: bool,
|
||||
restart_may_still_complete: bool,
|
||||
manual_recovery_required: bool,
|
||||
) -> RestartReport {
|
||||
RestartReport {
|
||||
stage,
|
||||
failure,
|
||||
service_failure: None,
|
||||
service_phase: None,
|
||||
systemd_unit: None,
|
||||
backup_path,
|
||||
restart_submitted,
|
||||
restart_may_still_complete,
|
||||
manual_recovery_required,
|
||||
}
|
||||
}
|
||||
|
||||
fn snapshot_from_document(document: &ConfigDocument) -> Snapshot {
|
||||
let mut snapshot = Snapshot::default();
|
||||
for client in document.peers().iter().filter_map(|peer| {
|
||||
@@ -1973,11 +1366,12 @@ restart_timeout_seconds = 30
|
||||
let id = app.next_restart_attempt;
|
||||
app.next_restart_attempt += 1;
|
||||
app.active_restart_attempt = Some(id);
|
||||
let attempt = super::RestartAttempt {
|
||||
let attempt = super::restart::RestartAttempt {
|
||||
id,
|
||||
saved: clone_saved(&app.saved),
|
||||
};
|
||||
app.mark_verified_with_attempt(
|
||||
super::restart::mark_verified_with_attempt(
|
||||
app,
|
||||
attempt,
|
||||
Some(observed),
|
||||
verified_at,
|
||||
@@ -4693,11 +4087,12 @@ restart_timeout_seconds = 30
|
||||
let id = app.next_restart_attempt;
|
||||
app.next_restart_attempt += 1;
|
||||
app.active_restart_attempt = Some(id);
|
||||
let attempt = super::RestartAttempt {
|
||||
let attempt = super::restart::RestartAttempt {
|
||||
id,
|
||||
saved: clone_saved(&app.saved),
|
||||
};
|
||||
let result = app.mark_verified_with_attempt(
|
||||
let result = super::restart::mark_verified_with_attempt(
|
||||
&mut app,
|
||||
attempt,
|
||||
Some(observed),
|
||||
"2026-10-20T08:30:00Z".to_owned(),
|
||||
|
||||
@@ -0,0 +1,670 @@
|
||||
//! Saved-generation-bound restart preparation, execution, and publication.
|
||||
//!
|
||||
//! This module coordinates the service job, runtime observation and comparison,
|
||||
//! durable verification bookkeeping, and applied state publication. Low-level
|
||||
//! service and runtime adapters remain separate.
|
||||
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use crate::backup::{self, BackupReason, HistoricalApplyStatus, SavedBytes};
|
||||
use crate::config::ResolvedSettings;
|
||||
use crate::domain::{PublicKey, Snapshot};
|
||||
use crate::persistence::{BackupComplete, PersistedState, PersistenceError, SavedStore};
|
||||
use crate::runtime::transport::TransportError;
|
||||
use crate::runtime::{self, RuntimeObservation, SavedPeerProjection, TechnicalMatch};
|
||||
use crate::service::{self, RestartDeadline, RestartExecution, RestartOutcome};
|
||||
use crate::vault::VaultBundle;
|
||||
use crate::wireguard::{ConfigDocument, SectionKind};
|
||||
use time::OffsetDateTime;
|
||||
use time::format_description::well_known::Rfc3339;
|
||||
|
||||
use super::{AppError, Application, clone_saved, historical_apply_from_saved, same_saved_bytes};
|
||||
|
||||
/// Read-only, saved-generation-bound restart preview for a future confirmation UI.
|
||||
pub struct RestartPreview {
|
||||
saved: SavedBytes,
|
||||
interface: String,
|
||||
unit: String,
|
||||
timeout: Duration,
|
||||
projection: SavedPeerProjection,
|
||||
expected_clients: Option<Snapshot>,
|
||||
has_draft: bool,
|
||||
}
|
||||
|
||||
impl RestartPreview {
|
||||
pub fn interface(&self) -> &str {
|
||||
&self.interface
|
||||
}
|
||||
|
||||
pub fn systemd_unit(&self) -> &str {
|
||||
&self.unit
|
||||
}
|
||||
|
||||
pub const fn has_draft(&self) -> bool {
|
||||
self.has_draft
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||
pub enum RestartChoice {
|
||||
Cancel,
|
||||
RestartSavedState,
|
||||
DiscardDraftAndRestart,
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||
pub enum RestartStage {
|
||||
Cancelled,
|
||||
Preparation,
|
||||
Backup,
|
||||
Service,
|
||||
RuntimeRead,
|
||||
Verification,
|
||||
Persistence,
|
||||
Complete,
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||
pub enum RestartFailureKind {
|
||||
InvalidOrStaleSavedGeneration,
|
||||
UnsafeSaveConfig,
|
||||
UnsupportedSavedShape,
|
||||
BackupFailed,
|
||||
DraftDiscardFailed,
|
||||
ServiceFailed,
|
||||
RuntimeReadFailed,
|
||||
RuntimeNoDevice,
|
||||
RuntimePermissionDenied,
|
||||
RuntimeMalformed,
|
||||
RuntimeTransport,
|
||||
RuntimeTimeout,
|
||||
RuntimeLimitExceeded,
|
||||
RuntimeMismatch,
|
||||
VerificationPersistenceFailed,
|
||||
}
|
||||
|
||||
/// Secret-free explanation for saved configuration that cannot be used as a
|
||||
/// restart verification target. The identity is a public WireGuard key.
|
||||
#[derive(Clone, Debug, Eq, PartialEq)]
|
||||
pub struct RestartDiagnostic {
|
||||
pub identity: Option<PublicKey>,
|
||||
pub section_line: Option<usize>,
|
||||
pub reason: &'static str,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Eq, PartialEq)]
|
||||
pub struct RestartReport {
|
||||
pub stage: RestartStage,
|
||||
pub failure: Option<RestartFailureKind>,
|
||||
pub service_failure: Option<service::RestartFailure>,
|
||||
pub service_phase: Option<service::RestartPhase>,
|
||||
pub systemd_unit: Option<String>,
|
||||
pub backup_path: Option<std::path::PathBuf>,
|
||||
pub restart_submitted: bool,
|
||||
pub restart_may_still_complete: bool,
|
||||
pub manual_recovery_required: bool,
|
||||
}
|
||||
|
||||
pub(super) struct RestartAttempt {
|
||||
pub(super) id: u64,
|
||||
pub(super) saved: SavedBytes,
|
||||
}
|
||||
|
||||
pub(super) struct PreparedRestart {
|
||||
pub(super) attempt_id: u64,
|
||||
pub(super) installed: SavedBytes,
|
||||
pub(super) settings: ResolvedSettings,
|
||||
pub(super) projection: SavedPeerProjection,
|
||||
pub(super) expected_clients: Option<Snapshot>,
|
||||
pub(super) backup_path: std::path::PathBuf,
|
||||
pub(super) deadline: service::RestartDeadline,
|
||||
}
|
||||
|
||||
pub(super) fn preview_restart(application: &Application) -> Result<RestartPreview, AppError> {
|
||||
let installed = application.store.read_saved()?;
|
||||
if !same_saved_bytes(&installed, &application.saved) {
|
||||
return Err(AppError::SavedGenerationChanged);
|
||||
}
|
||||
let settings = application.store.resolved_settings_for(&installed)?;
|
||||
let document = ConfigDocument::parse(&installed.server_config)?;
|
||||
if document
|
||||
.sections()
|
||||
.iter()
|
||||
.filter(|section| section.kind == SectionKind::Interface)
|
||||
.count()
|
||||
!= 1
|
||||
{
|
||||
return Err(AppError::InvalidSavedState);
|
||||
}
|
||||
if document.has_unsafe_save_config() {
|
||||
return Err(AppError::UnsafeSaveConfig);
|
||||
}
|
||||
let (projection, expected_clients) =
|
||||
restart_projection_from_document(&document).map_err(AppError::UnsafeSavedPeers)?;
|
||||
let expected_clients =
|
||||
expected_clients.filter(|snapshot| snapshot == application.draft.states.saved());
|
||||
Ok(RestartPreview {
|
||||
saved: clone_saved(&installed),
|
||||
interface: settings.interface_name,
|
||||
unit: settings.systemd_unit,
|
||||
timeout: settings.restart_timeout,
|
||||
projection,
|
||||
expected_clients,
|
||||
has_draft: application.has_draft(),
|
||||
})
|
||||
}
|
||||
|
||||
/// Validate the reviewed preview and preserve the established preparation
|
||||
/// order: finalize the pre-restart backup before discarding any draft, then bind
|
||||
/// the one-use attempt and initialize the single deadline before service work.
|
||||
pub(super) fn prepare_confirmed_restart(
|
||||
application: &mut Application,
|
||||
preview: RestartPreview,
|
||||
choice: RestartChoice,
|
||||
now: &impl Fn() -> Instant,
|
||||
) -> Result<PreparedRestart, RestartReport> {
|
||||
if choice == RestartChoice::Cancel {
|
||||
return Err(report(
|
||||
RestartStage::Cancelled,
|
||||
None,
|
||||
None,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
));
|
||||
}
|
||||
application.active_restart_attempt = None;
|
||||
if application.has_draft() != preview.has_draft
|
||||
|| (preview.has_draft && choice != RestartChoice::DiscardDraftAndRestart)
|
||||
|| (!preview.has_draft && choice == RestartChoice::DiscardDraftAndRestart)
|
||||
{
|
||||
return Err(report(
|
||||
RestartStage::Preparation,
|
||||
Some(RestartFailureKind::InvalidOrStaleSavedGeneration),
|
||||
None,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
));
|
||||
}
|
||||
let installed = match application.store.read_saved() {
|
||||
Ok(saved)
|
||||
if same_saved_bytes(&saved, &preview.saved)
|
||||
&& same_saved_bytes(&saved, &application.saved) =>
|
||||
{
|
||||
saved
|
||||
}
|
||||
_ => {
|
||||
application.active_restart_attempt = None;
|
||||
return Err(report(
|
||||
RestartStage::Preparation,
|
||||
Some(RestartFailureKind::InvalidOrStaleSavedGeneration),
|
||||
None,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
));
|
||||
}
|
||||
};
|
||||
let settings = match application.store.resolved_settings_for(&installed) {
|
||||
Ok(settings) => settings,
|
||||
Err(_) => return Err(preparation_failure()),
|
||||
};
|
||||
let document = match ConfigDocument::parse(&installed.server_config) {
|
||||
Ok(document) => document,
|
||||
Err(_) => return Err(preparation_failure()),
|
||||
};
|
||||
if document
|
||||
.sections()
|
||||
.iter()
|
||||
.filter(|section| section.kind == SectionKind::Interface)
|
||||
.count()
|
||||
!= 1
|
||||
{
|
||||
return Err(preparation_failure());
|
||||
}
|
||||
if document.has_unsafe_save_config() {
|
||||
return Err(report(
|
||||
RestartStage::Preparation,
|
||||
Some(RestartFailureKind::UnsafeSaveConfig),
|
||||
None,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
));
|
||||
}
|
||||
let (projection, expected_clients) = match restart_projection_from_document(&document) {
|
||||
Ok(projection) => projection,
|
||||
Err(_) => {
|
||||
return Err(report(
|
||||
RestartStage::Preparation,
|
||||
Some(RestartFailureKind::UnsupportedSavedShape),
|
||||
None,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
));
|
||||
}
|
||||
};
|
||||
let expected_clients =
|
||||
expected_clients.filter(|snapshot| snapshot == application.draft.states.saved());
|
||||
if settings.interface_name != preview.interface
|
||||
|| settings.systemd_unit != preview.unit
|
||||
|| settings.restart_timeout != preview.timeout
|
||||
|| projection != preview.projection
|
||||
|| expected_clients != preview.expected_clients
|
||||
{
|
||||
return Err(preparation_failure());
|
||||
}
|
||||
let historical_apply = match historical_apply_from_saved(&installed) {
|
||||
Ok(status) => status,
|
||||
Err(_) => return Err(preparation_failure()),
|
||||
};
|
||||
let backup_path = match backup::create_archive(
|
||||
application.store.archives_directory(),
|
||||
BackupReason::PreRestart,
|
||||
historical_apply,
|
||||
&installed,
|
||||
) {
|
||||
Ok(path) => path,
|
||||
Err(_) => {
|
||||
application.active_restart_attempt = None;
|
||||
return Err(report(
|
||||
RestartStage::Backup,
|
||||
Some(RestartFailureKind::BackupFailed),
|
||||
None,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
));
|
||||
}
|
||||
};
|
||||
|
||||
let id = application.next_restart_attempt;
|
||||
let Some(next_id) = id.checked_add(1) else {
|
||||
return Err(report(
|
||||
RestartStage::Preparation,
|
||||
Some(RestartFailureKind::InvalidOrStaleSavedGeneration),
|
||||
Some(backup_path),
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
));
|
||||
};
|
||||
application.next_restart_attempt = next_id;
|
||||
application.active_restart_attempt = Some(id);
|
||||
if choice == RestartChoice::DiscardDraftAndRestart && application.discard_draft().is_err() {
|
||||
application.active_restart_attempt = None;
|
||||
return Err(report(
|
||||
RestartStage::Preparation,
|
||||
Some(RestartFailureKind::DraftDiscardFailed),
|
||||
Some(backup_path),
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
));
|
||||
}
|
||||
|
||||
application.draft.states.runtime_knowledge_unknown();
|
||||
let deadline = service::RestartDeadline::new(now(), settings.restart_timeout);
|
||||
Ok(PreparedRestart {
|
||||
attempt_id: id,
|
||||
installed,
|
||||
settings,
|
||||
projection,
|
||||
expected_clients,
|
||||
backup_path,
|
||||
deadline,
|
||||
})
|
||||
}
|
||||
|
||||
/// Run the confirmed restart scenario after preparation has completed. The
|
||||
/// backup is already durable before this function is entered; exactly one
|
||||
/// shared deadline bounds the service job and the subsequent runtime read.
|
||||
pub(super) fn execute_confirmed_restart(
|
||||
application: &mut Application,
|
||||
prepared: PreparedRestart,
|
||||
service: impl FnOnce(&str, RestartDeadline) -> RestartExecution,
|
||||
runtime_read: impl FnOnce(&str, Duration) -> Result<RuntimeObservation, TransportError>,
|
||||
install: impl FnOnce(&SavedStore, &SavedBytes, BackupComplete) -> Result<(), PersistenceError>,
|
||||
now: impl Fn() -> Instant,
|
||||
) -> RestartReport {
|
||||
let PreparedRestart {
|
||||
attempt_id,
|
||||
installed,
|
||||
settings,
|
||||
projection,
|
||||
expected_clients,
|
||||
backup_path,
|
||||
deadline,
|
||||
} = prepared;
|
||||
let execution = service(&settings.systemd_unit, deadline);
|
||||
let job = execution.outcome;
|
||||
let submitted = execution.submitted;
|
||||
let service_phase = execution.phase;
|
||||
if job != RestartOutcome::Succeeded {
|
||||
application.active_restart_attempt = None;
|
||||
let may_complete = submitted
|
||||
&& !matches!(
|
||||
job,
|
||||
RestartOutcome::Failed(
|
||||
service::RestartFailure::JobFailed
|
||||
| service::RestartFailure::JobDependency
|
||||
| service::RestartFailure::JobTimedOut
|
||||
| service::RestartFailure::JobCanceled
|
||||
| service::RestartFailure::JobSkipped
|
||||
| service::RestartFailure::JobResultUnknown
|
||||
)
|
||||
);
|
||||
let mut report = report(
|
||||
RestartStage::Service,
|
||||
Some(RestartFailureKind::ServiceFailed),
|
||||
Some(backup_path),
|
||||
submitted,
|
||||
may_complete,
|
||||
submitted,
|
||||
);
|
||||
report.service_failure = match job {
|
||||
RestartOutcome::Failed(failure) => Some(failure),
|
||||
RestartOutcome::Waiting | RestartOutcome::Succeeded => None,
|
||||
};
|
||||
report.service_phase = Some(service_phase);
|
||||
report.systemd_unit = Some(settings.systemd_unit);
|
||||
return report;
|
||||
}
|
||||
let remaining = deadline.remaining_at(now());
|
||||
if remaining.is_zero() {
|
||||
application.active_restart_attempt = None;
|
||||
return report(
|
||||
RestartStage::RuntimeRead,
|
||||
Some(RestartFailureKind::RuntimeTimeout),
|
||||
Some(backup_path),
|
||||
true,
|
||||
false,
|
||||
true,
|
||||
);
|
||||
}
|
||||
let observation = match runtime_read(&settings.interface_name, remaining) {
|
||||
Ok(observation) => observation,
|
||||
Err(error) => {
|
||||
application.active_restart_attempt = None;
|
||||
return report(
|
||||
RestartStage::RuntimeRead,
|
||||
Some(match error {
|
||||
TransportError::NoDevice => RestartFailureKind::RuntimeNoDevice,
|
||||
TransportError::PermissionDenied => RestartFailureKind::RuntimePermissionDenied,
|
||||
TransportError::Malformed => RestartFailureKind::RuntimeMalformed,
|
||||
TransportError::Transport => RestartFailureKind::RuntimeTransport,
|
||||
TransportError::Timeout => RestartFailureKind::RuntimeTimeout,
|
||||
TransportError::LimitExceeded => RestartFailureKind::RuntimeLimitExceeded,
|
||||
}),
|
||||
Some(backup_path),
|
||||
true,
|
||||
false,
|
||||
true,
|
||||
);
|
||||
}
|
||||
};
|
||||
if deadline.is_expired_at(now()) {
|
||||
application.active_restart_attempt = None;
|
||||
return report(
|
||||
RestartStage::Verification,
|
||||
Some(RestartFailureKind::RuntimeTimeout),
|
||||
Some(backup_path),
|
||||
true,
|
||||
false,
|
||||
true,
|
||||
);
|
||||
}
|
||||
if runtime::compare_peers(&projection, &settings.interface_name, &observation)
|
||||
!= TechnicalMatch::Match
|
||||
{
|
||||
application.active_restart_attempt = None;
|
||||
return report(
|
||||
RestartStage::Verification,
|
||||
Some(RestartFailureKind::RuntimeMismatch),
|
||||
Some(backup_path),
|
||||
true,
|
||||
false,
|
||||
true,
|
||||
);
|
||||
}
|
||||
let Some(verified_at) = OffsetDateTime::now_utc()
|
||||
.replace_nanosecond(0)
|
||||
.ok()
|
||||
.and_then(|timestamp| timestamp.format(&Rfc3339).ok())
|
||||
else {
|
||||
application.active_restart_attempt = None;
|
||||
return report(
|
||||
RestartStage::Persistence,
|
||||
Some(RestartFailureKind::VerificationPersistenceFailed),
|
||||
Some(backup_path),
|
||||
true,
|
||||
false,
|
||||
true,
|
||||
);
|
||||
};
|
||||
let attempt = RestartAttempt {
|
||||
id: attempt_id,
|
||||
saved: clone_saved(&installed),
|
||||
};
|
||||
if mark_verified_with_attempt(application, attempt, expected_clients, verified_at, install)
|
||||
.is_err()
|
||||
{
|
||||
application.active_restart_attempt = None;
|
||||
return report(
|
||||
RestartStage::Persistence,
|
||||
Some(RestartFailureKind::VerificationPersistenceFailed),
|
||||
Some(backup_path),
|
||||
true,
|
||||
false,
|
||||
true,
|
||||
);
|
||||
}
|
||||
report(
|
||||
RestartStage::Complete,
|
||||
None,
|
||||
Some(backup_path),
|
||||
true,
|
||||
false,
|
||||
false,
|
||||
)
|
||||
}
|
||||
|
||||
/// Persist verified apply knowledge using the active restart's one-use attempt
|
||||
/// and exact saved generation. Session publication follows durable success.
|
||||
pub(super) fn mark_verified_with_attempt(
|
||||
application: &mut Application,
|
||||
attempt: RestartAttempt,
|
||||
observed_management_snapshot: Option<Snapshot>,
|
||||
verified_at_utc: String,
|
||||
install: impl FnOnce(&SavedStore, &SavedBytes, BackupComplete) -> Result<(), PersistenceError>,
|
||||
) -> Result<(), AppError> {
|
||||
if application.active_restart_attempt != Some(attempt.id) {
|
||||
return Err(AppError::PreRestartBackupRequired);
|
||||
}
|
||||
let installed = application.store.read_saved()?;
|
||||
if !same_saved_bytes(&installed, &application.saved)
|
||||
|| !same_saved_bytes(&installed, &attempt.saved)
|
||||
{
|
||||
return Err(AppError::SavedGenerationChanged);
|
||||
}
|
||||
let mut next_states = application.draft.states.clone();
|
||||
next_states.prepare_technically_verified_result(observed_management_snapshot)?;
|
||||
let mut persisted = match &installed.internal_state {
|
||||
Some(bytes) => PersistedState::from_toml(bytes)?,
|
||||
None => PersistedState::new(
|
||||
next_states
|
||||
.reserved_addresses()
|
||||
.iter()
|
||||
.map(ToString::to_string),
|
||||
None,
|
||||
)?,
|
||||
};
|
||||
persisted.reserved_addresses = next_states
|
||||
.reserved_addresses()
|
||||
.iter()
|
||||
.map(ToString::to_string)
|
||||
.collect();
|
||||
let historical_apply = HistoricalApplyStatus::Verified {
|
||||
verified_at_utc: verified_at_utc.clone(),
|
||||
};
|
||||
persisted.last_verified_at_utc = Some(verified_at_utc);
|
||||
let mut proposed = installed;
|
||||
proposed.internal_state = Some(persisted.to_toml()?);
|
||||
application.store.validate_saved(&proposed)?;
|
||||
if let Err(error) = install(&application.store, &proposed, BackupComplete(())) {
|
||||
application
|
||||
.first_delivery
|
||||
.invalidate_active_authorizations();
|
||||
match application.store.read_saved() {
|
||||
Ok(current) => {
|
||||
let document = ConfigDocument::parse(¤t.server_config).ok();
|
||||
let vault = current
|
||||
.vault
|
||||
.as_deref()
|
||||
.and_then(|bytes| std::str::from_utf8(bytes).ok())
|
||||
.and_then(|text| VaultBundle::from_toml(text).ok());
|
||||
if let Some(document) = document {
|
||||
application
|
||||
.first_delivery
|
||||
.retain_matching(&document, vault.as_ref());
|
||||
} else {
|
||||
application.first_delivery.clear();
|
||||
}
|
||||
}
|
||||
Err(_) => application.first_delivery.clear(),
|
||||
}
|
||||
return Err(error.into());
|
||||
}
|
||||
application.saved = proposed;
|
||||
application.draft.states = next_states;
|
||||
application.verified_saved_projection = Some(SavedPeerProjection::from_document(
|
||||
&application.draft.document,
|
||||
));
|
||||
application.historical_apply = historical_apply;
|
||||
application.active_restart_attempt = None;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn preparation_failure() -> RestartReport {
|
||||
report(
|
||||
RestartStage::Preparation,
|
||||
Some(RestartFailureKind::InvalidOrStaleSavedGeneration),
|
||||
None,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
)
|
||||
}
|
||||
|
||||
pub(super) fn report(
|
||||
stage: RestartStage,
|
||||
failure: Option<RestartFailureKind>,
|
||||
backup_path: Option<std::path::PathBuf>,
|
||||
restart_submitted: bool,
|
||||
restart_may_still_complete: bool,
|
||||
manual_recovery_required: bool,
|
||||
) -> RestartReport {
|
||||
RestartReport {
|
||||
stage,
|
||||
failure,
|
||||
service_failure: None,
|
||||
service_phase: None,
|
||||
systemd_unit: None,
|
||||
backup_path,
|
||||
restart_submitted,
|
||||
restart_may_still_complete,
|
||||
manual_recovery_required,
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) fn restart_projection_from_document(
|
||||
document: &ConfigDocument,
|
||||
) -> Result<(SavedPeerProjection, Option<Snapshot>), Vec<RestartDiagnostic>> {
|
||||
use crate::wireguard::{DocumentIssue, PeerIssue};
|
||||
|
||||
let mut diagnostics = Vec::new();
|
||||
for issue in document.issues() {
|
||||
let (line, reason) = match issue {
|
||||
DocumentIssue::MalformedSectionHeader { line } => (*line, "malformed section header"),
|
||||
DocumentIssue::DirectiveOutsideSection { line } => {
|
||||
(*line, "directive appears outside a section")
|
||||
}
|
||||
DocumentIssue::MalformedDirective { line } => (*line, "malformed directive"),
|
||||
DocumentIssue::ManagedMetadata(_) => continue,
|
||||
};
|
||||
diagnostics.push(RestartDiagnostic {
|
||||
identity: None,
|
||||
section_line: Some(line),
|
||||
reason,
|
||||
});
|
||||
}
|
||||
|
||||
for peer in document.peers() {
|
||||
let identity = peer
|
||||
.identity
|
||||
.or_else(|| peer.public_key_candidates.first().copied());
|
||||
for issue in &peer.issues {
|
||||
let reason = match issue {
|
||||
PeerIssue::MissingPublicKey => Some("peer is missing a public key"),
|
||||
PeerIssue::MalformedPublicKey { .. } => Some("peer has a malformed public key"),
|
||||
PeerIssue::DuplicatePublicKeyField { .. } => {
|
||||
Some("peer has duplicate public key fields")
|
||||
}
|
||||
PeerIssue::DuplicatePeerIdentity { .. } => {
|
||||
Some("public key is used by multiple peer sections")
|
||||
}
|
||||
PeerIssue::MalformedAllowedIp { .. } => {
|
||||
Some("peer has a malformed AllowedIPs item")
|
||||
}
|
||||
PeerIssue::UnsupportedAllowedIpShape { .. } | PeerIssue::MissingIpv4HostRoute => {
|
||||
None
|
||||
}
|
||||
};
|
||||
if let Some(reason) = reason {
|
||||
diagnostics.push(RestartDiagnostic {
|
||||
identity,
|
||||
section_line: Some(peer.section_line),
|
||||
reason,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let projection = SavedPeerProjection::from_document(document);
|
||||
if (projection.document_unverifiable
|
||||
|| !projection.unverifiable_peer_indices.is_empty()
|
||||
|| !projection.duplicate_public_keys.is_empty())
|
||||
&& diagnostics.is_empty()
|
||||
{
|
||||
diagnostics.push(RestartDiagnostic {
|
||||
identity: None,
|
||||
section_line: None,
|
||||
reason: "saved peer set is not uniquely verifiable",
|
||||
});
|
||||
}
|
||||
if !diagnostics.is_empty() {
|
||||
return Err(diagnostics);
|
||||
}
|
||||
|
||||
let mut clients = Vec::with_capacity(document.peers().len());
|
||||
for peer in document.peers() {
|
||||
let Some(identity) = peer.identity else {
|
||||
return Err(vec![RestartDiagnostic {
|
||||
identity: None,
|
||||
section_line: Some(peer.section_line),
|
||||
reason: "peer identity is unavailable",
|
||||
}]);
|
||||
};
|
||||
let Ok(client) = document.project_client(&identity) else {
|
||||
clients.clear();
|
||||
break;
|
||||
};
|
||||
clients.push(client);
|
||||
}
|
||||
let management_snapshot = (clients.len() == document.peers().len())
|
||||
.then(|| Snapshot::new(clients).ok())
|
||||
.flatten();
|
||||
Ok((projection, management_snapshot))
|
||||
}
|
||||
Reference in New Issue
Block a user