# Verification and restart recovery

Use this runbook to establish that the expected destination contents match the
configured source, or to choose a supported recovery path after an interruption.
It describes the current implementation; it does not certify a release or make
an unsupported recovery operation safe.

See the [current verification and recovery validation report](https://github.com/iqbalmd/cdc-pipeline-engine/blob/main/docs/support/verification-recovery-20260925.md)
for tested scenarios, results and remaining qualification limits.

The commands below use a native package installed at `/usr/bin/pgpipe`, its
`pgpipe` service account, and `/etc/pgpipe/pgpipe.yaml`. For a container or raw
binary, use that deployment's launcher and the non-root identity that owns its
configuration and state. Do not run a second pipeline alongside the existing
owner. Supply the same configured environment references and protected files
used by the service; do not put passwords or API tokens in command arguments.

## 1. Record the situation before changing it

1. Record the pgpipe version, pipeline name, configuration revision, source and
   destination identities, state backend, and affected tables. Retain the exact
   error, operation/job ID, saved checkpoint, source slot/WAL evidence, and the
   relevant logs. Keep credential files and reports containing primary-key
   identities in access-controlled evidence storage.
2. Check whether the process is streaming, paused, copying, reconnecting, or
   stopped behind a safety hold. Inspect table-level state and active jobs too.
   A healthy HTTP endpoint, **Streaming**, a recent heartbeat, or a zero-looking
   lag sample does not prove that all expected rows match. A stale or failed WAL
   sample is unknown, not zero backlog.
3. Choose the matching recovery branch below. Preserve state and protected
   evidence while investigating; a restart is not permission to skip a failed
   transaction, replace the source identity, or repeat destructive maintenance.

Useful native-package diagnostics:

```sh
/usr/bin/pgpipe --version
sudo systemctl status pgpipe --no-pager
sudo journalctl -u pgpipe -n 200 --no-pager
sudo -u pgpipe -- /usr/bin/pgpipe status -c /etc/pgpipe/pgpipe.yaml --json
sudo -u pgpipe -- /usr/bin/pgpipe validate -c /etc/pgpipe/pgpipe.yaml
```

Status may be incomplete when the API is unavailable or a local state file is
locked. Follow its reported evidence source and errors. Authenticated control
commands use `PGPIPE_API_TOKEN`; follow the deployment's existing protected
credential procedure. `validate` checks configuration without starting the
pipeline. It does not establish database readiness or data equality. Database
startup preflight runs when pgpipe starts; there is no separate `pgpipe preflight`
command.

## 2. Establish a stable comparison window

Ordinary verification reads separate source and destination repeatable-read
snapshots. They do not share one replication boundary. Active writes or ordinary
replication lag can therefore produce a temporary mismatch.

1. List every expected customer table and its mapped destination. Record the
   active table selection, exclusions, schema mapping, and transform plan. A
   report covering a subset is evidence only for that subset.
2. Stop application writes and schema changes affecting the selected source
   tables. Let already-open write transactions finish. Stop independent
   application, administrator, ETL, and maintenance writes/DDL affecting the
   mapped destination tables.
3. Keep pgpipe running and streaming until the selected data has caught up.
   Review current checkpoint/lag evidence, destination-heartbeat delivery when
   enabled, table state, and outstanding errors together. Resolve active
   backfills, repairs, reconnects, and safety holds before declaring this an
   acceptance window.
4. Run verification while that window remains stable. Preserve the report and
   the evidence of the write freeze. If writes resume or the comparison fails,
   establish a fresh window before accepting a later result.

**Pause replication does not create this window.** It stops pgpipe intake, while
source applications can still write. Runtime verification is not admitted while
pgpipe is paused. Do not pause the pipeline in place of quieting source
applications and allowing replication to catch up. See
[Whole-pipeline pause and resume](https://github.com/iqbalmd/cdc-pipeline-engine/blob/main/docs/pause-resume.md).

### Scope and unsupported comparisons

- The verifier compares the configured result of source transforms, filters,
  table/column routing, and destination schema mapping. A filtered or masked
  pipeline should match that intended result, not the untransformed source.
  Do not temporarily remove transforms to manufacture a match.
- Every compared table needs a source primary key. A table without one is
  `skipped`, even when CDC supports its replication identity. Transforms that
  change primary-key values are not supported by content verification and
  produce an error. Neither outcome is a successful comparison.
- Dashboard/API explicit selections must be configured exact `schema.table`
  names, not wildcard requests or arbitrary unconfigured tables. The textual
  selector cannot represent an identifier containing a dot. The CLI has no
  `--table` option and verifies its configured selection. Leave the dashboard
  selection empty to use the configured selection, then check the resolved
  table list in the report.
- The enabled managed source heartbeat is excluded from customer-content
  verification. Review every reported exclusion.
  Use destination-heartbeat freshness/alerts when enabled, or separate liveness
  monitoring. Do not repair a heartbeat timestamp to make delivery look healthy.

If the required scope cannot be compared with the supported verifier, record
that limitation and obtain a separate reviewed comparison. Do not label the
unverified scope as matched.

## 3. Run verification and inspect the complete result

### Dashboard and API

1. Sign in and open **Verify & Repair**.
2. Enter the intended exact tables, or leave **Tables** empty for the configured
   selection. Leave **Create diagnostic repair previews for mismatches**
   unchecked unless diagnostic candidates are required.
3. Select **Run verification**. Wait for completion, then open the run's
   **Details** in **Verification history**. Browser reload does not turn an
   accepted request into a completed result.
4. Apply the acceptance checks below to the report, not just its history badge.

The equivalent authenticated API request is:

```http
POST /api/verification/runs
Content-Type: application/json

{"tables":["public.orders"],"create_repair_jobs":false}
```

Use the deployment's verified HTTPS trust and authenticated request mechanism;
mutating browser-origin requests must satisfy the configured origin protection.
A `202` response means accepted. Read `GET /api/verification/status`, retain the
returned run ID, and fetch `GET /api/verification/runs/{id}` for its report.
`GET /api/verification/runs` lists history. `POST /api/verification/cancel`
requests cancellation; a canceled run is not acceptance evidence. A running
verification returns a conflict for another overlapping run; unavailable
pipeline state returns an unavailable response. Resolve the reported condition
instead of repeatedly submitting requests.

### Read-only CLI

Run as the service identity in the stable window:

```sh
sudo -u pgpipe -- /usr/bin/pgpipe verify \
  -c /etc/pgpipe/pgpipe.yaml --json --timeout 30s
```

Capture stdout and the command's exit status in the incident record. The report
contains bounded primary-key identities and checksums; it does not contain
arbitrary non-key row values. `--timeout` limits each verification query, not the
whole run. Runtime dashboard verification separately uses
`verification.max_duration` as a total time limit.

The command is read-only against source and destination. Its optional
`--persist-history` and `--create-repair-jobs` flags write pgpipe state and require
exclusive lifecycle ownership. **Stop the running pgpipe process before using
either flag**, while keeping the comparison window stable. Prefer the running
dashboard workflow when history is needed without stopping the service.
Creating repair candidates does not approve or execute a repair.

### Acceptance checks

Accept customer-content verification only when all of these are true:

1. The run completed for the reviewed source, destination, configuration, and
   stable comparison window. Its table mappings and transform plan match the
   intended replication contract.
2. Every expected customer table appears exactly once in the reviewed report
   scope, and every one has `status: matched`. Check the names; matching counts
   alone cannot prove that the correct tables were checked.
3. `summary.matched == summary.total`, and `summary.mismatched`,
   `summary.errored`, and `summary.skipped` are all zero. The expected customer
   scope must not be empty. Each excluded operational table has an understood
   reason and separate liveness coverage where needed.
4. No unresolved copy, apply, DLQ, identity, or recovery error contradicts the
   claim being made. Retain the report/run ID and checkpoint/window evidence
   before releasing the write freeze.

**Exit zero or an aggregate `matched` badge is insufficient.** The CLI exits
non-zero for mismatches or table errors by default, but skipped tables do not
make it fail. A report with both matched and skipped tables can have an
aggregate `matched` result. `--exit-code=false` disables even the normal
mismatch/error exit behavior; do not use it as an acceptance gate.

| Result | Action |
| --- | --- |
| `matched` for every expected table | Apply all acceptance checks, retain evidence, then release the reviewed write freeze. |
| `mismatch` | Inspect missing, extra, and changed identities. If writes or lag affected the window, establish a fresh stable window and rerun. A persistent mismatch requires a reviewed correction or repair; it does not authorize automatic overwrites. |
| `skipped` | The table was not compared, commonly because it has no primary key. Record unsupported coverage and arrange a reviewed alternative. |
| `error`, `timed_out`, `canceled`, or `history_error` | No complete accepted run. Resolve the reported cause and rerun; preserve any available evidence. |
| Repair `completed / inconclusive` | The repair finished its writes, but its final comparison did not establish a match. Keep evidence and the write freeze; use **Rerun verification** after establishing a stable window. This does not rerun the repair. |

Ordinary verification has no table status named `inconclusive` or `unsupported`.
Those words describe the operational conclusion, or the separate repair
workflow. Truncated mismatch detail is not a clean result: report-detail limits
bound retained keys, while exact mismatch totals continue counting. A second
match without a stable window adds diagnostic confidence but is not the same
consistency proof.

Repair verification can be rerun with
`POST /api/v2/repair/{id}/rerun-verification`. Follow the separate
[Repair Approval Operator Runbook](https://github.com/iqbalmd/cdc-pipeline-engine/blob/main/REPAIR_OPERATOR_RUNBOOK.md) before any repair
approval. Content equality alone does not authorize clearing a lifecycle fence
or an interrupted operation's evidence.

## 4. Ordinary process interruption or repeatable apply failure

Use this branch only when no incomplete maintenance operation, unavailable-WAL
condition, unknown-COMMIT outcome, or identity/fencing hold requires a branch
below.

1. Preserve the saved checkpoint, slot/WAL, state, configuration, and logs.
   Correct the actual connectivity, permissions, storage, schema, or destination
   constraint failure. Repeated restarts do not correct a deterministic error.
2. Retain the normal checkpoint-resume configuration. With a saved checkpoint,
   `snapshot.mode: initial` resumes it; `never` also resumes a saved checkpoint.
   Do not delete state or switch to a destructive mode merely to restart.
3. Validate configuration and restart through the deployment launcher:

   ```sh
   sudo -u pgpipe -- /usr/bin/pgpipe validate -c /etc/pgpipe/pgpipe.yaml
   sudo systemctl restart pgpipe
   sudo -u pgpipe -- /usr/bin/pgpipe status -c /etc/pgpipe/pgpipe.yaml --json
   ```

4. Review startup logs for the expected checkpoint and successful readiness
   checks. Confirm the relevant table states, error queues, and liveness. If
   startup reports a safety hold, stop the restart loop and follow that hold's
   recovery path. Use stable-window verification when data correctness needs to
   be established before applications resume.

A failed committed source transaction remains indivisible and uncheckpointed;
fix its cause so it can replay. `admin skip-commit` is a separate, audited
whole-transaction exception that intentionally omits those changes. It is not
ordinary recovery. The quarantine and `delete_insert` modes described as future
work in [Transaction Failure Recovery](https://github.com/iqbalmd/cdc-pipeline-engine/blob/main/TRANSACTION_FAILURE_RECOVERY.md) are not
runtime recovery options.

A stored Pause survives process restart. Wait for the dashboard's actual
control state; a successful process restart does not mean intake resumed. Use
the normal Resume workflow only after its WAL and safety checks succeed.
Pause/Resume cannot clear a stopped pipeline's failure or ownership hold.

## 5. Initial snapshot or deliberate full rebuild interrupted

An initial snapshot can commit some destination batches before failing. pgpipe
does not save its completion checkpoint after a failed copy or sequence
resynchronization. Partial rows are not a valid streaming baseline.

1. Stop the process if it is still running. Preserve the partial destination,
   logs, source slot/WAL, state, and configuration as incident evidence before
   any destructive correction.
2. Fix the copy error. Confirm that there is no interrupted live Add, repair,
   rebuild, or unknown-COMMIT hold requiring another section. Do not treat the
   absence of an initial completion checkpoint as permission to discard other
   saved evidence.
3. Choose an explicit scope with the incident owner. Protected first enrollment
   in `initial` mode requires empty mapped customer destination tables. If
   partial rows remain, use a DBA-reviewed emptying procedure before retrying
   `initial`, or approve a full selected-table rebuild as described below.
   There is no generic safe `TRUNCATE CASCADE` recipe for an unknown dependency
   graph.
4. For an approved full rebuild, while stopped configure:

   ```yaml
   snapshot:
     mode: always
     reset_destination: truncate
   ```

   This replaces **all selected mapped destination tables**, not just the table
   named in the error. Confirm source authority, compatible backups, the table
   scope and dependency constraints, and stop independent destination writers.
   pgpipe obtains its source snapshot before resetting destination data and
   does not use `CASCADE`. Verification exclusions do not narrow this destructive
   scope: review the full selected, mapped destination set, not just the
   customer tables in a verification report.
5. Validate and start through the normal launcher. Keep `always + truncate`
   unchanged until the copy completes. If interrupted again, investigate before
   restarting; the same configuration deliberately repeats the full copy.
6. After **Phase: Streaming**, with no **Backfilling** or active table operation,
   open **Configuration → Snapshot → Return to normal startup**. The server
   checks that the fresh snapshot checkpoint was saved before changing the
   configuration to `initial + error`. Complete stable-window verification and
   review outstanding errors before restoring application writes.

Do not change to `initial` during a deliberate rebuild, because an older saved
checkpoint may still exist. Do not use `never` to escape a partial initial
copy: without a checkpoint it starts from the slot's current position and
assumes a correctly preloaded destination. `snapshot_only` copies and exits
without a resumable checkpoint; it is not a substitute for snapshot plus CDC.

See [Snapshot and source-table recovery configuration](https://github.com/iqbalmd/cdc-pipeline-engine/blob/main/CONFIGURATION.md) for the
full scope and physical-identity requirements.

## 6. Ordinary live table Add/backfill interrupted

This is different from both initial snapshot and **Rebuild and re-add**.

1. Retain the batch/job ID, exact table selection and physical source identity,
   durable per-table phase, cutover/buffered LSNs, copied rows/chunks, local
   journals, publication/configuration state, and destination contents.
2. Review `GET /api/table-add-batches/{id}` or the dashboard's saved batch status
   where available. After restart, unfinished batch history is marked
   interrupted. Pending/copy/replay/failed live-backfill state deliberately
   blocks normal startup before replication begins.
3. Escalate to the incident owner and database administrator for a reviewed
   recovery plan. Automatic live-backfill resume and a generic ordinary-Add
   abandonment command are not implemented. The error's instruction to resolve
   saved state is not authorization to edit it.

Do not delete table state or journals, mark a table streaming, submit ordinary
Add again to clear the hold, or use the single-table rebuild-abandonment command
for an unrelated Add job. Changing to `always + truncate` does not bypass this
earlier startup gate. Preserve the original evidence while planning a compatible
restore or a separately established clean lifecycle; this runbook intentionally
provides no invented force/resume command.

## Table removal when the browser disconnects

Once pgpipe admits a table removal, reloading or closing the browser does not
cancel it. The running streaming session owns the operation through its saved
removal intent, publication change, identity guard, table list and configuration
update. Requests canceled before admission do not begin a removal. In a request
listing several tables, this applies to the admitted table; cancellation still
prevents later tables from being admitted.

If the response is lost, refresh the Tables page before trying again. A lost
response does not prove either failure or success. Check the table list and
removal history/log message. Successful removal preserves destination rows;
ordinary Add still requires the reviewed rebuild path for a removed table.

The operation has a one-minute context budget for database work and remains
bound to its original streaming session. This is not a promise that local
filesystem operations or locks always finish within one minute. Service
shutdown, a session failure, or a genuinely uncertain database/configuration
result can still interrupt it. An incomplete saved removal intent deliberately
blocks unsafe continuation or silent republication at restart.

For that safety block, preserve the configuration, checkpoint, source-table
bindings, publication membership and logs. Have the incident owner reconcile
the exact saved physical relation and selected-table set with the removal
intent before restarting. Do not delete the binding/state, reset the slot,
restore an active binding speculatively, or clear destination rows to bypass
the block. Browser-disconnect handling does not introduce automatic recovery
from a crashed or outcome-uncertain removal.

## 7. Interrupted Rebuild and re-add

### Single-table job

A preview that expired across restart needs a fresh preview. A provable
pre-reset failure may be classified as safe to retry. A successful operation
can be reconciled only from matching durable completion evidence. A possible
reset or incomplete/contradictory evidence remains `recovery_required`; a reset
is never blindly repeated.

To retire a failed, recovery-required **single-table** rebuild:

1. Stop pgpipe, keep destination writers stopped, and preserve the job, state,
   configuration, source/WAL identity and destination ledger evidence.
2. Review the exact saved job ID and the decision to leave its destination rows
   untrusted. Replace the example ID and reason consistently:

   ```sh
   sudo -u pgpipe -- /usr/bin/pgpipe admin abandon-table-rebuild \
     -c /etc/pgpipe/pgpipe.yaml \
     --job-id rebuild-123 \
     --reason "COPY failed after destination reset" \
     --confirm "ABANDON REBUILD rebuild-123" --yes
   ```

3. Require successful abandonment. The command acquires exclusive ownership,
   removes only the exact saved source relation from publication/configuration,
   retains its removal tombstone, and abandons its rebuild ledger. It leaves
   destination rows untouched and explicitly untrusted and does not advance the
   checkpoint. If interrupted, finish the same decision with the same operator
   and reason rather than inventing a new scope.
4. Start pgpipe without that table. Later use **Add Tables → Rebuild and re-add**
   for a fresh preview, explicit replacement approval, source-authoritative
   copy, verification, replay and activation. Only successful completion makes
   that table trusted and streaming again.

Retiring the last selected user table requires the managed source heartbeat to
remain enabled, or another selected user table. Respect a refusal rather than
manually emptying the selection. This command refuses related-table group jobs.

### Related-table group

Use **Review group recovery** in the authenticated dashboard when recovery mode
is available. It waits for the previous pipeline workers to stop and acquires
fresh exclusive ownership; ordinary dashboard mutations remain disabled.

1. Review the entire saved group, including members that were streaming before
   the rebuild. Enter the current exact confirmation and non-sensitive reason
   to **Retire the interrupted group**.
2. Require the durable retirement result. All reviewed members are removed
   together; existing destination rows remain untrusted. The checkpoint does
   not advance. If acknowledgement or cleanup is uncertain, review the same
   saved decision as the original operator and finish it.
3. After recovery completes and the process restarts, unrelated tables resume
   from the saved checkpoint. Select **Review fresh rebuild**, review the
   current dependency scope and row replacement, and approve a new job.
4. Keep independent destination writers stopped. The group workflow holds source
   writes to the reviewed tables through copying and verification; unrelated
   source tables may remain writable while replication commits queue. Require
   every group member's verified completion and activation before trusting it.

The API exposes `GET /api/table-rebuilds`, `GET /api/table-rebuilds/{id}`,
`POST /api/table-rebuilds/{id}/recovery/preview`, and
`POST /api/table-rebuilds/{id}/recovery/approve`. Use the current preview's proof
and exact approval fields; do not replay an expired rebuild approval or retire
individual group members. See [Reviewed related-table rebuild](https://github.com/iqbalmd/cdc-pipeline-engine/blob/main/RELATED_TABLE_REBUILD.md)
for supported group shapes, liveness requirements, and interrupted-decision
handling.

## 8. Unavailable WAL or unknown destination COMMIT

These are evidence-dependent recovery cases. Do not substitute a successful
status request or row-count check for proof that replay is safe.

### Missing slot or unavailable retained WAL

1. Preserve the checkpoint, slot name, current slot/WAL evidence, identities and
   configuration. A missing slot, `wal_status=lost`, or `restart_lsn` after the
   saved checkpoint means normal replay cannot be assumed safe.
2. Restore the original, proven-compatible source continuity from the recovery
   system, or obtain explicit approval to rebuild the complete selected
   destination scope. Merely creating the same slot name at current WAL does
   not restore the missing history.
3. For the same legitimate state and an unpaused pipeline, an explicitly
   approved startup recovery can use:

   ```yaml
   source:
     replication:
       missing_slot_policy: recreate_with_snapshot
   snapshot:
     mode: initial
     reset_destination: truncate
   ```

   Merge these fields into the existing configuration; retain all other source
   settings. This authorizes replacing **all selected destination data** when
   startup detects lost continuity. It is a full rebuild, not resume from the
   missing WAL. `never` is refused. Use the full-rebuild backups, scope review,
   write freeze and acceptance checks. Restore the normal `missing_slot_policy:
   fail` and non-destructive reset setting after successful recovery if future
   automatic full rebuilds are not intended.
4. A restored Pause/accepted Resume retains additional protection: it cannot
   authorize automatic slot recreation or destructive copying when WAL is
   unavailable. Keep the old state and use the reviewed fresh-state rebuild
   procedure below if a destructive recovery is approved. Do not delete the
   pause record or repeatedly restart to bypass it.

### Unknown or ambiguous destination COMMIT

1. Keep the exact source transaction, hold/error identity, and destination/state
   evidence. A connection failure while COMMIT was being acknowledged does not
   prove rollback. Do not retry a row subset, advance/reset the checkpoint,
   insert a skip marker, or assume upsert makes every replay safe.
2. Identify the path from the actual error and configuration. Ledger-backed
   apply commits the destination changes and its apply evidence together. Replay
   checks the same pipeline, commit LSN and XID and can skip already committed
   DML. Durable CDC uses that path; eligible grouped `best_effort` writes also
   use the ledger, while retaining their asynchronous crash-durability limits.
   Do not infer ledger protection for every `best_effort` transaction.
3. Snapshot/COPY and singleton paths without that ledger can leave a durable
   destination-COMMIT hold. The reported hold blocks restart across durability
   and snapshot modes. Preserve it. There is no operator command to clear it in
   place, even after independently reconciling row contents or side effects.
   Choose a reviewed compatible restore or the fresh-state rebuild below with the
   incident owner and DBA.
4. Treat DLQ retry evidence separately. `retry_outcome_unknown` blocks another
   **Retry** for that entry; only after independently reconciling its data and
   effects may explicit **Resolve** acknowledge the entry. A
   `retry_applied_resolution_pending` outcome means the write succeeded but
   persistence of its resolution failed; do not blindly apply it again.

`admin destination-fence inspect` checks authority compatibility, not unknown
COMMIT outcomes or row equality. **NO RECOVERY NEEDED** from that inspection
does not clear a COMMIT hold. Neither Pause/Resume nor a generic DLQ retry clears
a pipeline-wide hold. See [Transaction Failure Recovery](https://github.com/iqbalmd/cdc-pipeline-engine/blob/main/TRANSACTION_FAILURE_RECOVERY.md)
and the [target reconnect validation report](https://github.com/iqbalmd/cdc-pipeline-engine/blob/main/docs/support/target-reconnect-fix-20260925.md).

### Reviewed fresh-state full rebuild

This is the supported destructive route when the existing lifecycle cannot
resume, including a preserved pause with unavailable WAL or a durable COMMIT
hold. It is not permission to discard interrupted repair/Add evidence or to
claim those operations completed.

1. Stop pgpipe and any competing owner. Preserve the original state and complete
   recovery cohort described below. Confirm source authority, all selected
   destination tables that will be replaced, dependency constraints, and the
   independent destination-write freeze with the incident owner and DBA.
2. Configure a **new, empty state location** using the reviewed backend-specific
   procedure. Preserve the previous location intact. Configure
   `snapshot.mode: always` and `snapshot.reset_destination: truncate`. Keep the
   intended source, destination, table selection and non-secret configuration
   fixed through inspection and authorization.
3. Run the read-only inspection as the owning identity, using the real operator
   and approved incident/change reason:

   ```sh
   sudo -u pgpipe -- /usr/bin/pgpipe admin destination-fence inspect \
     -c /etc/pgpipe/pgpipe.yaml \
     --operator "alice@example.com" \
     --reason "approved full source-authoritative rebuild; CHG-5678"
   ```

4. Proceed only when it reports **READY FOR FULL REBUILD**. Review the complete
   warning and evidence, then run only the freshly printed
   `admin destination-fence authorize-rebuild` command with its exact
   `--confirm` value and `--yes`. Do not invent the confirmation or reuse an old
   one. The authorization changes destination safety records; the following
   start deletes/replaces the selected destination contents.
5. Retain the printed request ID. If authorization is interrupted or its
   acknowledgement is uncertain, run `inspect` again and retry only as it
   instructs with the same `--request-id`, operator and reason. Normal startup
   remains blocked while the authorization is incomplete.
6. Follow the command's deployment-specific start instruction. Keep
   `always + truncate` until the first destination-attested snapshot checkpoint
   advances. Once streaming and not backfilling, use **Configuration → Snapshot
   → Return to normal startup**. Complete the stable-window verification and
   acceptance checks before releasing the write freeze.

If inspection reports missing destination identity, damaged ownership markers,
or contradictory audit evidence, stop. Restore the matching original metadata
from the compatible backup. When that is unavailable, the reviewed alternative
is a genuinely new destination database and new empty state, leaving the old
evidence intact; do not invent an identity or copy old safety tables into the new
database. See [Destination-fence inspection and authorization](https://github.com/iqbalmd/cdc-pipeline-engine/blob/main/README.md#pgpipe-admin-destination-fence)
for the exact deployment and retry contracts.

## 9. Interrupted claimed repair

An unclaimed queued repair may still be admitted only while its deadline and
readiness checks hold. A claimed repair is not automatically resumed after
restart. Only positively proven pre-write cleanup can run automatically under
the separate cleanup bounds.

For `interrupted_partial`, `interrupted_ambiguous`, `interrupted_postwrite`,
failed cleanup, or `recovery_required`:

1. Stop pgpipe and keep source/destination application writes frozen for the
   affected selection.
2. Preserve the exact job and audit, state, local repair artifacts, source
   WAL/slot evidence, destination data, and protected fence/chunk ledgers.
3. Follow the [Repair Approval Operator Runbook](https://github.com/iqbalmd/cdc-pipeline-engine/blob/main/REPAIR_OPERATOR_RUNBOOK.md#restart-and-recovery-procedure)
   with the incident owner and DBA. Choose a reviewed clean lifecycle/rebuild
   or a proven compatible coordinated restore.
4. Re-establish configuration, identity/fencing, startup-readiness and content
   verification before allowing application writes or another repair.

Do not retry an old claimed/terminal job, edit its phase, delete a fence or ledger
row, or repoint the same state at another destination. Ordinary table rebuild
recovery and `admin abandon-table-rebuild` do not retire a repair job.

## 10. Keep a compatible recovery cohort

A backup of application destination rows alone is not a pgpipe recovery plan.
Before a destructive rebuild, abandonment decision, or restore, retain enough
evidence to recover a proven compatible cohort:

- Destination application data and protected pgpipe destination schemas,
  identities, ownership fences and apply/repair/rebuild ledgers.
- The matching pgpipe state backend, checkpoint, physical table bindings,
  lifecycle/control records, configuration, protected credential inputs, local
  operation artifacts, and the exact build/version.
- Source system/database/table identities and the slot/WAL history needed by
  the saved checkpoint. Reusing names does not establish physical continuity.
- PostgreSQL roles, ownership, ACLs and required grants. Database-only dumps
  commonly omit cluster-global roles; restore the protected safety-owner and
  runtime-role boundary as well as the tables.

Use a documented backend-consistent capture. For a plain BoltDB or SQLite file
backup, stop pgpipe and confirm it is quiescent; do not copy a live database file
and assume it is consistent. A documented backend-native snapshot is another
option. PostgreSQL state uses the database backup system's consistent method.
Record compatible restore points/timelines and the procedure used to relate
state to destination data, rather than assuming two timestamps prove alignment.

Never restore only state or only destination data to clear a blocker. A restore
into a different database identity is a new lifecycle unless a documented,
supported lineage recovery proves otherwise. Preserve the old evidence. After
restore, validate configuration, inspect identity/fencing through the supported
readiness checks, run `pgpipe repair check` where repair readiness is required,
and let startup preflight enforce its holds. Complete stable-window content
verification before restoring application writes.

The backup/PITR-specific commands and unsupported recovery decisions belong in
the installation's reviewed recovery plan. There is intentionally no generic
`force`, `retry interrupted repair`, `clear state`, or `delete evidence` step.