Skip to main content

migrate

The Unraid migrator (hoserva migrate): the pre-flight scan of a Flash Backup zip and its written go / no-go report. Nothing here writes to an Unraid disk or to the zip.

📄️The migration session and its report

The one migration session: its phase and, once a scan has finished, the report as rows. `phase` is `none` before any scan, `scanning` while a `migration_scan` job is queued or running, `scanned` once a report exists and `scan_failed` when the latest scan did not finish, including when its job was cancelled or dropped before it ran (its `scanError` says why; the report of an earlier scan, if there was one, is still returned). The rows name and count; they never quote a file's content. `getMigrationReport` returns the same report as a document. `flashDevices` lists the disks a scan can read as the Unraid USB stick (`startMigrationDeviceScan`): a FAT filesystem labelled `UNRAID` on a disk that is neither the boot disk nor in the array. It is empty, and `zipOnly` is true, once the session's report was made from a capture that says Unraid booted from an internal device: the Flash Backup zip is then the only source. It is empty, with `zipOnly` false, when no such disk is attached or this daemon cannot read one. The report's `review` holds what the Review step shows as data: the disk mapping table, the share preview, the boot mode and the capture's state. It comes from the same scan as the rows, which stay as they are, and is absent from a report made before it existed.

📄️Delete the migration session

Deletes the session, its report and the uploaded Flash Backup zip, which holds secrets (password hashes, SSH host keys, WireGuard and rclone config, the licence key, containers' environment). A scan of the Unraid USB stick keeps nothing of it: the stick is never written and nothing is copied from it. Succeeds when there is nothing to delete. Refused with 409 `scan_in_progress` while a scan runs, and with 409 `migration_in_progress` while an import is pending its point of no return (`startMigrationImport`): the scan's baseline is what the adopted disks are verified against. `startMigrationImport` with `undo` takes a pending import back, after which this succeeds.

📄️Scan a Flash Backup zip

Takes the Flash Backup zip and queues a `migration_scan` job (topology class, so no storage job runs beside it). The zip is kept in the daemon's state directory, readable by root only, as the session's source; it is never modified and never extracted: entries are read in memory. A scan replaces the previous session's report and zip once it finishes. Refused before anything is queued, with nothing kept: 400 `file_required` (no `file`), 400 `invalid_zip` (not a zip, an entry path with `..` or starting with `/`, or a duplicate entry), 413 `zip_too_large` (a zip over 2 GiB, refused as soon as the request body, which is the zip and its multipart framing, passes that size plus 1 MiB), 400 `invalid_flash_backup` (no usable `config/disk.cfg`) and 400 `unsupported_layout` (an Unraid version other than 6.12.x or 7.x, or a flash layout Hoserva does not recognise, unless `unverifiedLayout` is true). 409 `scan_in_progress` while a scan runs, 409 `migration_in_progress` while an import is pending its point of no return (`startMigrationImport`), and 501 `not_configured` when this daemon has no migration service. `unverifiedLayout` overrides only the layout refusal; the override is recorded in the report and printed at its top. The scan reads the disks Hoserva already inventories and SMART without waking a disk in standby. It also reads every data disk the capture records, through a read-only mount at a private mountpoint under the daemon's state directory (XFS without replaying its log), after the disk's read-only filesystem check: a disk that fails a check, or is a ZFS, encrypted or multi-device btrfs disk, is refused by name in the report and the scan goes on with the rest. Each data disk's files are listed and a sample hashed, as the baseline the verify phase compares against (`fullChecksums` hashes every file). Nothing is written to a source disk, and no disk stays mounted when the job ends, whether it succeeded, failed or was cancelled. The job reports its progress and can be cancelled.

📄️Scan the Unraid USB stick

Instead of the Flash Backup zip, reads Unraid's configuration from the USB stick, attached to this machine, and queues a `migration_scan` job. `device` must be one of the `flashDevices` `getMigration` offers. The stick is mounted read-only (never read-write) at a private mountpoint under the daemon's state directory for the one read made here before anything is queued and for the job's own read, and is unmounted after each; nothing is ever written to it and nothing is copied from it. The stick is the user's rollback. Refused before anything is queued: 400 `invalid_flash_device` (the device is not on offer: not a FAT filesystem labelled `UNRAID`, the boot disk, an array disk, or a filesystem UUID that another disk shares), 409 `zip_only_source` (the session's capture, or the stick's own, says Unraid booted from an internal device, whose ZFS boot pool Hoserva does not read; the zip is the only source), 409 `flash_device_unreadable` (it could not be mounted read-only or unmounted, or failed while it was read), 400 `invalid_flash_backup` (no usable `config/disk.cfg`) and 400 `unsupported_layout` (as for the zip, unless `unverifiedLayout` is true). 409 `scan_in_progress` while a scan runs, and 501 `not_configured` when this daemon has no migration service or cannot read a flash device. The result is the report the same flash's zip gives.

📄️Adopt the Unraid data disks

Phase C of the migration: queues a `migration_import` job (topology class) that adopts the Unraid data disks into the pool at `/mnt/user` without formatting them and without writing a byte to them. Each data disk is mounted by its own device, never by a filesystem UUID another disk may share, with `ro,norecovery` (XFS), `ro,noload` (ext4) or `ro,rescue=nologreplay` (btrfs), and the catch-all pool over them is read-only, with the share directory structure intact. The former parity and cache disks are recorded by identity and are neither formatted, mounted nor opened: formatting them is the point of no return. No `snapraid.conf` is generated, so no parity engine exists and no sync can run. While the import is pending (`getMigration` `phase` is `imported`), parity, array-write and topology jobs other than this one's own retry are refused with 409 `migration_in_progress`, a scan included, and `forgetMigration` is refused with the same code. `roles` is the disk-role mapping the user confirmed against the serial table, one entry per disk, keyed by `serial` or `wwn`; the `review` of `getMigration` proposes a role for each disk the capture names. A cache on a spare partition of the boot disk (`listDisks` `cachePartitions`) is keyed by that partition's `byId` and `partUuid` instead, with no serial or WWN. The mapping is refused with 400 `invalid_import_roles`, before anything is queued, unless every role names a disk the scan listed and this machine still has; a disk the scan refused, an Unraid boot device or the Unraid USB stick has no role but `ignore`; a disk `disks.ini` records as parity is never `data`, whatever filesystem it reports, and a disk it records as data is never `parity` or `cache`; a weak-identity disk is never parity; the disk this machine boots from is never parity or data and is the cache only by a spare partition of it; parity is one or two disks, each at least as large as the largest data disk; and every data disk has a filesystem Hoserva adopts whose UUID is not another data disk's; and the layout must be one `snapraid.conf` can be rendered for at the point of no return, which is the same check that step makes: the content-file copies (the parity disks + 2, on distinct devices) need the boot device, a cache that is a device of its own (a cache that is a partition of the boot disk is the boot device's copy) and enough data disks. A layout that falls short is refused with 400 `invalid_import_roles`, naming the shortfall and what to add (a data disk or a cache device), never left to fail at step 17. The Unraid USB stick in any role is refused with 409 `unraid_stick`. Also refused before queueing: 409 `confirmation_required` unless `confirm` is true, 404 `no_migration_report` before a scan, 409 `scan_not_finished` while a scan runs or the latest one failed, 409 `migration_no_go` when the report's verdict is no-go, 409 `scan_outdated` for a report made before the disk table existed, 409 `array_exists` when the array is not a pending import's, and 501 `not_configured`. The job reads every data disk's identity again and re-runs its read-only filesystem check immediately before mounting; a disk that changed since the request, or now fails, is refused and nothing is mounted. A failure after the array is recorded unmounts what was mounted, deletes the record and leaves no pool; a retry of the same mapping applies a recorded import again. Nothing is read from the Unraid flash: the import uses the report the scan stored.

📄️Verify the adopted disks against the scan's baseline

Step 16 of the migration, the last checkpoint before parity is touched: queues a `migration_verify` job (topology class, read-only, and admitted while the import is pending) that walks every adopted data disk through its read-only mount and every share through the read-only pool at `/mnt/user`, and compares what it finds with the scan's baseline. For each disk and each share it compares the file, symlink and special-file counts, the total bytes, every file's size, every symlink's target and every special file's type, and it hashes again exactly the files the baseline hashed (`fullChecksums` of the scan decides how many that is). The expected figures of a share are the union of the disks' baselines: a path two disks hold is shown once by the pool, from the first disk, and is listed in `duplicates`, never as missing or extra. Any difference, and any file or directory that cannot be read, fails the job and leaves `getMigration` in `verify_failed` with the result in `verify`; a verify can be run again, and a run clears the earlier result when it starts. A disk or the pool that the kernel's mount table does not show read-only is not read. Nothing is written to a source disk. The result is in `getMigration`; the job reports its progress and can be cancelled, which leaves a failed result. Refused before anything is queued: 409 `no_import_pending` unless an import is pending its point of no return (`startMigrationImport`), 409 `no_migration_baseline` when the scan recorded no baseline, and 501 `not_configured`.

📄️Cross the point of no return and initialise parity

Step 17 of the migration, the point of no return and the first step that writes to a disk of the old array: queues a `migration_parity` job (topology class) that formats the former Unraid parity disk(s) XFS and the cache (a whole disk, a spare partition of the boot disk, or partition 4 of an Unraid boot + data device and never the rest of that disk), records them as the array's own, mounts the data disks read-write, generates `snapraid.conf` with its content files placed on separate devices, applies what the import deferred (each share's cache mode and its top-level directory's setgid mode and group), wires the parity engine so sync, scrub and fix are available without a restart, and queues the initial `sync` as an ordinary sync job, which runs through the threshold guard like every other. The data disks are never formatted. Until the sync completes the array has no redundancy at all: `getMigration` `parityInit` states that window and what rollback means for this session's boot mode and layout, and the user is shown them before this is called.

📄️The template conversion preview

What the scan's conversion of the Flash Backup's Docker templates found: every template with its class and how it converted, every Compose Manager project, and the counts the report shows. The scan converts each template in memory with the converter `convertUnraidTemplate` runs, passing it the Docker networks of the Phase A capture, so a custom network's `docker network create` command is exact when the capture holds that network. Nothing is created, written under `/var/lib/hoserva/stacks/` or started. The session keeps only each template's outcome (its status and warning classes), never its content, so this answers without the zip; `getMigrationTemplate` builds a preview on request. `counts` cover the templates that had a container on the source server (autostart, running and stopped) and, without the capture's container list, every template (`allTemplates`); a template-only template is converted and previewable but is in `templateOnly` and not in the clean or warning counts. A Compose Manager project is previewed with its own `compose.yaml`, counted in `composeProjects` and never converted. 404 `no_migration_report` before a scan has finished, and 404 `no_template_preview` for a report made before scans converted templates (scan again).

📄️One template's conversion preview

The preview of one template or Compose Manager project from the latest scan, built when it is asked for by converting the template again from the Flash Backup zip the session keeps in the daemon's state directory, with the Docker networks of the Phase A capture; the session itself keeps no template content. It holds the source as the flash holds it, the generated Compose (the project's own `compose.yaml` for a project), every warning, including the writable-layer warning every converted template carries and any host path into another `/mnt/<pool>`, and the privileges the Compose content asks for. The source holds the template's environment, secrets included, which is why this is an admin operation, and the report's rows never quote it. In the generated Compose a variable the template masks (`Mask='true'`) with a value is a `${NAME}` reference, and its value goes to the `.env` of the stack `createMigrationStacks` creates, not to the Compose; this API never returns that `.env`, but the value is still in `source` and can be quoted by a warning about that variable. The value stays in the Compose, in clear, only when the variable's name cannot be a Compose reference (it is not a plain identifier of letters, digits and `_`) or is one a stack's `.env` may not define because Docker reserves it, which is reported as a `note` warning naming the variable. Every other variable's value, and a masked variable's empty value, is in the Compose as written. A Compose Manager project's `compose.yaml` is shown as the project holds it, whatever it contains. A template the converter could not read has `status` `failed` and an `error`, and no Compose. 404 `template_not_found` for a name the report does not list or a Compose Manager project whose `compose.yaml` is not in the source, `no_migration_report` before a scan has finished and `no_template_preview` for a report made before scans converted templates. 409 `template_source_unavailable` when the zip is not kept: it was removed, or the report was made from the Unraid USB stick, which nothing is copied from. A preview is never answered from a copy kept after the zip is gone.

📄️What the migration's Phase D can create, and what it has created

Phase D's container steps as data, from the latest scan. `templates` are every template of the scan grouped by `class` in the order the groups are shown (`autostart` in Unraid's own autostart order, then `running`, `stopped`, `template_only` and `unknown`); only a creatable template on Unraid's autostart list whose stack does not exist yet is `preselected`, so with no capture (class `unknown`) nothing is. Every one can still be selected. `composeProjects` are the Compose Manager projects, offered with their own `compose.yaml`; `byHand` are the containers created with `docker run` and their image, which nothing is generated for: they are recreated by hand. The preview of each (the generated Compose, every warning including the writable-layer warning) is `getMigrationTemplate`, and should be read before `createMigrationStacks`. `stacks` are the stacks `createMigrationStacks` created, in the order they are offered for starting (Unraid's autostart order first), each with its `state`; `awaiting` names the started stack that must be confirmed or stopped before another is started, and `next` the first stack not started yet; a stack that was started, stopped and never confirmed is offered as `next` only when none of those is left. `parityInitialized` is false until the migration is past its point of no return (`initializeMigrationParity`); the operations that create or start something refuse until then. This reads no disk. 404 `no_migration_report` before a scan has finished and 404 `no_template_preview` for a report made before scans converted templates.

📄️Create stopped Compose stacks from the scan's templates and projects

Step 19 of the migration: creates a Compose stack, stopped, through the stack layer (`createStack`) for each selected template or Compose Manager project of the scan, from the generated Compose the preview showed (`getMigrationTemplate`), or the project's own `compose.yaml`. Nothing is started and nothing is written outside the stacks directory. A stack is named after the template's `<Name>` (or the project's name), lowercased with every character a stack name may not hold turned into `-`.

📄️Start one migrated stack

Step 20 of the migration: queues the `stack_start` job (`startStack`) for a stack `createMigrationStacks` created, and records that it was started: the start is recorded before the job is queued, so a start that cannot be recorded queues nothing, and a start that is queued and whose job is then not recorded leaves the stack recorded as started with no job. Containers are started one at a time: this is refused with 409 `container_unconfirmed` while another migrated stack that was started is neither confirmed (`confirmMigrationContainer`) nor stopped, which includes one whose start job is still running or whose containers are running, and one recorded as started with no job. When the state of another started stack cannot be read (its start job or its containers), nothing is queued and the read failure is returned as it is, not as `container_unconfirmed`: 503 `docker_unavailable` when the Docker Engine cannot be reached, 500 otherwise. A failure to record the start is a 500. Refused before anything is queued with 409 `parity_not_initialized` until the migration is past its point of no return, 404 `migrated_stack_not_found` for a name the migration did not create, 409 `container_confirmed` for a stack already confirmed, 404 `stack_not_found` when the stack was removed since, and 409 `array_stopped` or 503 `array_state_unknown` as `startStack` answers. The stack's `waitSeconds` is what Unraid's autostart list waited after starting it, as a suggestion. Run `checkMigrationContainer` once the job has succeeded.

📄️Check that a started stack's containers see their data

The data check of step 20: for each bind mount of the stack's containers whose host path is under `/mnt/user` or `/mnt/cache`, whether the path exists and is not empty. A directory counts as empty when it holds no entry, a file when it has no byte; a path that cannot be read is `unreadable`, never `ok`. It reads one directory entry of each path (which can spin a disk up), when asked and never on a timer, and records that a check ran since the latest start, which `confirmMigrationContainer` needs. A stack with no bind mount under those paths has nothing to check and is `allOk`. Refused with 409 `container_not_started` for a stack that was not started, 409 `no_container` while the stack has no container (its start has not created one), 409 `parity_not_initialized` and 404 `migrated_stack_not_found` as `startMigrationContainer` does.

📄️Confirm a started stack sees its data

Records that the user read the data check and the started stack sees its data, which offers the next stack. Refused with 409 `container_not_started` for a stack that was not started, 409 `data_check_required` until `checkMigrationContainer` has run since the latest start, 409 `data_check_failed` when that check found a path that is missing, empty or unreadable unless `acceptFailedCheck` is true (a container whose data directory is meant to be empty), and 409 `container_not_running` when none of the stack's containers is running (stop the container instead, to go on without it). Confirming a confirmed stack changes nothing. 409 `parity_not_initialized` and 404 `migrated_stack_not_found` as `startMigrationContainer` does.

📄️The post-migration checklist

Phase D's closing steps as a checklist, each item's state taken from a record where one exists, so it cannot say something was done when it was not. The migration counts as finished exactly when the array record carries the time the point of no return finished (`initializeMigrationParity`), which is written in the same statement that ends it and by nothing else: `finished` is true and `finishedAt` is that time. A migration that is pending, part-way through its point of no return, or was undone has none, and so has an array created by hand or one that finished before this record existed; then `finished` is false, `finishedAt` is absent and `items` is empty: the checklist does not apply. `getMigration` keeps no 'complete' phase.

📄️Acknowledge a checklist item no record can show

Records that the user did `user_scripts` (worked through the scan's inventory) or `restore_drill` (deleted a file and recovered it with `hoserva fix --path`), with who made the acknowledgement and when; the restore drill's also records the latest succeeded `fix` job that had a `path`, when there is one. It is kept in the database and outlives `forgetMigration`. Refused with 409 `migration_not_finished` while the checklist does not apply (`getMigrationChecklist` answers `finished` false), and with 409 `checklist_item_has_record` for every other item, whose state comes from a record and is never set by hand. Acknowledging an acknowledged item changes nothing: the first acknowledgement is kept. Returns the item as it stands.