Windows 10 Startup & Repair Boot Loop

Posted on 2020-12-312020-12-28 by Savagedlight

Earlier this week, I was tuning my PC’s RAM settings, and entered “unstable territory”, which caused cold reboots immediately after login, and again during boot. After turning back the dial to the previously stable configuration, I figured it’d be good to go: But no. Now my Windows install was in an infinite “Startup & Repair” boot loop.

I therefore present to you the list of steps I went through, in case any of the steps are helpful to you. I restarted after each of the attempted fixes below:

Attempted fix 1: Use the go-to “Startup Repair” option. This told me it couldn’t fix the problem on my PC. I also attempted this from a freshly created Windows 10 installation USB stick, to no avail.
Attempted fix 2: Command prompt, “chkdsk c:” – this found no problems with the disk. Also ran “chkdsk c: /offlinescanandfix” with the same result (although much faster).
Attempted fix 3: “sfc /scannow /OFFBOOTDIR=c:\ /OFFWINDIR=c:\Windows” – no errors found. (It initially failed to run because it failed to start service. After repeated attempts, also manually starting the service which failed, I had to exit the command prompt and start it again, and re-issue the command, to successfully run.)
Attempted fix 4: System Restore. This failed spectacularly as it didn’t know where the Windows install was, and recommended I reboot and try again. I did – no avail. Attempted to run it manually in Command Prompt (“rstrui.exe /OFFLINE:C:\Windows” – according to this thread) and it would let me select restore points. However, it failed to restore a desktop.ini file, and therefore aborted the whole restore procedure. Go figure.
Attempted fix 5: Correcting the boot loader with the “BootRec” utility.

> c:
> cd c:\Windows\System32
> BootRec.exe /FixMbr
(OK)
> BootRec.exe /FixBoot
(OK)
>BootRec.exe /RebuildBCD
(Failed to add file which already exists)

Attempted fix 6: Correcting the boot loader by killing it with fire:

> c:
> cd c:\Windows\System32
> Diskpart
(Now entered the Diskpart utility)
>> List disk
(Only one disk)
>> Sel disk 0
>> List vol
(Three volumes. 0 was c:, 1 was recovery partition, 2 was UEFI FAT32 partition)
>> Sel vol 4
>> assign letter=V:
>> Exit
(Back in normal shell)
>V:
> format v: /FS:FAT32
(Opted yes to the prompt which warned me this was the end of the world)
> c:
> cd c:\Windows\System32
> bcdboot C:\windows /s V: /f UEFI
(Opted yes to the prompt which warned me this was the end of the world again - sorry folks, it's my fault)
> exit

And now, back to the welcome screen which previously only listed “Advanced Options” and “Shutdown PC” – it now listed an option for continuing to boot my PC. It successfully booted – and I now wonder, why didn’t the automated attempts of fixing the boot process detect the EFI partition was broken?

I guess Windows break easily, is why.

[How-To] Previous History on Windows clients with Samba and ZFS Snapshots

Posted on 2020-12-282020-12-29 by Savagedlight

This post describes my final configuration for making the “Previous Versions” fileshare feature work for Windows 10 clients connecting to a FreeBSD Samba server backed by ZFS, as well as how I got there. It involved reading through Samba documentation, code, and various posts on the internet, of which this mailing list entry was the most helpful, to figure out why things didn’t work as expected/documented, and how to circumvent that.

I’m using the Samba VFS module shadow_copy2 to achieve this, and have some observations about this module:

It has an unintuitive shadow:delimiter paramter – the contents of this parameter has to be at the beginning of the shadow:format parameter. It cannot be unset/set to blank, and it defaults to “_GMT“.
The shadow:delimiter parameter cannot start with a literal “-“, and the value of the parameter can’t be quoted or escaped. As an example, “\-” or “-” doesn’t work, whether they are quoted or not in the config.
According to documentation, shadow:snapprefix supposedly only supports “Basic Regular Expression (BRE)”. Although this module supports something which looks like simplified regular expression, it does not seem to support BRE. It also requires all special regular expression characters like ({}) (and possibly others) to be escaped with a backslash(\) character, even though the characters are part of the regex pattern and not to be treated as a literal character. This is not how regexes usually work.
I have not been successfull in using the circumflex/”not”(^) operator, the valid characters (“[]”) operator, the “0 or more” (*), nor the “one or more” (+) operators in regex here.

As such, I had to tweak my snapshot names to have a common ending sequence. The path of least resistance here was to make them all end on “ly” – such as “frequently“, “daily” etc. I also had to spell out every possible snapshot name in the regex.

Working smb.conf

I use “sysutils/zfstools” to create and manage snapshots on my FreeBSD file server, and I have configured it to store snapshots with date/time in UTC. As such, all snapshots are named in the pattern “zfs-auto-snap_(name_of_job)-%Y-%m-%d-%Hh%MU“. As an example, the “daily” snapshot created at 00:07 UTC on 28th December, 2020 is named “zfs-auto-snap_daily-2020-12-28-00h07U“.

[global]
# Previous History stuff
vfs objects = shadow_copy2
shadow:snapdir = .zfs/snapshot
shadow:localtime = false
shadow:snapprefix = ^zfs-auto-snap_(frequent){0,1}(hour){0,1}(dai){0,1}(week){0,1}(month){0,1}$
# shadow:format must begin with the value of shadow:delimiter, 
# and shadow:delimiter cannot start with -
shadow:delimiter = ly-
shadow:format = ly-%Y-%m-%d-%Hh%MU

Example Result

Example output, with a file server in UTC and client in UTC+0100:

Updates to this blog post

2020-12-29
- Removed incorrect “+” in vfs objects directive.
- Moved a paragraph from the ingress to the smb.conf subheading

FreeBSD+ZFS Without Drives

Posted on 2019-06-092019-06-09 by Savagedlight

This weekend I decided to set up a single-drive FreeBSD+ZFS system, and prove that you CAN remove (and replace) the only drive in a root ZFS pool without service interruption.

Recap of the video

Prequisite: A standard FreeBSD 12.0 install with root on ZFS, where the root pool is smaller than the amount of system memory. (in my case: 8GB system memory, 4GB root pool)
Replace existing drive with a memory-backed block device
Physically remove the existing drive
Verify the system still works
Physically attach new drive to system
Replace memory-backed block device with new drive
Bonus: Reboot system, verify it boots from new drive.

Scripts used in the demonstration are provided as a GitHub Gist.

FreeBSD: Semi-manual ZFS+UEFI installation

Posted on 2018-12-252020-12-28 by Savagedlight

This post will show how to install and update a FreeBSD ZFS+UEFI installation.

Start the installer normally, and go through the steps. When you get to the part where it asks whether you want to install to UFS, ZFS, etc, chose to open a shell.

Create the partition scheme for each drive you will be using in your root zpool, and make sure to use unique labels. Make sure to replace ‘ada0’ with whatever is appropriate for you.
gpart create -s gpt ada0 gpart add -t efi -s 800k ada0 gpart add -t freebsd-zfs -a 128m -l YourLabel ada0

I aligned the freebsd-zfs partition to 128MiB to ensure it’s 4k aligned, and to leave room for boot loader changes.

Create the zpool and add datasets, then exit the shell. The datasets for /usr and /var are not mounted, while their child datasets are mounted. This is because most of the data in /usr and /var belongs in the boot environment. Some of the subpaths have their own datasets because they should, in my opinion, be shared among boot environments.

zpool create -m none -o altroot=/mnt -O atime=off -O compress=lz4 sys gpt/YourLabel
zfs create -o canmount=off sys/ROOT
zfs create -o mountpoint=/ -o canmount=noauto sys/ROOT/default
zfs mount sys/ROOT/default
zfs create -o mountpoint=/var -o canmount=off -o compress=gzip-9 -o setuid=off -o exec=off sys/var zfs create sys/var/audit zfs create sys/var/log zfs create -o atime=on sys/var/mail zfs create -o atime=on sys/var/spool zfs create -o exec=on sys/var/tmp zfs create -o mountpoint=/usr -o canmount=off sys/usr zfs create -o compress=gzip-9 sys/usr/src zfs create sys/usr/obj zfs create -o canmount=off sys/data zfs create -o mountpoint=/usr/home -o setuid=off sys/data/homedirs zfs create -o mountpoint=/root sys/data/root

zpool set bootfs=sys/ROOT/default sys exit

Now the installer should continue doing its thing. Do what you’d normally do, but when it asks if you want to open a shell into the new environment, say yes.

Execute this commands to ensure ZFS works as expected:
echo 'opensolaris_load="yes"' >> /boot/loader.conf
echo 'zfs_load="yes" >> /boot/loader.conf
echo 'zfs_enable="YES"' >> /etc/rc.conf

Configure the UEFI partitions by doing the following for each drive that is a member of the ‘sys’ zpool: (remember to replace ‘ada0’ with whatever is appropriate for you)
dd if=/boot/boot1.efifat of=/dev/ada0p1

When upgrading FreeBSD, re-run the above command to apply new bootcode *after* having run installworld.

Supermicro X11SSZ-TLN4F fails to reboot, and BIOS update woes

Posted on 2017-09-06 by Savagedlight

A system which consistently fails to power-cycle. Really fun, especially when it generally sits in a corner not hooked up to a keyboard and/or monitor.

Debugging this issue, I discover the system has no trouble rebooting if I hit ctrl+alt+del before OS boots. But once it has left the BIOS sphere and booted something, be it FreeBSD or FreeDOS, telling it to reboot would make it try to reboot but fail to do anything after the “shutdown”. It would be powered on, fans running, but no POST, no VGA output, and the “power on LED” on the mainboard would not light up. No error beeps or codes.

I’ll try to upgrade BIOS to solve this as I’m on 1.0b and latest available is 2.0a, but doing that requires a reboot to make mainboard enter FLASH/UPDATE mode, which is where it once again fails and can’t get to the point where it actually flashes the BIOS.

I also had some trouble booting the FreeDOS USB stick, I suspect this has something to do with my UEFI/LEGACY settings. Simply switching to legacy or legacy+UEFI boot mode didn’t solve it, but after loading optimized defaults (or resetting BIOS by plucking out the battery, if that’s your thing) it would boot FreeDOS.

But back to actually getting to upgrade the BIOS when upgrading BIOS requires a successful reboot; which this mainboard can’t do, and is the reason I’m upgrading the BIOS in the first place:

After looking through the manual, I discover there’s a jumper that can be changed to force mainboard into FLASH/UPDATE mode from the get-go. I also discover this can be configured in the BIOS screen, so I opt for the second approach and try to upgrade the BIOS again. (The setting is located at: Advanced -> PCH-FW Configuration -> Me FW Image Re-Flash; set it to Enabled)

This time, it successfully flashed the BIOS at least. And the aforementioned Re-Flash setting is back to disabled. But the problem persists, and I’ll keep debugging it.

Venus: Semi-Manual FreeBSD 11-CURRENT AMD64 ZFS+UEFI Installation

Posted on 2016-02-022016-04-05 by Savagedlight

In this post I’ll be describing how to do a semi-manual installation of a FreeBSD 11 ZFS system with UEFI boot. Big thanks to Ganael Laplanche for this mailing list entry, as it was of great help. Some things have changed since then which makes the process a little simpler, and that’s why I’m writing this. :) I’ll also include some steps I consider best practices.

The steps outlined below are generalized from how I installed FreeBSD on my dev box named Venus.

As I’m writing this, the latest FreeBSD 11 snapshot is of r294912 (2016-01-27), and does not yet support automatic installation to ZFS on UEFI systems. I’m using this snapshot for installing the system.

Start the installer normally, and go through the steps. When you get to the part where it asks whether you want to install to UFS, ZFS, etc, chose to open a shell.

I aligned the freebsd-zfs partition to 1M to ensure it’s 4k aligned, and to leave room for boot loader changes. I specified a 55GB partition because my SATADOM’s are 64GB, and I want to leave some free space in case I need to replace one of them with another which isn’t the exact same size, and because I want to leave some room for other things such as a future log, cache or swap partition.

Create the zpool and add datasets, then exit the shell. All datasets within sys/ROOT/default are optional.
zpool create -m none -o altroot=/mnt -O atime=off -O checksum=fletcher4 -O compress=lz4 sys gpt/YourLabel zpool set bootfs=sys/ROOT/default sys zfs create -p sys/ROOT/default/var zfs create -o compress=gzip-9 -o setuid=off sys/ROOT/default/var/log zfs create -o compress=gzip-9 -o setuid=off sys/ROOT/default/var/tmp zfs create sys/ROOT/default/usr zfs create -o compress=gzip-9 sys/ROOT/default/usr/src zfs create sys/ROOT/default/usr/obj zfs create sys/ROOT/default/usr/local zfs create sys/data zfs create -o mountpoint=/usr/home -o setuid=off sys/data/homedirs zfs mount -a exit

Now the installer should continue doing its thing. Do what you’d normally do, but when it asks if you want to open a shell into the new environment, say yes.

Execute this commands to ensure ZFS mounts all datasets on boot:
echo 'zfs_enable="YES"' >> /etc/rc.conf

Configure the (U)EFI partitions by doing the following for each drive that is a member of the ‘sys’ zpool: (remember to replace ‘ada0’ with whatever is appropriate for you)
mkdir /mnt/ada0 newfs_msdos ada0p1 mount -t msdosfs /dev/ada0p1 /mnt/ada0 mkdir -p /mnt/ada0/efi/boot cp /boot/boot1.efi /mnt/ada0/efi/boot/BOOTx64.efi mkdir -p /mnt/ada0/boot cat > /mnt/ada0/boot/loader.rc << EOF unload set currdev=zfs:sys/ROOT/default: load boot/kernel/kernel load boot/kernel/zfs.ko autoboot EOF

At this time you can double check you have the expected file hierarchy in /mnt/ada0:
(cd /mnt/ada0 && find .)
Should output:
. ./efi ./efi/boot ./efi/boot/BOOTx64.efi ./boot ./boot/loader.rc

Now, if you had more than one drive, you can just copy the contents of /mnt/ada0 to the appropriate mountpoints. cp -R /mnt/ada0/ /mnt/ada1/

Remember to unmount the EFI partitions, then exit the shell and reboot into the new system. :)

Once you’re in the new system, you should create a read-only ZFS dataset for /var/empty.

PS: Similar to how you need to re-apply bootcode when upgrading zpool version, you should probably re-copy /boot/loader.efi to the EFI partition as ./efi/boot/BOOTx64.efi. I am not sure if this is strictly necessary… But it shouldn’t hurt. :) I’ll update this paragraph when I get a confirmation one way or the other.

Introduction: Venus, the FreeBSD dev box

Posted on 2016-02-022016-02-02 by Savagedlight

(Also known as: The Mini-ITX quarter-depth chassis that could fit a Micro-ATX mainboard)

I’ll be using this system for my FreeBSD hacking, but this post focus on the system hardware.

The story of this system started when I ordered my Super Micro quarter-depth (SC505-203B) Atom-based firewall named Kuiper from Nextron. The chassis specifications state it’s 24.9cm deep and 43.7cm wide (9.8″ and 17.2″ respectively), and would only fit mini-ITX boards.

As I was also interested in a Xeon E3v5 virtualization server, and would prefer it to be quarter-depth as well, I was a little disappointed that Super Micro didn’t have any mini-ITX mainboards for that platform. Nextron helpfully suggested that they could check if the Super Micro X11SSL-F mainboard (micro-ATX) would fit when building my firewall, as they had it on hand. It has the dimensions 24.4cm by 24.4 cm (9.6″ by 9.6″). I was not expecting it to fit as the depth of the mainboard was a mere 5 mm (0.2″) less than the chassis.

A few days later, when the firewall was built, they reported back: It fits! But they would have to sacrifice one of two 2×2.5″ drive bays. It was also a very, very snug fit, as can be seen in the image below. Excellent! I only need two data drives in that system anyway, and COULD use them as root drives if necessary.

Snug fit! This is NOT the I/O side!

I was happy. Now, considering this chassis was designed for Atom systems, the PSU would probably not be capable of powering a 80W CPU, for not to mention the potential cooling trouble. Nextron suggested getting a 45W CPU, but I decided to pay the premium of getting a low-powered CPU, the Xeon E3-1240L, with a TDP of only 25W.

System Parts
Chassis	SuperChassis CSE-505-203B
Mainboard	Super Micro X11SSL-F
CPU	Xeon E3-1240L
RAM	32GB: 2x 16GB DDR4 2133MHz ECC Unbuffered DIMM
HDD	2x Seagate Laptop Thin SSHD HDD/SSD Hybrid – 500GB SATA3 5400RPM 2.5″
SSD	2x Supermicro SATADOM 64GB MLC – Vertical (added later)

Picture Gallery

Overview of Venus’ interiors, as shipped from Vendor