View source for VM recovery via cloud console

<languages/>
<translate>

<!--T:3-->
If the VM can't be accessed anymore via SSH or via a local user, the OS can be booted into single user mode or a recovery kernel can be launched, which provides privileged access to the OS image. The only requirement is that the boot manager is accessible and can be modified.

== Debian10 recovery == <!--T:4-->

<!--T:5-->
The recovery procedure is not that easy and convenient, as you would expect from CentOS; the functionality is the same or at least similar. Most cloud images have the root account locked, so just booting single user won't help us. However, when a Linux-based system boots, regardless what flavor it is, the kernel gives up the control into userspace for all things related to userspace, like running daemons, etc. That is done as soon as all the hardware is initialized, then the kernel runs a single userspace binary, called the <code>init</code> process which always has PID1; in most recent distributions it is either <code>systemd</code>, <code>systemV</code> or <code>upstart</code>. Via the boot manager, we are able to modify that and tell the kernel to execute a shell instead and manually mount the image filesystem and do our recovery operations. The debian10 image comes with GRUB2 as well, but the menu looks a little different; however, the keys and key combinations we need to use are all the same. Boot or reboot the system until you see the GRUB menu, then hit <code>e</code> for <i>edit</i>. Remove the serial consoles and add <code>init=/bin/bash</code> to let the kernel know the new <code>init</code> process.

<!--T:6-->
Modify the line after <code>linux</code> like below:

<!--T:7-->
<code>linux  /boot/vmlinuz-4.19.0-6-cloud-amd64 root=UUID=d187d85e-8a80-4664-8b5a-dce4d7ceca9e ro  biosdevname=0 net.ifnames=0 console=tty0 init=/bin/bash</code>

<!--T:8-->
That will boot the kernel, initialize <code>initrd</code> and execute <code>/bin/bash</code> as the <code>init</code> process. Now, we basically landed in memory and are mounted r/o, since the userspace <code>init</code> process is supposed to take care of the root filesystem; the kernel just needs to know where to find it before it hands over the control. To do a useful recovery, the next steps will be to remount the initrd filesystem r/w, mount the OS image disk, chroot into it, set a root password and restart the VM. After a successful restart, we can login as root. Take note that bash has no <code>reboot</code> or any power control mechanism, so we have to unmount everything cleanly and stop the VM.

<!--T:9-->
Within our initrd remount the file system r/w:

<!--T:10-->
<code>mount -o remount,rw /</code>

<!--T:11-->
Mount /dev/vda1 (the first primary partition) to /mnt:

<!--T:12-->
<code>mount /dev/vda1 /mnt</code>

<!--T:13-->
We now have now the image root filesystem r/w mounted at <code>/mnt</code>, to use tools like <code>passwd</code> via chroot in there, we need to mount <code>/dev</code> to gain tty access and <code>/proc</code> and <code>/sys</code>, since we can then also access the network.

<!--T:14-->
<code>mount -o bind /proc /mnt/proc</code> <code>mount -o bind /sys /mnt/sys</code> <code>mount -o bind /dev /mnt/dev</code>

<!--T:15-->
Then chroot into <code>/mnt</code>, which will show an ioctl error for the terminal process group, we can ignore that. Now we can just use <code>passwd</code> to reset the root password, once done we leave the chroot via <code>Ctrl+D</code>, unmount out previously mounted mount points and restart the system by using the <code>Ctrl+Alt+Del</code> submit button on the OpenStack console page, you can also just stop and start the VM since we unmounted all real filesystems, they are already synced and all buffers flushed to the virtual disk. So we are strictly operating in memory, which is volatile anyway.

<!--T:16-->
After the VM has started, you can now login as user root with the password you have chosen. Once completed, remove the root password again, or disable direct root logins via ssh.

== CentOS7 recovery == <!--T:17-->

<!--T:18-->
Open the console via Horizon and reboot the VM; the <code>CtrlAltDel</code> button in the upper right corner can be used for that, unless you need to recover a kernel persistent kernel panic. At one point the boot manager shows up, which is currently for all cloud images GRUB or GRUB2. Other would work as well, they will only have a different key sequence to gain access to the append parameters for the kernel. Once the GRUB menu is visible, hit <code>e</code> on your keyboard to get into edit mode, you will see something like this:

<!--T:19-->
<pre>        insmod xfs
        set root='hd0,msdos1'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint='hd0,msdos1'  3ef2b806-efd7-4eef-aaa2-2584909365ff
        else
          search --no-floppy --fs-uuid --set=root 3ef2b806-efd7-4eef-aaa2-2584909365ff
        fi
        linux16 /boot/vmlinuz-3.10.0-1127.19.1.el7.x86_64 root=UUID=3ef2b806-efd7-4eef-aaa2-2584909365ff ro console=tty0 console=ttyS0,115200n8 crashkernel=auto console=ttyS0,115200 LANG=en_US.UTF-8
        initrd16 /boot/initramfs-3.10.0-1127.19.1.el7.x86_64.img
</pre>
Now, navigate to the line which starts with <code>linux16</code>. Here, all console parameters need to be removed. Since qemu uses the serial console (ttySX), we would have to go onto the compute node directly and attach it there to a terminal. The easier option is just to leave <code>console=tty0</code> in there. If we want to have the filesystem from the image mounted r/w we would have to change the parameter <code>ro</code> to <code>rw</code>, but that can be done later as well; if something needs to investigated, r/o is a very good option to leave timestamps intact on inodes. Centos has a parameter to interrupt the boot process in an early stage, which is <code>rd.break</code>. The linux16 line should then look like this (the order of the parameters do not matter):

<!--T:20-->
<code>linux16 /boot/vmlinuz-3.10.0-1127.19.1.el7.x86_64 root=UUID=3ef2b806-efd7-4eef-aaa2-2584909365ff ro rd.break console=tty0 crashkernel=auto LANG=en_US.UTF-8</code>

<!--T:21-->
To boot the kernel with the changes, hit <code>Crtl+x</code>. Under <code>/sysroot</code>, you will find the ro mounted filesystem from the image, you can chroot into it or modify it directly. To make it rw, it needs to be remounted: <code>mount -o remount,rw /sysroot</code>.

== CentOS8 recovery == <!--T:22-->

<!--T:23-->
The steps are very similar compared to the CentOS7 recovery procedure: the option <code>console</code> needs to initialize a tty instead of a serial console and <code>rd.break</code> will launch the recovery environment.

<!--T:24-->
e.g: <code>root=UUID=c7b1ead0-f176-4f23-b9c7-299eb4a06cef ro console=tty no_timer_check net.ifnames=0 crashkernel=auto</code>

<!--T:25-->
[[Category:Cloud]]
</translate>