VM recovery via cloud console: Difference between revisions

That will boot the kernel, initializes initrd and executes <code>/bin/bash</code> as the init process, now we basically landed in memory and are mounted r/o, since the userspace init process is supposed to take care of the root filesystem, the kernel just needs to know where to find it before it hands over the control. To do a useful recovery, the next steps will be remounting the initrd filesystem r/w, mount the OS image disk, chroot into it, set a root password and restart the VM. After a successfull restart, we can login as root. Take note that bash has no <code>reboot</code> or any powercontrol mechanism, so we have to unmount everything cleanly and stop the VM.

We have now the image root filesystem r/w mounted at /mnt, to use tools like <code>passwd</code> via chroot in there, we need to mount <code>/dev</code> to gain tty access and <code>/proc</code> and <code>/sys</code>, since we can then also access the network.

After the VM has started, you can now login as user root with the password you have chose. Once completed, remove the root password again, or disable direct root logins via ssh.

Open the console via Horizon and reboot the VM, in the upper right corner the <code>CtrlAltDel</code> button can be used for that, unless you need to recover a kernel persistent kernel panic. At one point the boot manager shows up, which is currently for all cloud images GRUB or GRUB2. Other would work as well, they will only have a different key sequence to gain access to the append parameters for the kernel. Once the GRUB menu is visible, hit <code>e</code> on your keyboard to get into edit mode, you will see something like below.

Now navigate to the line which starts with <code>linux16</code>, here all <code>console</code> parameters need to be removed, since qemu uses the serial console (ttySX), we would have to go onto the compute node directly and attach it there to a terminal. The easier option is just to leave <code>console=tty0</code> in there. If we want to have the filesystem from the image mounted r/w we would have to change the paramter <code>ro</code> to <code>rw</code>, but that can be done later as well if something needs to investigated r/o is a very good option to leave timestamps intact on inodes. Centos has a parameter to interrupt the boot process in an early stage, which is <code>rd.break</code>. The linux16 line should then look like the below (the order of the parameters do not matter):

To boot the kernel with the changes hit <code>Crtl+x</code>. Under <code>/sysroot</code> you will find the ro mounted filesystem from the image, you can chroot into it or modify it directly. To make it rw, it needs to be remounted: <code>mount -o remount,rw /sysroot</code>.

The steps are very similar compared to the CentOS7 recovery procedure, the option <code>console</code> needs to initialize a tty instead of a serial console and <code>rd.break</code> will launch the recovery environment.