Linux Mint 22 upgrade stories
3/Aug 2024The shorter story
Let’s start with a shorter first story. I had a relatively new installation of Linux Mint 21.3 on my laptop. sudo mintupgrade was what it takes to upgrade to Linux Mint 22. Easy.
There was an annoying issue with ssh agent and SSH_AUTH_SOCK being empty after upgrading, causing encrypted SSH key to prompt for passphrase on every use, e.g. git pull. The fix was to compile and install git-credential-libsecret, and to restart the shell or the system. I believe it used to use GNOME Keyring.
The longer story
There’s a longer story about upgrading my PC to Linux Mint 22 though. The system has been around for a much longer time, at least for a few major releases.
With the same sudo mintupgrade, first I had to remove a few 3rd party package repos, no big deal. Then I ran into the first roadblock on package upgrade.
kbd initramfs hook
E: /usr/share/initramfs-tools/hooks/kbd failed with return 1.
update-initramfs: failed for /boot/initrd.img-6.8.0-39-generic with 1.
dpkg: error processing package initramfs-tools (--configure):
installed initramfs-tools package post-installation script subprocess returned error exit status 1
/usr/share/initramfs-tools/hooks/kbd contained references to non-existent paths, e.g. it references /bin/setfont while the file was moved to /usr/bin/setfont in a recent update. This causes the hook to fail. I filed a bug report for this. The fix was to manually edit the hook file to use the correct paths.
sudo
Well, it wasn’t that straightforward to edit the hook, as it is owned by root. During upgrade, sudo wouldn’t work.
sudo: PAM account management error: Module is unknown
sudo: a password is required
There was nothing I could do at this point. Aborting the upgrade process.
Live USB & timeshift
Aborting the upgrade process left me with an unbootable system. The only way out was to boot with a live USB and restore the timeshift snapshot. It was the first time I restored a timeshift snapshot and it was really impressive. The system booted up again as if nothing happened.
sudo su
Fool me once, shame on you; fool me twice, shame on me. This time, I ran sudo mintupgrade while keeping a spare shell with sudo su on the side. I was met with the same kbd issue, and this time it was fixed by manually editing the hook.
kpartx initramfs hook
A similar issue came up (what’s wrong with noble packages?) for kpartx.
cp: cannot stat '/lib/udev/rules.d/56-dm-parts.rules': No such file or directory
cp: cannot stat '/lib/udev/rules.d/68-del-part-nodes.rules': No such file or directory
cp: cannot stat '/lib/udev/rules.d/95-kpartx.rules': No such file or directory
E: /usr/share/initramfs-tools/hooks/kpartx failed with return 1.
update-initramfs: failed for /boot/initrd.img-6.8.0-39-generic with 1.
run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1
dpkg: error processing package linux-image-6.8.0-39-generic (--configure):
installed linux-image-6.8.0-39-generic package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
initramfs-tools
linux-image-6.8.0-39-generic
After analyzing the kpartx and kpartx-boot package content, I noticed the outdated paths and again filed a bug report. The fix was also to manually edit the hook.
Package dependency problems
The next problem about package dependency was a nightmare.
dpkg --configure -a
dpkg: dependency problems prevent configuration of libc-bin:
libc-bin depends on libc6 (>> 2.39); however:
Version of libc6:amd64 on system is 2.35-0ubuntu3.8.
dpkg: error processing package libc-bin (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of rpcbind:
rpcbind depends on libc6 (>= 2.38); however:
Version of libc6:amd64 on system is 2.35-0ubuntu3.8.
rpcbind depends on libtirpc3t64 (>= 1.3.4+ds-1.1); however:
Package libtirpc3t64 is not installed.
dpkg: error processing package rpcbind (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of locales:
locales depends on libc-bin (>> 2.39); however:
Package libc-bin is not configured yet.
dpkg: error processing package locales (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of libc6-dbg:amd64:
libc6-dbg:amd64 depends on libc6 (= 2.39-0ubuntu8.2); however:
Version of libc6:amd64 on system is 2.35-0ubuntu3.8.
dpkg: error processing package libc6-dbg:amd64 (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of libtirpc-dev:amd64:
libtirpc-dev:amd64 depends on libtirpc3t64 (= 1.3.4+ds-1.1build1); however:
Package libtirpc3t64 is not installed.
dpkg: error processing package libtirpc-dev:amd64 (--configure):
dependency problems - leaving unconfigured
dpkg: error processing package libc6:amd64 (--configure):
package libc6:amd64 2.35-0ubuntu3.8 cannot be configured because libc6:i386 is at a different version (2.39-0ubuntu8.2)
dpkg: error processing package libc6:i386 (--configure):
package libc6:i386 2.39-0ubuntu8.2 cannot be configured because libc6:amd64 is at a different version (2.35-0ubuntu3.8)
dpkg: dependency problems prevent configuration of libnsl-dev:amd64:
libnsl-dev:amd64 depends on libtirpc-dev; however:
Package libtirpc-dev:amd64 is not configured yet.
dpkg: error processing package libnsl-dev:amd64 (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of libnsl2:amd64:
libnsl2:amd64 depends on libc6 (>= 2.33); however:
Package libc6:amd64 is not configured yet.
libnsl2:amd64 depends on libtirpc3t64 (>= 1.0.2); however:
Package libtirpc3t64 is not installed.
dpkg: error processing package libnsl2:amd64 (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of libnsl2:i386:
libnsl2:i386 depends on libc6 (>= 2.38); however:
Package libc6:i386 is not configured yet.
libnsl2:i386 depends on libtirpc3t64 (>= 1.0.2).
dpkg: error processing package libnsl2:i386 (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of valgrind:
valgrind depends on libc6 (>= 2.38); however:
Version of libc6:amd64 on system is 2.35-0ubuntu3.8.
valgrind depends on libc6-dbg; however:
Package libc6-dbg:amd64 is not configured yet.
dpkg: error processing package valgrind (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent processing triggers for man-db:
man-db depends on libc6 (>= 2.34); however:
Package libc6:amd64 is not configured yet.
dpkg: error processing package man-db (--configure):
dependency problems - leaving triggers unprocessed
Errors were encountered while processing:
libc-bin
rpcbind
locales
libc6-dbg:amd64
libtirpc-dev:amd64
libc6:amd64
libc6:i386
libnsl-dev:amd64
libnsl2:amd64
libnsl2:i386
valgrind
man-db
Error - Return code: 1
DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get install -fyq
Reading package lists...
Building dependency tree...
Reading state information...
Correcting dependencies... failed.
The following packages have unmet dependencies:
libc-bin : Depends: libc6 (> 2.39) but 2.35-0ubuntu3.8 is installed
libc6 : Breaks: libc6:i386 (!= 2.35-0ubuntu3.8) but 2.39-0ubuntu8.2 is installed
libc6:i386 : Breaks: libc6 (!= 2.39-0ubuntu8.2) but 2.35-0ubuntu3.8 is installed
libc6-dbg : Depends: libc6 (= 2.39-0ubuntu8.2) but 2.35-0ubuntu3.8 is installed
libnsl2 : Depends: libtirpc3t64 (>= 1.0.2) but it is not installed
libnsl2:i386 : Depends: libtirpc3t64:i386 (>= 1.0.2) but it is not installed
libtirpc-dev : Depends: libtirpc3t64 (= 1.3.4+ds-1.1build1) but it is not installed
rpcbind : Depends: libc6 (>= 2.38) but 2.35-0ubuntu3.8 is installed
Depends: libtirpc3t64 (>= 1.3.4+ds-1.1) but it is not installed
valgrind : Depends: libc6 (>= 2.38) but 2.35-0ubuntu3.8 is installed
E: Error, pkgProblemResolver::Resolve generated breaks, this may be caused by held packages.
E: Unable to correct dependencies
Error - Return code: 100
I noticed the :i386 packages and figured they should not be needed. But it was too late to remove :i386 packages now, because everything was already broken.
Remove i386 packages
Live USB and timeshift snapshot restore. I removed the i386 packages before sudo mintupgrade.
This Super User answer contains the commands to remove i386 packages:
dpkg --print-foreign-architectures
dpkg -l | grep i386
apt-get purge ".*:i386" --allow-remove-essential
dpkg --remove-architecture i386cannot divert directories
Continuing with package upgrades, and another error appeared.
dpkg-divert: error: cannot divert directories
Use --help for help about diverting files.
dpkg: error processing archive /var/cache/apt/archives/libc6_2.39-0ubuntu8.2_amd64.deb (--unpack):
new libc6:amd64 package pre-installation script subprocess returned error exit status 2
Errors were encountered while processing:
/var/cache/apt/archives/libc6_2.39-0ubuntu8.2_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
Error - Return code: 100
A quick Google search returned this glibc ubuntu package bug. Basically the fix was to download usrmerge package from online, because there is no noble package, only a jammy one, then install it with dpkg -i.
Package dependency problems (continued)
After fixing libc6 cannot divert directories, apt --fix-broken install did not go well.
Reading package lists...
Building dependency tree...
Reading state information...
Correcting dependencies... failed.
The following packages have unmet dependencies:
libatk-adaptor : Depends: libatk-bridge2.0-0t64 (>= 2.52.0-1build1) but it is not installed
Depends: libglib2.0-0t64 (>= 2.62) but it is not installed
libc-dev-bin : Depends: libc6 (> 2.39) but 2.35-0ubuntu3.8 is installed
Recommends: libc-devtools (> 2.39) but 2.35-0ubuntu3.8 is installed
libc6-dbg : Depends: libc6 (= 2.39-0ubuntu8.2) but 2.35-0ubuntu3.8 is installed
libc6-dev : Depends: libc6 (= 2.39-0ubuntu8.2) but 2.35-0ubuntu3.8 is installed
libc6-i386 : Depends: libc6 (= 2.39-0ubuntu8.2) but 2.35-0ubuntu3.8 is installed
libc6-x32 : Depends: libc6 (= 2.39-0ubuntu8.2) but 2.35-0ubuntu3.8 is installed
libcups2-dev : Depends: libcups2t64 (= 2.4.7-1.2ubuntu7.2) but it is not installed
libcupsimage2-dev : Depends: libcupsimage2t64 (= 2.4.7-1.2ubuntu7.2) but it is not installed
Recommends: libcupsfilters-dev but it is not installed
libgfapi0 : Depends: libtirpc3t64 (>= 1.1.4) but it is not installed
libgfrpc0 : Depends: libtirpc3t64 (>= 1.1.4) but it is not installed
libgfxdr0 : Depends: libtirpc3t64 (>= 1.1.4) but it is not installed
libglusterfs0 : Depends: libc6 (>= 2.38) but 2.35-0ubuntu3.8 is installed
Depends: libssl3t64 (>= 3.0.0) but it is not installed
libguestfs-xfs : Depends: libguestfs0t64 (= 1:1.52.0-5ubuntu3) but it is not installed
libtirpc-dev : Depends: libtirpc3t64 (= 1.3.4+ds-1.1build1) but it is not installed
locales : Depends: libc-bin (> 2.39) but 2.35-0ubuntu3.8 is installed
valgrind : Depends: libc6 (>= 2.38) but 2.35-0ubuntu3.8 is installed
E: Error, pkgProblemResolver::Resolve generated breaks, this may be caused by held packages.
E: Unable to correct dependencies
Error - Return code: 100
It was not helpful. No amount of dpkg --purge --force-depends and --force-all --configure -a worked.
Inspired by this blog post, I did aptitude install libc6. It presented a plan to resolve the dependencies and it worked magically. Note to self: aptitude is wonderful.
Upgrade successful
Finally, the upgrade was successful.
Running apt-get clean
DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get clean
Running update-initramfs -u
update-initramfs -u
update-initramfs: Generating /boot/initrd.img-6.8.0-39-generic
Upgrade successful! You can now close this terminal and reboot your computer.
Black screen
Upon reboot, I was welcomed with a black screen. Nice.
Recovery mode
Restarted again, this time into recovery mode, which is essentially with kernel param nomodeset, it showed the lightdm login screen. However, it would just loop back to the login screen after entering the password to start the default X desktop environment.
Logs from journalctl -f wasn’t too useful. No luck with messing with .XAuthority and sudo Xorg -configure. Installed the missing xserver-xorg-video-amdgpu package but it didn’t help either.
I switched to login to a Wayland session by selecting in the dropdown next to the username. That worked.
amdgpu
The display was a little sluggish in Wayland and I figured it was using the software renderer. The monitor name also didn’t show up in Display.
$ lsmod | grep amdgpu
$ lsmod returned nothing about amdgpu, which was strange, because I thought newer kernels would come with amdgpu module.
$ sudo modprobe amdgpu
modprobe: ERROR: could not insert 'amdgpu': Invalid argumentmodprobe returned an error.
Kernel Mode Setting (KMS)
The fact that it booted up in recovery mode but not normal mode was a hint. It was the nomodeset kernel parameter that caused recovery mode to work. Something was wrong with kernel not loading the amdgpu module.
According to Arch amdgpu docs,
Specify the correct module order
Make sure amdgpu has been set as first module in the Mkinitcpio#MODULES array, e.g.
MODULES=(amdgpu radeon).
There’s no Mkinitcpio in Ubuntu, but the equivalent would be /etc/initramfs-tools/modules.
After adding amdgpu to /etc/initramfs-tools/modules, it booted up without recovery mode. lsmod and inxi -G look good as well.
$ lsmod | grep amdgpu
amdgpu 17100800 39
video 73728 1 amdgpu
amdxcp 12288 1 amdgpu
i2c_algo_bit 16384 1 amdgpu
drm_exec 12288 1 amdgpu
gpu_sched 61440 1 amdgpu
drm_suballoc_helper 16384 1 amdgpu
drm_buddy 20480 1 amdgpu
drm_display_helper 237568 1 amdgpu
drm_ttm_helper 12288 1 amdgpu
ttm 110592 2 amdgpu,drm_ttm_helper
$ inxi -G
Graphics:
Device-1: AMD Navi 22 [Radeon RX 6700/6700 XT/6750 XT / 6800M/6850M XT]
driver: amdgpu v: kernel
Device-2: Logitech HD Pro Webcam C920 driver: snd-usb-audio,uvcvideo
type: USB
Display: wayland server: X.Org v: 23.2.6 with: Xwayland v: 23.2.6 driver:
X: loaded: amdgpu unloaded: fbdev,modesetting,vesa dri: radeonsi gpu: amdgpu
resolution: 2560x1440~144Hz
API: EGL v: 1.5 drivers: radeonsi,swrast
platforms: wayland,x11,surfaceless,device
API: OpenGL v: 4.6 compat-v: 4.5 vendor: amd mesa v: 24.0.9-0ubuntu0.1
renderer: AMD Radeon Graphics (radeonsi navi22 LLVM 17.0.6 DRM 3.57
6.8.0-40-generic)
API: Vulkan v: 1.3.275 drivers: N/A surfaces: xcb,xlib,wayland
Conclusion
In retrospect, installing a fresh system could have saved me 10 hours, but I am glad it is now fixed, despite the painful journey. I hope this will be useful for anyone who runs into similar issues.
Last but not least, thank Clem and team for the awesome work!