Ubuntu: apt-get upgrade makes logins fail!

[sethml]

Arrgh.  After doing an 'apt-get upgrade' of the stock Ubuntu install on the device, all attempts to log in start failing:

Code:

% ssh root@pluggy
root@pluggy's password:
Permission denied, please try again.
root@pluggy's password:

% ssh sethml@pluggy
sethml@pluggy's password:
Permission denied, please try again.
sethml@pluggy's password:

root@pluggy:~# su sethml
su: Authentication failure
(Ignored)
sethml@pluggy:/root$ id
uid=501(sethml) gid=501(sethml) groups=501(sethml)
sethml@pluggy:/root$ su root
Password:
su: Authentication failure
sethml@pluggy:/root$ id
uid=501(sethml) gid=501(sethml) groups=501(sethml)

(Yes, I'm typing the right password.)  Attempts to change the password also fail mysteriously:

Code:

root@pluggy:~# passwd root
Enter new UNIX password:
Retype new UNIX password:
passwd: Authentication token manipulation error
passwd: password unchanged

I suspect that the PAM stuff has changed in some incompatible way, but it's hard to figure out how.  There's nothing useful in /var/log/auth.log.

Once this has happened, if you reboot or log out of your existing shells, there's no good way to log back into the system!

Have other people had this problem?  Has anyone done an apt-get upgrade and not had their system become unusable?  Anybody figured out how to fix the problem?  Or even how to get PAM to give more useful debugging info?

[plugit]

Did you do an apt-get update first? I've updated/upgraded the stock fs several times without failure...

[sethml]

Yes, I did the "apt-get update" first.  I've done this twice now - once with the firmware that came on the plug, the second time after reflashing uImage to uImage.sheeva.20090319 (after pain figuring out how to change the uBoot partition size) and reflashing the root image to ubuntu-9.0.5.Release.jffs2.

Arrgh - adding 'single' to the kernel command line doesn't help get the system back:

Code:

Give root password for maintenance
(or type Control-D to continue):
Login incorrect.

[fjr]

What you need on your kernel command line is rather 'init=/bin/bash' and when you are at the prompt, start by mounting the procfs otherwise many commands will fail:
mount -t proc proc /proc

[sethml]

Thanks fjr.  At this point I've wiped the ubuntu install, so I can't test that.  Adding "init=/bin/sh" to the kernel command line didn't work - it got through the kernel boot and then just sat there.  I can't imagine why bash would behave differently, but if I end up in this situation again I'll try it.

plugit: have you don't an apt-get upgrade recently?  Perhaps ubuntu recently broke PAM somehow?

[KaiBo]

I experienced that problem a week ago too. After assuming the worst (I had an SSH-Port forwarded over night) I noticed as well that it seems to have something to do with an upgrade. However, after several installing-testing-cycles it miraculously ^*1 worked and kept working ever since. I have not been able to find out what exactly caused that behavior but my last assumption was renaming the system. Can you confirm that?


^*1How is that spelled?

[sethml]

I just did a clean reinstall of the ubuntu-9.0.5 image and then did pretty much exactly:

• boot into new image
• comment out the "supersede" line from /etc/dhcp3/dhclient.conf
• ifdown eth ; ifup eth0
• apt-get update
• apt-get upgrade

After that logging in still worked.  I changed my hostname, but with "hostname pluggy.ofb.net" and editing /etc/hostname, and logging in still works.  I rebooted, and logging in still works.  So at this point my best theory is that editing the hostname before upgrading screws things up somehow, which seems very unlikely, but I don't have a better idea.  I'm not about to wipe and upgrade again just to find out.

Sometimes I get pretty frustrated with how fragile and voodoo Linux has become...

[sethml]

I've written up a HOWTO on the wiki with some of my hard-earned wisdom:
  http://plugcomputer.org/plugwiki/index.php/New_Plugger_How_To

If anybody figures out this login issue, please update the wiki page with your solution so others don't have to go through it!

The wiki documentation really could use more cleanup, but I think I need to go dig in the garden instead.

[bfmorgan]

I've just gone thru 2 re-install with updates. I believe the update that is killing the ability to login after the update is the apt-get dist-upgrade step.

As soon as I started the apt-get dist-upgrade I tried to start another window and login, no luck.

I'm still looking at the log and will get back as soon as I figure this out.

[Gothnet]

I had this problem too.

I can't tell you the root cause but I can tell you the easy way to fix it.

Connect oover serial,
Add init=/bin/bash to the bootargs in uboot
Run passwd -d root
run passwd root
Enter password twice

For some reason it will now let you set the password. I tried a load of other things and wasted a lot of time on this, but that's what worked. It seems to have some sort of problem setting the password if there's one there, even if you've deleted it from /etc/shadow, but allowing the passwd program to remove it and then add a new one in works fine.

What's causing it is a mystery. I even tried running strace against the passwd executable and couldn't see anything obviously wrong.

[catch03]

I had problems with Gothnet's solution. I would log in over serial, and still get the

Code:

root@DB88FXX81:/# passwd root
Enter new UNIX password:
Retype new UNIX password:
passwd: Authentication token manipulation error
passwd: password unchanged

which was really frustrating...

Took me forever to get this fixed.
1st, check /etc/shadow and /etc/passwd for your accounts.

2nd, check /etc/shadow for duplicates. For one of my accounts, it was in triplicate! So leave the first copy and delete the remaining duplicates.

3rd, run a 'passwd -d (username)' this deletes the stored password for the user.

4th, run 'passwd (username)' to set the new password!

Example for steps 3 and 4:

Code:

root@DB88FXX81:/# passwd -d root
Password changed.
root@DB88FXX81:/# passwd root
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully

5th, reboot the device and take off "init=/bin/bash" in the bootargs. Don't forget saveenv!

6th, finally, 'reset' and enjoy

Hope this helps!

[bpsbr_ernie]

Easiest way I found was:

Right after the apt-get upgrade

1) pwconv (to sync the passwd and shadow files)
2) passwd root (to set the root password again)

[gaudencio]

ok, I just got my box and on the upgrade got this message
  | Running services and programs that are using NSS need to be restarted, otherwise they might   |
  | not be able to do lookup or authentication any more (for services such as ssh, this can       |
  | affect your ability to login). Please review the following space-separated list of init.d     |
  | scripts for services to be restarted now, and correct it if needed.     

I'm guessing that's what's causing the problems. I tried adding sshd to the list (which just had cron and samba I think) but when it came to restart the daemon it spurted out some message. Either way, I had no problem logging into the box after the update, so, in summation......who knows.

Just thought that info might benefit someone

[youmustbejoking]

I added just 'ssh' instead of 'sshd' and that seemed to do it.  Haven't logged out yet, but it seemed to go through without reporting an error.