I got a call and several messages this morning from a client about her Drupal 6 site being down. Its on a pretty stout VPS running smoothly with NGINX with server monitoring. Trying to access the site, I got a 404 error and I couldn’t SSH in. Turns out it was a hardware problem on the server side… so nothing we did. The VPS host fixed it quickly and we were back up… but while browsing the site was fine, no one could log in… not even me! It turned out that the server crash had messed up a few different things, but it took some doing to figure out what. I’m posting this as a note for myself and others who may come this way later.
First I checked the basics:
Clear browser cookies for the site and try again
Check the $base_url variable in settings.php
Check the $cookie_domain variable in settings.php
But – this didn’t fix it… which meant I need to dig in a bit more
Managing Drupal when you can’t log in
The first problem I had was just figuring out how to do the basics… I mean, clearing the cache is Drupal 101 (akin to rebooting your OS), but I couldn’t even log in so how could I clear cache? The answer is Drush. For those who don’t know, Drush is an interface for your server that lets you do Drupal stuff from the command line (see http://drush.org/ for a list of commands). I quickly installed it, and was able to clear the cache easily:
Yup… thats pretty easy. Unfortunately, this didn’t do the trick. So – I figured the next thing to do was to turn on Watchdog (the error log) and see what errors I might be getting when logging in:
NOTE: you should usually leave this OFF on production servers by default, and can leave the Syslog module turned on… then you should be able to access Drupal errors in the system error log. I was just doing this to make it easy for myself
// turn on the database logging
drush pm-enable dblog
// try logging in and going to admin again to generate errors
This showed that the sessions table was “broken” and in need of repair (along with some boost tables). This makes sense if the server shut down right in the middle of processing requests, especially since this is a busy site. Now, there are many ways to handle this – and the easiest is to just TRUNCATE the sessions table. You can also opt for the easy solution (as I did), and just log into phpMyAdmin and “repair” the table. But, I wanted to be sure that people weren’t going to still be trying to log in while I messed around. So – I put the site into maintenance mode:
// put into maintenance mode
drush vset site_offline 1
// take out of maintenance mode
drush vset site_offline 0
Drush plus Drupal functions
While in maintenance mode, I was able to repair the tables and clear the errors… but I STILL couldn’t log in. At this point I knew my settings.php file was ok, the DB tables were ok, AND I was still able to check and see that watchdog wasn’t repporting any new errors… it was just giving me access denied. I also looked and my new sessions were being recorded in the DB sessions table.
So that really only left one thing: rebuilding the node access table. This can easily be done when you are logged in as admin by going to Admin >> Content management >> Post settings (in Drupal 6). But again… I can’t login… However, because Drush works through a bootstrapped version of the site, I have access to Drupal functions:
drush php-eval ‘node_access_rebuild();’
That did it! I was able to log in and everything else was fine after that. It did take some time for me to work through this, but now that I have, it shouldn’t take but a few minutes if I ever need to do this again. This would have been a nightmare without drush, and I am so glad that I had enough familiarity to know that this was an option. I’ve been using drush for awhile to check status and update modules… but that was mostly because its just so much faster than doing it by hand. This is the first time drush saved my day, and I have the feeling it won’t be the last.
A note on Drupal 7
The above was written about a Drupal 6 site… but drush is drush. I was using drush version 5, but version 4 has most of the same commands. A few details above might change, but in theory most all of this should also work for 7. But – be mindful.