Monday, September 13, 2010

Install pair of KVM servers for Web and DB

This post is for my personal reference. I'm using custom scripts to manage the installation of KVM servers, so the information in here will probably not be useful to anyone else.

Examine the files in /root/vmbuilder/appliances/mysql to see if anything needs updating. For instance, vmbuilder.partition may need to be adjusted. Also examine /root/vmbuilder/appliance/jetty files. If changes are significant, then copy the appliance folder and create a new appliance folder. For instance:

cd /root/vmbuilder/appliances
cp -a mysql mysql-production

Then, inside /root/vmbuilder, and as the root user:

./run_vmbuilder.sh -a mysql -h db1 -i xxx.xxx.xxx.219 -d domainname.com -uusername -n "Real User Name"
./run_vmbuilder.sh -a jetty -h web1 -i xxx.xxx.xxx.220 -d domainname.com -uusername -n "Real User Name"
At the end of each command, output similar to this will be shown:

Your server login details:

Hostname: db1.domainname.com
IP Number: xxx.xxx.xxx.219
Username: username
Temporary Password: vr545Lz4

Next, log into each system with the password provided:

ssh username@db1.domainname.com

Upon logging in, I will be prompted to change your password.

Then, from my local system, install ssh key to each server, entering my new password when prompted:

ssh-copy-id username@db1.domainname.com

Now, log into the new MySQL server and run the following commands. Use KeepPassX to generate a strong password for mysql root, and store login information in KeyPassX for future reference.

date
more /etc/timezone
sudo dpkg-reconfigure tzdata
sudo dpkg-reconfigure mysql-server-5.1
mysql -uroot -p mysql

Create a new database with the following commands:

show databases;
create database mydbname;
grant all privileges on mydbname.* to dbuser@localhost identified by 'mypassword';
grant all privileges on mydbname.* to dbuser@"%" identified by 'mypassword';
flush privileges;
exit;

On the Web server, run these commands:

date
more /etc/timezone
sudo dpkg-reconfigure tzdata
mkdir bin
mkdir web
mkdir -p backups/database
mkdir -p backups/logs
mkdir -p backups/webapps
mkdir -p backups/jcr
cd bin
vi editdb
vi savedb
chmod 755 *
crontab -e

Insert the following into editdb:

#!/bin/sh

##########################
# Configuration settings #
##########################

# Database server
SERVER=db1.domainname.com

# Database username
USER=dbuser

# Password to access database
PASS=mypassword

# Name of the mysql database
DBNAME=mydbname

mysql -u$USER -p$PASS -h$SERVER $DBNAME

Insert the following into savedb:

#!/bin/sh

##########################
# Configuration settings #
##########################

# Database server
SERVER=db1.domainname.com

# Database username
USER=dbuser

# Password to access database
PASS=mypassword

# Name of the mysql database
DBNAME=mydbname

# Folder to store backups
LOCATION=$HOME/backups/database

NOW=`date +%Y%m%d-%H%M%S`

mysqldump -u$DBUSER -p$DBPASS -h$DBSERVER $DBNAME > $LOCATION/$DBNAME.$NOW.sql
Set the following crontab:

# m h dom mon dow command
29 */12 * * * /home/username/bin/savedb
35 0 * * * /home/username/bin/savejcr

Now, lets get Jetty configured properly:

cd ~/jetty/contexts
mv javadoc.xml javadoc.xml.bak
mv test-jndi.xml test-jndi.xml.bak
mv test.xml test.xml.bak
vi web.xml
cd ..
mv webapps webapps.orig
mkdir webapps
vi start.sh
vi stop.sh
cd webapps
scp otherhost:/path/to/war/myfile.war ROOT.war

Insert the following into web.xml:

<?xml version="1.0"  encoding="ISO-8859-1"?>
<!DOCTYPE Configure PUBLIC "-//Mort Bay Consulting//DTD Configure//EN" "http://jetty.mortbay.org/configure.dtd">


<!--
Configure a custom context for the static site.

This context contains only a ServletHandler with a default servlet
to serve static html files and images.
-->

<Configure class="org.mortbay.jetty.handler.ContextHandler">
  <Call class="org.mortbay.log.Log" name="debug"><Arg>Configure web.xml</Arg></Call>
  <Set name="contextPath">/web</Set>
  <Set name="resourceBase"><SystemProperty name="user.home" default="."/>/web/</Set>
  <Set name="handler">
    <New class="org.mortbay.jetty.handler.ResourceHandler">
      <Set name="welcomeFiles">
        <Array type="String">
          <Item>index.html</Item>
        </Array>
      </Set>
      <Set name="cacheControl">max-age=3600,public</Set>
    </New>
  </Set>
</Configure>

Change start.sh so it looks similar to this (change variables as appropriate):
#!/bin/sh

java -DSTOP.PORT=8079 -DSTOP.KEY=stopkey -Denv=prod -Dwicket.configuration=deployment -Dlog4j.configuration=log4j.prod.properties -Duser.timezone=US/Pacific -jar start.jar etc/jetty-logging.xml etc/jetty.xml &

Change stop.sh to have the same STOP.KEY value.

Copy any files into the /web folder as necessary.

If moving a host from one KVM server to another, and the IP number should stay the same, do the following on the KVM guest:

ifconfig
sudo vi /etc/network/interfaces
sudo /etc/init.d/networking restart
ifconfig

Add something similar to the bottom of interfaces:

auto eth0:0
iface eth0:0 inet static
address xxx.xxx.xxx.220
netmask 255.255.255.0
network xxx.xxx.xxx.0
broadcast xxx.xxx.xxx.255
gateway xxx.xxx.xxx.1
# dns-* options are implemented by the resolvconf package, if installed
dns-nameservers xxx.xxx.xxx.3
dns-search domainname.com

Then, on the KVM Host, make sure to configure a firewall file. For instance, /etc/firewall.d/db1.domainname.com would contain:

# This file is processed by /etc/init/firewall.conf

# the primary IP address for this VM
IP="xxx.xxx.xxx.220"

# tcp ports that should be universally opened to the entire Internet
OPENTCPPORTS="22 80 8080 443 8443"

# udp ports that should be universally opened to the entire Internet
OPENUDPPORTS=""

# tcp ports that should be open only to hosts within this netblock segment
SEGMENTTCPPORTS=""

# udp ports that should be open only to hosts within this netblock segment
SEGMENTUDPPORTS=""

# ports that redirect to the port number + 8000. For instance, port 80 -> 8080.
REDIRECT="80 443"

# IPs and blocks that should have full access to the VE's services
# DMZS="1.2.3.0/24 5.6.7.8/32"
DMZS=""

# IPs and blocks that should be entirely blocked from the VE's services
BANNED=""

Restart the firewall:

sudo stop firewall
sudo start firewall

Friday, September 10, 2010

Server disk and raid problems

I have two KVM servers that are pretty much identical, both running RAID1. I rarely log into the host systems, but did so today because I needed to create a new host on one of them. It turns out that I couldn't do anything because the host's drive had been set to read-only mode. I checked the other system, and it too was in read-only mode!

Upon researching the problem, I found that sdb appeared to be hosed. If the server experiences a disk problem, it automatically sets itself to read-only mode. However, I didn't know this because I haven't been good at monitoring the host servers, and all of the guest servers have been running just fine. Here is a snippet from the dmesg command:

[2116582.870838] Aborting journal on device dm-0:8.
[2116582.871461] EXT4-fs error (device dm-0): ext4_journal_start_sb: Detected aborted journal
[2116582.871512] EXT4-fs (dm-0): Remounting filesystem read-only

Upon looking at /var/log/syslog, I found the last entry to be dated Jun 18 on one server and Aug 7 on the other. I need to monitor these servers better...

I also found that both systems were running on only a single drive out of the RAID1 set. The output of cat /proc/mdstat indicated only one drive one active on each system. I don't know if this was related to the problems with sdb on each system, but it needed to be dealt with.

To fix the problem, I had to reboot each server (shutdown -r now). When the server came up, it told me I needed to manually run fsck, which I did. Then I rebooted again and the server came up properly. So next, I added the missing drive back into the raid array:

mdadm --manage /dev/md0 --add /dev/sdb1

Checking /proc/mdstat showed the partitions syncing. I rebooted just to make sure it all came back up properly, which it did. One of the servers finished syncing a little more than an hour later, the other one looks like it will take closer to 2 hours.

I do this so rarely, that I had forgotten the commands to use, but an earlier blog entry helped jog my memory.