Install pair of KVM servers for Web and DB

This post is for my personal reference. I'm using custom scripts to manage the installation of KVM servers, so the information in here will probably not be useful to anyone else.

Examine the files in /root/vmbuilder/appliances/mysql to see if anything needs updating. For instance, vmbuilder.partition may need to be adjusted. Also examine /root/vmbuilder/appliance/jetty files. If changes are significant, then copy the appliance folder and create a new appliance folder. For instance:

cd /root/vmbuilder/appliances
cp -a mysql mysql-production

Then, inside /root/vmbuilder, and as the root user:

./run_vmbuilder.sh -a mysql -h db1 -i xxx.xxx.xxx.219 -d domainname.com -uusername -n "Real User Name"
./run_vmbuilder.sh -a jetty -h web1 -i xxx.xxx.xxx.220 -d domainname.com -uusername -n "Real User Name"

At the end of each command, output similar to this will be shown:

Your server login details:

Hostname: db1.domainname.com
IP Number: xxx.xxx.xxx.219
Username: username
Temporary Password: vr545Lz4

Next, log into each system with the password provided:

ssh [email protected]

Upon logging in, I will be prompted to change your password.

Then, from my local system, install ssh key to each server, entering my new password when prompted:

ssh-copy-id [email protected]

Now, log into the new MySQL server and run the following commands. Use KeepPassX to generate a strong password for mysql root, and store login information in KeyPassX for future reference.

date
more /etc/timezone
sudo dpkg-reconfigure tzdata
sudo dpkg-reconfigure mysql-server-5.1
mysql -uroot -p mysql

Create a new database with the following commands:

show databases;
create database mydbname;
grant all privileges on mydbname.* to dbuser@localhost identified by 'mypassword';
grant all privileges on mydbname.* to dbuser@"%" identified by 'mypassword';
flush privileges;
exit;

On the Web server, run these commands:

date
more /etc/timezone
sudo dpkg-reconfigure tzdata
mkdir bin
mkdir web
mkdir -p backups/database
mkdir -p backups/logs
mkdir -p backups/webapps
mkdir -p backups/jcr
cd bin
vi editdb
vi savedb
chmod 755 *
crontab -e

Insert the following into editdb:

#!/bin/sh

##########################
# Configuration settings #
##########################

# Database server
SERVER=db1.domainname.com

# Database username
USER=dbuser

# Password to access database
PASS=mypassword

# Name of the mysql database
DBNAME=mydbname

mysql -u$USER -p$PASS -h$SERVER $DBNAME

Insert the following into savedb:

#!/bin/sh

##########################
# Configuration settings #
##########################

# Database server
SERVER=db1.domainname.com

# Database username
USER=dbuser

# Password to access database
PASS=mypassword

# Name of the mysql database
DBNAME=mydbname

# Folder to store backups
LOCATION=$HOME/backups/database

NOW=`date +%Y%m%d-%H%M%S`

mysqldump -u$DBUSER -p$DBPASS -h$DBSERVER $DBNAME > $LOCATION/$DBNAME.$NOW.sql

Set the following crontab:

# m h dom mon dow command
29 */12 * * * /home/username/bin/savedb
35 0 * * * /home/username/bin/savejcr

Now, lets get Jetty configured properly:

cd ~/jetty/contexts
mv javadoc.xml javadoc.xml.bak
mv test-jndi.xml test-jndi.xml.bak
mv test.xml test.xml.bak
vi web.xml
cd ..
mv webapps webapps.orig
mkdir webapps
vi start.sh
vi stop.sh
cd webapps
scp otherhost:/path/to/war/myfile.war ROOT.war

Insert the following into web.xml:

<?xml version="1.0"  encoding="ISO-8859-1"?>
<!DOCTYPE Configure PUBLIC "-//Mort Bay Consulting//DTD Configure//EN" "http://jetty.mortbay.org/configure.dtd">


<!--
Configure a custom context for the static site.

This context contains only a ServletHandler with a default servlet
to serve static html files and images.
-->

<Configure class="org.mortbay.jetty.handler.ContextHandler">
  <Call class="org.mortbay.log.Log" name="debug"><Arg>Configure web.xml</Arg></Call>
  <Set name="contextPath">/web</Set>
  <Set name="resourceBase"><SystemProperty name="user.home" default="."/>/web/</Set>
  <Set name="handler">
    <New class="org.mortbay.jetty.handler.ResourceHandler">
      <Set name="welcomeFiles">
        <Array type="String">
          <Item>index.html</Item>
        </Array>
      </Set>
      <Set name="cacheControl">max-age=3600,public</Set>
    </New>
  </Set>
</Configure>

Change start.sh so it looks similar to this (change variables as appropriate):

#!/bin/sh

java -DSTOP.PORT=8079 -DSTOP.KEY=stopkey -Denv=prod -Dwicket.configuration=deployment -Dlog4j.configuration=log4j.prod.properties -Duser.timezone=US/Pacific -jar start.jar etc/jetty-logging.xml etc/jetty.xml &

Change stop.sh to have the same STOP.KEY value.

Copy any files into the /web folder as necessary.

If moving a host from one KVM server to another, and the IP number should stay the same, do the following on the KVM guest:

ifconfig
sudo vi /etc/network/interfaces
sudo /etc/init.d/networking restart
ifconfig

Add something similar to the bottom of interfaces:

auto eth0:0
iface eth0:0 inet static
address xxx.xxx.xxx.220
netmask 255.255.255.0
network xxx.xxx.xxx.0
broadcast xxx.xxx.xxx.255
gateway xxx.xxx.xxx.1
# dns-* options are implemented by the resolvconf package, if installed
dns-nameservers xxx.xxx.xxx.3
dns-search domainname.com

Then, on the KVM Host, make sure to configure a firewall file. For instance, /etc/firewall.d/db1.domainname.com would contain:

# This file is processed by /etc/init/firewall.conf

# the primary IP address for this VM
IP="xxx.xxx.xxx.220"

# tcp ports that should be universally opened to the entire Internet
OPENTCPPORTS="22 80 8080 443 8443"

# udp ports that should be universally opened to the entire Internet
OPENUDPPORTS=""

# tcp ports that should be open only to hosts within this netblock segment
SEGMENTTCPPORTS=""

# udp ports that should be open only to hosts within this netblock segment
SEGMENTUDPPORTS=""

# ports that redirect to the port number + 8000. For instance, port 80 -> 8080.
REDIRECT="80 443"

# IPs and blocks that should have full access to the VE's services
# DMZS="1.2.3.0/24 5.6.7.8/32"
DMZS=""

# IPs and blocks that should be entirely blocked from the VE's services
BANNED=""

Restart the firewall:

sudo stop firewall
sudo start firewall

Server disk and raid problems

I have two KVM servers that are pretty much identical, both running RAID1. I rarely log into the host systems, but did so today because I needed to create a new host on one of them. It turns out that I couldn't do anything because the host's drive had been set to read-only mode. I checked the other system, and it too was in read-only mode!

Upon researching the problem, I found that sdb appeared to be hosed. If the server experiences a disk problem, it automatically sets itself to read-only mode. However, I didn't know this because I haven't been good at monitoring the host servers, and all of the guest servers have been running just fine. Here is a snippet from the dmesg command:

[2116582.870838] Aborting journal on device dm-0:8.
[2116582.871461] EXT4-fs error (device dm-0): ext4_journal_start_sb: Detected aborted journal
[2116582.871512] EXT4-fs (dm-0): Remounting filesystem read-only

Upon looking at /var/log/syslog, I found the last entry to be dated Jun 18 on one server and Aug 7 on the other. I need to monitor these servers better...

I also found that both systems were running on only a single drive out of the RAID1 set. The output of cat /proc/mdstat indicated only one drive one active on each system. I don't know if this was related to the problems with sdb on each system, but it needed to be dealt with.

To fix the problem, I had to reboot each server (shutdown -r now). When the server came up, it told me I needed to manually run fsck, which I did. Then I rebooted again and the server came up properly. So next, I added the missing drive back into the raid array:

mdadm --manage /dev/md0 --add /dev/sdb1

Checking /proc/mdstat showed the partitions syncing. I rebooted just to make sure it all came back up properly, which it did. One of the servers finished syncing a little more than an hour later, the other one looks like it will take closer to 2 hours.

I do this so rarely, that I had forgotten the commands to use, but an earlier blog entry helped jog my memory.

Using string-based property accessors in javascript

When it comes to javascript development, I rely heavily on using plain javascript objects for data, often loading JSON data from an async server call. Below is some typical JSON that I might load after it has been processed into a plain javascript object. Note that the loaded data contains an array of users along with extra data is used for pagination purposes (count, page, and max).



When using this data, I typically directly access the properties in these objects. I'm not sure if there is a recognized name for doing this, but I'm going to call it direct property access. And I'd say that in many cases, this is the best solution for the job. Here's an example of direct property access:



However, as the complexity of my applications have increased, I've found many situations where using direct property access becomes cumbersome. For these situations, I've been exploring the use of what I'll call string-based property access. I'm finding that there are real use cases where doing it makes sense.

Use Case #1

I need to find a deeply nested property value in an object. But there is a chance that some of the properties in the path could be undefined. I want simple code to access this property if it exists, and not some large chunk of code with many tests along the way for undefined values. And I want to specify a default value if anything in the path is undefined. Compare this:



To this (and then imagine something many levels deep):



Using string-based property access seems to be the more elegant solution, especially when you are dealing with many levels in depth. It avoids littering your code with lots of excess code. In my situation, I'd rather have my overall JS download size be smaller and put a little extra processing onto the client.

I found a very basic resolve implementation but am not currently using it. Note that it doesn't handle anything besides dot separators (no arrays, etc.), and it doesn't have the 3rd default value as an option.

There is also a large and flame-filled discussion about the best way to do string-based property access on comp.lang.javascript. This conversation gets into handling arrays, array-like property access, and so forth. But for many scenarios, handling just simple dot notation might be good enough.

Use Case #2

I want to localize my application, so I've defined a JS object as a lookup table of strings. I want to insert values from my database into these strings, but might put them in different locations depending upon the language. Here's an example lookup table:



Take a look at my StackOverflow question for some ideas on how to solve this. Again, the most elegant solution seems to be using string-based property access.

Use Case #3

I want the custom UI components in my application to all access data in the same manner. Each component would have a wrapper object around my actual data object (which I call a Model). Each component would know how to get data from a Model, and the Model would know how to grab my data (from an internal JS object, an HTML5 store, from a remote JSON call, etc.). When my components are created, the data doesn't necessarily have to exist first, and could be loaded as a different part of the lifecycle.

This is the way that the Apache Wicket framework does things, and I believe that it is the right solution for the problem. Even though it is for Java, there are some very good reasons to look to it for ideas and inspiration. Wicket's PropertyModel (extends Model) object uses string based property accessors very effectively.

Wicket uses Models and PropertyModels as wrappers around your POJOs. This enables the framework to have a consistent interface when dealing with data objects, as it simply needs to know how to work with a Model instead of each specific POJO. It also provides a means to detach POJOs from the models and then reattach as necessary. The solution works really well, but relies on string-based property accessors.

In addition, Wicket's CompoundPropertyModel concept could be utilized in a javascript application. That would really help simplify my component/client code and offload the data access into reusable objects. Even if you aren't a Java programmer or hate Java, many concepts introduced by Wicket are worth considering, especially the component-based nature of the framework and the way it wraps Models around data.

Conclusion

At this time, I don't have fully implemented string-based property access method in place yet. I'm currently using a combination of methods to see what works best. I'm leaning toward going with a simpler solution that doesn't catch all possible strings and just handles dot notation in the strings. But before I commit to this, I'm going to play with it some more to see if anything comes up where I might need a more robust solution.

If anyone has thoughts on any of this, or links to other resources I should look at, I'm all ears!

My latest electronics purchase

I just purchased a Pioneer VSX-1020-K Audio/Video Receiver. From what I've read online, it looks like it will totally meet the needs of my home theater. I will update this post once I have it set up and review it.

Problem with google-checkout-java-sdk and JDK1.6

I was using google-checkout-java-sdk in a custom online shopping solution. When I switched the project's POM from java 1.5 to 1.6, it seems that google checkouts would no longer work. I was getting this error when attempting to send an order to Google Checkout:



The root of this problem was difficult to find, but with some googling it sounds like it has something to do with changes in Sun's rt.jar. I experimented with removing xerces dependencies or using different versions of it (2.0.2, 2.9.1). That didn't work. So then I added xalan-2.7.1 to my POM, and things started to work again!



I haven't really gotten to the root of this yet, but at least I can make my client happy again. Would love to hear from anyone who knows more about this.

Blog template updated, includes syntax highlighter

I just updated my blog template to something that was wider, that's why it looks different now. I needed it wider to better display code snippets now that I added SyntaxHighlighter. I installed it by following Carter Cole's howto.

Using Wicket with Shiro for authentication and authorization

In my Wicket application, I'm using Apache Shiro for both authentication and authorization. I needed powerful authorization features that would have taken a lot more work with Wicket alone. Personally, unless you have dead-simple requirements, I'd suggest learning how to do it with shiro and saving yourself some hassles in the future.

To simplify integrating wicket with shiro, I created the wicket-shiro project hosted at wicketstuff. Note that although I created the project, it is pretty much just a port of someone else's wicket-jsecurity implementation. I didn't really write much code for it, I just took the time to port it from jsecurity to shiro 1.0, clean up some things, and add a wicket-shiro-spring-hibernate example. I also wrote up some information about it here:

http://wicketstuff.org/confluence/display/STUFFWIKI/wicket-shiro

The wicket-shiro project is currently commented out of the wicketstuff pom because of the lack of shiro being available via maven, but hopefully I'll update the project with real shiro maven resources soon. However, you can still check it out and use it, just mvn install it into your local mvn repo.

So here's a quick tutorial on setting up a user's permissions from data stored in your business objects. I'm using a service layer with a Hibernate data store. A Member contains a username and password and a reference to a Person object. A Person contains a set of ProjectParticipations -- all of the projects that this member works with. I need a member to only be able to perform actions on projects they are participating in, but to see a list of all projects and join those projects if desired. Also, if a member creates a project or a project creator gives them administrative permissions, they should have access to additional management features for that project.

To do this, I extend AuthorizingRealm and injected my MemberService into it. This service is used to build the permissions for a user. Here's a simplified version of mine -- it's certainly not perfect and major improvements are planned, but it should get you started:

Then, in a wicket page, I have this:

If the member manages a project, then he has the "projectmgr" role which has the "project:viewControls" permission. So the above will only show a menu of project management options to members who have that particular permission. If the member doesn't manage any projects, then they won't see that menu option.

You can also dynamically change permissions during runtime. For instance, if a member who doesn't manage a project creates a new project, then they should immediately get the viewControls permission so they can access the management features. You can force this refresh with this:

Hopefully this gets you started using Wicket with Shiro for authorization!

Tauren

Repairing a corrupt eclipse workspace

I've had Eclipse crash on me many times in the past and left things in a state where I couldn't get the workspace to start back up properly. Eclipse would lock-up when restarting. I have always fixed it by simply creating a new workspace and then rechecking out my projects from SVN. But that is a tedious process, and can be a real hassle.

So this time I decided to find a better way. Victor Igumnov provided the solution in his blog. Thanks Victor!

This solved my problem for the most part, however, if you are using a multi-module maven project with eclipse and the m2eclipse plugin, a simple project import didn't get things set back up as they were.

Before eclipse crashed, I had checked out a maven project with subversive using the "Check out as Maven project" feature. When you do this, root-level eclipse projects are created for all maven modules and submodules that are being checked out.

After the crash, doing an "Import -> Existing projects into Workspace" would only create a root-level eclipse project for the parent maven project. Instead, to get back root-level eclipse projects for all maven submodules, you need to do an "Import -> Maven Projects".

After doing that, my workspace was back up and running as it had been before.

---

UPDATE: 2/4/2010

An alternative method that I discovered might work as well, but I still ran into some problems and had to resort to the method above. If it had worked, it would have enabled me to get back up and running even faster. I had problems even with the first method, so I'll try this method again next time this problem happens and see if I have any better luck.

A MyEclipse forum posting gave me the clue for this alternative method. Simply rename a directory called .projects and restart eclipe. Then, once it has started, shut it down, delete the newly created .projects folder and rename the original back to .projects. When Eclipse is started up again, everything seems to work properly. The .projects directory is located at:

%WORKSPACE%/.metadata/.plugins/org.eclipse.core.resources/.projects

When I tried this, Eclipse started up without problems. However, one of my projects was missing its pom.xml, which made it so that none of the source folders were showing up. All I did was right-click on the project and do a "Refresh" (or F5). It then attempted to rebuild the project. Unfortunately, it would finish the build and seemed to keep repeating going from 67% to 70% and then back again.

In the end, I restarted Eclipse, but then it froze on me again during startup. So I resorted to the first solution at the top of this posting. But even after doing it, the build would repeat over and over again. Eclipse had crashed when I was doing some refactoring, and my code was in an invalid state. Once I corrected it, then the build completed properly. So it is possible that this alternative method would have worked after all.