In my previous post on Different JVM settings for different Tomcat instances, I briefly touched on using tomcat-users.xml to allow access to the the Server Status page. This page, along with the Tomcat Manager application (amongst others), are installed by default when you deploy Tomcat. Access to them is controlled via the tomcat-users.xml file.

As we saw in the previous article, we can enable access by defining a role, and subsequently a user who is part of that role.

Why doesn't Tomcat ship with a default account?: Simple. It's a security issue. Instead of providing that default account, Tomcat is locked down completely, and it's up to us to enable access.

Let's try accessing the Tomcat Manager without having any users defined in tomcat-users.xml.

As you can see, access is restricted. However, what it does do is tell us (some of) the types of roles available to us.

Let's look at our tomcat-users.xml file again. Now, based on what you see in the file, and what we added in the previous article, I'm hoping most (if not all) of you are a little concerned by the fact passwords are stored in plain text.

Thankfully this file isn't directly accessible from a web browser, however it can be read by anyone who has access to the server.

Realms

Tomcat Realms are simply a way of "storing" usernames, passwords and roles. The default out of the box realm used is "UserDatabaseRealm". This particular realm reads clear text passwords out of tomcat-users.xml

We're going to look at a couple of alternative realms, namely: MemoryRealm and JDBCRealm. There are other realms available to you, and if your interested, you can read up on them at: http://tomcat.apache.org/tomcat-7.0-doc/realm-howto.html

MemoryRealm

What we're going to do is change the realm to "MemoryRealm", which whilst still reading the user details out of tomcat-users.xml, will now do so using a specified encrypted format: SHA, MD2 or MD5.

The algorithm must be supported by the java.security.MessageDigest class

Open up CATALINA_HOME/instance/conf/server.xml

Find "UserDatabaseRealm", as below, and comment it out.

Next, add the following Realm inside the Host block, which is just below the section you commented out.

Save the file, and exit.

We now need to create a SHA encrypted version of the password you want to use. Tomcat is useful here as it provides a script to do just that. Inside CATALINA_HOME/bin, run the following:

As a proper example:

You now want to replace your password (in tomcat-users.xml) with the encrypted version.

Now restart Tomcat and you should be able to login.

JDBCRealm

To configure the JDBCRealm, we need to do a couple of things first:

1. Create tables and columns in your database of choice (we're going to use MySQL in this scenario)
2. Configure a 'tomcat' user to allow Tomcat to talk to MySQL
3. Drop the MySQL connector JAR file into place
4. Set up our Realm

Use the following scripts to create your tables:

(I've created a dedicated database called TomcatAuthentication)

We need to populate our tables with data, so now run the following:

We now need to grant a 'tomcat' user read access to these tables, so run the following from your favourite MySQL client tool:

The next thing we need to do is get a hold of the MySQL JDBC driver, which we can download from: http://dev.mysql.com/downloads/connector/j/

Download the file, extract the contents, and then copy the JAR file (in my case mysql-connector-java-5.1.20-bin.jar) to CATALINA_HOME/lib

Now we want to add a Realm to our servers.xml file to support JDBC.

Add the following to our definition (replacing the MemoryRealm):

I've already replace the appropriate entries above to those we're using in this article.

Now you can restart Tomcat, and when you try to access the Tomcat Manager or Service Status page, authentication will now take place against MySQL.

You can obviously harden your password further by using SSL, and locking down access to internal IPs or even just localhost, and we'll look at this in a future article.

SSL certificates expire and need renewed regularly, which involves at least replacing the certificate file and possibly the key and chain certificate files too, running "apachectl configtest" to make sure everything is properly in place and then restarting. Quick, simple, and therefore quite surprising when Apache dies immediately afterwards.

SSL certificates and keys need to be matched - the key must be the one used to generate the signing request for the certificate. As wonderful as it otherwise is, "apachectl configtest" does not test for this! If for whatever reason they don't match then Apache can't handle SSL requests and will refuse to start and you'll see this in the error logs :

[Fri Feb 17 21:58:56 2012] [error] Unable to configure RSA server private key
[Fri Feb 17 21:58:56 2012] [error] SSL Library Error: 185073780 error:0B080074:x509 
certificate routines:X509_check_private_key:key values mismatch

This shouldn't happen if you have been careful with your key, request and certificate generation, but sometimes life isn't straightforward - security considerations can mean it's impossible to request SSL certificates on behalf of customers. Non-technical customers can sometimes get mixed up during the process and use the wrong keys and CSRs which means you end up with a mismatch.

Now you have to track down the correct pair of key and certificate files - and quickly, as at this point Apache is down and will not come back up until you fix it or disable SSL. Luckily the OpenSSL command line tool can help.

The first step is to find the correct certificate file. OpenSSL will tell you everything you need to know to track down the right one. For each of your possible certificate files you can run the following, replacing MYDOMAIN.crt with your certificate file :

openssl x509 -noout -text -in MYDOMAIN.crt

This will print a lot of detailed information, of which we are only interested in the first few lines :

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 198312 (0x306a8)
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=GB, O=Fuzzy Orange, CN=Quick Example SSL CA
        Validity
            Not Before: Nov  8 12:32:56 2011 GMT
            Not After : Aug 27 04:55:46 2012 GMT
        Subject: serialNumber=1pfX346-sw4eFZmll0aBd3dWkwKKjLDx, C=GB, 
O=WWW.MYDOMAIN.COM, OU=FO3269373, OU=Fuzzy Orange, CN=WWW.MYDOMAIN.COM

The key elements here are the validity dates and the O and CN elements of the subject - these let you know what domain this certificate is for and the dates it can be used between. When you've found the certificate that you should be using, you then need to know it's modulus to compare with your keys. Once again OpenSSL can do this for you - simply run the following, once again replacing MYDOMAIN.crt with your certificate file :

openssl x509 -noout -modulus -in MYDOMAIN.crt

This will print a large string of characters :

Modulus=F06D8B592B348B8D6704B05496CC3E094F875E6A6C5219DA33A...

This modulus string will only match up with the correct key for this certificate, so all we need to do now is check the modulus of your possible key files. Run this to see it, replacing MYDOMAIN.key with your key file :

openssl rsa -noout -modulus -in MYDOMAIN.key

Once again you'll get a large string of characters, this time of the key's modulus. Compare it to the certificate's modulus as extracted before - when you find a match then you have found your key and certificate pair.

Now quickly copy these matching certificate and key files to the location where Apache expects them and start it up again - this time it should run as normal and not report any errors. And next time around, use OpenSSL to verify the keys and certificates match in advance!

We've been doing a fair bit of work with Tomcat for a while now. It was partly instigated by the knowledge that ColdFusion 10 was going to use Tomcat under the hood (in replace of JRun), but we've also been doing work deploying Railo as well as some Java apps (deploying as WAR files).

One of the things we're so used to is the ability to run multiple instances of ColdFusion on JRun, and we wanted to do the same with Tomcat. As such, this entry is a generic walkthrough to get up and running with multiple instances of Tomcat.

The first thing you need to do is get a hold of Tomcat. Go grab the most recent version from http://tomcat.apache.org (v7.0.27 as of writing).

We like to install Tomcat versions inside a master ApacheTomcat folder, e.g. /opt/ApacheTomcat/apache-tomcat-7.0.27

Once you've extracted Tomcat, take a look at the contents of the folder. You'll find some text files, and more importantly a bunch of folders:

• bin
  • contains all the binaries and scripts for running Tomcat
• conf
  • configuration files including server.xml and web.xml
• lib
  • the libraries that make Tomcat work
• logs
  • funnily enough, logs
• temp
  • used for temporary files
• webapps
  • this is where we put our apps
• work
  • folder used by Tomcat to write out files needed during runtime, including generated code for JSPs, class files, serialised sessions

An important part of any Tomcat installation is environment variables. There are 5 different variables that interact with Tomcat, two of which are mandatory:

Mandatory

• CATALINA_HOME
• JAVA_HOME

Optional

• CATALINA_BASE
• CATALINA_TMPDIR
• CLASSPATH

All three optional variables can be calculated using CATALINA_HOME

The usual way

Under a normal install, we really only care about CATALINA_HOME. By default, this variable points to the Tomcat root folder, so using our set up, it points to /opt/ApacheTomcat/apache-tomcat-7.0.27

This gives access to the /bin and /lib folders, and allows all other environment variables to be set when we run startup.sh (found in the /bin folder).

And this is the important part. Instead of allowing CATALINA_BASE to be set automatically, we want to explicitly set it. That's the key to multiple instances.

Configuring Multiple Instances

Create two new folders inside your CATALINA_HOME folder, e.g.

/opt/ApacheTomcat/apache-tomcat-7.0.27/instance1
/opt/ApacheTomcat/apache-tomcat-7.0.27/instance2

Copy the following folders (and their contents) from CATALINA_HOME, and paste them into the two directories you created above:

• /conf
• /logs
• /temp
• /webapps
• /work

(You might want to delete the contents of your copied log folders).

Next we need to configure the 3 different ports that each Tomcat instance will use. These are the connector port, the AJP port, and the shutdown port.

• Connector port
  • What Tomcat uses to communicate with the outside world. Default: 8080
• AJP port
  • If you're going to be connecting to Apache (or IIS), then you'll be looking to use AJP rather than the "basic" connector. AJP is a binary protocol and is therefore faster than the default connector. Default: 8009
• Shutdown port
  • Tomcat uses a port as part of it's shutdown process. It communicates with this port to shutdown cleanly, and therefore it has to be unique per instance. Default: 8005

Let's start by opening and editing /instance1/conf/server.xml. Find the blocks similar to below for the three connector ports

Change the ports to something like:

Do the same for the second instance, providing different port numbers again.

Let's now create some startup/shutdown scripts for our instances. Create a new file called instance1-startup.sh in CATALINA_HOME/bin

Add the following to the script:

Create another new file called instance1-shutdown.sh in CATALINA_HOME/bin, and add the following:

Do the same for instance2, being careful to change the paths.

You can now run ./instance1-startup.sh and ./instance1-shutdown.sh to start/stop your first instance, as well as your other scripts to control your other instances.

Congratulations, you're now running multiple instances of Tomcat, which can be accessed via (depending on the port numbers you chose above)
http://localhost:8181
http://localhost:8282

You can now deploy your ColdFusion and Railo WAR files directly into an instances webapps folders, e.g.
/opt/ApacheTomcat/apache-tomcat-7.0.27/instance1/webapps

Migrating your Linux server from the Rackspace US Cloud to the UK

We here at Fuzzy Orange love the Cloud. We make extensive use of cloud services, particularly at Rackspace. When we started running cloud servers at Rackspace they only offered this service in the US but now it's offered in the UK too and we're migrating our servers over.

Rackspace have provided a detailed set of instructions which do the job well, but are maybe not the best at explaining what's going on. There are also one or two little "gotchas" not mentioned there.

Now, the first thing that should be noted is that depending on the complexity of the setup on your US cloud server it may be easier to just move files by hand. For example, if you just have a little web server with static content, it would be quicker and easier just to create a new UK cloud server and copy your content across manually.

Important things you need to know

1. US and UK usernames and API keys

The Rackspace migration scripts use the Cloud Files API to copy things around, so you will need valid usernames and API keys for both the US and UK cloud.

2. US and UK servers will run simultaneously

You need to have both servers up and running for the whole procedure, so you will be charged for both until everything is complete.

3. Python 2.6 on your UK server

Older Linux distributions ship with Python 2.4, which isn't good enough to run the Rackspace migration scripts. If you're so inclined you can build Python 2.6 from scratch, but it's easier to grab the ActiveState Python 2.6 installer and use that.

4. Files can't be changed during the process

Well, they can, but you'll have to transfer any changed files manually after the process is complete. When taking the source snapshot of your US cloud server you have to make sure no files are open for writing - this means databases either shut down or open read-only.

5. It will take about a day for this to complete

After you've created your migration image and run the Rackspace migration scripts you need to create a support ticket wait for Rackspace to do some work behind the scenes to link your US image to your UK server. It will take about a day or so for this to happen.

Step by step

1. Create the destination server in the UK cloud

Create a new, blank server in the UK cloud. Make sure it matches exactly your US server for memory, disk size and OS version.

2. Create an on-demand backup on the UK server

This is where the US image is eventually copied across to. Call this something like "MigrationImage" so you'll know what it's for.

Once it's complete, go to Cloud Files and look inside the cloudservers folder. Take note of the .yml file for your image for later (e.g. MigrationImage_20110607_123456.yml)

3. Prepare the US server

You need to prepare the server for an on-demand backup image. This means making sure no files are opened for writing, which means any databases need to be shut down or set to read-only.

4. Create an on-demand backup on the US server

This becomes the source image for your UK server. Call this something like "MigrationImageSource" so you'll know what it's from

Once you have the on-demand backup of the US server you can optionally restart your databases or make them read-write again, but understand that changes from this point on won't be copied by the migration scripts so you will have to deal with them manually afterwards.

Just like the UK image, go to Cloud Files and take note of the name of the .yml file for use later.

5. InstallPython 2.6 on your UK server

Check the version of Python you have installed by just running "python --version". If don't have Python installed at all, try to install it using your system's package manager (e.g. "yum install python" or "apt-get install python")

If you don't have at least Python 2.6.0 then you'll need to install it by hand. You can download the source and compile from scratch, but as you're probably only going to be using it this once for the migration, you might find the ActivePython from ActiveState easier to install - just remember to add the ActivePython bin directory to the beginning of $PATH once you're done.

6. Install PyYAML on your UK server

PyYAML is a Python module the migration scripts need to run. Download it from http://pyyaml.org/download/pyyaml/PyYAML-3.09.tar.gz, untar it to a folder and run "python setup.py install" to install it.

7. Download the migration script on your UK server

Get the Rackspace migration script from http://c857.r57.cf3.rackcdn.com/migrationscript_v1.0.zip and, if you like (and I'd recommend that you do so), verify the checksum with that on the Rackspace FAQ page. Unzip the file into a folder.

8. Run the migration script on your UK server

Now you need to run the command on your UK server to copy the migration image over from the US to the UK. The format of the command is :

python cpp.py -i -v US_username:US_apikey@cloudservers/source.yml UK_username:UK_apikey@cloudservers/

You need to replace US_username, US_apikey, source.yml, UK_username and UK_apikey with your details so the command ends up looking like :

python cpp.py -i -v usaccount:rapovn09202v97vouovn29v2v1@cloudservers/MigrationImage_20110607_123456.yml ukaccount:99hiefoid90ouopbv2poxcox89@cloudservers/

This will take a while to run depending on the size of the image to be migrated. It will print progress as it goes. Don't worry about getting it wrong, it handles errors gracefully and you can run it again. If you have an exceptionally large server image it may be worth running this inside Screen or some other tool that allows you to disconnect from the session without terminating your login.

9. Raise a support ticket in the UK portal

Once the migration script has completed you have to raise a support ticket in the UK cloud portal. The Rackspace FAQ has the correct wording to use in the ticket details, but additionally I would suggest you add the name of the US and UK servers.

10. Restore the migrated image to the UK server

Once you have notification from Rackspace support that they have finished migrating your image you should be able to restore the image to your UK server. The migration overwrites the on-demand UK backup you took at the start of all this with the US on-demand image, so restore using that.

Once this is done the UK server is now an identical copy of the US server at the point the on-demand image was taken there.

11. Change network settings on the UK server

Sometimes this happens automatically, but sometimes it doesn't. If you can't immediately access your UK server on the new IP address it has been assigned, you'll need to use the Cloud console to connect to it directly to change the network settings.

The Cloud portal already gives you the primary IP of your cloud server. Rackspace support assure me that for all cloud servers the netmask is always 255.255.255.0 and the default gateway is the same as the IP, but with the last octet changed to 1 - so, if your primary IP is 33.44.55.66, your default gateway is 33.44.55.1. DNS servers should be 83.138.151.80 and 83.138.151.81.

12. Test your UK server

Now you need to test the new UK server to make sure it's all working. Start any databases you had previously stopped or bring them out of read-only mode.

13. Sync up any changed content

If any content has changed on the US server since you started this process then this is the time to copy it across to the new UK server.

14. Change DNS names to point at UK server

If you have any public DNS records that refer to the US server you probably want to change them now to point at the new UK server.

15. Delete the US server

Once you are totally, 100% sure that the UK server has migrated correctly, all content has been copied and any DNS entries have been re-pointed towards it, you can delete your US server.

Remember that you are still charged even if your US server is switched off, so you need to delete it entirely. If desired, create an on-demand backup to Cloud Files before doing so you can re-create it if necessary.

And there you go - your US cloud server is now running in the UK.