DCM4CHEE Log file analysis with Chainsaw

Posted by Martin P in DCM4CHEE      No comments

Jul
25

DCM4CHEE is quite a complex beast, consisting of around 70 discrete services – on top of the many services delivered as part of the JBOSS application server DCM4CHEE relies upon.  Many of those services offer multi-threaded services – so even on a server with a modest workload, there can be periods when a lot is going on – all of which is being logged to the central log file.

The log file is the first recourse when problems occur – or even if something happens that is simply unexpected.  It can, however, be quite a challenge to untangle the log messages originating from different services or threads, and that is made even worse if there are stack dumps throughout the file.  There is a way to make troubleshooting a little easier, thankfully.

Apache Chainsaw is a Java GUI application that accepts input from the log4j logging framework that DCM4CHEE uses, and offers a couple of features which make the log output easier to manage:

• A single entry for each event.  While in the standard log file, events may run to several lines (and stack dumps run to several pages), Chainsaw displays each event as a single line with drill-down to more details if appropriate.
• Potentially sophisticated filtering abilities which allow (for example), display of events originating from a specific thread.

Installation

Installation couldn’t be easier – given an appropriate Java runtime is in place.  On my Ubuntu laptop, unzipping the ‘standalone’ package and running chainsaw.sh was all that was needed.  There is a chainsaw.bat for Windows users and a .dmg easy-to-install package for Mac users.

Log File Configuration

The default configuration that comes with DCM4CHEE doesn’t work with Chainsaw, though.  We need to tweak it just a little.  In the DCM4CHEE configuration file server/default/conf/jboss-log4j.xml, below the lines

<appender name=”FILE” class=”org.jboss.logging.appender.RollingFileAppender”>
……
……
</appender>

.. add in the following lines:

<appender name=”XML”  class=”org.jboss.logging.appender.RollingFileAppender”>
<errorHandler class=”org.jboss.logging.util.OnlyOnceErrorHandler”/>
<param name=”File” value=”${jboss.server.log.dir}/server.xml”/>
<param name=”Append” value=”false”/>
<param name=”MaxFileSize” value=”10000KB”/>
<param name=”MaxBackupIndex” value=”1″/>

<layout class=”org.apache.log4j.xml.XMLLayout”/>

</appender>

.. which will result in a second log file being produced.  The normal ‘server.log’ will be untouched, but you will also have a server.xml which can be opened in Chainsaw.  The XML formatting is less efficient than the standard formatting – so you won’t get as much in the way of time coverage in the same 10MB space but that can be adjusted if necessary.

Once in Chainsaw, the events can be filtered using the ‘Refine focus’ field by adding in rules against which any of the event fields can be tested.  Even easier, any of the event fields displayed in the main window has a right-click context menu, which includes ‘Add to refine focus field’.  Picking this re-filters the display accoring to the value in that field (e.g. a specific Thread ID):

A new book for MySQL, and a toolkit for MySQL AND PostgreSQL

Posted by Martin P in DCM4CHEE, High Availability      No comments

Jul
14

So I came across a new book on MySQL replication, which looks pretty useful although a little expensive. The author biographies are particularly impressive – these are the guys who really should know about replication.  One of those is the architect of MySQL’s row-based replication which set me thinking.

A while back I was implementing replication on a couple of DCM4CHEE servers and noticed that the master and slave became increasingly out of sync.  At the time, the only option in MySQL was statement-based replication and I attributed (with no real evidence) the issue to known limitations of that form of replication.  In any case – the alternative (DRBD) turned out to be quite adequate and that is the strategy I’ve used since.

However, more recently (November 2008), MySQL 5.1 included row-based replication and I never went back to review the strategy.  Seeing this new book prompted me to spend a little time checking what the state of MySQL row-based replication is.

To cut a long story short, I ended up discovering Maatkit – “power tools for open source databases”, which, while written primarily with MySQL in mind, supports other open source databases (like PostgreSQL for example) and is certainly going to join the rest of my arsenal.  The first use will be mk-table-checksum – designed to allow comparison of master and slave to ensure replication consistency.

I’ll try out the row-level locking and will a new toolset, can be comfortable that it works the way it should (or indeed, not).

BTW the O’Reilly site has a sample chapter which describes nicely the process of setting up simple replication – worth a look if you haven’t tried it before.