Plague Quake 2 Devastation update

I’ve started working on a stats page for Plague Devastation, my Quake 2 Devastation server, so that I can automate the collection and parsing of the XML log files produced after every match. Right now I have a basic collection mechanism that runs every twenty minutes that collects any log files produced and ups them to an S3 bucket at AWS.

My intention is to have a utility server pull the data from the S3 bucket and shove it into an Aurora MySQL cluster that can be accessed by a player who wants to see how awesome he/she is.

How to Enable SNMP on ESXi 5

There are a few steps involved in getting SNMP functional on ESXi. They go something like this.

Set the SNMP community string
Enable the SNMP service
Add necessary firewall rules
Enable the added firewall rule
Restart the SNMP daemon

SSH into the ESX host (which will require turning on the SSH daemon on the ESX host via the vSphere client), then paste the following into the terminal window after editing the field to your own SNMP community string.

esxcli system snmp set --communities <mycommunity>
esxcli system snmp set --enable true
esxcli network firewall ruleset set --ruleset-id snmp --allowed-all true
esxcli network firewall ruleset set --ruleset-id snmp --enabled true
/etc/init.d/snmpd restart

Installing Mellanox Connect-X 10g Ethernet (MT25418) on ESX 5.1

Part of building my ESX cluster in my garage was the addition of a 10 gigabit network for the storage layer. This storage layer allows all of the ESX hosts to talk to my FreeNAS iSCSI target which will host VMs and whatnot.  The basic components of this storage fabric are a Fujitsu XG2000 16-port 10gig Ethernet switch and four HP Infiniband 4X DDR Dual Port HBAs which are basically rebranded Mellanox ConnectX Infiniband adapters – model number MT25418.

I flashed the HP HBA with the following steps to turn it into a stock Mellanox MT25418 card with the following steps run from a windows 10 box. Briefly it goes something like:

Install Mellanox Firmware Tools (MFT) - WinMFT x64
Install MLNX_VPI - Mellanox Virtual Protocol Interconnect (VPI Drivers)

##get device name:  
Open Command Prompt
mst status

## Firmware to Mellanox 2.9.1
flint -d mt25418_pciconf0 -i fw-25408-2_9_1000-MHGH28-XTC_A2-A3.bin -nofs -allow_psid_change burn

Then install the drivers. The drivers are the Mellanox mlx4_en bundle v1.6.1.2 that I have copied to my cloudfront distribution here.

To install the drivers, I first had to turn on SSH on the host, which requires a few steps:
– Download an install the vSphere client locally
– launch the client and point it to the ESX host
– Under Configuration – Under Security Profile – Under Service, click properties
– click SSH, click options, click start click ok, click ok

Then copy the driver bundle to /tmp/ on the host using WinSCP.
Now launch a putty session to the esx host and run the following command:

esxcli software vib install -d /tmp/

Then reboot.

When the host comes back up, there will be new entries for the Mellanox adapter

Plague reborn in Devastation Quake ][

The last update for Plague was over fourteen years ago. My kids didn’t exist yet. The idea of my kids didn’t exist yet.

But now that I am moving out of my colo and splitting my gear between a 42U cabinet in my garage and hosting it in Amazon Web Services, it’s time to reorganize some things. In that light I have started up the Quake 2 Devastation server on a virtual machine in my garage and put up the old site to talk about it. It’s very, very out of date, but it’s kinda fun to see the origins of this server circa 2003.

I am also working on a statistics page that processes the Devastation server game logs. So far it’s a fairly simple PHP setup that pushes the raw XML data generated by Q2Dev into an Aurora MySQL database. Where it gets tricky is writing the SQL queries and PHP code that parses the data and generates meaningful and interesting match information.

Stay tuned!

Create an Ubuntu AMI on Amazon Web Services with AWS utilities baked in.

I am in the middle of migrating my site out of my datacenter into Amazon Web Services and as a part of that I decided to build a WordPress site using the AWS reference architecture and best practices (more about that in a later post). I also plan on incorporating Linux more into my architecture as an excuse to finally dig into this OS that I’ve been avoiding for the twenty-plus years of my career as a “Windows guy”.

I’ll be using both the Amazon Linux flavor and Ubuntu 16 LTS as my web servers that will drive,, and but Ubuntu does not come with AWS utilities built in like Amazon Linux does, so I plan on baking my own Amazon Machine Image (AMI) to use as my baseline. Here are the steps I performed to build my base image:

I started off by increasing the SSH timeout because I’m lazy and hate logging into a server after a timeout:

sudo nano /etc/ssh/sshd_config

and add the following two lines at the end:

ClientAliveInterval 120
ClientAliveCountMax 720

Then restart the sshd service or reboot.

sudo /etc/init.d/ssh restart

When the server has come back up and/or you have logged back in, update the base image with all the latest Ubuntu patches and fixes.

sudo apt dist-upgrade -y
sudo apt update -y 
sudo apt upgrade -y

Since the primary function of these servers will be as a part of an autoscaling group referencing a common file area, I want the Ubuntu NFS client installed by default.

sudo apt install nfs-common -y

I’ve missed this step before so I like to make sure that I verify that the nfs-common package is installed

sudo dpkg -l |grep -i nfs-common

Next install binutils, the GNU Binary Utilities. They are a set of programming tools for creating and managing binary programs and we’ll need them for compiling the EFS utilities below.

sudo apt install binutils

Next install stunnel. Stunnel is a SSL encryption wrapper for network services and is a requirement for accessing encrypted EFS instances.

sudo apt update
sudo apt install stunnel4

Now we get to download, compile and install AWS EFS utilities.

git clone
cd efs-utils
sudo ./
sudo apt-get -y install ./build/amazon-efs-utils*deb

Lastly we install python and pip, which is used to install python packages. Since the AWS CLI is Python based, we need it.

sudo apt update
sudo apt install -y python python-pip

Then install the AWS Command Line Interface

pip install awscli --upgrade --user

And since we are here and have probably received a prompt, update PIP to the latest version.

pip install --upgrade pip

Presto! We are all set to shut down this instance and bake it into an AMI.

ZFS reading material

Comparing ZFS, traditional RAID and mirroring:

The great thing about ZFS for home servers is that it gives you the power of RAID without the need to pay for expensive RAID-cards. In fact, the RAID options offered by ZFS rival those of really big, powerful and expensive enterprise disk systems! And all that with cheap, consumer-grade disks.

Ten Ways To Easily Improve Oracle Solaris ZFS Filesystem Performance:

ZFS can be a very fast file system. ZFS is mostly self-tuning and the inherent nature of the algorithms behind ZFS help you reach better performance than most RAID-controllers and RAID-boxes – but without the expensive “controller” part.

Frequently Asked Questions About Flash Memory (SSDs) and ZFS:

Here’s a collection of frequently asked questions about flash memory (also known as solid state disks, or SSDs) and ZFS, with answers and some useful links, and an index, too.

And lastly:

ZFS Best Practices Guide

Memory Pressure in SQL Server 2012

I have been working on a SQL Server 2012 database project using Folding@home data pulled from here every hour. As a part of that project I’ve been working on performance tuning my queries to be responsive enough to run from a public web page dynamically and I’ve been digging into memory optimizations and the concept of memory pressure- simply the amount of work the physical RAM in your server is having to do and whether or not it is a bottlenexk in the optimal performance of your application. Here are some quick cliff notes about some memory metrics to watch:

Total Server Memory / Target Server Memory Ratio
This measurement should be at around 100% once the steady state is reached.

Page Life Expectancy
Several authorities on the subject, including SQL Server’s Customer Advisory Team (CAT) [3], indicate that the value for this counter should be at least 300 for OLTP applications. SQL Server’s CAT also indicates that values for this counter should never quickly drop by 50% or more.

Buffer Cache hit ratio
OLTP applications it should equal or exceed 98%

Page reads / sec, Page writes / sec and Lazy writes / sec Measurements
If Lazy writes/sec is consistently experiencing non-zero values with a low PLE and elevated values for Page reads/sec and Page writes/sec the server is experiencing buffer pool contention and you will need to go about troubleshooting this problem further.

Memory Grants Pending
SQL Server Customer Advisory Team recommends that the value of this measurement should always be less or equal to 1.

Clipped from:

Re-synchronizing Windows passwords with the “Stored User Names and Passwords” tool

Recently I was having a problem on a Windows 7 PC where Outlook 2013 would randomly ask me for a password to connect to an Exchange mailbox. No matter what I did, I couldn’t get it to remember the new password and ticking the checkbox to remember the password did nothing.

I used this command:

%windir%\system32\rundll32.exe keymgr.dll, KRShowKeyMgr

from a command prompt to open the “stored user names and Passwords” tool where I could edit the credential directly.


windows server clustering

I was experimenting with windows server clustering On Windows Server 2003 and snapped these photos of my lab.



My intention was to build a tiered web architecture using multiple load balanced web servers for the app and web layer and a clustered SQL Server setup for the back end. And additional file server would host static content for the web servers.

My lab consists of five IBM eServer x336 1U boxes with dual dual-core Xeon processors and 8GB of RAM each. They are connected via 4gb Fiber channel to a HP MSA1000 SAN.


My desk with two PCs, three monitors and a subwoofer. The cabinet with the lab is on the right.


Top of the cabinet from the rear.


The bottom of the cabinet. The fiber connections are barely visible in orange.


Side shot. There’s a HP desktop in there acting as a domain controller.


Another shot of my work area.











Project: Home made 21U rack

I finally got around to building a cabinet for all of my computer gear. I have a lot of PCs for various reasons and decided to get rackable cases for them and stack them all up to keep things organized and accessible. Plus racks are seksi. I like woodworking, and am developing my skills there so I decided to build a basic rack from wood, thereby making it cool looking and keeping the cost down. 21U prebuilt racks are expensive.

I’d already built the rack when I decided to take pics for a project post. Here’s the back of the rack already built without the top after a first base coat of stain.

The back after the second and final coat of stain:


The wheels on the rack go ’round and ’round:


The rack rails, purchased from $63 for 21U:

The cabinet with the rails added:

With the top shelf added:


Finished, baby. Yeah. I used a 2×6 on the front so that the cabinet would extend past the fronts of the racked equipment enough that I can add a door to the cabinet should I chose:



Racking my computers:

More gear added:

All cabled up:

My workspace. Time to repaint the floor. Concrete paint sucks…

Some time I’ll probably add a door and sides, but for now I’m keeping it open because I like looking at the gear inside.
The total cost of this project was around $100 for the 21U rack rail kit and the lumber.