LOGO

Home Sweet Home

       I thought it would be fitting that the first project I document on DigiFAIL.com would be the creation of its home, the Mainframe Mark 2. Mainframe Mk2 was designed to replace the original Mainframe, which was the first Linux computer I ever put together. It served me well as file server for the LAN, but had started to show its age. At only 400 MHz, anything much more taxing than serving files over SMB/NFS was toying with disaster. Once I decided to actually put up a world-accessible website, I knew the old girl was going to need to be retired. Hardware performance aside, I also wanted to modernize its capabilities. I wanted to push more network management responsibilities onto the one box, and at the same time address the biggest concerns I had with the original Mainframe, namely power consumption, lack of redundancy, physical size, and noise.

To this end I started the design for Mainframe Mk2 around one of the new Atom boards, as these give an excellent power consumption to performance ratio. They also lend themselves well to fan-less (or at least, less fans...) operation, which would address the noise issue. Most importantly, they are standard x86 machines, which makes them much more accessible. You can do much better in terms of energy efficiency with embedded ARM devices, but they are considerably more difficult to setup and maintain.

Hardware Design

Board        The core of the Mk2 is an Atom 230 board with 2 GB of PC2 5300 RAM. The case is a Jetway JC-300 which features a 60 Watt integrated PSU. Cooling in the stock case is decent, though I did add a Vantec heat spreader to the RAM and changed the CPU fan out for a Scythe SY124010L which is much quieter.

Because I wanted to maximize my power efficiency, I decided to keep the OS on flash rather than a traditional HDD. A proper SSD was a bit out of the budget, so I decided to go with a CF card in an IDE adapter. With modern wear leveling on the card, and some software configuration tricks, there isn't much risk of burning the card out with repetitive I/O. But really, with as cheap as flash media has gotten, blowing it up after a few years isn't much of a concern.

Redundancy was a big goal on this build, but my energy and size goals were in direct opposition of that. Even if I didn't mind spending the energy on it, the case wasn't large enough to hold two 3.5 inch drives anyway. I wasn't sure how to handle this problem at the start of the project, but decided to push ahead anyway and hope something came to me later.

Early Problems

       I was looking to minimize how many wires would be crammed into the case, so my next purchase was CF to IDE adapter that plugged directly into the motherboard from DealExtreme (SKU 2720) It worked out great in my head, but when I connected it up to the board I realized that the Vantec heat spreader was so thick that it pushed on the adapter.

It occurred to me that the problem was really that the LEDs and the jumper pins on the adapter were sticking out too far. If I removed them and moved them to the back side (at least the jumper anyway, LEDs weren't strictly necessary), the adapter would just fit. So off I went with my soldering iron and removed the three LEDs and shorted the jumper out to "Master".

Adapter

The finished product looked excellent. The jumper was small enough that it wouldn't touch the metal of the heat spreader, and aside from the loss of the LEDs, I was very happy with this fix. The voltage selection jumper could have gone too, but it was just as well to bend it up slightly, as that was farther to the side and didn't come directly in contact with the spreader.

It was actually just as well that this little problem came up now, as the original case was destroyed in transit and I was waiting on its replacement the whole time. So this gave me a nice diversion as I waited for the new one to get here. Sure, it would have been nice to do a dry run with everything in the case, but no big deal. A few days after I finished the modifications to the adapter board, the case got here. So I mounted it all up in the case and took a look at it...

Wait...what?

Little big...

Oh, I see what happened here. In my zeal to fix the adapter issue, I failed to take into account the physical dimensions of the case. So I had a nice modified little adapter which could never have possibly worked in the first place. At least it was only $3...

Second CF Adapter

       Clearly I needed to completely rethink the CF adapter issue. It would have much much easier to just give up here and install the OS to the main drive, but I wasn't giving up that easy. As I had mentioned earlier, this case has a tray where a slimline optical drive is supposed to go. Since I wasn't putting any optical drive in this machine, I had that whole area free in the case. It was just a matter of figuring out how to get a CF adapter mounted to it.

To start, I got another CF adapter from DealExtreme (SKU 10309), this time one that had the female IDE connector at a right angle to the rest of the adapter. My idea was to mount the adapter in upside down, with the IDE connector hanging over the edge of the tray. That was the only way I would have the vertical clearance to get it inside the case. But the tray was clearly not setup for this, so I had to make a few modifications.

Tray Modification

The first thing I had to do was cut the tab that would have held the side of the optical drive. I ground this down level with the rest of the tray, making sure there were no jagged bits sticking up that might short the CF adapter that would eventually sit on top of it. I then drilled a hole for the CF adapter to mount to, and epoxied a nut on the underside that I could thread a screw into. Finally, I just had to find the right diameter screw and cut it down to length. It is a little tight on the inside with the primary storage HDD in place, but nothing too bad.

Then it was simply a matter of getting the IDE ribbon short enough and folded in such a way that it wouldn't intrude too much in the case. In case you are wondering, it is very hard finding IDE cables under 6 inches in length. I was about to make my own IDE cable, until I found a very short one in an old OEM Dell. I pulled it out, and with some tight folds I managed to get it to fit in exactly the space available between the RAM and the side of the tray. I was initially a little concerned about how tight everything was, especially with how it would effect airflow, but so far there doesn't appear to be any serious problems. In the following you can see just how tight a fit everything is, with the IDE ribbon cable packed in alongside the PSU wires.

Adapter in place

You have probably noticed that the adapter is in upside-down... Obviously I would have rather it been oriented the proper way, but it just wasn't possible with how tight everything is in the case. Still, there is no problem getting the card in and out in this configuration, as there is enough space to the front of the case that you can pull the whole card out and clear of the adapter without any trouble. You may also have noticed that the adapter is only mounted with the single screw on the one side. The other side is laying on top of the edge of the tray, which makes a nice flat platform at about the same height as the mounted side. Of course it does move around slightly, but as the server will not be living a very mobile life I don't think it is that big of an issue. Plus, the adapter and card assembly is so light that there isn't much chance of it banging around in there.

Storage System

       What is a file server without storage, right? Now that I had a filesystem to install my OS on, I needed to turn my attention to exactly how I was going to mount the drives in this thing. The primary storage drive was easy enough, it goes into the bay right below the tray where the CF adapter is mounted. I used a 1 TB WD Green Drive (WD10EADS), which puts energy efficiency and low-noise over raw performance by throttling itself down during periods of inactivity, and only running at a relatively low RPM even when it is active. On a server like this, I don't need a high-revving drive to get instant access to many small files like you would want on a desktop machine, I will mainly be dealing with streaming larger files one at a time. In this scenario, the WD Green Drive excels. I found some 6 inch SATA cables on NewEgg.com which were the exact size needed to plug the drive into the board with no slack, so everything went in fine there.

But as I mentioned earlier, there is no place for a second HDD in this case. I was ready to either get an external drive to do the backups on, or risk another few years without them, when I noticed the open area next to the CF adapter. It wasn't large enough to fit another 3.5 drive, but I wondered if a laptop HDD wouldn't fit in there. Pulling out a spare SATA laptop drive I had, I found that the area next to the CF adapter was almost exactly the same size as a 2.5 laptop HDD. So I went back onto NewEgg and ordered a 320 GB Hitachi Travelstar, another of the 6 inch SATA cables, and a 4 pin molex to SATA power adapter, as the case's PSU only had one SATA power plug.

Like last time, this was going to involve some modification to the optical drive tray, which was increasingly less and less designed for the things I was trying to do with it. I had to cut the holes on one end of the tray into an elongated slit so I could adjust the drive back and forth, and the other end needed a new hole drilled. Basically, the existing hole pattern was completely wrong for mounting a HDD to it.

Mounting holes

Once the new holes were cut into the tray, the rest was easy. I just had to tuck in the molex to SATA power adapter, and connect the short SATA cable to the motherboard. Like last time I was only able to mount the drive on one side, but again, given how little this machine will ever move and how light the components are, I am not too worried about it. I should also mention at this point that the server is oriented such that it is laying on its side, meaning gravity will pull both the CF adapter and the laptop HDD down onto their mounting points, rather than towards the surface of the tray.

In the end, all three storage devices for Mainframe Mk2 are mounted to the same piece of metal. A nice little package if I do say so myself. It all came together exceptionally well, considering that I had absolutely no plan on how to do this when I bought the case. Here you can see the CF adapter, WD Green Drive, and Hitachi laptop drive all mounted in their final positions.

Completed Hardware

Software Additions

       The server runs Slackware 13.0, which was released a few days before the build was complete. I tried to keep the install as close to stock as possible in terms of installed software so I could keep it synced up with security updates without having to rebuild too many packages manually. I found that on the older server I ended up rebuilding most of the packages from source as the install had drifted so far from stock that nothing worked without a rebuild anymore. Still, there were a few things I wanted to change.

The main software additions were lighttpd and DenyHosts. I didn't use Apache as my situation didn't require all of the features and capabilities that Apache offers, and frankly, it would simply have consumed more resources than necessary for a simple site like this.

DenyHosts is a nice little script that parses the SSHd logs and adds IPs to /etc/hosts.deny if they attempt to brute force the SSH server. Anyone who has ever run an Internet-facing SSH server knows that you are likely to see a few hundred SSH brute force attempts a day, so something like this comes in handy. Personally, I have been hit with as many as 30,000 failed SSH login attempts in a single night, so something like this was high on my list for the new server build. DenyHosts also has a nice function where it will download IPs which other DenyHosts installations have blocked recently, which should help to preemptively block attacks.

In addition to building the packages, I also went and wrote proper rc.d files for both of them, so they function like native Slackware packages. But as they are not part of the default boot scripts, they don't start unless manually called from rc.local, which is demonstrated a little farther down the page.

Storage Configuration

       The main change in regards to the storage devices used in the server was limiting the I/O on the CF card in an attempt to squeeze as much life out of it as I could. This consists of mounting the CF card with options like noatime to limit redundant I/O and using a non-journalizing filesystem. This is the /etc/fstab file from the server:
# Root filesystem on Compact Flash
/dev/hda1        /              ext2    defaults,noatime,nodiratime    1   1

# Primary storage drive
/dev/sda1	/mnt/storage	xfs	defaults,logbufs=8		1   1

# Backup drive
/dev/sdb1	/mnt/backup	ext4	defaults			1   1	

# /tmp on RAM
tmpfs   /tmp   tmpfs   defaults,nosuid,size=64M,mode=1777  		0   0 

# Serve web pages out of RAM
tmpfs  /var/www/htdocs  tmpfs  noexec,nosuid,size=24M,gid=www,mode=0775 0   0

# System mounts
devpts           /dev/pts         devpts      gid=5,mode=620   		0   0
proc             /proc            proc        defaults        		0   0
#tmpfs            /dev/shm         tmpfs       defaults         	0   0
Here you can see the changes I made for the root filesystem and both of the HDDs. I chose to run the whole OS out of one partition for simplicity, though usually I would break it up into a few partitions. You can also see that I used different filesystems for the two HDDs. I used the high-performance XFS where speed counted (the drive users would be interfacing with) and the slightly more prosaic EXT4 for the backups. Beyond performance, my logic was that if there was some sort of software related filesystem failure on one drive, the other would in theory be unaffected. So far I haven't needed to test this out...

In the end, I simply setup some scripts to rsync my critical data between the primary 3.5 drive and the backup 2.5 drive. In keeping with my efficiency goals, the laptop drive is configured to go to sleep after 1 minute of inactivity, and the rsync process is done every 24 hours to prevent excessive read/write operations. The backup could probably stand to be done a little more often, but the main drives are not written to that often so it hasn't been an issue yet. Indeed, it can go a few days before rsync even needs to make any changes to the backup drive.

TMPFS Configuration

       You can see above I also did a 64 MB tmpfs "partition" on RAM. This is used as a general file cache, and holds most of the logs. I didn't put everything on tmpfs, as I would like at least the core kernel logs to survive a reboot, but things like the denyhosts logs certainly didn't need to live on non-volatile flash.

The common approach to this is simply symlinking the various log files from /var/log onto files under /tmp. In this scenario, your software continues logging as normal, never realizing it is actually going into /tmp. This is well and good, and what I had done in the past, but I wanted something a little more elegant this time around. So I went into the configuration file for each daemon and changed its log location to /tmp/log. I then went through each daemon's entry under /etc/logrotate.d/ and made sure logrotate was aware of the new log locations so they would get compressed and rotated out properly.

The problem with this method is that on reboot, there won't be a /tmp/log to save to, and your daemons are going to fail. So we need to setup a sane hierarchy under /tmp for our applications to use. But we need to be careful to do this before the daemons start, or else they will just silently fail. The following is the /etc/rc.d/rc.local file from the server, which sets up the files and directories under /tmp and then starts the daemons. This assumes that you have the proper rc.service files installed on your system, like I have included in my Slackware packages of those daemons.

#!/bin/sh
#
# /etc/rc.d/rc.local:  Local system initialization script.
#
# Put any local startup commands in here.  Also, if you have
# anything that needs to be run at shutdown time you can
# make an /etc/rc.d/rc.local_shutdown script and put those
# commands in there.

# Set storage drive to powerdown after 15 minutes
echo "Configuring power management on disk drives:"
hdparm -S 180 /dev/sda
# Set backup drive to powerdown after 1 minute of idle time
hdparm -B 1 -S 12 /dev/sdb
echo ""

# Create directories in volatile storage, need to do this before
# daemons start up and want to log
echo "Populating /tmp on tmpfs..."
mkdir -p /tmp/web
chmod -R 777 /tmp

# Permissions for logs should be a little different
# lighttpd doesn't run as root, so it must own log directory
mkdir -p /tmp/log/lighttpd
chown lighttpd:lighttpd /tmp/log/lighttpd
chmod 755 /tmp/log/lighttpd

# Samba runs as root, so it doesn't matter
mkdir /tmp/log/samba

# Start the Network Time Protocol daemon:
# We need to start this later than in rc.M because of the tmpfs log.
# This means that the time won't be 100% accurate for initial logs, but I
# really don't think that is a problem.
if [ -x /etc/rc.d/rc.ntpd ]; then
    /etc/rc.d/rc.ntpd start
fi

# Start lighttpd instead of Apache
if [ -x /etc/rc.d/rc.lighttpd ]; then
    /etc/rc.d/rc.lighttpd start
fi

# Start VSFTPD in standalone mode:
if [ -x /etc/rc.d/rc.vsftpd ]; then
    /etc/rc.d/rc.vsftpd start
fi

# Start DenyHosts
if [ -x /etc/rc.d/rc.denyhosts ]; then
    /etc/rc.d/rc.denyhosts start
fi

# EOF
Note that I start NTPd from rc.local instead of its normal spot in rc.M. As explained in the comment, rc.M runs before rc.local, so if we let it start there it won't have a log file to use. Interestingly enough, this build has taught me that for whatever reason, daemons generally won't create a log file if one isn't found, they will either continue running without logging or fail silently. I am not sure what the logic is behind this, but it seems rather odd.

You can also see at the top of rc.local where I apply the hdparm settings for the two HDDs. I use some very intense power management options on the drives, putting them to sleep as quickly and as often as possible. For the backup drive, the power management settings are so hardcore that within a second or so of the last file operation finishing, the drive is already back in sleep mode. For the primary drive, I was a little more relaxed, though it still goes to sleep after a few minutes. In practice, this causes a few-second lag the first time a file is accessed from the drive after a period of inactivity. For my purposes this is fine, but for some it might be an issue.

Dynamic DNS

       Usually this should be pretty straight forward, but it took me a bit of hacking around to get it working the way I wanted, so I thought I might as well document what I did.

I get my domain name from LQ Consulting which is an off-shoot of one of my favorite online forums, LinuxQuestions.org. The cost of a domain name from them is a bit more than other registrars, but the profits go right into supporting LQ, so I don't mind. Unfortunately, and I hate to talk down on the service, LQ Consulting doesn't appear to be anything more but a side-project of LQ's owner, Jeremy Garcia. There is no support to speak of, and all of the FAQs and documentation available are ripped right from the support pages of other registrars and web hosts.

The biggest problem was with dynamic DNS, which allows me to keep the domain name even when my IP changes (which isn't often, but does happen from time to time on Verizon FiOS). According to their "support" information, LQ Consulting's dynamic DNS service should be supported by a number of Unix/Linux scripts, which they give the links to. But after trying each one that was linked, I found that not a single one actually supports whatever scheme they were using. Worse, nowhere does it say which dynamic DNS service they are actually running, so I had to try and figure it out on my own.

The first clue was in their support pages. It said that once you got the software installed, point it to their update URL: dynamic.name-services.com. Opening that link in the browser returned the following string:

;URL Interface
;Machine is Reseller9-SJL
Command= Language=eng ErrCount=1 Err1=An invalid command was specified ResponseCount=1 ResponseNumber1=304150 ResponseString1=Validation error; invalid ; command MinPeriod= MaxPeriod=10 Server=Reseller9-SJL Site= IsLockable= IsRealTimeTLD= TimeDifference=+0.00 ExecTime=0.016 Done=true
I had absolutely no idea what that meant, but some searches on Google eventually pointed me in the right direction. Apparently this is the same dynamic domain name service run by eNom. Checking the various dynamic DNS update scripts provided for Linux by LQ Consulting, it turns out that none of them actually had support for eNom. Incredible.

After awhile I managed to find the script enom-ddns.sh, which I got working with a few slight modifications:

#!/bin/bash
# 
# Modified version of enom-ddns.sh - eNom Dynamic DNS Update with Bash

hostname='@'
password='XXXXXXXXXXXXXXXXX'
zonename='digifail.com'

# Must have the PERL module URI::Escape
url_hostname=$(echo $hostname | perl -MURI::Escape -lne 'print uri_escape($_)')
url_password=$(echo $password | perl -MURI::Escape -lne 'print uri_escape($_)')
url_zonename=$(echo $zonename | perl -MURI::Escape -lne 'print uri_escape($_)')

enom_url='http://dynamic.name-services.com/interface.asp?Command=SetDNSHost'
enom_url="${enom_url}&HostName=${url_hostname}"
enom_url="${enom_url}&DomainPassword=${url_password}"
enom_url="${enom_url}&Zone=${url_zonename}"

# For debug, echo the URL that will be called
#echo $enom_url
curl $enom_url &> /dev/null
Here you can see that for the hostname I used the character "@", which the DNS service interprets as essentially as a wildcard. So using "@" as the hostname and "digifail.com" as the zone name tells the dynamic DNS service that any hostname on digifail.com (I.E. www, ftp, etc) should resolve to the same IP.

Running this every hour has been working fine for awhile now, outside of one strange incident where eNom apparently forgot the password for the account, so the domain name stopped updating. That is where the "echo $enom_url" line comes in. If you are ever having problems with dynamic DNS, uncomment that line and then copy the URL this script generates into your web browser. The message you get back should give you an idea as to what the problem is.

The Journey Begins

       I can't really end this on a "Conclusion" or anything like that, as the end of the server setup was really just the start of a long journey. For the next 5 years or so this machine is going to be on the front lines; handling all of the internal responsibilities a network of this size requires, and keep DigiFAIL running for anyone who might be so inclined as to take a look. If you are reading this, then at least that second part seems to be working still. But now that the Mk2 has been running for a few months, I have gotten a pretty good feel for what worked and what might give me some trouble down the road.

I'll update this page if any problems come up with the original hardware or software, but until then, onwards and upwards!

UPDATE: October 2012

       Hurricane Sandy is battering the East Coast of the US, and the server's UPS system is now running on fumes. I'll have to shut down the server before it runs out of power, but not before documenting its current uptime, which stands as a testament to how well the setup has worked so far:

Pre-Sandy

UPDATE: October 2013

       Proving the universe is not without a sense of humor, one year later another storm has managed to knock the server out of commission. This time, a close lightning strike apparently damaged the Jetway's internal PSU, limiting its output power. I was able to limp along for a few days by taking the hard drives offline, which gave me some time to order a picoPSU-120 kit to re-power it with.

For anyone else out there with a Jetway JC-300, I'm happy to report that the picoPSU fits perfectly inside, I didn't even have to widen the hole on the back where the AC/DC adapter plugs in, it was the same size as the original one. Given the picoPSU has an actual support and upgrade path should anything go wrong, I would definitely suggest making the swap before the case's internal PSU dies on you.

picoPSU Installed