Physically separate networks to finally fix server issues


I’ve been fighting with all of my network stuff for the longest time, and I think I finally have it licked (again).

Ever since I moved the site over to Linux, I’ve had weird, unexplained issues, most often the network would stop responding with no warning.  I wouldn’t get any response from a terminal, and going to the physical machine would show the PC is extremely show and the cursor would barely respond.  The only solution would be to reboot.  Looking through the logs I would see that a certain process ID was hung up, but now that the system rebooted I couldn’t see what ID was associated with what program.

Over the past year and a half I thought the culprit could have been several things… sound drivers, Realtek network cards (which I disabled and replaced with Intel), bad drivers, custom firewall issues, network link speed auto-adjusting, FIOS adapter needing to be reset, and I have addressed all of these things only to have the problem continue to occur.

Maybe there were multiple issues, and when the network goes down it’s not always the same issue.  Sometime the FIOS adapter needs to be reset, sometimes the firewall needs to be rebooted, and sometimes the server just stops responding.  The server issue could be the sound daemon, which I uninstalled (and honestly why is it enabled by default on a server distribution?).  After doing research it looed like the Realtek network cards don’t like to play nice with the default drivers in RHEL or CentOS, and compiling your own drivers was no help.  I finally bought an Intel card and it seemed pretty stable for a few weeks, but still continued to happen.  I finally disabled the Realtek network cards in the motherboard’s BIOS, but RHEL still wanted to use cards that weren’t even there.  The latest theory is to disable defragging the network stack, which happens after so much time or so much traffic.  If this continues to happen then I’m going to dump RHEL and go all Ubuntu.

Last night the IPcop firewall I’ve been using for years just stopped responding.  Since Frontier Communications is so persnickety about me using THEIR craptastic router in order to be supported, I started using it.  The home internet came up fine, but trying to get a server up was a complete bear.  I finally was able to get it up by setting up port forwarding rules (enabling all web server ports to direct to the internal server), but it was extremely slow and would drop a lot of traffic.

I ended up reinstalling the IPcop firewall (after a complete dusting and upgrade to SSD), and I could connect to the firewall, but it didn’t seem to be directing traffic to the Internet.  When you have four possible network ports, it takes a lot of trial and error to determine which physical port has been assigned to which network.  I guess I should write down the MAC address of each port and label them from the outside…

The problem with having so many layers to your network is like having optional features on your car: it’s just more stuff that can go wrong.  Going simple is the best way to make things just work, and I’m tired of jumping the moment I get a “your site is down” alert, which seems to happen at least once a week.  The reason I moved everything to Linux is to be super-stable and worry-free.

I finally fixed everything (I hope) by doing what I should have done a year ago: physically separate the home and server networks.  Now everything seems to be running like a champ, but time will tell if everything is truly fixed.

Alan is a web architect, stand-up comedian, and your friendly neighborhood Grammar Nazi. You can stalk him on the Interwebs via Google+, Facebook and follow his ass on Twitter @ocmodshop.