At approximately 2:15pm today our site went offline. Other servers in the data-center were fine, and our server was responding to ping and UDP traffic seemed fine, but any TCP traffic (the kind that is necessary for the operation of our site) was not able to make it past the switch.

Early investigation seemed to hit that we were the victim of a DoS (Denial of Service) attack. The question was out whether we were the intended victim or not.

Upon exhaustive inspection of all the network equipment in the path of our server, technicians manged to find a switch that was operating poorly and further diagnosis deemed it to be a firmware failure.

A temporary fix was put in place, all affected systems came online, and a plan was made for further upgrades and a permanent fix.

Within the next week we could have a couple minutes of downtime again, while we implement the permanent fix so please stay tuned.

Also, as a side note, we will likely be moving servers again in a couple months, from the current data-center to another data-center where we can do better monitoring of network services, and make sure our downtime is minimized.