What happened to the site? (AKA I’ve been working on the server, all the live long day)

Published on by Josh Bray

computer-and-cat-o

Well, hel­lo every­one! You might have no­ticed that SuperNerdLand has been down since last week, and maybe won­der­ing what hap­pened. While I’m tak­ing a bit of a break from restor­ing ar­ti­cles on here, let me fill you all in on the short details.

April 20th, ap­prox. 6am, our serv­er got hacked. From all looks, it was a brute force en­try be­cause we had a lot of iffy look­ing bot scan­ning ac­tiv­i­ty go­ing on in our logs just a day or two be­fore the serv­er was ex­ploit­ed, and we did have an uptick in what our WordPress se­cu­ri­ty was au­to­block­ing when look­ing back at that time frame. After that, it got turned into a zom­bie and was part of a bot­net send­ing spam mails and at­tempt­ing to scan oth­er IPs for a few hours un­til the dat­a­cen­ter got told by the ISPs some­thing was hap­pen­ing, and the DC told my VM ad­min. She then shut off the net­work adapter un­til I could look at it.

Long sto­ry short, af­ter look­ing into logs to see what hap­pened I de­cid­ed to just pull a bit from the serv­er then nuke the ma­chine to re­in­stall everything. 

While I did not see any ac­tiv­i­ty to point to­wards data/password theft, if you had an ac­count on our site then you should change your pass­word. Even though the pass­words were en­crypt­ed in our data­base, change them any­way. It’s just a good idea. 

Even if you didn’t have an ac­count on our site. Change your pass­words reg­u­lar­ly peo­ple. It’s just good practice. 

Anyway, I’ll sum up an­oth­er lengthy part of the sto­ry where my back­ups were cor­rupt and I could only re­cov­er bits by say­ing just that, and fast for­ward to where we are at now.

The serv­er has been re­stored with some ex­tra se­cu­ri­ty in mind — for the hard­ware and the site. We are also hav­ing to re­store the ar­ti­cles. Not all by hand, mind you. We were able to im­port some of what got pulled from the bad back­ups and what could be scrapped from archive.org (spe­cial thanks to har­ta­tor who made this awe­some ruby script to down­load whole sites from the archive.org). Between these we are able to re­cov­er all the ar­ti­cles that were not drafts on the backend.

We are just clean­ing up the im­port­ed ar­ti­cles, and re-inserting im­ages were need­ed ATM. We are go­ing to have a first pass done by tonight, with a large chunk of the site go­ing “live” to­mor­row morn­ing. By Wednesday pass num­ber two of the restora­tion will be done, and by Friday every­thing should be back to where it was in a more com­plete sense. 

I want to apol­o­gize for the down­time every­one! And thank you for the pa­tience. We are work­ing hard to get back on­line for you, and all bookmarks/links should be work­ing just fine af­ter the restoration. 

Also, if you no­tice any­thing par­tic­u­lar­ly finicky in the next few days don’t hes­i­tate to yell at us and tell us to fix it! 

Be back soon lovelies!

The fol­low­ing two tabs change con­tent below.
My Twitter profile

Josh Bray

Josh has worked in IT for over 15 years. Graduated Broadcasting school in 2012 with a fo­cus on A/V pro­duc­tion. Amateur pho­tog­ra­ph­er with a pas­sion to make things work… by any means nec­es­sary. Editor-in-Chief and do-er of tech things at SuperNerdLand
My Twitter profile

Latest posts by Josh Bray (see all)

Scroll to top