Smarter Sheep

2:00 pm - just finished 2 days of work moving pages around in our Sitefinity based website. Performance was dead slow so it took much longer than it should have.

2:05 pm - testing all the pages in the website menu structure. Hmm.. none of the most important ones are working. I’m getting a generic error message that something failed in the framework.

2:10 pm - logging back in as admin. Can’t edit the pages either. Same generic error message. I’ve seen this before with the framework and I’m concerned.

2:15 pm - go to sitefinity website and login to get latest framework patch which i’ve been avoiding. Hope applying this will fix the issue.
2:20 pm - download completes, unzipping and getting ready to upload to site. Good thing I made a backup last week.
2:25 pm - Put up a website maintanance page and start uploading Sitefinity patch… nearly 3600 files…

3:01 pm - 1 cell call, 1 google chat and 1 text message from 3 different people that the main website was down. No maintanance page or anything just a big server error.

3:02 pm - panic
3:03 pm - @#$%$#
3:04 pm - time of repentance and healing (had to keep it short for obvious reasons)
3:05 pm - checking site, ftp into server, start checking logs
3:08 pm - no logs to be found, editing global.asax file and turn on custom logging feature I wrote for such an occasion.
3:12 pm - log into host account and resetting asp.net
3:20 pm - still no luck…so editing web.config to turn off any caching etc and checking settings
3:30 pm - locating backups
3:35 pm - modifying backup web.config
3:40 pm - short prayer
3:41 pm - ftp and upload 7500 files to restore website file structure from backup
3:44 pm - realize this is going to take awhile… more repentence
3:50 pm - opening more sockets in the ftp client..hey hey.. that helped … short praise time
4:01 pm - backup restored..
4:02 pm - site still not coming up
4:04 pm - log back into hosting server and remove and reapply .net.
4:08 pm - site comes up yeah!
4:10 pm - testing menus because something doesn’t look right. A large group of pages won’t come up without the generic application error message.
4:15 pm - log into admin interface on website
4:16 pm - admin interface errors out
4:17 pm - sinking feeling
4:18 pm - panic, prayer, repentance (I’ve almost got this down now)
4:20 pm - try several other routes to login and finally get in. I can navigate anywhere within admin interface except to the pages screen which is where I need to go to try and repair any of the pages. Same error message there that I get on all the failing pages.
4:30 pm - create a trouble ticket with hosting company to see if they have anything in their logs that may help.
5:00 pm - still working on admin interface but no luck, hosting company responds that their are no errors logged in IIS so it looks like it’s all in our Sitefinity framework.
5:15 pm - make descision that we are going to lose the last several days of work on the site and have to restore database backups.
5:18 pm - log support ticket with hosting company to restore last week’s backup.
6:35 pm - hosting company says they have it done.
6:40 pm - something is wrong. Doesn’t look like they restored anything. data is the same.

7:00 pm - 1:00 am working on other things while waing for hosting company to reply to my ticket that something is wrong with their restoration of the database.
1:00 am - 1:40 am - tickets back and forth with hosting company as they realize they were trying to restore wrong database on wrong domain…. scary! they have always been superb in the past… must be a new guy having a bad night as well.
2:15 am - database backup finally successfully restored.
2:30 am - main church website is finally up and running again, but without any of the recent changes so it looks like a “do over”
2:35 am - testing pages and logging in to admin interface of website framework
2:40 am - looks like all is well and we are back to where we started… except for our main xml files on the site rotator got nuked during all this and I have to recreate them.
3:00 am - testing in several different browsers and it looks like we are back online.
3:05 am - sigh of relief mixed with thoughts of having to redo all the work and how to do it without the framework corrupting our data again.
3:10 am - prayer and thanks to God for getting us through another day in the church IT department. I’ve learned long ago that things could be much worse