Our main website for work is hosted in the cloud and is maintained by a company we\u2019ve worked with for over 15 years. It\u2019s generally very reliable and so are the periodic patches\/updates they do. <\/p>\n\n\n\n
There was a patch scheduled for a recent Wednesday night and I made a mental note to check on things on Thursday morning – just in case. This check is usually nothing more than logging in to the website and making sure things are where I expect them to be – then updating the team.<\/p>\n\n\n\n
Thursday morning I opened up my email to a flurry of system messages indicating an extended downtime – much longer than we usually see.\u00a0 \u00a0 I quickly checked and the system let me log in – and we appeared to be up and running – so my message to the team was that things looked a little bumpier than usual<\/em> and we should be a little extra vigilant in our review.\u00a0<\/p>\n\n\n\n And then things started to get weird<\/em>.<\/mark><\/p>\n\n\n\n I edited a page just fine, but the second page wouldn\u2019t save. I could download an image, but not upload it to a new folder. If I changed the format and tried again, it would work. Some pages wouldn\u2019t update and some would. <\/p>\n\n\n\n I reached out to my colleagues and their testing was all over the map. Some things that worked for me didn\u2019t work for them. In a couple of cases, an entire function wouldn\u2019t load. <\/p>\n\n\n\n Then I got a note from another editor on campus reporting a new problem. <\/p>\n\n\n\n But, there was no pattern. My notes continued to grow with problems and exceptions – making it challenging to even figure out how to report. I opened a ticket with our support, assumed it was the patch, and shared all the conflicting notes.<\/p>\n\n\n\n I updated the team and we continued to try and test. It wasn\u2019t time to update all the editors across campus, but I was starting to think about that email. <\/p>\n\n\n\n Support got back to me asking for screenshots and noting they were looking into it. Which suggested that this was just us and not a buggy patch that they would roll back from all the clients.<\/p>\n\n\n\n I sent out a note to the editors and explained where we were and that they should stop editing until we had more information. I promised to keep them in the loop.<\/p>\n\n\n\n My boss was out this week and I sent a note to his peers to update them in case they got any questions. <\/p>\n\n\n\n And then, things got worse.<\/p>\n\n\n\n I got another update from support – another person this time – and they reported that the night before (and continuing into the day) our website was the subject of a DDOS attack. <\/p>\n\n\n\n DDOS stands for Distributed Denial of Service – where hundreds or thousands of compromised computers in a network are directed to send potentially millions of requests to a website. <\/p>\n\n\n\n This is usually more than enough to crash the site. And keep it down until they stop. <\/p>\n\n\n\n Our support had been battling this overnight and managed to get some degree of stability by blocking traffic from 14 different countries. It cut the attack in half and got the site up and running again, but since the attack was ongoing it made the editing unstable. <\/p>\n\n\n\n And, they had tried to notify us of the problem by emailing a staff member that hadn\u2019t worked at UA in over 5 years.\u00a0 Sigh.<\/p>\n\n\n\n I was supposed to lead a training session\/office hours meeting with about 30 people that afternoon, but knew there was no way I could do that and keep on top of the problem. <\/p>\n\n\n\n I hadn\u2019t created the meeting so I couldn\u2019t cancel it in the system, but sent out an email to the potential participants to let them know the situation – and promised to join the call anyway for a few minutes in case anyone missed the cancellation notice.<\/p>\n\n\n\n I updated the team again, then got on a call with support about the situation and options – which ran long so I had to have another colleague hop on the call in my place to explain what was going on. <\/p>\n\n\n\n With the support call finished, I joined the call and used that time to copy the information from the support tickets to a separate system used by our IT department. <\/p>\n\n\n\n When I ended that call, our IT department reached out and suggested the proposed solution from support sounded like a scam. <\/p>\n\n\n\n The only good news was that during the support call it appeared that the attack had halted, for the time being. <\/p>\n\n\n\n I sent another note to the editors explaining where we were and told them I would do more testing in the evening. I finished up the workday with a call to the team on our next steps. <\/p>\n\n\n\n After a short break to eat, I logged in back and did another round of testing and tried to catch up on updates.\u00a0<\/p>\n\n\n\n On Friday, I did more testing with the team and then sent a note to the editors to let them back into the system. <\/p>\n\n\n\n As the morning went on, support started to pressure me to make the switch to this new system. From my own research it had some serious problems and I needed a consultation with our IT department to clarify. We ultimately decided to leave things as they were for over the weekend and regroup on Monday. <\/p>\n\n\n\n In the meantime, a potential law student reached out from another country and reported through the channels that they couldn\u2019t access our website.<\/p>\n\n\n\n Support confirmed the error they were getting matched with their country being blocked. I explained what was going on and promised a quick resolution. <\/p>\n\n\n\n It was an exhausting couple of days, but I did get some good feedback from the editors about how I handled the situation and I did my best with what I had at every step.<\/p>\n\n\n\n Still, I feel responsible as the caretaker and webmaster of the site. <\/p>\n\n\n\n I had some homework over the weekend to get a better handle on the solution – a CDN (Content Delivery\u00a0 Network) – and we regrouped on Monday to keep working the problem.<\/p>\n\n\n\n Though, I suspect I won\u2019t sleep well until we\u2019re finally past this hurdle since we could easily get attacked again – and there\u2019s no way of knowing why. <\/p>\n\n\n\n Update: the CDN is working, though not without some bumps. Our log-in stopped working for a bit and we had to work around that. There\u2019s also an update delay we\u2019re trying to get used to. And I\u2019ve got some subsites to add to a list to resolve. <\/p>\n\n\n\n But, we\u2019re generally safer and now more stable. <\/p>\n\n\n\n Though, I kinda wish it had just been a buggy patch we could roll back.<\/p>\n","protected":false},"excerpt":{"rendered":" Our main website for work is hosted in the cloud and is maintained by a company we\u2019ve worked with for over 15 years. It\u2019s generally very reliable and so are the periodic patches\/updates they do. There was a patch scheduled for a recent Wednesday night and I made a mental note to check on things […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"_bluesky_dont_syndicate":"","_bluesky_syndication_accounts":"","_bluesky_syndication_text":"","footnotes":""},"categories":[1],"tags":[],"class_list":["post-3252","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/thunderofwade.com\/blog\/wp-json\/wp\/v2\/posts\/3252","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/thunderofwade.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/thunderofwade.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/thunderofwade.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/thunderofwade.com\/blog\/wp-json\/wp\/v2\/comments?post=3252"}],"version-history":[{"count":1,"href":"https:\/\/thunderofwade.com\/blog\/wp-json\/wp\/v2\/posts\/3252\/revisions"}],"predecessor-version":[{"id":3253,"href":"https:\/\/thunderofwade.com\/blog\/wp-json\/wp\/v2\/posts\/3252\/revisions\/3253"}],"wp:attachment":[{"href":"https:\/\/thunderofwade.com\/blog\/wp-json\/wp\/v2\/media?parent=3252"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/thunderofwade.com\/blog\/wp-json\/wp\/v2\/categories?post=3252"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/thunderofwade.com\/blog\/wp-json\/wp\/v2\/tags?post=3252"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}