This is intensely nerdy, but I’ll try to translate.

A few days ago, I got asked to update a link on one of our pages for the new graduate course bulletin. The link was on a code-heavy page in two parts that listed the details of our graduate programs.  The page changes depending on what link the user clicks on the page before.

The change was easy – I’m all about updating links – but something went wrong. The link fixed easy and the re-publish of the page was smooth – though the system insisted that I re-publish the other section on the page even though I hadn’t touched it.

And then it broke.  Instead of the information about the page, it was a mess of code dumped out on the page.  

We had a problem.

I tried to revert the stuff I had changed and the link went back – but it was still broke.

So, we tried to look at the same page in our test (DEV) environment.  And the page – both parts – was exactly the same as Production.  Except, it worked.

It was like  1 + 1 = pickle.   Everything was the same, except one worked and one didn’t.

The whole team got in on this and we brainstormed so hard there were clouds in my office.

The users were starting to call about the broken-ness and our support with the vendor was going nowhere.  And then I had a thought.

I dug into the history of the part of the page I hadn’t touched – but the system had insisted I republish too – and found the problem.

My former co-worker had last worked on this years ago and his entry was in the system as an editor.  However, after that was an entry from a prior student assistant that had worked for us.  He had looked at the page, switched modes from code to WYSIWYG (what you see is what you get.  Yes, that’s what it stands for), which broke the page.

But, he didn’t publish it so it never went live.  And no one else worked on that page.  And so we sat on a freaking time bomb that lasted right up until I re-published that page and broke everything.  And that was after a clone of the entire system and database and at least 2 upgrades

Dammit, Joey!  🙂

And all the code looked the same between instances since it was really broken.  Just, waiting.

I broke this the rest of the way in DEV as a test, then reverted it back to the last good version.  The page broke as we expected and then came right back.  I did the same thing in Production and then rang the Victory Bell (™) like a boss cause it was all good. 

It was exactly what Batman would have done.  Or, maybe, Oracle.