For most Information Technology professionals I know there is no worse time than when something goes terribly wrong. We aren’t talking about hardware failures or the normal every day software bugs and issues. We don’t get excited about those. They are what we deal with every day and we just see them as the busy work we do that interrupts us from our everyday jobs of creating magic and solving problems for our enterprise. No I’m talking about the catastrophic event that never should have happened. The point at which we realize that we aren’t going to be able to solve the problem without significant loss of information or expenditure.
Just such an event happened to me this past week. A personal mistake caused a significant data loss. During the aftermath there was a point where I realized that our backup plan while well designed wasn’t working perfectly and we had gaps in our coverage. This meant that there was that one horrible moment where I got a heavy feeling in my stomach and realized that we were going to have to go and talk to the rest of the company about the impact on them. In ordinary situations the conversation would have been fairly painless as our backups would have restored everything or almost everything and we would have only had to explain the inconvenience of waiting for the restore to be completed and the small gap of information that had to be recreated. This was not the case this time. There was a large gap of data that had to be recreated manually and as much as I would have liked to there was no way for me or my staff to recreate it, it would have to be done by the employees who had generated the data in the first place.
As I was thinking over these things over night and planning the conversations I would need to have in the office the next day it occurred to me that I was dreading certain conversations more than others and the ones I was dreading the most made me think about why I do what I do and why it is satisfying for me personally. I discovered as I lie awake that night that I wasn’t dreading the conversation with my manager (who is not an IT person) or the other managers in the company. There was nothing to do with them but be open and honest about the mistake (mine personally as it turns out) and the failure in our backup coverage, outline the steps we took to recover what we did, and describe the changes we were already in the process of making to fix the issue in the future. Indeed when I had that conversation with my manager he asked me exactly those questions and didn’t waste anyone’s time with useless recriminations and I expected nothing less from my experiences with him.
The Conversations I dreaded the most were those with the operations staff. The people who came in to work that morning only to be told by me that a personal mistake I had made was going to make them reenter work they had already done. This is what I desperately wanted to avoid. The part that I would have run away from if it had been possible to do so. The look on their faces gave me a physical pain in my heart like a knife had been plunged deep into my chest. I would have given almost anything to have been able to tell them that I had found a way around them having to do this work again.
As I examined that I realized that this wasn’t because it was terribly hard on them. It was an annoyance for them, some more than others and I certainly feel bad that they had to have their time wasted by this mistake, but the real answer I think was because I see myself as the hero. I see myself speaking up in a meeting finding a way to get something done that can’t be done, or to reduce the effort employees are going to have to make to do something. I see myself fixing problems and solving mysteries that employees of the company have been unable to solve on their own. This event rips off the fantasy that I can solve anything, that I can fix it. That I am the IT magician that makes miracles happen.
I have a feeling, just an instinct, that many IT professionals see themselves the same way. It is our way to feel some control over life even if it is only virtual. It is our way of making a difference. We get a thrill out of the feeling we get when someone comes to us with a difficult problem and we can solve it in a few minutes and our customer walks away happy and feeling that we have saved the day. This week I was faced with the unenviable position of having my cape ripped away and my powers exposed as so much smoke and mirrors. It wasn’t a pleasant experience and I am eagerly awaiting the nightmare’s end.