Yesterday was NOT a fun day at work.
As I’m sitting here sipping five year plus expired Pepto-Bismal straight from the bottle and frantically trying to keep my systems stable enough to produce the newspaper despite the completely random network, I wonder if management is ever going to “get it.”
Management outsourced our infrastructure, and wonder why it costs more and delivers less. The answer is ownership. We care about the stuff under our watch and will do just about anything to keep things running long enough to get the product out. The outsource company (IBM) cares about the $$$ in the contract, and will do just about anything to stick to the letter of that contract to maximize their profit (including things that appear to be downright shady).
Our network went buggy a couple days ago. They blamed a WINS/DNS server installed at a remote facility by somebody else and called it “done”. No “why was this the problem”. No “we found this“. Basically a WAG (wild ass guess). As an engineer, I figured out long ago that things don’t fix themselves. If you can’t point at a cause and say, “This IS the problem and here’s why.” with 100% certainty, then the problem is NOT fixed.
So of course, it wasn’t fixed. From 10am yesterday until after 5pm the network was so unstable that most folks couldn’t work. These folks would be our reporters, writers, editors, ad builders, etc.
As applications guys, all we could do was swear creatively and try to keep our applications up…when we could talk to them that is. One of my major ones for instance processes and delivers ALL the page images that are rendered for output…and that’s a relatively simple one. The Ad production and Page building systems are simply immense. They don’t much like network fluginess (that’s a technical term). It was a struggle to keep them stable.
This is the fourth time in less than a year that we’ve come very close to NOT publishing due to the same problem! In the 21 years prior to that…well…we’ve had problems, no doubt, but we’ve never been so cut off from all alternatives. We had creative talent and dedicated folks that figure out a way to get the job done. Here, we were pretty much sitting on our hands waiting for somebody else to get something working. We were also saying stuff like, “Well, if I had that (insert resource here) that management’s given away to a third-party company, then I could do (insert pretty cool thing here that gets the paper out).
What’s different? Ownership. We are newspaper guys. We are production. We believe in what we do. We have a personal stake in making sure this paper publishes. For the outsourcers? It’s just a job. They don’t understand our industry, much less the lengths we’ll go to in order to get the job done. They are still in the process of taking all those extra tools away from us. The day they accomplish that, well, it’ll mark the beginning of the end. Without creative, dedicated, talented, newspaper people at the controls, we simply won’t publish. Going to be a sad day…that.
IBM doesn’t like creativity. IBM doesn’t like alternate workflows. IBM doesn’t like change. IBM likes procedures. For them, the job is a success if they followed the procedure…even if the client died and his house is on fire.
Did we get a paper out? Well, yes. There are still enough dedicated and creative folks scattered about so that we can accomplish the impossible.
Did we get the paper we WANTED to produce out? Well, not so much. Again, the time lost cannot be recovered…we can’t simply delay the production…so the quality and no doubt the content suffered.
If I had my way the picture above would fill every space that we didn’t get to put what we intended in.
So. They rebooted some stuff. Hey, now it’s fixed! (yeah, right). What was the problem? Where? Why? What’s been done to actually FIX it?
Heh. That answer will come with the next failure.
Oh, and in case you thought I was exaggerating about the 5+ year old Pepto-Bismal?
Ugh. Still works pretty good though…and that’s a good thing. I’m watching the pings for several of my systems begin to sporadically time out.