The difference between theory and practice...

Sat Aug 02 07:47:00 -0700 2008
manage

...is greater in practice than in theory.

It's a favourite saying of mine, and it certainly applies in spades to IT.

I should be sat here typing this via my new whizzo NTL Business 20 mbit cable connection, having dumped my old Virgin 20 mbit cable connection because of intolerable levels of "traffic shaping" restricting my alleged 20 mbit connection to around 4 mbit, irrespective of ports or protocols. Monday 28th July 2008 was changeover day.

So on Monday 28th the NTL guys duly turn up with an identical Scientific Atlanta Webstar 2100 cable modem to the one already sat here.

After all, there is absolutely no technical reason whatsoever that two separate arms of the same parent company cannot share the same fibre infrastructure (and "last mile, actually last 50 yds in my case, coax) and deliver services to both domestic and business customers.

So the Virgin webstar 2100 is unplugged, and the NTL one is plugged in, the tech pulls out his mobile phone and speaks to another tech and tells him my account details and the MAC address of the new NTL webstar 2100, it doesn't work immediately so they go away saying not to worry it will propogate and go live by midnight.

What has followed since then is a tale of woe.

To be sure, when you ring NTL business you speak to an english person in england, not an indian in india who insists you reboot the modem before doing anything else, and generally speaking the support guys and gals working for NTL are more clued up, but....

They are JUST as capable as their Virgin counterparts to making promises to ring back that they do not keep, and answering questions in all sorts of manners that do not actually promise you anything concrete by any given time.

For my part I was playing it cool, thinking, "let's get the bloody connection sorted first, then screw them for some compo", because, after all, this is supposed to be a BUSINESS connection, with appropriate level of service.

So there I am on Friday (1st Aug) afternoon at 3 pm, not having heard back from the person I spoke to at 11 am who promised faithfully that they would ring me back and tell me what the hell is going on, it's time to get insistent, or a little bit anyway.

Bottom line here is NTL business just happened to have a bunch of new (router etc) hardware go live on Monday 28th, and the bottom line on that was that due to this the cable modems (new installs, new config) can get a downstream lock, but they cannot get an upstream lock and therefore cannot effectively log on to the network or accept the "poke" from the network admins to config and activate them.

Basically the problem here is one of IP addresses, both allocation and routing. The new upgraded NTL hardware isn't handling IP addresses properly... trust me to choose that week to change...

But anyway, back to the manager who I finally get through to, he admits that there is no way my connection is going to be live before the end of next week, eg Thursday or Friday 7th or 8th of Aug, and suggests before I do that a formal complaint is filed.

I agree wholeheartedly, pointing out that being unable to secure any form of connectivity is, (NTL having cancelled my Virgin account) to put it mildly, embarrassing, not merely financially inconvenient, for a business.

I dropped broad hints about the maxed out config of the NTL webstar 2100, 48 mbit down and 5 mbit up...

So how come you're reading this?

If you think it was a DOCSIS hack you're wrong, it would have worked, until I was caught, done for theft of service and banned for life, nope, you can't beat a bit of social engineering and getting the hack done by an insider, leaving your hands clean, and the bit of social engineering was speaking to a fellow tech head and telling them the truth, and by the way buddy, what's the chances of just routing my traffic through the Virgin routers and ebabling my cable modem MAC through that until NTL business can get their shit together... "hang on a minute, sending the config file now..."

So while Customer Services were busy not ringing me back, or bullshitting me when then they did ring me back or when I rang them, some lonely Tech managed to band aid the problem until it could be fixed properly, and it didn't take 3 hours of his time like Customer Services, it took 5 minutes, and got NTL a lot more patient customer.

The really funny bit is the aforementioned NTL manager, he had apparently noticed what yon tech had done, and was about to do something about it when one of HIS senior managers (who I happen to know socially and professionally) looked over his shoulder and said "Ah, I see John's account is being taken care of, well done." so the aforesaid manager gives me a courtesy call to inform me that my complaint has been lodged "at the highest levels" and was there anything else he could do for me while we both waited for my connection to be sorted next week?

So, in theory, transferring from one branch to another is easy, and in theory the practice of changing is not much different from the theory, but it practice it ain't easy at all, and in practice the gap between the practice and the theory is big enough for entire business models to disappear down, screaming, into a bottomless pit.

The only ones who get my sympathy here are the techs that do the install, they are invariably the only direct human contact between customers and the company, so they get all the crap, even though they warned against trying to roll out a new system on a live network, even though they are actually the only ones who can eventually fix things and give me my connection...

WTF?

Sat Aug 02 13:56:39 -0700 2008
manage

Big company makes mistake. You are offline for a few days, maybe even a week. As if crap like this didn't happen with small companies.Except that it does. And often, the small company is worse because you have only one or two guys to work with - you can't just call back and get somebody else who happens to know how to help you. Instead, you just have the same guy telling you the same thing and you get nowhere.

I recently had my backup colo go redundant on their network feeds. A good idea, except that they didn't let me know and something borked on the routing tables so that all outbound traffic (from my server) worked fine, but all inbound traffic was null routed.My network monitor reported "all good" because the servers successfully reported in. But I was down for almost 3 days as far as my customers went! That hurt, but there wasn't much I could do once I found out about it...

Nameless big company can't offer you "personal service" - but that's often a good thing because "personal" service could just as easily be bad service as good. Sorry you're off-line. Good luck. But sometimes your whining gets to be a bit much!

WTF?
Sat Aug 02 14:22:23 -0700 2008
manage

But sometimes your whining gets to be a bit much!

I know, I must be the customer from hell, having the sheer unmitigated effrontery to expect a company to actually deliver something other than lies and bullshit in exchange for my hard cash.

It really is so unreasonable of me to expect that the force of Law that supports all business transactions applies to them as deliberately as it applies to me.

It is quite ridiculous of me to expect a company to display even the slightest hint of ability at their core business, the vegetable stand only has rocks, the butcher only has dead flies, and the cableco only has a bunch of brochures for a mailshot... you'd think I was asking for them to deliver me a 1,000 kg payload into geo stationary orbit.

When I get bullshitted solidly for a week I'm whining.

When I point out that I am only able to type this reply now because I have abilities that 99.9% of other BUSINESS customers will not have, or will not employ, I'm being tiresome, like it is such a major achievement to get someone connected to a network that was designed from the ground up in the first place to be resilient and route around damage, and you talk like I'm asking for 5 cents a gallon gas, all I can burn, and being unreasonable.

When I point out that THEY offered, legally an "invitation to treat" which I offered and they accepted, this thing that I am not getting, I did not go to them an ask for a bespoke service I suppose that means nothing either.

WTF?
Sat Aug 02 14:56:16 -0700 2008
manage

I think his point is with a business contract, you get an SLA.  You are paying extra to get that service and they can't deliver.

Yes, shit happens.  But you should expect a vendor to be fully forthcoming about it, not hem and haw and beat around the bush.

Depending on how much the contract is, I'd expect regular updates as to what happened, why and what they are doing to make certain it doesn't happen again.

The fact that people put up with it is why a lot of this still behavior persists.

And the last time my network was down for more than an hour because of a mistake on my ISP's end, I got 4 box tickets to the season opener Indianapolis Colts vs Chicago Bears game in Indy as well as credit on my account and a written explanation of what/when/why and why it won't happen again.

What did you get for your 3 days down?  You should at least get some serious credit for the account.

WTF?
Sat Aug 02 20:49:14 -0700 2008
manage

If he has an SLA he should take a good lookie at it and exercise his options or STFU. For my 3 days down, I got a good lesson in reviewing a contract. Shit does, in fact, happen, and most contracts are written to accomodate the occasional oops. Has he, in fact, taken a look at what he actually agreed to?

My guess is that he has actually looked at it, decided he didn't like it, and decided to bitch here rather than pay for an actual SLA, because in truth, an actual SLA with teeth doesn't come cheap.

And what it really comes down to is this: the myth of the 5 nines. http://www.mohawksoft.org/?q=node/38 In reality, it's rarely possible to achieve more than 2-3 nines or so without insane expense, creative interpretation, or sheer dumb luck. Let's grind the numbers, shall we? 

99% = ~ 3.65 days/year.

99.9% = ~.365 days/year (about 8 hours 45 minutes)

99.99% = ~ .0365 days/year (about 45 minutes)

99.999% = ~ .00365 days/year (about 5 minutes)

As an actual, hosted application provider, we've sustained about 99.95% uptime during business hours for the past 4 years. Despite a strict SLA, we've never once even been threatened when faced with a few hours of downtime every year or so. And let's be realistic - a single bad reboot can kick you down below your 5 nines, but a decent quality server can usually sustain 3-4 nines without too much trouble if it's not overloaded. It's the usually part that freaks people out, because even quality systems occasionally get nailed with a bad part or two.

Crap happens. Remember when 365 Main went down? It was a terrible event for me, ruining my whole day and costing my company considerably in expenses, even though we had everything ready. Redundant power? Check. Triple-redundant network feeds? Check. Redundant servers? Check. Quality equipment? Check. Uptodate software? Check. Properly secured network, with best practices applied? Check.

This is one of the best hosting centers available anywhere! And it was taken down because of a brownout and a backup generator trip circuit that got "confused". It took me a full day to get 100% service back up, though we did have our core services up in about 2 hours. (luckily enough, one of the servers came back up rather than trip on the brownout power condition - our downtime would have been much higher otherwise. Yes, that's a double-failure - the backup power failed to trip, and the power supply on that server failed to trip. It was even more dumb luck that the server that came back up is the one that was otherwise responsible for backups, so it had all the current data already on it)

Even when you have every "i" dotted and every "t" crossed, something will happen that will blindsight you, despite all your best planning, redundancy on site, redundant "disaster recovery" off-site hosting, integrated backup strategies and daily backup recovery/verification. Let me tell you... all this is damned expensive, time-consuming, and still worth every dang bit, because if you don't do ALL of this, (and maybe more!) the downtime can be days, weeks, or even permenant.

Is GF paying for any of this? Somehow, I doubt it, from hearing words like "cable modem". His "business plan" is probably better termed "fixed-IP consumer service" with a fancy name and a $30/month price boost.

And that's why I construe his rant as... whining.

WTF?
Sun Aug 03 04:09:57 -0700 2008
manage

And that's why I construe his rant as... whining.

Mr Pot, meet Mr Kettle...

Fawkes didn't blow 50k on a pink slip SLA so he's just a luser and deserves everything he gets....

When Fawkes was in hosting and lot's of "pro" outfits went down because like you they talked redundancy but it actual fact had all their eggs in one basket he didn't go down, for the basically three years the business was running uptime was a straight 100%, despite considerably less than that from any individual component.

But that's chalk and cheese, and having owned and run a hosting biz I know of what I speak.

You talk of 99.95% uptime for yourself, and you talk like you live in an island and the only goal is your servers staying up, nobody else needs to be online, whereas I realised that keeping my servers up was only half the battle, the bigger half was keeping customers connected.

My uptime with NTL business for the (projected) first two weeks, or about 4% of a year, is a big fat 0%.

50% would do, connecting every other hour for just 60 minutes, 25% would do, connecting one hour in four, but no, I get a big fat 0% and just because I haven't dropped 50k on an SLA whoop de do.

The fact is this is NTL BUSINESS, that is all they do, and they are incapable of rectifying a problem WITH THEIR OWN SYSTEMS that the user themselves can and did rectify in literally 5 minutes by picking up the phone and talking to a tech.

SLA doesn't come into it, this is core competence, this is American Airlines with a bunch of new Airbuses and staff who know how to run the Staten Island Ferry, with a couple of ex military multi-jet pilots floating around in the engine room.

This is after all a website owned and run by an Open Source talking head, I wonder how many Open Source coders Bruce knows who are, from your perspective, serious players who can drop 50k on a T3 and SLA for their personal connection, shit, there are admins on here on bloody dial up, which is more robust than the service NTL business have provided me with so far.

BASIC CONNECTIVITY is what we are talking about here, that thing that mom & pop or lone geek ISP's used to manage to provide 20 years ago when it was a damn sight harder and more technically challenging than it is today.

Am I paying for this? Fuckin A I am paying for this, at a 2 to 1 exchange rate its 100 bucks a month, and don't forget I've bought backbone bandwidth with SLA and all that crap and I know what it costs per mbit, THIS EXACT SAME COMPANY can connect me at THIS EXACT SAME ADDRESS with THIS EXACT SAME CABLE & HARDWARE at up to not just T1 or T3 but by changing ONLY the cable modem DS3 (45 mbit) and by replacing the "last mile" which is in fact 50 yards of coax with fibre I can have OC48 (2 gbit) right here, switched and carried by the SAME fucking network and hardware that is going to take two weeks to provide me with 20/1 mbit.

So what we have here, from your perspective, is FedEx is great because you drop 5k a month on an SLA and each parcel costs you 100 bucks to ship, but your customers pay 20 bucks a parcel and it may or may not get to the destination, never mind in one piece, but hey, that's ok because YOUR parcels are delivered, and you really do not care what that says about YOU as a business.

Sorry dude, it only does what it says on the tin if you give us your firstborn son and all your daughters.

WTF?
Sun Aug 03 06:32:06 -0700 2008
manage

I worked in the telecom industry for 7 years. Lucent, then Alcatel, specializing in ATM, Frame Relay and Optical Switching for the first 5, then CALEA and security for the last 2.  I'm very familiar with 5 nines and what it takes and costs, since one of the things I worked on was switching equipment for 911 emergency systems and banking systems.

Yes, shit happens.  Again, the issue is transparency and root cause analysis. Is a provider honest and forthcoming about what and why?  What is the plan to make sure this doesn't happen again?

Guy's issue isn't that something happened, it is how he was treated by the company.

WTF?
Sun Aug 03 07:10:55 -0700 2008
manage

Guy's issue isn't that something happened, it is how he was treated by the company.

Amen, though I must add that since my "problem" was easily routed around by a tech stepping outside SOP for 5 minutes it has to be said that the culture that caused me to be treated this way is almost certainly also responsible for the fact that what should have been such a trivial problem in the first place is allowed to balloon into what is tantamount to a two week outage, which Customer Services then has to bullshit me about.

The difference between theory and practice...
Sun Aug 03 11:28:37 -0700 2008
manage

Thought I'd actually read this since it's one of Guy's rare articles under 6k.

Couple of points Guy seems to have missed:

- no matter what country they're in front line tech support follows scripts because they're the filter to keep the idiots who just have the modem unplugged from bothering the "real" techs.  This is a fact of life in the support workd, live with it.

- secondly, and perhaps more importantly, WTF were you thinking cutting over a business critical connection with zero service overlap?!?  As everyone else has rightly pointed out, shit happens.  No matter what the shit turned out to be, or who's fault it was specifically, it's nobody else's fault that you didn't allow for that in your execution of the project.  (I.e. get the new service working first and then still wait a week or two after getting things switched over before even thinking of disconnecting the old service.)  Remind me again how many years you've been in this industry?  Egads man...

The difference between theory and practice...
Sun Aug 03 12:02:57 -0700 2008
manage
  1. it wasn't business critical.
     
  2. I had an overlap planned, NTL screwed it by killing the Virgin config.
     
  3. Sure, if I was serious about it the level best I can actually do at this location is buy xDSL from a BT umbrella corporation or reseller, AND cable (as I am) from Virgin / NTL, for some form of redundancy
     
  4. NB despite not being connected until the end of next week I am managing to post here, so I'm not that dumb
The difference between theory and practice...
Sun Aug 03 12:03:03 -0700 2008
manage
  1. it wasn't business critical.
     
  2. I had an overlap planned, NTL screwed it by killing the Virgin config.
     
  3. Sure, if I was serious about it the level best I can actually do at this location is buy xDSL from a BT umbrella corporation or reseller, AND cable (as I am) from Virgin / NTL, for some form of redundancy
     
  4. NB despite not being connected until the end of next week I am managing to post here, so I'm not that dumb
The difference between theory and practice...
Sun Aug 03 12:03:06 -0700 2008
manage
  1. it wasn't business critical.
     
  2. I had an overlap planned, NTL screwed it by killing the Virgin config.
     
  3. Sure, if I was serious about it the level best I can actually do at this location is buy xDSL from a BT umbrella corporation or reseller, AND cable (as I am) from Virgin / NTL, for some form of redundancy
     
  4. NB despite not being connected until the end of next week I am managing to post here, so I'm not that dumb
The difference between theory and practice...
Sun Aug 03 12:03:11 -0700 2008
manage
  1. it wasn't business critical.
     
  2. I had an overlap planned, NTL screwed it by killing the Virgin config.
     
  3. Sure, if I was serious about it the level best I can actually do at this location is buy xDSL from a BT umbrella corporation or reseller, AND cable (as I am) from Virgin / NTL, for some form of redundancy
     
  4. NB despite not being connected until the end of next week I am managing to post here, so I'm not that dumb
The difference between theory and practice...
Sun Aug 03 12:03:16 -0700 2008
manage
  1. it wasn't business critical.
     
  2. I had an overlap planned, NTL screwed it by killing the Virgin config.
     
  3. Sure, if I was serious about it the level best I can actually do at this location is buy xDSL from a BT umbrella corporation or reseller, AND cable (as I am) from Virgin / NTL, for some form of redundancy
     
  4. NB despite not being connected until the end of next week I am managing to post here, so I'm not that dumb
The difference between theory and practice...
Sun Aug 03 12:03:19 -0700 2008
manage
  1. it wasn't business critical.
     
  2. I had an overlap planned, NTL screwed it by killing the Virgin config.
     
  3. Sure, if I was serious about it the level best I can actually do at this location is buy xDSL from a BT umbrella corporation or reseller, AND cable (as I am) from Virgin / NTL, for some form of redundancy
     
  4. NB despite not being connected until the end of next week I am managing to post here, so I'm not that dumb

Not really important but...

Mon Aug 04 06:24:01 -0700 2008
manage

I've always preferred:

 

The difference between theory and practice is that in theory there is no difference between theory and practice.

 

There is an implied "...but in practice, there always is."

 

That is all.