Fighting BT Wholesale

BT Wholesale Engineer Quotes

2008-03-17T16:37:00.004Z

Sometimes, I really do wish that I knew what the hell was going on in BT Wholesale/Openreach. Here is a small selection of fun quotes I've gathered from BT Wholesale engineers, either stated to customers of mine, or directly to myself.

"This wouldn't have happened if you were with BT."
Told to a customer by engineer when leaving their premises after fixing a fault.
"We seem to have lost that line."
Frames engineer calling myself about an SDSL fault.
"You wouldn't happen to have the frame mapping information for that line would you?"
Asked of myself by the same engineer as previous (frame mapping information would be documentation of where the line is within the telephone exchange).
"This is definitely a line fault."
Engineer to a customer.
"OR-15 fault proved to cp equipment"
Same engineer, same fault, on the BT Wholesale fault report to me.

Long time, no update....

2008-03-17T16:20:00.003Z

It's been a while since I updated this thing. No excuses, other than being busy busy busy both in work, and my home life.

I have read a little bit of criticism about this blog, stating that complaints like these should be brought up with Ofcom; that's all well and good, until you've had to deal with the unholy combination that is Ofcom and BT Wholesale.

Ofcom is of course the telecoms watchdog. However as far as watchdogs go, Ofcom are the toothless old mutt with mange that can barely stand up under their own power. The complaints my company makes to Ofcom with regards to BT Wholesale's behavior are done so weekly. But unless you can produce actual hard evidence, they are not interested. Ofcom do not want to hear about long-term observations and conjecture about BT Wholesale's internal workings. If you cannot show Ofcom a definitive "BT Wholesale told us to go fuck ourselves" memo signed by someone in BT Wholesale... they will not listen. Because of BT Wholesale's lack of openness, and ability to ensure you cannot provide any evidence (as all evidence ultimately is provided by BT Wholesale's ordering/fault systems), most Ofcom meetings in my experience end up as a he-said she-said argument.

The purpose of this blog however, was, and still is, to give the general public some idea of just what is going on behind the scenes of your ISP, and why it can be so difficult to get your faulty DSL line working again.

eCo Broadband Ordering

2007-04-20T13:42:00.000Z

"eCo Broadband" is the ordering system provided by BT Wholesale to ISPs for ordering ADSL on phone lines. And it's been broken since the start.

eCo is based on Siebel CRM (which BT is a partner for). Siebel CRM is a Customer Relationship Management system. For those of you who don't know what that is, CRM is a system intended for internal use by a company to manage customer contacts, basic accounting etc etc. BT have basically bodged it into an external ordering system. The system itself is an ActiveX thin-client terminal, run within Internet Explorer (interpret this as slow, and doesn't work in IE7 or Firefox).

To give you an overview of just how complicated BT makes things, below is a flowchart laying out the various systems and whatnot that BT Wholesale runs. The orange section is the ONLY section that the ISP has visibility of (this is probably by no means accurate, but is what we have built up knowledge of over the years of dealing with BT Wholesale).

Careful study of this diagram will show you the following (and very basic) implementation flaws:

People working in BT Wholesale use a different interface and database to what the ISP uses, for almost every department and function, and sometimes even have several differing interfaces showing the same or similar information.
There is a single piece of software that connects all the databases together - this frequently fails and causes orders placed on the ISP systems to not be visible on the internal BT systems, modifications don't propagate correctly etc etc.
Internal to BT, there appear to be several interfaces into the system offering differing views, and which also have the ability to alter orders and assets within the system. This can only make tracking down faulty systems near impossible.

There is also an XML interface intended for bulk orders, however after trying to tell the developer of the XML interface that it was so broken as to be unusable, I was told:

There's nothing wrong with the XML interface. Although there are several service providers who have decided to use software that can automate the keyboard and mouse, and use that with the eCo interface to place their orders.

Apparently he was so full of himself he couldn't see the flaw in his argument. If the XML system is so broken that even large providers are resorting to screen-scraping the eCo system.. there's something really, really wrong with it.

Additionally, around 3-4 times a week and usually during peak hours, either or both the MAC generation and WOOSH diagnostics systems will crash hard under the load. BT have not as yet seen fit to upgrade these systems so that when ISPs are open for business, they can actually conduct it.

I should also add that I have heard from several people working in some of the much larger ISPs (eg. Easynet, Sky, Plus.net) that they do not use either eCo or the XML system, and in apparently have access to a much more detailed and useful system.

BT Retail also appear to have full access to the BT Wholesale backend systems, unlike any other ISP. Once too often BT Retail have been able to give details of a fault to one of my customers.

So, that's a very quick explanation and run through of what eCo Broadband is, and why it's the bane of every provider.

Current eCo Broadband problems

2007-04-19T21:20:00.000Z

BT Wholesale's ordering system (used by ISPs to place orders for ADSL) is currently suffering a major fault, which one person the BT Wholesale provisions desk admitted to.

Over recent weeks, orders placed on lines have been rejected with a nonsensical error: ADSL order cancelled, reorder as provision. This doesn't make sense, as the original order was a provision order.

One BT wholesale monkey admitted that they are seeing a great deal of these errors, and admitted that their ordering systems were being migrated to a new platform, and some orders were getting dropped in between the systems. Except it's much worse than that.

We've found that some orders are being canned on the eCo frontend, but not on the backend systems, or in some cases, only partially cancelled on the backend systems. Numerous orders have resulted in an ADSL tag appearing on the line (which means that it's impossible to attempt a second order without getting it cleared), or even an apparent ADSL sync, but no ability to login.

Of course, BT Wholesale managers are making all manner of excuses for their botched system's screw-ups. We've had:

There's open orders on the line, they'll need to talk to their voice provider. Blame circle tactic.
There's a tag on the line, you'll need to talk to the tags team. Again with the blame circle, and one that can take the tags team a week or more to clear - and during this time who gets the blame from the customer? That's right, the ISP.
Sir, the order was cancelled. Yes, of course it was cancelled. You're screwed up system cancelled for no reason.
It looks like the telephone exchange is out of capacity. This is a lie. The exchanges rarely reach capacity, and are normally upgraded long before they reach saturation.

Of course, even pointing out that it seems highly unusual that you've just had 5 orders in a row go down the tubes is greeted with an instant:

There is nothing wrong with our system. The order was cancelled because [insert mystical reason pulled out of the manager's ass].

I should also mention as a side note, that an additional problem that appears to have cropped up is that "modify" orders (those are orders which change the speed, contention or care package on a line) are intermittently causing instantaneous cease orders (cancel the line, no more DSL).

So, if you've just ordered ADSL/broadband, and your ISP has told you that BT are cancelling it? Don't blame your ISP.

Six Degrees of BT

2007-04-18T22:02:00.000Z

The first thing that you should understand about BT, is how BT is (in theory) split up.

BT Retail - Responsible for your telephone voice service usually, plus BT Internet/Yahoo! Internet.
BT Wholesale - Sells data services, including broadband to nearly all ISPs that use BT landlines; with the exception of LLU providers - they simply use BT Wholesale to place orders to move lines onto their own equipment. Theoretically, all broadband providers have equal access (here's the first secret, they don't).
BT Openreach - The voice service equivalent of BT Wholesale. Not a lot to say here, except that these guys for some strange reason actually tend to be the most helpful out of all three BT companies. Unfortunately, they won't come into this blog much.

First thing you should know, is that although BT like to claim that all three companies are seperate, they're not really. A BT engineer may work on a wholesale job one day, and be working on an Openreach job the next.

Secondly, is that even though BT Retail and BT Wholesale systems (databases, ordering and reporting systems) are supposed to be separate, they're not really. All too often I have seen BT Retail have access to systems that they are not supposed to have. However, the illusion of separation is purely to keep Ofcom off their back.

Thirdly and lastly, is something we like to call the "blame circle" in my workplace, and the unreal seperation of the BT groups only helps them to do this. The blame circle is when one group will pass off a problem onto another group until it eventually comes full circle. For example, a ADSL fault is reported. Wholesale will state that it's a problem with the voice service, and the customer will need to contact their provider... which is 99 times out of 100, BT Retail. BT Retail phone monkeys are apparently taught that at even the slightest mention of "fault" and "broadband" to retort with "Sorry sir, but you'll have to call your ISP. This is a broadband fault." and so on.

Next time, we'll explain the BT eCo Broadband ordering system, how and why BT Retail are not made to use it, and why it is the greatest source of frustration for a lot of providers.

Introduction

2007-04-18T21:52:00.000Z

I have been working in an ISP environment for 8 years. In this time, ADSL (or broadband) and SDSL services have risen to prominence in the UK, to the point where they it is now cheaper to have a broadband service than it is to have a dialup service.

However, the main sticking point in this is that British Telecom (BT) still maintains it's monopoly over the UK telecoms industry. Ofcom - the telecoms watchdog - is toothless, and frequently fails to do anything that actually punishes BT in any fashion.

This is a blog about my daily encounters with the minions of BT.