eCo Broadband Ordering

"eCo Broadband" is the ordering system provided by BT Wholesale to ISPs for ordering ADSL on phone lines. And it's been broken since the start.

eCo is based on Siebel CRM (which BT is a partner for). Siebel CRM is a Customer Relationship Management system. For those of you who don't know what that is, CRM is a system intended for internal use by a company to manage customer contacts, basic accounting etc etc. BT have basically bodged it into an external ordering system. The system itself is an ActiveX thin-client terminal, run within Internet Explorer (interpret this as slow, and doesn't work in IE7 or Firefox).

To give you an overview of just how complicated BT makes things, below is a flowchart laying out the various systems and whatnot that BT Wholesale runs. The orange section is the ONLY section that the ISP has visibility of (this is probably by no means accurate, but is what we have built up knowledge of over the years of dealing with BT Wholesale).

Careful study of this diagram will show you the following (and very basic) implementation flaws:

• People working in BT Wholesale use a different interface and database to what the ISP uses, for almost every department and function, and sometimes even have several differing interfaces showing the same or similar information.
  
• There is a single piece of software that connects all the databases together - this frequently fails and causes orders placed on the ISP systems to not be visible on the internal BT systems, modifications don't propagate correctly etc etc.
• Internal to BT, there appear to be several interfaces into the system offering differing views, and which also have the ability to alter orders and assets within the system. This can only make tracking down faulty systems near impossible.

There is also an XML interface intended for bulk orders, however after trying to tell the developer of the XML interface that it was so broken as to be unusable, I was told:

There's nothing wrong with the XML interface. Although there are several service providers who have decided to use software that can automate the keyboard and mouse, and use that with the eCo interface to place their orders.

Apparently he was so full of himself he couldn't see the flaw in his argument. If the XML system is so broken that even large providers are resorting to screen-scraping the eCo system.. there's something really, really wrong with it.

Additionally, around 3-4 times a week and usually during peak hours, either or both the MAC generation and WOOSH diagnostics systems will crash hard under the load. BT have not as yet seen fit to upgrade these systems so that when ISPs are open for business, they can actually conduct it.

I should also add that I have heard from several people working in some of the much larger ISPs (eg. Easynet, Sky, Plus.net) that they do not use either eCo or the XML system, and in apparently have access to a much more detailed and useful system.

BT Retail also appear to have full access to the BT Wholesale backend systems, unlike any other ISP. Once too often BT Retail have been able to give details of a fault to one of my customers.

So, that's a very quick explanation and run through of what eCo Broadband is, and why it's the bane of every provider.

Current eCo Broadband problems

BT Wholesale's ordering system (used by ISPs to place orders for ADSL) is currently suffering a major fault, which one person the BT Wholesale provisions desk admitted to.

Over recent weeks, orders placed on lines have been rejected with a nonsensical error: ADSL order cancelled, reorder as provision. This doesn't make sense, as the original order was a provision order.

One BT wholesale monkey admitted that they are seeing a great deal of these errors, and admitted that their ordering systems were being migrated to a new platform, and some orders were getting dropped in between the systems. Except it's much worse than that.

We've found that some orders are being canned on the eCo frontend, but not on the backend systems, or in some cases, only partially cancelled on the backend systems. Numerous orders have resulted in an ADSL tag appearing on the line (which means that it's impossible to attempt a second order without getting it cleared), or even an apparent ADSL sync, but no ability to login.

Of course, BT Wholesale managers are making all manner of excuses for their botched system's screw-ups. We've had:

• There's open orders on the line, they'll need to talk to their voice provider. Blame circle tactic.
  
• There's a tag on the line, you'll need to talk to the tags team. Again with the blame circle, and one that can take the tags team a week or more to clear - and during this time who gets the blame from the customer? That's right, the ISP.
  
• Sir, the order was cancelled. Yes, of course it was cancelled. You're screwed up system cancelled for no reason.
• It looks like the telephone exchange is out of capacity. This is a lie. The exchanges rarely reach capacity, and are normally upgraded long before they reach saturation.

Of course, even pointing out that it seems highly unusual that you've just had 5 orders in a row go down the tubes is greeted with an instant:

There is nothing wrong with our system. The order was cancelled because [insert mystical reason pulled out of the manager's ass].

I should also mention as a side note, that an additional problem that appears to have cropped up is that "modify" orders (those are orders which change the speed, contention or care package on a line) are intermittently causing instantaneous cease orders (cancel the line, no more DSL).

So, if you've just ordered ADSL/broadband, and your ISP has told you that BT are cancelling it? Don't blame your ISP.

Six Degrees of BT

The first thing that you should understand about BT, is how BT is (in theory) split up.

• BT Retail - Responsible for your telephone voice service usually, plus BT Internet/Yahoo! Internet.
  
• BT Wholesale - Sells data services, including broadband to nearly all ISPs that use BT landlines; with the exception of LLU providers - they simply use BT Wholesale to place orders to move lines onto their own equipment. Theoretically, all broadband providers have equal access (here's the first secret, they don't).
• BT Openreach - The voice service equivalent of BT Wholesale. Not a lot to say here, except that these guys for some strange reason actually tend to be the most helpful out of all three BT companies. Unfortunately, they won't come into this blog much.
  

First thing you should know, is that although BT like to claim that all three companies are seperate, they're not really. A BT engineer may work on a wholesale job one day, and be working on an Openreach job the next.

Secondly, is that even though BT Retail and BT Wholesale systems (databases, ordering and reporting systems) are supposed to be separate, they're not really. All too often I have seen BT Retail have access to systems that they are not supposed to have. However, the illusion of separation is purely to keep Ofcom off their back.

Thirdly and lastly, is something we like to call the "blame circle" in my workplace, and the unreal seperation of the BT groups only helps them to do this. The blame circle is when one group will pass off a problem onto another group until it eventually comes full circle. For example, a ADSL fault is reported. Wholesale will state that it's a problem with the voice service, and the customer will need to contact their provider... which is 99 times out of 100, BT Retail. BT Retail phone monkeys are apparently taught that at even the slightest mention of "fault" and "broadband" to retort with "Sorry sir, but you'll have to call your ISP. This is a broadband fault." and so on.

Next time, we'll explain the BT eCo Broadband ordering system, how and why BT Retail are not made to use it, and why it is the greatest source of frustration for a lot of providers.

Introduction

I have been working in an ISP environment for 8 years. In this time, ADSL (or broadband) and SDSL services have risen to prominence in the UK, to the point where they it is now cheaper to have a broadband service than it is to have a dialup service.

However, the main sticking point in this is that British Telecom (BT) still maintains it's monopoly over the UK telecoms industry. Ofcom - the telecoms watchdog - is toothless, and frequently fails to do anything that actually punishes BT in any fashion.

This is a blog about my daily encounters with the minions of BT.