Jul 5, 2007

It was my great pleasure on Jun 26 to be a part of the Network World IT Roadmap event in Santa Clara! For this event, I was interviewed in front of a live audience of about 60 CIOs for the Voice over IP (VoIP,) Convergence, and Collaboration track by Johna Till Johnson, president and chief research officer at Nemertes Research. For those who are interested, some highlights of our conversation are below:

You did a network overhaul that included a VOIP implementation a number of years ago, correct - 2003?
Yes, we were fairly early adopters of the technology. In fact, 802.3af (PoE) had just been finalized a few months prior to our first (pilot) implementation. We had the switches on order, but because of obvious delays in finalizing the switch hardware to meet the spec, we had to borrow some proprietary inline power injectors from 3Com to power our phones until the new switches came in. We probably received the first PoE switches off the line from 3Com.

How large is the implementation?
17 locations that are separated geographically. About 800 handsets now, a variety of hard sets, some softphones, wireless.

Can you tell us a little about the network overhaul that was going on at the same time as your VOIP implemenation?
We were fortunate enough to be looking at a network overhaul at the time, so we really had the opportunity to start with a clean slate. Most of the schools were still running on 10-year-old 10Base-T hubs, with 10Base-2 coaxial cable links tying together distant school buildings, and 128K bit/sec ISDN-based WAN. We moved to a managed, multi-service network on Fast and Gigabit Ethernet switches from 3Com, a fiber backbone, with T1s multiplexed to a DS3 at the central office. All in all we added about 250 switches with about 7000 ports, and pulled 200 miles of copper, 15 miles of fiber.

One of our key goals was to create a truly converged network, rather than segregating everything with vlans and multiple backbone links. We can take a phone, plug it into any port, and expect it to set itself up and work without intervention. It's based on a distributed model, in which network call processors (NCPs) installed at every site handle the call handoffs and distribution, voicemail, and system configuration. Basically we've become a voice shop and a data shop, which has increased our workload a little since we used to outsource all our voice to a local service provider.

What were some of the challenges you had with the implementation?
We decided early on to do most of the installation ourselves using in-house staff. We had to learn the nuances of voice and dial plans up front, which was a bit challenging, and lined up the pilot install with the manufacturer, so that we could shadow them through the process. We probably saved about a half million dollars in install fees and made ourselves more self-sufficient, but it was a challenge getting it all done.

We also had heat and power issues. Between the significant increase in port density, and the larger power draw from the PoE, we quickly found that our UPSs couldn't keep up in our higher density racks, so we had to double their size. In addition, our MDF locations run a bit hotter, so we had to re-work our A/C at a couple of our sites.

We also ran into NAT challenges, since every site uses NAT. We only have 4 class C address ranges to work with, which nets us about 1000 public addresses. This obviously isn't enough to handle our several thousand devices across multiple sites, so we subnet everything down to 30 valid ips per site, with everything behind them NATed.

Of course, we didn't discover that NAT was a problem until site number 2, when we couldn't make site to site calls, so we had to scramble a bit to find a creative ways to route around the NAT gateways and get voice traffic to function. We use our layer 3 core switches to route around the NAT gateways for voice traffic only.

What is the benefit of that approach?
Since the phones are getting private ip addresses, we have an inherent increase in security, since they are non-routable and therefore impossible to attack from the outside. We can still connect hard sets or softphones from outside using VPN connections and for our people out on conferences or off-site for some other reason, but we have zero risk of outside attack.

The 3Com system adds another layer to this, in that the phones all run Layer 2 for local calls, and only grab an ip address for site to site calls. This not only reduces our ip address management overhead, but makes the phones themselves difficult to attack, since they only have addresses for a brief period.

How did you go about selecting a vendor for your implementation?

We established our requirements and office/classroom standards up front, which remained relatively fluid as we discovered new opportunities while researching vendors. VoIP enables a number of new capabilities that one wouldn't think of when coming from a traditional TDM background. One of our key requirements was the ability to converge voicemail into our email system for unified messaging. We were adamant about this not being a proprietary solution that required a specific email system - we very much wanted it to be based on open-standards to avoid any sort of vendor lock-in. The 3Com solution uses standard IMAP, which makes it very easy to include voice mail messages in any email client our users might choose.

We formed a committee early on to lay everything out, which was made up of key unit leaders, office managers, and a few parent volunteers who were professional networking consultants. This helped to not only involve those who the technology would impact, but also to establish expectations up front, so that noone would be surprised when the system was implemented. We called in the vendors directly for demonstrations, and revised our documents/standards throughout the process. We even enlisted the top choices to write bid specifications/RFPs for their own products, from which we lifted the key details for our own specs. Finally, we hosted a "technology showcase" to which we invited all of our manager level staff from across the district for an overview of the technologies, providing them an opportunity to learn about the systems and to ask questions of the vendors directly - again, managing expectations.

What kind of benefits are you realizing with respect to VOIP?
Most school districts don't have fully functional phones in every classroom - they typically have a rudimentary handset, often without buttons on it for dialing out, and rarely with individual voicemail. Our biggest goal was to provide full voice function to each classroom, incl. voicemail and doing unified messaging.

Also key for us has been the abilty to direct-dial people by their extension anywhere in the district, without having to route through and around people. This has been a huge productivity gain. One of the interesting things we did was to go with a site code/extension model, which allowed us to assign consistent extensions from site to site. Basically, the office manager is always 180, nurse is always 192, etc., so all our people have to know is the site codes to make a call from site to site. All of our classrooms are already numbered - room 1 is x101, etc. so if a user knows what room a teacher is in, they can dial them directly.

The system also empowers the district through its flexibility. It is quite easy to set up a new office with several phones, their own auto attendant, voicemail, and the like for a specific project in a matter of minutes, rather than days. This really can't be done with traditional telephony - you would have to call in cable people, etc.

We now handle all of our moves, adds and changes in-house using existing cabling infrastructure, and gain on the maintenance side of things, since we are not maintaining separate networks.

And, of course, we have the expected savings on our phone bills. No more Centrex mileage charges, which amounted to several thousand dollars a month, reduced long-distance charges, etc., etc.

What has changed since you originally installed the network?

We've grown quite a bit, adding a few new schools and offices, but stuck with the same platform; it's worked well for us. We've continue to grow and see people doing more voice applications in general.

What advice do you have for other would-be VOIP implementers today?

  • Plan your standards early
  • Set your expectations before you start shopping.
  • Spend a lot of time planning the implementation; even if you're not involved in the implementation initially, if you can take over that role down the road you can save a lot of money and make yourself a lot more agile; it's easier and quicker to respond to issues.
  • Training will be a factor for your voice guys, who don't understand data, and data guys, who don't understand voice. That's a consideration that must be made up front.
  • Infrastructure planning is critical . we were replacing all of our infrastructure, but that is not always the case. Need to test existing stuff fully.
  • Consider your network topology - what your plans are. Do you want to vlan this thing, separate voice and data, or do you want it as flexible as possible?

With regards to standards, are you now using things like SIP?
SIP was too new when we were implementing, and really didn't work (it still doesn't in many respects.) When you think about what SIP is actually for - interoperability - and then look at how it is implemented today, you'll notice that people aren't really using it for that. Everyone pretty much uses the same phones and equipment across their organization. This is because there are still problems getting vendor A's SIP product to work with vendor B's. Of course, if you buy all of vendor A's gear, then everything works great. But if you try a mix, you still have to sacrifice features due to inconsistent standards implementation. It is getting better - you can transfer more than 9 out of 10 calls between most vendors' phones successfully - you couldn't do that a few years ago, which wasn't good enough for me (somehow I didn't think it would meet our end users' expectations, either.) Thus, if all of your equipment is coming from one vendor anyway, why do we care about SIP?

Our system actually is SIP-compliant, but we don't run it in SIP mode. We like the layer 2 functionality of it because we don't have to manage a large IP infrastructure and have a whole separate set of IP addresses for all of our phones. They all run layer 2 locally and only run layer 3 TCP w/real IP addresses if they're communicating site-to-site, which dramatically eases the management level, and has actually enabled us to do the routing around the NAT quite easily. This would have been far more difficult to do with SIP, since we would have had to do VLANs, separate out the DHCP, etc., etc.


Post a Comment

Post a Comment