The following is a "special letter" that I wrote for SNS News Service, "[t]he most accurate predictive letter in computing and telecommunications, read by industry leaders worldwide". It was an honor to be given the opportunity to contribute to such a respected publication.

Infrastructure 2.0 and the New Data Center Culture

Prior to the 1920s, before widespread electronic information technology was but a glimmer in the eyes of all but the most outrageous academics, clerks kept the wheels of commerce churning. Involved in calculating mathematical solutions to business and scientific formulas, clerks were known as computers, and were employed across the world of commerce to calculate everything from basic accounting to the most sophisticated calculus problems.

In the 1920s and ’30s, it became clear that a new branch of science – computer science – was developing ideas into technology that could automate those same functions. Computers delivered the same tactical skills to the business, at less cost, with more reliability. Over the next several decades, the term clerk was relegated to government bureaucrats and legal interns, displaced forever by machines from IBM, Digital Equipment Corp., and others.

Ironically, the information technology organizations that formed to tend the new electronic computer are perhaps the one place in business that still relies on an army of “clerks” to keep things running smoothly – system, storage, and network administrators. This is about to change, however – a trend driven by virtualization and next-generation data center technologies, most notably Infrastructure 2.0.

The “Clerks” in Your Data Center

Take a look at your IT operations teams. The odds are very high that you’ll find them staffed with well-trained system, storage, and network administrators working hard to keep up with the flow of trouble tickets and provisioning requests produced in the normal course of operations.

This volume of tactical expertise is necessary, because while IT has done an amazing job of automating processes and functions in business, the vast majority of IT is controlled through manual processes. In fact, spreadsheets, basic database applications targeted at helping humans keep track of their work, and simple scripts to automate manual tasks are the most common uses of computing to support IT.

These ad hoc tools had better be good. The loss of a critical IT system often means a severe loss of revenue. For example, after a June 2008 outage at online retailing giant Amazon, CNET reported that the business may have lost as much as $31,000 per minute, based on $4.13B in global revenue the previous quarter. That’s $1.86M per hour.

Needless to say, keeping business systems running is a high priority, and organizations have been willing to build large IT staffs to do so as a result. That labor is expensive.

A 2006 study by IT analyst firm IDC (“IDC: 2006 Worldwide IT Spending on Servers, Power and Cooling, and Management/Administration”) on the costs of owning and running servers in an enterprise finds that those costs have and will continue to increase rapidly, at least through 2010. What is most interesting about the report, however, is that the vast majority of the increase – from about $20B nationwide in 1996 to about $160B estimated in 2010 – comes from increased system administration and management costs. The number of people and skill sets required to run computing is an increasing burden on corporate IT.

Perhaps the most dramatic example of a largely tactical discipline is network administration, in part because it is the least automated of all data-center disciplines. Record keeping is often performed on spreadsheets. Administration is largely done through “old-school” textual user interfaces with cryptic commands and non-intuitive data and function sets. It takes real expertise to tend to the routers and switches that form the basis of a network infrastructure, but most of that expertise is applied through highly manual processes.

So IT relies on “clerks” to get the network job done.

Diseconomies and the Velocity of Change

All of that is finally changing through advanced technologies being introduced to data-center networks under the umbrella of Infrastructure 2.0. Most notably, the role of the tactical network administrator – responding to trouble tickets one-by-one as they arrive in a queue – is about to disappear forever.

The story of the timely passing of the network clerk has its roots in two things: the increasing velocity of change in enterprise information technology, and diseconomies of scale in network management.

Business demand for more IT to decrease costs and increase revenue – for everything from back-office functions to online marketing and e-commerce to Web 2.0 social networking and customer interaction systems – is driving incredible demand for IT resources. This demand has greatly increased the rate at which new applications are provisioned into the data center, which has led to a rapid increase in the number of devices to be managed, which, in turn, has put a strain on everything from physical space and power usage to the number of IP addresses consumed.

The nonprofit organizations responsible for allocating these addresses report that they will run out of unused addresses anywhere from late 2010 to early 2012 – assuming there is no mad land-grab for the remaining addresses. (It should be noted that there is a relatively new addressing protocol, “IPv6,” that organizations will be encouraged to adopt, which offers 3.4 X 10³⁸ assignable addresses – more than enough to last for the foreseeable future.)

This rapid increase in scale, coupled with the largely manual operations of the networking infrastructure, causes a second problem: the costs of network operations actually increase as IT needs grow. One striking example of this effect is the cost of managing IP addresses.

IP address management costs come as a surprise to many executives. IP addresses are those little four-part numerical tags that tell networking equipment both the identity of a device on the network and where to find that device. They typically look something like this: “192.168.123.231.” IT organizations purchase blocks of these addresses from their network access provider, which in turn acquires blocks of addresses from a Regional Internet Registry – one of several nonprofit companies chartered with managing elements of the common Internet infrastructure.

Infoblox, an Infrastructure 2.0 vendor, commissioned a study from Computerworld that examines the costs of domain-name (i.e., “cocacola.com”) and IP-address management (IPAM) for a variety of IT organizations, ranging from small businesses to the largest enterprises. The survey measured the cost of managing each IP address assigned to the target companies; one finding stood out:

The annual cost to manage each IP address is directly correlated with organization size. Enterprise organizations have higher costs per IP address, with an average of $9.19 annually. SMB organizations, on the other hand, report an average annual cost of $7.12 for each IP address. The overall annual average among all organization sizes is $8.10.

Much of this, according to the Computerworld study, is a reflection of the lack of automation in even the largest companies. According to the survey, a full 63% of large enterprises still rely on spreadsheets to track IP address assignments.

Virtualization Changes Everything

Diseconomies of scale are impacting bottom lines, and that is driving savvy organizations to look for ways to avoid these additional costs. The current boom in virtualization technologies is a clear example of this trend.

Server virtualization hit the IT marketplace in a big way with the maturation of special software, called a hypervisor, produced by enterprise virtualization heavyweight VMware and others. A hypervisor can be thought of as software that enables fractional usage of a physical server’s resources, much like dividing an office building into several leasing units.

A CIO Magazine survey from early 2008 (sponsored by VMware and chip manufacturer Intel) found that a full 89% of IT organizations surveyed had already implemented virtualization in production environments, or planned to do so in the next 12 months.

The reason for this boom is a familiar refrain for many: virtual server technologies increase utilization of physical server infrastructure, delaying the need to lay out more capital for servers to support new or growing application systems. Translation: virtualization saves money.

Most data centers adopted virtualization strictly to consolidate physical servers through fractional use. Interestingly enough, that reduction never happened for many. Instead, as physical capacity was freed up from one application, it was quickly consumed by the demands of other, often new, IT demands.

However, virtualization opened up other opportunities that will likely turn out to be much more valuable. First, an application running in a virtual machine can be moved around the network at will, without losing client connections or missing a calculation – and can utilize any physical server running a compatible hypervisor.

Now the application environment can be managed completely separately from the physical infrastructure. This has had a powerful effect on IT operations, as now servers no longer need to be tailored to a specific application that will run on them, but can be purchased according to a uniform design to support virtualization as a whole. Given the complexity of today’s highly heterogeneous data centers, it is easy to see why this homogeneity would be so desirable.

Virtualization also enables levels of automation that were previously impractical with highly customized physical infrastructure. As the virtual infrastructure has to be completely controlled through a computer program, it has not taken long for IT operations to begin to drive out the manual tasks that were once required to provision, maintain, recover, and retire computer servers in the past.

All of this automation speeds up the rate at which IT operations happen. This has driven a litany of change in the data center in recent months, including major technology advancements under the banner of Infrastructure 2.0.

Infrastructure 2.0

Infrastructure 2.0 represents the new wave of dynamic networking services and management tools, designed to systematically drive out the manual tasks and record systems that network administrators everywhere have come to rely on.

Examples of Infrastructure 2.0 systems include:

• Core network automation systems that manage network infrastructure, such as DNS/DHCP infrastructure, IP address management (IPAM), and network access control (NAC)
• Utility and cloud-computing infrastructure systems that automate provisioning of networking to physical and virtual systems as application service levels demand
• Unified computing environments that converge compute, virtualization, and Infrastructure 2.0-enabled network platforms into a homogenous, centrally managed IT infrastructure

Not that spreadsheets and command lines didn’t work before now, but the rush to embrace the economics of data center virtualization and cloud computing means it has become impossible to keep up with tactical network administration by hand. The days of the static network diagram, printed on the IT department plotter with a detailed schema of the data center, each line connecting a port on a switch to a port on a server or storage device with an accompanying IP address, will soon vanish forever.

As mentioned above, in the coming years, the practice of queuing requests for changes until someone can get to them – a practice that just doesn’t work in an on-demand world – will also fade away. Remember the diseconomies of scale for IP address management: network automation isn’t a “nice to have” anymore; it is a definite “must.”

What this means for network administrators is two-fold. First, if the bulk of tactical tasks – configuring servers, doling out IP addresses, assigning host names, routing past failures, etc. – are going to be turned over to automated systems, the network administrator’s role changes from tactical to strategic. CIOs will be demanding not that their network administrators can keep the network running by fixing it as fast as it breaks, but by identifying the policy sets that keep the network running automatically, and by recommending hardware and software systems that tighten the enforcement of these policies.

Second, this same argument applies to other system administrators as well – server administrators, storage administrators, and so on. Automation is having the same effect on system administrators everywhere. In fact, many thought leaders in the enterprise computing contend that there will no longer be the traditional cultural divide between server, storage, and network administrators. Instead, we see a convergence of the Infrastructure Architect and Data Center Operator roles. In other words, the responsibility of network operations falls under the auspices of general IT strategists.

What Is a Clerk to Do?

All of this is extremely frightening for IT tacticians. There is a palpable backlash against automation in many IT departments: a steady flow of skepticism about sharing resources, virtualization benefits, and the likelihood of true utility computing.

These skeptics miss the point entirely, however. Infrastructure 2.0 is not a technological ideal being pushed on the market from the top down. It is, instead, the recognition of work being done at the grassroots level of information technology organizations. Virtualization has led to automation, which has led to a rethinking of how data centers and networks are operated.

Anyone who values a career in IT operations or network administration should right now get trained on core Infrastructure 2.0 technologies such as VMware Virtual Infrastructure, Cisco’s Nexus switches (and upcoming Unified Computing offerings), and network specific tools, like the Infoblox Network Services Suite.

There is an old saying in IT circles: Enterprise IT has done a brilliant job of automating everything but IT. We are finally getting around to getting the job done. The result, as in most automation trends in industrialization, is the displacement of some roles and increased opportunity for others. The role of the clerk was eliminated from most business functions in this way, replaced in part by the more strategic knowledge worker.

The tactical IT administrator is about to become another excellent example of the effects of automation – thanks in large part to Infrastructure 2.0.