The Truth about the Wireless Bandwidth "Crisis"

(Part 2 of "Who Will Pay for Mobile Data?")

There's a big nasty dilemma hidden at the heart of mobile computing: No one knows how we'll pay for all that mobile data we're supposed to use in the next few years. The question doesn't get much publicity, but it drives some of the most intense debates in mobile, including net neutrality and the wireless bandwidth "crisis."

This is the second of a three-part series on the issue. In Part 1 (link), I talked about the tech industry's unlimited vision for the growth of mobile data, and why I think it won't come true because we'll run out of people willing to pay at the current rates for data service

In this part, I will talk about the alternate scenario, in which most people are willing to pay for mobile data and adoption of it continues to accelerate. In this case, the mobile operators will need to invest urgently in increased capacity, and even with that investment I think we'll eventually run out of cellular bandwidth.

This means the operators face two conflicting possible futures. In one, growth is about to slow down and they don't need to invest in a bigger network. In the other, they need to invest urgently in additional network capacity. For telecom execs, it's a bet-your-career choice with no clear winner. So, naturally, they are trying to get someone else to pay for the investment. That's the real cause of the rhetoric about a wireless "crisis," and it's driving much of the net neutrality debate.

To understand why this is happening, let's start with a look at the physics and economics of a cellular data network...


Mobile data doesn't scale like fixed-line broadband

When mobile operators in the US and Europe first built out their 3G networks, they miscalculated what people would do with them. They expected that new, relatively low-bandwidth mobile services like a simplified version of the Internet (called WAP) and picture messaging (MMS) would be the dominant source of data traffic on the network, and they structured it accordingly. But those new data services failed to take off, and the operators were left with a ton of excess capacity. Desperate to generate any revenue from their new networks, they offered fire-sale data plans for the newly-emerging smartphones. It didn't matter if the operator made a good profit off a smartphone data plan -- with the network already built and sitting idle, any data revenue was better than none at all.

So the operators in many regions gave us low-cost or unlimited data plans. Those plans set customer expectations for how they should use mobile data and what it would cost in the future.

Overcapacity continued on most mobile networks until the launch of the iPhone in 2007. We tend to forget about it today, but the iPhone was the first smartphone to make PC-style browsing practical and attractive for most smartphone users. The result was an explosion of mobile browsing, and almost overnight mobile data networks supporting the iPhone started to go from overcapacity to overload.

The reason for the overload was simple -- people in the developed world learned to browse first on their PCs, most of which have high-speed wired connections to the Internet. Bandwidth on these connections isn't infinite, but it's large enough that activities like file sharing and watching videos are mainstream.

When people started doing PC-style browsing with their smartphones, they brought their PC browsing habits with them. Unfortunately, the cellular wireless networks don't have nearly the same data capacity as the wired networks. So mainstream browsing behavior on a PC turns out to be excessive browsing on a smartphone, especially if you use a lot of YouTube.

Even if you're not a big video user, the normal sorts of messaging and web traffic created by a PC can overload a wireless network. A typical PC has a more or less continuous connection to the web, so instant messages and web app updates can ping back and forth constantly. But to conserve battery life and stretch network resources, a smartphone doesn't talk to a cellular network continuously; it basically says hello to the network, sends a message or a bit of data, and then says goodbye. Each little message and each app ping creates its own set of hellos and goodbyes. Send too many and they can overwhelm an operator's servers. Web apps send too many.

The operators, of course, can add additional wireless capacity to cope with the increased traffic, and they have been doing so. Cisco estimates that the total capacity of the world's wireless networks will increase by about 10x from 2010 to 2015. But these additions eventually run into physics problems. There's only so much information you can squeeze into a certain amount of wireless spectrum. At some point the cellular infrastructure overloads, and as an operator you have some ugly choices:

--You can add a lot more cell towers, reducing the size of each radio cell and therefore increasing the number of devices you can support. Unfortunately, this is extremely expensive -- not just to build the towers themselves, but for the fiber optic cables connecting them, the servers to manage them, and for the lobbying you must do to overcome political opposition to additional towers.

--You can offload data traffic to local wired connections. The operators are already pushing this hard, via WiFi. Some are also encouraging the installation of femtocells in individual homes and businesses. (A femtocell is basically a micro cell tower in a box the size of a wifi router. It gives you cellular service inside a building or business, using wired broadband to communicate back to the cellular network.) But that too is expensive: at around $200 a pop (link), it would cost about $13 billion to attach a femtocell to every one of the 67 million consumer broadband lines in the US. Plus there's the support cost for installing them, and the expense to put millions more cells in businesses, and the cost to buy the back-end servers necessary to support them. Nevertheless, some very smart people watching the mobile data market believe that femtocells are essential to the future of mobile data (link). Cisco estimates that offloading of some sort will handle about 20% of mobile traffic by 2015, and up to 40% in some countries.

--You can buy more spectrum, but there are huge licensing costs associated with that, not to mention the cost of retrofitting your cell towers for the new frequencies and replacing all of the phones in the installed base.

Unfortunately, even if you make all of the changes above, there are some very convincing arguments that it won't be enough, quickly enough, to head off a capacity crunch if current trends continue. The growth rate of smartphones, tablets, and wireless notebooks will swamp the cellular infrastructure no matter what. Folks in the mobile industry have taken to calling this the "Moore's Law vs. Shannon's Law" problem, with Moore's Law representing the exponential growth of computing power, and Shannon's Law the fundamental limits on how much data you can push over a particular chunk of spectrum. Reinforcing the Shannon bandwidth limits is the fact that some other critical elements in the mobile data infrastructure can't keep up with Moore's Law. The number of cell sites can't increase exponentially, and handset battery capacity is barely growing at all. A crunch is inevitable at some point.

So the people who tell you that cellular wireless will replace wired broadband just don't understand the physics involved. An outstanding summary of the situation was written by Martyn Roetter, a telecom consultant (link).

Here's the key paragraph:

"Until and unless the current laws of physics are invalidated in ways that remove current limits on spectrum capacity such as are embodied in Shannon’s Law, the future will see: (a) The vast majority of broadband traffic (as distinct from numbers of broadband subscriptions) continuing to be carried (delivered and transmitted) over fixed access networks; and (b) Demands for broadband traffic from wireless or mobile subscribers outstrip the capacity of all the bandwidth available for radio access networks to handle it, even with the use of the new spectrum that can be allocated and the deployment of more spectrally efficient technologies... Bandwidth within one optical fiber is vastly greater than all the bandwidth that might theoretically be made available for mobile communications, even if every megahertz were to be refarmed for mobile services. A single mode fiber has a bandwidth of as much as 100,000 GHz, or 100 terahertz, whereas total valuable spectrum for mobile communications provides bandwidth of no more than at most 3 GHz."

Got it? What he's saying is that wired broadband traffic can continue to grow exponentially, which will create demand for mobilizing that traffic through cellular wireless -- which the cellular networks can't handle.

If data traffic continues to grow at its current pace, we're headed for a situation in which the cellular networks will be overloaded no matter what we do.


Rock, meet hard place

So we have two possible scenarios for the future of mobile data. In the segmented scenario I discussed in part 1, we run out of customers willing to pay for mobile data plans, and the growth of mobile data slows down. In the consensus scenario, customer demand continues to increase, and we run out of cellular network capacity.

These conflicting scenarios are terrifying to the mobile operators because there's no way to tell for sure which one will happen. If you knew for sure that demand was going to continue to grow, you'd invest heavily in capacity, and also start raising data prices to restrain the growth in demand to something you can actually deliver. But if demand is about to stop growing, investing in capacity and raising prices is exactly the wrong thing to do. You'll end up with excess capacity, and the price hikes will make demand stop growing even faster.

This table summarizes the dilemma (click on it to see a larger version):



An economist would tell you that this will all sort itself out in the long term, and I'm sure it will in 20 years or so. But in the meantime, in the real world, the operators have to invest in infrastructure years before the demand arrives. If you're an executive at a major operator, it is almost impossible to get the forecast right. That means you will probably either overbuild the network, wasting billions of dollars and putting your career at risk; or you will underbuild, losing share to competitors and putting your career at risk.

You can't win. It's like one of those Star Trek episodes where Captain Kirk destroys the rogue computer by putting it in a logical loop (link). If you watch closely at tech conferences, you can see the smoke seeping out of the ears of telecom execs.

Faced with this dilemma, those telecom execs naturally are trying to find a third option: Get someone else to pay for mobile data. There are a couple of options:


Option 1: Have the government pay for mobile data

I doubt that most governments would pay to make wireless data completely free for everyone, but I was surprised when I found out how much governments are already paying for mobile. For example, the US government subsidizes mobile phone service for millions of unemployed people (because it helps with their job searches; they need phone numbers so employers can all to offer them jobs). I could easily imagine that benefit being extended to include mobile data, on the assumption that poor people need access to job boards (how we'll avoid paying for their YouTube and Kongregate usage I don't know).

Governments are also being lobbied to give special regulatory treatment to wireless data. The rhetoric around a "wireless spectrum crisis" is being used to influence governments. The focus of this lobbying in the US is on taking spectrum away from the TV networks and supplying it to the mobile operators. Effectively that is a financial subsidy for the operators -- if the government forces the transfer the operators will have to pay less, and will get the spectrum faster, than if they were to purchase it on the open market.

Here's how the lobbying works. This is an excerpt from an e-mail sent to me recently by a PR firm working for a group called the Internet Innovation Alliance:

"IIA's Blog: The Spectrum Clock is Ticking
Writing for Forbes, Lawrence J. Spiwak, President of the Phoenix Center for Advanced Legal and Economic Public Policy Studies, warns Congress that more spectrum needs to be freed up for mobile broadband and it needs to be freed up soon: Like it or not, the clock is ticking on spectrum exhaustion, both for consumers and our public safety professionals. Unless we want a market characterized by higher prices, failed data sessions, dropped calls and stifled innovation, policymakers need to implement a cohesive spectrum policy with a large degree of urgency."

At first glance, that reads like a standard plea from a bunch of web companies worried about the mobile network getting overloaded. But the backstory is that both IIA and the Phoenix Center are reportedly funded by the mobile operators (link, link). So this isn't an independent assessment of the situation, it's the operators sending us a message. And the message is: "Give us more bandwidth or we'll trash your phone service." I think that's a bit disingenuous -- unless the operators seriously mismanage their networks, we won't end up with both bad service and higher prices. But they're right that without more spectrum we'll definitely get one or the other.


Option 2: Make web companies pay for mobile data

Several of the leading operators in Europe recently argued that big tech companies like Apple and Google should be forced to pay to use the wireless networks. Although they don't put it this way, they're asking the big Internet companies to subsidize mobile data plans for users (link).

The CEO of Telefonica said the web companies "use Telefonica’s networks for free, which is good news for them and a tragedy for us. That can’t continue."

Here's the CEO of France Telecom (link):

"The real risk of everything is collapse. Nobody utters this loudly enough, but the real issue for the world is a collapse of the network or some local collapses. We are the people with pipes. We are supposed to invest heavily in pipes in order to bring the capacity which is necessary to sustain the explosion of consumption and usage and data traffic in our networks. At the same time, the people that create this traffic…are not really incentivized to manage properly, globally, the traffic. There is an unbalance in the overall system, which in our view is a major problem. It is totally impossible to absorb such an explosion in traffic without first, clearly investing massively in spectrum and equipment, and second, without introducing some new pricing approaches."

This is the heart of the whole debate about net neutrality. I believe it's not really about mobile operators trying to give an advantage to their own services, it's mostly about the operators trying to open up a second revenue stream because they're afraid they can't get enough revenue from users to support future growth.

For the operators, charging web companies a fee seems intensely attractive because the fee could be scaled to the amount of traffic they generate (unlike the flat-rate data plans that users prefer), forcing the web companies to use bandwidth more efficiently. It also would let operators increase their revenue without directly reducing user demand. Basically, the web companies would subsidize a shift from wired to wireless computing.


The third option

I can see why the operators are pushing on both of these options. They're in a difficult situation, and it would be very helpful to them if somebody bailed them out (link). I might be trying the same things if I worked for an operator. But there is a third option for managing cellular data overload, and it deserves to get a lot more attention:

Raise prices.

In almost every other industry in the world, you're responsible for charging enough money to support your business. Yes, sometimes you have to make investments before you know how much demand there will be, and yes, sometimes that creates a lot of risk for your company. But that's why they pay you the big bucks, Mr. or Ms. CEO.

I don't understand how we as a society came to the conclusion that wireless data should be different. Is there some religious commandment that people must be allowed to stream Netflix on the subway? Or maybe those big Cisco growth forecasts have led us to think that endless growth of mobile data is a ravenous beast that will cause immense suffering if it's not fed more bandwidth.

Baloney. If the network is overloaded, raise your prices until you either get enough money to expand the network, or you force people to use less data. If you want network bandwidth used more efficiently, show users the cost of the data they use and they'll demand more efficient apps and devices on their own.

I bet a price increase from $50 a month to $80 a month for mobile data would end the bandwidth crisis overnight.

Not only is higher pricing the simplest way to manage network overload, it's going to happen no matter what we do. Even if we give the operators all the bandwidth from the TV networks, and get the web companies to subsidize wireless service, all that will do is delay the crunch for a few years. More traffic will switch from fixed-line to wireless until once again the network saturates and prices go up. It is inevitable.


What it means

When we plan for the future of mobile, we need to be realistic, and a little bit humble, about what we can change and what we can't. We need to learn to live with the things we can't change, and focus on doing a good job of managing the things we can.

Here's my list of the things I think we can't change about mobile data because they are driven by economics and physics:

--Most data traffic will be wireless only for the last 100 feet (30 meters) that it travels from your device to the nearest hotspot (whether it's WiFi, femtocell, or something else). So we need to be careful about our terminology. Most data could well be technically "wireless" in the sense that it passes through WiFi at the end, but that is a meaningless distinction for the purposes of this article; most of it won't pass through the cellular data network.
--Most of us will continue to have some sort of broadband cable connecting to our homes and offices, or to a point very close by (like the lamp post in the street outside your window). Forget those visions of cellular replacing the wired broadband network; in the developed world it can't happen.
--The cost per-byte of cellular data will be significantly higher than the cost per-byte of wired data. The difference will be large enough that we'll be aware of it and it will alter the way we use our devices.
--Flat rate unlimited cellular data contracts will go up in cost, or will be replaced by much more variable pricing for most users. This is already underway at some operators. For example, Verizon is rumored to be about to move from $30 per month for unlimited data to a tiered plan that ranges from $30 per month for two gigabytes to $80/month for 10 gigs (link). I don't usually like consumer price hikes, but in this case the change is long overdue.
--As the relative cost of mobile data rises, most of us will use cellular data primarily as a supplement to the wired network when we're on the go. We'll become religious about turning on WiFi in our smartphones and tablets, and making sure it can connect at home and at work. Because cellular data is more expensive, many of us will try to avoid using very data-heavy apps on the cellular network.

This means cellular data use won't be carefree. That may not sound like a big difference, but in consumer terms I think it is. We've been making the assumption that cellular data can directly replace fixed-line data, just as cellular phones replaced fixed line phones for many people. "Go ahead! Use it anywhere! Be free!" But for an aggressive user of mobile data, that can't happen. Our use of cellular data is going to be much more nuanced, managed, and carefully thought out than our use of cellular voice. I think many of us will look at our cellular data budgets the same way we look at our automobile budgets. Some people will spend more, some less, but I think most of us will be aware of the cost and manage it actively.

The wired Internet will continue to set the tune. The ongoing role of fixed-line broadband means that many leading-edge web apps will continue to be designed around the capacity and responsiveness of fixed-line networks. This is another subtle but very important difference, because it means the mobile operators will continue to play catch-up to customer expectations set on the wired networks. There will be exceptions; some features of cellular data (such as location) will drive unique mobile apps. But in most application categories, rather than shaping the future of the Internet, mobile operators in the developed world will be pushed to deliver an Internet experience that evolved on fixed-line networks.


Here are the things I think we can change about cellular data:

--We can alter the share of total data traffic that moves through the cellular networks. By transferring spectrum and giving the operators other favorable treatment, we can make the overall capacity of the cellular data networks higher than it would have been otherwise. Basically, we can make the mobile operators bigger. That may delay the onset of mobile network congestion, and enable some classes of web applications to be more successful in cellular (for example, low-res video streaming). That can have a big impact on individual users and app companies. It will also have a big impact on the ultimate revenue and profitability of the mobile operators, which is why they are lobbying so hard.
--We can probably change the size of the average mobile data bill, but only temporarily. The more revenue streams we give to the operators, the more mobile data we'll probably get for a given user price. However, as I mentioned above, keep in mind that if mobile data is made cheaper, people will use more of it, which will eventually saturate the network and cause prices to rise. So any money we save on our mobile data bills will probably be temporary.
--The decisions we make in the next few years will profoundly change the economic structure of the wireless data industry. Changes in regulations and pricing rules will have a huge impact on the ability of small companies to compete with large ones in mobile, and will determine who pays for the whole thing. This could decide whether the mobile internet looks more like the wired Internet (low barriers to entry, lots of companies) or cable television (high barriers to entry, dominated by a few big players).

I think the most important thing about the three points above is that they're all driven by government regulation. The rules we set for the mobile Internet are going to determine the ultimate size of the mobile operators, how they are funded, how competition works in mobile data, and how much power is held by the various players.

That scares me. I prefer to have winners and losers in a market chosen by customer decisions, not government ones. You can't blame the mobile operators, or the big web companies like Google, for lobbying the government on these issues. But I don't think their interests are necessarily the same as the rest of the industry, let alone consumers. Also, most of the big players are driven by quarterly revenue, and in some cases they are pushing for changes that I think will help them in the short term but would actually hurt them in the long run.

I wish there were some scenario in which we could tell governments just to butt out and let the market decide, but governments are already deeply involved in allocating spectrum, and there's no practical way to undo that. So I think it's important that we all have a very thorough, open discussion of the government decisions to be made and the sort of wireless industry they'll produce.

That's what I'll cover tomorrow.

_____

In part 3, which I'll post tomorrow (link), I'll give my ideas on how we should structure the mobile data market. I'll also talk about the opportunities this new world of mobile data will create for companies in mobile. In the meantime, I welcome your comments and questions. This is a big, complex issue, and I don't pretend to have it all figured out.

The mobile data apocalypse, and what it means to you

The mobile industry is now completing a huge shift in its attitude toward mobile data. Until pretty recently, the prevailing attitude among mobile operators was that data was a disappointment. It had been hyped for a decade, and although there were some successes, it had never lived up to the huge growth expectations that were set at the start of the decade. Most operators viewed it as a nice incremental add-on rather than the driver of their businesses.

But in the last year or so, the attitude has shifted dramatically from "no one is using mobile data" to "oh my God, there's so much demand for mobile data that it'll destroy the network." A lot of this attitude shift was caused by the iPhone, which has indeed overloaded some mobile networks. But there's also a general uptick in data usage from various sources, and the rate of growth seems to be accelerating.

Extrapolating the trend, most telecom analyst firms are now producing mobile data traffic forecasts that look something like this:




The forecasts are driven by a couple of simple observations:

--Smartphones produce much more data traffic than traditional mobile phones. Cisco estimates that a single smartphone produces as much data traffic as 40 traditional feature phones. So converting 10 million people from feature phones to smartphones is like adding 390 million new feature phone users, in terms of impact on the data network. The more popular smartphones get, the busier the network becomes.

--A notebook PC generates far more traffic than a smartphone. According to Cicso, a single notebook computer generates the same data traffic as 450 feature phones. As notebook users convert to 3G-enabled netbooks and add 3G dongles to their computers, they dramatically increase the data traffic load on the network.

You can read Cisco's analysis here.

This becomes especially interesting when you look at the forecasts for growth of 3G-equipped netbooks and notebooks. Mobile operators in many countries have started subsidizing sales of those devices if you pay for a data service plan. It's an attractive deal for many people. Say your son or daughter is going off to college. Do you buy them a regular notebook computer and also pay for the DSL service to their apartment, or do you buy them a 3G data plan for about the same price as DSL and get the netbook for free?

The forecasting firm In-Stat recently predicted that by 2013, 30% of all notebook computers will be sold through mobile operators and bundled with 3G data plans (link). Notebook computer sales worldwide are about 150 million units a year, so that's 45 million new 3G notebooks a year -- or the data equivalent of adding 20 billion more feature phones to the network every year.

Jeepers.

These forecasts are producing a behind-the-scenes panic among mobile network operators. The consensus is that there's no way their networks can grow quickly enough to support all that data traffic. There are several reasons:

--They can't afford to build that much infrastructure.

--Even if they could afford the buildout, they won't have enough bandwidth available to carry all that data, even with 4G.

--Traffic-shaping techniques like tiered pricing and usage caps can't restrain usage growth enough to save them, because

--Fear of losing customers to a competitor will force them to continue to subsidize sales of 3G dongles and offer relatively generous caps in their data plans.

There are a number of projections that show the operators losing money on wireless data a few years from now, as costs continue to increase faster than revenue. The danger isn't so much that they will all go broke, but they're very afraid that they'll turn into zero-profit utilities.

Many operators now seem to be counting on WiFi as their ultimate savior. The theory is that if they can offload enough of the data traffic from their networks to WiFi base stations connected to wired networks, then maybe other measures like 4G, usage caps, and aggressive improvements to the network will let them squeak through.

It's an ironic situation. For a long time the mobile operators thought of themselves as the future lords of data communication. All devices would have 3G connections, the thinking went, and the fixed-line data carriers such as Comcast and BT would fade away just like the fixed-line voice companies are doing.

Instead, the new consensus is that we're moving to a world where the fixed-line vendors will be expected to carry most consumer data traffic for the foreseeable future. They'll provide your wireless connectivity at home and work, while the mobile network will fill in the gaps when you're on the move. The area of disagreement, of course, is who will get the majority of the access revenue. We'll let the fixed-line and mobile operators argue over that one; I want to talk about some of the other impacts of this weird new hybrid wireless world that we're heading into.

(I touched on some of this in my post on net neutrality a couple of weeks ago (link), but I want to go into more detail here.)


The brave new world of scarce mobile bandwidth

Built-in WiFi is now good. For a long time many mobile operators resisted selling smartphones with WiFi built in. They viewed WiFi networks as competitors for customer control, and wanted to prevent usage of them. Now that they see WiFi as their savior, the operators are suddenly encouraging its inclusion in phones. Don't be surprised if in the near future it becomes impossible to get a subsidized price for any smartphone that doesn't have WiFi built in.


Traffic shaping is a fact of life, and a likely source of irritation. Many mobile operators are starting to limit the performance of applications that consume the most data bandwidth (today that's mostly video and file sharing). It's already being done today, and in most cases the operators won't even tell you they're doing it, unless the government requires them to. Certain apps will just communicate more slowly, or fail altogether, when the network gets busy.

There are a couple of exceptions where operators have been more public about their traffic shaping activity. The 3 network in the UK recently announced restrictions (link). And O2 in the UK has given details on exactly which applications it restricts in its home wireless data service (link).

Current traffic shaping hasn't generated a firestorm of complaints from the average customer (as distinct from net neutrality advocates), in part because it is very hard for users to tell why a website runs slowly on a particular day. But as mobile traffic continues to increase, operators are going to find that it's cheaper to ratchet up the restrictions bit by bit rather than pay for more capacity. Eventually people will notice, and I worry that we'll end up in a situation in which the operators carefully balance out how much they can piss off their customers without creating an outright revolt. It's a lot like the way the US airline industry operates today, and it's a miserable experience for everyone involved.

What to do. There are better ways to shape traffic. I think operators should give customers more information on how much data they're using at any given time, so they can manage it themselves. Then let them make an informed decision about which apps they'll use their bandwidth on. It would be relatively simple to create an on-screen widget showing how much data is being transferred at any time, just like the signal strength and battery life indicators on today's phones.

It's also possible to create some APIs that would tell a website how much bandwidth is available to it, so the developer could adjust its features accordingly. This idea is being tossed around between web companies and operators, but I don't know how much is actually being done about it.

Combine those changes with usage-based pricing (my next point) and customers will shape their own traffic. Then there won't be any need for covert manipulation of the network.


Say hello to capped data plans. Completely unlimited wireless data plans are not sustainable long term; the economics of them just don't work. And in fact, virtually no data plans today are completely uncapped; there is almost always some fine print about the maximum amount of traffic allowed before surcharges kick in or the user is tossed off the network.

Some people are saying that the operators should go back to charging by the byte, and in some parts of the world (particularly Asia), there is a long history of per-byte pricing. But the experience in most of the world has been that per-byte pricing makes users so nervous about their expenses that they won't use data services at all.

(DoCoMo in Japan has an interesting hybrid approach (link) in which it charges per-packet until the user hits a maximum charge of about $70 per month. Additional usage beyond that cap is free. So that's capped pricing rather than capped usage. This reduces customer fear of accidentally running up a gigantic bill, but I wonder how DoCoMo prevents power users from flooding the network with traffic. Maybe there's a second, hidden cap on total usage.)

What to do. I think the right answer in most of the world is going to be flat-rate data plans in which there's a clearly-communicated cap, with tiered charges beyond that. The cap will need to be set at a level that moderate users won't ever reach, so they don't become gun-shy about data. To alleviate the fear of accidentally running up a huge bill, there will also need to be an on-device meter showing how much of the user's monthly data allocation has been used (just telling them to go look at a website is not enough; it should be on-screen). I'm told that on-screen meters like this are already being offered on netbooks by some European operators.

Today most operators are pretty up-front about communicating the data limits when a computer is connected to a mobile network. But many of them are still deceptive toward smartphone customers. AT&T's Smartphone Personal service, for example, promises the following for $35 a month:

Included Data: Unlimited; Additional data: $0 per MB

Sounds pretty straightforward. No asterisks, no fine print. But if you click on the terms of service (link), you'll find a long list of banned application types, followed by this general provision:

"AT&T reserves the right to (i) deny, disconnect, modify and/or terminate Service, without notice, to anyone...whose usage adversely impacts its wireless network or service levels or hinders access to its wireless network... and (ii) otherwise protect its wireless network from harm, compromised capacity or degradation in performance."

In other words, if the network is getting slow, they can do anything to your service, at any time, without notice.

There is also a hidden 5G per month maximum:

"If you are on a data plan that does not include a monthly MB/GB allowance and additional data usage rates, you agree that AT&T has the right to impose additional charges if you use more than 5 GB in a month."

This is not just an American problem. Orange in the UK calls its iPhone data service "unlimited," but there's a footnote saying that "unlimited" actually means 750 megabytes a month, a surprisingly low cap compared to AT&T's.

If we're ever going to collectively manage mobile network overload, we'll all need to be much more up-front about the way it operates and what a particular service plan will and won't do.


Is residential 3G really a good idea? Especially in Europe, it's common for operators to tell people that they should ditch their DSL or cable modem at home and replace it with a 3G modem. That works out well only when the network has excess capacity. As soon as the networks start to get congested, the operators will need to offload traffic to residential WiFi routers connected to DSL or cable. If those residential fixed lines have been removed, the operators can't offload.

What to do. I think this one is going to be self-limiting. Once 3G bandwidth gets scarce, the operators will realize that they can get a lot more revenue feeding data to smartphones than to PCs. The math works like this: With a given amount of bandwidth, you could support a single notebook computer and charge about $50 a month, or support 11 smartphones at $30 a month each. Hmm, $330 a month versus $50, seems like a pretty easy decision.

But there are two circumstances in which it would make sense for the operators to keep subsidizing PC sales:

1. If smartphone sales plateau. If this happens, eventually the network will catch up with demand and then there will be excess capacity for PCs; or

2. If operators can route most of the actual data traffic from PCs through WiFi connected to landlines. In this case they could sell you data plans knowing that you won't affect their networks much. That brings us to the next point...


Operators have a huge vested interest in unlocking WiFi access points. Most WiFi access points today are encrypted and inaccessible to other devices in the area. I think there's a strong financial incentive for mobile operators to work with fixed-line access companies to get those access points unlocked. The benefit for the wireless companies is clear -- the more WiFi points they can talk to, the fewer cell towers they need to build. But the benefits for the fixed-line operators are much less clear. Why should they help the mobile operators with their bandwidth crunch?

What to do. The ideal situation would be a revenue-sharing deal in which the operators share some money with the fixed-line companies to encourage them to open up access to their networks. In this scenario, your DSL or cable provider would give you a WiFi router that has been pre-configured to automatically and securely share excess bandwidth with mobile devices in the area. Your own traffic would get priority, but any extra capacity could be shared automatically. The benefit for you as a consumer would be a free router, and/or a lower DSL bill as the cable company passes along some of the revenue it gets from the mobile operators.

The effectiveness of this sort of approach is going to depend on the relative cost for an operator of subsidizing a set of WiFi base stations in an area, versus the cost of installing more wireless capacity. I wonder about weird scenarios like a DSL provider auctioning off excess WiFi capacity to wireless operators in a particularly congested area.


Femtocells for the rest of us. Another very logical step for the operators is to start pushing femtocells aggressively. (Femtocells are radios that work like a short-range cell tower, but are the size of a WiFi router. You connect one to your DSL or cable line, and it offloads traffic from the wireless network. Link)

What to do. Today femtocells are generally sold as signal boosters in areas with marginal wireless coverage. But in the future I think it may make sense for operators to give away femtocells, or at least subsidize them, for customers who live in areas where the data network is congested.


What it all means: Fixed-mobile convergence with a twist

If you step back from the details, the big picture is that we really need a single integrated data network that encompasses mobile and fixed connections, and switches between them seamlessly. People have been talking about this sort of thing for years (check out the Wikipedia article on fixed-mobile convergence here), but the focus has generally been on handing voice calls between WiFi and cellular. That's hard to do technologically (because you can't interrupt a voice conversation during the handover for more than a fraction of a second). Besides, it doesn't solve a significant customer problem -- the voice network isn't the thing that's overloaded.

The place where we could really, really use fixed-mobile convergence is in data. I'm worried, though, that the intense competition between the wireless and wired worlds will make it difficult and slow to achieve the coordination needed. This might be a useful place for government to put its attention. Not in terms of regulating the integrated network into existence (that would be the kiss of death), but to grease the skids for cooperation between the mobile and fixed-line worlds.


Just one more thing...

Everything above is based on the assumption that those Cisco and analyst forecasts are correct. But Cisco has a vested interest in hyping fear of the data apocalypse (Emergency! Buy more routers now!!), and my general rule about tech analysts is that every time they all agree on something you should bet against them.

There is a genuine crunch in mobile data capacity going on at the moment; you can read about network outages caused by the iPhone even today. And I can assure you that for every network failure you read about, there are dozens of other failures and near-failures that don't get reported. Many wireless data networks are very stressed.

And the situation will get worse.

But there's no such thing as infinite demand. At some point the growth of mobile data will slow down, and it's very important to try to estimate how and when that'll happen, so we as an industry do not overshoot too badly. The question isn't whether the growth forecasts are wrong, it's when they will be wrong.

I'll write about that next week...

A web guy and a telecom guy talk about net neutrality

It was a nondescript bar in the American Midwest, the sort of place where working men drop in at the end of the day to unwind before they head home. You wouldn't expect to find two senior business executives there, and as I sat in the empty bar at midday I wondered if maybe my contact had given me a bad lead. But then the door opened and a general manager from one of the leading web companies walked in, followed by a senior VP from one of the US's biggest mobile network operators. I hunched down in the shadows of a corner booth and typed notes quietly as they settled in at the bar.

Bartender: What'll you have?

Telecom executive: Michelob Light.

Web executive: I'll have a Sierra Nevada Kellerweis.

Bartender: Keller-what?

Web executive: Um, Michelob Light.

Telecom executive: Thanks for coming. Did you have any trouble finding the place?

Web executive: All I can say is thank God for GPS. I've never even been on the ground before between Denver and New York.

Telecom executive: I wanted to find someplace nondescript, so we wouldn't be seen together. The pressure from the FCC is bad enough already, without someone accusing us of colluding.

Web executive: No worries, my staff thinks I'm paragliding in Mexico this weekend. What's your cover story?

Telecom executive: Sailboat off Montauk.

Web executive: Sweet. So, you wanted to talk about this data capacity problem you have on your network...

Telecom executive: No, it's a data capacity problem we all have. Your websites are flooding our network with trivia. The world's wireless infrastructure is on the verge of collapse because your users have nothing better to do all day than watch videos of a drunk guy buying beer.

Web executive: Welcome to the Internet. The people rule. If you didn't want to play, you shouldn't have run the ads. Remember the promises you made? "Instantly download files. Browse the Web just like at home. Stream HD videos. Laugh at an online video or movie trailer while travelling in the family car."

Telecom executive: That was our marketing guys. They don't always talk to the capacity planners. Besides, who could have known that the marketing campaign would actually work?

Web executive: Don't look at me. I've never done a marketing campaign in my life. I think you should just blame it on A--

Telecom executive: You promised, no using the A-word.

Web executive: Sorry. But I still don't see why this is a problem. Just add some more towers and servers and stuff.

Telecom executive: It's not that simple. The network isn't designed to handle this sort of data, and especially not at these volumes. Right now our biggest problem is backhaul capacity -- the traffic coming from the cell towers to our central servers. But when we fix that, the cell towers themselves will get saturated. Fix the towers and the servers will fall over somewhere. It's like squeezing a balloon. We have to rebuild the whole network. It's incredibly expensive.

Web executive: So? That's what your users pay you for.

Telecom executive: But most of them are on fixed-rate data plans. So when we add capacity, we don't necessarily get additional revenue. It's all expense and no profit. At some point in the not-too-distant future, we'll end up losing money on mobile data.

Web executive: Bummer.

Telecom executive: More like mortal threat. Fortunately, we've figured out how to solve the problem. The top five percent of our users produce about 50% of the network's total traffic. So we're just going to cap their accounts and charge more when they go over.

Web executive: Woah! Hold on, those are our most important customers you're talking about. You can't just shut them down.

Telecom executive: The hell we can't. They're leeches using up the network capacity that everyone else needs.

Web executive: Consumers will never let you impose caps. You told them they had unlimited data plans, that's the expectation you set. You can't go back now and tell them that their plans are limited. They won't understand -- and they won't forgive you.

Telecom executive: First of all, the plans were never really unlimited in the first place. There's always been fine print.

Web executive: Which no one read.

Telecom executive: Off the record, you may have a point. On the record, the fact is that you can retrain users. Look, you grew up in California, right?

Web executive: What does that have to do with anything?

Telecom executive: Once upon a time, there weren't any water meters in California. Now most of the major cities have them, and they'll be required everywhere in a couple of years. Something that was once unlimited became limited, and people learned to conserve.

Web executive: The difference is, I can read my water meter. You make a ton of money when people exceed their minutes or message limits, and you don't warn them before they do it. If you play the same game with Internet traffic, it'll scare people away from using the mobile web -- or worse yet you'll invite in the government. Look what happened with roaming charges in Europe.

Telecom executive: Jeez, don't even think about that. Okay, so we'll need to add some sort of traffic meter so people will know how much data they're using when they load a page.

Web executive: Great, that'll discourage people from using Yahoo.

Telecom executive: Huh?

Web executive: Oops, did I say that out loud?

Telecom executive: Then there's the issue of dealing with websites and apps that misuse the network.

Web executive: Not this again.

Telecom executive: I'm not talking about completely blocking anything, just prioritizing the traffic a little. Surely you agree that 911 calls should get top priority on the network, right?

Web executive: Of course.

Telecom executive: And that voice calls should take priority over data?

Web executive: I don't know about that.

Telecom executive: Oh come on, what good is a telecom network if you can't make calls on it?

Web executive: (sighs) Yeah, okay.

Telecom executive: So then what's wrong with us prioritizing, say, e-mail delivery over video?

Web executive: Because when you start arbitrarily throttling traffic, I can't manage the user experience. My website will work great on Vodafone's network but not on yours, or my site will work fine on some days and not on others. How do you think the customers will feel about that?

Telecom executive: Not as angry as they will be if the entire network falls over. Listen, we're already installing the software to prioritize different sorts of data packets. We could be throttling traffic today and you wouldn't even know it.

Web executive: But people will eventually figure it out. They'll compare notes on which networks work best and they'll migrate to the ones that don't mess with their applications. Heck, we'll help them figure it out. And if that's not enough, there's always the regulatory option. The Republicans are out of office. They can't protect you on net neutrality any more.

Telecom executive: You think you're better at lobbying the government than we are? We've been doing it for 100 years, pal. Besides, we have a right to protect our network.

Web executive: You mean to protect your own services from competition!

Telecom executive: Parasite!

Web executive: Monopolist!

Telecom executive: That's it! It's go time!

They both stood. The telecom guy grabbed a beer bottle and broke it against the bar, while the web guy raised a bar stool over his head. Then the bartender pulled out a shotgun and pointed it at both of them.

Bartender: Enough! I'm sick of listening to you two. Telecom guy, you're crazy if you think people will put up with someone telling them what they can and can't do on the Internet. The Chinese government can't make that stick, and unlike them you have competitors.

Web executive: See? I told you!

Bartender: Shut up, web guy! You keep pretending that the wireless network is infinite when you know it isn't. If you really think user experience is important, you need to start taking the capabilities of the network into account when you design your apps.

Web executive: Hey, he started it.

Telecom executive: I did not!

Bartender: I don't care who started it! Telecom guy, you need to expose some APIs that will let a website know how much capacity is available at a particular moment, so they can adjust their products. And web guy, you need to participate in those standards and use them. Plus you both need to agree on ways to communicate to a user how much bandwidth they're using, so they can make their own decisions on which apps they want to use. That plus tiered pricing will solve your whole problem.

Telecom executive: Signaling capacity too. Don't forget signaling.

Bartender: That's exactly the sort of detail you shouldn't confuse users with. Work it out between yourselves and figure out a simple way to communicate it to users. Okay?

Web executive: I guess.

Telecom executive: Yeah, okay.

Bartender. Good. Now sit down and start over by talking about something you can cooperate on.

Telecom executive: All right. Hey, what's that guy doing in the corner? Is that a netbook?

Web executive: He's a blogger!

Bartender: There's no blogging allowed in here!

Telecom executive and web executive: Get him!

I ran. Fortunately, the bar had a back door. Even more fortunately, the web guy and the telecom guy got into an argument over who would go through the door first, and I was able to make my escape.

So I don't know how the conversation ended. But I do know that I wish that bartender was running the FCC.

Everything you always wanted to know about web community, and then some

It's been a long time since I posted, and I apologize. In addition to doing a bunch of work for clients at Rubicon, I have been preoccupied with a big strategy project we just finished, on online community. We released the results today, and I think you might find them interesting, so here's a summary.

In our strategy work with tech companies, we're frequently asked about web communities -- how they operate, what they can and can't do, and how a company should work with them. The companies we deal with generally fall into three camps when it comes to community:

--Many companies are still learning about online community and don't know what to do or what to expect.

--Some companies have already tried some online community activity, but were disappointed -- often because they attracted only a few enthusiasts rather than the masses of end users they expected.

--And of course some companies run successful web communities, either as a sideline or as their core business. They're very hungry for information on how other communities operate, and insights on what they could do better.

To help deal with all of those questions, we conducted a study of online community in the US. We surveyed more than 3,000 US web users on their overall internet usage, and then dived deep on their use of online communities and what impact those communities have on their lives. I know some of you will be disappointed that the survey is US-only; we'd be delighted to repeat the research in other parts of the world if anyone wants to sponsor it ;-)

Meanwhile, here are some of the key things we learned:

--Small groups of enthusiasts dominate most online conversations, but that doesn't mean online communities matter only to a narrow segment of people. Most web users read community content rather than contributing to it, and are strongly influenced by the things they see there, especially product reviews and recommendations. Those reviews are now second only to word of mouth as a purchase influencer for web users.

--Because most web users are voyeurs more than contributors, if you're running an online discussion, you should think of it as theatre -- it's a performance in which the community leader(s) interact with a small group of contributors for the education and amusement of the rest of us. All the web's a stage, but we're not all players in it. At least not equal players.

--This means companies that turn away from web communities because they're populated mostly by enthusiasts are missing the point. They've mistaken their fellow actors for the audience. If you're running a community, you need to take care of the active participants in a community so that the rest of the audience will watch and learn.

--If a company needs more incentive to work with the Internet, it turns out that the web has also become the number two source of product support information for web users. After checking the manual, web users are more likely to check a company's website for information or search the web than they are to take traditional steps like calling the manufacturer or asking a dealer. That will be comforting to many companies that want to reduce their support costs, since phone support is very expensive. But how many of those companies have bothered to put detailed support information online, and make sure it's well indexed by the search engines?

--There are an enormous number of tidbits in the study regarding web use. A few items that stood out to me include:

• -About a quarter of US web users say they have dated someone they first met online.
• -Although Twitter and SecondLife get a lot of press, their audiences are very narrow when you compare them to major social sites like MySpace, Facebook, and even LinkedIn.
• -Yahoo is the second most valued website in the minds of US web users, after Google. It's ahead of major web properties like YouTube and Facebook. All of the negative press about Yahoo sometimes makes people forget how strong its user base is.
  
• -The major social networks are much more satisfying and useful to teens than they are to adults. In fact, satisfaction with the social sites declines steadily after age 14.
• -Since we're coming up on an election in the US, it's interesting to look at political ties on the Internet. Democrats are more active online than Republicans, and say the web has a greater influence on their behavior, including voting.
• -Young people dominate online conversations, with people 22 and under producing about half of all user-generated content and comments. So if you sometimes feel like you're dealing with kids online, it may be because you are.

A full report on the findings is available in PDF format here (link). I know many people don't like to read PDFs online, and besides you can't easily comment on them or link to sections in them. So we've also posted the report online, cut into several sections for easy reading:

Part one summarizes the report, and gives detailed information on the use of community online, and what that means for business (link). This is the section that gives information on community usage rates and the "most frequent contributors" who dominate online conversations.

Part two discusses the leading web destinations in the US, measured in several ways, and discusses the role of community in them (link). This is where you'll find learn about the remarkable membership rate of Classmates.com, the #3 social community site in the US when measured by profiles. We also answer questions like: "Is Facebook more popular than MySpace?" "What do web users value more, Wikipedia or NYTimes.com?" and "What are the top five most-visited types of website?"

Part three talks about the role of community sites in the social lives of Americans (link). This is where we compare Republicans and Democrats online, and look at how different age groups use the social networking sites. We answer questions like: "How many young people lie about their age to get access to websites?" and "How many web users create fake identities online?"


What does it mean to mobile?

I always get that question when I post something that's not directly related to the mobile industry. In part, my answer is that I think this information's useful to anyone who's interested in technology. You're soaking in the internet, and it's good to understand how it works.

But community users don't limit themselves to only PC access, so online community will have a big effect on the mobile industry. It's important to understand what's really happening in the wired internet so you can sort out which mobile web opportunities have the best prospects. For example, I saw a presentation by someone senior at a mobile company the other day, and he was touting the importance of Twitter as a driver for the mobile web. I almost laughed out loud, because I had just been working on our report, and I knew how tiny the Twitter audience is. It's a classic case of people in the tech industry assuming something they use a lot is also being adopted by the masses.

That's not to beat up on Twitter specifically; they have a great base of users. But mainstream they ain't.

The other thing mobile people should think about a lot is the huge role that young people play in the generation of online content. As a group they are vastly more active online than older people. We aren't sure exactly what the cause is. Some of it may be that high school and college students just have a lot more time on their hands, and they spend some of it posting to the web. But probably also some of it is generational. Whatever the cause, it's likely that young users will be some of the heaviest drivers of use of the mobile web, especially the uploading of content and comments.

Not that Apple needs another advantage, but that probably plays to the strengths of the iPhone, because Apple has such a good franchise with young people in the US. It also helps to explain why RIM is trying to reach out to young people. In Europe, I think Nokia and SonyEricsson have a better chance at those users, because their brands are strong with young people. But they'll need the right products as well, which is a subject for other posts...

The three laws of technology strategy

The other day when I was writing about the fate of mobile apps (link), I mentioned one of the laws of technology strategy. It made me realize that although we in the industry talk about those laws all the time, I've never seen them all written down in one place. There are probably more than three laws, but these are my favorites. Please post a comment if you want to add some more.

Here we go, twenty years of industry experience boiled down to three lines:


1. An elegant business model paired with mediocre technology beats an elegant technology paired with a mediocre business model.

To put it another way, if you create a marvelous tech product that has no way of making money, you get a long and passionate entry on Wikipedia. If you create a lousy tech product that prints money, you get to be Bill Gates.

Windows is the best case study here, but this one has been proven over and over again in the history of the tech industry. But companies keep tripping over it because they're often run by engineers who have been trained to value technical elegance as an end in itself.

Don't get me wrong, elegance is great. The most wonderful tech companies are those that combine elegant products and great business models. But you must pay the bills or you don't get to keep playing. And wads of money can buy a lot of patches and kludges.


2. Design for a need, not a desire.

A serial entrepreneur once expressed this to me nicely: "I focus on aspirin issues." In other words, if someone has a serious enough problem that they feel pain, they'll be much more likely to pay money for an answer. (I wish I could remember who told me that -- I'd like to credit him by name.)

Very often tech companies will fall in love with a concept that is compelling to people in the company, but not to non-technologists. They'll convince themselves that people will want it because, well, they ought to want it.

A related problem: A company will come up with a product that's nice, but doesn't really address an aspirin problem. You know you have this problem when someone in the company says that need a marketing campaign to explain to people why they should want the product. The really good products need marketing for visibility, not persuasion.

I think this is the underlying problem behind most failed web applications. They do something interesting, as opposed to something compelling.

What makes this whole problem especially tough is that you can't just ask customers what they need. They aren't engineers, they don't understand what you could build. All they'll ask you for is improvements on the products they already have today. What you have to do is get inside the customers' heads, understand how they live, and figure out what you could do to improve their lives. That's what the best product managers do.


3. Software designed for one platform usually fails on another.

We teach this one to ourselves every time the industry goes through a platform transition, and then we promptly forget it again:

A computing platform isn't just a technology, it's a mindset, with a huge set of unstated assumptions about customers and business practices attached to it. When you port software from one platform to another, you take those assumptions along with you, and usually they don't fit.

This is why the software leaders in one generation of computing usually fail in the next generation. Check it out -- which software products led in the DOS world? Lotus, WordPerfect, Ashton-Tate. Did any of them thrive in the Windows/Mac world? Nope.

Then did the software leaders in Windows/Mac -- Adobe, Microsoft, Symantec, Intuit -- dominate in the Internet? Nope, the new startups without the mental baggage dominated.

Which leads to an interesting question: Do you think the leaders of mobile Internet will be the same companies that led the PC Internet? Or is the next Adobe/Lotus/Google a little startup out there, rethinking what it means to be connected in a mobile setting?

Think about it.

Mobile applications, RIP

Summary: The business of making native apps for mobile devices is dying, crushed by a fragmented market and restrictive business practices. The problems are so bad that the mobile web, despite its many technical drawbacks, is now a better way to deliver new functionality to mobiles. I think this will drive a rapid rise in mobile web development, largely replacing the mobile app business. This has huge implications for mobile operators, handset companies, developers, and users.


The decline of the mobile software industry

Mobile computing is different from PC computing.

For the last decade, that has been the fundamental rule of the mobile data industry. It was the central insight of Palm Computing's "Zen of Palm" philosophy. Psion came up with similar ideas, and you can hear echoes of them from every other successful mobile computing firm: Mobile computers are used differently from PCs, and therefore must be designed differently.

We all assumed this also meant mobile devices needed a whole mobile-specific software stack, including an operating system and APIs designed specifically for mobility, and native third-party applications created from the ground up for mobile usage.

That's what we all believe, but I'm starting to think we got it wrong.

Back in 1999 when I joined Palm, it seemed we had the whole mobile ecosystem nailed. The market was literally exploding, with the installed base of devices doubling every year, and an incredible range of creative and useful software popping up all over. In a 22-month period, the number of registered Palm developers increased from 3,000 to over 130,000. The PalmSource conference was swamped, with people spilling out into the halls, and David Pogue took center stage at the close of the conference to tell us how brilliant we all were.

It felt like we were at the leading edge of a revolution, but in hindsight it was more like the high water mark of a flash flood. In the years that followed, the energy and momentum gradually drained out of the mobile applications market.

The problem wasn't just limited to Palm; the level of developer activity and creativity that we saw in the glory days of Palm OS hasn't reappeared on any mobile platform since. In fact, as the market shifted from handhelds to smartphones, the situation for mobile app developers has become substantially worse.

That came home to me very forcefully a few days ago, when I got a call from Elia Freedman. Elia is CEO of Infinity Softworks, which makes vertical market software for mobile devices (tasks like real estate valuation and financial services). He was one of the leaders of the Palm software market, with a ten year history in mobile applications.

I eventually moved on from Palm, and Elia branched out into other platforms such as Blackberry. But we've kept in touch, and so he called recently to tell me that he had given up on his mobile applications business.

Elia gave me a long explanation of why. I can't reproduce it word for word (I couldn't write that fast), but I've summarized it with his permission here:

Two problems have caused a decline the mobile apps business over the last few years. First, the business has become tougher technologically. Second, marketing and sales have also become harder.

From the technical perspective, there are a couple of big issues. One is the proliferation of operating systems. Back in the late 1990s there were two platforms we had to worry about, Pocket PC and Palm OS. Symbian was there too, but it was in Europe and few people here were paying attention. Now there are at least ten platforms. Microsoft alone has several -- two versions of Windows Mobile, Tablet PC, and so on. [Elia didn't mention it, but the fragmentation of Java makes this situation even worse.]

I call it three million platforms with a hundred users each (link).

The second technical issue is certification. The walls are being formed around devices in ways they never were before. Now I have to certify with both the OS and with each carrier, and it costs me thousands of dollars. So my costs are through the roof. On top of that, the adoption rate of mobile applications has gone down. So I have to pay more to sell less.

Then there's marketing. Here too there are two issues. The first is vertical marketing. Few mobile devices align with verticals, which makes it hard for a vertical application developer like us to partner with any particular device. For example, Palm even at its height had no more than 20% of real estate agents. To cover our development costs on 20% of target customer base, I had to charge more than the customers could pay. So I was forced to make my application work on more platforms, which pushed me back into the million platforms problem.

The other marketing problem is the disappearance of horizontal distribution. You used to have some resellers and free software sites on the web that promoted mobile shareware and commercial products at low or no charge. You could also work through the hardware vendors to get to customers. We were masters of this; at one point we were bundled on 85% of mobile computing devices. We had retail distribution too.

None of those avenues are available any more. Retail has gone away. The online resellers have gone from taking 20% of our revenue to taking 50-70%. The other day I went looking for the freeware sites where we used to promote, and they have disappeared. Hardware bundling has ended because carriers took that over and made it impossible for us to get on the device. Palm used to have a bonus CD and a flyer that they put in the box, where we could get promoted. The carriers shut down both of those. They do not care about vertical apps. It feels like they don't want any apps at all.

You can read more of Elia's commentary on his weblog (link).

Add it all up, and Elia can't make money in mobile applications any more. As he told me, "Mike, it's time for you to write the obituary for mobile apps." More on that later.

Although it's a very sad situation, if Elia's experience were an isolated story I'd probably just chalk it up to bad luck on the part of a single developer. But it mirrors what I've been hearing from a lot of mobile app developers on a lot of different operating systems for some time now. The combination of splintering platforms, shrinking distribution channels, and rising costs is making it harder and harder for a mobile application developer to succeed. Rather than getting better, the situation is getting worse.

I've always had faith that eventually we would solve these problems. We'd get the right OS vendor paired with a handset maker who understood the situation and an operator who was willing to give up some control, and a mobile platform would take off again. Maybe not Palm OS, but on somebody's platform we'd get it all right.

I don't believe that any more. I think it's too late.


The mistake we made

We told ourselves that the fundamental rule of our business was: Mobile is different. But we lost sight of an even more fundamental law that applies to any computing platform:

A platform that is technically flawed but has a good business model will always beat a platform that is elegant but has a poor business model.

Windows is the best example of inelegant tech paired with the right business model, but it has happened over and over again in the history of the tech world.

In the mobile world, what have we done? We created a series of elegant technology platforms optimized just for mobile computing. We figured out how to extend battery life, start up the system instantly, conserve precious wireless bandwidth, synchronize to computers all over the planet, and optimize the display of data on a tiny screen.

But we never figured out how to help developers make money. In fact, we paired our elegant platforms with a developer business model so deeply broken that it would take many years, and enormous political battles throughout the industry, to fix it -- if it can ever be fixed at all.

Meanwhile, there is now an alternative platform for mobile developers. It's horribly flawed technically, not at all optimized for mobile usage, and in fact was designed for a completely different form of computing. It would be hard to create a computing architecture more inappropriate for use over a cellular data network. But it has a business model that sweeps away all of the barriers in the mobile market. Mobile developers are starting to switch to it, a trickle that is soon going to grow. And this time I think the flash flood will last.

If you haven't figured it out yet, I'm talking about the Web. I think Web applications are going to destroy most native app development for mobiles. Not because the Web is a better technology for mobile, but because it has a better business model.

Think about it: If you're creating a website, you don't have to get permission from a carrier. You don't have to get anything certified by anyone. You don't have to beg for placement on the deck, and you don't have to pay half your revenue to a reseller. In fact, the operator, handset vendor, and OS vendor probably won't even be aware that you exist. It'll just be you and the user, communicating directly.

Until recently, a couple of barriers prevented this from working. The first was the absence of flat-rate data plans. They have been around for a while in the US, but in Europe they are only now appearing. Before flat-rate, users were very fearful of exploring the mobile web because they risked ending up with a thousand-Euro mobile bill. That fear is now receding. The second barrier was the extremely bad quality of mobile browsers. Many of them still stink, but the high quality of Apple's iPhone browser, coupled with Nokia's licensing of WebKit, points to a future in which most mobile browsers will be reasonably feature-complete. The market will force this -- mobile companies how have to ship a full browser in order to keep up with Apple, and operators have to give full access to it.

There are still huge problems with web apps on mobile, of course. Mobile web apps don't work when you're out of coverage, they're slow due to network latency, and they do not make efficient use of the wireless network. But I believe it will be easier to resolve or live with these technical drawbacks in the next few years than it will be to fix the fundamental structural and business problems in the native mobile app market.

In other words, app development on the mobile web sucks less than the alternative.

Here's a chart to help explain the situation. Imagine that we're giving a numerical score to a platform, rating its attractiveness to developers. Attractiveness is defined as the technical elegance of the platform multiplied by how easy it is for developers to make money from it. The attractiveness score for native mobile app development looks like this over time:



This is why mobile app developers are in trouble. Even though the base of smartphones has been growing, and the platforms themselves have become more powerful, the market barriers have been growing even faster. So attractiveness has been dropping.

Now add in mobile web development:



Based on what I'm hearing from mobile developers, the lines just crossed. The business advantages of mobile web development outweigh its technical limitations. More importantly, if you look at where the lines are going, the advantage of mobile web is going to grow rapidly in the future.

I'm not saying all native mobile development is dead. In fact, we're about to see the release of Apple's native development tools for the iPhone, and as Chris Dunphy just pointed out to me, they are sure to result in a surge of native development for that platform. But I think even a rapidly-growing base of iPhones can't compare to the weight of the whole mobile phone market getting onto a consistent base of browsers.


What it all means

If you're a mobile developer, you should consider stopping native app development and shifting to a mobile-optimized website. That's what Elia did, and he said it's amazing how much easier it is to get things done. Even mobile game developers, who you'd think would be the last to abandon native development, are looking at web distribution (link; thanks to Mike Rowehl for pointing it out).

See if you can create a dumbed-down version of your application that will run over the mobile web. If the answer is yes, do it. If the answer is no, try to figure out what technology changes would let you move to the web, and watch for those changes to happen.

There are exceptions to any rule, and I think it makes sense to keep doing native development if your app can't work effectively over the web, and it's a vertical application so popular that you can get about $50 or more in revenue per copy. In that situation, you probably have enough resources to stay native for the time being. But even you should be monitoring the situation to see when you can switch to the web, because it will cut your expenses.

If you're a mobile customer, make sure your next smartphone has a fully functional browser that can display standard web pages. And get the best deal you can on a flat-rate data plan; you'll need it.

If you're an operator or a handset vendor, get used to life as a dumb pipe. By trying to control your customers and make sure you extract most of the revenue from mobile data, all you've done is drive developers to the Web, which is even harder to control. You could have had a middle ground in which you and mobile developers worked together to share the profits, but instead you've handed the game to the Google crowd.

Congratulations.


Oh, about that obituary...

In loving memory of the mobile applications business. Adoring child of Java, Psion, Palm OS and Windows Mobile; doting parent of Symbian, Access Linux Platform, and S60; constant companion of Handango and Motricity. Scared the crap out of Microsoft in 2000. Passed away from strangulation at the hands of the mobile industry in 2008. Awaiting resurrection as a web service in 2009. In lieu of flowers, the family asks that you make a donation to the Yahoo takeover defense fund.