So much of the world seems to imagine that in each technology company, tucked somewhere, neatly in the corner of safe, there lies their cherished “intellectual property.”

The magic scroll, the secret formula, the special code, the transformative prototype — a boon which, if somehow, released to their competitors would instantly transform their lead into a trivial advantage.

Far be it from me to underestimate the value of an idea, but I think the error that most are making here is in underestimating the effort required in actually making something work. More than three years ago, I had the central ideas for what became LightSail Energy. Over that period of time, I and my extraordinarily talented colleagues have invested, collectively, the greatest efforts of our careers into developing the product and the understanding necessary to make it, a process that involved hundreds of experiments, thousands of decisions, and tens of thousands of tasks. We have almost two years ahead of us before our first product even ships.

One can not simply steal the kind of knowledge and expertise so developed. The momentum developed by one technical group cannot be simply transplanted into a competitor, it transcends documentation. No — it resides in our greatest assets: our people, the minds we’ve trained, the conventional wisdom we’ve transformed, the reputation that, through our trials, we have deeply entrenched.

While startups are growing, while their greatest advances and products are in their future, they need scarce not worry about a competitor stealing their work or ideas. Startups are in a race against time, not others.

It is even difficult to get people to steal your ideas. If only it were so easy to change the world.

Theft

This whole picture changes once a product is released, once made whole.

A novel may be simply transcribed, code copied. An engine, reverse engineered. Once there’s proof that something works, it’s easy for some to imagine that they could simply copy it.

But it takes much more than an opportunistic interest to bring most things to market. Even as Facebook began their long ascent, they feared that Google, or someone else who knew what they were doing, would just make their product. “And look how long it took them!” Mark Zuckerberg exclaims.

Google couldn’t simply steal Facebook, even though they knew how it worked, even though they had access to the clientside code. There was a barrier — the users and data of their growing population. And if one is going to simply copy something, one might as well try to improve upon it. The same urge to simply copy a work now becomes a stroke of inspiration.

The crucial factor? The common thief is lazy, and the lazy thief is thwarted. As you see, the thief is rarely a person of great motivation, excepting for personal vendettas. If other victims make better targets, one is safe. If all victims defend themselves more vigorously than the notion of honest work in the lazy thief’s mind, an honest society becomes an inevitability.

Theft != Transcription != Transformation != Inspiration

We must distinguish these concepts. There are simple semantic distinctions that our law and policy makers continuously evade. But they are incredibly important.

• Theft implies that the rightful person, an owner is denied access to something of value.
• Transcription is simply the copying of something, leaving the original intact.
• Transformation is the manipulation of a work into something of a different essential quality, message, or utility.
• Inspiration is a transformation less direct, acted upon through the medium of our imagination.

The great danger of laws that ignore these is not that they will prevent theft, but that they will so heavyhandedly prevent transformation and inspiration: the engines of our entire civilization.

Copyright has its merits, but most importantly, compared to patents, it induces limited collateral damage. Authors are protected, by property laws and window locks, from the most egregious of violations, theft, and by copyright, from commercial and transcriptions of their works, which might, it could be said, constitute theft of the market for their authorship.

Where copyright is dangerous is where it spreads. It spreads to non-commercial sharing of fragments, in music, criticism, and art, to the use and transformation of fragments. It spreads to the prevention of the dissemination of works to those who cannot pay, wouldn’t pay, could never pay. It prevents even the growth of the stature of the artist: it dramatically tips the balance of power of such industries away from the artist to those with the organizational resources to enforce their copyright monopoly on others: in music, the record labels, in movies, Hollywood, in science, Reed-Elsevier, Springer-Verlag. Their corporate lobbies will declare that they are protecting the artist, but in reality, the artist is humanized. The artist plays only a small role: second fiddle to the giant, thrumming machines of distribution, promotion, copyright enforcement, and market analysis in the publication industry’s leviathan mass.

What does this do?

Musicians, actors, filmmakers, and authors are enthralled to the callous calculations of multinational corporations, by structure insensitive to the local, cultural sensibilities that artists wish to convey. Those artists outside of a mature mass market industry where the promotion machine, defended by copyright, can create hits by bulldozing over works of artistic merit, are steadily seduced by those monied coffers: sell-out or be squeezed out. Indie artists are remarkable for their resilience and their art, but also for their poverty.

Remixes are prevented. The sampling of other’s work is believed to be theft. Internet services engaged in the promotion of new works are embargoed by those entities enforcing copyright. The remixing of footage of ten thousand films, which, as YouTube amply demonstrates, are deemed not inspiration nor transformation but the acts of criminals. And scientists, like me, outside of academia, outside of institutions which can mindlessly purchase the scientific journals of highest repute, are systematically shut out of the products of academic scientists, the works of public investment, which should rightly be the domain of everyone. Scientists are seduced by the well defended and financially supported reputations of journals in much the same way as artists are seduced by the distribution and glamor of the labels and studios, as models are seduced into posing and surrendering their image, for their glamor, their paltry salary, their many admiring eyes, their fame.

A better system, one could imagine, would draw the distinction between the theft of property, the theft of market share or opportunity, and the transcription or transformation implied by these other examples. A remix scarcely steals the market of the original work unless unreferenced. An immigrant entrepreneur scarcely steals the jobs of the natives unless they hire only immigrants as well — and even then this is unclear.

The heavyhanded application of copyright law is tantamount to the mislabeling of transcription or transformation as theft. If we are to grow as a knowledge economy, we must not commit such a grave error.

But all of these problems pale in comparison to the collateral damage done by the patent system.¹

Criminalizing Inspiration

It is a peculiar feature about a patented invention that it need not actually work.

It need not actually satisfy any needs.

It need not be, on its own, economically viable.

It need not ever been intended to be reduced to practice, nor spread out into the world.

It is an even more peculiar feature of patents that they do not grant you any rights, that is, other than that of taking away rights.

Rights to use of equipment that you own.

Rights to a methodology of medical practice.

Rights to manufacture or sale or application of an invention.

But most importantly, rights to inventions that you neither described nor anticipated, but that some aspect of your patent, another invention happens to incorporate.

Even if your patent discusses only the barest of sketches, and all of the hard work, and the vast majority of the good ideas necessary, were the result of other minds, whether independently, or by inspiration arising from the original work.

The patent system, then, makes a terrible sacrifice. Our physical property laws protect our stuff. Copyright, to a great extent, protects our creative, transcribable works. But patent law, in shoring up the defenses against these other violations, ‘protects’ us against, and criminalizes, inspiration.

But the patent system continues to grow. Business model patents. Medical patents. Use patents. Design patents. Continuations, and continuations in part. Nations even measure their inventive efforts by their cumulative accretion of patent applications. The scope of this heavyhanded mechanism continues unrelentingly, and unrepentant, chanting their mantra “We are protecting our ideas. Ideas have value.”

Ideas do have value. Great value. But the value of inspiration, of innovation, of allowing someone to make an improvement on an unfinished, or incompletely adapted idea, and bring it out into the world, is far greater.

If the ideas for stories could be patented, modern artists as great as J.R.R. Tolkien, George Lucas, and Steven Spielberg would have been sued as derivative.  The hero’s journey deeply underlies many of their works, in many forms. And who would have patented the love song?

SOPA – The Thermonuclear Option

The absolute misapprehension of these semantic differences, and the total disregard for collateral damage, in the past months reached a fever pitch with the introduction of the SOPA or Stop Online Piracy Act. It seemed as if everything that could be wrong with it, was.

True, as its proponents claim, it would give the corporate copyright and distribution monopolists one more tool to prevent sharing from degrading their dying business model.

But in a SOPA world, if one person shares one element and one corporation makes one complaint, then in one moment with zero due process and zero transparency, a website can be blocked, and the possibility for any transcription, transformation, or inspiration destroyed.

But not just for the offending material. For everything.

Share, once, the wrong content to Wikipedia, and the entire project, the greatest encyclopedia of all time, one of the greatest efforts of all of civilization, is threatened with extinction.

SOPA has been prevented — so far. But what halted process was that the ‘technical’ aspects of the internet confused our lawmakers. It is deeply disturbing that it was not the semantic distinctions between theft and inspiration, or the threat of inordinate collateral damage, that halted the efforts of SOPAs proponents. It makes one fear their judgments in other matters equally.

While it is tempting to make an analogy to our current middle eastern conflict, it would not, in truth, reflect our military operations adequately. The military aspired to surgical precision. Predator drones. Counter-insurgency tactics.

SOPA represents a different stance. To threaten wikipedia with destruction is to threaten to vaporize the nations thought to harbor Osama bin Laden. SOPA is absolutely the thermonuclear option. It, and the efforts behind it, must be stopped.

What this Means for Startups

Do not be threatened by others copying your idea. Do not even be threatened by others copying an unfinished product. They cannot copy you, nor the imagined futures in your head, nor the organization that you’ve built, nor the reputation you’ve gained.

Your job is to create something wonderful, get it out in the world, and make it so convenient and clear that you should be the one to buy from, that you should be the one to trust, that hardly anyone would attempt to compete with you. iTunes costs money, but is so superior an experience to Kazaa that hardly anyone would choose the latter.

Once you’ve released your product, your goal is to stay ahead of it. To improve it, refine it, and when the time comes, to supersede it — to have the success of your past project propel you into the next.

On rare occasions, a work or invention may operate, its works hidden, for the relevant time period of the interest of its creator. A high tech company might build their product in China, but integrate a single element, hard to make, hard to understand, at headquarters on american shores. A piece of software might require a special key; a chemical process an essential catalyst.

A business might hold a monopoly over these trade secrets for as long as they can, perhaps to wring continued business benefits out of it. This may provide some advantage.

But it will not last. At best, it will buy you time. And at worst, keeping secrets will hamper your own work; your story, your promotion, and all the internal communication of the company. Communication is hard enough when people are open and honest. Operating on need-to-know bases is torture — you don’t know what you need to know. Worst of all, it will keep you in the past — a cruel death to the innovative spirit, and a poor trade for a temporary advantage over a determined competitor.

Trade Trade Secrets

This risks of people discovering the secrets of your work are, frankly, almost always overstated, and the advantages of sharing, truly underrated.

We live in a global world. Interested, helpful parties can emerge from any of its corners. The more that you share, and the clearer that you make it, the further your reach. Helpful parties from any corner can bring gifts, information, criticism, or their own efforts. So much of what we now are at LightSail emerged from the people who over time approached us, fascinated by our mission.

We have secrets, of course. But it is impossible to track them all without hampering every conversation. So we will stay open. Not wide open — not exhibitionist — we can’t spend all of our time showing the world who we are and what we do, but open. We will let the conversation flow. And just as often as we share what we’re doing, people share amazing ideas of their own.

So don’t just keep trade secrets. Trade them.

Footnotes

1 – Notably, the one area in which patents are decently functional is the one where they are most similar to copyright: pharmaceutical patents. It is unambiguous whether a drug is chemically identical to another, just as is it unambiguous whether it is, despite a different printing process, the same book. Pharmaceutical patents are the exception that prove the rule.

We’ve launched our new website, and finally revealed the technology that we’ve developed and we think is going to change the world — regenerative air energy storage!

LightSail set out to prove that the science of our regenerative air energy storage concept works, and we have answered that challenge with a triumphant yes!

LightSail's Industrial Scale Prototype

We built an industrial scale machine by modifying a commercial natural gas compressor. We changed the cylinder head, added nozzles, replaced valves to allow reversibility, coated the surfaces to prevent corrosion, and threw our minds and hearts at the problem of showing that our approach could dramatically increase the efficiency of compressed air energy storage. Without water spray, and without burning natural gas, previous attempts at storing energy in compressed air topped out at less than 50% thermal efficiency — ok for a backup system, but not enough to change the world. This year, we aimed at greater than 80% thermal efficiency, at a high RPM (and therefore power), to show that unlike what people had assumed, high efficiency does not mean sacrificing performance.

We met or exceeded all our technical targets — demonstrating record breaking performance at the same time as record breaking thermodynamic efficiency — conclusively demonstrating our water spray heat transfer idea behind our regenerative air energy storage concept is effective at industrial scale.

Afterglow: the day I presented to Bill Gates

We presented to Bill Gates, a limited partner in the fund that invested in us. He was super excited by the potential of our project — that if we hit our targets it would change the world.

We spoke before hundreds of policy makers and energy executives, and helped instate groundbreaking legislation supporting energy storage in California.

Governor Jerry Brown

We have settled on our ultimate product architecture and design — a huge accomplishment. We’ve got a long way to go, but our models predict our experimental results within 5% RME accuracy, so we have some real confidence that it will hit all our hoped for technical specs.

We truly defined our market and value proposition. We’re aiming to make renewables plus energy storage a better and less expensive way to provide high value peak power than what the conventional sources — natural gas peakers, diesel gensets, and extra transmission wires – can muster.

The Trillion Dollar Formula

This is an utterly enormous market; at least a trillion dollars in size over the next couple decades.

IEA Estimates of Energy Infrastructure Investment Over 2008-2030. More than 30% could be economically addressed by renewables + energy storage

We’ve found that we’re uniquely positioned to reach that target, providing the lowest levelized cost of dispatchable electricity of any source, way ahead of our competitors.

But most of all, we’re excited about changing the world. Not only does energy storage make a renewables based grid possible, it also makes it economical. That’s the key to changing the world!

We’ve been working hard to uncover the greatest, most urgent opportunities for energy storage worldwide, and the opportunities we’ve turned up are simply massive. Energy storage is just what’s needed in places as diverse as Hawaii, Texas, Ireland, California, Paris, Denmark, Iceland, Nova Scotia, New York City, Australia, Chile, Dubai, India, and Subsaharan Africa. The scale and diversity of opportunities were astonishing. The most amazing thing? The willingness of governments to put their feet forward and most towards a future that’s right. We have been cynical; we believed that only once we had a full product, a long history, and economic parity under the most conservative of assumptions would governments move. We were proven wrong. Governments are leading the world into a clean future of energy. It’s utilities that are pushing back!

Steve the Redeemer

Take Iceland — a country of stark beauty. More than 80% of the country’s electricity is exported in the form of aluminum — the processing of which is one of the most energy intensive for any widely used material in the modern world. This single industry represents 40% of the Icelandic economy.

The Hellisheidi Geothermal Plant in Iceland's Golden Circle

Essentially 100% of their grid electricity comes from their amazing geothermal and hydroelectricity resources, and an enormous amount of their heating comes from geothermal cogeneration. Iceland is a land of abundant green energy.

Gullfoss -- the golden falls.

There’s a catch, though. Transmitting power across the sparse, weatherbeaten land is an expensive, unreliable proposition, where remote locations risk being knock completely off-grid with each storm. This is worse than it seems — if power is cut to aluminum smelters, the aluminum freezes, severely damaging the equipment. To backup the geothermal and hydro plants, then, industries have had to co-locate with diesel gensets — hardly a solution in light of the self reliance and environmental commitment of the Icelanders. We intend to replace these gensets completely. But we can do a lot more.

Low-temperature geothermal heat is available nearly everywhere in Iceland, and we can harness it. By expanding air at a higher temperature (and therefore pressure and volume) than when it was compressed, we get more mechanical energy out than we needed to compress it. This allows us to convert heat energy into mechanical energy, and from there, electricity. So instead of sitting idly like backup diesel gensets, our machines can be producing clean, geothermal energy, constantly; leaving the compressed air available for bursts of power when the grid fails.

So, we met with Iceland’s Minister of Energy — a former thermodynamics professor at Lund University, who bemoaned parliament’s inability to understand the concept of exergy.

We have therefore ‘rebranded’ our efforts. From now on, we have an initiative in ‘energy quality management.’ This they understand.

He understood the implications of an economical energy storage and geothermal electric generator immediately, and urged us to consider a project in Iceland. This is exactly the sort of progressive movement that governments are making and utilities resist. But we will overcome their skepticism! Stay tuned.

Catching My First Wave - A Good Omen

Of course, it wasn’t all business. If your mission call upon you to travel, it is your duty to truly experience the place. So I took the time harness some of nature’s forces myself. After I visited the grid operator and wind farms of Hawaii, I learned to surf!

We have continued to hire and improve our utterly world class team. We’re almost 30 people now, but I can tell you I have never before seen or even imagined such a diversity interests or depth of talent in a group. I work with the most amazing people I’ve met in my life! It is amazing to see how rapidly people are growing, but even more amazing to see how much more we can accomplish as a team. There are things that we literally couldn’t do on our own given all the time in the world — we have such a diverse set of skills in the company that we can make amazing things happen.

It was our first Burning Man. Our minds were blown. It is more than a festival, more than an amazing city. It is the most spiritually profound, unashamedly sensual, and maniacally creative place I’ve ever been.

Whiteout

Offering to the Sun

Deep Playa

We travelled as the chefs of the Airship Victoria last year; an airship project that eventually intends to hoist a Tesla-coil based lightning musical instrument. The camp, directly on esplanade, next to the flaming lotus girls, the sonic cannon, the flamethrower organ, and a 24 hour bar, was a surreal experience.

The Airship Victoria

It felt like… the future! It turns out, in the future, there are lots of lights, people float around on bikes, and jellyfish hover and flow.

Wonderful. But the main thing about the future is that people can’t help but be caught in the moment.

Stargate

Our camp featured tesla-coil concerts, and there, was, admittedly, high drama before the balloons were successfully fully deployed. Despite some initial setbacks, eventually the camp lifted their payload high into the air. In a city confined to a flat lakebed, the balloons added a third dimension to the playascape.

It’s impossible to describe the sense of flow one achieves in such a dizzying storm of self expression. We danced in drum circles in the nude, rode art cars and floated glowing jelly-fish, windsurfed and found inner peace. But what was most dazzling of all was the temple.

The Temple of Transition

A strikingly elegant wooden structure, built in just 10 weeks by inspired volunteers, the temple was a deeply spiritual place of reckoning. The visitors, pilgrims of every creed, came and prayed, and made offerings for their loved ones, those who that had left them, those who they had left behind. Poems, and pictures, incense and chants, old clothes or talismans, and cherished items of every description, laid respectfully to rest, ultimately fated to return to the atmosphere aflame.

Steve was so overcome that he bent down on one knee and made an offering to his mother, a brilliant opera singer, who left the world when far too young.

I miss you so much mom. You would have loved this place. I will love you forever.

As the temple’s towers, lean and graceful, slowly surrendered to the flames, glowing sparks rose deep from the inferno, and like wisps were carried up towards the heavens. The temple of transition, once a place of cool respite, now glowed brighter than the noonday sun. The crowd gasped as a shower of blue leapt out from the flames. Someone, days before, hid fireworks that launched streams of blue from the middle of the swirling firestorm, but in that splendid moment, it was impossible not to see those glowing blue apparitions, lifted high into the glowing sky, as souls, let finally free.

Meditation, Release, a Moment of Inner Peace

Upon our return, it seemed as if the whole of LightSail met us with faces silently asking us to bring them next year! We will.

This year, we’re starting a camp — tentatively named “Cleantech”. A solar powered shower and water recovery/purification system of our own design. It will be beautiful and efficient and environmentally friendly. Our kind of project!

LightSail's Firehouse Lab

At the end of the year, having wrapped up our work at our firehouse lab, having shown all we can with our current industrial scale prototype, we moved into our new facility — the former Scharfenberger chocolate factory, in which we will design, test , and manufacture our first product line. It is an amazing space. We will do outstanding work there, and we will be happy and proud.

As the move in completed, the holidays arrived, giving us the occasion to throw a lab-warming party for our friends and family. It was absolutely amazing. I felt as if the party unfolded as a microcosm of the entire project. It began with a simple idea: “let’s have a holiday party,” which lead to the conclusion “we clearly must have it at our new space,” and from that point, it took on a life of its own, spearheaded by people of admirable competence and outstanding creativity.

We were blown away by it all. The founders had no idea! Everywhere you looked there was perfectly executed brilliance.

Enter the space, we’re greeted with placards describing what all of the work is, what each of the rooms are, how each of the items work. There were demonstrations of our tank technology, our electronics and controls, our machineshop and quality assurance, our water spray lab, and even our original prototype (built in Ed’s garage using scrap parts and ebay!) We had no idea it would be there, and were blown away to see it!

Humble Beginnings: The Original LightSail Prototype, hydraulic, quirky, built of scrap, sweat, and parts ordered off ebay.

The original machine used a hydraulic approach — slower, with less power per unit mass or cost, and with higher inefficiencies, but we conclusively proved we could control the temperature of the air during operation, and control the valves to let only an amount of air in that would expand down to 1 atmosphere — yielding the very highest efficiencies. It was a cheap, quick way to show that some of our main ideas worked, and that we could build something. We sure have come a long way from that!

Travis O’Guin and his band played an incredible set of dixieland Jazz of some of the past century’s greatest compositions (ever wonder how “hit me baby one more time” is in dixieland jazz? Amazing.) Ed broke out into dance with a series of dancers, and the LightSail toddlers couldn’t resist the beat!

Machinemaster Todd Bowers breaking it down for Professor Robert Dibble and wife Helen

The machineshop was running — demonstrations included a CNC lathe disco ball, a hula dancer shaking it to an earthquake powered by the CNC mill, and just-in-time manufacturing of LightSail Branded Bottle Openers!

Dave Sprinkle spent years in the racing industry, but it's cupcakes that bring this smile to his face...

But what really stole the party were the cupcake cars, brought in by the brilliant Keith Johnson and his merry friends Lisa Pongrace and Greg Solberg. Our partygoers insist they’re even more fun to drive than a Tesla.

It was an unbelievable way to ring-in the new year. This is going to be a great one. That everyone injected such creativity and excellence in such a gathering just shows how much people care about their work and their team and this company and how high a standard they have for themselves. It seems as if everything at LightSail is like that — our people perform at a higher level than us founders can even think to ask of them, or indeed, even to imagine.

Energy Standout of the Year, Forbes 30 Under 30. As shot by the iconic Harry Benson.

To top it all off, we received coverage from none other than Forbes Magazine. I am honored to be highlighted as the standout in the field of energy in the Forbes 30 under 30 ranking! My extended family is finally less suspicious of my dropout ways. What a relief! I had a wonderful time at home with my brothers and little cousins and found to my amazement that my family had founded four businesses between us since we last visited. I guess it’s in the genes.

Christmas in Nova Scotia

This year looks to be even better. It feels like we’re reaching escape velocity.

I am honored to have been elected a mentor for the Thiel 20 under 20 Fellowship. These kids aren’t waiting to change the world, they’re just going out and doing it — I am so excited to be working with them!

I have been tapped to judge the Nova Scotia Cleantech Open, remarkable not only for being in my home province, but also for its amazing quality, rigor, and prize money ($100k free money with $200k of seed investment available.) I’m joining Matthew Nordan, of Venrock and Lux Research, whose work and judgement I have always admired greatly. His “The State of Cleantech VC is already a classic in the field.

We’re working full-speed on our product and technology, and are rallying allies across the planet to realize a wonderful number of as yet unannounced projects and partnerships.

Last year was an amazing year, but I have a feeling this one will be even better.

I am so happy to be alive at this moment in history. Great things are afoot. The winds are changing.

Skin Cancer

One of my first serious startup business efforts involved skin cancer: melanoma, specifically, by far the most malignant and dangerous type. It turns out that one can really do a pretty good job in terms of detecting melanoma in the early stages, when it’s still relatively easy to treat. This is in part because it is a cancer on the surface of the skin, and typically the cancerous, rapidly multiplying melanocytes produce an excess of melanin in patterns of characteristic irregularity. In other words, one can spot melanoma in ugly moles. If (a) one knows what to look for and (b) actually does look, in the vast majority of cases you can catch the disease before it spreads, and thus, save a life. The World Health Organization pegs melanoma deaths at 48,000 per year.¹

The ABCD's of Melanoma. Part of the ABCDs for detection of melanoma. On the left side from top to bottom: melanomas showing (A) Asymmetry, (B) a border that is uneven, ragged, or notched, (C) coloring of different shades of brown, black, or tan and (D) diameter that had changed in size. The normal moles on the right side do not have abnormal characteristics (no asymmetry, even border, even color, no change in diametry).

The chain breaks in both places. There are papers² that estimate the sensitivity of physicians to melanomas at 86%, which is not bad, if they happen to look, but not great either. The big problem is that it’s by no means certain that the doctor will look: most melanomas are still found first by the patient. That is, if they’re found at all before it’s too late.

A friend of mine refused to see a dermatologist, despite her friends urgings. Once she finally did, the cancer had grown out of control. She did survive, but she had to suffer through an extremely painful excision followed by reconstructive surgery followed by a recovery period nearly a year long. Prevention and early detection are the key weapons against disease.

This could have been prevented. There’s a heck of a lot you can learn from a photograph, even one taken from a cellphone. From that photograph, a trained individual can determine whether the mole is a cause for concern — whether you’re really just fine or whether one should be examined more closely by a specialist. There are even papers showing how an algorithm can be made to be as accurate as physicians in diagnosing the cancer.³

It seemed like a great hack. Within a week or so, I rallied some friends of mine, threw together an application to Y Combinator, and tried to put a business together.

Whatever the technical challenges were, I was unprepared for the legal and social challenges I was to face. The partners at Y Combinator did like the idea on some level, but declined to fund us. In Paul Graham’s words

“The trouble with the melanoma detection idea is that you’d spend most of your time dealing with legal and regulatory crap. That sort of work doesn’t really take advantage of your skills.”

There were other problems too (we were rather cavalier about the quality of images we’d get in practice from widely varying cameras, lenses and lighting conditions), but in the end it was the specter of law and regulation that cowed us. I still maintain that we could build this, and while perhaps it might not have been the absolute best focus for my efforts or launching point for my career, I still think we could have done a lot of good.⁴

What concerns me most isn’t that we couldn’t get the melanoma detection idea to take off. It’s that anyone trying to work in the field of medicine is going to have to slog through legal and regulatory crap. There’s a lot that can be done with simple little pieces of software and hardware to help people look after their health. Detection is the first line of defense. But as long as we persist in the current regulatory environment, where detection aids must disclaim whatever they imply or slog through the same lengthy and expensive FDA approval process regardless of their capacity for harm, entrepreneurs, engineers, doctors and scientists are going to be limited in the good that they can do, as well as the bad. Lesson one.

[JessNordell notes that both SpotCheck and SkinScan have taken up the charge! ]

The Electrical Grid

It doesn’t take too long before the familiar seems sane, and what we grow up with seems to have always been, bequeathed to us by ancestors blessed with the virtues of invention, vigor and foresight. I remember, as a kindergardener, when my mother took me to the local power plant, chimneys three hundred feet tall, how awed I was by the sights and sounds — the enormous burners and boilers, steam blasting through turbines powering vast generators, electrons pumping through transformers in gray substations of metal ringed insulators, transformers, cylinders, a geometrical landscape adorned with fins, cathedrals of wire, lines buzzing. Deep sounds metal clangs reverberated through the boiler room, amid hot steel, as my tiny feet traced halting paths along innumerable iron grates. Five years old, seeing this, I imagined the builders of these machines surely must have known what they were doing. Promethean gods walked the earth: behold! This was their handiwork.⁵

It shocked me to find out how the electrical grid really worked.

The grid, as it exists today, has almost no capacity to store energy. This means that at every instant in time, the amount of energy going on to the grid must almost exactly equal the amount of energy coming off. Otherwise, the grid rapidly fails in its operation, rising too high or too low in voltage, or shifting frequency, or succumbing to noise, spelling disaster to everything dependent upon a smooth, steady stream of power for its operation. That is to say, nearly every device of the modern world.

And while we have some provisions to redirect power on the grid (primarily, giant, motor driven switches of a kind not unlike those which control your household lightbulbs), our ability to throttle the amount of power we put onto the grid is slow and expensive. Gas turbines are the power plants that can be controlled most readily, however their power must be throttled over a period of nearly 15 minutes. When they do this, however, they operate away from their most efficient operating point, meaning that lots of fuel is burned, power is expensive, and more CO2 goes out the stack into the air. Hydropower stations can be throttled nearly as readily as gas turbines, but they can operate in a wider range before they lose most of their efficiency. Coal, petroleum, and nuclear plants, on the other hand require periods on the order of days to speed up and slow down — it takes about that long for them to relax to a thermal equilibrium, keeping thermal stresses within safe bounds.

However, the electrical demand is quite unpredictable, and it can vary significantly in a period much shorter than the 15 minutes it takes for our fastest power plants to significantly alter their power output. So to respond in faster timescales, power plants must run as ‘spinning reserves.’ This is where fuel is burning and the turbine is spinning, constantly, in expectation of a fluctuation. There are now massive power plants, burning fuel this very second, for no purpose other than to smooth out the rapid fluctuations on the grid. I hope I don’t need to point out how crazy this is.

What’s more, electrical supply from renewables is very unpredictable, especially whatever’s coming from distributed power. Wind power is the least expensive (in fact, according to Prof. Marc Jacobson of Stanford University, considerably less expensive than coal power!)⁶, but in terms of supplying controllable, usable power, wind is terrible! Wind power varies rapidly and widely, is extremely difficult to predict, is quite correlated over large regions, and, what’s probably the worst, wind power comes mostly in the dead of night, when nobody needs it.

Two days of output and wind speed from a four section wind farm. See the sudden drop? That's why we need energy storage if wind is to be a major part of our grid.

The entire world has wired itself up with an electrical grid that is fundamentally insane, and as we plug in renewables to solve the other problem with the electrical grid, controlling it is only getting harder.

We need a way to store energy. The current state-of-the-art⁸ is pumping water up a hill to store energy (essentially using a hydropower plant backwards), and letting it run down to retrieve it. This works well and is relatively inexpensive if you have the ideal geography, but those plant sites are mostly taken up: from now on pumped hydro energy storage is either going to be a lot more expensive or a lot more exotic (current plans call for giant undersea bladders, or the use of enormous underground aquifers).

Pumped Hydro Energy Storage

There are a number of other energy storage technologies, including electrochemical batteries, flywheels, and compressed air. There are good people working on every one of these approaches, and from our perspective, technically, there’s a good chance that at least one of us will find a solution allowing for energy storage so efficient and inexpensive that renewables, especially wind power, are economically competitive on the world scale, and on an unsubsidized basis.

However, among the many obstacles we find along the way appears the government. Despite their problems, technologies now exist that can both regulate the grid on short time scales (that is, dampening the second to second and minute to minute fluctuations) and, over a longer period, buy unneeded power in bulk at low prices and sell it when needed at high prices. If, on a single device, you can only provide the short or long term service, the economics don’t make sense. Luckily, it is technically feasible to do both simultaneously, on the same device, and thus the economics for the current technology can work out, in at some cases. The electrical grid can be buffered by energy storage, and utilities could make money doing it, to boot! How sensible.

Unfortunately, it’s illegal!⁷

Let me explain.

Transmission assets, like roads, railways, telephone networks and the electrical grid, have been shown to naturally tend towards a monopoly. In the case of the electrical grid, after a number of political fights the following deal was struck: the government would grant the electrical grid transmission monopoly to a single entity (in some cases more specifically the last few legs leading from the power plants to your home) in exchange for certain powers. In particular, the utility could, using its monopoly pricing power, impose practically any price on the unwitting public, which would pay just the same — they have almost no choice and electricity demand is notoriously inelastic. However, it would be restrained to charge a price only up to a certain regulated return on their capital, and no more. The regulatory commission has other powers, such as being able to define which sorts of investments and projects can proceed, and it places on the utility certain contractual demands to supply power.

Therein lies the first problem. In effect, were one selling power equipment for the grid, one’s real customer is not the utility, but the regulatory commission. It’s in the utility’s interest to buy the most capital intensive equipment that the regulatory commission will allow, and to incur the greatest expenses. After that, it may jack up prices such that it achieves its regulated rate of return, but now, due to its greater expense, it is now earning a greater profit as a greater entity for more investor dollars.

I never imagined such a perverse set of incentives. There are only three reasons for which utilities regulated in this way will look beyond the most expensive corner of the status quo. Firstly, in order to reduce risk on their capital investments, secondly, to explore the use of alternative technologies and broaden their strategic options, and finally, out of the goodness of their hearts. They’ve actually done a lot on that last count; that renewable energy is deployed at all seems to me shining evidence of a desire to do good. But utilities are conservative organizations, they’ve already figured out how to run their capital investments with very low risks (lower than most new technologies could possibly manage) and dollars for technology exploration are in short supply.

So, there’s the first shocker. Utilities with a regulated rate of return (I think this is most of them, though I’d love a better quantification for this) have essentially no incentive to save money, and hence no incentive to try out new money saving technology. If we’re to sell to anyone we have to sell to the regulatory boards, who are even more conservative. Oy.

Secondly, as the government is fond of doing, parts of the electrical utility business have been broken up, and they are not even allowed to talk with one another and share information, much less operate with the same hardware. The transmission department at PG&E is kept sealed off from the generation department at PG&E. Unfortunately, despite the fact that it is ludicrous, that it is patently insane to operate an electrical grid without energy storage (just as it’s insane to operate a network with no buffer, a computer with no cache or memory), there is no energy storage department at PG&E, or at least it doesn’t fit into the existing regulatory structure. There are no requirements for a certain amount of energy storage, and even the things that you could do to save money for the utility are unattractive for the reasons described above, and even those are unlikely with current technology to be good investments because they are both risky and it is illegal to operate the same device for both the generation department (people are saying now that storing cheap energy at night and selling it at a higher price during the day is called ‘generation’ for some reason) and the transmission department (regulating short term fluctuations in the grid and storing energy to ease congestion is called ‘transmission’ for some reason). So, a sensible and practically necessary thing (energy storage) is not allowed to be run in a manner which with existing technology which could save the public and the utility money and could enable the economic competitiveness of renewables. But that doesn’t matter because utilities aren’t trying to save money anyway.

Wow.

A couple of these problems can be avoided in the new, deregulated, ‘ancillary services’ markets. These deregulated markets, humorously enough, are markets for regulation — utilities and electric co-operatives pay a premium for MW scale power slices, so as long as you’re in the right spot and can deliver power on extremely short notice (people are pushing this as short as 4 seconds) you can stand to make a fair bit of money on a deregulated market selling to utilities. There are a few companies (Beacon Power is a prominent one) that try to make systems to do this now: in addition to the technology they’ve developed they’ve put in an incredible amount of political effort to get utilities to open up and deregulate their (*snicker*) regulation markets.

Unfortunately, the economics are still somewhat marginal. Current energy storage technologies are really expensive compared with the wasteful throttling of fossil fuel power plants. Better technologies (we’re working on one) might be able to make a really substantial profit. And if you can do energy arbitrage (buying power low, selling it high), or do something else that’s useful with your energy storage device, you can make even more money. Unfortunately, as above, it’s illegal to sell regulation services and do energy arbitrage on the grid — you have to do something else. (If any investors are reading this, don’t worry, we’ve got a few tricks up our sleeves!)

Someday, by our actions and with the help of others, enough of these problems will be solved so that the electrical grid can be buffered and made sane, that the public will save money by using energy storage, and renewable energy will be free to take over the world.

Someday.

Until then, we’ve got a lot of work to do. Here I thought we were working on a complete, technical solution to solve a massive problem in the world. A straight shot. And what do we find? Legal and regulatory crap, my nemesis, as defined by Paul Graham.

Lesson two.

So it goes.

A Patent Puzzle

Inventing things is what I do. It’s an inseparable part of who I am. I can’t get three chapters into a book before I have an idea for a new technology, product, startup, or hack. Parties are brainstorming sessions. I pester romantic partners as we fall asleep.

Some of my ideas are good. Some, not so good. Those that pass all my filters probably each have a decent shot at solving a major problem in a way fundamentally better than what’s come before with an approach that’s within my eventual capability to execute. Maybe it’s a little better than 1 in 3. I can’t really know until I try. But because I have so many ideas, and possibly because of practice, they number in the hundreds. If they were they only idea I had, I could really see myself giving them a go. And succeeding. I am not at all unique in this.

One thing that helps tremendously is to be able to talk about one’s ideas. Maybe even bring another person in to help work on bringing them to life. Oftentimes one’s ideas are too much for one to handle at some time, or maybe one is missing some crucial missing piece that needs just the right person or the right piece of knowledge.

If good ideas are too plentiful to fully utilize by oneself, if the difficult part of profiting from a good idea is bringing them to life, and if being open about them aids in this process, then ideas should be free.

Or, at least, it should not make one unduly vulnerable to reveal one’s ideas. One should at least be free to let one’s ideas free.

My friend, let me tell you about the patent system.

If I hold a patent on something (a widget), it does not allow me to build, license or sell the patented widget. It merely allows me to prevent others from building, licensing, or selling that widget (or anything else that falls under the widget classes and widget constituents for patented widget uses under the widget claims).

If my widget involves some device (a sprocket) that someone else (Eve) has patented, Eve can prevent me from building, licensing, or selling my widget unless I purchase a license for Eve’s sprocket.

If my something absolutely requires some crummy little item (a doohickey) that I forgot to mention or didn’t quite yet invent in my papers, my blog posts, my patent applications or my brochures, the sum total of my disclosure to the public domain and the patent office, and Eve notices this, she can patent said doohickey and the rational use of her doohickey and extort an outrageous price from me before I can sell any widgets at all.

Herein lies the problem.

Race to the Sea!

The Race to the Sea of the First World War, 1914. While the war was still mobile, both sides attempted to find an open flank. This began a race that ended when it reached the sea. Before these lines were drawn, armies were mobile. Afterward both sides became mired in grueling siege trench warfare from nearly unassailable positions. An analogy with patent strategy is apt: intellectual property is territory, and you do not want to be surrounded.

If you mention almost anything specific about what you’re doing and how you’re doing it, then you’re making yourself vulnerable. Eve can now take a look at what you’ve done, run through possible improvements, and possible requirements that you either have not claimed or disclosed or have not invented yet, and patent the ones that will hurt the most. Once you’ve given away your position, she can surround you, choking you off from the possibility of profiting from your work.

The important patent on the sewing machine was the sewing pin. With a tiny little hoop through the head. It is easy to forget the little things.

In the current legal climate, there are three options.

1. Stay hidden indefinitely. Protect everything that you’ve got by trade secret. If you personally (with or without corporate backing) expect to be able to bring your ideas to their full potential, this might be best for both you and the world.

2. Race to the sea. Patent your work — stake your ground, and reveal your position. Once your first patents are filed, you have to, urgently and carefully, check to make sure you’ve thought everything through. Are the relevant uses and markets patented? Are all of the incorporated or required devices available as commodities, or public domain, or inexpensively licensable, or do you own those patents yourself? Are you absolutely sure that you haven’t left anything critical out — that there’s nothing that you still need to invent that someone else might patent first? Are substantial improvements and variants of your design included in the first patent? There is a relatively short delay (on the order of a year) between when you file a patent and it appears for publication. If any of the vulnerabilities describe above exist, you’ve got until the publication date to protect yourself before Eve can make her attack. The clock is ticking.

3. Use your judgment, talk and write as freely as you desire, and cross your fingers. Hope that no one patents around you, and no one litigates.

I personally would love to be able to open source many of my patentable ideas. Engines. Desalination plants. Solar collectors. Refrigerators. Waste water mining. Wind turbines. Boats. Planes. A crowd beacon. Medical devices. Heat lamps. Translucent coatings. Metallurgical processes. Bioreactors. You name it. But the legal climate does not make it very safe for me to do so. If I write about them too freely, Eve might patent around me — now neither I nor anyone else can pursue my idea without encumbrance. If I protect myself by racing to the sea, I need to engage in an extraordinary investment of time, energy, and money on an idea that I probably crystallized in three seconds. It’s clear that even in the best of circumstances I’d be rate limited, but for almost all practical purposes this would become an untenable demand on my time (unless I have the resources of Nathan Myhrvold at my disposal.) Finally, I can protect myself by keeping my ideas trade secrets. Unless they are one of the few which I actually find the time and resources to seriously work on, this accomplishes exactly nothing. There is no pile of precious ideas I keep to stroke at night. I do not hoard them.

Some members of the free software community have been brainstorming ideas which have some relevance to tackling this problem. They include the Patent Commons and retaliation provisions in the GPL v3.⁹ Unfortunately they really do not provide full protection. Patents can only prevent people from building or selling things. They are weapons. Patent trolls, on the other hand, don’t build or sell anything. There is nothing to retaliate with, nothing to counter-sue, there is no center, no target. Patent trolls occupy an ecological niche in the legal landscape equivalent to cell based terrorist organizations in today’s political world. A sufficiently determined and evil person could extract billions of dollars from the world and destroy billions of dollars in wealth via long and painful lawsuits, simply by surrounding the patents and inventions that the world now relies upon. Perhaps at this time our only real protection is how grueling and soul destroying that path to fortune is versus the creation of wealth.

The patent system is supposed to encourage innovation. But I think what it’s done is cause everyone to overvalue ideas, and that has shifted the balance of power away from those who want to turn their ideas into something concrete and towards, marginally, those who have the ideas in the first place, but much more worrisomely, those who wish to wage war with them.

I’d love a discussion on how to fix this system. Specifically, not what to fix, but how: what specific actions can we take to make a difference? Quite apart from this, what should I do? Suppose I want nothing more than to do good for the world, to let my ideas, if they so deserve, blossom fully into life. Just talk about them and hope there’s no lawsuit? I do not want to be thinking about this!

Lesson three.

More legal and regulatory crap. Maybe I shouldn’t be so surprised: without law, there’s no formal property, no real capitalism, pretty much just anarchy, really. Of course it undergirds business. But what’s strange is that it’s constraining my thought. I just want to talk to people about my ideas, help the world, and be good. This shouldn’t be so hard!

So it goes.

Post Script: Trevor Blackwell of AnyBots and Y Combinator gives a thoughtful reply on Hacker News as to how the regulations that come into being are so malformed.

Big companies are smarter than gov’t regulators, they understand their business better, and they have a longer time horizon. So when the government comes around to regulate them, they think “OK, how can we turn this into a huge barrier to entry for new competitors?” They have large lobbyist and strategy budgets. They generally win.

The most misregulated industries in the US are energy, medical, and transportation. So there are lots of glaring inefficiencies, but they are there for a reason. Technologists assume the reason is stupidity and that clever inventions can fix things. Frustration ensues.

References

1 – Lucas, R. Global Burden of Disease of Solar Ultraviolet Radiation, Environmental Burden of Disease Series, July 25, 2006; No. 13. News release, World Health Organization

2 – Delays in diagnosis and melanoma prognosis (II): the role of doctors. Richard MA.

3 – Neural Network Diagnosis of Malignant Melanoma From Color Images, Ercal F. et al.

4 – I was told later by an entrepreneur formerly employed by the FDA that with a sufficiently awesome disclaimer one can put nearly anything on the market (“Nothing this program tells you means anything”) though I am really not sure how well this works in practice. I have read horror stories of the offices of a breast cancer detection aid being raided by the FBI — the case had to go up the the supreme court before they were acquitted.

5 – I do remember, however, challenging my tour guide in the waiting room when my cartoon host ‘Zappy the Electron’ claimed that both electricity and electrons traveled at the speed of light!

6 – Jacobson, M. Z., and G. M. Masters G. M., Exploiting Wind Versus Coal

7 – I should mention that I only know about these problems in the context of the USA regulatory framework. I’m sure there are analogous problems elsewhere, but I have not examined them in sufficient detail.

8 – State-of-the-art if one is considering cost effectiveness, and not energy density! Pumped hydro plants are gargantuan.

9 – This is a good discussion: Copyleft versus Patents: The Open Source Legal Battle, By François Lévêque and Yann Ménière

organizations
relationships
communities

as living entities

as where no one cell holds the soul of a person
no one person, incarnate
may form all essence of a community
nor live throughout its lifetime
wholly part

as one cell expires
one soul retires
others, some new, fill their place

generations cycle
yet something remains
an living entity unto itself
though it may be true
that some people or ideas
resemble vital organs

the liver, to filter
the stomach, to process
the lung, to supply
the mind, to direct
the heart, to power

too often
transplanted,
perfectly good hearts
from one being to another
from one community to another
raise immune reactions
edicts like allergens
mobs like histamines

without disarmament
no transplant succeeds
no hire, restructuring, or revolution carried through
without somehow disabling
or surviving
societal antibodies

for some, it seems
while becoming part of a great organization
or movement
or organism
they enter through digestive tracts
to supply the raw stuff
the best
carbon, iron, calcium, oxygen
skills, goals, ideas, selves
are prepared
and cooked
chewed
dissolved
filtered
processed
and made, finally, into a part of something new
to fit some other master plan
a genetic blueprint
an ideology
a mission
a politic

for some
no greatness of which they are part
could come to redefine them
their raw stuff
the parts comprising
can find no better molecule
no further local minima
no structure more solid
between surrounding stomach walls
they are indigestible

or too delicate
too beautiful
too unique
to be eaten

their soul to realize
as part only whole
of something profound,
meaningful,
believed in
or as the kernel of a gem
of another body
of an essence of its own
starting anew

So prediction is fundamentally at the heart of a technologist’s work. At the highest level, we must predict to find what work focus on, and what future to aim for.

You might then think that prediction, as a skill, is worthy of practice. And practice it gets. In living rooms, in pubs and classrooms and yearbooks and dial-in talkshows and newspapers and blogs and comment threads and slashdot and every polluted corner of our existence, you find evidence: prediction is practiced all the time.

There’s a problem. In most areas of the technologist’s pursuit, it’s easy to see whether you’ve done well. Code should compile. Planes should fly. Cars should go. Bridges should stay up. We have a lot of honesty in our discipline, much of it because we are blessed with tests that we find hard to fool.

A typical test for predictions, on the other hand, is whether the story sounds good at the pub. You make some exclamation. People nod and clap. Everyone forgets.

This would be fine if you’re just looking for some conversation. But if you are, like technologists fundamentally in the business of creating the future, it becomes lot more troublesome. We are left to ignore predictive incompetence until reality slaps us coldly across the face. We are flying blind.

Taking a cue from Trevor Blackwell, I’ve decided to inject some rigor into my life: when I make predictions, instead of casting them abstractly into the air, I’ll post them here: einfall.slinkset.com. And I won’t delete my predictions — if they turn out wrong, I’ll keep them there, as permanent reminders to learn from.

Through accountability, honesty. Through honesty, improvement.

Thanks to Trevor Blackwell for the inspiration, and John and Brett from Slinkset for the List Hosting.

Notes: a friend of mine noted that most of my predictions seem ‘pessimistic’, in the sense that they take the form of ‘X will not Y.’ I would have to agree with him. But this is largely a byproduct of how these predictions were made – they’ve come from studying some field, working in it for a while, and coming to the creeping realization that one or more of the current approaches were doomed. Besides, much of the skill of experts comes from the ability to ignore false trails.

Further Reading: An excellent site for major predictions (often with significant wagers) is Long Bets.

This may sound weird, but I pretty much choose employment based on the promise of quality work. Other factors fade into irrelevance.

When I say quality of work, I mean more than the work environment, more than the magnitude of technical challenges, and more than the IQ of those I’d be working with. I want the opportunity to walk paths with the greatest hope of leading to first-class work. Nobel-prize winning kind of work.¹ This force guides me, and so inevitably I tend upstream of technological change. Money and prestige are mere proxies for what I really want: to develop and inspire fundamental changes in the way people live. That doesn’t mean I need tackle the greatest problems humanity now faces (yet). What matters is that I, personally, have a reasonable approach. So I must always remind myself to, as Richard Hamming says, ‘plant the little acorns from which the mighty oak trees grow’ — because small projects can, swiftly and strikingly, grow momentum and value.¹

I find little value in submitting myself to some company culture. I instead mean to develop my professional values, ambitions, and goals: for example, I would like to develop new methods, make them available by open sourcing them and make them popular by evangelizing them. I’d love to be given the chance to teach what I’ve learned. Excellent people bring ideas and perspective to a communities of makers. Given time, I think this will evolve naturally into a company culture worth having.

Most big companies grow faster than they could build trust, to a size greater than strong values can be supported. Natural culture is the product of alignment of creative philosophies, and in BigCo, this is too often replaced with virtual company nationalism. Fascism even. I find this is more than distasteful. I haven’t really learned how to work within it at all. I could devote my efforts to such an organization only were there deeply meaningful work to be done. And why bother?

Angling to be upstream of technological change, I bait unusual questions and find surprising answers. Give me the choice between a VP position at a big five media company with oodles of benefits, and, say, work at an early netscape or google for a totally minimal salary, and I’ll choose the latter every time. I’m pulled towards organizations where I can learn about organizing, rather than learning about institutional tradition. It’s not important for me to learn about how to run a large organization: if ever I do, I won’t follow of the paths of current captains of industry. Instead, I intend to help grow large, leaderless, open organizations, and so I’d do almost anything for a chance to work with Caterina Fake, or Linus Torvalds.

I want to work on something I find deep personal meaning in. I strongly believe in supporting open culture. I don’t think I’d work for long in games or entertainment unless it could influence some social change. I worked at MochiMedia because it made possible an income stream for small independent developers where none existed before. This finally opened up professional game development from BigCos. Now, much innovation in gaming emerges from bedroom studios. Independent game developers can now commit to their art in a way they before could not.

Similarly, I’d work at YouTube rather than Hulu, even though one’s a startup and the other isn’t, because they’re more interested in involving everyone in the process. As Clay Shirky says, they’re interested in ‘finding the mouse’.

I want to work somewhere where I can truly make a difference. Why am I working in technology at all? Archimedes once said, ‘If you give me a lever and a place to stand, I can move the world.’ Technology is my lever. I need only find place to stand. This makes me wary of startups that try to do good, but aren’t particularly focused on doing it efficiently. I wouldn’t work for most charities. There’s too little pressure on them to focus — the tempering influence of market competition is replaced by government demands for ‘accountability’, which arn’t nearly so powerful.

There are numerous ‘ecogreen’ websites out there that try to promote simple, green ways of living. These may be virtuous, however, in terms of minimizing environmental impact I think they’re somewhat irrelevant.² Saving plastic bags won’t lift a toe on our carbon footprint unless we find ways to either cut down on air and automobile travel, or do it more efficiently. And on carbon footprints — global warming is, I think, a red herring — there are thousands of nasty effects of pollution from, say, coal-fired power plants that will hit even if global warming doesn’t occur (though I think, probably, it will). Too much of China now wears breathing masks.³

I can see myself dedicating myself to the right company, so long as our goal, philosophies, and ambitions align. Yet these are stringent requirements. So far, then, I’ve found it necessary to reserve some energy and time for my own projects. So I must be open with companies: with most, I want only consulting work, to help them with some particular project, idea or problem. And I want to be completely, totally honest with everyone about it, because so far, the high road has never let me down.

Sincerely,
Danielle

———

Notes:

[1] – From the classic talk by Richard Hamming, You and Your Research. I don’t, particularly, apologize for my ambition here. Why shouldn’t I try to do first class work? The Nobel prize winning part is purely incidental. But this is the kind of work I mean — a significant contribution, one that people can build upon.

[2] – The free, online book Sustainability – Without the Hot Air is an excellent read. It is the first thing I’d suggest to someone interested in seriously starting into environmental matters. I shouldn’t claim that small contributions to green living are completely irrelevant — each does have some small effect. Perhaps raising the issue of green living in our collective consciousness will have an effect greater at second order than I imagined. But so many of our behaviors are misplaced. Many people, for example, go out of their way to buy ‘sustainable’ products at Whole Foods, say, when in reality, longer vehicle trips do more damage than almost anything you could buy. Many things are sold in a way to make you feel good about buying them. They don’t have any real effect!

[3] – This is worded provocatively, but pollution in China is a growing catastrophe. See ‘As China Roars, Pollution Reaches Deadly Extremes’ and ‘Where Breathing is Deadly’.

———

Thanks to Alex Lang, Ma’ayan Bresler, Nick Pilon, Colin Percival, Michael Nielsen, and Joel Muzzerall for reading drafts of this, and Charles Beatty, for sparking it.

PS: Certain misconceptions have been raised. Some feel that this is one demand of an over privileged generation. I reply to this here. Additionally, I am not, in fact, abandoning my startup. But I do need money, and a visa, so I am looking into either employment or seed funding.

True to form, when a discussion appeared on Hacker News I rushed to comment, and this article erupted from that attempt. The current scientific understanding of our cosmic origins is a mystery to the public at large, but it was only after I noticed the bewilderment of my fellow hackers that I realized how poor a job we scientists have done in conveying the motivation behind our discoveries.

This article represents an attempt to replace that sense of bewilderment with that of wonder. I want more than to explain what cosmologists believe. I want give people a deep sense of why we believe it, of how we’ve come to our current understanding, and of why we care.

Look close, and it seems the universe is lopsided.

The cosmic microwave background (CMB) is like a snapshot of the early universe. It was once all hot plasma, gas so hot that the atoms inside it were broken up. Because it was hot, it emitted light. Because it was dense, it was opaque: the light emitted couldn’t just pass through, instead it had to bounce around. But once cool enough, the universe became transparent: all the light could now travel freely. It was as if the photographic shutter of the universe was lifted.

The light from this moment became the cosmic microwave background radiation. Because the universe seemed to have cooled at almost exactly the same time everywhere, the CMB is, unlike almost everything else in astronomy¹, a picture of the entire universe at almost exactly the same moment in time. It is the best picture we have of the structure of the early universe.

The universe appears to have expanded evenly since then. We know it’s expanding now. Light is like a wave. Since the speed of light is constant, an illuminated object moving towards us has its wave crests squish together, turning bluer, and an object moving away from us has the distance between crests expand, turning more red. This is called a red shift. Since he knew the colors of certain celestial objects, Edwin Hubble was able to observe that the further something is from us, the more red-shifted its light, and therefore the faster it is speeding away.

Since we know that the early universe was hot, dense and small, and we know now that it’s cooler, sparse, big, and expanding, we can reasonably deduce that, long ago, there was a Big Bang. The universe exploded.

Strikingly, the CMB is almost the same everywhere you look. There are minor fluctuations, but even they seem to have the same distribution everywhere. The CMB, our best picture of the early universe, is extraordinarily smooth. It is one of the smoothest things ever observed in nature. This might not seem like a mystery. You might imagine that anything expanding, hot and dense would look roughly the same in all directions. It needn’t. Nebulae are formed by exploding stars, and they aren’t particularly smooth. In fact, in nature, it would seem, more often than not, that explosions are messy.

In 1981, Alan Guth suggested what might be called a ‘recipe for a universe’: inflation theory. Until then, nobody had come up with any good ideas for why the universe was so smooth and even. It is as if God² had pressed the entire universe with a cosmic clothes iron.

Guth said, suppose you started with pretty much any initial universe. Suppose you also had an extremely strong, extremely smooth field of energy. If this field started dumping energy into the rest of the universe, it would also evenly expand space itself.³ The universe would undergo a period of exponential expansion — inflation — having the effect of flattening and smoothing the rest of the universe. Inflation is God’s clothing iron.

A flat, smooth universe isn’t the only thing that inflation predicted. For example, at small physical scales, quantum mechanical fluctuations persist. During inflation these fluctuations are blown up as well, and these would seed, almost entirely, the cosmological structure of the universe. We see these fluctuations in the CMB. According to inflation, they are tiny quantum fluctuations blown up to a cosmic scale. They are, quite literally, the ancestors of our galaxies.

It wasn’t just that there were fluctuations. Inflation theory predicted a very specific distribution and type⁴. When people finally had the technological capability to check, that’s just what they found. The universe appeared, at a cosmic scale, astonishingly consistent with this simple theory. Yet recently our observational capacities have improved. A CMB survey called WMAP has uncovered several surprising and unexplained features, not all of which fit well with the our previous inflation theories.

If you divided the sky in half by tracing the orbit of the earth around the sun⁵, and compared, in each half, the size of big fluctuations, those between 3 and 5 degrees wide, you would come to the conclusion that one side has fluctuations outweighing the other by an alarmingly large amount. One side of the universe is bumpier than the other. Moreover, the difference is larger than would be accounted for by randomness, at least 99 times out of 100.⁶

This asymmetry looks real. It has been checked against every known experimental error and background effect astrophysicists have been able to think of. And if it is real, our previous inflation theories, with one field of energy to inflate the early universe, won’t work. They can’t account for this anomaly.

The authors Erickcek, Kamionkowski, and Carroll don’t merely point out this problem. They posit a solution. They describe another inflation model, consistent with our new observations. They suggest the universe had not one, but many fields of universe inflating energy. There’s just one problem. At least one of these fields needs to be asymmetric.

Where could such an asymmetry come from? It is possible that we’ll never know. Cosmology offers us the hope of uncovering consistent, compelling stories of our origins. Thousands of independent observations fit neatly in cosmology’s book. But while we may discover a few lost pages from our first chapters, we may never know all reasons why our book was written in the first place.⁷

Nevertheless, the authors make an exciting point. Wherever the asymmetry in the inflation field came from, it must have existed before inflation. It must have existed before the big bang. We had once imagined that time before our explosive beginnings would forever remain a mystery. Yet hidden in the CMB are hints of times earlier still. In this wonderful piece of work, the authors carefully consider what anomalies in the CMB could mean. And in the process, they may have discovered a way to look farther into the past than ever before.

Notes:

[1] – Since light moves at a finite speed, when we see something far away, we’re seeing light emitted in the past. What we see of something a light-year away is (at least) one year old.

[2] – I mean ‘God’ here as in a figure of speech. Feel free to substitute ‘Mother Nature’, ‘Allah’, or the ‘Flying Spaghetti Monster’ while reading.

[3] – What does it mean, exactly, for energy to expand space itself? It’s roughly analogous to blowing up a balloon. We know that the gravity of the universe, just like the elastic outside walls of a balloon, pull its contents inward. In a balloon, air pressure pushes against that inward force of the walls. During cosmic inflation, the inflationary force pushes against gravity. There’s one important difference though. We don’t actually know what the inflationary force is. Air blows up our balloons, but we have few clues as to what blew up the universe.

[4] – The quantum fluctuations predicted by inflation follow a nearly-scale-invariant random Gaussian distribution. These fluctuations show up in the CMB, and for the most part follow these predictions pretty closely.

[5] – The line dividing the two halves of the sky here is called the ecliptic.

[6] – Formally, this statement is true at at least the 99% confidence level.

[7] – There are some questions forever beyond our grasp. Even if we knew from where the Big Bang had come, we could always probe further, and ask where that came from.

An article on one bright young man, Moshe, recently appeared on Hacker News. For a long time I’ve been meaning to write about the subject, and what was to be a simple comment morphed into this essay.

The story of educational acceleration is an old one. Curious, bright children learn and explore rapidly on their own, and interactively with their parents. The world is like a playground for the growing mind. The child takes in everything. Eventually, these children find themselves mired in school’s morass. There are new adventures: more kids, older kids, a new environment. Yet kept in one place, individual attention of parents replaced by lectures from often overtaxed and uninterested teachers, their minds are left to go fallow. While some of school is new, and quite enjoyable, boredom and obedience, for the curious child, is torturous, a fact which lucky children and mindful parents come to confront.

Alternatives appear: skipping grades, dropping out, home-schooling, gifted programs, science fairs, participating in the popularity game, sports, focusing on musical or athletic achievement, playing hookie, becoming jaded.

After entering junior high I pretty much stopped responding to the world at large. Life rapidly degenerated. I quickly dropped out, and luckily my parents didn’t make me go back. At that time both of my parents were very busy with work, and so homeschooling couldn’t work for long. We discovered that college was much cheaper than private school, which didn’t seem very good anyway. We argued my way in.

I was given a huge amount of freedom. Like Moshe’s parents, and like many others, mine urged me to slow down — advice I tried to take, though my eyes were always bigger than my appetite. I was amongst a culture that respected me for good reasons. I shared so much more than with my age peers. I was learning so much. A growth spurt hid my relative youth — soon nobody suspected I was so young. I truly enjoyed my time in college. Nothing brightens a teenager’s world more than responsibility, respect, and freedom.

There are dangers. Early fame is a minefield, for many reasons, in many fields. I think it’s hard for most people to live up to past successes, the ‘promise’ they had shown. But there are worse dangers. Two extremes haunt the gifted:

It is easy to feel that what one has achieved is in some sense artificial. That one has been built up out of nothing — that really, respect is in a large part undeserved. ‘Imposter syndrome’ is a common term for this feeling. People can feel themselves to profoundly wanting. Phony. But this is by no means exclusive to the academically accelerated. It follows early achievement wherever it can be found.

Alternatively, one can fall into a trap of picking activities on the basis of their impressiveness. One can miss out on more intellectually simulating, enjoyable, and valuable things in life, all in order to participate in a vapid achievement contest. One strives to be the youngest smartest fastest most daring most broadly educated highest iq’d phd’d nobel prized ivy leagued quantum genius prodigy in the world ever.

This happens way too often. And it affects all of us. One can feel so small next to the prominent. Long shadows can reach deep into other’s egos. Perhaps worse, all of us have felt a need to be great at times. It can become overpowering.

It’s a trait that is unfortunately encouraged by some parents who should know better. It’s certainly encouraged by society at large. The wiser guardians of bright kids sometimes say that raising them is like having to constantly pull on the brakes. They must. The world keeps pushing them.

One of the saddest things is that it’s all futile anyway. Inevitably, you grow up. Prodigies become ex-prodigies. This is lost on too many people.

Beyond questions of ego and achievement, there’s another problem with acceleration. Those who advance so quickly do so by letting delight pull them. One falls in love with the rhythm of important, challenging work, with the excitement, with ideas, and with the culture of the mind.

But you can’t live life like this forever. One must learn other lessons while growing up. At some point, we all wish for a pause button. We all need time to pull our identities together. We all need time to relax, and there is much that we need that cannot be provided by an overscheduled life. It is far too easy, with successes behind you, with fame around you, with drive and courage within you, and with a love of your work pulling you, to let that pause button go, and to refrain from asking for respite before hitting rock bottom. I know. It happened to me.

It felt like the world was ending. I felt like I was abandoning my love and my duty. I felt incredibly selfish. And I was exhausted.

It was not so bad, it turns out. My life rallied. It is difficult to explain how much better I feel. And surprisingly, taking a break, and taking time to reset myself personally, has even started to improve my work. Much of that many be a function of happiness. But part of this may help even the happiest of people. Pressure to do great things can rush life. It is so hard to pursue daring thoughts in a life too hurried.

So my advice to Moshe, and anyone who finds a glimmer of recognition in these words: stay curious, let your love of something pull you, but do not hastily give all of yourself and find there is nothing left to give. Don’t engage in the achievement rat race. Listen to praise from those you respect, but ignore it from those you don’t. Push back when society eggs you on. Focus on things that matter the most to you alone. Ignore fears of being an imposter — be authentic to yourself, that is enough. Keep your hands on the brakes. Enjoy your freedom of time and mind. Pursue daring thoughts when the urge strikes you. And when your heart begs pause of the world, listen.

Further Reading:

The letter arrived silently. I’m embarrassed to admit that my body read like a chapter on the stages of grief. Shock. My stomach churned as I turned inward. Admonishments ‘not to take rejection personally’ meekly confronted universal doubts. Egos struggled against a rethinking of everything. Hours of discussion lapsed. Plans of what to do were floated, accepted, rejected, forgotten. Night passed to sunrise before sleep. Denial. I woke up recalling a story of one rejection mailed out accidentally, to a startup later to succeed. Thoughts strayed from their success – all I could register was the possibility of a mix-up. Anger. I stewed. ‘It doesn’t matter what they think. I know where they’re coming from, I know what they must think of us. They’re wrong! And we don’t need them anyway.’

Finally, acceptance.

YCombinator has its own personality. Their opinions are their own. They have perspective and wisdom and the will to apply it. But they have no more dominion over truth than the rest of us, nor are they immune to the blindspots that all beings must endure, nor would they suggest otherwise. It’s taken much for me to fully understand this

For years I’ve contemplated PG’s word as philosophy. It has changed the way I think. As PG writes, heroes are those of whom you’d ask “what they’d do in the same situation,” as for years I’ve asked myself. He is a hero.

But PG isn’t my conception of him, no matter how close his words have touched me. YC isn’t what I know of it, no matter how right I’d judged it to fit. And we are not what any others, no matter how wise, can ever hope to judge us.

Once can’t come to truly know people by their writings alone. Yet in my imagination I felt I could. Upon rejection, it couldn’t feel like a group in Mountain View had overlooked our ideas in favor of ideas more compelling to them. Their judgment felt universal. It took hours to move past the looming question: ‘How are we deficient?

Our idea seemed nothing but wonderful. Yet after taking a few steps away, and looking back, there are perfectly valid reasons for skepticism. PG is even on record stating that building applications for people interested in local events is a ‘perennial tarpit’. This isn’t a YC idea, even if it was YC inspired.

It started with a question: could software help people connect in real life? Nothing we knew about was any good. Nobody sane would sift through lists of events online. Dating sites mostly sucked. Put people in the position of a judge, and romance cannot bloom. Chat rooms sucked. There are trolls and perverts everywhere. And the internet at large has no location. Rarely would you find someone nearby. Even more rarely could you expect to know someone by their online persona: my YC dilemma writ large. And we knew, or believed, or hoped. There must be a better way.

But how could they believe us? With neither users nor customers nor a finished product, we have no proof of what is perhaps best described as a social experiment. All we have is code, determination, and the belief in our hearts that we are on the right track.

That belief can only come from imagining our success as we do. But our imagination is not theirs. Without proof, others cannot take the same leaps of faith. It is so easy to take rejection as a challenge to ourselves. It is only a rejection of our rendition. We are unproven.

How were we deficient? Perhaps we don’t see the problems others have seen: traps laying in wait for us just beyond the bend. Or perhaps it is because our imagination, try though we might, could not be made to fit with our heroes, and we were so certain of our shared viewpoint that we were blind to this possibility.

Much attention is being paid to YCombinator clones of late, and there are murmurs of the movement forcast more springing up. When the first batch appeared, they pointed out that YC would eventually struggle with deal-flow. Plenty of good startups were being passed over, so why not fund those? Too little effort was invested in explaining why one might prefer them to YCombinator, dooming them to the appearance of funding YC rejects. Later attempts focussed on a different limitation: bright entrepreneurs must contend with, in addition to harrowing expense, culture shock, and commitment, an immigration system unwilling to receive those planning to work for themselves.

Seed funding is limited by a third resource. For proven teams one needn’t far sight to bet dollars on success. But to bet on the earliest of early startups, run by strangers, not friends, toiling without traction, with working, complete technology, takes more than guts. To back such a startup requires they imagine the success for themselves.

There is hope for new players in the global seed funding game. But none will succeed as YCombinator does, winning successes from strange ideas with unproven players, unless like YCombinator, they cultivate a personality of their own, and call upon their imagination to see as the bright minds applying dream.

It’s not given much press, but the the Halting Problem is intimately related to Gödel’s First Incompleteness Theorem. Indeed it produces it as a correllary. Historically, Gödel’s incompleteness results were proved by hacking arithmetic into a Turing complete system, and this is still how they’re explained today.

There’s a one-to-one bijection between computability of a function and provability of a statement. Hence, the short, and generally accessible proof that the Halting problem is not in general computable for an arbitrary input is also a proof of the ‘most important, surprising result in logic’, namely, that some results, which have may have a perfectly valid truth-value outside a system, cannot be proven within it. One only needs the notion of a computer to follow this line of thinking, which is, in essence, what Gödel did. But the Halting problem is much easier to grasp. I’ve had children understand it, though it does take some walking through!

The interesting thing about the Halting problem is that it’s unsolvable in full generality, independent of whatever special capabilities the system has available. To see this clearly, consider the proof.

Question: Does there exist a (halting) program H which, given any program P, figure out if it would halt, for any input I?

Assume there exists such a program H. Construct a program T as follows.

(Program P, Input I) => (Boolean Halts):
if H(P,I) is true run forever
otherwise halt

Now, call T on itself, with itself as an input. Our assumption presupposes that H always halts. If T would halt on input T, then T will run forever. And if T would run forever on input T, then T would halt. This is a contradiction, so no such program H would exist.

This becomes philosophically and theologically relevant because it doesn’t at all depend upon the form that H might take. Suppose one has funky laws of physics, or happens upon a ‘book of truths’. Or suppose one is some kind of omniscient god, able to just ‘know’ whether something halts or doesn’t. All of these can be wrapped up by saying they’re part of H, whatever H may be, and even if you assume that H¹ always exists and is well-defined for some set, you cannot also have H be well-defined for programs which include calls to H.

In short, an omniscient entity can’t exist. Not even a god. When speaking amongst nontechnical types this conclusion is usually met with some form of drama: praise or otherwise.

‘Omniscience’, as it happens, doesn’t survive close inspection.

Notes:

[1] Taking the existence of H as given, convention names it a ‘halting oracle’.

The central limit theorem states that if you have many small, independent, random variables, then their sum is distributed approximately as a bell curve. Strikingly, almost everything is made up of many small parts, and these parts don’t tend to influence each other very much.

So much of what can measure seems to fit a bell curve. This is why the normal distribution works. Because this assumption tends to work well, it is usually taken as a matter of course. Students are taught it, lecturers preach it, researchers apply it, and startlingly few stop to question it.

Suppose the variables are not small, or suppose they’re not independent. Suppose, under certain conditions, the value of one variable would seriously effect another. Suppose we’re talking about the buildup of snow on a mountain slope. Most of the time, snowflakes can gradually build, without significant effect. But once enough builds, you don’t find snowflakes resting calmly upon a drift. What you find is an avalanche.

The sum total of snowflake movement isn’t what we might expect. The snowflakes on the top used to be lightly packed by the new, gradually coming down. The snowflakes on the bottom used to just sit there. But they’re not just sitting there. They’re moving fast, and they’re moving down.

The central limit theorem doesn’t always hold! If you have a model where, most of the time, a change in one part doesn’t  effect another much, but some of the time it really does, then you can’t assume that your outcomes will follow a bell curve. Entirely different outcomes are possible.

Traditionally, when you have some measurement that’s far outside of normal, you call this an outlier. In much statistical analysis, these are ignored or thrown out (for example, even the most extraordinary measurements won’t effect a median). This is useful if you want to study the ordinary behavior of something. But sometimes, the information you gain from outliers is by far the most interesting. Some of the outliers are expected in normal distributions (bell curves). But some of them are outliers because the model doesn’t apply. Some outliers are avalanches.

We started from an iffy assumption. Not everything is made up of independent random variables. Parts effect each other. Sometimes it’s violent, like a chain reaction, or our avalanche. And sometimes, it’s magical. Life is an example of this. If the chemicals that made up our bodies didn’t bond so strongly to one another, our DNA would unwind, and you’d be a puddle.

Statisticians try to account for this. One example on the tips of our tongues lately: finance. Roughly speaking, the ‘beta’ used in Finance means the volatility of a stock with the linear correlations with the company’s existing portfolio factored out. Despite this, there seem to be ‘six-sigma events’ happening all the time, things which, according to the theories, and ‘ordinary’ date, really shouldn’t be happening at all. ¹ What’s going on?

When small effects just add up simply, you can model them by what’s called a linear model (so called because if you add up small things along an XY graph, you’ll get a line). This doesn’t always work, and in fact, most interesting phenomena are non-linear.

Models are limited. They break down. And one can’t really account for every possibility. Financial markets can collapse due to dustbowls, and furthered by widespread investor panic. The destructive power of armies during world wars can be dwarfed by exhaustion and a powerful flu, though normally flu is beaten with chicken soup.² And planets full of ten story tall reptiles can be wiped out by meteorites. No number of small rocks would matter — usually they just ping off them. Bet they didn’t see that coming.

It isn’t too long before one fears running directly into Godel’s Incompleteness Theorems: checking all assumptions isn’t merely inconvenient, for most interesting problems, it’s impossible.³

The next time you hear about your monthly six, or seven, or eight sigma event, keep this in mind. Outliers are where the model breaks down. They happen more often than standard models would expect, and they often point out problems in the understanding of the system. If they start happening all the time, start mistrusting your statisticians. And your wall street.⁴

One can show the recursiveness of the model verification problem. Suppose one says to you: drug x worked better than drug y, with a 95% confidence. You might reply: what’s the confidence of that confidence? Somewhere along the line, they’ll have to say 100, and then they need to prove consistency. One can show that there are theories with unknown external variables and relationships (as are dealt with in statistics) aren’t even formally recursively enumerable,⁵ but even if they were, no such theory could contain a statement of its own consistency.

Back on planet earth, is there anything we should be worried about? Some who talk about the climate claim that, since basically the temperature of the earth just goes up and down naturally, it’s not really anything to worry about. But outliers can mark where models break down. Where systems change. This is a chart of temperature over the past few centuries. You may notice, to the right, an outlier. Maybe it’s telling us something.⁶

Further Reading:

1 – The Psychology of Human Misjudgment – Charlie Munger

“Now let’s talk about efficient market theory, a wonderful economic doctrine that had a long vogue in spite of the experience of Berkshire Hathaway. In fact o­ne of the economists who won — he shared a Nobel Prize — and as he looked at Berkshire Hathaway year after year, which people would throw in his face as saying maybe the market isn’t quite as efficient as you think, he said, “Well, it’s a two-sigma event.” And then he said we were a three-sigma event. And then he said we were a four-sigma event. And he finally got up to six sigmas — better to add a sigma than change a theory, just because the evidence comes in differently. [Laughter] And, of course, when this share of a Nobel Prize went into money management himself, he sank like a stone.”

2 – Wikipedia on the Spanish Flu.

3 – For the curious, a quick way to show this is by adding a ‘randomness’ oracle to a turing machine, and showing that you still can’t solve the halting problem.

4 – Wikipedia on the 2007 subprime mortgage financial crisis.

5 – As wikipedia states, a recursively enumerable language is a formal language for which there exists a Turing machine (or other computable function) which will enumerate all valid strings of the language.

6 – (update) For further musings on uncertainty and randomness, a terrific book is Nassim Taleb’s The Black Swan. A related essay is available here. Also intriguing is the working paper “Extreme Sample Selection Bias: Conditions that Cause the Correlation Between Two Variables to Switch Signs” by Tim Groseclose

“Everyone always discusses algorithms and such on this forum. However, there is one trait that is perhaps equally important as knowledge of algorithms – good problem solving strategy. By that I mean not jumping into solutions too quickly, thinking through everything before writing any code and such. Can anyone share what they’ve figured out that helps them? I’ve noticed that I have several flaws when I solve problems in general. I am too quick to start writing code. Also, I don’t think through my algorithms thoroughly. Also, there are many other things. Does anyone have similar flaws that they’ve already passed. If you have, please share your strategies for getting better at problem solving in general. Anything would be helpful, since I have the TCHS tournament coming up and I want to go home with some sort of prize.”

I wrote back, on May 10th ’07:

One thing I want to start doing is to make estimates of the implementation time of parts of my solution, how difficult each will be to code correctly/debug, and how sure I am that my method will work in the first place.

I’ve gotten very good at making estimates of time and space constraints. Now they are second nature. But knowing how best to solve a problem practically is still another area which I need to improve.

For example, for the last hard problem on TopCoder (circa May 9th, sorry, you’ll have to register to see it). I broke the problem down. Six steps toward victory! Five of those steps were really quite simple, though a few bugs crept in. The sixth was brutal — I could solve the problem by implementing a complicated comparison function object which recursively called itself on sorted sets of children. It was really crazy. To get it all working properly took nearly eight hours after the match.

Each bug, if not isolated, tends to increase debugging time by something like the square of the number of interacting components: the rate at which the number of dependencies would grow. Perhaps it would be simpler to either independently verify each component (by having pre-written tested functions or methods), or to organize the program, inasmuch as possible, as a linear pipeline or other simple acyclic graph. In fact, one might take a few seconds just to sketch the program idea out — how strongly coupled is it? What can I be sure of? How much of a pain will it be to debug? Can I make it less so? If not, can I anticipate debugging requirements and build in information gathering tools?

Furthermore, I was lured by that fact that, conceptually, ‘most’ of the problem could be coded simply. This blinded me to the fact that the code that was doing all of the work was almost completely unsupported by the auxiliary framework I had envisioned, and that the whole thing was a tightly entangled mess. Maybe the other coders can handle it. But I think it’s mostly because they know ways to keep things from failing — keep bugs isolated and visible enough to squash.

What I really needed to do was find a data structure or recurrence that was simple enough to verify piecemeal. I cannot be trusted to write bugless code. In fact, I should guard against it by breaking things down. This has the added benefit of making more clear what I need to do. I would need to remember less. Increasing my effective working memory has an impressively super-linear effect on my problem solving ability, how fast I can learn things, and also how much fun I’m having. I’ve come to realize this only recently, while studying for PhD qualifiers using index cards, scrap paper, two levels of books, one rough, one ‘canonical’, a wiki system of notes and a flashcard engine we made for it.

The last thing — and I think this is really more rudimentary, but I’ve yet to learn my lesson, know exactly what your variables are. In all of my experience this is the worst problem I have. Again and again I find myself with a few minutes to spare before the end of the coding phase (for readers unfamiliar with TopCoder, to get any points you need to have a solution that is (almost) exactly correct, submitted before the 1:15hr deadline). There is some minor error which I’ve detected. The tests give one more in some cases, one less in others, maybe it fails on an edge case. The tinkerer in me asks ‘hey! what happens if I turn this knob? Cross that t? Increment that i?’. So, I try it. Does it ever work? No. Not one damned time.

It’s voodoo programming, sure, and I make fun of it when I see others do it, but there’s just something — as that clock is ticking down, for some reason it feels like the competition is not about making sense of problems but just trying loads of things really really fast. And when I’ve defined an ‘r’ or an ‘x’ or a ‘q’ as some kind nifty function, where I only know the ‘approximate’ relationship with what I’m doing (up to a least significant bit, or maybe it’s part of the state of my program’s implicit automata that I used to speed up coding), when I’ve done this and the program nearly works except for this minor case or that minor case, it takes me nearly forever to disentangle what something is from what it is supposed to be. Even though I feel like I’m so close, when the pressure is on I just can’t handle it. It’s just way too complicated to do in my head when I keep having to watch the clock.

——

One advantage of TopCoder is that you’ll be constantly stretching your abilities. You’ll be solving many more problems than you otherwise might, and because of the time pressure you’ll learn faster, and it will stick better. I came across apropos comments by Philip Greenspun

1. A person won’t become proficient at something until he or she has done it many times. In other words., if you want someone to be really good at building a software system, he or she will have to have built 10 or more systems of that type.
2. A person won’t retain proficiency at a task unless he or she has at one time learned to perform that task very rapidly. Learning research demonstrates that the skills of people who become accurate but not fast deteriorate much sooner than the skills of people who become both accurate and fast.

and

Whatever the training task, the pace must be ruthlessly brisk. The learner should be expected to build at the same pace as an experienced developer. The difference between the learner and the wizard is that you expect the learner to make a lot of mistakes. The system as built may be awkward or not handle error cases properly. That’s okay. Training research shows that if you get speed now you can get quality later. But if you don’t get speed you will never get quality in the long run. We practice this technique in 6.916, Software Engineering for Web Applications, our course at MIT. Each student builds five database-backed Web applications during the 13-week semester. The first few that they build, during the course of the problem sets, are not necessarily elegant or optimal, but by the end of the semester they’ve become remarkably proficient, especially when you consider that each student is taking three or four other classes.

This rings remarkably true. Periods where I’ve needed to learn a skill in a short period of time have inevitably resulted in me learning them well. TopCoder is a terrific example. So was the high school math I needed to catch up on when I found myself in the university math department at 12. Cramming seems to have gotten a bad rap. I learn best intensely.

It’s not overarching. One of the most striking counterexamples is my friend Joel. When I met him in first year, he had already read code complete, and was working on his own games. He painstakingly engineered every assignment to be bug free. Extensible. Readable. Coherent. Most of us rationalised such behavior as making sense only because later assignments built upon the previous.

Joel lost the floppy he kept his work on. Most of us would have been upset. He was unphased. He did all his engineering for practice, and was happy to be able to do it again.

On TopCoder, he isn’t always the fastest. But he is the only competitor I know with a perfect accuracy rating. I don’t believe this skill was developed under time pressure. But I do believe that it entailed long hours, and sincere commitment to build something well. That is quite rare.

——

It’s been a while since I’ve done any of these contests: things keep getting in the way. Making rent, and taking care of other basic necessities has impeded my time for fun. Hopefully my skill wouldn’t have deteriorated in the interim. I actually have hope that I’ve unlearned more of the bad habits than my good. :-)

Further Reading

Misof’s description of the solution to the “RussianCheckers” TopCoder problem highlights a crucial part of the problem solving process: understanding how to reliably, quickly, and accurately implement a particular solution. The problem statement gives all the information one needs to solve the problem, but the task of implementation is so hard that only two competitors manage. Petr, the winner of the match, has a solution that’s a paragon of readability.

Tonight it’s winter in Berkeley. 53 degrees and raining, and outdoors, warmed by a heat-lamp, sheltered by an awning. I draw spiced apple cider through my lips. Classical music plays. An earbudded minority vote silently with their ears. Old men watch hooded students roll down the hills towards Telegraph Ave, Berkeley’s epicenter of hippiedom. Moist, newspapers ink the hands of activists, busily plotting the victories in the years long struggle to ‘save the oaks’. A young man lids a drink and smiles at me. Separated by glass, headphones, and 12 feet, I smile back. We wave.

There’s something magical about this place.

I don’t know anyone here. To arrive I flew four thousand miles from my place of growth. This place isn’t home. Yet there are few places that attract me so strongly. Modern life has been made private. And in doing so, life’s become a little lonely.

Builders of great cities have long understood that life would, but for misfortune, consist of more than work and one’s home. The vibrancy, energy, and community grown in what are sometimes called ‘third places’ played part in much of the world’s social, political and intellectual revolutions. The roles that the Roman forae, French salons, and English learned societies played in scholarship has been tremendous, as has been the influence of American chautauquas, worker’s taverns, and artist’s ghettos in social and political spheres. These public, accessible, talkative, comfortable playful places are magnets for folks of many stripes. Creativity can thrive there. Unconstrained by work’s implied unity of purpose, and decoupled from the tight bonds around one’s family and home, third places give marginal people, ideas, and voices room to grow, people to hear them, perspectives to challenge them, and food to help keep the conversation going.

Yet in much of the world, third spaces are dying, or being replaced by poor substitutes. Nearly half the USA lives in a suburb. Some try to define suburbs with density requirements, or as satellites of more major cities. But that doesn’t seem to capture the idea: almost all of San Jose is a suburb. Los Angeles, despite its skyscrapers, feels desolate in the same way, albeit in a different intensity as Jacksonville, Florida. We’ve heard it all before suburbs have too much pavement, too much repetition, too many box stores, too many cars and vans and soccer moms and private schools. But nothing so much makes a suburb if not a lack of third places. It may be the defining characteristic. There’s nowhere to go. There’s nothing to do.

So people fight back.

They watch TV; sitcoms and reality TV, predominantly: short, canned glimpses into lives where people are living. The same people, born of similar neighborhoods decades later, would watch YouTube over Friends, prefer to text instead of talk, and paste pictures in MySpace instead of a scrapbook. They’d hangout in different places, most marginal, most online. Yet despite these differences, the core behavior remains the same. People reach out subconsciously and compulsively to the world. They have a social itch, and nothing in their known universe will let them can scratch it the old fashioned way. Times have changed. Things are just too far away. Instead, they substitute.

Centuries ago, Shakespeare’s melanchoic Jaques proclaimed:

“All the world’s a stage,
And all the men and women merely players;
They have their exits and their entrances;
And one man in his time plays many parts…”

If the world can be a stage, then the web is valued stagehand. Millions now live essentially online. Working means sitting behind a computer. Home means sitting behind a computer. Hanging out means sitting behind a computer. The web has made possible access to millions more places than urban planners had dreamed of. It deeply affects our culture. Yet, in many ways, it comes up short. The web doesn’t serve apple cider, nor can it provide heat lamps for the evening chill. Eyes I catch won’t smile back at me. I won’t be able to catch eyes at all. To how many does it occur that when reading the New York Times, or Reddit, or YouTube, hundreds of eyes are following the same pages they are? Does it occur to them that comments on popular postings represent the opinions of only an assertive minority? Does it occur to the millions of highschoolers using Wikipedia that they can change a page at will? That thousands of others have requested the same references for the same civil war paper as they? So much interactivity of the physical world has been lost in the transition to the web. How much can we get back?

——

PS: To all you aspiring entrepreneurs, ThirdPlace.com appears to be parked. The tagline?

ThirdPlace.com

The Leading Competition Site on the Net

——

Further reading:

rokhayakebe describes online places as cities in a comment on Paul Graham’s Cities and Ambition

“I think the biggest shift in deciding where to live is happening “online” rather than “offline”. I am an online nomad.

I never lived and never will live in MySpace. I do not like the MySpacian’s message (Hey let’s try to see who has more friends and hookups).

I sometimes spend time at Facebook. I lived there for a little while until I realized I am not so much into keeping in touch and I had no friends in the few hours I spent in college. When they open their borders, that’s when I found that I do not like the Facebookies message (You should throw more pies and send more kisses)

I vacationed at Twitter, but it is not really my cup of tea. I still don’t get their message (Life is a popularity contest).

So Where do I live? Well, I live mostly in HN. Although I sometimes get into arguments with the habitants, I have yet to find another city that beats the intelligence, vibe, energy and support I witness here. I take a daily ride to Techcrunch City and NYT, but I make sure I come back home to HN and mingle with the people who live here.”