Somewhere in the world there are magicians who are still talking about a magic trick they simply can’t explain… or perhaps they have solved the mystery. The linchpin is a single idea. One fact.

Almost 20 years ago I crossed an ocean to attend my first live magic conference. The headline acts/speakers included Jeff McBride, Eugene Burger, and Darwin Ortiz; all major influences of my work and I couldn’t pass up the opportunity to meet and work with them in person. I went alone and knew nobody attending, although that would change before the convention even began.

In the restaurant on the eve of the convention, I dined solo and mostly people-watched. I watched the retirees who have devoted a portion of their newfound free time to rekindling a childhood hobby. I watched the one or two young kids with their tiger-parent relentlessly molding them in to future Vegas headliners, and I watched the adolescents who spend every waking moment drilling the most difficult sleights and the most elaborate flourishes. There was a whole group of the latter engaged in what could be best described as the legerdemain equivalent of a rap battle, each attempting to one-up the displays of dexterity demonstrated by the last. With nothing to do (and nothing to lose) I approached this group. Minutes later, I left them speechless.

Interrupting, I borrowed a deck from a member of this group and proceeded to start dealing the cards, somewhat haphazardly, into two piles. “Do you know ‘out of this world’?” I inquired, as I continued the seemingly random dealing of the cards. “It’s the trick where you deal the cards into two piles, but somehow all the reds are together and all the blacks are together.” With no moves and no hesitation, I turned over the pile on my left, then the pile on my right. The cards were, indeed, separated by color. They were speechless until one member of the group finally exclaimed (only half-joking) “What is this sorcery?!” I simply smiled and walked off. As I left, the conversation shifted to theorizing about what just happened and, I am told, continued the rest of the night–and the rest of the convention.

Depending on your background, the description above might sound like just another card trick but, to that group, it was a miracle. It was, indeed, a variation of a well-known card plot called ‘out of this world’ created by Paul Curry in 1947. The effect is the colors are separated (usually by the audience participant) but, in the original, there are necessary extra steps to make the magic happen. In my demonstration, none of those extra steps took place. The cards could not be in any known order since I directly borrowed a shuffled deck in use. The cards had no marks or preparations, it was just a regular deck. There could be no sleight of hard as the eagle-eyes there knew what to look for and didn’t blink once.

It turns out that what I did will easily fool 99 out of 100 magicians. Being fooled or not hinges on knowing one single fact. Much of magic operates this way. I’m often asked if magic ever “fools” me. The answer is yes, but rarely. In the 25 years I have been performing professionally, I have read hundreds of books from obscure old manuscripts to mainstream works. I’ve learned thousands of magic principles and most magic tricks rely on only one or two. The saying really is true, once you know the secret, the mystery is gone (at least for that one trick).

This is an example of linchpin knowledge. Facts, or ideas, or patterns that single-handedly bring the whole solution together; the one dot that connects all the dots.

What is This Sorcery?

In my career as a magician, I have heard this question many times; from magicians and lay-audiences alike. Usually phrased as “What is this sorcery?” but perhaps more aptly it should be “What is this, sorcery?” to which, the answer is unequivocally “yes.”

My long-time friend and mentor, Jeff McBride, who was the headline act at the 2004 Daytona Beach Festival of Magic, defined sorcery simply as “access to sources.” Whether you are a modern day wonder-worker, or a “sorcerer” from a bygone era, the sources are always knowledge or technology. Historically the knowledge was available in the form of books, manuscripts, or oral teaching; today we have the world-wide-web, video, zoom lessons and more. Without the knowledge obtained from these “sources” the uninitiated are left struggling to explain the inexplicable and often assumed the “sources” must be supernatural. As Arthur C. Clarke once famously stated, “Any sufficiently advanced technology is indistinguishable from magic” and the technology we create today truly is magical.

In one of my favorite conference talks I have had the privilege to watch, Jessica Kerr approached the debate of whether software is “art” or “craft.”

“Software is not craft. We aren’t housepainters… Software is not art. Software is the next thing after art. It is something new, that we’ve never been able to do before.”

-Jessica Kerr - The origins of opera and the future of Programming

Just like sorcerers, magicians, and wizards; we use our knowledge to conjure a new reality and future for humanity. We, too, rely on sources be they literal sources (thanks Richard Stallman!), books, courses, StackOverflow, or the combined knowledge of our teams. We are knowledge workers. Knowledge is our strength, knowledge is our weakness.

Enter the Linchpins

“A linchpin is the person or thing that serves as the essential element in a complicated or delicate system or structure—the one that holds everything together.”

-Dictionary.com

In 2017, the April 1st edition of New Scientist magazine centered around knowledge with a number of articles on the topic. One such article by Sean O’Neill considered how much one could know and introduced the idea of a personbyte - the amount of knowledge one person can reasonably learn in a lifetime. The limitation is not necessarily what our brains can store (current estimates suggest our brains can store one petabyte of data) but rather the rate at which we can learn. Our field of software engineering vastly exceeds one personbyte, and is continuing to grow exponentially.

There will always be gaps in our knowledge. Often these gaps are merely tactical or syntactic; “How do I do X with framework Y?” or “How do I apply construct A in language B?” These are known unknowns and rarely are these linchpins. The unknown unknowns, however, are more insidious and it is here that most linchpins live.

The landscape of software is changing. The development work of the next decade will undoubtedly involve creating solutions to problems that currently seem impossible. There are almost certainly business problems you could begin to solve today with the help of a single linchpin fact, but the challenge is finding it. We are almost never aware of missing linchpin knowledge. These are the unknown unknowns, residing in the complex cynefin domain.

(By Tom @ thomasbcox.com - Own work - a re-drawing of the prior artwork found here that incorporates more recent changes, such as renaming "Simple" to "Clear"., CC BY-SA 4.0)

While we could once coast on familiar patterns and tools, the needs currently emerging require more than just gluing together packages inside of frameworks. Moreover, with technologies like GPT4 and CoPilot, the value of that aspect of programming is diminishing rapidly. Large language models (LLMs) trained on a vast source-code corpus perform tasks effectively in the “Clear” cynefin domain and will undoubtedly be a productivity boost for developers; the real-work that takes place in the complicated and complex domains will still require experienced, creative, thinking developers. This is the value proposition of the 21st century software engineer; how we think, how we solve problems, and how we connect diverse ideas into novel solutions. In short, the future of software development requires radically different thinking to that which is currently commonplace. The most valuable programmars of the next decade will be those who can discover the linchpins and translate these into a vision and a direction that teams and organizations can follow.

Discovering Linchpins - More Thinking, Less Typing

If, as Donald Rumsfeld says, “There are things we don’t know we don’t know” how can we discover these things? It starts with spending more time thinking and exploring the problem space and spending less time exploring the solution space.

In 1972, Allen Newell and Herbert Simon published “Human Problem Solving” based on their decades of work that eventually earned them a Turing Award. In this book, they first described the concepts of “problem space” and “solution space.” These concepts were further popularized by Dan Olsen in his book “The Lean Product Playbook.

In software engineering, the solution space is the domain of all the interesting technologies, tools, patterns, frameworks, languages, and libraries. They are fun to play with, they look good on our resumes, and they are where we as an industry spend most of our time. The technology problems of the past were typically so homogeneous that we could leap directly into the solution space and have a reasonable probability of success. The majority of books, tutorials, conference talks, etc. focus on building depth within the solution space. The solution space has the answers, but do we really know the questions?

“Forty-two,” said Deep Thought, with infinite majesty and calm.

It was a long time before anyone spoke. Out of the corner of his eye Phouchg could see the sea of tense expectant faces down in the square outside.

“We’re going to get lynched aren’t we?” he whispered.

“It was a tough assignment,” said Deep Thought mildly.

“Forty-two!” yelled Loonquawl. “Is that all you’ve got to show for seven and a half million years’ work?”

“I checked it very thoroughly,” said the computer, “and that quite definitely is the answer. I think the problem, to be quite honest with you, is that you’ve never actually known what the question is.”

-Douglas Adams, The Hitchhikers Guide to the Galaxy

The Problem Space, while typically less interesting from an engineering perspective, is where the questions reside. Only by exploring the problem space can we begin to ask the right questions, and only by asking the right questions can we consistently map problems to solutions. Discovering linchpin knowledge starts with skillfully navigating the problem space.

Understanding the problem and the business context and asking the right questions at the outset of a project will often lead engineers into surprising and unexpected places. Jumping directly to the solution space, while interesting, rarely yields new insights; typically we either get more of what we already have or waste time reinventing the wheel (or reinventing the flat tire).

Discovering Linchpins - Cultivate Breadth and Range

Filling in linchpin gaps requires looking at problems from new and unique perspectives. We gain these perspectives when we deviate from the course of deep specialization and explore ideas from outside of our core domain.

Specialization tends to shut off the wide-band tuning searches and thus to preclude further discovery.

-R. Buckminster Fuller

There has always been value in specialization; cultivating depth in a particular area or skill. In Malcolm Gladwell’s book, Outliers, he popularized the 10,000 hours hypothesis. In David Epstein’s book Range he argues that Gladwell’s narrow approach to mastery and success works well in simple or predictable domains (so-called “kind” domains in Epstein’s parlance) but not necessarily in complex and unpredictable domains (what Epstein refers to as “wicked” domains). In these pursuits, it is those with broader and more diverse experiences, perspectives, and cognitive tools that will thrive. The linchpins to modern, complex problems often reside in these broader perspectives.

Modern work demands knowledge transfer: the ability to apply knowledge to new situations and different domains. Our most fundamental thought processes have changed to accommodate increasing complexity and the need to derive new patterns rather than rely only on familiar ones.

-David Epstein, Range

In 2004, I fooled that collection of extremely knowledgeable and talented magicians because I had taken a broader approach to my study of magic. Their specialization was trumped by my range.

Early in my magical journey I read everything I could get my hands on, including an obscure book published in 1902 and titled “The Expert at the Card Table.” In this treatise, the pseudonymous author detailed the techniques they developed for cheating at cards over the course of their life. Although the fields of card magic and card cheating seem closely adjacent, there are significant distinctions between the two. Further exploration led me to the discovery of an almost entirely forgotten utility used by the card cheats of yesteryear; something called a “shiner.”

Although those magicians were highly-skilled and very knowledgeable, the absence of that one single fact left them with no ability to explain what they had just seen. It was specialization that fooled them.

While all I managed to do was fool a group of magicians at a convention a couple decades ago, similar tangents into adjacent fields have changed the entire course of history. I wonder where we would all be today had engineering and mathematics student Claude Shannon not taken a philosophy class while studying at the University of Michigan back in the early 1930s. Although this particular academic digression did little to support deep specialization in Shannon’s chosen subjects it did cultivate range and ultimately furnished him with the crucial linchpin to a future problem. He connected the 90-year-old work, “The Law’s of Thought”, by George Boole–who had the novel idea of representing something that was true with a “1” and false with a “0” to support logical reasoning–with his work designing electrical circuits. He could have spent years reinventing Boole’s life’s work, but instead he was able to make major progress in a short amount of time by having access to key linchpin knowledge.

“it just happened that no one else was familiar with both fields at the same time”

-Claude Shannon

This was neither the first nor last time that linchpin knowledge changed the course of technology. Another notable example comes from a story Steve Jobs recounted on June 12, 2005 when he delivered the commencement address to the graduating class at Stanford University. In this speech Jobs described how dropping out of college allowed him to follow his curiosity and cultivate range which, again, forever changed the course of modern computing.

“…much of what I stumbled into by following my curiosity and intuition turned out to be priceless later on. Let me give you one example:

Reed College at that time offered perhaps the best calligraphy instruction in the country. Throughout the campus every poster, every label on every drawer, was beautifully hand calligraphed. Because I had dropped out and didn’t have to take the normal classes, I decided to take a calligraphy class to learn how to do this. I learned about serif and sans serif typefaces, about varying the amount of space between different letter combinations, about what makes great typography great. It was beautiful, historical, artistically subtle in a way that science can’t capture, and I found it fascinating.

None of this had even a hope of any practical application in my life. But 10 years later, when we were designing the first Macintosh computer, it all came back to me. And we designed it all into the Mac. It was the first computer with beautiful typography. If I had never dropped in on that single course in college, the Mac would have never had multiple typefaces or proportionally spaced fonts. And since Windows just copied the Mac, it’s likely that no personal computer would have them. If I had never dropped out, I would have never dropped in on this calligraphy class, and personal computers might not have the wonderful typography that they do. Of course it was impossible to connect the dots looking forward when I was in college. But it was very, very clear looking backward 10 years later.”

-Steve Jobs

The key is to let your curiosity and intuition drive your breadth. Not everything you learn or do has to have immediate utility to your work; you never know when the random, seemingly irrelevant idea or perspective you develop today will be just what you need to solve a future problem.

Again, you can’t connect the dots looking forward; you can only connect them looking backward. So you have to trust that the dots will somehow connect in your future.

-Steve Jobs

Discovering Linchpins - Diversity and the Wisdom of Crowds

There is incredible value in developing your own cognitive diversity. You should absolutely open your mind to new ways of looking at the world, open yourself to new and diverse experiences, and be open to exploring ideas that don’t have obvious/immediate value. Sometimes these experiences arm us with a future linchpin, other times they simply provide a foundation that opens a new adjacent-possible linchpin. We don’t learn in large, discrete chunks; we learn in tiny deltas. Each delta creates new, fertile soil for the next idea to take root.

That being said, as valuable as this all is, it takes time and we still have the fundamental limit of the personbyte; the ceiling on what one person can learn and know in their lifetime. If you truly want to “10X” your creativity and problem-solving, it is important to not only scale up your own broad perspectives and multi-modal thinking, but also scale out at the same time by increasing the number of people interacting with–and thinking about–the problem. In other words, as the problems we aim to solve become harder and more complex; the value of diversity and teamwork becomes even greater.

Looking back through the history of science, many of the major breakthroughs came from “lone innovators” and often these were amateur or hobbyist scientists. As the “low-hanging fruit” in science began to be exhausted, new breakthroughs required more creativity and insight thinking. A great illustration is the distribution of Nobel prizes throughout the years.

The first 15 Nobel prizes were awarded to a total of 19 people. Contrast that with the last 15, which have been allocated across 42 recipients. This is particularly notable given that at most only three people can win. More to this point, in each of the last 15 prizes, several people felt left out and that their contributions weren’t recognized meaning the number of people who materially contributed to these scientific breakthroughs is much higher. Similarly the most influential scientific papers of recent decades involve larger and larger teams of authors, particularly among the frequently-cited works. In the complex problem domains, there are often many linchpins (or steps to discover the linchpin) which are often distributed across multiple minds.

“Literally everyone you will ever meet knows something that you don’t.”

-Bill Nye (The Science Guy)

While there is truth (and wisdom) in this quote from Bill Nye, knowing “something you don’t” will be of limited utility if that “something” isn’t the linchpin to your particular problem. You will dramatically increase the possibility of finding a novel solution to a problem by building and working with a diverse team.

In recent years there has been an increased emphasis on diversity initiatives in organization and, although this has resulted in a considerable amount of polarizing politicization, the fact of the matter is diverse teams consistently outperform homogeneous teams. When an individual possesses breadth of knowledge, perspective, and range; it is much easier to see the hidden connections across domains; with a similarly diverse team this effect is multiplied.

Many organizational diversity efforts focus on identity diversity (i.e. diversity you can see) although here we’re talking about cognitive diversity (i.e. diversity of thought, perspective, and mental models). The two aren’t mutually exclusive by any stretch, in fact diverse identities almost universally drive diverse perspective (and, by definition, diverse thought and mental models). Diversity initiatives are valuable as they help us avoid a natural bias toward homophily.

LLMs execute well in the simple domain, consequently we must move up the value chain. The ability to build and nurture effective teams is imperative. Beyond diversity, there must be a foundation of trust and willingness to engage in productive conflict otherwise all the cognitive diversity in the world will still result in groupthink. Encourage your teams to grow their own breadth and range, and lead a culture where every perspective has a voice; you never know where the linchpin is hiding.

Magic and Ego

Regrettably, the magic community consists of a considerable number of members primarily driven by ego. Performances in the style of “Look what I can do that you can’t” or “I know a secret that you don’t” grow tiresome quickly. I believe magic should be centered around giving the gift of astonishment to an audience, not showing off. Likewise, I believe software should be about building amazing things with an amazing team.

The most damaging attitudes in software today is the belief that “I have been writing software for X years, I know what I’m doing” all the while being blissfully unaware of a missing linchpin in the approach/solution while the project accelerates to full speed before hitting a brick wall. In my career I have seen more than one project crash and burn in cultures where this attitude is pervasive. This thinking is fatal in the complex domain.

There was a once a time where one could enjoy a reasonably successful career as the person with “all the answers.” Like the Nobel prizes and scientific discoveries of yesteryear; the low-hanging fruit have been all-but exhausted (and the rest will be soon be automated). It’s not about ego, it can’t be about ego. The emerging challenges of our industry require many minds and many perspectives. We may be in organizations with an antiquated culture focused on individual performance and it is up to us to demonstrate how much more a talented and diverse team is capable of.

I used to hoard my “magician foolers” now I use them as teachable moments to help other magicians think differently and do my part to elevate the art of magic as a whole. I’m even here breaking one of the most sacred rules of magic in an attempt to elevate software as a whole.

Conclusion

Our industry is shifting under our feet. The problems we must solve in the coming years will be challenging. Cognitive diversity in yourself and your teams will be your ace-in-the-hole. Remember that, just like in magic, it’s easy once you know the secret!