The Road Not Taken, No Not That One, You Know, The Other One

1961. Robert McNamara, the recently installed Secretary of Defense in President John Fitzgerald Kennedy’s cabinet, sits in his Pentagon office at a nine-foot-long mahogany desk, polished to a mirrorlike shine. He is wearing a dark suit, his thick brown hair slicked back on his head and parted in the middle, old fashioned wire-rim spectacles framing his mirthless eyes, his jaw clenched tight, a severe expression on his face, looking very much the Presbyterian elder he is.

The door opens, and Edward Lansdale, career spy and counterinsurgency specialist, enters. He has handsome, movie star good looks and a neat moustache. A look of determination is on his face, with a hint of a maverick smile, and he is carrying a box of what seem to be weapons and camping gear, caked with mud and even blood.

MCNAMARA is making notes on some graph paper.

MCNAMARA: [Stiffly] Colonel Lansdale, good to see you.

LANSDALE: Good morning, Mr Secretary.

MCNAMARA: Lansdale, we’re doing a systems analysis on our policy in South Vietnam. I understand you’ve just returned from a trip there and I need your report on the situation. You have nine and a half minutes for this briefing.

LANSDALE strides over to the desk and upends the content of the box. Out spill handmade pistols and knives, old French rifles, and bamboo punji sticks, all over the desk with a clatter, with a few falling on the floor.

LANSDALE: The enemy in Vietnam uses these weapons – and they were using them just a little bit ago before I got them. Many of them are barefoot or wear sandals. They wear black pajamas, usually, with tatters or holes in them. I don’t think you’d recognize any of them as soldiers, but they think of themselves that way. The people that are fighting there, on our side, are being supplied with our weapons and uniforms and shoes and all of the best that we have; and we’re training them. Yet, the enemy is licking our side. Always keep in mind about Vietnam, that the struggle goes far beyond the material things of life. It doesn’t take weapons and uniforms and lots of food to win. It takes something else, ideas and ideals, and these guys are using that something else. Let’s at least learn that lesson.

MCNAMARA stares at his soiled desk, blinking.

MCNAMARA: I see. [Stands up.] Colonel Lansdale, you can’t substitute emotions for reason.

LANSDALE: [Chuckles] It substituted just fine when we made those Marxists on Luzon think their villages were attacked by vampires.

MCNAMARA, somewhat fussily, fishes out his graph paper and pencil from under the weapons and other junk now on his desk. He walks to a clearer part of the desk and places it down.

MCNAMARA: Lansdale. As I said before, we are performing an extensive systems analysis on the situation in South and North Vietnam, and very much need to capture all the factors at play. I’m not sure if you are familiar with systems analysis. This is a process we used when I was an executive at Ford Motors. I have a list of seventy three factors our staff have so far found, including food supply, ammunition, rice production, oil imports, and so on. We’ll crunch the numbers, and once the analysis is complete, the output of the model will give us a clear path to victory.

LANSDALE glances at the list.

LANSDALE: Mr Secretary, your list is incomplete. You’ve left out the most important factor of all.

MCNAMARA: What is it?

LANSDALE: Well, it’s the human factor. You can put it down as the X factor.

MCNAMARA writes down “X Factor” on the graph paper.

MCNAMARA: What does it consist of?

LANSDALE: What the people out on the battlefield really feel; which side they want to see win and which side they’re for at the moment. That’s the only way you’re going to ever have this war decided.

MCNAMARA: Ah. Good point, but we’ve got that actually. Over here, see: “Volunteer signups”, “Ho Chi Minh “uncle-ization” ratio”, “Negative reviews on Saigon embassy facebook page”, “Mao Zedong cat pun frequency”, “GI sales of Conrad short stories”.

LANSDALE: I see. Well, what about the V factor?

MCNAMARA writes down “V Factor” on the graph paper.

MCNAMARA: What does it consist of?

LANSDALE: Vampires.

MCNAMARA: Ok. Ah, we’ll make sure to give that, uh, the appropriate weighting.

LANSDALE: Mr Secretary, there’s no mathematical formula for the human spirit.

MCNAMARA: That’s true. Some of those smart IBM boys we seconded from Cambridge found that a fifteen dimension vector including poetry writing and fish sauce consumption was a passable proxy in the Indochinese context, though.

LANSDALE: Oh, ok. Well that sounds just dandy. Don’t forget there are three brands of fish sauce popular in the south though – you should really track the lot.

MCNAMARA: Interesting. I’ll put those IBM boys onto it.


MCNAMARA: Good. This has been really useful, but I’ve got something else to do now. Oh, by the way, the intern you put on the Saigon embassy Twitter account is going great. The analytics are through the roof!

LANSDALE: [Snaps fingers, points back to MCNAMARA and smiles winningly.] I’ll pass it on.

MCNAMARA: Thanks Ed.

LANSDALE: Thanks Bob.

Rather loosely adapted from Chapter 22 of Max Boot’s recent biography of Lansdale, The Road Not Taken.


Boot – The Road Not Taken: Edward Lansdale and the America Tragedy In Vietnam
Brecher, Ames – War Nerd Podcast Episode 39
McNamara – In Retrospect
Wintermute, Boot – Max Boot interview on The Road Not Taken


Kafka As Deterritorializing Stream Function

Sometimes he accompanied her on her errands in the city, where everything had to be carried out in the utmost hurry. Then she would almost run to the next subway station, Karl with her bag in his hand, the journey went by in a flash, as if the train were being carried away without any resistance, already they were getting off, clattering up the stairs instead of waiting for the elevator that was too slow for them, the large squares from which the streets flowed out in a starburst emerged and brought a tumult of streamed lines of traffic from all sides, but Karl and Therese hurried, tightly together, to the different offices, cleaners, warehouses and stores which weren’t easy to contact by telephone in order to make orders or complaints, generally trivial things.

– Kafka, Amerika, Hofman trans.

“No one is better than Kafka at differentiating the two axes of the assemblage and making them function together,” say Deleuze and Guattari, and though they are referring to the Czech writer, it applies quite well to the open source distributed message queue too, even though the quote was written thirty years before its invention. Kafka in either form effects both decoupling of components and a disintegration of content. Models, formats and codes are first broken down, then made available to be reassembled in other ways.

I admit that when I first heard of it, naming an information processing system after a writer famous for depictions of absurd, violent and impenetrable bureaucracy did strike me as bold. It evokes The Departure (Der Aufbruch) as router documentation, or An Imperial Message (Eine kaiserliche Botschaft) as a Service Level Agreement. Perhaps we can think of the system developers as finally addressing Franz’s eloquent, frustrated, bug reports. Jay Kreps, the namer of Apache Kafka and one of the co-creators (with Narkhede and Rao), simply explains that he wanted his high performance message queue to be good at writing, so he named it after a favourite prolific writer. Fortunately complete publication and reading doesn’t also require the process dying of tuberculosis, followed by a decades-long legal case. Even if Kreps did have a deeper correspondence in mind, if I were forced to explain the name all the time, I might just smile and point at my Franz fridge magnet too.

The D/G quote is from the Postulates of Linguistics chapter, concerned with the way meaning is imposed on communicating agents and their intertwining systems.

On a first, horizontal, axis, an assemblage comprises two segments, one of content, one of expression. On the one hand it is a machinic assemblage of bodies, of actions and passions, an intermingling of bodies reacting to one another; on the other hand it is a collective assemblage of enunciation, of acts and statements, of incorporeal transformations attributed to bodies. Then on a vertical axis, the assemblage has both territorial sides, or reterritorialized sides, which stabilize it, and cutting edges of deterritorialization, which carry it away. No one is better than Kafka at differentiating the two axes of the assemblage and making them function together.

– Deleuze and Guattari, November 30, 1923: Postulates of Linguistics, A Thousand Plateaus

Deleuze and Guattari are secret pomo management consultants at heart, and as their dutiful intern I have accordingly expressed the great men’s vision as an Ansoff Matrix slide for distribution to valued stakeholders.

Compare this Jay Kreps slide from Strange Loop 2015, itself entirely representative of a million whiteboard sketches accompanying middleware everywhere:

For middleware, what D/G call the cutting edges of deterritorialization, we might call a payload codec. The data structure used within the producing process is disassembled, scrambled into a bucket of bytes, then carried away along a line of flight – in this case a Kafka topic. A Kafka topic is a transactional log, a persistent multi-reader queue where the removal policy is decoupled from reader delivery, and retention is instead controlled by time or storage space windows. (Blockchains are public transaction logs optimised for distributed consensus and no retention limit. Hence their inherent parliamentary slowness.) The consumer then uses its own codec to reterritorialize the data – making it intelligible according to its own data model, and within its own process boundary. Though nowadays the class signature of the messages may match (say both sides use the JVM and import the definition from the same library), at a bare minimum the relationship of those messages to other objects and functions within the process differs.

In Anti-Oedipus, D/G call reading a text “productive use of the literary machine”, and it’s along those lines that the quote continues:

No one is better than Kafka at differentiating the two axes of the assemblage and making them function together. On the one hand, the ship-machine, the hotel-machine, the circus-machine, the castle-machine, the court-machine, each with its own intermingled pieces, gears, processes, and bodies contained in one another or bursting out of containment (see the head bursting through the roof). On the other hand, the regime of signs or of enunciation: each regime with its incorporeal transformation, acts, death sentences and judgements, proceedings, “law”. […] On the second axis, what is compared or combined of the two aspects, what always inserts one into the other, are the sequenced or conjugated degrees of deterritorialization, and the operations of reterritorialization that stabilize the aggregate at a given moment. K., the K.-function, designates the line of flight or deterritorialization that carries away all of the assemblages but also undergoes all kinds of reterritorializations and redundancies – redundancies of childhood, village life, bureaucracy, etc.

 – Deleuze and Guattari, ATP ibid.

Cataloguing these correspondences between D/G’s description of Kafka and the software that bears his name is not intended to ignore that their contact is a kind of iconographic car accident. Kafka is a famous, compelling writer, and frequent cultural reference point, after all. The collision of names reveals structural similarities that are usually hidden.

In the Kreps talk above, titled in Deleuzian fashion “Apache Kafka and the Next 700 Stream Processing Systems”, he also introduces a stream processing API to unify the treatment of streams and tables. The team saw this as crucial to Kafka’s identity as a streaming platform rather than just a queue, and delayed calling Kafka 1.0 for years, until this component was ready. Nomadic messages escape through the smooth stream space, before capture and transformation in striated tablespace as rows in data warehouses.

First version of Tetris.

This unification of batches and streams echoes a similar call in computational theory by Eberbach. Turing computation is built around batches. Data is available in complete form at input on the Turing machine tape, then the program runs, and if it terminates, a complete output is available on the same tape. The theory of computability and complexity are built around this same encapsulated box of space and time. Much of what computers do in 2018 is actually continual computation – the reacting to events or processing streams of data that have no semantically tied termination point. That is, though the process may terminate, that isn’t particularly relevant to any analysis of computational complexity or performance we want to do. When editing a document on a computer, you care about the responsiveness keystroke to keystroke, not the entire time editing the document as if it were one giant text batch.  

By contrast, modern computing systems process infinite streams of dynamically generated input requests. They are expected to continue computing indefinitely without halting. Finally, their behavior is history-dependent, with the output determined both by the current input and the system’s computation history.

 – Eberbach, Goldin and Wegner – Turing’s Ideas and Models of Computation

Streams are a computational model of continuation, and therefore infinity. In their wide-ranging 2004 paper, Eberbach and friends go on to argue for models of Super-Turing computation. This includes alternative theoretical models such as the π-calculus and the $-calculus, new programming languages, and hardware architectures.

We conjecture that a single computer based on a parallel architecture is the wrong approach altogether. The emerging field of network computing, where many autonomous small processes (or sensors) form a self-configured network that acts as a single distributed computing entity, has the promise of delivering what earlier supercomputers could not.

 – Eberbach et al, ibid.

These systems are now coming into existence. Through co-ordination with distributed registries (Zookeeper here), and with the improved deployment and configuration baseline devops techs have brought in, spinning up new nodes or failing over existing nodes is autonomously self-configured. Complete autonomy isn’t there, but it seems a high and not particularly desirable bar for many systems. Distributed systems and streams predate Apache Kafka, nor shall it be the last one. Yet is marks a moment where separate solutions for managing streams, tables and failover are concretized in a single technical object, a toolbox for the streaming infrastructure of infinity.


CCRU – Ccru Writings 1997-2003
Cremin – Exploring Videogames with Deleuze and Guattari: Towards an affective theory of form
Deleuze and Guattari – Anti-Oedipus
Deleuze and Guattari – A Thousand Plateaus
Eberbach, Goldin and Wegner – Turing’s Ideas and Models of Computation
Kafka, Hofman trans. – Amerika (The Man Who Disappeared)
Kreps, Narkhede and Rao – Kafka – A Distributed Messaging System For Log Processing
Kreps – Apache Kafka and the Next 700 Stream Processing Systems (talk)
Narkhede – Apache Kafka Goes 1.0
Pajitnov – Tetris (game)
Stopford – The Data Dichotomy: Rethinking the Way We Treat Data and Services
Thereska – Unifying Stream Processing and Interactive Queries in Apache Kafka

Pogromon Go

It’s not fair to blame it on Pokémon, really. They weren’t the worst of it by far, though the limited edition Pikajew did echo the unfortunate spirit of the times.

The Pure Tribe had their own apps, monsters and backends. Variations. Innovations. Memes and games catch on and evolve. The whole time they were annotating and mapping. Highlighting targets in bold colours: signs, shrines, grandmothers, foxes. Sharing high scores on the day. Trading in parts. People and drones had got in plenty of practice beforehand. “Gotta catch ‘em all!”

Fads come and go.


Mountain City

The mountain city of Mount Huan is acknowledged to be one of the wonders of my home district, even though many a year goes by when it is not seen at all. A few years ago, the lawyer Sun Yun’nian was drinking with some mates on his verandah when suddenly they noticed a lone apartment block on the mountain opposite, rising up far into the deep blue sky. They looked at each other in sheer disbelief, as they knew of no condominium in that vicinity and had not had a chance to buy off the plan. Then a vast number of skyscrapers with blobject glass curves, abstract sculptures, antennas, and wall-gardens came into view, an unsolicited message from city hall arrived reminding them to vape responsibly, and they realized it was the Mountain City of Mount Huan.
Presently the expressways and light rail of the outer city became visible, and within them they could distinguish countless storied buildings, temples and residences. Suddenly a great wind arose, dust blew in, and the city could scarcely be seen any longer. By and by the wind subsided, the air cleared, and the city had vanished, save for one tall tower. Each storey of this tower had been pierced by sixty shuttered windows, all of which had been thrown open and let through the light from the sky on the other side. One could count the storeys of the tower by the rows of dots. The higher they were, the smaller they became, until by the eightieth story they resembled tiny stars, and above that they became an indistinguishable blur of twinkling lights disappearing into the heavens. It was just possible to make out tiny figures on the tower, some hurrying about, others leaning, or standing.
A little while longer, and the tower began to decrease in size, until its roof could be seen. One by one, pieces the size of an apartment or a gondola would detach, descend at an orderly pace down the side of the building, and roll away, disappearing from view. The tower continued shrinking still further to the height of a stadium, and then a car, then a bean, until finally it could not be seen at all.
It’s said you can fly to Mount Huan, if you need to, and that maps work fine in the city centre, but are glitchy to the point of useless in the suburbs. You have to get a local app: I forget its name. A determined walker can take in the whole layout of the city – its markets, its users, its parks. It is in no respect different from a city in our world. The annual “Ghost City Marathon” has become popular in recent years, and is well regarded, though there are problems recording accurate times.


Pu Songling, Strange Tales From A Chinese Studio, Minford trans. Story 89 from Minford / Story 251 in Zhang Youhe is a model for this text.
Pu Songling, Liaozhai Zhiyi, 聊齋誌異會校會注會評本, Zhang Youhe ed, 1978.