Kafka As Deterritorializing Stream Function

Sometimes he accompanied her on her errands in the city, where everything had to be carried out in the utmost hurry. Then she would almost run to the next subway station, Karl with her bag in his hand, the journey went by in a flash, as if the train were being carried away without any resistance, already they were getting off, clattering up the stairs instead of waiting for the elevator that was too slow for them, the large squares from which the streets flowed out in a starburst emerged and brought a tumult of streamed lines of traffic from all sides, but Karl and Therese hurried, tightly together, to the different offices, cleaners, warehouses and stores which weren’t easy to contact by telephone in order to make orders or complaints, generally trivial things.

– Kafka, Amerika, Hofman trans.

“No one is better than Kafka at differentiating the two axes of the assemblage and making them function together,” say Deleuze and Guattari, and though they are referring to the Czech writer, it applies quite well to the open source distributed message queue too, even though the quote was written thirty years before its invention. Kafka in either form effects both decoupling of components and a disintegration of content. Models, formats and codes are first broken down, then made available to be reassembled in other ways.

I admit that when I first heard of it, naming an information processing system after a writer famous for depictions of absurd, violent and impenetrable bureaucracy did strike me as bold. It evokes The Departure (Der Aufbruch) as router documentation, or An Imperial Message (Eine kaiserliche Botschaft) as a Service Level Agreement. Perhaps we can think of the system developers as finally addressing Franz’s eloquent, frustrated, bug reports. Jay Kreps, the namer of Apache Kafka and one of the co-creators, simply explains that he wanted his high performance message queue to be good at writing, so he named it after a favourite prolific writer. Fortunately complete publication and reading doesn’t also require the process dying of tuberculosis, followed by a decades-long legal case. Even if Kreps did have a deeper correspondence in mind, if I were forced to explain the name all the time, I might just smile and point at my Franz fridge magnet too.

The D/G quote is from the Postulates of Linguistics chapter, concerned with the way meaning is imposed on communicating agents and their intertwining systems.

On a first, horizontal, axis, an assemblage comprises two segments, one of content, one of expression. On the one hand it is a machinic assemblage of bodies, of actions and passions, an intermingling of bodies reacting to one another; on the other hand it is a collective assemblage of enunciation, of acts and statements, of incorporeal transformations attributed to bodies. Then on a vertical axis, the assemblage has both territorial sides, or reterritorialized sides, which stabilize it, and cutting edges of deterritorialization, which carry it away. No one is better than Kafka at differentiating the two axes of the assemblage and making them function together.

– Deleuze and Guattari, November 30, 1923: Postulates of Linguistics, A Thousand Plateaus

Deleuze and Guattari are secret pomo management consultants at heart, and as their dutiful intern I have accordingly expressed the great men’s vision as an Ansoff Matrix slide for distribution to valued stakeholders.

Compare this Jay Kreps slide from Strange Loop 2015, itself entirely representative of a million whiteboard sketches accompanying middleware everywhere:

For middleware, what D/G call the cutting edges of deterritorialization, we might call a payload codec. The data structure used within the producing process is disassembled, scrambled into a bucket of bytes, then carried away along a line of flight – in this case a Kafka topic. A Kafka topic is a transactional log, a persistent multi-reader queue where the removal policy is decoupled from reader delivery, and retention is instead controlled by time or storage space windows. (Blockchains are public transaction logs optimised for distributed consensus and no retention limit. Hence their inherent parliamentary slowness.) The consumer then uses its own codec to reterritorialize the data – making it intelligible according to its own data model, and within its own process boundary. Though nowadays the class signature of the messages may match (say both sides use the JVM and import the definition from the same library), at a bare minimum the relationship of those messages to other objects and functions within the process differs.

In Anti-Oedipus, D/G call reading a text “productive use of the literary machine”, and it’s along those lines that the quote continues:

No one is better than Kafka at differentiating the two axes of the assemblage and making them function together. On the one hand, the ship-machine, the hotel-machine, the circus-machine, the castle-machine, the court-machine, each with its own intermingled pieces, gears, processes, and bodies contained in one another or bursting out of containment (see the head bursting through the roof). On the other hand, the regime of signs or of enunciation: each regime with its incorporeal transformation, acts, death sentences and judgements, proceedings, “law”. […] On the second axis, what is compared or combined of the two aspects, what always inserts one into the other, are the sequenced or conjugated degrees of deterritorialization, and the operations of reterritorialization that stabilize the aggregate at a given moment. K., the K.-function, designates the line of flight or deterritorialization that carries away all of the assemblages but also undergoes all kinds of reterritorializations and redundancies – redundancies of childhood, village life, bureaucracy, etc.

 – Deleuze and Guattari, ATP ibid.

Cataloguing these correspondences between D/G’s description of Kafka and the software that bears his name is not intended to ignore that their contact is a kind of iconographic car accident. Kafka is a famous, compelling writer, and frequent cultural reference point, after all. The collision of names reveals structural similarities that are usually hidden.

In the Kreps talk above, titled in Deleuzian fashion “Apache Kafka and the Next 700 Stream Processing Systems”, he also introduces a stream processing API to unify the treatment of streams and tables. The team saw this as crucial to Kafka’s identity as a streaming platform rather than just a queue, and delayed calling Kafka 1.0 for years, until this component was ready. Nomadic messages escape through the smooth stream space, before capture and transformation in striated tablespace as rows in data warehouses.

First version of Tetris.

This unification of batches and streams echoes a similar call in computational theory by Eberbach. Turing computation is built around batches. Data is available in complete form at input on the Turing machine tape, then the program runs, and if it terminates, a complete output is available on the same tape. The theory of computability and complexity are built around this same encapsulated box of space and time. Much of what computers do in 2018 is actually continual computation – the reacting to events or processing streams of data that have no semantically tied termination point. That is, though the process may terminate, that isn’t particularly relevant to any analysis of computational complexity or performance we want to do. When editing a document on a computer, you care about the responsiveness keystroke to keystroke, not the entire time editing the document as if it were one giant text batch.  

By contrast, modern computing systems process infinite streams of dynamically generated input requests. They are expected to continue computing indefinitely without halting. Finally, their behavior is history-dependent, with the output determined both by the current input and the system’s computation history.

 – Eberbach, Goldin and Wegner – Turing’s Ideas and Models of Computation

Streams are a computational model of continuation, and therefore infinity. In their wide-ranging 2004 paper, Eberbach and friends go on to argue for models of Super-Turing computation. This includes alternative theoretical models such as the π-calculus and the $-calculus, new programming languages, and hardware architectures.

We conjecture that a single computer based on a parallel architecture is the wrong approach altogether. The emerging field of network computing, where many autonomous small processes (or sensors) form a self-configured network that acts as a single distributed computing entity, has the promise of delivering what earlier supercomputers could not.

 – Eberbach et al, ibid.

These systems are now coming into existence. Through co-ordination with distributed registries (Zookeeper here), and with the improved deployment and configuration baseline devops techs have brought in, spinning up new nodes or failing over existing nodes is autonomously self-configured. Complete autonomy isn’t there, but it seems a high and not particularly desirable bar for many systems. Distributed systems and streams predate Apache Kafka, and nor shall it be the last one. Yet it is a moment where separate solutions for managing streams are concretized, tables and failover in a single technical object, a toolbox for the streaming infrastructure of infinity.

References

CCRU – Ccru Writings 1997-2003
Cremin – Exploring Videogames with Deleuze and Guattari: Towards an affective theory of form
Deleuze and Guattari – Anti-Oedipus
Deleuze and Guattari – A Thousand Plateaus
Eberbach, Goldin and Wegner – Turing’s Ideas and Models of Computation
Kafka, Hofman trans. – Amerika (The Man Who Disappeared)
Kreps, Narkhede and Rao – Kafka – A Distributed Messaging System For Log Processing
Kreps – Apache Kafka and the Next 700 Stream Processing Systems (talk)
https://youtu.be/9RMOc0SwRro
Pajitnov – Tetris (game)
Stopford – The Data Dichotomy: Rethinking the Way We Treat Data and Services https://www.confluent.io/blog/data-dichotomy-rethinking-the-way-we-treat-data-and-services/
Thereska – Unifying Stream Processing and Interactive Queries in Apache Kafka https://www.confluent.io/blog/unifying-stream-processing-and-interactive-queries-in-apache-kafka/

Advertisements

Pogromon Go

It’s not fair to blame it on Pokémon, really. They weren’t the worst of it by far, though the limited edition Pikajew did echo the unfortunate spirit of the times.

The Pure Tribe had their own apps, monsters and backends. Variations. Innovations. Memes and games catch on and evolve. The whole time they were annotating and mapping. Highlighting targets in bold colours: signs, shrines, grandmothers, foxes. Sharing high scores on the day. Trading in parts. People and drones had got in plenty of practice beforehand. “Gotta catch ‘em all!”

Fads come and go.

 

Mountain City

The mountain city of Mount Huan is acknowledged to be one of the wonders of my home district, even though many a year goes by when it is not seen at all. A few years ago, the lawyer Sun Yun’nian was drinking with some mates on his verandah when suddenly they noticed a lone apartment block on the mountain opposite, rising up far into the deep blue sky. They looked at each other in sheer disbelief, as they knew of no condominium in that vicinity and had not had a chance to buy off the plan. Then a vast number of skyscrapers with blobject glass curves, abstract sculptures, antennas, and wall-gardens came into view, an unsolicited message from city hall arrived reminding them to vape responsibly, and they realized it was the Mountain City of Mount Huan.
Presently the expressways and light rail of the outer city became visible, and within them they could distinguish countless storied buildings, temples and residences. Suddenly a great wind arose, dust blew in, and the city could scarcely be seen any longer. By and by the wind subsided, the air cleared, and the city had vanished, save for one tall tower. Each storey of this tower had been pierced by sixty shuttered windows, all of which had been thrown open and let through the light from the sky on the other side. One could count the storeys of the tower by the rows of dots. The higher they were, the smaller they became, until by the eightieth story they resembled tiny stars, and above that they became an indistinguishable blur of twinkling lights disappearing into the heavens. It was just possible to make out tiny figures on the tower, some hurrying about, others leaning, or standing.
A little while longer, and the tower began to decrease in size, until its roof could be seen. One by one, pieces the size of an apartment or a gondola would detach, descend at an orderly pace down the side of the building, and roll away, disappearing from view. The tower continued shrinking still further to the height of a stadium, and then a car, then a bean, until finally it could not be seen at all.
It’s said you can fly to Mount Huan, if you need to, and that maps work fine in the city centre, but are glitchy to the point of useless in the suburbs. You have to get a local app: I forget its name. A determined walker can take in the whole layout of the city – its markets, its users, its parks. It is in no respect different from a city in our world. The annual “Ghost City Marathon” has become popular in recent years, and is well regarded, though there are problems recording accurate times.

References

Pu Songling, Strange Tales From A Chinese Studio, Minford trans. Story 89 from Minford / Story 251 in Zhang Youhe is a model for this text.
Pu Songling, Liaozhai Zhiyi, 聊齋誌異會校會注會評本, Zhang Youhe ed, 1978.

Metis and General Intellect

“The general intellect” can be interpreted as tacit craft knowledge embedded in individual cunning and social relations. This definition sets general intellect in contrast and opposition to formal information systems. Framing it this way may not be completely true to historic usage, but has some revealing consequences. It applies to either abstract information machines like traditional bureaucracies, or concretized ones, like specific computer programs. Craft and process arguments in software development are also a lens on this transformation of skills in cognitive capitalism, information valorization and their relationship with bureaucracy and flows of stupidity. To expand the general intellect is to accelerate.

The term the general intellect (sometimes general social knowledge) originates from Marx and was used by operaismo thinkers to grapple with the emergence of cognitive capitalism. Virno states:

Whereas money, the “universal equivalent” itself, incarnates in its independent existence the commensurability of products, jobs, and subjects, the general intellect instead stabilizes the analytic premises of every type of practice. Models of social knowledge do not equate the various activities of labor, but rather present themselves as the “immediate forces of production.”

They are not units of measure, but they constitute the immeasurability presupposed by heterogeneous operative possibilities.

They are not “species” existing outside of the “individuals” who belong to them, but axiomatic rules whose validity does not depend on what they represent. Measuring and representing nothing, these technico-scientific codes and paradigms manifest themselves as constructive principles.

From here I suggest general intellect as a form of tacit and social knowledge, of metis, defined in contrast to formal knowledge, numeric or systematic knowledge. The general intellect is cognition but it is not data. It is highly contextualized, by experience, locality and specific social relations. James C. Scott offers the knowledge embodied by a soccer team or a ship’s pilot as good examples of metis. He also uses it in contrast to systematic forms of knowledge used in high modernist, often Fordist projects of top-down political control, state or corporate.

At the economy scale, the general intellect also seems to have an intersection with the idea of “institutions” in economic development. Institutions established ongoing government policies but also more abstract things like property rights and the rule of law; they are not primarily buildings, but persistent social relations, not commoditized or readily transferable between nations. Acemoglu and colleagues found significant correlation between historic settler mortality and modern economic success, putting forward the type of colonial institutions as the causative link.

Though this is oppositional with computation as a form of knowledge, the two are complementary in production. Operators draw on the general intellect to make machines work and produce things. This is true of concrete machines, like a coffee maker, or abstract machines in the Simondon / Deleuze / Guattari sense. And machines, especially large abstract machines, make use of operator black boxes to be effective. Traditional bureaucracy before the advent of modern computers and networks is an example of an abstract information machine which uses human operators as black box components – for instance to persist information to longer term storage, by writing on paper.

If we look at the skilled cognition involved in designing complex machinery, such as computer programming, we find the general intellect in the sense of individual and organizational craft knowledge. The rise of agile software development techniques, emphasising teamwork and craft skills over Fordist or high modernist planning, is one example of this over the last fifteen years. Yet the act of programming depends very much on an individual mental model, as pointed out by Naur. Programming is not typing; the main productive activity in programming is building a coherent mental model, the actual executable code produced is a side-effect. “Programming in this sense must be the programmers’ building up knowledge of a certain kind, knowledge taken to be basically the programmers’ immediate possession” (Naur). The spread of algorithms and software throughout society would then suggest a shattering of the general intellect into millions of shards of specific intellects. The general intellect – the entirety of system relations – could decay even as systemic shards expand in sophistication.

None of this is deny a certain translatability from metis to formalized knowledge. It can all be boiled down to bits in the end. Translation for functional use is costly and lossy, though. The mechanics of deep learning parallel this metic transformation from formal data structures into occluded knowledge. To understand how a deep learning system internally navigates a problem space requires a separate systematic analysis alien to the learning mechanism of the algorithm. Deep learning is a localized preview of machinic metis.

A countertrend to the fragmentation of general intellect might be the success of open source, but the point of open source is precisely to make the executable details of machines more readily available through social processes. It is a common platform rather than a general intellect, where evolution of the platform happens through patches (explicit formal communication) rather than primarily through evolved social understanding, though those dynamics still exist. It is striking that open source communities are organized primarily around a specific machine or platform rather than user products. This is true from the GNU C compiler through to the Apache web server and git source control. They echo Simondon’s critique of objects made in capitalism not evolving but merely accumulating features. Simondon’s comments on technical culture also parallel general social knowledge:

Now that he is a technical being no longer, man is forced to find for himself a position in the technical ensemble that is something other than the position of individual.

The trend for computer programming to promote skilling up for designers but sometimes exporting deskilling elsewhere was noted by Mackenzie in 1984; it’s because capitalism is a valorizing process rather than a deskilling process per se. Likewise there are deskilling trends in the software industry around outsourcing highly specific work to remote or offshore teams, so long as it promotes valorization (increases shareholder value).

The frustrations of working in or with a bureaucracy are often those of being a black box cog in a larger abstract machine, either through alienation from the meaning of the work, or because the work actually causes an undesirable effect which conflicts with personal goals, or even the stated goal of the organization itself. That is a form of stupidity but relates to all bureaucracy. Deleuze and Guattari say in capitalism:

The apparatus of antiproduction is no longer a transcendent instance that opposes production, limits it, or checks it; on the contrary, it insinuates itself everywhere in the productive machine and becomes firmly wedded to it in order to regulate its productivity and realize surplus value which explains, for example, the difference between the despotic bureaucracy and the capitalist bureaucracy.

eg in a despotic state the army may come and confiscate food and labour from a subsistence farmer when fighting a war, but in capitalism this military anti-production is in the form of a military-industrial complex, production interleaved with anti-production. Yet those critiques could apply to Soviet socialism too; only capitalism manages to create demand and ensure lack in the midst of overabundance.

Deleuze takes stupidity as an inability to dissociate from presuppositions, sense rather than common sense. Contemplating flows of stupidity, I am reminded of the slogan of engineer Jesse Robbins for making useful things in corporate bureaucracies: “Don’t fight stupid, make more awesome”. This could also serve as an accelerationist slogan, and can be critiqued the same way. Are you pushing forward as an elite ignoring politically hard problems, or are you building a concrete alternative solution that will attract change? Are you wasting time trying to change a system accelerated beyond human comprehension, or are you using accelerated human components to build a new system?

References

Acemoglu et al – The Colonial Origins of Comparative Development
Beck et al – Agile Software Development Manifesto
Deleuze and Guattari – The Anti-Oedipus
Garton – Excavating the Origins of Accelerationism
Land – A Quick and Dirty Introduction to Accelerationism
Mackenzie – Marx and the Machine
Naur – Programming As Theory Building
Scott – Seeing Like A State
Simondon – On The Mode of Existence of Technical Objects
Virno – Virtuosity and Revolution: The Political Theory of Exodus, in Radical Thought In Italy
Williams / Srnicek – #accelerate Manifesto for an Accelerationist Politics