Reducing Abstraction in Code

Abstraction is something we are taught to value as programmers, and the process of finding common patterns across parts of a system is one programmers are usually good at. There is another equally important process of improving systems by collapsing redundancy and abstraction. Gilbert Simondon names this “concretization”.

Primitives

Concretization removes parts and makes machines more specific. A simple example is the abbreviation of clutter by replacing with clearer syntax. Say in Python

if available == True:
   reserve()

to

if available:
   reserve()

Or in JUnit:

assertTrue(false);

to

fail();

These are behaviour-preserving design improvements, or in other words, refactorings. They often turn up in novice programmer code or code written by people new to a language or toolset. Other primitive concretizing refactorings might be dead code removals, such as Remove Unused Parameter.

Another primitive concretization step is recognizing that a variable with a highly general type can be typed more precisely. A String, byte[] or a void* are highly general types, in that they can hold pretty much anything. Replacing with a more specific type usually relies on a precondition, either implicitly or explicitly.

int age = Integer.parseInt(ageStr);

In this case the potential throwing of NumberFormatException entails an implicit precondition. The concretizing step is the refactoring that introduces the typed variable.

Wait, but isn’t the problem with using Strings and primitive objects everywhere that they lack abstraction? Yes. They indicate that the code lacks an explicit model, or in other words, abstractions. They also indicate the code lacks concretizations – specifics from the problem domain that make it a well-focused machine. (Lacking both abstraction and concretization indicates ontological slime, a wonderful term from William Wimsatt, and perhaps the topic of another post.)

For a multi-line example of primitive concretization, consider this refactoring available when going from Java 1 to 5:

Iterator expenseIter = expenses.iterator();
while (expenseIter.hasNext()){
  Expense expense = (Expense)expenseIter.next();
  sum += expense.getExpenseValue();
}

to

for (Expense expense: expenses){
  sum += expense.getExpenseValue();
}

This mirrors the evolution of Java itself as a technical object and iteration as a technical concept. I’ve written about Simondon and the history of looping at more length elsewhere. Specialization and reduction are near-synonyms more frequently used in programming, but because of the clearer relationship to abstraction, and the connection to Simondon, I stick with concretization here, at the cost of a few more syllables. (Reification is a different concept, in my opinion.)

Interleaving Abstraction and Concretization

The adjunction of a supplementary structure is not a real progress for the technical object unless that structure is concretely incorporated into the ensemble of the dynamic system of its operation. – Simondon, Mode of Existence of Technical Objects, Mellamphy trans.

Design improvements often include both abstracting and concretizing steps. The feeling is of abstraction clearing space and concretization then making better use of it.

Michael Feathers’ use of characterization tests is an example of starting a design process with concretization.

    @Test
    public void removesTextBetweenAngleBracketPairs() {
        assertEquals("", Pattern.formatText(""));
    }

Characterization tests stabilize the function of the machine by pinning down very specific behaviors in the form of facts. This then allows a round of refactorings and rewrites. The immediate next step would often be abstracting refactorings such as Extract Method and Extract Class (naming a clump of things introduces an abstraction and an indirection).

Arlo Belshee’s Naming Is A Process also interleaves abstracting and concretizing steps.

Missing to Nonsense – Abstraction
Nonsense to Honest – Concretization
Honest to Honest and Complete – Concretization
Honest and Complete to Does the Right Thing – Abstraction
Does the Right Thing to Intent – Concretization
Intent to Domain Abstraction – Abstraction

A number of these steps, especially in the later half, themselves consist of interleaved abstracting and concretizing sub-steps. Eg in Honest and Complete:

1/ Use Introduce Parameter Object. Select just the one parameter you want to encapsulate. Name the class Foo and the parameter self. (Abstraction)
2/ Use Convert To Instance Method on the static. Select the parameter you just introduced. (Abstraction)
3/ Improve the class name (Foo) to at least the Honest level. (Concretization)
4/ Go to the caller of the method. Select the creation of the new type. Introduce parameter to push it up to the caller’s caller. (Abstraction)
5/ Convert any other uses of the parameter you are encapsulating to use the field off the new class. (Concretization)

Belshee’s process, using names as the signposts for improving code, is a wonderful combination of practical walkthrough and a theory of programming. It even seems to put living flesh on my skeletal wish for Name Oriented Software Development, though, eg, stronger tool and language support for consistent dictionaries are needed to realize the full vision.

Executable Theory

This kind of divergence of functional aims is a residue of abstract design in the technical object, and the progress of a technical object is definable in terms of the progressive reduction of this margin between functions in plurivalent structures. – Simondon, ibid

Every abstraction, even one as small as an extracted method, is also a theory. These little theories then need to be applied and refined to ensure a coherent system. What Simondon saw in the evolution of mechanical engines and other industrial era machines, we can observe at smaller scale and higher frequency when engineering in our more plastic computational material.

Simondon describes machines as becoming more concrete over time, finally reaching a highly focused state where each part cleanly supports the functions of others in an overall system. He also states that the introduction of a new theory is the invention of a new machine. So perhaps he would disagree that the process is cyclical.

We can, perhaps, reconcile this if we think of each software function or class as a small widget in a larger system. In this sense of the widget = machine = function, every new method is a new Simondonian machine. This also suggests that software rarely progresses to the refined machines he describes, but is more usually an assembly of semi-refined widgets. Which sounds about right.

Once you realise abstraction and concretization are complementary, anti-parallel processes, you start noticing it everywhere. I suspect casual design phrases like “nice abstraction” are actually misleading. Ohm’s Law is a nice abstraction; modern chips that rely on parasitic capacitance in a material context of silicon are well-built machines. In working software, a nice abstraction is also a nice concretization: a well-formed widget within a coherent machine.

All problems in computer science can be solved by another level of indirection, except of course for the problem of too many indirections. – David Wheeler

The Bureaucracy of Automatons

An introduction to the notes on Confucian Software.

Software and the Sage

Among the many dissimilarities between software and gentlemen of the classical Chinese Spring and Autumn Period, two in particular stand out. One existed in a pre-scientific feudal society on an agricultural technological and economic base, and the other presupposes the scientific method and a modern (or post-modern) industrial base. Secondly, the concept of virtue or potency (德) is central to The Analects, but software artifacts are, in our day and age, non-sentient. Morality requires some degree of self-awareness – of consciousness – and so software does not itself practice virtue any more than a spoon or a lawnmower.

The immediate relevance, for a developer, of the Analects, are the two other grand concerns of Confucius, which are existential fundaments of software. These are names (名), and the rites (礼).

Continue reading

Early Thoughts On Large Scale Scrum

Some people lose their sense of perspective. I’ve lost my sense of scale. – Will Self, Scale

We learn, in computing, that choice of data structure determines the algorithms one can run, their complexity, and the corresponding performance characteristics of an application. To talk of an algorithm without a data structure is to risk meaninglessness, for the two exist in a system, like the hub and spoke of a bicycle wheel. Likewise, Scrum and Scrum-like techniques are software processes, but they are just as fundamentally about changing team structure as about the various meetings and practices then enabled. Waterfall processes are pipelines of handoff between pinned threads organized by a task allocator-manager. Scrum is a common thread pool pulling work from a priority queue, then creating changes to a shared blackboard of state, which a product manager thread reacts to by adding or editing tasks on the queue. It’s model-view-controller: the code is the model, the production system is the view, and the product owner is the controller.

When organizations find success with agile, or Scrum, or something they call that which improves their delivery, they might reasonably want to expand on that success, simply because many organizations are larger than a team of seven people. Hence the demand for agile processes at scale. The demand, or criticism, of agile techniques ignoring large teams has been there for pretty much the whole history of the movement, but fifteen years in, we now have some road tested processes that try to address it directly. One of them is Large Scale Scrum (LeSS), and I had the good fortune this year to be introduced to it by one of its creators, Bas Vodde. That three day course and a bit of reading is the main source for these notes. There’s a lot of good material on the LeSS site.

Vodde and Larman say that Large Scale Scrum is Scrum, with some justification. There are various ways to combine small teams in delivering solutions for a client; you can federate, you can have cross cutting program managers and architects of some kind, create product governance committees, and so on. LeSS chooses to scale beyond one team working on a product by keeping the single product work queue pattern, and increasing the thread pool. The worker thread that pulls issues is now a team instead of a developer. That’s it. There is still only one product owner. Everyone in the organization works in the same sprint.

As in Scrum, everything else about the process is a support activity to facilitate humans maintaining that queue data structure. For scrum masters this is usually a combination of presiding over Scrum rituals and malediction of alternative work channels born of earlier software traditions – destruction of Old Ways, like long specification documents or individual users pushing their own priorities on particular developers. Other functions such as managers and sales exist not to direct development tasks but to secure resources from the larger organizational and commercial milieu.

This focus on the product at the heart of the software organization gives a rather existential question great importance: what exactly is the product? Why, exactly, is your organization here? What is the meaning of work life? Vodde and Larman don’t put the questions quite that way, but they do make product definition a key part of LeSS adoption, and strongly favour more expansive definitions based on actual user experience. This may seem obvious on the surface, but is the sort of obvious that is easily lost in the complex interactions of a large established organization. If your business provides sports news to its users, your real product is probably not a statistics vending web service with three “internal clients”. Or at least if you define your team’s product that way, you won’t be as effective at helping your users do something; it will at best be a local optimization. Part of the training was trying to define this product boundary for real systems we worked on, and real organizations. Often the best product definition cuts across senior managerial hierarchies, and this may limit what is the practical product definition can be in the near term. Crossing an organizational boundary usually requires coordination overhead that splits the thread pool of development teams, or introduces multiple product owners, both of which muddle the pattern. LeSS then uses the definition of done to ratchet the team closer to a product delivery mindset in every sprint. You are done only when the user experiences a change in the product.

LeSS mandates feature teams, which are fashionable at the moment, for mostly good reasons. This is again to support the work queue data structure and reinforce a whole-product view across the development teams. The argument is that programmers can learn, and the cost of context switches is worth it for product knowledge, reduced handoff queuing and integration risk, and flexibility of always having developers working on the most valuable features. Bas in particular emphasized learning as at the heart of software development. “When you switch components, velocity will go down, and that’s a good thing!” he said at multiple points during the three days. “It means you’re learning.”

Large Scale Scrum owes much of its success to Bas Vodde’s worldview of constructive organizational nihilism. “You have to give up the idea that things happen for a reason,” he says. “Or because someone decides them.” When I described it this way on morning three of the course Bas laughed, and tilted his head; then someone else asked what nihilism was. Study of nineteenth century philosophy isn’t a focus of a typical software engineering education, though Bas as it happens has squeezed some into his schedule around coding, becoming fluent in Chinese, raising a young family and keeping a day job of ripping internal corporate bureaucracies to pieces in the cause of building better software. He hadn’t seen Scrum and nihilism as linked.

Nihilism is the position that things have no inherent meaning, or, for Bas Vodde, that organizations have no inherent meaning. I call it constructive as I think he likes building useful things, and that’s what the most powerful parts of LeSS are about too. LeSS draws on Lean Engineering in its move away from an idea of best practice to one of continuous learning. In Taylorist scientific management, specialist managers discover a best practice and then encode it as a procedure to be followed by the organization. Nietzsche famously said that God was dead, that we had killed him, and set up Truth in his place, as another lie. Taylorism has the same tendency: it sets up procedures, and a hierarchical organizational tree, as an organizational singular Truth which is received by workers and mediated by a priesthood of managers. LeSS is pretty explicit about wanting to tear that down.

Hello? Is it Truth You're Looking For?

Bas himself, responding to an earlier version of this post, says his position is a little more moderate: that organizations usually have a purpose, but the things they actually do don’t achieve it, or often have any purpose at all, if you look for them.

The LeSS training explicitly frames itself against Ford and Taylor – sections contrasting Taylor with the Toyota way were part of the first day – but the argument is strictly pragmatic. Taylor’s techniques suit a world where large numbers of lightly schooled agricultural workers are coming to work in a factory, in industrial capitalism. It’s not well suited for knowledge workers building software under cognitive capitalism. You don’t need to discard God or Truth in general, both of which engineers can be fond of, to recognize that they aren’t a good model for software development, or other processes of design invention. An idea of a negotiated social truth, the artifacts and understanding of which mutates over time and depend on context, is rather more recognizable. That’s why the agile manifesto says it favours what it does; it is all about rejecting attempts to fixate on a flakily specified organizational high modernist Truth in favour of grappling with messy user reality. (Since the training I found Paulo Virno and the operaismo thinkers already made this link between nihilism and just-in-time production; eg quote and interview.)

Vodde and Larman tear down the idea of a single managerial Truth, then, but they also erect a replacement: the product.

Scrum is a strict process, but productive even in partial use. Ron Jeffries describes this himself when he describes many organizations as not really doing Scrum, and this is echoed by plenty of agile coaches, though they by their role tend to see this as a gap rather than positive variation. Let’s call this use of only some tools from the Scrum toolbox following a Scrum-like process. Scrum isn’t designed to work like this: it’s not a rational unified monstrosity where you are theoretically supposed to start by editing 90% of it out. You may have also added extra guff. Many teams running Scrum-likes may be doing it based on misconceptions, they may be Scrum-but, and they may well benefit from coaching. My guess, from the teams I’ve seen, is that most get some benefit from a short defined iteration (ie a sprint, though an over in Test match cricket might have been a better metaphor). They also benefit from a retrospective and a short daily standup meeting, even if the rest is a bit of a muddle. Crucially, Scrum-like teams also accommodate experiments and localization while keeping a common jargon.

Large Scale Scrum seems to have less room for this variation, because LeSS is Scrum, but scaled up, so each ritual and artifact bears more process weight. It has to do this because much variation will come from organizational inertia and dysfunction that will destroy the queue-worker thread structure at the heart of Scrum.

Scrum, with its focus on a single product owner and a single team, could conceal its essentially political nature pretty well. It was very directly about work that individuals in a small team do, so it could be introduced organically, at a team level. LeSS is about rebuilding your organization around a product, so it’s too big to hide. It is explicitly revolutionary and constitutional in its impact on office politics. For implementations smaller than nine teams, it even prescribes transitioning in a single day, starting a new organizational epoch with a dramatic software process Year Zero. This new Dictatorship of the Product has a politics which are collaborative and decentralizing, but shouldn’t be mistaken for something democratic. The rule of the product turns on the fulcrum of the product owner’s vision. Feature team programmer übermensch escape the shackles of component specialization, spreading their influence across technologies and languages. And of course the waterfall hierarchies of Taylorism were even less democratic and collaborative. LeSS might instead, when working well, have the character of an austere republicanism. There is a separation of powers, there is an executive product owner, and the duty to the Product takes the place of duty to the State (or The People).

Scrum, and Agile before it, build in the idea that the business and technology organizations are separate, and that one serves the other. In many industries that’s simply a recognition of organizational reality. The Scrum community aren’t deaf to it, and they tend to translate it into advice on hunting down, catching and breaking in a product owner. It’s pretty decent advice on the whole, if that’s what you want. In something like a startup, that may not make sense. Perhaps the lead developer has the best idea of the product, so role separation is meaningless. If, like Giles Bowkett, you want Scrum to die in a fire, I’d guess you’d want Large Scale Scrum to die in a fire too; maybe a larger one.

The most passionate argument in the three days of the course – and there were a few good ones – wasn’t about turning org charts upside down, or sidelining managers, or because I called Bas a nihilist, or what a product really is. It was about Jira. Bas recommends using a simple tool for the feature backlog, preferably a spreadsheet, and almost certainly not Jira. The argument, which has something to it, is that Jira is lightweight enough to be used, but heavyweight enough to destroy team ownership and collaborative planning. I have seen this too: it gets worse the more you type in meetings and the more workflow you add.

Say what you will about the tenets of National Socialism, Dude, at least it has traceability.

Say what you will about the tenets of National Socialism, Dude, at least it has traceability.

The Jira users were ready to circle the wagons and have it prised from their cold dead hands. I had forgotten how important the immediate tactility of sticky notes on boards is to a certain type of agilist. “Our ‘process’ is mainly github”, Bowkett says. The toolset for remote collaboration is much richer now, and the assumed developer toolset is a much stronger foundation. (Are you old enough to have had a serious argument about whether to use source control in your career? How about unit tests? With a senior developer?) Both Vodde and Bowkett are right to say tools shape processes. We default to their defaults. It made me realize we had been conflating the feature backlog and the sprint tasks for years, because that’s what’s easy in Jira. We also use Jira to coordinate across multiple cities, sometimes within teams – but I already knew that wasn’t Scrum.

One of the strengths of Scrum is its clarity of definition: you can talk clearly about whether or not you are following it, it doesn’t pretend to fit existing teams without change, and the same is true of LeSS. Vodde and Larman’s insight that making better software in large organizations requires refactoring the organization is spot on. LeSS is an organizational pattern for the product development interface. It’s not the only such pattern. MVC isn’t the only user interface pattern either, but its variants dominate the design space for a reason. If you don’t have the problems of product definition, flexibility and responsiveness in large organizations LeSS was designed to address, and you ship useful features at high quality every week already, don’t use it. For myself, and I suspect many software development organizations, those problems are of daily relevance. Software politics is the art of the possible. A software process coup d’état isn’t possible over here tonight; I’ll probably mangle and cajole a while instead.

Iteration

Abstraction is a concept familiar to programmers, and a term in common use. Abstraction is often discussed as a quality of code, but it can also describe a process of technology changing in a particular way over time. Gilbert Simondon, among others, offers the term concretization to describe a kind of anti-parallel process where technology components become more specific and effective over time, as designs evolve.

That introduction is pretty abstract. Examples can be seen in the changing design of loops.

Continue reading

Duckrabbit Bugfeature

Programmer folk art, origin unknown

Programmer folk art, origin unknown

516. If somebody showed me the figure and asked me ‘What is that?’, I could answer him only that way. –I couldn’t answer: ‘I take that to be a . . .’ or ‘Probably that is a . . .’ Any more than I take letters to be this or that when I’m reading a book.
— Wittgenstein, Philosophical Investigations

Duckrabbit. Source: Wikimedia