Navigating Remote Collaboration: A Diary of Project Recovery — Epoche C1
26 October: what actually happened The Aurora project's backend integration slipped because the QA team discovered an incompatibility in an API — the interface through which one module calls another — several weeks after the development team had marked that module complete. Everything else in this entry follows from that single fact, and my first account of it was wrong in a way worth setting down, because the wrong account produced two of the three remedies we adopted. What I wrote at the time was that our remote setup had created silos, and that this was not a technical oversight so much as a failure to communicate. Both halves of that are confused. The second sets up an opposition that does not exist: an interface defect between two modules owned by two groups is a communication failure, in a precise and well-documented sense, and treating the technical and organisational descriptions as alternatives is what stopped me looking for the mechanism. The first blames remoteness for something that the evidence says would have happened at fifty metres. This entry sets out what the research on distributed development actually establishes, what our delay should have been expected to cost, why the assumptions on both sides formed as they did, and which of our three fixes will still be running in April. Conway's law: the defect was in the shape of the organisation Melvin Conway (1968) published an observation that has held up for over half a century and that names our problem exactly: an organisation designing a system is constrained to produce a design whose structure mirrors the organisation's own communication structure. His reasoning is nearly tautological once stated. Any two components that must interface require the people building them to agree on the interface; if there is no communication path between those people, no agreement is reached; and the design that results is one in which the components that were talked about together are coupled, and the boundaries fall where the conversations did not happen. Applied here, the API incompatibility is not evidence that we failed to communicate about the design. It is the design, faithfully recording the communication structure that produced it. The boundary between our development and QA groups is the boundary at which the specification became ambiguous, because that is the boundary across which nobody was obliged to reach agreement. Conway's law does not say this is avoidable by trying harder. It says that if you want a clean interface at a particular seam, you must first create a communication path across that seam and give someone responsibility for it — which is, as it happens, exactly what our first remedy did, though we arrived at it by instinct rather than by argument. James Herbsleb and Rebecca Grinter (1999) followed a large distributed development split across sites in England and Germany and found the same pattern in detail. The teams had a written interface specification, which is what a Conway-aware manager would prescribe. It was not sufficient, and the reason is the interesting part: specifications are necessarily incomplete, and the gaps in them are ordinarily closed by unplanned conversation — someone wanders over and asks what a field is supposed to contain when it is empty. Across sites this did not happen, so the gaps were closed instead by each side making a plausible assumption. Their teams, like mine, discovered the mismatches at integration. Reading their account, the sentence in my earlier entry — that developers assumed QA had the latest specification while QA assumed developers had validated against staging — stops looking like a local failure of diligence and starts looking like the expected output of the arrangement. What the delay should have been expected to cost I described the delay as unexpected. On the published evidence it was the base rate. Herbsleb and Audris Mockus (2003) analysed several years of change records from a large software organisation, comparing individual work items that involved people at a single site with comparable items that spanned sites. The measure was elapsed time from the opening of a change request to its completion. Cross-site items took roughly two and a half times as long as same-site items of comparable size. Their analysis of why is the part that matters for management: the difference was largely accounted for by the number of people who ended up involved in a cross-site item, rather than by the items being intrinsically harder. Distribution does not make the work more difficult. It makes more people necessary, and each additional person is a delay. Two things follow that I did not appreciate in October. The first is that our schedule was built on same-site productivity assumptions for work that was structurally cross-site, which means the plan was wrong before anyone made a mistake. The second is that "we lost weeks" is not, by itself, evidence of a failure — a factor of two and a half is what the literature would have predicted, and the useful question is not why we were late but whether we were later than two and a half times. What co-location actually supplies, and why fifty metres is already remote My entry attributed the problem to the absence of the spontaneous informal check-ins that occur naturally in a shared office. This is the received explanation and it is half right in a way that matters, because the received version implies that returning to an office would fix it. Thomas Allen (1977) measured the relationship directly in research and engineering organisations, plotting the probability that two engineers communicate at least once a week against the physical distance between their workplaces. The resulting curve — since named after him — falls extremely steeply over the first few tens of metres and then flattens into a low plateau. By around fifty metres of separation the probability has dropped to near its asymptotic value, and moving the two people further apa