The lower-post-volume people behind the software in Debian. (List of feeds.)

[ This blog was crossposted on Software Freedom Conservancy's website. ]

I am honored to be a co-author and editor-in-chief of the most comprehensive, detailed, and complete guide on matters related to compliance of copyleft software licenses such as the GPL. This book, Copyleft and the GNU General Public License: A Comprehensive Tutorial and Guide (which we often call the Copyleft Guide for short) is 155 pages filled with useful material to help everyone understand copyleft licenses for software, how they work, and how to comply with them properly. It is the only document to fully incorporate esoteric material such as the FSF's famous GPLv3 rationale documents directly alongside practical advice, such as the pristine example, which is the only freely published compliance analysis of a real product on the market. The document explains in great detail how that product manufacturer made good choices to comply with the GPL. The reader learns by both real-world example as well as abstract explanation.

However, the most important fact about the Copyleft Guide is not its useful and engaging content. More importantly, the license of this book gives freedom to its readers in the same way the license of the copylefted software does. Specifically, we chose the Creative Commons Attribution Share-Alike 4.0 license (CC BY-SA) for this work. We believe that not just software, but any generally useful technical information that teaches people should be freely sharable and modifiable by the general public.

The reasons these freedoms are necessary seem so obvious that I'm surprised I need to state them. Companies who want to build internal training courses on copyleft compliance for their employees need to modify the materials for that purpose. They then need to be able to freely distribute them to employees and contractors for maximum effect. Furthermore, like all documents and software alike, there are always “bugs”, which (in the case of written prose) usually means there are sections that are fail to communicate to maximum effect. Those who find better ways to express the ideas need the ability to propose patches and write improvements. Perhaps most importantly, everyone who teaches should avoid NIH syndrome. Education and science work best when we borrow and share (with proper license-compliant attribution, of course!) the best material that others develop, and augment our works by incorporating them.

These reasons are akin to those that led Richard M. Stallman to write his seminal essay, Why Software Should Be Free. Indeed, if you reread that essay now — as I just did — you'll see that much of the damage and many of the same problems to the advancement of software that RMS documents in that essay also occur in the world of tutorial documentation about FLOSS licensing. As too often happens in the Open Source community, though, folks seek ways to proprietarize, for profit, any copyrighted work that doesn't already have a copyleft license attached. In the field of copyleft compliance education, we see the same behavior: organizations who wish to control the dialogue and profit from selling compliance education seek to proprietarize the meta-material of compliance education, rather than sharing freely like the software itself. This yields an ironic exploitation, since the copyleft license documented therein exists as a strategy to assure the freedom to share knowledge. These educators tell their audiences with a straight face: Sure, the software is free as in freedom, but if you want to learn how its license works, you have to license our proprietary materials! This behavior uses legal controls to curtail the sharing of knowledge, limits the advancement and improvement of those tutorials, and emboldens silos of know-how that only wealthy corporations have the resources to access and afford. The educational dystopia that these organizations create is precisely what I sought to prevent by advocating for software freedom for so long.

While Conservancy's primary job provides non-profit infrastructure for Free Software projects, we also do a bit of license compliance work as well. But we practice what we preach: we release all the educational materials that we produce as part of the Copyleft Guide project under CC BY-SA. Other Open Source organizations are currently hypocrites on this point; they tout the values of openness and sharing of knowledge through software, but they take their tutorial materials and lock them up under proprietary licenses. I hereby publicly call on such organizations (including but not limited to the Linux Foundation) to license materials such as those under CC BY-SA.

I did not make this public call for liberation of such materials without first trying friendly diplomacy first. Conservancy has been in talks with individuals and staff who produce these materials for some time. We urged them to join the Free Software community and share their materials under free licenses. We even offered volunteer time to help them improve those materials if they would simply license them freely. After two years of that effort, it's now abundantly clear that public pressure is the only force that might work0. Ultimately, like all proprietary businesses, the training divisions of Linux Foundation and other entities in the compliance industrial complex (such as Black Duck) realize they can make much more revenue by making materials proprietary and choosing legal restrictions that forbid their students from sharing and improving the materials after they complete the course. While the reality of this impasse regarding freely licensing these materials is probably an obvious outcome, multiple sources inside these organizations have also confirmed for me that liberation of the materials for the good of general public won't happen without a major paradigm shift — specifically because such educational freedom will reduce the revenue stream around those materials.

Of course, I can attest first-hand that freely liberating tutorial materials curtails revenue. Karen Sandler and I have regularly taught courses on copyleft licensing based on the freely available materials for a few years — most recently in January 2017 at LinuxConf Australia and at at OSCON in a few weeks. These conferences do kindly cover our travel expenses to attend and teach the tutorial, but compliance education is not a revenue stream for Conservancy. While, in an ideal world, we'd get revenue from education to fund our other important activities, we believe that there is value in doing this education as currently funded by our individual Supporters; these education efforts fit withour charitable mission to promote the public good. We furthermore don't believe that locking up the materials and refusing to share them with others fits a mission of software freedom, so we never considered such as a viable option. Finally, given the institutionally-backed FUD that we've continue to witness, we seek to draw specific attention to the fundamental difference in approach that Conservancy (as a charity) take toward this compliance education work. (My recent talk on compliance covered on LWN includes some points on that matter, if you'd like further reading.)


0One notable exception to these efforts was the success of my colleague, Karen Sandler (and others) in convincing the OpenChain project to choose CC-0 licensing. However, OpenChain is not officially part of the LF training curriculum to my knowledge, and if it is, it can of course be proprietarized therein, since CC-0 is not a copyleft license.

Posted Wed Apr 26 00:07:00 2017 Tags:

I support the open letter by Drupal developers protesting the attempted expulsion of Larry Garfield from the Drupal commmunity.

As a Drupal contributor who has never in any respect attempted to tie the project to his beliefs or lifestyle, Garfield deserves the right to be judged by his code alone. That is the hacker way; competence is all that matters, and no irrelevance like skin color or shape of genitals or political beliefs or odd lifestyle preference should be allowed to matter.

That I even need to say this in 2017 is something of a disgrace. The hacker culture already had judge-by-the-code-alone figured out forty years ago when I was a n00b; the only reason it needs to be said now is that there’s been a recent fashion for “social justice” witch hunting which, inevitably, has degenerated into the sort of utter fiasco the Drupal devs are now protesting.

Thomas Paine said it best: “He that would make his own liberty secure, must guard even his enemy from oppression; for if he violates this duty, he establishes a precedent that will reach to himself.”

It doesn’t matter how much you dislike Larry Garfield’s personal kinks. If you don’t defend him now, you may have nobody to defend you when some self-declared commissar of political and sexual correctness – or just a censorious project lead like Dries Buytaert – decides that you should be declared an unperson.

You shall judge by the code alone. That is the only social equilibrium that doesn’t degenerate into an ugly bitchfest with expulsions controlled by whatever happens to be politically on top this week. It was the right community norm forty years ago, and remains so today.

Posted Tue Apr 18 16:26:03 2017 Tags:

My last G+ post reported this:

Something out there kills about one oceangoing ship a week.

It is probably freakishly large waves – well outside the ranges predicted by simple modeling of fluid dynamics and used to set required force-tolerance levels in ship design. Turns out these can be produced by nonlinear interactions in which one crest in a wave train steals energy from its neighbors.

Much more in the video.

So go watch the video – this BBC documentary from 2002 on Rogue Waves. It’s worth your time, and you’ll learn some interesting physics.

As I’m watching, I’m thinking that the really interesting word they’re not using is “soliton”. And then, doing some followup, I learn two things: the solutions to the nonlinear Schrödinger equation that describe rogue waves are labeled “Peregrine solitons”, despite not actually having the non-dissipative property of your classical soliton; and it is now believed that the S.S. Edmund Fitzgerald was probably wrecked by a rogue wave back in ’75.

In a weird way this made it kind of personal for me. I used to joke, back when people knew who he was, that Gordon Lightfoot and I have exactly the same four-note singing range. It is a fact that anything he wrote I can cover effectively; I’ve sung and played The Wreck of the Edmund Fitzgerald many times.

So, I’m texting my friend Phil Salkie (he who taught me to solder, and my reference for the Tinker archetype of hacker) about this, and we started filking. And here’s what eventually came out: Wreck of the Edmund Fitzgerald, the science! version:

The lads in the crew saw that soliton come through
It stove in the hatches and coamings
Her hull broached and tore, she was spillin’ out ore
That rogue put an end to her roamings.

Does anyone know where the Gaussian goes
When the sea heights go all superlinear?
A Schrödinger wave for a watery grave
It’ll drown both the saint and the sinner.

That is all.

Posted Mon Apr 17 00:34:49 2017 Tags:
At the American Chemical Society meetings drug companies disclose recent new drugs to the world. Normally, the chemical structures are already out in the open, often as part of patents. But because these patents commonly discuss many compounds, the disclosures are a big thing.

Now, these disclosure meetings are weird. You will not get InChIKeys (see doi:10.1186/s13321-015-0068-4) or something similar. No, people sit down with paper, manually redraw the structure. Like Carmen Drahl has done in the past. And Bethany Halford has taken over that role at some point. Great work from both! The Chemical & Engineering News has aggregated the tweets into this overview.

Of course, a drug structure disclosure is not complete if it does not lead to deposition in databases. The first thing is to convert the drawings into something machine readable. And thanks to the great work from John May on the Chemistry Development Kit and the OpenSMILES team, I'm happy with this being SMILES. So, we (Chris Southan and me) started a Google Spreadsheet with CCZero data:



I drew the structures in Bioclipse 2.6.2 (which has CDK 1.5.13) and copy-pasted the SMILES and InChIKey into the spreadsheet. Of course, it is essential to get the stereochemistry right. The stereochemistry of the compounds was discussed on Twitter, and we think we got it right. But we cannot be 100% sure. For that, it would have been hugely helpful if the disclosures included the InChIKeys!

As I wrote before, I see Wikidata as a central resource in a web of linked chemical data. So, using the same code I used previously to add disclosures to Wikidata, I created Wikidata items for these compounds, except for one that was already in the database (see the right image). The code also fetches PubChem compound IDs, which are also listed in this spreadsheet.

The Wikidata IDs link to the SQID interface, giving a friendly GUI, one that I actually brought up before too. That said, until people add more information, it may be a bit sparsely populated:


But others are working on this series of disclosures too, and keep an eye on this blog post, as others may follow up with further information!
Posted Fri Apr 14 13:39:00 2017 Tags:

In ancient lore, a molly-guard was a shield to prevent tripping of some Big Red Switch by clumsy or ignorant hands. Originally used of the plexiglass covers improvised for the BRS on an IBM 4341 after a programmer’s toddler daughter (named Molly) frobbed it twice in one day

The Great Beast of Malvern, the computer designed on this blog for performing repository surgery, sits to the left of my desk. This is Zola the cat sitting on it, as he sometimes does to hang out near one of his humans.

What you cannot quite see in that picture is the Power Switch of the Beast, located near the right front corner of the case top – alas, where an errant cat foot can land on it. Dilemma! I do not want to shoo away the Zola, for he is a wonderfully agreeable cat. On the other hand, it is deucedly inconvenient to have one’s machine randomly power-cycled while hacking.

Fortunately, I am a tool-using sophont and there is an elegant solution to this problem.

The solution is an ordinary refrigerator magnet. Being a magnet, it adheres to the ferrous-metal case top just firmly enough not to be easily pushed aside by a cat foot, but is readily moved by human fingers. Here it is in the ‘unsafe’ position:

And in the ‘safe’ position:

Thus, your word of the day: “zola-guard”. Like a molly-guard, but for cats.

Posted Thu Apr 13 14:32:22 2017 Tags:

This is a public service announcement: E-Shielder Security, describing itself as “leading importers and suppliers of high end electronic technology solution systems” is a gang of idiotic scum.

Yesterday they posted a Hacktivists on the rampage in 2017, which largely reproduced my Hacker Archetypes post.

They did so in obvious ignorance of who the hackers I was referring to actually are, going off on a tear about “hacktivists”. That term is, in general, a flare-lit clue that the person using it is either an idiot or a vandal trying to cloak destructive behavior in respectability – real hackers are proud of what they do, take responsibility for it, and don’t wear masks (with a limited exception for those under direct threat from totalitarian governments). In this case it was clearly idiocy.

Mere idiocy turned into something nastier. I left a comment on the post pointing out their error, something I had clear standing to do as the author of the article they were quoting.

The comment was suppressed. That was scummy behavior; thus “idiotic scum”.

Don’t do business with these clowns. Warn your friends. Propagate this widely, the clowns deserve some serious reputation damage.

Addendum: Title amended because the article may have originated at CyberSec Buzz, another ‘security’ blog run by drivelheads who obviously have no fscking idea what they’re talking about. It has been taken down where I originally found it.

Posted Wed Apr 12 12:00:56 2017 Tags:

There’s a book about martial arts called On the Warrior’s Path that tries to understand the differing psychologies of martial artists through the lens of half a dozen archetypes – Seeker, Ronin, Tribal Warrior, and others.

I have not yet read the book, but my friend and regular A&D commenter Susan Sons reports having found it very effective for motivating young and newbie martial artists. “It gave them their first glimpse of what they were trying to become,” she reports, “They both knuckled down not just in the obvious physical parts of training, but in the mental aspects, far more than they had before and far more than their age/experience peers.”

So, Susan had the idea that it might be a good idea to develop a parallel gallery of hacker archetypes to help motivate newbies. We brainstormed this on IRC for a while. One thing that had been blocking Susan is that, by her own report, she sucks at naming things. I, on the other hand, am pretty good at that; I was able to come up with names that helped the archetypes develop more definition.

We don’t think this is a complete set, and some of the names might change. But it’s enough of a start for some public brainstorming.

Also note: no hacker is only one of these, but in talking about a number of mutual friends we found it was always pretty easy to agree on both the friend’s dominant archetype and the secondary one that they display most after it. I think this is an indication that we are, even if imperfectly, zeroing in on real traits.

Here they are. Descriptions mostly Susan, names mostly me.

Algorithmicists: Very good at algorithms and sustained, intricate coding. Have mathematical intuition, and are one of the two types (with Architect) that have the highest tolerance for complexity. They like the idea of correctness proofs and think naturally in terms of invariants. They gravitate to compiler-writing and crypto. Often solitary with poor social skills; have a tendency to fail by excessive cleverness. Never let them manage anyone!

Tinkerers: Hackers who are drawn to crossovers with the physical world – will design hardware as cheerfully as software. One of the two types (with Prankster) most likely to be lockpickers and locksmiths. Know practical electronics (including analog and RF), adept at reverse-engineering. When you can get them to pull their heads out of the details (which they may resist pretty hard) they make terrific whole-systems engineers.

Architects: The guys who are fascinated by, and very good at, blocking out architecture in complex systems. Kings of the productive refactor. Have an acute feel for design patterns and can see around corners in design space. Strong drive to simplify and partition; “It’s not done until it’s elegant.” The Architect failure mode is to lose sight of the ground. Architects don’t necessarily have communications skills; if they do, they can make worthy team leads.

Sharpshooters: Tenacious detail-obsessives who are most comfortable with a bottom-up view of code and like rifle-shooting bugs more than almost anything else. In past times they would have been happy writing assembler. Polar opposite of the Architect, very productive when paired with one (and vice-versa). Not a good bet for managing anything.

JOATs: The great strengths of the jack-of-all-trades are adaptability, fast uptake of new ideas, and mental flexibility. The JOAT doesn’t do any one thing better than the other types, but can do a bit of everything – including people and social engineering. The JOAT failure mode is to try to do everything themselves. A JOAT is more likely than other types to make an excellent team lead, as long as he or she remains aware enough to delegate deep technical decisions to others.

Pranksters: Their natural bent is adversarial – they’re great at thinking up ways to disrupt and subvert systems (or just put them to unexpected and hilarious uses). They gravitate to infosec and test engineering. The really good ones can social-engineer people more ruthlessly and effectively than any of the other types.

Castellans: Supreme control freaks who get their power from focusing on what they’re responsible for and knowing it inside out. Castellans memorize manuals; they love language-lawyering, process automation, and vacuuming up domain-specific knowledge about whatever they’re working on. Old-school sysadmins are often castellans: “That will never happen on my system” is their promise (and of course Pranksters love to prove them wrong).

Translators: The type that bridges between human and machine: tends to excel at UI/UX development, documentation, policy and supply-chain stuff, requirements analysis, user training, and so on. Highly social, less hard-core technical than others, but in a way that helps them help other hackers understand how non-hackers see and interact with technology. Some of them make good project managers, but like JOATs they need to understand their technical limitations and mostly leave the hard decisions to types that naturally swim in deeper technical waters. Of all the types, Translators are the least likely to self-identify as hackers even if they are intimate with the culture and working within it.

What archetypes, if any, are we missing? Are there places where the boundaries need adjusting?

(Oh, and me? Mostly Architect with a side of Algorithmicist and a touch of JOAT.)

Final note: This post is being edited as we collect more feedback. “Translators” weren’t in the first version at all.

Posted Mon Apr 3 18:49:36 2017 Tags:

A ‘pidgin’ is a language formed by contact between speakers of different languages. A ‘creole’ is what happens when a pidgin becomes a birth language for children raised where a pidgin is spoken. Pidgins are simple languages, stripped to the running gears, Often creoles re-complexify in later generations, retaining grammar mostly from one parent language and vocabulary mostly from the other.

My interest in the historical linguistics of pidgins and creoles began a very long time ago when I noticed that pidgins, wherever they arise, are usually morphologically a lot like English – analytic (positional) grammar with few inflections, SVO order oftener than can be accounted for by the fact that English is often one of the parent languages. Why should this be?

Nicholas Ostler’s excellent Empires of the Word deepened the question by proposing that analytic SVO grammar is the common factor in languages like English, Chinese and Malay that have been very successful at spreading from their original homelands. In his account, that is because this class of language has the lowest complexity barrier to acquisition for adult speakers.

That would explain pidgins all right – they look like they do because they’re invented by adults as the simplest possible way to establish communication. And English, with similar traits, is a non-pidgin that has spread like crazy because it combines the prestige of the Anglosphere with being exceptionally easy for native speakers of other languages to learn.

Er, but why is English like that in the first place?

A few years back I tripped over – and was instantly fascinated by – the notion that English is best understood historically as an old creole. Most educated people know the ha-ha-only-serious line about English being the result of attempts by Norman men-at-arms to pick up Saxon barmaids; this refers to pidgin formation between late Anglo-Saxon and old French after 1066, with later Middle English viewed as the succeeding creole.

But there may have been an even more important creolization 150 years previously when Vikings conquered the region of central England known as the Danelaw and their West Norse collided with midlands Anglo-Saxon of the time. Midlands Anglo-Saxon could be considered to have been replaced by an Anglo-Norse creole with a simplified version of old Anglo-Saxon grammar and a lot of Norse vocabulary (including items as basic as most of the pronouns that English has used since).

So, double creolization with the grammar getting simplified at each stage. But wait! There’s more! We don’t know for sure because the Anglo-Saxons who invaded Britain after the Roman collapse in 410 weren’t literate, but it is quite possible that there was an even earlier creolization following on their domination of the Celtic natives.

So, running this forward in time, the oldest Anglo-Saxon collides with Celtic languages; the resulting creole becomes middle Anglo-Saxon. Which then collides with Old Norse; the creole from this becomes late Anglo-Saxon. That is what collides with Old French and, badda boom badda bing, Middle English.

At each stage the language grammar gets progressively more stripped to its running gears – more analytic, more SVO, more pidgin-like – only to partly re-complexify (as creoles do) and pick up loads of vocabulary items along the way.

Unfortunately we have only a badly foreshortened view of all this because the Anglo-Saxon manuscripts are mostly from the late period. The only creolization process we can more or less track is the latest of the three; the previous two have to be inferred from, for example, the fact that late Anglo-Saxon had already adopted the pronouns of West Norse.

The other problem with this theory is actually a problem with linguists. Ever since philologists started reconstructing the history of the Indo-European language group in the 1800s, linguists have loved nice tidy evolutionary tree structures. Crosslinks that mess up that picture, like sprachbunds or creole formations, are not loved. There is still – at least it seems to me, as an amateur but careful observer – a tendency in academia to want to banish pidgin and creole formation to the periphery, denying that these can be central features of the history of “great” languages.

Part of this comes from historical connotation; pidgin and creole formation were first noticed in the record of the Age of Exploration, when Europeans were having a high old time rambling all over the globe trading with, warring on, enslaving, and having sex with various kinds of dusky-skinned natives. Pidgin/creole formation has still got an associative whiff about it of dandies keeping octoroon mistresses that’s a bit unseemly.

OK, so what if we try looking past that? What happens if, instead of admitting there’s been language hybridization and applying the label “creole” only when forced and the contact event was recent, we think of creolization as a language-formation process that is historically normal under certain circumstances, and go looking for examples and recurring patterns?

Now, much of the rest of this is mostly me being speculative. And I’m not a trained historical linguist with a PhD union card, so its unlikely any of the pros will care much what I think. But I see some fascinating vistas opening up.

For one thing, there are recurring patterns. I can see four: trade creoles, conquest creoles, camp creoles, and court creoles. The lines among these are not perfectly sharp, and sometimes a creole will be repurposed after it forms, but they illustrate four basic modes of formation.

A trade creole is a language evolved from a trade pidgin. There are numerous examples in the South Pacific, of which Tok Pisin in Papua New Guinea is among the best known; these are recently formed within the last 200 years and linguists do apply the label “creole” to them.

On the other hand, Swahili seems to be an old trade creole, resulting from contact between coastal Bantu languages in West Africa and Arab traders and beginning to form not long after 600CE. Linguists normally don’t label it as a creole, but it has kept the pattern of simplified and regularized grammar relative to its root languages. One marker of this is that unlike most of the area’s Bantu languages, Swahili has no tonal system – a feature contact pidgins invariably drop.

The Indonesian archipelago is rife with dozens of trade creoles formed by contact among different Austronesian languages and others (including Chinese and Sanskrit). National Indonesian and Malay are themselves well understood to be old trade creoles, though in the normal way of such things the “creole” label is seldom applied.

A conquest creole is a case like Middle English where an incoming military elite forms a contact pidgin with the natives that displaces the native “pure” language. I have previously noted that this seems to fit what happened to middle Anglo-Saxon in the Danelaw beginning 150 years earlier. If there was a still earlier creolization event mixing Anglo-Saxon and Celtic, Middle Anglo-Sazion too would have been a conquest creole.

A thing to look for in conquest creoles is the formation of a creole continuum in which the language of the invaders is the acrolect, a relatively less modified version of the indigenous language is the basilect, and individuals routinely code-switch from lower to higher forms and vice-versa without recognizing that this involves not just a change in vocabulary but substantial morphological shifts as well. I don’t think you get this in trade creoles, where social relationships along the contact front are more horizontal.

Another example, helpfully demonstrating multiple creolizations in a language’s back story in case we are tempted to think of English as unique, is Maltese – grammar from Arabic conquerors, vocabulary from Romance-speaking subjects. Recently (last 150 years) “old” Maltese has been largely replaced by an Anglo-Maltese creole.

Next, the camp creole. This is a creole formed from a pidgin invented as a military command language in a multilingual empire.

The best known of these is Hindi/Urdu, the latter name for which literally means “camp language” (it’s related to Mongol “ordu” and the derived English word “horde”). It originated as a military pidgin formed by contact among a largish group of related North Indian languages around the 7th century CE (so, about as old as Swahili).

I tripped over a minority theory of the origin of modern German a while back. The usual story about this is that it’s the language of Luther’s Bible, but apparently some experts think it is more properly viewed as having originated more recently as a military pidgin in Frederick the Great’s Prussia. At the time there were different so-called “dialects” of German that were quite mutually unintelligible, so this theory makes functional sense.

My source even proposed that this is why modern German tends to verb-final order in sentences – as a way of making command verbs more prominent.

The label “creole” is, as you are probably expecting by now, not generally applied to either Hindi or Modern German. In the case of Hindi, though, it fits the generally accepted interpretation of the language’s history. And I’m betting the military-pidgin account of modern German hasn’t gotten quite the attention it deserves.

Our fourth variety is the court creole. This is like a camp creole, but instead of arising from a military pidgin it develops among the ruling elite in the capital of a multi-lingual empire. Because once your polity gets past a certain size you need to recruit administrators, servants, concubines and whatnot from places where the language isn’t yours, and they then have the same contact problem as a military camp.

There are a couple of solutions to this problem that don’t involve spinning up a new language. You might be able to actually impose your language on the natives; the Romans were pretty effective at this. You might be able to adapt a camp creole that your armies already speak; this is how Hindustani became a court language with a literary tradition. But if all else fails, your capital is going to grow its own creole because it has to.

The type example I had in mind for “court creole” when I began writing was Mandarin Chinese, but there was the problem that it’s tonal, a feature normally lost during pidginization. On the other hand, Wikipedia says straight up that Mandarin arose during the Ming dynasty “as a practical measure, to circumvent the mutual unintelligibility of the varieties of Chinese”, which certainly sounds like my court-creole notion in action.

The standard account describes Mandarin as a “koine”, which is elsewhere defined as a contact language arising among mutually intelligible varieties – and thus without undergoing drastic complexity reduction through a pidginization phase. That would remove the mystery about the retention of tonality.

But something is off here. We actually know that mutual unintelligibility was a problem at the time; one Emperor put a complaint on record that he couldn’t understand the speech of certain provincial officials, and founded language academies to attack the problem. This doesn’t really sound like the conventional koine-formation story.

We might be running up against an edge case for the “koine” and “creole” categories where it’s difficult to know which applies. Or linguists might be exhibiting their usual flinch about using the label “creole” outside of a dandies-and-octoroons situation. It’s difficult to know and I am certainly too ignorant of historical Chinese linguistics to justify a strong opinion.

What I think we can propose is that the specialists ought to take a fresh look at the period sources and see if they can detect any traces of what looks like pidginization (with its characteristic loss of grammatical complexity) in the period when Mandarin was forming.

What we can say without dispute is that like English and Malay (but, to be fair, unlike Swahili) modern Mandarin has retained a lot of pidgin-like traits commonly found in creoles – SVO analytic grammar, simple phonology, a spoken form easily acquired by adults from other language groups (even as its written form is infamously difficult).

Which brings us to the end of my speculation. I wish I had an unambiguous example of a court creole to lay down, but just this brief survey should make clear that creolization events have been both more common and far more important than you’d think from the linguistics textbooks.

Even the oldest attested human language may have been a creole. There are structural and lexical indications in Sumerian that it may have fused from a couple of rather dissimilar languages spoken still earlier in the Fertile Crescent!

So, hey, academic linguists, stop being such prudes about language hybridization, eh? It’s limiting your vision.

Posted Sun Apr 2 14:56:20 2017 Tags:

This is a higher-level explanation of the inputfd protocol RFC I sent to the list on March 31. Note that this is a first draft, this protocol may never see the light of the day or may be significantly altered before it lands.

First, what is it? inputfd is a protocol for a Wayland compositor to pass a file descriptor (fd) for an input device directly to the client. The client can then read events off this fd and process them, without any additional buffering in between. In the ideal case, the compositor doesn't care (or even notice) that events are flowing between the kernel and the client. Because the compositor sets up the fd, the client does not need any special privileges.

Why is this needed? There are a few input devices that will not be handled by libinput. libinput is the stack for those devices that have a direct interaction with the desktop - mice, keyboards, touchpads, ... But plenty of devices do not want or require desktop interactions. Joysticks/gamepads are the prime example here, but 3D mice or VR input devices also come to mind. These devices don't control the cursor on the desktop, so the compositor doesn't care about the device's events. Note that the current draft only caters for joysticks, 3D mice etc. are TBD.

Why not handle these devices in libinput? Joysticks in particular are so varied that a library like libinput would have to increase the API surface area massively to cater for every possibility. And in the end it is likely that libinput would merely buffer events and pass them on almost as-is anyway. From a maintenance POV - I don't want to deal with that. And from a technical POV - there's little point to have libinput in between anyway. Furthermore, it's already the case that gaming devices are opened directly by the application rather than going through X. So just connecting the clients with the device directly has a advantages.

What does the compositor do? The compositor has two jobs: device filtering and focus management. In the current draft, a client can say "give me all gaming devices" but it's the compositor that decides which device is classified as such (see below for details). Depending on seat assignment or other policies, a client may only see a fraction of the devices currently connected.

Focus management is relatively trivial: the compositor decides that a client has focus now (e.g. by clicking into the window) and hands that client an fd. On focus out (e.g. alt-tab) the fd is revoked and the client cannot read any more events. Rinse, wash, repeat as the focus changes. Note that the protocol does not define how the focus is decided, that task is up to the compositor. Having focus management in the compositor gives us a simple benefit: the client can't hog the device or read events when it's out of focus. This makes it easy to have multiple clients running that need access to the same device without messing up state in a backgrounded client. The security benefit is minimal, few people enter their password with a joystick. But it's still a warm fuzzy feeling to know that a client cannot read events when it's not supposed to.

What devices are applicable devices? This is one of the TBD of the protocol. The compositor needs to know whether a device is a gaming device or not, but the default udev tag ID_INPUT_JOYSTICK is too crude and provides false positives (e.g. some Wacom tablets have that tag set). The conversion currently goes towards having a database that can define this better, but this point is far from decided on.

There are a couple of other points that are still being discussed. If you have knowledge in game (framework) development, do join the discussion, we need input. Personally I am far from a game developer, so I cannot fathom all the weird corner cases we may have to deal with here. Flying blind while designing a protocol is fun, but not so much when you then have to maintain it for the next 10 years. So some navigational input would be helpful.

And just to make sure you join the right discussion, I've disabled comments here :)

Posted Sat Apr 1 06:12:00 2017 Tags:

I’ve been ill with influenza and bronchitis for the last week. Maybe this needs to happen more often, because I had a small but fundamental insight into network scaling theory a few minutes ago.

I’m posting it here because I think my blog regulars cast a wide enough net to tell me if I’ve merely rediscovered a thing in the existing literature or, in fact, nobody quite got here before.

Back in the naive days of the dot-com boom people used to talk about Metcalfe’s Law: the value of a node of n networks scales as O(n**2). This heuristic drove a lot of the early excitement about Internet-related stocks.

But then the boom busted, and in 2006 along come Odlyzko and Tilly to explain that. What they said is: empirically, networks actually seem to top out at O(n log n) value, and this is superlinear but way lower than O(n**2), and thus dot-com boom fall down go bust.

You can read the Odlyzko/Tilly paper here. It’s good; it’s lucidly written and deserves its status as a seminal classic. The explanation of O(n log n) that the authors give is that in a world where not all connections have equal value, people build only the connections with the best cost-benefit ratio, and due to an effect called the “gravity law” the value of traffic between any two nodes falls off superlinearly with distance. This produces a substantial disincentive to build long-distance links, leading to a network of clusters of clusters with O(n log n) link density and value scaling.

After Odzylko/Tilly, complexity theorists looked at real-world networks and found that they frequently evolve towards a topology that is self-scaling or fractal – clusters of clusters at any scale you examine. Circulatory systems in the body, neural networks in the brain, road and rail networks in human cities, the Internet itself – over and over, we find self-scaling nets anywhere evolution is trying to solve optimal-routing problems.

So here is my small stone to add to the Odlyzko/Tilly edifice: their assumption in 2006 was stronger than it needed to be. You still get selective pressure towards an O(n log n) self-scaling network even if the cost of connections still varies but the value of all potential connections is equal, not variable. The only assumptions you need are much simpler ones: that the owner of each node has a finite budget for connection-building small in relation to the cost of providing links to all nodes, and that network hops have a nonzero cost.

To see why, we need to recognize a new concept of the “access cost” of a node. The value of a node is by hypothesis constant: the access cost is the sum over all other nodes of any cost metric over a path to each node – distance, hop count, whatever.

in this scenario, each node owner wants to find the best links to the network, but the valuation minimizes access costs . Under this assumption, everyone is still trying to solve an optimal routing problem, so you still get self-scaling topology and O(n log n) statistics.

That’s it. Same result as Odlyzko/Tilly but with weaker assumptions. Put their case and my case together and you have this:

The value of a network of n nodes will rise as O(n log n) under the following assumptions: (a) node hops have variable costs, and (b) each node owner has a small budget.

Is this in the literature anywhere? If not, I guess it’s worth my time to do a more formal writeup.

Posted Tue Mar 28 20:16:44 2017 Tags:

This blog post is brought to you by a recent bad experience I had watching a 5-minute clip from Big Bang Theory on the recommendation of a friend who thought I might find it amusing.

Bleagh. This is supposed to be a show about geniuses? It’s not. It’s a show about a dimwit’s idea of what bright people are like. The slowest person in my peer group could out-think and out-create any of these sad-sack imitations of “smart” on any day of the week.

These actors are not bright, and don’t know how to fake it on screen. It occurred to me that I have seen this pulled off occasionally; the example that leaps to mind was Jennifer Love-Hewitt playing a bright scientist opposite Jackie Chan in Tuxedo (2003). She did a good enough job that I was later quite surprised at how relatively free of the ravages of intelligence she sounds in propria persona.

Ms. Love-Hewitt must have been at least smart enough to know that she should emulate the mannerisms of very bright people, and then set about doing it. After thinking about this, I thought it would be entertaining (and possibly useful) to compile some actionable advice for actors finding themselves in a similar situation.

Here goes a list of bright-person behavior signals which, while not universal, are very common…

Bright people have very precise diction and tend to self-assimilate to educated speech norms even if their formal education is minimal. Enunciate as crisply as you can. If the character is designed to have a regional or lower-class accent, dial it down a little. [UPDATE: I may have overgeneralized a bit here. Strongly true of STEM geeks, but maybe not as reliably of other kinds of brights.]

Bright people concentrate. Their casual attention to a task or person is as intense as most peoples’ full attention. So fixate on those targets – not to the point of being glassy-eyed about it, but to the point where stillness and attention dominate your body language.

Bright people spend a larger fraction of their time in an ‘on’ state of mental alertness or conscious thought than non-brights do. This has consequences in visual saccades that are easy to see – with a little practice, you can grade people by intelligence in bank or movie-theater lines by watching eye movements. Look for relatively little time spent in a defocused, half-asleep state – or, conversely, lots of time when the eyes are tracking or making motions indicative of either imaginative activity or memory retrieval. Thus, when you play a bright person, always be looking at something.

Do not fall into the robot trap. Bright people are not emotionless, not at all. They do tend to be more introspective and more controlled, which makes their emotional signaling less obvious. A good way to approach this mental stance is to behave like someone who is seeing wry, dry humor in everything.

The most common minor failure mode of bright people in dramatic situations is that they’ll have visible difficulty tolerating stupid behavior by those around them. The thing to get is that this is not egotism and shouldn’t be pushed out that way unless the character is an asshole by design; playing it not as assholery but as weary exasperation is usually truer.

Bright people move differently. This one is complicated. If they’re naturally physically graceful, the always-on/full-attention trait amplifies that a lot. If they’re physically clumsy, they may still exhibit startling if confined physical competence in trained skills – typing, playing a musical instrument, martial arts, whatever. What ties this together is that they’re good at all kinds of learning, including learning to use whatever physical ability they have efficiently.

Posted Tue Mar 28 00:03:50 2017 Tags:

My exercise in how small you can make a version-control system and still have it be useful, src, does seem to have a significant if quiet fanbase out there. I can tell because patches land in my mailbox at a slow but steady rate.

As the blurb says: Simple Revision Control is RCS/SCCS reloaded with a modern UI, designed to manage single-file solo projects kept more than one to a directory. Use it for FAQs, ~/bin directories, config files, and the like. Features integer sequential revision numbers, a command set that will seem familiar to Subversion/Git/hg users, and no binary blobs anywhere.

Posted Sun Mar 26 12:56:28 2017 Tags:

A few minutes ago here at chez Raymond, my friend John Desmond says: “So, have you heard about the new Iraqi national anthem?”

I said “Uh, OK, I’m braced for this. What about it?”

He said “In the good old Sumer time.”

I pointed a finger at him and said “You’re Akkad!”

Yes. Yes, we probably do both deserve a swift kicking.

Posted Wed Mar 22 04:03:23 2017 Tags:

When you've emptied a cupboard, put masking tape across it, ideally in a colour that's highly visible. This way you immediately see know which ones are finished and which ones still need attention. You won't keep opening the cupboard a million times to check and after the move it takes merely seconds to undo.

Posted Fri Mar 17 02:00:00 2017 Tags:

Was looking at using zstd for backup, and wanted to see the effect of different compression levels. I backed up my (built) bitcoin source, which is a decent representation of my home directory, but only weighs in 2.3GB. zstd -1 compressed it 71.3%, zstd -22 compressed it 78.6%, and here’s a graph showing runtime (on my laptop) and the resulting size:

zstandard compression (bitcoin source code, object files and binaries) times and sizes

For this corpus, sweet spots are 3 (the default), 6 (2.5x slower, 7% smaller), 14 (10x slower, 13% smaller) and 20 (46x slower, 22% smaller). Spreadsheet with results here.

Posted Thu Mar 9 00:53:39 2017 Tags:

I encourage all of you to either listen to or read the transcript of Terry Gross' Fresh Air interview with Joseph Turow about his discussion of his book “The Aisles Have Eyes: How Retailers Track Your Shopping, Strip Your Privacy, And Define Your Power”.

Now, most of you who read my blog know the difference between proprietary and Free Software, and the difference between a network service and software that runs on your own device. I want all of you have a good understanding of that to do a simple thought experiment:

How many of the horrible things that Turow talks about can happen if there is no proprietary software on your IoT or mobile devices?

AFAICT, other than the facial recognition in the store itself that he talked about in Russia, everything he talks about would be mitigated or eliminated completely as a thread if users could modify the software on their devices.

Yes, universal software freedom will not solve all the worlds' problems. But it does solve a lot of them, at least with regard to the bad things the powerful want to do to us via technology.

(BTW, the blog title is a reference to Philip K. Dick's Minority Report, which includes a scene about systems reading people's eyes to target-market to them. It's not the main theme of that particular book, though… Dick was always going off on tangents in his books.)

Posted Tue Feb 14 03:30:00 2017 Tags:

There are a lot of problems in our society, and particularly in the USA, right now, and plenty of charities who need our support. The reason I continue to focus my work on software freedom is simply because there are so few focused on the moral and ethical issues of computing. Open Source has reached its pinnacle as an industry fad, and with it, a watered-down message: “having some of the source code for some of your systems some of the time is so great, why would you need anything more?”. Universal software freedom is however further from reality than it was even a few years ago. At least a few of us, in my view, must focus on that cause.

I did not post many blog posts about this in 2016. There was a reason for that — more than any other year, work demands at Conservancy have been constant and unrelenting. I enjoy my work, so I don't mind, but blogging becomes low priority when there is a constant backlog of urgent work to support Conservancy's mission and our member projects. It's not just Conservancy's mission, of course, it's my personal one as well.

For our 2016 fundraiser, I wrote last year a blog post entitled “Do You Like What I Do For a Living?”. Last year, so many of you responded, that it not only made it possible for me to continue that work for one more year, but we were able to add our colleague Brett C. Smith to our staff, which brought Conservancy to four full-time staff for the first time. We added a few member projects (and are moving that queue to add more in 2017), and sure enough — the new work plus the backlog of work waiting for another staffer filled Brett's queue just like my, Karen's and Tony's was already filled.

The challenge now is sustaining this staffing level. Many of you came to our aid last year because we were on the brink of needing to reduce our efforts (and staffing) at Conservancy. Thanks to your overwhelming response, we not only endured, but we were able to add one additional person. As expected, though, needs of our projects increased throughout the year, and we again — all four of us full-time staff — must work to our limits to meet the needs of our projects.

Charitable donations are a voluntary activity, and as such they have a special place in our society and culture. I've talked a lot about how Conservancy's Supporters give us a mandate to carry out our work. Those of you that chose to renew your Supporter donations or become new Supporters enable us to focus our full-time efforts on the work of Conservancy.

On the signup and renewal page, you can read about some of our accomplishments in the last year (including my recent keynote at FOSDEM, an excerpt of which is included here). Our work does not follow fads, and it's not particularly glamorous, so only dedicated Supporters like you understand its value. We don't expect to get large grants to meet the unique needs of each of our member projects, and we certainly don't expect large companies to provide very much funding unless we cede control of the organization to their requests (as trade associations do). Even our most popular program, Outreachy, is attacked by a small group of people who don't want to see the status quo of privileged male domination of Open Source and Free Software disrupted.

Supporter contributions are what make Conservancy possible. A year ago, you helped us build Conservancy as a donor-funded organization and stabilize our funding base. I now must ask that you make an annual commitment to renewal — either by renewing your contribution now or becoming a monthly supporter, or, if you're just learning about my work at Conservancy from this blog post, reading up on us and becoming a new Supporter.

Years ago, when I was still only a part-time volunteer at Conservancy, someone who disliked our work told me that I had “invented a job of running Conservancy”. He meant it as an insult, but I take it as a compliment with pride. In fact, between me and my colleague (and our Executive Director) Karen Sandler, we've “invented” a total of four full-time jobs and one part-time one to advance software freedom. You helped us do that with your donations. If you donate again today, your donation will be matched to make the funds go further.

Many have told me this year that they are driven to give to other excellent charities that fight racism, work for civil and immigration rights, and other causes that seem particularly urgent right now. As long as there is racism, sexism, murder, starvation, and governmental oppression in the world, I cannot argue that software freedom should be made a priority above all of those issues. However, even if everyone in our society focused on a single, solitary cause that we agreed was the top priority, it's unlikely we could make quicker progress. Meanwhile, if we all single-mindedly ignore less urgent issues, they will, in time, become so urgent they'll be insurmountable by the time we focus on them.

Industrialized nations have moved almost fully to computer automation for most every daily task. If you question this fact, try to do your job for a day without using any software at all, or anyone using software on your behalf, and you'll probably find it impossible. Then, try to do your job using only Free Software for a day, and you'll find, as I have, that tasks that should take only a few minutes take hours when you avoid proprietary software, and some are just impossible. There are very few organizations that are considering the long-term implications of this slowly growing problem and making plans to build the foundations of a society that doesn't have that problem. Conservancy is one of those few, so I hope you'll realize that long-term value of our lifelong work to defend and expand software freedom and donate.

Posted Mon Feb 13 15:20:00 2017 Tags:

There are a lot of problems in our society, and particularly in the USA, right now, and plenty of charities who need our support. The reason I continue to focus my work on software freedom is simply because there are so few focused on the moral and ethical issues of computing. Open Source has reached its pinnacle as an industry fad, and with it, a watered-down message: “having some of the source code for some of your systems some of the time is so great, why would you need anything more?”. Universal software freedom is however further from reality than it was even a few years ago. At least a few of us, in my view, must focus on that cause.

I did not post many blog posts about this in 2016. There was a reason for that — more than any other year, work demands at Conservancy have been constant and unrelenting. I enjoy my work, so I don't mind, but blogging becomes low priority when there is a constant backlog of urgent work to support Conservancy's mission and our member projects. It's not just Conservancy's mission, of course, it's my personal one as well.

For our 2016 fundraiser, I wrote last year a blog post entitled “Do You Like What I Do For a Living?”. Last year, so many of you responded, that it not only made it possible for me to continue that work for one more year, but we were able to add our colleague Brett Smith to our staff, which brought Conservancy to four full-time staff for the first time. We added a few member projects (and are moving that queue to add more in 2017), and sure enough — the new work plus the backlog of work waiting for another staffer filled Brett's queue just like my, Karen's and Tony's was already filled.

The challenge now is sustaining this staffing level. Many of you came to our aid last year because we were on the brink of needing to reduce our efforts (and staffing) at Conservancy. Thanks to your overwhelming response, we not only endured, but we were able to add one additional person. As expected, though, needs of our projects increased throughout the year, and we again — all four of us full-time staff — must work to our limits to meet the needs of our projects.

Charitable donations are a voluntary activity, and as such they have a special place in our society and culture. I've talked a lot about how Conservancy's Supporters give us a mandate to carry out our work. Those of you that chose to renew your Supporter donations or become new Supporters enable us to focus our full-time efforts on the work of Conservancy.

On the signup and renewal page, you can read about some of our accomplishments in the last year (including my recent keynote at FOSDEM, an excerpt of which is included here). Our work does not follow fads, and it's not particularly glamorous, so only dedicated Supporters like you understand its value. We don't expect to get large grants to meet the unique needs of each of our member projects, and we certainly don't expect large companies to provide very much funding unless we cede control of the organization to their requests (as trade associations do). Even our most popular program, Outreachy, is attacked by a small group of people who don't want to see the status quo of privileged male domination of Open Source and Free Software disrupted.

Supporter contributions are what make Conservancy possible. A year ago, you helped us build Conservancy as a donor-funded organization and stabilize our funding base. I now must ask that you make an annual commitment to renewal — either by renewing your contribution now or becoming a monthly supporter, or, if you're just learning about my work at Conservancy from this blog post, reading up on us and becoming a new Supporter.

Years ago, when I was still only a part-time volunteer at Conservancy, someone who disliked our work told me that I had “invented a job of running Conservancy”. He meant it as an insult, but I take it as a compliment with pride. In fact, between me and my colleague (and our Executive Director) Karen Sandler, we've “invented” a total of four full-time jobs and one part-time one to advance software freedom. You helped us do that with your donations. If you donate again today, your donation will be matched to make the funds go further.

Many have told me this year that they are driven to give to other excellent charities that fight racism, work for civil and immigration rights, and other causes that seem particularly urgent right now. As long as there is racism, sexism, murder, starvation, and governmental oppression in the world, I cannot argue that software freedom should be made a priority above all of those issues. However, even if everyone in our society focused on a single, solitary cause that we agreed was the top priority, it's unlikely we could make quicker progress. Meanwhile, if we all single-mindedly ignore less urgent issues, they will, in time, become so urgent they'll be insurmountable by the time we focus on them.

Industrialized nations have moved almost fully to computer automation for most every daily task. If you question this fact, try to do your job for a day without using any software at all, or anyone using software on your behalf, and you'll probably find it impossible. Then, try to do your job using only Free Software for a day, and you'll find, as I have, that tasks that should take only a few minutes take hours when you avoid proprietary software, and some are just impossible. There are very few organizations that are considering the long-term implications of this slowly growing problem and making plans to build the foundations of a society that doesn't have that problem. Conservancy is one of those few, so I hope you'll realize that long-term value of our lifelong work to defend and expand software freedom and donate.

Posted Mon Feb 13 15:20:00 2017 Tags:

libinput has a couple of features that 'automagically' work on touchpads such as disable-while-typing and the lid switch triggered disabling of touchpads and disabling the touchpad when an external mouse is plugged in [1]. But not all of these features make sense on all touchpads. For example, an Apple Magic Trackpad doesn't need disable-while-typing because unless you have a creative arrangement of input devices [2], the touchpad won't be where your palm is likely to hit it. Likewise, a Logitech T650 connected over a unifying receiver shouldn't get disabled when the laptop lid closes.

For this to work, libinput has some code to figure out whether a touchpad is internal or external. Initially we had some code to detect this but eventually moved this to the ID_INPUT_TOUCHPAD_INTEGRATION property now set by udev's hwdb (systemd 231 and later). Having it in the hwdb makes it quite trivial to override locally where the current rules are insufficient (and until the hwdb is fixed, thanks for filing a bug). We still have the fallback code though in case the tag is missing. On a sufficiently modern distribution, udevadm info /sys/class/input/event4 for your touchpad device node should show something like ID_INPUT_TOUCHPAD_INTEGRATION=internal.

So for any feature that libinput adds for touchpads, we only enable it where it makes sense. That's why your external touchpad doesn't trigger disable-while-typing or the lid switch.

[1] ok, I admit, this is something we should've left to the client, but now we have the feature.
[2] yes, I'm sure there's at least one person out there that uses the touchpad upside down in front of the keyboard and is now angry that libinput doesn't allow arbitrary rotation of the device combined with configurable dwt. I think of you every night I cry myself to sleep.

Posted Fri Feb 10 00:27:00 2017 Tags:

My New Years resolution is to restart blogging.jigsawfish2

Bufferbloat is the most common underlying cause of most variable bad performance on the Internet; it is called “lag” by gamers.

Trying to steer anything the size of the Internet into a better direction is very slow and difficult at best. From the time changes in the upstream operating systems are complete to when consumers can buy new product is typically four years caused by the broken and insecure ecosystem in the embedded device market. Chip vendors, box vendors, I’m looking at you… So much of what is now finally appearing in the market is based on work that is often four years old. Market pull may do what push has not.

See What to do About Bufferbloat for general information. And the DSLReports Speedtest makes it easy to test for bufferbloat. But new commercial products are becoming increasingly available.  Here’s some of them.

Introduction

The fq_codel & cake work going on in the bufferbloat project is called SQM – “smart queue management.” This SQM work is specifically targeted at mitigating the bufferbloat in the “last mile,” your cable/DSL/fiber connection, by careful queue management and an artificial bandwidth bottleneck added in your home router (since most modems do no perform flow control to the home router, unfortunately).

Modems require built in AQM algorithms, such as those just beginning to reach the market in DOCSIS 3.1. I just ordered one of these for my house to see if it functions better than the SQM mitigation (almost certainly not), but at least these should not require the manual tuning that SQM does.

To fix bufferbloat in WiFi requires serious changes in the WiFi driver in your home router (which typically runs Linux), and in your device (laptop/phone/tablet).  The device driver work was first released as part of the LEDE project, in January 2017 for initially just a couple of WiFi chip types.

Evenroute IQrouter

First up, I’d like call out the Evenroute IQrouter, which has a variant of SQM that deals with “sag”. DSL users have often suffered more than other broadband users, due to bad bloat in the modems compounded by minimal bandwidth, so the DSL version of the IQrouter is particularly welcome.   Often DSL ISP’s seem to have the tendency (seemingly more often than ISPs with other technologies) to under provision their back haul, causing “sag” at different times of day/week.  This makes the static configuration techniques we’ve used in LEDE/OpenWrt SQM ineffective, as you have to give away too much bandwidth if a fixed bandwidth is used.  I love the weasel words “up to” some speed used by many ISPs. It is one thing for your service to degrade for a short period of days or weeks while an ISP takes action to provision more bandwidth to an area; it is another for your bandwidth to routinely vary by large factors for weeks/months and years.

I sent a DSL Evenroute IQrouter to my brother in Pennsylvania recently and arranged for one for a co-worker, and they are working well, and Rich Brown has had similarly good experiences. Evenroute has been working hard to make the installation experience easy. Best yet, is that the IQrouter is autoconfiguring and figures out for you what to do in the face of “sag” in your Internet service, something that may be a “killer feature” if you suffer lots of “sag” from your ISP. The IQrouter is therefore the first “out of the box” device I can recommend to almost anyone, rather than just my geek friends.

The IQRouter does not yet have the very recent wonderful WiFi results of Toke and Dave (more about coming this in a separate post), but has the capability for over the air updates and one hopes debloated WiFi and ATF will come to it reasonably soon. The new WiFi stack is just going upstream into Linux and LEDE/OpenWRT as I write this post.  DSL users seldom have enough bandwidth for the WiFi hop to be the bottleneck; so the WiFi work is much more important for Cable and fiber users at higher bandwidth than for DSL users stuck at low bandwidth.

Ubiquiti Edgerouter

I’ve bought an Ubiquiti Edgerouter X on recommendation of Dave Taht but not yet put it into service. Router performance can be an issue on high end cable or fiber service. It is strictly an Ethernet router, lacking WiFi interfaces; but in my house, where the wiring is down in the basement, that’s what I need.  The Edgerouter starts at around $50; the POE version I bought around $75.

The Edgerouter story is pretty neat – Dave Taht did the backport 2? years back. Ubiquti’s user community jumped all over it and polished it up, adding support to their conf tools and GUI, and Ubiquiti recognized what they had and shipped it as part of their next release.

SQM is available in recent releases of Ubituiti’s Edgerouter firmware.  SQM itself is easy to configure. But the Edgerouter overall requires considerable configuration before it is useful in the home environment, however, and its firmware web interface is aimed at IT people rather than most home users. I intend this to replace my primary router TP-Link Archer C7v2 someday soon, as it is faster than the TP-Link since Comcast keeps increasing my bandwidth without asking me.  I wish the Ubiquiti had a “make me into a home router” wizard that would make it immediately usable for most people, as its price is low enough for some home users to be interested in it.   I believe one can install LEDE/OpenWrt on the Edgerouter, which I may do if I find its IT staff oriented web interface too unusable.

LEDE/OpenWrt and BSD for the Geeks

If you are adventurous enough to reflash firmware, anything runnable on OpenWrt/LEDE of the last few years has SQM available. You take the new LEDE release for a spin. If your router has an Ath9k WiFi chip (or a later version of the Ath10k WiFi chip), or you buy a new router with the right chips in them, you can play with the new WiFi goodness now in LEDE (noted above). There is a very wide variety of home routers that can benefit from reflashing. Its web UI is tolerably decent, better than many commercial vendors I have seen.

WiFi chip vendors should take careful note of the stupendous improvements available in the Linux mac802.11 framework for bufferbloat elimination and air time fairness. If you don’t update to the new interfaces and get your code into LEDE, you’re going to be at a great disadvantage to Atheros in the market.

dd-wrt, asuswrt, ipfire, all long ago added support for SQM. It will be interesting to see how long it takes them to pick up the stunning WiFi work.

The pcengines APU2 is a good “DIY” router for higher speeds. Dave has not yet tried LEDE on it yet, but will. He uses it presently on Ubuntu….

BSD users recently got fq_codel in opnsense, so the BSD crowd are making progress.

Other Out of the Box Devices

The Turris Omnia is particularly interesting for very fast broadband service and can run LEDE as well; but unfortunately,  it seems only available in Europe at this time.  We think the Netduma router has SQM support, though it is not entirely clear what they’ve done; it is a bit pricey for my taste, and I don’t happen to know anyone who has one.

Cable Modems

Cable users may find that upgrading to a new DOCSIS 3.1 modem is helpful (though that does not solve WiFi bufferbloat).  The new DOCSIS 3.1 standard requires AQM.  While I don’t believe PIE anywhere as good as fq_codel (lacking flow queuing), the DOCSIS 3.1 standard at least requires an AQM, and PIE should help and does not require manual upstream bandwidth tuning.  Maybe someday we’ll find some fq_codel (or fq_pie) based cable modems.  Here’s hoping…

Under the Covers, Hidden

Many home routers vendors make bold claims they have proprietary cool features, but these are usually smoke and mirrors. Wireless mesh devices without bufferbloat reduction are particularly suspect and most likely to require manual RF engineering beyond most users. They require very high signal strength and transfer rates to avoid the worst of bufferbloat. Adding lots more routers without debloating and not simultaneously attacking transmit power control is a route to WiFi hell for everyone. The LEDE release is the first to have the new WiFi bits needed to make wireless mesh more practical. No one we know of has been working on minimizing transmit power to reduce interference between mesh nodes. So we are very skeptical of these products.

There are now a rapidly increasing number of products out there with SQM goodness under the covers, sometimes implemented well, and sometimes not so well, and more as the months go by.

One major vendor put support for fq_codel/SQM under the covers of one product using a tradename, promptly won an award, but then started using that tradename on inferior products in their product line that did not have real queue management. I can’t therefore vouch for any product line tradename that does not acknowledge publicly how it works and that the tradename means that it really has SQM under the covers. Once burned, three times shy. That product therefore does not deserve a mention due to the behavior of the vendor. “Bait and switch” is not what anyone needs.

Coming Soon…

We have wind of a number of vendors’ plans who have not quite reached the market, but it is up to them to announce their products.

If you find new products or ISP’s that do really well, let us know, particularly if they actually say what they are doing. We need to start some web pages to keep track of commercial products.


Posted Thu Feb 2 08:00:00 2017 Tags: