The lower-post-volume people behind the software in Debian. (List of feeds.)
In my last blog post I expressed my severe disappointment with the gap between the Rust language in theory (as I had read about it) and the Rust language in practice, as I encountered it when I actually tried to write something in it.
Part of what I hoped for was a constructive response from the Rust community. I think I got that. Amidst the expected volume of flamage from rather clueless Rust fanboys, several people (I’m going to particularly call out Brian Campbell, Eric Kidd, and Scott Lamb) had useful and thoughtful things to say.
I understand now that I tested the language too soon. My use case – foundational network infrastructure with planning horizons on a decadal scale – needs stability guarantees that Rust is not yet equipped to give. But I’m somewhat more optimistic about Rust’s odds of maturing into a fully production-quality tool than I was.
Still, I think I see a problem in the assumptions behind Rust’s development model. The Rust community, as I now understand it, seems to me to be organized on a premise that is false, or at least incomplete. I fear I am partly responsible for that false premise, so I feel a responsibility to address it square on and attempt to correct it.
Technically, I greatly admire Rust’s crate system. I see its continuity with earlier experiments in the same general direction – Perl CPAN, Python Pip. But crates are easier to use than their predecessors and get more details right. The differences are evolutionary, but they matter – they add up to making crates a near ideal tool for enabling code re-use and a decentralized swarm attack on the design space around Rust.
It’s “let a thousand modules bloom”, and exactly the kind of approach I advocated in my foundational work on the open-source development model. I pushed hard in that direction back around the turn of the century because that’s what the times demanded; our main problem then was getting out from under overcontrolling practices that were holding back software engineering and serving both its craftsmen and its customers rather poorly.
Now, confronting a purer expression of those decentralist ideas than we then knew how to build, I worry that I may have encouraged a kind of utopianism – a belief that if we just cultivate enough decentralization and divergence of approach, good whole systems will just sort of coalesce out of the chaos without anyone having to make hard decisions.
But evolution doesn’t work that way. Ratcheting up adaptive fitness requires not just mutation but selective pressure. Alternatives need to be winnowed as well as generated. Markets (even reputation markets) have penalties as well as rewards – if you offer an inferior product, people won’t buy it and eventually you need to go do something else to survive.
Which brings me directly to what bothers me about the crate system and the sociology behind it – I don’t see any pruning. Worse, I don’t see much understanding of the need for it. A lot of Rustaceans don’t seem to grasp why, when the question is “where do I get feature X?” the answer “oh, there are 23 crates for that” is objectively terrifying.
How many of those crates are student exercises or plain junk? How do I tell which ones are still maintained? How do I audit for quality? How do I form expectations about which will still be maintained in ten years? How do I even find all the crates that might be relevant to my interests?
This isn’t a question that comes up so often with respect to (say) Python because Python has an effective form of curation – blessing things into the standard library, at which point their alternatives generally disappear. In effect, Python modules are quality-filtered on the taste of the BDFL and the devteam.
(This is, in fact, why Benevolent Dictator For Life is such a common governance model in our otherwise rather anarchic community. Experience has shown that curation by one person with a clear design vision generally beats attempts to design by committee.)
The same question comes up at other levels. At the whole-program level, we have operating-system distributions precisely in order to curate the riotous diversity of software into semi-coherent wholes with an implied minimum quality threshold. Again: selective pressure.
Rust seems to be missing any analogous mechanism, and that worries me a lot. It’s what I meant when I said in my last post that the swarm attack seems to be failing.
To sharpen this point, I’ll tell you what I think “success” looks like. As a potential Rust user, what I want to do is be able to go from a given feature requirement to one high-quality implementation with an implicit long-term stability guarantee. This, folks, is what a high-quality production language looks like, and not by accident – it minimizes discovery costs for actual production users.
Getting to this from the crate ecology as it is now conceived is not just a technology problem, it’s a a challenge to how the Rust community imagines itself. And, as I trust I’ve made clear, not a unique one. Any sort of open-source ecology eventually has to face a similar transition to maturity.
It could be as conceptually simple as adding a rating system to crates.io, allowing the user to filter on ratings, and having one of the ratings being “Approved by the core team; we take responsibility for this.” But someone is going to have to decide. Who is it going to be? And who decides who will decide?
Backing up a bit: in scheduling theory and operations research there’s a class of problems called “explore/exploit dilemmas”. They show up everywhere where you have limited resources to invest and have to choose between generating options so you’ll have good ones later and harvesting the rewards from those you already have. They show up in the strategy games I like to play, especially in pick-up-and carry games where (say) you’re running a simulated railroad to try to make the money the fastest. Early in the game, you’re mainly interested in investing so you can build an efficient network. But later on you have to back off building track and pile up money using what you have, otherwise you’ll be overtaken by players who chose that transition point more shrewdly.
Rust has built a good machine for exploring. But you guys have competition out there: Go, Nim, Swift, D, and maybe other languages that aren’t on anyone’s radar yet. I think you’re going to need to make that turnover into exploit mode before you know it, and that this is tied into your challenges about curation and governance.
Here’s my advice, if you’ll take it: get conscious!. Grapple with what success looks like from a potential user’s point of view, the challenge of curation, and the problem of who decides. It’s either that, or remain a toy language with an inward-facing community forever.
I wanted to like Rust. I really did. I’ve been investigating it for months, from the outside, as a C replacement with stronger correctness guarantees that we could use for NTPsec.
I finally cleared my queue enough that I could spend a week learning Rust. I was evaluating it in contrast with Go, which I learned in order to evaluate as a C replacement a couple of weeks back.
My chosen learning project in Rust was to write a simple IRC server. As a service daemon with no real-time requirements written around a state machine of the kind I can code practically in my sleep, I thought this would make a good warmup for NTP.
In practice, I found Rust painful to the point of unusability. The learning curve was far worse than I expected; it took me those four days of struggling with inadequate documentation to write 67 lines of wrapper code for the server.
Even things that should be dirt-simple, like string concatenation, are unreasonably difficult. The language demands a huge amount of fussy, obscure ritual before you can get anything done.
The contrast with Go is extreme. By four days in of exploring Go I had mastered most of the language, had a working program and tests, and was adding features to taste.
Then I found out that a feature absolutely critical for writing network servers is plain missing from Rust. Contemplate this bug report: Is there some API like “select/poll/epoll_wait”? and get a load of this answer:
We do not currently have an epoll/select abstraction. The current answer is “spawn a task per socket”.
Upon further investigation I found that there are no proposals to actually fix this problem in the core language. The comments acknowledge that there are a welter of half-solutions in third-party crates but describe no consensus about which to adopt.
Not only that, but it seems the CSP implementation in Rust – the most tractable concurrency primitive – has limits that make it unusable for NTPsec’s purposes (only selection over a static set of channels is possible) and there is some danger that it might be removed entirely!
I have discovered that Rust is not, or not yet, a language suitable for long-term infrastructure work. It suffers from a damning combination of high barriers to entry and technical deficiency.
What is worse is that Rust’s community seems to be unable to fix or even clearly grasp these problems.
I think one problem here is the absence of a strong BDFL providing tasteful design direction and setting priorities. The Rust community appears to have elected to use the decentralized nature of their crate system (which undeniably has some very nice technical properties) to execute a swarm attack on the design space around the language. Which is wonderful in theory but seems to be failing in practice.
UPDATE: This post attracted two kinds of pro-Rust response. One was stupid flamage from zealots. The other was thoughtful commentary from a few people close to the core of the Rust community. The latter group has convinced me that there is considerable awareness of the problems I ran into; a couple even agreed, after analysis, that Rust is at present a poor fit for NTPsec’s requirements. This gives me hope that the Rust of five years from now may become the mature and effective replacement for C that it is not yet.
This post describes the synclient tool, part of the xf86-input-synaptics package. It does not describe the various options, that's what the synclient(1) and synaptics(4) man pages are for. This post describes what synclient is, where it came from and how it works on a high level. Think of it as a anti-bus-factor post.
The most important thing first: synclient is part of the synaptics X.Org driver which is in maintenance mode, and superseded by libinput and the xf86-input-libinput driver. In general, you should not be using synaptics anymore anyway, switch to libinput instead (and report bugs where the behaviour is not correct). It is unlikely that significant additional features will be added to synclient or synaptics and bugfixes are rare too.
synclient's interface is extremely simple: it's a list of key/value pairs that would all be set at the same time. For example, the following command sets two options, TapButton1 and TapButton2:
The -l switch lists the current values in one big list:
synclient TapButton1=1 TapButton2=2
The commandline interface is effectively a mapping of the various xorg.conf options. As said above, look at the synaptics(4) man page for details to each option.
$ synclient -l
LeftEdge = 1310
RightEdge = 4826
TopEdge = 2220
BottomEdge = 4636
FingerLow = 25
FingerHigh = 30
MaxTapTime = 180
A decade ago, the X server had no capabilities to change driver settings at runtime. Changing a device's configuration required rewriting an xorg.conf file and restarting the server. To avoid this, the synaptics X.Org touchpad driver exposed a shared memory (SHM) segment. Anyone with knowledge of the memory layout (an internal struct) and permission to write to that segment could change driver options at runtime. This is how synclient came to be, it was the tool that knew that memory layout. A synclient command would thus set the correct bits in the SHM segment and the driver would use the newly updated options. For obvious reasons, synclient and synaptics had to be the same version to work.
8 or so years ago, the X server got support for input device properties, a generic key/value store attached to each input device. The keys are the properties, identified by an "Atom" (see box on the side). The values are driver-specific. All drivers make use of this now, being able to change a property at runtime is the result of changing a property that the driver knows of.
synclient was converted to use properties instead of the SHM segment and eventually the SHM support was removed from both synclient and the driver itself. The backend to synclient is thus identical to the one used by the xinput tool or tools used by other drivers (e.g. the xsetwacom tool). synclient's killer feature was that it was the only tool that knew how to configure the driver, these days it's merely a commandline argument to property mapping tool. xinput, GNOME, KDE, they all do the same thing in the backend.
How synclient works
The driver has properties of a specific name, format and value range. For example, the "Synaptics Tap Action" property contains 7 8-bit values, each representing a button mapping for a specific tap action. If you change the fifth value of that property, you change the button mapping for a single-finger tap. Another property "Synaptics Off" is a single 8-bit value with an allowed range of 0, 1 or 2. The properties are described in the synaptics(4) man page. There is no functional difference between this synclient command:
and this xinput command
Both set the same property with the same calls. synclient uses XI 1.x's XChangeDeviceProperty() and xinput uses XI 2.x's XIChangeProperty() if available but that doesn't really matter. They both fetch the property, overwrite the respective value and send it back to the server.
xinput set-prop "SynPS/2 Synaptics TouchPad" "Synaptics Off" 1
Pitfalls and quirks
synclient is a simple tool. If multiple touchpads are present it will simply pick the first one. This is a common issue for users with a i2c touchpad and will be even more common once the RMI4/SMBus support is in a released kernel. In both cases, the kernel creates the i2c/SMBus device and an additional PS/2 touchpad device that never sends events. So if synclient picks that device, all the settings are changed on a device that doesn't actually send events. This depends on the order the devices were added to the X server and can vary between reboots. You can work around that by disabling or ignoring the PS/2 device.
synclient is a one-shot tool, it does not monitor devices. If a device is added at runtime, the user must run the command to change settings. If a device is disabled and re-enabled (VT-switch, suspend/resume, ...), the user must run synclient to change settings. This is a major reason we recommend against using synclient, the desktop environment should take care of this. synclient will also conflict with the desktop environment in that it isn't aware when something else changes things. If synclient runs before the DE's init scripts (e.g. through xinitrc), its settings may be overwritten by the DE. If it runs later, it overwrites the DE's settings.
synclient exclusively supports synaptics driver properties. It cannot change any other driver's properties and it cannot change the properties created by the X server on each device. That's another reason we recommend against it, because you have to mix multiple tools to configure all devices instead of using e.g. the xinput tool for all property changes. Or, as above, letting the desktop environment take care of it.
The interface of synclient is IMO not significantly more obvious than setting the input properties directly. One has to look up what TapButton1 does anyway, so looking up how to set the property with the more generic xinput is the same amount of effort. A wrong value won't give the user anything more useful than the equivalent of a "this didn't work".
If you're TL;DR'ing an article labelled "the definitive guide to" you're kinda missing the point...
Every once in a while I post something just to have it handy as a reference for the next time I have to deal with a galloping case of some particular kind of sloppy thinking. That way I don’t have to generate an individual explanation, but can simply point at my general standards of evidence.
This one is about accusations of sexism, racism, and other kinds of prejudice in the open-source culture.
First, a statement of principle: in the hacker culture, you should be judged by your work and your work alone. It is wrong for anyone to be attacked, belittled, or excluded because of the color of their skin, the shape of their genitals, what they like to do with their genitals, their politics, their religion, or any other irrelevancy. We are, and should remain, a place where those marginalized because of some aspect of their meatspace identities can find community and a place to do fulfilling work.
I have always stood up for this norm, and will till I die. If I am presented with evidence that anyone in the community has violated it, I will exert whatever authority I have as a tribal elder to condemn that violation and point the community in an ethically correct direction.
That said, in practice I believe such violations are very rare. So rare, in fact, that I cannot now say I know with certainty of even one. On the other hand, I do know of a lot of accusations having been flung by three categories of people: the mentally disturbed, drama queens, and political carpetbaggers. Of these, I consider the last – people seeking social and political power that they have not earned through the merit of their work – to be the most dangerous, enough so that they cannot merely be ignored but must be actively countered and ejected from our community.
In the remainder of this post I will explain what you need to do to present me with a prejudice-related grievance in order to get my full attention. I cannot enforce these standards on other elders or anyone else, but I recommend them to all.
First: Be humble. Don’t walk in assuming your outrage over whatever injustice is bothering you entitles you to dictate to us. It doesn’t – and, anyway, hackers are often prickly, countersuggestible people who don’t take well to what they perceive as attempts to jerk them around, so you’ll self-sabotage if you come on too strong. We have lots of work to do and limited patience for distractions; your cause may be important, but you are not, so start humble and reasonable and stay that way.
Second: show me your code. I want to see URLs to public repositories with your commits in them. (OpenHub statistics will do for a first cut.) Your credibility goes up with commit volume and number of different projects. and especially with the number of other people you have collaborated with.
In theory, I might be open to other metrics than commit volume for people who aren’t primarily software engineers. But that’s an edge case; the point is, whether it’s lines of code or Thingiverse objects or PCB layouts, I want to see evidence of contributed work.
There are three reasons I filter on this. One is that if you don’t contribute to the open-source work, I don’t consider that you have earned the right to lecture me or the open-source community on how we should behave or think.
Another is that if you haven’t put in time playing well with others, any claim you make to know how the community operates and whether it in fact suffers from ingrained prejudices is ungrounded. You need to know our problems, our adaptive strategies, and the reasons we organize and communicate the way we do before your opinion will be worth anything. Make your bones, get that experience, then maybe we can talk.
A third reason is that this is an extremely difficult filter for the people who generate false positives – the mentally disturbed, the drama queens, and the political carpetbaggers – to actually pass. Usually they suffer from a combination of stupidity, laziness, and antisociality that prevents them from contributing effectively. By stopping them here at an objective criterion we can avoid more difficult arguments about later filters.
Third: show me your evidence. I want to see evidence of specific harm, attack, or attempts to exclude, on identifiable victims, by identifiable perpetrators. It isn’t sufficient to say, for example, “Women (or black people, or gays) can’t get their patches accepted, or are sexually/racially taunted on forums.” and then wave your hands as though the accusation itself is to be treated as evidence and anyone demanding specifics is part of the problem.
I want to see concrete evidence of specific incidents – mailing-list traffic, IRC captures, pointers to web pages. If you can’t produce that evidence, you aren’t having a problem with the public behavior of hackers (or anyone else) and I can’t address it.
Fourth: Do not ever try to kafkatrap me. You do that, your credibility goes to negative infinity and stays there. You not only discredit yourself, you damage your allies and your cause.
Fifth: Convince me that you’re actually talking about anyone who actually regards me as a tribal elder. This means that you can’t go on about gamers, or 4chan, or neoreactionaries, or “brogrammers”, or any one of three dozen other on-line cultures or population categories in which a reasonable person might (rightly or wrongly) read evidence of bigotry, and expect me to care more than in a general, abstract way. They aren’t my people or my problem; you need to go find their tribal elders and complain to them.
If, on the other hand, you had a bad experience somewhere else and insist on sweeping open-source hackers into the same bucket because we look something like those people, or smell like them, or whatever…then you are the problem.
Sixth: If you have evidence of a specific instance, and want to persuade me that it is an index for a general pattern of prejudiced or hostile or belittling behavior, then come equipped with a generative theory of why your experience far from me differs from the almost ideally unprejudiced behavior I have observed near me over nearly forty years.
That is, you need to explain why I should consider that your claim of systemic prejudice flatly contradicts my everyday experience of hackers not caring about anything but quality of work. As a feminist would say, you need to not deny my lived experience.
Note: theories of the form “You’re blinded by your own prejudices/privilege” are kafkatrapping; see above, these will just set your credibility to negative infinity. Bring a theory which can be tested by falsifiable consequences, or don’t bother.
Generally, remember that neither I nor my community have a lot of patience for sloppy thinking, special pleading, or lazy guilt-tripping. If your reaction to this advice is to dismiss these as defensive rationalizations for not giving a shit, then we don’t give a shit about you.
If you pass all these filters, maybe you have something to teach me, and maybe you’ll get to see what I’m like when I am righteously pissed off because hacker norms have been violated in a serious way. It is part of my job to come down like the wrath of God when that happens – it’s what my community trusts and expects me to do.
That is all.
Back when I was in university, I visited a bookstore and found, in the physics section, a group of books on relativity. One of the shortest books with the least glossy cover had a surprising author: Albert Einstein. Surprising? Well, he formed the theory, but everyone knows he wrote science papers, not books, and for advanced physicists, not average people, right?
Apparently not. You can still buy Relativity, the Special and General Theory on Amazon, including a Kindle format for $0.99. From Einstein's own description, "The present book is intended, as far as possible, to give an exact insight into the theory of Relativity to those readers who, from a general scientific and philosophical point of view, are interested in the theory, but who are not conversant with the mathematical apparatus of theoretical physics."
Sure enough, the book is very readable, and didn't require any advanced math in order to understand the concepts. Einstein himself understood his theory so well that he was able to explain it in terms an average person can understand.
In contrast, the other relativity books on the same shelf were more complicated and harder to read. Maybe the authors understood just as well as Einstein, and had the benefit of decades more research in the area, but somehow they managed to obscure the main point. Perhaps part of genius, or some kinds of genius, is just the ability to explain your deep thoughts to others.
Anyway, I took that as an important lesson: don't be afraid to go back to the original source when you want to learn something. It's not always as intimidating as it sounds.
Which brings us to this Christmas holiday, where I had a few hours here and there to catch up on reading, and I set out to learn about the quality revolution in manufacturing (generally agreed to have started in Japan, albeit with ironic American influence). I wanted to know: What was the big insight? Did westerners finally catch on? Does it all just seem obvious to us now? And even more importantly, does it apply to software development, or is software too different from manufacturing for the concepts to really apply? (In particular, it seems clear that manufacturing quality is largely about doing a one-time design and reliably producing zillions of identical items, and the problems seem to be mainly in the reliable reproduction. In software, reproduction is the easy part.)
In my stint as a "Semiconductor Industry Veteran" (The Guardian's words, not mine for four months in the late 1990s, I had heard buzzwords related to this topic. Notably: Toyota Production System (TPS), Six Sigma, Zero Defects, and Total Quality Management. I didn't know much about them, just the names. Well, I had heard that in the TPS, any worker could push a big red button and stop the assembly line if they found a defect, but I didn't really see how that could help. Six Sigma sounded fishy... six sigma from what? Just three sigma is better than 99%, so Six Sigma must be essentially Zero Defects, which sounds nice, it must be what Japanese companies were doing. Total Quality Management sounds like a scam just from the name, so let's just ignore that one.
So it's a vacation, but I have limited time, what should I read? After a
bit of investigation, one thing the 1990's Japanese System
fiction management consultants all seemed to have in common was an
influence by W.
Edwards Deming, an American who apparently either co-invented or at
least evangelized all this stuff around Japan (I'm still not quite sure).
Maybe let's go there and see if we can find the original source.
(It later turned out that Deming was in turn heavily influenced by Walter A. Shewhart's work in the 1920s, but that's going back a bit far, and it seems like Deming generalized his work to a bunch of new areas, so let's stick with Deming for now.)
A quick trip to Amazon brought me to Deming's last book, The New Economics for Industry, Government, Education, which wins the dubious award for "most overpriced e-book by a dead guy I have ever seen." But on the other hand, if it's that expensive, maybe there's something good in there, so I bought it.
It was even better than I thought. I've never highlighted so many passages in a single book before. I don't even like highlighting passages in books. And this is a pretty short book. It gets right to the point, demolishing everything I thought I knew. (It also includes some well-placed jabs at Six Sigma, a management technique supposedly based on Deming's methods!)
(Around the part of the book where he was insulting Six Sigma, I thought I'd better look it up to see what it's about. It's indeed pretty awful. Just to start with, the "six" sigma are just because they count the entire range from -3 standard deviations to +3 standard deviations. That's an error margin +/- 3 stddev, or ~99.7%. It's not "six" standard deviations by any normal human's terminology. It gets worse from there. Safe to ignore.)
Anyway, back to demolishing everything I thought I knew. I was looking for advice on how to improve quality, which means I was kinda surprised when early in the book I found this:
I listened all day, an unforgettable Thursday, to 10 presentations, reports of 10 teams, on reduction of defects. The audience was composed of 150 people, all working on reduction of defects, listening intently, with touching devotion to their jobs. They did not understand, I think, that their efforts could in time be eminently successful-no defects-while their company declines. Something more must take place, for jobs.
In a book that I thought was about reducing defects, this was a little disconcerting.
We move on to,
Have you ever heard of a plant that closed? And why did it close? Shoddy workmanship? No.
Um, what is this? Maybe I should go get a Total Quality Management book after all. (Just kidding. I refuse to acknowledge any religion with a name that dumb.)
He moved on to blast a few more things that many of us take as axiomatic:
The effect of Hard work? Best efforts? Answer: We thus only dig deeper the pit that we are in. Hard work and best efforts will not by themselves dig us out of the pit. [...]
Ranking is a farce. Apparent performance is actually attributable mostly to the system that the individual works in, not to the individual himself. [...]
In Japan, the pecking order is the opposite. A company in Japan that runs into economic hardship takes these steps:2 1. Cut the dividend. Maybe cut it out. 2. Reduce salaries and bonuses of top management. 3. Further reduction for top management. 4. Last of all, the rank and file are asked to help out. People that do not need to work may take a furlough. People that can take early retirement may do so, now. 5. Finally, if necessary, a cut in pay for those that stay, but no one loses a job. [...]
Appointment of someone as Vice President in Charge of Quality will he disappointment and frustration. Accountability for quality belongs to top management. It can not he delegated. [...]
It is wrong to suppose that if you can't measure it, you can't manage it-a costly myth. [...]
If you can accomplish a [numerical quality/goal] goal without a [new] method, then why were you not doing it last year? There is only one possible answer: you were goofing off. [...]
The bulletin America 2000: An Educational Study, published by the Secretary of Education, Washington, 18 April 1991, provides a horrible example of numerical goals, tests, rewards, but no method. By what method? [...]
It is important that an aim never be defined in terms of a specific activity or method. [...]
This diagram [process flow diagram], as an organization chart, is far more meaningful than the usual pyramid. The pyramid only shows responsibilities for reporting, who reports to whom. It shows the chain of command and accountability. A pyramid does not describe the system of production. It does not tell anybody how his work fits into the work of other people in the company. If a pyramid conveys any message at all, it is that anybody should first and foremost try to satisfy his boss (get a good rating). The customer is not in the pyramid. [...]
Suppose that you tell me that my job is to wash this table. You show to me soap, water, and a brush. I still have no idea what the job is. I must know what the table will be used for after I wash it. Why wash it? Will the table be used to set food on? If so, it is clean enough now. If it is to be used for an operating table, I must wash the table several times with scalding water, top, bottom, and legs; also the floor below it and around it. [...]
The U.S. Postal Service is not a monopoly. The authorities of the postal service are hampered by Congress. If the U.S. Postal Service were a monopoly, there would be a chance of better service. [...]
The interpretation of results of a test or experiment is something else. It is prediction that a specific change in a process or procedure will be a wise choice, or that no change would be better. Either way the choice is prediction. This is known as an analytic problem, or a problem of inference, prediction. Tests of significance, t-test, chi-square, are useless as inference-i.e., useless for aid in prediction. Test of hypothesis has been for half a century a bristling obstruction to understanding statistical inference. [...]
Management is prediction. [...] Theory is a window into the world. Theory leads to prediction. Without prediction, experience and examples teach nothing. To copy an example of success, without understanding it with the aid of theory, may lead to disaster.
I could keep quoting the quotes. They are all about equally terrifying. Let me ruin any suspense you have might left: you aren't going to fix quality problems by doing the same thing you were doing, only better.
This book also quickly answered my other questions: yes, it is applicable to software development. No, westerners have still not figured it out. No, software will not get better if we just keep doing what we were doing. Yes, Japanese manufacturers knew this already in the 1980s and 1990s.
This post is getting a bit long already, so let's skip over more ranting and get to the point. First of all, I hope you read Deming's book(s). This one was very good, quite short, and easy to read, like Einstein's Relativity was easy to read. And you know it works, because his ideas were copied over and over, poorly, eventually into unrecognizability, ineffectiveness, and confusion, the way only the best ideas are.
Just for a taste, here's what I think are the two super important lessons. They may seem obvious in retrospect, but since I never really see anybody talking about them, I think they're not obvious at all.
The first lesson is: It's more important for output to have a continuous (usually Gaussian) distribution than to be within specifications. Western companies spend a lot of time manufacturing devices to be "within spec." If an output is out of spec, we have to throw it away and make another one, or at least fix it. If it's in spec, that's great, ship it! But... stop doing that. Forget about the spec, in fact. Real-life continuous processes do not have discontinuous yes/no breakpoints, and trying to create those requires you to destabilize and complexify your processes (often by adding an inspection or QA phase, which, surprisingly, turns out to be a very bad idea). I don't have time to explain it here, but Deming has some very, very strong opinions on the suckiness of post-manufacturing quality inspection.
The second lesson is: Know the difference between "special causes of variation" and "common causes of variation." Special causes produce outputs outside the continuous statistical distribution you were expecting. In my line of work, the continous variable might be Internet throughput, which varies over time, but is pretty predictable. A special cause would be, say, an outage, where you get 0 Mbps for a while before the outage is fixed. (On the other hand, if you're running an ISP, the pattern of your outages may actually be its own statistical distribution, and then you break it down further. Drunk drivers knocking down poles happens at a fairly predictable rate; accidentally deleting your authentication key on all the routers, causing a self-induced accidental outage is a special event.)
The reason it's so important to differentiate between special and common causes is that the way you respond to them is completely different. Special causes are usually caused by, and thus can be fixed or avoided by, low-level individual workers, in the case of manufacturing or repetitive work. In software (although Deming doesn't talk about software), special causes are weird bugs that only trigger in certain edge cases but cause major failure when they do. Special causes can often be eliminated by just fixing up your processes.
Common causes are simple variation inside your normal statistical distribution. It's tempting to try to fix these, especially when you're unlucky enough to get an outlier. Wifi throughput got really low for a few seconds, lower than two standard deviations fro the mean, isn't that weird? We got a surprisingly large number of customer service calls this week, much more than last week, what did we do wrong? And so on.
The great insight of Deming's methods is that you can (mostly) identify the difference between common and special causes mathematically, and that you should not attempt to fix common causes directly - it's a waste of time, because all real-life processes have random variation. (In fact, he provides a bunch of examples of how trying to fix common causes can make them much worse.)
Instead, what you want to do is identify your mean and standard deviation, plot the distribution, and try to clean it up. Rather than poking around at the weird edges of the distribution, can we adjust the mean left or right to get more output closer to what we want? Can we reduce the overall standard deviation - not any one outlier - by changing something fundamental about the process?
As part of that, you might find out that your process is actually not in control at all, and most of your problems are "special" causes. This means you're overdriving your process. For example, software developers working super long hours to meet a deadline might produce bursts of high producitivity followed by indeterminate periods of collapse (or they quit and the whole thing shuts down, or whatever). Running them at a reasonable rate might give worse short-term results, but more predictable results over time, and predictable results are where quality comes from.
My job nowadays is trying to make wifi better. This separation - which, like I said, sounds kind of obvious in retrospect - is a huge help to me. In wifi, the difference between "bugs that make the wifi stop working" and "variable performance in wifi transfer speeds or latency" is a big deal. The former is special causes, and the latter is common causes. You have to solve them differently. You probably should have two different groups of people thinking about solving them. Even further, just doing a single wifi speed test, or looking at the average or a particular percentile to see if it's "good enough" is not right. You want to measure the mean and stddev, and see which customers' wifi is weird (special causes) and which is normal (common causes).
Anyway, it's been a good vacation so far.
This is a common source of confusion: the legacy X.Org driver for touchpads is called xf86-input-synaptics but it is not a driver written by Synaptics, Inc. (the company).
The repository goes back to 2002 and for the first couple of years it Peter Osterlund was the sole contributor. Back then it was called "synaptics" and really was a "synaptics device" driver, i.e. it handled PS/2 protocol requests to initialise Synaptics, Inc. touchpads. Evdev support was added in 2003, punting the initialisation work to the kernel instead. This was the groundwork for a generic touchpad driver. In 2008 the driver was renamed to xf86-input-synaptics and relicensed from GPL to MIT to take it under the X.Org umbrella. I've been involved with it since 2008 and the official maintainer since 2011.
For many years now, the driver has been a generic touchpad driver that handles any device that the Linux kernel can handle. In fact, most bugs attributed to the synaptics driver not finding the touchpad are caused by the kernel not initialising the touchpad correctly. The synaptics driver reads the same evdev events that are also handled by libinput and the xf86-input-evdev driver, any differences in behaviour are driver-specific and not related to the hardware. The driver handles devices from Synaptics, Inc., ALPS, Elantech, Cypress, Apple and even some Wacom touch tablets. We don't care about what touchpad it is as long as the evdev events are sane.
Synaptics, Inc.'s developers are active in kernel development to help get new touchpads up and running. Once the kernel handles them, the xorg drivers and libinput will handle them too. I can't remember any significant contribution by Synaptics, Inc. to the X.org synaptics driver, so they are simply neither to credit nor to blame for the current state of the driver. The top 10 contributors since August 2008 when the first renamed version of xf86-input-synaptics was released are:
There's a long tail of other contributors but the top ten illustrate that it wasn't Synaptics, Inc. that wrote the driver. Any complaints about Synaptics, Inc. not maintaining/writing/fixing the driver are missing the point, because this driver was never a Synaptics, Inc. driver. That's not a criticism of Synaptics, Inc. btw, that's just how things are. We should have renamed the driver to just xf86-input-touchpad back in 2008 but that ship has sailed now. And synaptics is about to be superseded by libinput anyway, so it's simply not worth the effort now.
8 Simon Thum
10 Hans de Goede
10 Magnus Kessler
13 Alexandr Shadchin
15 Christoph Brill
18 Daniel Stone
18 Henrik Rydberg
39 Gaetan Nadon
50 Chase Douglas
396 Peter Hutterer
The other reason I included the commit count in the above: I'm also the main author of libinput. So "the synaptics developers" and "the libinput developers" are effectively the same person, i.e. me. Keep that in mind when you read random comments on the interwebs, it makes it easier to identify people just talking out of their behind.
A long-standing criticism of libinput is its touchpad acceleration code, oscillating somewhere between "terrible", "this is bad and you should feel bad" and "I can't complain because I keep missing the bloody send button". I finally found the time and some more laptops to sit down and figure out what's going on.
I recorded touch sequences of the following movements:
- super-slow: a very slow movement as you would do when pixel-precision is required. I recorded this by effectively slowly rolling my finger. This is an unusual but sometimes required interaction.
- slow: a slow movement as you would do when you need to hit a target several pixels across from a short distance away, e.g. the Firefox tab close button
- medium: a medium-speed movement though probably closer to the slow side. This would be similar to the movement when you move 5cm across the screen.
- medium-fast: a medium-to-fast speed movement. This would be similar to the movement when you move 5cm across the screen onto a large target, e.g. when moving between icons in the file manager.
- fast: a fast movement. This would be similar to the movement when you move between windows some distance apart.
- flick: a flick movement. This would be similar to the movement when you move to a corner of the screen.
Next, I ran this through a libinput 1.5.3 augmented with printfs in the pointer acceleration code and a script to post-process that output. Unfortunately, libinput's pointer acceleration internally uses units equivalent to a 1000dpi mouse and that's not something easy to understand. Either way, the numbers themselves don't matter too much for analysis right now and I've now switched everything to mm/s anyway.
A note ahead: the analysis relies on libinput recording an evemu replay. That relies on uinput and event timestamps are subject to a little bit of drift across recordings. Some differences in the before/after of the same recording can likely be blamed on that.
The graph I'll present for each recording is relatively simple, it shows the velocity and the matching factor.The x axis is simply the events in sequence, the y axes are the factor and the velocity (note: two different scales in one graph). And it colours in the bits that see some type of acceleration. Green means "maximum factor applied", yellow means "decelerated". The purple "adaptive" means per-velocity acceleration is applied. Anything that remains white is used as-is (aside from the constant deceleration). This isn't really different to the first graph, it just shows roughly the same data in different colours.
Interesting numbers for the factor are 0.4 and 0.8. We have a constant acceleration of 0.4 on touchpads, i.e. a factor of 0.4 "don't apply acceleration", the latter is "maximum factor". The maximum factor is twice as big as the normal factor, so the pointer moves twice as fast. Anything below 0.4 means we decelerate the pointer, i.e. the pointer moves slower than the finger.
The super-slow movement shows that the factor is, aside from the beginning always below 0.4, i.e. the sequence sees deceleration applied. The takeaway here is that acceleration appears to be doing the right thing, slow motion is decelerated and while there may or may not be some tweaking to do, there is no smoking gun.
The slow movement shows that the factor is almost always 0.4, aside from a few extremely slow events. This indicates that for the slow speed, the pointer movement maps exactly to the finger movement save for our constant deceleration. As above, there is no indicator that we're doing something seriously wrong.
The medium movement gets interesting. If we look at the factor applied, it changes wildly with the velocity across the whole range between 0.4 and the maximum 0.8. There is a short spike at the beginning where it maxes out but the rest is accelerated on-demand, i.e. different finger speeds will produce different acceleration. This shows the crux of what a lot of users have been complaining about - what is a fairly slow motion still results in an accelerated pointer. And because the acceleration changes with the speed the pointer behaviour is unpredictable.
The medium-fast movement shows almost the whole movement maxing out on the maximum acceleration factor, i.e. the pointer moves at twice the speed to the finger. This is a problem because this is roughly the speed you'd use to hit a "mentally preselected" target, i.e. you know exactly where the pointer should end up and you're just intuitively moving it there. If the pointer moves twice as fast, you're going to overshoot and indeed that's what I've observed during the touchpad tap analysis userstudy.
The fast movement shows basically the same thing, almost the whole sequence maxes out on the acceleration factor so the pointer will move twice as far as intuitively guessed.
So does the flick movement, but in that case we want it to go as far as possible and note that the speeds between fast and flick are virtually identical here. I'm not sure if that's me just being equally fast or the touchpad not quite picking up on the short motion.
Either way, the takeaway is simple: we accelerate too soon and there's a fairly narrow window where we have adaptive acceleration, it's very easy to top out. The simplest fix to get most touchpad movements working well is to increase the current threshold on when acceleration applies. Beyond that it's a bit harder to quantify, but a good idea seems to be to stretch out the acceleration function so that the factor changes at a slower rate as the velocity increases. And up the acceleration factor so we don't top out and we keep going as the finger goes faster. This would be the intuitive expectation since it resembles physics (more or less).
There's a set of patches on the list now that does exactly that. So let's see what the result of this is. Note ahead: I also switched everything from mm/s which causes some numbers to shift slightly.
The super-slow motion is largely unchanged though the velocity scale changes quite a bit. Part of that is that the new code has a different unit which, on my T440s, isn't exactly 1000dpi. So the numbers shift and the result of that is that deceleration applies a bit more often than before.
The slow motions are largely unchanged but more deceleration is now applied. Tbh, I'm not sure if that's an artefact of the evemu replay, the new accel code or the result of the not-quite-1000dpi of my touchpad.
The medium motion is the first interesting one because that's where we had the first observable issues. In the new code, the motion is almost entirely unaccelerated, i.e. the pointer will move as the finger does. Success!
The same is true of the medium-fast motion. In the recording the first few events were past the new thresholds so some acceleration is applied, the rest of the motion matches finger motion.
Medium-fast motion now matches the finger speed except at the beginning where some acceleration was applied.
The fast and flick motion are largely identical in having the acceleration factor applied to almost the whole motion but the big change is that the factor now goes up to 2.3 for the fast motion and 2.5 for the flick motion, i.e. both movements would go a lot faster than before. In the graphics below you still see the blue area marked as "previously max acceleration factor" though it does not actually max out in either recording now.
In summary, what this means is that the new code accelerates later but when it does accelerate, it goes faster. I tested this on a T440s, a T450p and an Asus VivoBook with an Elantech touchpad (which is almost unusable with current libinput). They don't quite feel the same yet and I'm not happy with the actual acceleration, but for 90% of 'normal' movements the touchpad now behaves very well. So at least we go from "this is terrible" to "this needs tweaking". I'll go check if there's any champagne left.
It’s now just a bit over a month since Election Day, and I’m starting to be seriously concerned about the possibility that the U.S. might become a one-party democracy.
Therefore this is an open letter to Democrats; the country needs you to get your act together. Yes, ideally I personally would prefer your place in the two-party Duverger equilibrium to be taken by the Libertarian Party, but there are practical reasons this is extremely unlikely to happen. The other minor parties are even more doomed. If the Republicans are going to have a counterpoise, it has to be you Democrats.
Donald Trump’s victory reads to me like a realignment election, a historic break with the way interest and demographic groups have behaved in the U.S. in my lifetime. Yet, Democrats, you so far seem to have learned nothing and forgotten nothing. Indeed, if I were Donald Trump I would be cackling with glee at your post-election behavior, which seems ideally calculated to lock Trump in for a second term before he has been sworn in for the first.
Stop this. Your country needs you. I’m not joking and I’m not concern-trolling. The wailing and the gnashing of teeth and the denial of reality have to end. In the rest of this essay I’m not going to talk about right and wrong and ideology, I’m going to talk about the brutal practical politics of what you have to do to climb out of the hole you are in.
We need to start with an unsparing assessment of that hole.
First, your ability to assemble a broad-based national coalition has collapsed. Do not be fooled into thinking otherwise by your popular vote “win”; that majority came entirely from the West Coast metroplex and disguises a large-scale collapse in popular support everywhere else in the U.S. Trump even achieved 30-40% support in blue states where he didn’t spend any money.
County-by-county psephological maps show that your base is now confined to two major coastal enclaves and a handful of university towns. Only 4 of 50 states have both a Democratic-controlled legislature and a Democratic governor. In 2018 that regionalization is going to get worse, not better; you will be defending 25 seats in areas where Trump took the popular vote, while the Republicans have to defend only 8 where Clinton won.
Your party leadership is geriatric, decades older than the average for their Republican counterparts. Years of steady losses at state level, masked by the personal popularity of Barack Obama, have left you without a bench to speak of – little young talent and basically no seasoned Presidential timber under retirement age. The fact that Joseph Biden, who will be 78 for the next Election Day, is being seriously mooted as the next Democratic candidate, speaks volumes – none of them good.
Your ideological lock on the elite media and show business has flipped from a powerful asset to a liability. Trump campaigned against that lock and won; his tactics can be and will be replicated. Worse, a self-created media bubble insulated you from grasping the actual concerns of the American public so completely that you didn’t realize the shit you were in until election night.
Your donor advantage didn’t help either. Clinton outspent Trump 2:1 and still lost.
Your “coalition of the ascendant” is sinking. Tell all the just-so stories you like, but the brute fact is that it failed to turn out to defeat the Republican candidate with the highest negatives in history. You thought all you had to do was wait for the old white men to die, but anybody who has studied the history of immigration in the U.S. could have told you that the political identities of immigrant ethnic groups do not remain stable as they assimilate. You weren’t going to own the Hispanics forever any more than you owned the Irish and the Italians forever. African-Americans, trained by decades of identity politics, simply failed to show up for a white candidate in the numbers you needed. The sexism card didn’t play either, as a bare majority of married women who actually went to the polls seem to have voted for Trump.
But your worst problem is less tangible. Trump has popped the preference-falsification bubble. The conservative majority in most of the U.S. (coastal enclaves excepted) now knows it’s a conservative majority. Before the election every pundit in sight pooh-poohed the idea that discouraged conservative voters, believing themselves isolated and powerless, had been sitting out several election cycles. But it turned out to be true, not least where I live in the swing state of Pennsylvania, where mid-state voters nobody knew were there put Trump over the top. Pretty much the same thing happened all through the Rust Belt.
That genie isn’t going to be stuffed back in the bottle. Those voters now know they can deliver the media and the coastal elites a gigantic fuck-you, and Republicans know the populist techniques to mobilize them to do that. Trump’s playbook was not exactly complicated.
Some Democrats are beginning to talk, tentatively, about reconnecting to the white working class. But your real problem is larger; you need to make the long journey back to the political center. Not the center you imagine exists, either; that’s an artifact of your media bubble. I’m pointing at the actual center revealed by psephological analysis of voter preferences.
That center is far to the right of what you would prefer. For that matter it is rather to the right of where I would prefer – but facts are facts and denying them isn’t going to help. You Democrats need to think about what it takes to be competitive on a continuum where Fox News is barely right of center, Mitt Romney was an out-of-touch liberal, and as near as I could tell the politician who actually nailed the psephological center in 2008 was none other than Sarah Palin.
If you do not do this thing, you will continue to lose.
Again, I emphasize that I am not issuing an ideological prescription here. I am not arguing in this essay that the present Democratic platform and strategy is wrong in an abstract moral sense, but rather that that it has become suicidal practical politics. Trump has dynamited almost every connection it had to winning elections, and smarter Republicans than Trump will take the lesson going forward.
Before I get to suggesting some changes, I want to point out that the results of the dominance Republicans have already achieved are going to make your problems even worse than they look now. Those problems don’t end with not having a farm team. State-level control means the Republicans will largely determine redistricting in the 2020 census. Their ability to pass voter-ID laws will surely hurt you as well.
I also need to point out that you shouldn’t count on Republican failure to save you. Yes, I know Democrats tell themselves Republican “hard right” policy actually implemented will alienate so many voters that they’ll come running back to your party. But you also thought Hillary was inevitable and how did that work out for ya? Trump’s popularity has risen as his program becomes clearer. You need to be positioned so that you can cope with outcomes other than catastrophic disenchantment with Trumpian populism.
So, what can you do?
The most obvious thing is that you have to stop contemptuously dismissing the largest single demographic segment of the American electorate. Because believe me, they noticed. So did their wives and children.
This has larger implications than you may yet understand. It’s not just that you need to take any Democrat who uses the phrase “angry white men” out to the woodshed and beat him or her with a strap until he/she wises up. The whole apparatus of racial and ethnic identity politics is turning in your hand, reversing (like your old-media dominance) from an asset to a liability.
(Just to drive the point home, the gender card doesn’t work any more either. Trump is a feminist’s worst nightmare. He won anyway. He came close enough to winning the entire female vote to trigger bitter post-election denunciations of American women in general by feminists – which pretty much epitomizes the sort of reaction that isn’t going to help you.)
Your best plausible case is that the minority groups you counted on passively fail to add up to a winning coalition in the future, as they did this cycle. Your worst – and increasingly likely – case is that white people now begin voting as something like an ethnic bloc. This is, after all, how you’ve been teaching other ethnic groups to play the game since the 1960s.
You will not prevent this development by screaming “racism!”. Here’s a hot tip: people you dismiss as retrograde scum will not, in general, vote for you. In fact, one of the things you Democrats most urgently need to do is banish “racism” and “sexism” from your political vocabulary.
While these words point at some real problems, they are also a trap. They lead you to organize your political pitch around virtue-signaling, exclusion and demonization. That, in turn, can be successful (though repulsive) politics when it’s used against a minority to mobilize a majority or plurality. But you’re in the opposite situation now. You were trapped by your own privilege theory. You demonized a plurality of American voters, and in return they gave you Trump.
If you continue to do this, you will lose.
It is irrelevant whether an actual plurality of American voters actually are as racist and sexist as you think. They don’t think they are, and they’re fed up with being hectored about it. This isn’t 1965, and your ability to tap into a substratum of guilt by white people who deep down know they were in the wrong is gone. What that same move brings up now is resentment.
Speaking of virtue signaling, another thing you need to give up is focusing on peacock issues (like, say, transgender rights) while ignoring pocketbook problems like the hollowing out of middle-class employment.
Again, this advice has nothing to do with the rights or wrongs of individual peacock issues and more with a general sense that the elites are fiddling while Rome burns. For the first time since records have been kept, U.S. life expectancy went down during the Obama years, led by a disturbing rise in suicides and opiate addiction among discouraged unemployed in flyover country. A Democratic Party that fails to address that while it screws around with bathroom-law boycotts is willfully consigning itself to irrelevance.
Many of Trump’s “pro-working-class” policies are objectively terrible; a new wave of trade protectionism is, for example, bound to have dire long-term consequences. But that doesn’t matter, in a political competitive sense, until you Democrats have something to answer him with.
Right now, you have nothing. You have less than nothing, because your instinctive solution repels the Trump plurality. They don’t want welfare, they want jobs and dignity and a modicum of respect. (And, just as a reminder, not to be dismissed as retrograde racists and sexists.)
Now we need to talk about guns.
This is a more particular issue than I’ve touched so far, but it’s one that cuts straight to the heart of the self-alienation of the Democratic Party from the political center.
Again, I’m not going to address the rights and wrongs of gun policy here, just its practical political ramifications. A quarter century ago Bill Clinton – who is as shrewd a practical politician as has ever operated in the U.S. – warned his fellow Democrats that pushing gun control was a sure way to lose more voters than it gained. They ignored his advice and got shellacked in the 1994 elections.
Today voter support for personal firearms rights is at an unprecedented high. This is revealed both in polls and in the wave of state-level liberalizations of concealed-carry laws. One of Trump’s most popular first-hundred-days promises is nationwide concealed-carry reciprocity. From the fact that gun control was slow party suicide in 1994 we can deduce that it’s even worse practical politics today.
And yet, the Democratic Party line is still hostile to gun rights, and less than six months ago its leaders and captive pundits were talking up Australian-style gun confiscation.
If you continue to do this, you will lose.
The Democratic line on gun policy is a perfect symbol of everything that has become disconnected about the party. It reads as corrosive disrespect for middle-Americans who like their firearms, think of themselves as a nation of armed citizens rather than cowering subjects, and use their guns responsibly. It reeks of class warfare, urban elites against flyover-country proles. It’s disempowering, not empowering. It is, in short, a perfect focus for anti-Democratic populist anger.
Here’s what I’ve been building up to:
You Democrats don’t just need to reform your gun policy, you need to reform your attitude towards the voters to a place from which your present policy looks as vicious and absurd as it does to them.
You Democrats don’t just need to reform your rhetoric about racism and sexism, you need to reform your attitude towards the voters to a place from which your present rhetoric looks as vicious and absurd as it does to them.
It’s all of a piece. You’ve forgotten how to be the party of the people. Trump was the price of that forgetfulness. Now, you need to relearn it, for all our sakes.
The alternative is that something like the Republicans, or possibly worse, dominates American politics for the foreseeable future. I don’t want that, and you should fear it more than I do.
So get your act together now.
A short while ago, I asked a bunch of people for long-term touchpad usage data (specifically: evemu recordings). I currently have 25 sets of data, the shortest of which has 9422 events, the longest of which has 987746 events. I requested that evemu-record was to be run in the background while people use their touchpad normally. Thus the data is quite messy, it contains taps, two-finger scrolling, edge scrolling, palm touches, etc. It's also raw data from the touchpad, not processed by libinput. Some care has to be taken with analysis, especially since it is weighted towards long recordings. In other words, the user with 987k events has a higher influence than the user with 9k events. So the data is useful for looking for patterns that can be independently verified with other data later. But it's also useful for disproving hypothesis, i.e. "we cannot do $foo because some users' events show $bla".
One of the things I've looked into was tapping. In libinput, a tap has two properties: a time threshold and a movement threshold. If the finger is held down longer than 180ms or it moves more than 3mm it is not a tap. These numbers are either taken from synaptics or just guesswork (both, probably). The need for a time-based threshold is obvious: we don't know whether the user is tapping until we see the finger up event. Only if that doesn't happen within a given time we know the user simply put the finger down. The movement threshold is required because small movements occur while tapping, caused by the finger really moving (e.g. when tapping shortly before/after a pointer motion) or by the finger center moving (as the finger flattens under pressure, the center may move a bit). Either way, these thresholds delay real pointer movement, making the pointer less reactive than it could be. So it's in our interest to have these thresholds low to get reactive pointer movement but as high as necessary to have reliable tap detection.
General data analysis
Let's look at the (messy) data. I wrote a script to calculate the time delta and movement distance for every single-touch sequence, i.e. anything with two or more fingers down was ignored. The script used a range of 250ms and 6mm of movement, discarding any sequences outside those thresholds. I also ignored anything in the left-most or right-most 10% because it's likely that anything that looks like a tap is a palm interaction . I ran the script against those files where the users reported that they use tapping (10 users) which gave me 6800 tap sequences. Note that the ranges are purposely larger than libinput's to detect if there was a significant amount of attempted taps that exceed the current thresholds and would be misdetected as non-taps.
Let's have a look at the results. First, a simple picture that merely prints the start location of each tap, normalised to the width/height of the touchpad. As you can see, taps are primarily clustered around the center but can really occur anywhere on the touchpad. This means any attempt at detecting taps by location would be unreliable.
The analysis of time is more interesting: There are spikes around the 50ms mark with quite a few outliers going towards 100ms forming what looks like a narrow normal distribution curve. The data points are overlaid with markers for the mean , the 50 percentile, the 90 percentile and the 95 percentile . And the data says: 95% of events fall below 116ms. That's something to go on.
The analysis of distance is similar: Most of the tap sequences have little to no movement, with 50% falling below 0.2mm of movement. Again the data points are overlaid with markers for the mean, the 50 percentile, the 90 percentile and the 95 percentile. And the data says: 95% of events fall below 1.8mm. Again, something to go on.
Now let's combine the two. Below is a graph mapping times and distances from touch sequences. In general, the longer the time, the longer the more movement we get but most of the data is in the bottom left. Since doing percentiles is tricky on 2 axes, I mapped the respective axes individually. The biggest rectangle is the 95th percentile for time and distance, the number below shows how many data points actually fall into this rectangle. Looks promising, we still have a vast majority of touchpoints fall into the respective 95 percentiles though the numbers are slightly lower than the individual axes suggest.
So based on the above, we learned one thing: it would not be reliable to detect taps based on their location. But we also suspect two things now: we can reduce the timeout and movement threshold without sacrificing a lot of reliability.
Verification of findings
Based on the above, our hypothesis is: we can reduce the timeout to 116ms and the threshold to 1.8mm while still having a 93% detection reliability. This is the most conservative reading, based on the extended thresholds.
To verify this, we needed to collect tap data from multiple users in a standardised and reproducible way. We wrote a basic website that displays 5 circles (see the screenshot below) on a canvas and asked a bunch of co-workers in two different offices  to tap them. While doing so, evemu-record was running in the background to capture the touchpad interactions. The touchpad was the one from a Lenovo T450 in both cases.
We asked each user three questions: "do you know what tapping/tap-to-click is?", "do you have tapping enabled" and "do you use it?". The answers are listed below:
- Do you know what tapping is? 33 yes, 12 no
- Do you have tapping enabled? 19 yes, 26 no
- Do you use tapping? 10 yes, 35 no
Ok, let's look at the data sets, same scripts as above:
What does this mean for users?
Based on the above, the libinput thresholds will be reduced to 100ms and 1.3mm. Let's see how we go with this and then we can increase it in the future if misdetection is higher than expected. Patches will on the wayland-devel list shortly.
For users that don't have tapping enabled, this will not change anything. All users who have tapping enabled will see a more responsive cursor on small movements as the time and distance thresholds have been significantly reduced. Some users may see a drop in tap detection rate. This is hopefully a subconscious enough effect that those users learn to tap faster or with less movement. If not, we have to look at it separately and see how we can deal with that.
If you find any issues with the analysis above, please let me know.
 These scripts analyse raw touchpad data, they don't benefit from libinput's palm detection
 Note: mean != average, the mean is less affected by strong outliers. look it up, it's worth knowing
 X percentile means X% of events fall below this value
 The Brisbane and Paris offices. No separate analysis was done, so it is unknown whether close proximity to baguettes has an effect to tap behaviour
 i.e. the effect of users learning how to use a system that doesn't work well out-of-the-box. This may result in e.g. quicker taps from those that are familiar with the system vs those that don't.
Recently, on a mailing list I frequent, one of the regulars uttered the following sentence: “I’m told Breitbart is the preferred news source for the ‘alt-right’ (KKK and neo-nazis)”.
That was a pretty glaring error, there.
I was interviewed on Breitbart Tech once. I visit the site occasionally. I am not affiliated with the alt-right, but I’ve been researching the recent claims about it. So I can supply some observations from the ground.
First, while I’m not entirely sure of everything the alt-right is (it’s a rather amorphous phenomenon) it is not the KKK and neo-Nazis. The most that can truthfully be said is that ‘alt-right’ serves as a recent flag of convenience to which some old-fashioned white supremacists are busily trying to attach themselves.
Also, the alt-right is not Donald Trump and his Trumpkins, either. He’s an equally old-fashioned populist continuous with Willam Jennings Bryan and Huey Long. If you tossed a bunch of alt-right memes at him, I doubt he’d even understand them, let alone agree.
The defining characteristic of the alt-right is, really, corrosive snarkiness. To the extent an origin can be identified, it was as a series of message-board pranks on 4chan. There’s no actual ideological core to it – it’s a kind of oppositional attitude-copping without a program, mordantly nasty but unserious.
There’s also some weird occultism attached – the half-serious cult of KEK, aka Pepe, who may or may not be an ancient Egyptian frog-god who speaks to his followers via numerological coincidences. (Donald Trump really wouldn’t get that part.)
Some elements of the alt-right are in fact racist (and misogynist, and homophobic, and other bad words) a la KKK/Nazi, but that’s not a defining characteristic and it’s anyway difficult to tell the genuine haters from those for whom posing as haters is a form of what 4chan types call “griefing”. That’s social disruption for the hell of it.
It is worth noting that another part of what is going on here is a visceral rejection of politically-correct leftism, one which deliberately inverts its premises. The griefers pose as racists and misogynists because they think it’s the most oppositional stance they can take to bullies and rage-mobbers who position themselves as anti-racists and feminists.
My sense is that the true haters are a tiny minority compared to the griefers and anti-PC rejectionists, but the griefers are entertained by others’ confusion on this score and don’t intend to clear it up.
Whether the alt-right even exists in any meaningful sense is questionable. To my anthropologist’s eye it has the aspect of a hoax (or a linked collection of hoaxes) being worked by 4chan griefers and handful of more visible provocateurs – Milo Yiannopolous, Mike Cernovich, Vox Day – who have noticed how readily the mainstream media buys inflated right-wing-conspiracy narratives and are working this one for the lulz. There’s no actual mass movement behind their posturing, unless you think a thousand or so basement-dwelling otaku are a mass movement.
I know Milo Yannopolous slightly – he is who interviewed me for Beitbart – and we have enough merry-prankster tendency in common that I think I get how his mind works. I’m certain that he, at any rate, is privately laughing his ass off at everyone who is going “alt-right BOOGA BOOGA!”
And there are a lot of such people. What these provocateurs are exploiting is media hysteria – the alt-right looms largest in the minds of self-panickers who project their fears on it. And of course in the minds of Hillary Clinton’s hangers-on, who would rather attribute her loss to a shadowy evil conspiracy than to a weak candidate and a plain-old bungled campaign.
I’m worried, however, that that the alt-right may not remain a loose-knit collection of hoaxes – that the self-panickers are actually creating what they fear.
For there is a deep vein of anti-establishment anger out there (see Donald Trump, election of). The alt-right (to the limited and conditional extent it now exists) could capture that anger, and its provocateurs are doing their best to make you think it already has, but they’re scamming you – they’re fucking with your head. The entire on-line ‘alt-right’ probably musters fewer people than the Trumpster’s last victory rally.
It’s a kind of dark-side Discordian hack in progress, and I’m concerned that it might succeed. Vox Day is trying to ideologize the alt-right, actually assemble something coherent from the hoaxes. He might succeed, or someone else might. Draw some comfort that it won’t be the Neo-Nazis or KKK – they’re real fanatics of the sort the alt-right defines itself by mocking. Mein Kampf and ironic nihilism don’t mix well.
The best way to beat the “alt-right” is not to overestimate it, not to feed it with your fear. If you keep doing that, the vast majority of the rootless and disaffected who have never heard of it might decide there’s a strong horse there and sign on.
Oh, and a coda about Breitbart: anyone who thinks Breitbart is far right needs to get out of their mainstream-media bubble more. Compared to sites like WorldNetDaily or FreeRepublic or TakiMag or even American Thinker, Breitbart is pretty mild stuff.
All those fake-news allegations against Breitbart are pretty rich coming from a media establishment that gave us Rathergate, “Hands up don’t shoot!”, and the “Jackie” false-rape story and was quite recently exchanging coordination emails with the Clinton campaign. Breitbart isn’t any more propagandistic than CBS or Rolling Stone, it’s just differently propagandistic.
xinput is a tool to query and modify X input device properties (amongst other things). Every so-often someone-complains about it's non-intuitive interface, but this is where users are mistaken: xinput is a not a configuration UI. It is a DUI - a developer user interface  - intended to test things without having to write custom (more user-friendly) for each new property. It is nothing but a tool to access what is effectively a key-value store. To use it you need to know not only the key name(s) but also the allowed formats, some of which are only documented in header files. It is intended to be run under user supervision, anything it does won't survive device hotplugging. Relying on xinput for configuration is the same as relying on 'echo' to toggle parameters in /sys for kernel configuration. It kinda possibly maybe works most of the time but it's not pretty. And it's not intended to be, so please don't complain to me about the arcane user interface.
 don't do it, things will be a bit confusing, you may not do the right thing, you can easily do damage, etc. A lot of similarities...
This post mostly affects developers of desktop environments/Wayland compositors. A systemd pull request was merged to add two new properties to some keyboards: XKB_FIXED_LAYOUT and XKB_FIXED_VARIANT. If set, the device must not be switched to a user-configured layout but rather the one set in the properties. This is required to make fake keyboard devices work correctly out-of-the-box. For example, Yubikeys emulate a keyboard and send the configured passwords as key codes matching a US keyboard layout. If a different layout is applied, then the password may get mangled by the client.
Since udev and libinput are sitting below the keyboard layout there isn't much we can do in this layer. This is a job for those parts that handle keyboard layouts and layout configurations, i.e. GNOME, KDE, etc. I've filed a bug for gnome here, please do so for your desktop environment.
If you have a device that falls into this category, please submit a systemd patch/file a bug and cc me on it (@whot).
This post applies to most tools that interface with the X server and change settings in the server, including xinput, xmodmap, setxkbmap, xkbcomp, xrandr, xsetwacom and other tools that start with x. The one word to sum up the future for these tools under Wayland is: "non-functional".
An X window manager is little more than an innocent bystander when it comes to anything input-related. Short of handling global shortcuts and intercepting some mouse button presses (to bring the clicked window to the front) there is very little a window manager can do. It's a separate process to the X server and does not receive most input events and it cannot affect what events are being generated. When it comes to input device configuration, any X client can tell the server to change it - that's why general debugging tools like xinput work.
A Wayland compositor is much more, it is a window manager and the display server merged into one process. This gives the compositor a lot more power and responsibility. It handles all input events as they come out of libinput and also manages device's configuration. Oh, and instead of the X protocol it speaks Wayland protocol.
The difference becomes more obvious when you consider what happens when you toggle a setting in the GNOME control center. In both Wayland and X, the control center toggles a gsettings key and waits for some other process to pick it up. In both cases, mutter gets notified about the change but what happens then is quite different. In GNOME(X), mutter tells the X server to change a device property, the server passes that on to the xf86-input-libinput driver and from there the setting is toggled in libinput. In GNOME(Wayland), mutter toggles the setting directly in libinput.
Since there is no X server in the stack, the various tools can't talk to it. So to get the tools to work they would have to talk to the compositor instead. But they only know how to speak X protocol, and no Wayland protocol extension exists for input device configuration. Such a Wayland protocol extension would most likely have to be a private one since the various compositors expose device configuration in different ways. Whether this extension will be written and added to compositors is uncertain, I'm not aware of any plans or even intentions to do so (it's a very messy problem). But either way, until it exists, the tools will merely shout into the void, without even an echo to keep them entertained. Non-functional is thus a good summary.
pastebins are useful for dumping large data sets whenever the medium of conversation doesn't make this easy or useful. IRC is one example, or audio/video conferencing. But pastebins only work when the other side looks at the pastebin before it expires, and the default expiry date for a pastebin may only be a few days.
This makes them effectively useless for bugs where it may take a while for the bug to be triaged and the assignee to respond. It may take even longer to figure out the source of the bug, and if there's a regression it can take months to figure it out. Once the content disappears we have to re-request the data from the reporter. And there is a vicious dependency too: usually, logs are more important for difficult bugs. Difficult bugs take longer to fix. Thus, with pastebins, the more difficult the bug, the more likely the logs become unavailable.
All useful bug tracking systems have an attachment facility. Use that instead, it's archived with the bug and if a year later we notice a regression, we still have access to the data.
If you got here because I pasted the link to this blog post, please do the following: download the pastebin content as raw text, then add it as attachment to the bug (don't paste it as comment). Once that's done, we can have a look at your bug again.
I’ve been pretty quiet lately, other than short posts on G+, because I’ve been grinding hard on NTPsec. We’re coming up on a 1.0 release and, although things are going very well technically, it’s been a shit-ton of work.
One consequence is the NTPsec Project Blog. My first major post there expands on some of the things I’ve written here about stripping crap out of the NTP codebase.
Expect future posts on spinoff tools, the NTPsec test farm, and the prospects for moving NTPsec out of C, probably about one a week. I have a couple of these in draft already.
I pushed the patch to require resolution today, expect this to hit the general public with libinput 1.6. If your graphics tablet does not provide axis resolution we will need to add a hwdb entry. Please file a bug in systemd and CC me on it (@whot).
How do you know if your device has resolution? Run sudo evemu-describe against the device node and look for the ABS_X/ABS_Y entries:
if the Resolution value is 0 you'll need a hwdb entry or your tablet will stop working in libinput 1.6. You can file the bug now and we can get it fixed, that way it'll be in place once 1.6 comes out.
# Event code 0 (ABS_X)
# Value 2550
# Min 0
# Max 3968
# Fuzz 0
# Flat 0
# Resolution 13
# Event code 1 (ABS_Y)
# Value 1323
# Min 0
# Max 2240
# Fuzz 0
# Flat 0
# Resolution 13
The Fedora Change to retire the synaptics driver was approved by FESCO. This will apply to Fedora 26 and is part of a cleanup to, ironically, make the synaptics driver easier to install.
Since Fedora 22, xorg-x11-drv-libinput is the preferred input driver. For historical reasons, almost all users have the xorg-x11-drv-synaptics package installed. But to actually use the synaptics driver over xorg-x11-drv-libinput requires a manually dropped xorg.conf.d snippet. And that's just not ideal. Unfortunately, in DNF/RPM we cannot just say "replace the xorg-x11-drv-synaptics package with xorg-x11-drv-libinput on update but still allow users to install xorg-x11-drv-synaptics after that".
So the path taken is a package rename. Starting with Fedora 26, xorg-x11-drv-libinput's RPM will Provide/Obsolete  xorg-x11-drv-synaptics and thus remove the old package on update. Users that need the synaptics driver then need to install xorg-x11-drv-synaptics-legacy. This driver will then install itself correctly without extra user intervention and will take precedence over the libinput driver. Removing xorg-x11-drv-synaptics-legacy will remove the driver assignment and thus fall back to libinput for touchpads. So aside from the name change, everything else works smoother now. Both packages are now updated in Rawhide and should be available from your local mirror soon.
What does this mean for you as a user? If you are a synaptics user, after an update/install, you need to now manually install xorg-x11-drv-synaptics-legacy. You can remove any xorg.conf.d snippets assigning the synaptics driver unless they also include other custom configuration.
 "Provide" in RPM-speak means the package provides functionality otherwise provided by some other package even though it may not necessarily provide the code from that package. "Obsolete" means that installing this package replaces the obsoleted package.
[ This blog was crossposted on Software Freedom Conservancy's website. ]
As I mentioned in an earlier blog post, I had the privilege of attending Embedded Linux Conference Europe (ELC EU) and the OpenWrt Summit in Berlin, Germany earlier this month. I gave a talk (for which the video is available below) at the OpenWrt Summit. I also had the opportunity to host the first of many conference sessions seeking feedback and input from the Linux developer community about Conservancy's GPL Compliance Project for Linux Developers.
ELC EU has no “BoF Board” where you can post informal sessions. So, we scheduled the session by word of mouth over a lunch hour. We nevertheless got an good turnout (given that our session's main competition was eating food of about 15 people.
Most notably and excitingly, Harald Welte, well-known Netfilter developer and leader of gpl-violations.org, was able to attend. Harald talked about his work with gpl-violations.org enforcing his own copyrights in Linux, and explained why this was important work for users of the violating devices. He also pointed out that some of the companies that were sued during his most active period of gpl-violations.org are now regular upstream contributors.
Two people who work in the for-profit license compliance industry attended
as well. Some of the discussion focused on usual debates that charities
involved in compliance commonly have with the for-profit compliance
industry. Specifically, one of them asked
how much compliance is
enough, by percentage? I responded to his question on two axes.
First, I addressed the axis of
how many enforcement matters does the GPL
Compliance Program for Linux Developers do, by percentage of products
violating the GPL? There are, at any given time, hundreds of
documented GPL violating products, and our coalition works on only a tiny
percentage of those per year. It's a sad fact that only that tiny
percentage of the products that violate Linux are actually pursued to
On the other axis, I discussed the percentage on a per-product basis.
From that point of view, the question is really:
Is there a ‘close
enough to compliance’ that we can as a community accept and forget
about the remainder? From my point of view, we frequently compromise
anyway, since the GPL doesn't require someone to prepare code properly for
upstream contribution. Thus, we all often accept compliance once someone
completes the bare minimum of obligations literally written in the GPL, but
give us a source release that cannot easily be converted to an upstream
contribution. So, from that point of view, we're often accepting a
less-than-optimal outcome. The GPL by itself does not inspire upstreaming;
the other collaboration techniques that are enabled in our community
because of the GPL work to finish that job, and adherence to
the Principles assures
that process can work. Having many people who work with companies in
different ways assures that as a larger community, we try all the different
strategies to encourage participation, and inspire today's violators to
become tomorrow upstream contributors — as Harald mention has already
That same axis does include on rare but important compliance problem: when a violator is particularly savvy, and refuses to release very specific parts of their Linux code (as VMware did), even though the license requires it. In those cases, we certainly cannot and should not accept anything less than required compliance — lest companies begin holding back all the most interesting parts of the code that GPL requires them to produce. If that happened, the GPL would cease to function correctly for Linux.
After that part of the discussion, we turned to considerations of corporate contributors, and how they responded to enforcement. Wolfram Sang, one of the developers in Conservancy's coalition, spoke up on this point. He expressed that the focus on for-profit company contributions, and the achievements of those companies, seemed unduly prioritized by some in the community. As an independent contractor and individual developer, Wolfram believes that contributions from people like him are essential to a diverse developer base, that their opinions should be taken into account, and their achievements respected.
I found Wolfram's points particularly salient. My view is that Free Software development, including for Linux, succeeds because both powerful and wealthy entities and individuals contribute and collaborate together on equal footing. While companies have typically only enforce the GPL on their own copyrights for business reasons (e.g., there is at least one example of a major Linux-contributing company using GPL enforcement merely as a counter-punch in a patent lawsuit), individual developers who join Conservancy's coalition follow community principles and enforce to defend the rights of their users.
At the end of the session, I asked two developers who hadn't spoken during
the session, and who aren't members of Conservancy's coalition, their
opinion on how enforcement was historically carried out by
gpl-violations.org, and how it is currently carried out by Conservancy's
GPL Compliance Program for Linux Developers. Both responded with a simple
it seems like a good thing to do; keep doing
I finished up the session by inviting everyone to the join the principles-discuss list, where public discussion about GPL enforcement under the Principles has already begun. I also invited everyone to attend my talk, that took place an hour later at the OpenWrt Summit, which was co-located with ELC EU.
In that talk, I spoke about a specific example of community success in GPL enforcement. As explained on the OpenWrt history page, OpenWrt was initially made possible thanks to GPL enforcement done by BusyBox and Linux contributors in a coalition together. (Those who want to hear more about the connection between GPL enforcement and OpenWrt can view my talk.)
Since there weren't opportunities to promote impromptu sessions on-site, this event was a low-key (but still quite nice) start to Conservancy's planned year-long effort seeking feedback about GPL compliance and enforcement. Our next session is an official BoF session at Linux Plumbers Conference, scheduled for next Thursday 3 November at 18:00. It will be led by my colleagues Karen Sandler and Brett Smith.