The lower-post-volume people behind the software in Debian. (List of feeds.)

This is an update on our efforts to upgrade the TLS stack in Mono.

You can skip to the summary at the end if you do not care about the sausage making details.

Currently, TLS is surfaced in a few places in the .NET APIs:

  • By the SslStream class, which is a general purpose class that can be used to turn any bidirectional stream into an TLS-powered stream. This class is what currently powers the web client in Mono.
  • By the HttpWebRequest class, which provides .NET's HTTP client. This in turn is the foundation for the modern HttpClient, both the WCF and WebServices stacks as well as the quick and dirty WebClient API.

HttpClient is in particular interesting, as it allows for different transports to be provided for it. The default implementation in .NET 4.5 and Mono today is to use an HttpWebRequest-based implementation. But on Windows 10, the implementation is replaced with one that uses WinRT's HTTP client.

Microsoft is encouraging developers to abandon HttpWebRequest and instead adopt HttpClient as it both async-friendly and can use the best available transport given on a specific platform. More on this in a second.

Mono's Managed TLS

Mono currently only supports TLS 1.0.

This is the stack that powers SslStream and HttpWebRequest.

Last year we started an effort to bring managed implementations of TLS 1.2 and TLS 1.1. Given how serious security has become and how many holes have been found in existing implementation, we built this with an extensive test suite to check for conformance and to avoid common exploits found in implementation mistakes of TLS. This effort is currently under development and you can see where it currently lives at mono-tls module.

This will give us complete TLS support for the entire stack, but this work is still going to take a few months to audit.

Platform Specific HttpClients

Most of the uses for TLS today is via the HTTP protocol, and not over custom TLS streams. This means that it is more important to get an HTTP client that supports a brand new TLS stack, than it is to provide the SslStream code.

We want to provide native HttpClient handlers for all of Mono's supported platforms: Android, iOS, Mac, Linux, BSD, Unix and Windows.

On iOS: Today Xamarin.iOS already ships a native handler, the CFNetworkHandler. This one is powered by Apple's CFNetwork stack. In recent years, Apple has improved their networking stack, and we now I strongly recommend using Paul Bett's fantastic ModernHttpClient which uses iOS' brand new NSUrlSession and uses OkHttp on Android.

On Android: in the short term, we recommend adopting ModernHttpClient from Paul Betts (bonus points: the same component works on iOS with no changes). In the long term, we will change the default handler to use the Android Java client.

In both cases, you end up with HTTP 2.0 capable clients for free.

But this still leaves Linux, Windows and other assorted operating systems without a regular transport.

For those platforms, we will be adopting the CoreFX handlers, which on Unix are powered by the libcurl library.

This still leaves HttpWebRequest and everything built on top of it running on top of our TLS stack.

Bringing Microsoft's SslStream and HttpWebRequest to Mono

While this is not really TLS related, we wanted to bring Microsoft's implementations of those two classes to Mono, as they would fix many odd corner cases in the API, and address limitations in our stack that do not exist in Microsoft's implementation.

But the code is tightly coupled to native Windows APIs which makes the adoption of this code difficult.

We have built an adaptation layer that will allow us to bring Microsoft's code and use Mono's Managed TLS implementation.

SslStream backends

Our original effort focused on a pure managed implementation of TLS because we want to ensure that the TLS stack would work on all available platforms in the same way. This also means that all of the .NET code that expects to control every knob of your secure connection to work (pinning certificates or validating your own chains for example).

That said, in many cases developers do not need this capabilities, and in fact, on Xamarin.iOS, we can not even provide the functionality, as the OS does not give users access to the certificate chains.

So we are going to be developing at least two separate SslStream implementations. For Apple systems, we will be implementing a version on top of Apple's SSL stack, and for other systems we will be developing an implementation on top of Amazon's new SSL library, or the popular OpenSSL variant of the day.

These have the advantage that we would not need to maintain the code, and we benefit from third parties doing all the hard security work and will be suitable for most uses.

For those rare uses that like to handle connections manually, you will have to wait for Mono's new TLS implementation to land.

In Summary

Android, Mac and iOS users can get the latest TLS for HTTP workloads using ModernHttpClient. Mac/iOS users can use the built-in CFNetworkHandler as well.

Soon: OpenSSL/AppleSSL based transports to be available in Mono (post Mono 4.2).

Soon: Advanced .NET SSL use case scenarios will be supported with Mono's new mono-tls stack

Soon: HttpWebRequest and SslStream stacks will be replaced in Mono with Microsoft's implementations.

Posted Thu Aug 27 16:32:32 2015 Tags:

When I built the Great Beast of Malvern, it was intended for surgery on large repositories. The specific aim in view was to support converting the NetBSD CVS to git, but that project is stalled because the political process around NetBSD’s decision about when to move seems to have seized up. I’ve got the hardware and software ready when they’re ready to move.

Now I have another repo-conversion job in the offing – and it does something I thought I’d never see. The working set exceeds 32GB! For comparison, the working set of the entire NetBSD conversion tops out at about 18GB.

What, you might well ask, can possibly have a history that huge? And the answer is…GCC. Yes, they’re looking to move from Subversion to git. And this is clearly a job for the Great Beast, once the additional 32GB I just ordered from Newegg arrives.

Posted Tue Aug 25 23:25:54 2015 Tags:

Sometimes, the best encouragement you can get in a martial-arts class is silence.

Once a month my school, which normally teaches a combination of wing chun kung fu and Philippine blade/stick fighting, gets a visit from Sifu Jerry Devone, who teaches pure traditional Wing Chun at a level a bit higher than our Sifu Dale Yeager.

Sifu Jerry is a nice guy, but it’s not difficult to find videos of him almost casually destroying other kung fu players in ring fights. He shows the same soft/hard combination as an instructor – never yells at anyone, but demands precision and perfection and often gets it, even from unpromising students.

Tonight he told the four senior students (including Cathy and myself) to line up facing him and do the Siu Nim Tao form with him watching for defects. From some instructors this would be a terrifying prospect, with anticipation of a humiliating ass-chewing to follow. While Sifu Dale wouldn’t exactly humiliate someone who screwed up, he might make snarky theatrical jokes about bad performance in a half-laughing-with, half-laughing-at manner. Neither of these is Sifu Jerry’s style – he’d just quietly correct in a way that would make you grimly determined to get it right next time.

Still, I felt rather stressed. I know the motions of Siu Nim Tao – it’s not a complex form, and it doesn’t require anything I’m bad at like high kicking – but it’s subtle. There are fine details in it, and the devil is in those details, and in getting the overall flow and timing right.

You have some subtle stylistic choices about how to do forms, and one of them is power vs. grace. At the power extreme you do it in crisp, hard motions that deliver power in each movement. At the grace extreme you do it as a continuous flow with beautiful motions and smooth, inevitable transitions. The failure more of power is to be abrupt, herky-jerky and disconnected; the failure mode of grace is to be pretty, limp, and pointless.

What’s tricky is that there is variation among instructors among what power-to-grace ratio they actually want. It’s not even a constant; instructors may want slow and graceful one day, to isolate a motion they want to improve you on, but fast and powerful the next day to emphasize a combat application.

I wasn’t really worried about grossly screwing up a move, I was worried about doing Siu Nim Tao with good energy and style – and I was especially worried that the way I find natural to do it, which is definitely on the power side, might not be right. Or, anyway, not what Sifu Jerry wanted.

I suppressed my misgivings and did my Siu Nim Tao – crisp, snappy, a little fast, real power in the punches and cocked-wrist strikes and the one blade-hand move near the end meant to break up a wrist-grab. We all went through it, three or four times, with Sifu Jerry strolling back and forth in front of the line watching us narrowly.

Five or six minutes later, three of the four students had received minor corrections of their form. Amazingly, none of them was me!.

I got it right! While I managed to refrain from grinning like a fool, I felt almost absurdly triumphant. Especially when Sifu gave a mini-lecture on doing the form moves with definition, like you’re using them in a fight. That lecture wasn’t meant for me – I might make other kinds of mistakes, but I hadn’t fluffed that one and it’s unlikely I will.

Of such quiet victories, built up over years, are expertise and confidence made.

Posted Fri Aug 21 04:08:49 2015 Tags:

A couple of stories by Charles Babcock and (my coincidentally old friend) Steven J. Vaughan-Nichols have mentioned the existence of an ‘NTPsec’ project being funded by the Core Infrastructure Initiative as an alternative and perhaps eventual replacement for the reference implementation of Network Time Protocol maintained by Harlan Stenn and the Network Time Foundation.

I confirm that NTPsec does exist, and that I am deeply involved in it.

The project has not yet officially released code, though you can view a preliminary web page at ntpsec.org. For various complicated political reasons a full public discussion of the project’s genesis and goals should wait until we go fully public. You probably won’t have to wait long for this.

I can, however, disclose several facts that I think will be of interest to readers of this blog…

NTPSec is a fork of the Mills implementation of NTP (which we think of as “NTP Classic”). Early major objectives include security hardening, removal of the pre-POSIX legacy cruft that presently makes NTP Classic hard to audit and maintain, and building a really good test suite so the suite can demonstrate its correctness.

I am deeply involved, and have been working hard on this project behind the scenes for about eight months (this in part accounts for the light blogging recently). I am the architecture/technology lead, and presently the most active coder. A&D regular Susan Sons (aka HedgeMage) is also on the team, and in fact recruited me onto it.

Some team members (including me) are being paid to work full-time on this project. More may be hired. For that to happen, more funding commitments have to land. And probably will land; we’re hearing a lot of clamor from industry for a better-maintained, more secure NTP and have been pressed to release somewhat sooner than I would prefer.

I do expect this to have some negative impact on the amount of time I spend on other projects. But one of the reasons I took the gig is that GPSD is now sufficiently mature and stable not to need large amounts of my time. And time service is really, really important.

There is enough technical work on this project to which I am near-ideally suited to keep it top of my list for a minimimum of 2.5 to 3 years. That’s even if I don’t end up designing the next generation NTP protocol, an outcome I now consider to have over 50% odds.

Those of you guessing that my recent work on improving and documenting time service for GPSD led into this project are of course correct. But there’s more to it than that; it turns out that NTP daemons have a remarkably large amount in common with gpsd. Both are network-aware device managers, with closely comparable KLOC scale and porting challenges. They share arcana like dealing with 1PPS signals, and quite a bit of specialized knowledge just maps right across.

Another aspect of my skills profile that fits me well for the project is knowledge of ancient pre-standardization Unix APIs acquired over four decades. NTP is as full of this stuff as GPSD used to be before I removed it all several years back, and one of the principal tasks is to remove the cruft from NTP in order to reduce code volume and attack surface. We have already ripped out approximately 17% (40KLOC) of the NTP Classic codebase.

Finally, let me note that this code is not really living up to its reputation for impenetrability. There’s a longstanding legend that only Dave Mills ever really understood the Byzantine time-synchronization algorithms at NTP’s heart, but I used to be a mathematician and I think I already get most of it outside of a few arcana about statistical filtering of noisy signals. And most of the code isn’t those Byzantine algorithms anyway, but rather the not terribly surprising plumbing around them. Modifying it is high-end systems programming and not for the faint of heart, to be sure, but it’s not a thesis research project.

I think any top-grade systems engineer with a solid background in applied math or physics could grok NTP, really. Either that or, as I joked on G+, I actually have “read ancient code” as a minor superpower. Which joke I report mainly because I think Dave Taht was much funnier when he figuratively raised a Spock-like eyebrow back at me and said “Minor?”

Posted Wed Aug 19 00:43:47 2015 Tags:

Thanks to edmundedgar on reddit I have some more accurate data to update my previous bandwidth growth estimation post: OFCOM UK, who released their November 2014 report on average broadband speeds.  Whereas Akamai numbers could be lowered by the increase in mobile connections, this directly measures actual broadband speeds.

Extracting the figures gives:

  1. Average download speed in November 2008 was 3.6Mbit
  2. Average download speed in November 2014 was 22.8Mbit
  3. Average upload speed in November 2014 was 2.9Mbit
  4. Average upload speed in November 2008 to April 2009 was 0.43Mbit/s

So in 6 years, downloads went up by 6.333 times, and uploads went up by 6.75 times.  That’s an annual increase of 36% for downloads and 37% for uploads; that’s good, as it implies we can use download speed factor increases as a proxy for upload speed increases (as upload speed is just as important for a peer-to-peer network).

This compares with my previous post’s Akamai’s UK numbers of 3.526Mbit in Q4 2008 and 10.874Mbit in Q4 2014: only a factor of 3.08 (26% per annum).  Given how close Akamai’s numbers were to OFCOM’s in November 2008 (a year after the iPhone UK release, but probably too early for mobile to have significant effect), it’s reasonable to assume that mobile plays a large part of this difference.

If we assume Akamai’s numbers reflected real broadband rates prior to November 2008, we can also use it to extend the OFCOM data back a year: this is important since there was almost no bandwidth growth according to Akamai from Q4 2007 to Q7 2008: ignoring that period gives a rosier picture than my last post, and smells of cherrypicking data.

So, let’s say the UK went from 3.265Mbit in Q4 2007 (Akamai numbers) to 22.8Mbit in Q4 2014 (OFCOM numbers).  That’s a factor of 6.98, or 32% increase per annum for the UK. If we assume that the US Akamai data is under-representing Q4 2014 speeds by the same factor (6.333 / 3.08 = 2.056) as the UK data, that implies the US went from 3.644Mbit in Q4 2007 to 11.061 * 2.056 = 22.74Mbit in Q4 2014, giving a factor of 6.24, or 30% increase per annum for the US.

As stated previously, China is now where the US and UK were 7 years ago, suggesting they’re a reasonable model for future growth for that region.  Thus I revise my bandwidth estimates; instead of 17% per annum this suggests 30% per annum as a reasonable growth rate.

Posted Sat Aug 15 04:54:38 2015 Tags:

There’s a significant debate going on at the moment in the Bitcoin world; there’s a great deal of information and misinformation, and it’s hard to find a cogent summary in one place.  This post is my attempt, though I already know that it will cause me even more trouble than that time I foolishly entitled a post “If you didn’t run code written by assholes, your machine wouldn’t boot”.

The Technical Background: 1MB Block Limit

The bitcoin protocol is powered by miners, who gather transactions into blocks, producing a block every 10 minutes (but it varies a lot).  They get a 25 bitcoin subsidy for this, plus whatever fees are paid by those transactions.  This subsidy halves every 4 years: in about 12 months it will drop to 12.5.

Full nodes on the network check transactions and blocks, and relay them to others.  There are also lightweight nodes which simply listen for transactions which affect them, and trust that blocks from miners are generally OK.

A normal transaction is 250 bytes, and there’s a hard-coded 1 megabyte limit on the block size.  This limit was introduced years ago as a quick way of avoiding a miner flooding the young network, though the original code could only produce 200kb blocks, and the default reference code still defaults to a 750kb limit.

In the last few months there have been increasing runs of full blocks, causing backlogs for a few hours.  More recently, someone deliberately flooded the network with normal-fee transactions for several days; any transactions paying less fees than those had to wait for hours to be processed.

There are 5 people who have commit access to the bitcoin reference implementation (aka. “bitcoin-core”), and they vary significantly in their concerns on the issue.

The Bitcoin Users’ Perspective

From the bitcoin users perspective, blocks should be infinite, and fees zero or minimal.  This is the basic position of respected (but non-bitcoin-core) developer Mike Hearn, and has support from bitcoin-core ex-lead Gavin Andresen.  They work on the wallet and end-user side of bitcoin, and they see the issue as the most urgent.  In an excellent post arguing why growth is so important, Mike raises the following points, which I’ve paraphrased:

  1. Currencies have network effects. A currency that has few users is simply not competitive with currencies that have many.
  2. A decentralised currency that the vast majority can’t use doesn’t change the amount of centralisation in the world. Most people will still end up using banks, with all the normal problems.
  3. Growth is a part of the social contract. It always has been.
  4. Businesses will only continue to invest in bitcoin and build infrastructure if they are assured that the market will grow significantly.
  5. Bitcoin needs users, lots of them, for its political survival. There are many people out there who would like to see digital cash disappear, or be regulated out of existence.

At this point, it’s worth mentioning another bitcoin-core developer: Jeff Garzik.  He believes that the bitcoin userbase has been promised that transactions will continue to be almost free.  When a request to change the default mining limit from 750kb to 1M was closed by the bitcoin lead developer Wladimir van der Laan as unimportant, Jeff saw this as a symbolic moment:

What Happens If We Don’t Increase Soon?

Mike Hearn has a fairly apocalyptic view of what would happen if blocks fill.  That was certainly looking likely when the post was written, but due to episodes where the blocks were full for days, wallet designers are (finally) starting to estimate fees for timely processing (miners process larger fee transactions first).  Some wallets and services didn’t even have a way to change the setting, leaving users stranded during high-volume events.

It now seems that the bursts of full blocks will arrive with increasing frequency; proposals are fairly mature now to allow users to post-increase fees if required, which (if all goes well) could make for a fairly smooth transition from the current “fees are tiny and optional” mode of operation to a “there will be a small fee”.

But even if this rosy scenario is true, this begsavoids the bigger question of how high fees can become before bitcoin becomes useless.  1c?  5c?  20c? $1?

So What Are The Problems With Increasing The Blocksize?

In a word, the problem is miners.  As mining has transitioned from a geek pastime, semi-hobbyist, then to large operations with cheap access to power, it has become more concentrated.

The only difference between bitcoin and previous cryptocurrencies is that instead of a centralized “broker” to ensure honesty, bitcoin uses an open competition of miners. Given bitcoin’s endurance, it’s fair to count this a vital property of bitcoin.  Mining centralization is the long-term concern of another bitcoin-core developer (and my coworker at Blockstream), Gregory Maxwell.

Control over half the block-producing power and you control who can use bitcoin and cheat anyone not using a full node themselves.  Control over 2/3, and you can force a rule change on the rest of the network by stalling it until enough people give in.  Central control is also a single point to shut the network down; that lets others apply legal or extra-legal pressure to restrict the network.

What Drives Centralization?

Bitcoin mining is more efficient at scale. That was to be expected[7]. However, the concentration has come much faster than expected because of the invention of mining pools.  These pools tell miners what to mine, in return for a small (or in some cases, zero) share of profits.  It saves setup costs, they’re easy to use, and miners get more regular payouts.  This has caused bitcoin to reel from one centralization crisis to another over the last few years; the decline in full nodes has been precipitous by some measures[5] and continues to decline[6].

Consider the plight of a miner whose network is further away from most other miners.  They find out about new blocks later, and their blocks get built on later.  Both these effects cause them to create blocks which the network ignores, called orphans.  Some orphans are the inevitable consequence of miners racing for the same prize, but the orphan problem is not symmetrical.  Being well connected to the other miners helps, but there’s a second effect: if you discover the previous block, you’ve a head-start on the next one.  This means a pool which has 20% of the hashing power doesn’t have to worry about delays at all 20% of the time.

If the orphan rate is very low (say, 0.1%), the effect can be ignored.  But as it climbs, the pressure to join a pool (the largest pool) becomes economically irresistible, until only one pool remains.

Larger Blocks Are Driving Up Orphan Rates

Large blocks take longer to propagate, increasing the rate of orphans.  This has been happening as blocks increase.  Blocks with no transactions at all are smallest, and so propagate fastest: they still get a 25 bitcoin subsidy, though they don’t help bitcoin users much.

Many people assumed that miners wouldn’t overly centralize, lest they cause a clear decentralization failure and drive the bitcoin price into the ground.  That assumption has proven weak in the face of climbing orphan rates.

And miners have been behaving very badly.  Mining pools orchestrate attacks on each other with surprising regularity; DDOS and block withholding attacks are both well documented[1][2].  A large mining pool used their power to double spend and steal thousands of bitcoin from a gambling service[3].  When it was noticed, they blamed a rogue employee.  No money was returned, nor any legal action taken.  It was hoped that miners would leave for another pool as they approached majority share, but that didn’t happen.

If large blocks can be used as a weapon by larger miners against small ones[8], it’s expected that they will be.

More recently (and quite by accident) it was discovered that over half the mining power aren’t verifying transactions in blocks they build upon[4].  They did this in order to reduce orphans, and one large pool is still doing so.  This is a problem because lightweight bitcoin clients work by assuming anything in the longest chain of blocks is good; this was how the original bitcoin paper anticipated that most users would interact with the system.

The Third Side Of The Debate: Long Term Network Funding

Before I summarize, it’s worth mentioning the debate beyond the current debate: long term network support.  The minting of new coins decreases with time; the plan of record (as suggested in the original paper) is that total transaction fees will rise to replace the current mining subsidy.  The schedule of this is unknown and generally this transition has not happened: free transactions still work.

The block subsidy as I write this is about $7000.  If nothing else changes, miners would want $3500 in fees in 12 months when the block subsidy halves, or about $2 per transaction.  That won’t happen; miners will simply lose half their income.  (Perhaps eventually they form a cartel to enforce a minimum fee, causing another centralization crisis? I don’t know.)

It’s natural for users to try to defer the transition as long as possible, and the practice in bitcoin-core has been to aggressively reduce the default fees as the bitcoin price rises.  Core developers Gregory Maxwell and Pieter Wuille feel that signal was a mistake; that fees will have to rise eventually and users should not be lulled into thinking otherwise.

Mike Hearn in particular has been holding out the promise that it may not be necessary.  On this he is not widely supported: that some users would offer to pay more so other users can continue to pay less.

It’s worth noting that some bitcoin businesses rely on the current very low fees and don’t want to change; I suspect this adds bitterness and vitriol to many online debates.

Summary

The bitcoin-core developers who deal with users most feel that bitcoin needs to expand quickly or die, that letting fees emerge now will kill expansion, and that the infrastructure will improve over time if it has to.

Other bitcoin-core developers feel that bitcoin’s infrastructure is dangerously creaking, that fees need to emerge anyway, and that if there is a real emergency a blocksize change could be rolled out within a few weeks.

At least until this is resolved, don’t count on future bitcoin fees being insignificant, nor promise others that bitcoin has “free transactions”.


[1] http://www.coindesk.com/bitcoin-mining-pools-ddos-attacks/ “Bitcoin Mining Pools Targeted in Wave of DDOS Attacks” Coinbase 2015

[2] http://blog.bettercrypto.com/?p=1131 “Block Withholding Attacks – Recent Research” N T Courtois 2014

[3] https://bitcointalk.org/index.php?topic=327767.0 “GHash.IO and double-spending against BetCoin Dice” mmtech et. al 2013

[4] https://www.reddit.com/r/Bitcoin/comments/3c8tq2/questions_about_the_july_4th_bip66_fork/cstgajp “Questions about the July 4th BIP66 fork”

[5] https://twitter.com/petertoddbtc/status/608475414449778688 “350,000 full nodes to 6,000 in two years…” P Todd 2015

[6] https://getaddr.bitnodes.io/dashboard/?days=365 “Reachable nodes during the last 365 days.” Bitnodes.io

[7] https://bitcointalk.org/index.php?topic=532.msg6306#msg6306 “Re: Scalability and transaction rate” Satoshi 2010

[8] http://www.mail-archive.com/bitcoin-development@lists.sourceforge.net/msg08161.html “[Bitcoin-development] Mining centralization pressure from non-uniform propagation speed” Pieter Wuille 2015

Posted Tue Aug 4 02:32:27 2015 Tags:

So here’s how my day went….

I started off trying to convert a legacy manual page to asciidoc. Found that pandoc (which could be the target of a whole separate rant, because it totally sucks at translating anything with tables in it) won’t do that.

@PUSH…

But it will convert DocBook to asciidoc. OK, so I can use my doclifter tool to convert the manual page to DocBook, then DocBook to asciidoc via pandoc I try this, and doclifter promptly loses its cookies.

@PUSH…

Huh? Oh, I see an [nt]roff request in there I’ve never seen before. Must fix doclifter to handle that. Hack hack hack – done. Push the fix, think “I ought to ship a doclifter release”

@PUSH…

I look at the doclifter repo. I see that the commit graph has an ugly merge bubble in it from where I forgot a –rebase switch when I was pulling last. It’s the pointless kind of bubble where someone else’s patch commutes with mine so the history might as well be linear and easier to read.

You know, before I ship it…I was planning to move that repo from the downstairs machine to GitLab anyway, I might as well fix that bubble in the process….

@PUSH…

Now how do I do that? Hm, yeah, this patch-replay sequence will do it. I ought to can that procedure into a script because I’m doubtless going to have to do it again. (I hates pointless merge bubbles, I hates them forever…) Script done. Hm. I should publish this.

@PUSH…

OK, throw together a man page and README and Makefile. Oh, right, I need a project logo for the web page. What should it be? Into my mind, unbidden, enters the image of the scrubbing bubble animated characters from an advertising campaign of my childhood. Win!

@PUSH…

I google for images, find one, and GIMP out a suitable piece within individual scrubbing bubble in it. Snrk, yeah, that’s funny. Scale it to 64×64 and go.

@POP…

Funny logo achieved.

@POP…

OK, version 1.0 of git-debubble gets published.

@POP…

git-debubble gets applied to the doclifter repo,

@POP…

…which I then publish to GitLab.

@POP…

Now I can convert the manual page to DocBook XML…

@POP…

…which can then be converted to asciidoc.

I have a lot of days like this.

I think there ought to be a term for a sequence of dependency fulfillments that is a lot like yak shaving, except that something useful or entertaining independently of the original task gets emitted at each stage.

Posted Sat Aug 1 20:12:47 2015 Tags:

If you hate pointless merge bubbles as much as I do, you’ll kick yourself because you didn’t think of this.

But don’t feel bad. I should have, a lot sooner, myself.

Voila! git-debubble.

Posted Sat Aug 1 17:36:36 2015 Tags:
The latest feature release Git v2.5.0 is now available at the usual places. It is comprised of 583 non-merge commits since v2.4.0, contributed by 70 people, 21 of which are new faces.

One interesting change is to git help. We now list commands, grouped by the situation in which you would want to use them. This came from discussion on usability, inspired by one of the talks at GitMerge conference we had in spring.

Among notable new features, some of my favourites are:
  • A new short-hand branch@{push} denotes the remote-tracking branch that tracks the branch at the remote the branch would be pushed to.
  • git send-email learned the alias file format used by the sendmail program.
  • Traditionally, external low-level 3-way merge drivers are expected to produce their results based solely on the contents of the three variants given in temporary files named by %O, %A and %B placeholders on their command line. They are now additionally told about the final path (given by %P).
  • A heuristic we use to catch mistyped paths on the command line git cmd revs pathspec is to make sure that all the non-rev parameters in the later part of the command line are names of the files in the working tree, but that means git grep string -- \*.c must always be disambiguated with --, because nobody sane will create a file whose name literally is asterisk-dot-see.  We loosen the heuristic to declare that with a wildcard string the user likely meant to give us a pathspec. So you can now simply say git grep string \*.c without --.
  • Filter scripts were run with SIGPIPE disabled on the Git side, expecting that they may not read what Git feeds them to filter. We however treated a filter that does not read its input fully before exiting as an error.  We no longer do and ignore EPIPE when writing to feed the filter scripts.

    This changes semantics, but arguably in a good way.  If a filter can produce its output without fully consuming its input using whatever magic, we now let it do so, instead of diagnosing it as a programming error.
  • Whitespace breakages in deleted and context lines can also be painted in the output of git diff and friends with the new --ws-error-highlight option.
  • git merge FETCH_HEAD learned that the previous "git fetch" could be to create an Octopus merge, i.e. recording multiple branches that are not marked as "not-for-merge"; this allows us to lose an old style invocation git merge msg HEAD commits... in the implementation of git pull script; the old style syntax can now be deprecated (but not removed yet).
There are a few "experimental" new features, too. They are still incomplete and/or buggy around the edges and likely to change in the future, but nevertheless interesting.
  • git cat-file --batch learned the --follow-symlinks option that follows an in-tree symbolic link when asked about an object via extended SHA-1 syntax.  For example, HEAD:RelNotes may be a symbolic link that points at Documentation/RelNotes/2.5.0.txt.  With the new option, the command behaves as if HEAD:Documentation/RelNotes/2.5.0.txt was given as input instead.

    This is incomplete in at least a few ways.
    (1) A symbolic link in the index, e.g. :RelNotes, should also be treated the same way, but isn't. (2) Non-batch mode, e.g. git cat-file --follow-symlinks blob HEAD:RelNotes, may also want to behave the same way, but it doesn't.
  • A replacement mechanism for contrib/workdir/git-new-workdir that does not rely on symbolic links and make sharing of objects and refs safer by making the borrowee and borrowers aware of each other has been introduced and accessible via git worktree add. This is accumulating more and more known bugs but may prove useful once they are fixed.

Posted Mon Jul 27 21:18:00 2015 Tags:

Disclaimer: I have almost no experience cooking Vietnamese, this is just a distillation of various recipes I’ve only read.

Lamb steak (or other meat: just did this with rib-eye steak, was delicious)
Onion
Sugar
Garlic
Crushed chilli (or other form of chilli)
Vinegar
Soy sauce
Fish sauce

First cook the lamb (or other) steak until browned outside but rare inside. Set aside and slice into thin slices during the remaining cooking.

Slice onions then fry gently in oil until transparent. Add sugar (quite a lot – for two I use a couple of tablespoons at least), garlic, chilli and fry until the sugar is lightly caramelised. Throw in equal parts of vinegar, soy and fish sauce (roughly speaking, you want a little more total liquid than you had sugar), stir in and bring to the boil. Cook until slightly thickened. Add the sliced meat and stir-fry until cooked to your liking.

Serve with boiled rice and something green.

Posted Thu Jul 23 06:48:28 2015 Tags:

Hello Internet! I wanted to share some updates of Roslyn and Mono.

We have been working towards using Roslyn in two scenarios. As the compiler you get when you use Mono, and as the engine that powers code completion and refactoring in the IDE.

This post is a status update on the work that we have been doing here.

Roslyn on MonoDevelop/XamarinStudio

For the past year, we have been working on replacing the IDE's engine that gives us code completion, refactoring capabilities and formatting capabilities with one powered by Roslyn.

The current engine is powered by a combination of NRefactory and the Mono C# compiler. It is not as powerful, comprehensive or reliable as Roslyn.

Feature-wise, we completed the effort, and we now have a Roslyn-powered branch that uses Roslyn for code completion, refactoring, suggestions and code formatting.

In addition, we ported most of the refactoring capabilities from NRefactory to work on top of Roslyn. These were quite significant. Visual Studio users can try them out by installing the Refactoring Essentials for Visual Studio extension.

While our Roslyn branch is working great and is a pleasure to use, it also consumes more memory and by extension, runs a little slower. This is not Roslyn's fault, but the side effects of leaks and limitations in our code.

Our original plan was to release this for our September release (what we internally call "Cycle 6"), but we decided to pull the feature out from the release to give us time to fix the leaks that affected the Roslyn engine and tune the performance of Roslyn running on Mono.

Our revisited plan is to ship an update to our tooling in Cycle 6 (the regular feature update) but without Roslyn. In parallel, we will ship a Roslyn-enabled preview of MonoDevelop/XamarinStudio. This will give us time to collect your feedback on performance and memory usage regressions, and time to fix the issues before we make Roslyn the default.

Roslyn as a Compiler in Mono

One of the major roadblocks for the adoption of Roslyn in Mono was the requirement to generate debugging information that Mono could consume on Unix (the other one is that our C# batch compiler is still faster than Roslyn).

The initial Roslyn release only had support for generating debug information through a proprietary/native library on Windows, which meant that while Roslyn could be used to compile code on Unix, the result would not contain any debug information - this prevented Roslyn from being useful for most compilation uses.

Recently, Roslyn got support for Portable Program Database (PPDB) files. This is a fully documented, open, compact and efficient format for storing debug information.

Mono's master release contains now support for using PPDB files as its debug information. This means that Roslyn can produce debug information that Mono can consume.

That said, we still need more work in the Mono ecosystem to fully support PPDB files. The Cecil library is used extensively to manipulate IL images as well as their associated debug information. Our Reflection.Emit implementation will need to get a backend to generate PPDBs (for third party compilers, dynamic code generators) and support in IKVM to produce PPDB files (this is used by Mono's C# compiler and other third party compilers).

Additionally, many features in Roslyn surfaced bloat and bugs in Mono's class libraries. We have been fixing those bugs (and in many cases, the bugs have gone away by replacing Mono's implementation with implementations from Microsoft's Reference Source).

Posted Tue Jul 21 16:00:37 2015 Tags:

Below is an outline of the various types of touchpads that can be found in the wild. Touchpads aren't simply categorised into a single type, instead they have a set of properties, a combination of number of physical buttons, touch support and physical properties.

Number of buttons

Physically separate buttons

For years this was the default type of touchpads: a touchpad with a separate set of physical buttons below the touch surface. Such touchpads are still around, but most newer models are Clickpads now.

Touchpads with physical buttons usually provide two buttons, left and right. A few touchpads with three buttons exist, and Apple used to have touchpads with a single physical buttons back in the PPC days. Touchpads with only two buttons require the software stack to emulate a middle button. libinput does this when both buttons are pressed simultaneously.


A two-button touchpad, with a two-button pointing stick above.

Note that many Lenovo laptops provide a pointing stick above the touchpad. This pointing stick has a set of physical buttons just above the touchpad. While many users use those as substitute touchpad buttons, they logically belong to the pointing stick. The *40 and *50 series are an exception here, the former had no physical buttons on the touchpad and required the top section of the pad to emulate pointing stick buttons, the *50 series has physical buttons but they are wired to the touchpads. The kernel re-routes those buttons through the trackstick device.

Clickpads

Clickpads are the most common type of touchpads these days. A Clickpad has no separate physical buttons, instead the touchpad itself is clickable as a whole, i.e. a user presses down on the touch area and triggers a physical click. Clickpads thus only provide a single button, everything else needs to be software-emulated.


A clickpad on a Lenovo x220t. Just above the touchpad are the three buttons associated with the pointing stick. Faint markings on the bottom of the touchpad hint at where the software buttons should be.

Right and middle clicks are generated either via software buttons or "clickfinger" behaviour. Software buttons define an area on the touchpad that is a virtual right button. If a finger is in that area when the click happens, the left button event is changed to a right button event. A middle click is either a separate area or emulated when both the left and right virtual buttons are pressed simultaneously.

When the software stack uses the clickfinger method, the number of fingers decide the type of click: a one-finger is a left button, a two-finger click is a right button, a three-finger click is a middle button. The location of the fingers doesn't matter, though there are usually some limits in how the fingers can be distributed (e.g. some implementations try to detect a thumb at the bottom of the touchpad to avoid accidental two-finger clicks when the user intends a thumb click).

The libinput documentation has a section on Clickpad software button behaviour with more detailed illustrations


The touchpad on a T440s has no physical buttons for the pointing stick. The marks on the top of the touchpad hint at the software button position for the pointing stick. Note that there are no markings at the bottom of the touchpad anymore.

Clickpads are labelled by the kernel with the INPUT_PROP_BUTTONPAD input property.

Forcepads

One step further down the touchpad evolution, Forcepads are Clickpads without a physical button. They provide pressure and (at least in Apple's case) have a vibration element that is software-controlled. Instead of the satisfying click of a physical button, you instead get a buzz of happiness. Which apparently feels the same as a click, judging by the reviews I've read so far. A software-controlled click feel has some advantages, it can be disabled for some gestures, modified for others, etc. I suspect that over time Forcepads will become the main touchpad category, but that's a few years away.

Not much to say on the implementation here. The kernel has some ForcePad support but everything else is spotty.


Note how Apple's Clickpads have no markings whatsoever, Apple uses the clickfinger method by default.

Touch capabilities

Single-touch touchpads

In the beginning, there was the single-finger touchpad. This touchpad would simply provide x/y coordinates for a single finger and get mightily confused when more than one finger was present. These touchpads are now fighting with dodos for exhibition space in museums, few of those are still out in the wild.

Pure multi-touch touchpads

Pure multi-touch touchpads are those that can track, i.e. identify the location of all fingers on the touchpad. Apple's touchpads support 16 touches (iirc), others support 5 touches like the Synaptics touchpads when using SMBus.

Pure multi-touch touchpads are the easiest to support, we can rely on the finger locations and use them for scrolling, gestures, etc. These touchpads usually also provide extra information. In the case of the Apple touchpads we get an ellipsis and the orientation of the ellipsis for each touch point. Other touchpads provide a pressure value for each touch point. Though pressure is a bit of a misnomer, pressure is usually directly related to contact area. Since our puny human fingers flatten out as the pressure on the pad increases, the contact area increases and the firmware then calculates that back into a (mostly rather arbitrary) pressure reading.

Because pressure is really contact area size, we can use it to detect accidental palm contact or thumbs though it's fairly unreliable. A light palm touch or a touch at the very edge of a touchpad will have a low pressure reading simply because the palm is mostly next to the touchpad and thus the contact area itself remains small.

Partial multi-touch touchpads

The vast majority of touchpads fall into this category. It's the half-way point between single-touch and pure multi-touch. These devices can track N fingers, but detect more than N. The current Synaptics touchpads fall into that category when they're using the serial protocol. Most touchpads that fall into this category can track two fingers and detect up to four or five. So a typical three-finger interaction would give you the location of two fingers and a separate value telling you that a third finger is down.

The lack of finger location doesn't matter for some interactions (tapping, three-finger click) but it can cause issues in some cases. For example, a user may have a thumb resting on a touchpad while scrolling with two fingers. Which touch locations you get depends on the order of the fingers being set down, i.e. this may look like thumb + finger + third touch somewhere (lucky!) or two fingers scrolling + third touch somewhere (unlucky, this looks like a three-finger swipe). So far we've mostly avoided having anything complex enough that requires the exact location of more than two fingers, these pads are so prevalent that any complex feature would exclude the majority of users.

Semi-mt touchpads

A sub-class of partial multi-touch touchpads. These touchpads can technically detect two fingers but the location of both is limited to the bounding box, i.e. the first touch is always the top-left one and the second touch is the bottom-right one. Coordinates jump around as fingers move past each other. Most semi-mt touchpads also have a lower resolution for two touches than for one, so even things like two-finger scrolling can be very jumpy.

Semi-mt are labelled by the kernel with the INPUT_PROP_SEMI_MT input property.

Physical properties

External touchpads

USB or Bluetooth touchpads not in a laptop chassis. Think the Apple Magic Trackpad, the Logitech T650, etc. These are usually clickpads, the biggest difference is that they can be removed or added at runtime. One interaction method that is only possible on external touchpads is a thumb resting on the very edge/immediately next to the touchpad. On the far edge, touchpads don't always detect the finger location so clicking with a thumb barely touching the edge makes it hard or impossible to figure out which software button area the finger is on.

These touchpads also don't need palm detection - since they're not located underneath the keyboard, accidental palm touches are a non-issue.


A Logitech T650 external touchpad. Note the thumb position, it is possible to click the touchpad without triggering a touch.

Circular touchpads

Yes, used to be a thing. Touchpad shaped in an ellipsis or circle. Luckily for us they have gone full dodo. The X.Org synaptics driver had to be aware of these touchpads to calculate the right distance for edge scrolling - unsurprisingly an edge scroll motion on a circular touchpad isn't very straight.

Graphics tablets

Touch-capable graphics tablets are effectively external touchpads, with two differentiators: they are huge compared to normal touchpads and they have no touchpad buttons whatsoever. This means they can either work like a Forcepad, or rely on interaction methods that don't require buttons (like tap-to-click). Since the physical device is shared with the pen input, some touch arbitration is required to avoid touch input interfering when the pen is in use.

Dedicated edge scroll area

Mostly on older touchpads before two-finger scrolling became the default method. These touchpads have a marking on the touch area that designates the edge to be used for scrolling. A finger movement in that edge zone should trigger vertical motions. Some touchpads have markers for a horizontal scroll area too at the bottom of the touchpad.


A touchpad with a marked edge scroll area on the right.
Posted Tue Jul 21 04:40:00 2015 Tags:

Just a quick note that we’ve released the latest stable NetworkManager, version 1.0.4.  This is mainly a bugfix release, though it does have a couple new features and larger changes.  Some of those are:

  • Some configuration options can now be changed without restarting the NM daemon.  Those include the ‘dns’, ‘connectivity’, and ‘ignore-carrier’ settings.
  • Devices that have only an IPv6 link-local address are no longer assumed to be connected; by default whenever an interface is set “up” the kernel will assign an IPv6 link-local address which could theoretically be used for communication.  NetworkManager used to interpret this as the interface being available for network communication, while this is rarely what users want or expect.
  • Correct routing is now maintained when two interfaces of the same priority are connected to the same network.
  • udev rules can now be used to tell NetworkManager to manage or unmanage specific devices.
  • Connections with children (bridge, bond, team, etc) can now optionally bring up their slave interfaces when the master is started.
  • Many, many bugs and crashers have also been fixed in the core daemon, the libraries, and nmcli.

Grab the 1.0.4 release here!

We’re well into the development cycle of NetworkManager 1.2 as well, where among other things, we’ll finally be moving to GDBus instead of dbus-glib.  We’ll also have support for structured logging with journald, indicating that certain connections are metered, advertising LLDP-discovered information, built-in IPv4 link-local addressing to get rid of avahi-autoipd, improvements to Wi-Fi Access Point scanning, less verbose logging by default, improvements to DNS handling, and more.  Look for it later this year!

 

Posted Thu Jul 16 16:20:47 2015 Tags:

In a perfect world, any device that advertises absolute x/y axes also advertises the resolution for those axes. Alas, not all of them do. For libinput, this problem is two-fold: parts of the touchscreen API provide data in mm - without knowing the resolution this is a guess at best. But it also matters for touchpads, where a lack of resolution is a lot more common (though the newest generations of major touchpad manufacturers tend to advertise resolutions now).

We have a number of features that rely on the touchpad resolution: from the size of the software button to deciding which type of palm detection we need, it all is calculated based on physical measurements. Until recently, we had code to differentiate between touchpads with resolution and most of the special handling was a matter of magic numbers, usually divided by the diagonal of the touchpad in device units. This made code maintenance more difficult - without testing each device, behaviour could not be guaranteed.

With libinput 0.20, we now got rid of this special handling and instead require the touchpads to advertise resolutions. This requires manual intervention, so we're trying to fix this in multiple places, depending on the confidence of the data. We have hwdb entries for the bcm5974 (Apple) touchpads and the Chromebook Pixel. For Elantech touchpads, a kernel patch is currently waiting for merging. For ALPS touchpads, we ship size hints with libinput's hwdb. If all that fails, we fall back to a default touchpad size of 69x55mm. [1]

All this affects users in two ways: one is that you may notice a slightly different behaviour of your touchpad now. The software-buttons may be bigger or smaller than before, pointer acceleration may be slightly different, etc. Shouldn't be too bad, but you may just notice it. The second noticeable change is that libinput will now log when it falls back to the default size. If you notice a message like that in your log, please file a bug and attach the output of evemu-describe and the physical dimensions of your touchpad. Once we have that information, we can add it to the right place and make sure that everyone else with that touchpad gets the right settings out of the box.

[1] The default size was chosen because it's close enough to what old touchpads used to be, and those are most likely to lack resolution values. This size may change over time as we get better data.

Posted Thu Jul 16 11:28:00 2015 Tags:

Most of you by now have probably seen Conservancy's and FSF's statements regarding the today's update to Canonical, Ltd.'s Ubuntu IP Policy. I have a few personal comments, speaking only for myself, that I want to add that don't appear in the FSF's nor Conservancy's analysis. (I wrote nearly all of Conservancy's analysis and did some editing on FSF's analysis, but the statements here I add are my personal opinions and don't necessarily reflect the views of the FSF nor Conservancy, notwithstanding that I have affiliations with both orgs.)

First of all, I think it's important to note the timeline: it took two years of work by two charities to get this change done. The scary thing is that compared to their peers who have also violated the GPL, Canonical, Ltd. acted rather quickly. As Conservancy pointed out regarding the VMware lawsuit, it's not uncommon for these negotiations to take even four years before we all give up and have to file a lawsuit. So, Canonical, Ltd. resolved the matter at least twice as fast as VMware, and they deserve some credit for that — even if other GPL violators have set the bar quite low.

Second, I have to express my sympathy for the positions on this matter taken by Matthew Garrett and Jonathan Riddell. Their positions show clearly that, while the GPL violation is now fully resolved, the community is very concerned about what the happens regarding non-copylefted software in Ubuntu, and thus Ubuntu as a whole.

Realize, though, that these trump clauses are widely used throughout the software industry. For example, electronics manufacturers who ship an Android/Linux system with standard, disgustingly worded, forbid-everything EULA usually include a trump clause not unlike Ubuntu's. In such systems, usually, the only copylefted program is the kernel named Linux. The rest of the distribution includes tons of (now proprietarized) non-copylefted code from Android (as well as a bunch of born-proprietary applications too). The trump clause assures the software freedom rights for that one copylefted work present, but all the non-copylefted ones are subject to the strict EULA (which often includes “no reverse engineer clauses”, etc.). That means if the electronics company did change the Android Java code in some way, you can't even legally reverse engineer it — even though it was Apache-licensed by upstream.

Trump clauses are thus less than ideal because they achieve compliance only by allowing a copyleft to prevail when the overarching license contradicts specific requirements, permissions, or rights under copyleft. That's acceptable because copyleft licenses have many important clauses that assure and uphold software freedom. By contrast, most non-copyleft licenses have very few requirements, and thus they lack adequate terms to triumph over any anti-software-freedom terms of the overarching license. For example, if I take a 100% ISC-licensed program and build a binary from it, nothing in the ISC license prohibits me from imposing this license on you: “you may not redistribute this binary commercially”. Thus, even if I also say to you: “but also, if the ISC license grants rights, my aforementioned license does not modify or reduce those rights”, nothing has changed for you. You still have a binary that you can't distribute commercially, and there was no text in the ISC license to force the trump clause to save you.

Therefore, this whole situation is a simple and clear argument for why copyleft matters. Copyleft can and does (when someone like me actually enforces it) prevent such situations. But copyleft is not infinitely expansive. Nearly every full operating system distribution available includes an aggregated mix of copylefted, non-copyleft, and often fully-proprietary userspace applications. Nearly every company that distributes them wraps the whole thing with some agreement that restricts some rights that copyleft defends, and then adds a trump clause that gives an exception just for FLOSS license compliance. Sadly, I have yet to see a company trailblaze adoption of a “software freedom preservation” clause that guarantees copyleft-like compliance for non-copylefted programs and packages. Thus, the problem with Ubuntu is just a particularly bad example of what has become a standard industry practice by nearly every “open source” company.

How badly these practices impact software freedom depends on the strictness and detailed terms of the overarching license (and not the contents of the trump clause itself; they are generally isomorphic0). The task of analyzing and rating “relative badness” of each overarching licensing document is monumental; there are probably thousands of different ones in use today. Matthew Garrett points out why Canonical, Ltd.'s is particularly bad, but that doesn't mean there aren't worse (and better) situations of a similar ilk. Perhaps our next best move is to use copyleft licenses more often, so that the trump clauses actually do more.

In other words, as long as there is non-copylefted software aggregated in a given distribution of an otherwise Free Software system, companies will seek to put non-Free terms on top of the non-copylefted parts, To my knowledge, every distribution-shipping company (except for extremely rare, Free-Software-focused companies like ThinkPenguin) place some kind of restrictions in their business terms for their enterprise distribution products. Everyone seems to be asking me today to build the “worst to almost-benign” ranking of these terms, but I've resisted the urge to try. I think the safe bet is to assume that if you're looking at one of these trump clauses, there is some sort of software-freedom-unfriendly restriction floating around in the broader agreement, and you should thus just avoid that product entirely. Or, if you really want to use it, fork it from source and relicense the non-copylefted stuff under copyleft licenses (which is permitted by nearly all non-copyleft licenses), to prevent future downstream actors from adding more restrictive terms. I'd even suggest this as a potential solution to the current Ubuntu problem (or, better yet, just go back upstream to Debian and do the same :).

Finally, IMO the biggest problem with these “overarching licenses with a trump clause” is their use by companies who herald “open source” friendliness. I suspect the community ire comes from a sense of betrayal. Yet, I feel only my usual anger at proprietary software here; I don't feel betrayed. Rather, this is just another situation that proves that saying you are an “open source company” isn't enough; only the company's actions and “fine print” terms matter. Now that open source has really succeeded at coopting software freedom, enormous effort is now required to ascertain if any company respects your software freedom. We must ignore the ballyhoo of “community managers” and look closely at the real story.


0Despite Canonical, Ltd.'s use of a trump clause, I don't think these various trump clauses are canonically isomorphic. There is no natural mapping between these various trump clauses, but they all do have the same effect: they assure that when the overarching terms conflict with the a FLOSS license, the FLOSS license triumphs over the overarching terms, no matter what they are. However, the potential relevance of the phrase “canonical isomorphism” here is yet another example why it's confusing and insidious that Canonical, Ltd. insisted so strongly on using canonical in a non-canonical way.

Posted Wed Jul 15 22:45:10 2015 Tags:

The libinput test suite takes somewhere around 35 minutes now for a full run. That's annoying, especially as I'm running it for every commit before pushing. I've tried optimising things, but attempts at making it parallel have mostly failed so far (almost all tests need a uinput device created) and too many tests rely on specific timeouts to check for behaviours. Containers aren't an option when you have to create uinput devices so I started out farming out into VMs.

Ideally, the test suite should run against multiple commits (on multiple VMs) at the same time while I'm working on some other branch and then accumulate the results. And that's where git notes come in. They're a bit odd to use and quite the opposite of what I expected. But in short: a git note is an object that can be associated with a commit, without changing the commit itself. Sort-of like a post-it note attached to the commit. But there are plenty of limitations, for example you can only have one note (per namespace) and merge conflicts are quite easy to trigger. Look at any git notes tutorial to find out more, there's plenty out there.

Anyway, dealing with merge conflicts is a no-go for me here. So after a bit of playing around, I found something that seems to work out well. A script to run make check and add notes to the commit, combined with a repository setup to fetch those notes and display them automatically. The core of the script is this:


make check
rc=$?
if [ $? -eq 0 ]; then
status="SUCCESS"
else
status="FAIL"
fi

if [ -n "$sha" ]; then
git notes --ref "test-$HOSTNAME" append \
-m "$status: $HOSTNAME: make check `date`" HEAD
fi
exit $rc
Then in my main repository, I add each VM as a remote, adding a fetch path for the notes:

[remote "f22-libinput1"]
url = f22-libinput1.local:/home/whot/code/libinput
fetch = +refs/heads/*:refs/remotes/f22-libinput1/*
fetch = +refs/notes/*:refs/notes/f22-libinput1/*
Finally, in the main repository, I extended the glob that displays notes to 'everything':

$ git config notes.displayRef "*"
Now git log (and by extension tig) displays all notes attached to a commit automatically. All that's needed is a git fetch --all to fetch everything and it's clear in the logs which commit fails and which one succeeded.

:: whot@jelly:~/code/libinput (master)> git log
commit 6896bfd3f5c3791e249a0573d089b7a897c0dd9f
Author: Peter Hutterer
Date: Tue Jul 14 14:19:25 2015 +1000

test: check for fcntl() return value

Mostly to silence coverity complaints.

Signed-off-by: Peter Hutterer

Notes (f22-jelly/test-f22-jelly):
SUCCESS: f22-jelly: make check Tue Jul 14 00:20:14 EDT 2015

Whenever I look at the log now, I immediately see which commits passed the test suite and which ones didn't (or haven't had it run yet). The only annoyance is that since a note is attached to a commit, amending the commit message or rebasing makes the note "go away". I've copied notes manually after this, but it'd be nice to find a solution to that.

Everything else has been working great so far, but it's quite new so there'll be a bit of polishing happening over the next few weeks. Any suggestions to improve this are welcome.

Posted Tue Jul 14 20:40:00 2015 Tags:

I think it’s weird that I have to write this post in 2015, but earlier today I had to explain to someone with the technical skills to submit a good patch that he was doing the process wrong in some basic and extremely annoying ways.

Googling revealed that most explanations of patch etiquette are rather project-specific in their advice. So I’m going to explain the basics of patch submission that apply to just about any open-source project, with a focus on how to do it right when you aren’t a regular committer (that is, it’s what’s often called a drive-by patch). Here we go…

Let’s start with the blunder that motivated this post. It was email that inquired “Have you had a chance to review my previously submitted patch?” What was infuriating was that said inquiry did not identify or include a copy of that drive-by patch, didn’t identify what revision of the project it was made against, and for that matter didn’t even name the project.

This violated the first rule of getting your patch accepted, which is to minimize the cognitive load on the person processing it. That person may see dozens of patches a week on multiple projects (and frequently does, if it’s me). Expecting him to remember lots of context and match it to your name when you’re not already a regular committer that he recognizes is a good way to make yourself disliked.

So make it easy. When you mail a drive-by patch, always specify the target project it applies to, and what base revision of the project it applies to (by commit ID or date). The hapless n00b I am now excoriating replied, when asked what revision his patch was against, replied “HEAD” and it was all I could do not to leap through the screen and throttle him. Because of course HEAD now is not necessarily the same revision it was when he composed the patch.

Good etiquette is different if you have a well-established working relationship with the person you are submitting to. If I get a git-format patch submission from a regular contributor whom I recognize to one of my projects. he doesn’t have to repeat that context; I can generally assume it was made from a recently-synced repository clone and will apply against head. But I can only be relaxed about this if I am sure he will be explicit about the target project and base revision if it’s for a different project.

Another way of minimizing cognitive load is to make the patch self-explaining. Always include an explanation of the patch, its motivation, and its risks (in particular, if the patch is untested you should say so).

Hapless n00b failed to do this. I had to Google for the name of the principal function in his patch to learn that it’s a code-hardening move derived from OpenBSD security work. Even though it’s a good patch, and I merged it, that’s more work than he should have made me do. Through not being explicit, he actually slowed down the merge of a fix for a potential security issue by a couple of weeks.

Even if you are not sending a git-format patch, the git conventions for change comments are good to follow. That is, a summary line in imperative form (“Fix bug 2317.”; “Implement feature foo.”) should be optionally followed by a blank line and explanatory paragraphs.

Really good bug-fix patches include a recipe for reproducing the problem that motivated them. Because I can test those instantly, I can merge them instantly. On projects with a regression-test suite, you scale the heights of best practice by including a patch bands that enhance the test suite to demonstrate the correctness of the patched code. When you do that, I love you and want to see more patches from you.

Here’s a mistake the n00b thankfully did not make: his patch was small and simple, easily reviewed. When in doubt, send a patch series of simple changes rather than a huge rollup patch that does multiple things – that sort of mess is very likely to be rejected because reviewing it hurts the maintainer’s brain.

At least he shipped in diff -u format; it’d been years since I saw even a n00b use the horrible default -e format, which is very brittle in the face of any change to the target code at all (never do this!). diff -u -p format, which tries to put the name of the affected function in the header before each patch band, is better (this what git format-patch does).

Maintainers vary in whether they like patches to be plain text inline in the message or a MIME attachment. I myself am relatively indifferent to this, but others have strong preferences. If you are not sure, include an offer to resend in the form the maintainer prefers in your cover note.

On the other hand, unless the patch is large enough to blow an email size limit (which suggests it’s much too large) do not ever send it as an http/ftp link to be chased to the patch text. Your maintainer might be on the road with spotty network access when he tries to process it. Even if he isn’t, every step of hand-work he has to do (including trivial steps like typing a wget command) increases his process friction, which is unkind and reduces the odds that your patch will be merged.

It is irritating to the maintainer to get a patch against an old release, it is very irritating to get a patch for a problem he/she has already fixed, and it is extremely irritating to get a patch for a problem that the maintainer has to do extra work to find out he has already fixed. Find the project repository, clone it, test your issue, and make the patch against head. This will, among other things, greatly reduce the odds of your patch failing to apply.

Many maintainers (including me) dislike unnecessary merge bubbles. If you write a patch series using git that takes a while to complete, rebase it against current head before you ship. Again, this maximizes the chances that it will apply cleanly.

I’m going to finish with a preference of mine which while not necessarily universal practice, at least won’t offend anyone. If you have to refer to a previous commit in your change comment, please do not use commit IDs like git hashes or Subversion revision numbers to do it. Instead, quote the summary line of the commit and if in doubt cite the author and date.

There are two good reasons for this practice. One is that it’s human-friendly; a patch summary line is a much better cue to memory than a numeric or hex cookie. The other is that at some time in the future the repository might undergo surgery or conversion that will turn that cookie into a turd. Be kind to future maintainers by using a reference format that won’t become invalid and require hand translation.

Posted Mon Jul 13 12:31:42 2015 Tags:
Visualization of functional groups.
Public domain from Wikipedia.
In the past 50 years we have been trying to understand why certain classes of compounds show the same behavior. Quantum chemical calculations are still getting cheaper and easier (though, I cannot point you to a review of recent advances), but it has not replaced other approaches, as is visible in the number of QSAR/descriptor applications of the CDK.

Functional Group Ontology
Sankar et al. have developed an ontology for functional groups (doi:10.1016/j.jmgm.2013.04.003). One popular thought is that subgroups of atoms are more important than the molecule as a whole. Much of our cheminformatics is based on this idea. And it matches what we experimentally observe. If we add a hydroxyl or an acid group, the molecule becomes more hydrophylic. Semantically encoding this clearly important information seems important, though intuitively I would have left this to the cheminformatics tools. This paper and a few cited papers, however, show far you can take this. It organizes more than 200 functional groups, but I am not sure where the ontology can be downloaded.

Sankar, P., Krief, A., Vijayasarathi, D., Jun. 2013. A conceptual basis to encode and detect organic functional groups in XML. Journal of Molecular Graphics and Modelling 43, 1-10. URL http://dx.doi.org/10.1016/j.jmgm.2013.04.003

Linking biological to chemical similarities
If we step aside from our concept of "functional group", we can also just look at whatever is similar between molecules. Michael Kuhn et al. (of STITCH and SIDER) looked into the role of individual proteins in side effect (doi:10.1038/msb.2013.10). They find that many drug side effects are mediated by a selection of individual proteins. The study uses a drug-target interaction data set, and to reduce the change of bias due to some compound classes more extensively studies (more data), they removed too similar compounds from the data set, using the CDK's Tanimoto stack.

Kuhn, M., Al Banchaabouchi, M., Campillos, M., Jensen, L. J., Gross, C., Gavin, A. C., Bork, P., Apr. 2014. Systematic identification of proteins that elicit drug side effects. Molecular Systems Biology 9 (1), 663. URL http://dx.doi.org/10.1038/msb.2013.10

Drug-induced liver injury
These approaches can also be used to study if there are structural reasons why Drug-induced liver injury (DILI) occurs. This was studied in this paper Zhu et al. where the CDK is used to calculate topological descriptors (doi:10.1002/jat.2879). They compared explanatory models that correlate descriptors with the measured endpoint and a combination with hepatocyte imaging assay technology (HIAT) descriptors. These descriptors capture phenotypes such as nuclei count, nuclei area, intensities of reactive oxygen species intensity, tetramethyl rhodamine methyl ester, lipid intensity, and glutathione. It doesn't cite any of the CDK papers, so I left a comment with PubMed Commons.

Zhu, X.-W., Sedykh, A., Liu, S.-S., Mar. 2014. Hybrid in silico models for drug-induced liver injury using chemical descriptors and in vitro cell-imaging information. Journal of Applied Toxicology 34 (3), 281-288. URL http://dx.doi.org/10.1002/jat.2879
Posted Sun Jul 12 09:39:00 2015 Tags:
Tool validation
The first paper this week is a QSAR paper. In fact, it does some interesting benchmarking of a few tools with a data set of about 6000 compounds. It includes looking into the applicability domain, and studies the error of prediction for compounds inside and outside the chemical space defined by the training set. The paper indirectly uses the CDK descriptor calculation corner, by using EPA's T.E.S.T. toolkit (at least one author, Todd Martin, contributed to the CDK).

Callahan, A., Cruz-Toledo, J., Dumontier, M., Apr. 2013. Ontology-Based querying with Bio2RDF's linked open data. Journal of Biomedical Semantics 4 (Suppl 1), S1+. URL http://dx.doi.org/10.1186/2041-1480-4-s1-s1

Tetranortriterpenoid
Arvind et al. study tetranortriterpenoids using a QSAR approach involving COMFA and the CPSA descriptor (green OA PDF). The latter CDK descriptor is calculated using Bioclipse. The study finds that using compound classes can improve the regression.

Arvind, K., Anand Solomon, K., Rajan, S. S., Apr. 2013. QSAR studies of tetranortriterpenoid: An analysis through CoMFA and CPSA parameters. Letters in Drug Design & Discovery 10 (5), 427-436. URL http://dx.doi.org/10.2174/1570180811310050010

Accurate monoisotopic masses
Another useful application of the CDK is the Java wrapping of the isotope data in the Blue Obelisk Data Repository (BODR). Mareile Niesser et al. use Rajarshi's rcdk package for R to calculate the differences in accurate monoisotopic masses. They do not cite the CDK directly, but do mention it by name in the text.

Niesser, M., Harder, U., Koletzko, B., Peissner, W., Jun. 2013. Quantification of urinary folate catabolites using liquid chromatography–tandem mass spectrometry. Journal of Chromatography B 929, 116-124. URL http://dx.doi.org/10.1016/j.jchromb.2013.04.008
Posted Sat Jul 11 13:22:00 2015 Tags:

Last night f2pool mined a 1MB block containing a single 1MB transaction.  This scooped up some of the spam which has been going to various weakly-passworded “brainwallets”, gaining them 0.5569 bitcoins (on top of the normal 25 BTC subsidy).  You can see the megatransaction on blockchain.info.

It was widely reported to take about 25 seconds for bitcoin core to process this block: this is far worse than my “2 seconds per MB” result in my last post, which was considered a pretty bad case.  Let’s look at why.

How Signatures Are Verified

The algorithm to check a transaction input (of this form) looks like this:

  1. Strip the other inputs from the transaction.
  2. Replace the input script we’re checking with the script of the output it’s trying to spend.
  3. Hash the resulting transaction with SHA256, then hash the result with SHA256 again.
  4. Check the signature correctly signed that hash result.

Now, for a transaction with 5570 inputs, we have to do this 5570 times.  And the bitcoin core code does this by making a copy of the transaction each time, and using the marshalling code to hash that; it’s not a huge surprise that we end up spending 20 seconds on it.

How Fast Could Bitcoin Core Be If Optimized?

Once we strip the inputs, the result is only about 6k long; hashing 6k 5570 times takes about 265 milliseconds (on my modern i3 laptop).  We have to do some work to change the transaction each time, but we should end up under half a second without any major backflips.

Problem solved?  Not quite….

This Block Isn’t The Worst Case (For An Optimized Implementation)

As I said above, the amount we have to hash is about 6k; if a transaction has larger outputs, that number changes.  We can fit in fewer inputs though.  A simple simulation shows the worst case for 1MB transaction has 3300 inputs, and 406000 byte output(s): simply doing the hashing for input signatures takes about 10.9 seconds.  That’s only about two or three times faster than the bitcoind naive implementation.

This problem is far worse if blocks were 8MB: an 8MB transaction with 22,500 inputs and 3.95MB of outputs takes over 11 minutes to hash.  If you can mine one of those, you can keep competitors off your heels forever, and own the bitcoin network… Well, probably not.  But there’d be a lot of emergency patching, forking and screaming…

Short Term Steps

An optimized implementation in bitcoind is a good idea anyway, and there are three obvious paths:

  1. Optimize the signature hash path to avoid the copy, and hash in place as much as possible.
  2. Use the Intel and ARM optimized SHA256 routines, which increase SHA256 speed by about 80%.
  3. Parallelize the input checking for large numbers of inputs.

Longer Term Steps

A soft fork could introduce an OP_CHECKSIG2, which hashes the transaction in a different order.  In particular, it should hash the input script replacement at the end, so the “midstate” of the hash can be trivially reused.  This doesn’t entirely eliminate the problem, since the sighash flags can require other permutations of the transaction; these would have to be carefully explored (or only allowed with OP_CHECKSIG).

This soft fork could also place limits on how big an OP_CHECKSIG-using transaction could be.

Such a change will take a while: there are other things which would be nice to change for OP_CHECKSIG2, such as new sighash flags for the Lightning Network, and removing the silly DER encoding of signatures.

Posted Wed Jul 8 03:09:59 2015 Tags: