The lower-post-volume people behind the software in Debian. (List of feeds.)

Since the AppStore launched, developers have complained about the review process as too strict. Applications are mostly rejected either for not meeting requirements, not having enough functionality or circumventing Apple’s business model.

Yet, the AppStore reviews are too lax and they should be much stricter.

Let me explain why I think so, what I believe some new rules need to be, and how the AppStore can be improved.

Prioritizing the Needs of the Many

Apple states that they have 28 million registered developers, but I believe that only a fraction of those are actively developing applications on a daily basis. That number is closer to 5 million developers.

I understand deeply why developers are frustrated with the AppStore review process - I have suffered my fair share of AppStore rejections: both by missing simple issues and by trying to push the limits of what was allowed. I founded Xamarin, a company that built tools for mobile developers, and had a chance to become intimately familiar with the rejections that our own customers got.

Yet, there are 1.5 billion active Apple devices, devices that people trust to be keep their data secure and private. The overriding concern should be the 1.5 billion active users, and not the 0.33% (or 1.86% if you are feeling generous).

People have deposited their trust on Apple and Google to keep their devices safe. I wrote about this previously. While it is an industry sport to make fun of Google, I respect the work that Google puts on securing and managing my data - so much that I have trusted them with my email, photographs and documents for more than 15 years.

I trust both companies both because of their public track record, and because of conversations that I have had with friends working at both companies about their processes, their practices and the principles that they care about (Keeping up with Information Security is a part-time hobby of minex).

Today’s AppStore Policies are Insufficient

AppStore policies, and their automated and human reviews have helped nurture and curate the applications that are available. But with a target market as large and rich as iOS and Android these ecosystems have become a juicy target for scammers, swindlers, gangsters, nation states and hackers.

While some developers are upset with the Apple Store rejections, profiteers have figured out that they can make a fortune while abiding by the existing rules. These rules allow behaviors that are in either poor taste, or explicitly manipulating the psyche of the user.

First, let me share my perspective as a parent, and

I have kids aged 10, 7 and 4, and my eldest had access to an iPad since she was a year old, and I have experienced first hand how angering some applications on the AppStore can be to a small human.

It breaks my heart every time they burst out crying because something in these virtual worlds was designed to nag them, is frustrating or incomprehensible to them. We sometimes teach them how to deal with those problems, but this is not always possible. Try explaining to a 3 year old why they have to watch a 30 seconds ad in the middle of a dinosaur game to continue playing, or teach them that at arbitrary points during the game tapping on the screen will not dismiss an ad, but will instead take them to download another app, or direct them another web site.

This is infuriating.

Another problem happens when they play games defective by design. By this I mean that these games have had functionality or capabilities removed that can be solved by purchasing virtual items (coins, bucks, costumes, pets and so on).

I get to watch my kids display a full spectrum of negative experiences when they deal with these games.

We now have a rule at home “No free games or games with In-App Purchases”. While this works for “Can I get a new game?”, it does not work for the existing games that they play, and those that they play with their friends.

Like any good rule, there are exceptions, and I have allowed the kids to buy a handful of games with in-app purchases from reputable sources. They have to pay for those from their allowance.

These dark patterns are not limited applications for kids, read the end of this post for a list of negative scenarios that my followers encountered that will ring familiar.

Closing the AppStore Loopholes

Applications using these practices should be banned:

  • Those that use Dark Patterns to get users to purchase applications or subscriptions: These are things like “Free one week trial”, and then they start charging a high fee per week. Even if this activity is forbidden, some apps that do this get published.

  • Defective-by-design: there are too many games out there that can not be enjoyed unless you spend money in their applications. They get the kids hooked up, and then I have to deal with whiney 4 year olds, 7 year olds and 10 year old to spend their money on virtual currencies to level up.

  • Apps loaded with ads: I understand that using ads to monetize your application is one way of supporting the development, but there needs to be a threshold on how many ads are on the screen, and shown by time, as these apps can be incredibly frustrating to use. And not all apps offer a “Pay to remove the ad”, I suspect because the pay-to-remove is not as profitable as showing ads non-stop.

  • Watch an ad to continue: another nasty problem are defective-by-design games and application that rather than requesting money directly, steer kids towards watching ads (sometimes “watch an ad for 30 seconds”) to get something or achieve something. They are driving ad revenue by forcing kids to watch garbage.

  • Install chains: there are networks of ill-behaved applications that trick kids into installing applications that are part of their network of applications. It starts with an innocent looking app, and before the day is over, you are have 30 new scammy apps installed on your machine.

  • Notification Abuse: these are applications that send advertisements or promotional offers to your device. From discount offers, timed offers and product offerings. It used to be that Apple banned these practices on their AppReview guidelines, but I never saw those enforced and resorted to turning off notifications. These days these promotions are allowed. I would like them to be banned, have the ability to report them as spam, and infringers to have their notification rights suspended.

  • ) Ban on Selling your Data to Third Parties: ban applications that sell your data to third parties. Sometimes the data collection is explicit (for example using the Facebook app), but sometimes unknowingly, an application uses a third party SDK that does its dirty work behind the scenes. Third party SDKs should be registered with Apple, and applications should disclose which third party SDKs are in use. If one of those 3rd party SDKs is found to abuse the rules or is stealing data, all applications that rely on the SDK can be remotely deactivated. While this was recently in the news this is by no means a new practice, this has been happening for years.

One area that is grayer are Applications that are designed to be addictive to increase engagement (some games, Facebook and Twitter) as they are a major problem for our psyches and for our society. Sadly, it is likely beyond the scope of what the AppStore Review team can do. One option is to pass legislation that would cover this (Shutdown Laws are one example).

Changes in the AppStore UI

It is not apps for children that have this problem. I find myself thinking twice before downloading applications with "In App Purchases". That label has become a red flag: one that sends the message "scammy behavior ahead"

I would rather pay for an app than a free app with In-App Purchases. This is unfair to many creators that can only monetize their work via an In-App Purchases.

This could be addressed either by offering a free trial period for the app (managed by the AppStore), or by listing explicitly that there is an “Unlock by paying” option to distinguish these from “Offers In-App Purchases” which is a catch-all expression for both legitimate, scammy or nasty sales.

My list of wishes:

  • Offer Trial Periods for applications: this would send a clear message that this is a paid application, but you can try to use it. And by offering this directly by the AppStore, developers would not have to deal with the In-App purchase workflow, bringing joy to developers and users alike.

  • Explicit Labels: Rather than using the catch-all “Offers In-App Purchases”, show the nature of the purchase: “Unlock Features by Paying”, “Offers Subscriptions”, “Buy virtual services” and “Sells virtual coins/items”

  • Better Filtering: Today, it is not possible to filter searches to those that are paid apps (which tend to be less slimy than those with In-App Purchases)

  • Disclose the class of In-App Purchases available on each app that offers it up-front: I should not have to scroll and hunt for the information and mentally attempt to understand what the item description is to make a purchase

  • Report Abuse: Human reviewers and automated reviews are not able to spot every violation of the existing rules or my proposed additional rules. Users should be able to report applications that break the rules and developers should be aware that their application can be removed from circulation for breaking the rules or scammy behavior.

Some Bad Practices

Check some of the bad practices in this compilation

Posted Thu Sep 24 21:51:27 2020 Tags:

Some bad app patterns as some followers described them

You can read more in the replies to my request a few weeks ago:

Posted Thu Sep 24 02:33:00 2020 Tags:

This is the continuation from these posts: part 1, part 2, part 3 and part 4.

In the posts linked above, I describe how it's possible to have custom keyboard layouts in $HOME or /etc/xkb that will get picked up by libxkbcommon. This only works for the Wayland stack, the X stack doesn't use libxkbcommon. In this post I'll explain why it's unlikely this will ever happen in X.

As described in the previous posts, users configure with rules, models, layouts, variants and options (RMLVO). What XKB uses internally though are keycodes, compat, geometry, symbols types (KcCGST) [1].

There are, effectively, two KcCGST keymap compilers: libxkbcommon and xkbcomp. libxkbcommon can go from RMLVO to a full keymap, xkbcomp relies on other tools (e.g. setxkbmap) which in turn use a utility library called libxkbfile to can parse rules files. The X server has a copy of the libxkbfile code. It doesn't use libxkbfile itself but it relies on the header files provided by it for some structs.

Wayland's keyboard configuration works like this:

  • the compositor decides on the RMLVO keybard layout, through an out-of-band channel (e.g. gsettings, weston.ini, etc.)
  • the compositor invokes libxkbcommon to generate a KcCGST keymap and passes that full keymap to the client
  • the client compiles that keymap with libxkbcommon and feeds any key events into libxkbcommon's state tracker to get the right keysyms
The advantage we have here is that only the full keymap is passed between entities. Changing how that keymap is generated does not affect the client. This, coincidentally [2], is also how Xwayland gets the keymap passed to it and why Xwayland works with user-specific layouts.

X works differently. Notably, KcCGST can come in two forms, the partial form specifying names only and the full keymap. The partial form looks like this:


$ setxkbmap -print -layout fr -variant azerty -option ctrl:nocaps
xkb_keymap {
xkb_keycodes { include "evdev+aliases(azerty)" };
xkb_types { include "complete" };
xkb_compat { include "complete" };
xkb_symbols { include "pc+fr(azerty)+inet(evdev)+ctrl(nocaps)" };
xkb_geometry { include "pc(pc105)" };
};
This defines the component names but not the actual keymap, punting that to the next part in the stack. This will turn out to be the achilles heel. Keymap handling in the server has two distinct aproaches:
  • During keyboard device init, the input driver passes RMLVO to the server, based on defaults or xorg.conf options
  • The server has its own rules file parser and creates the KcCGST component names (as above)
  • The server forks off xkbcomp and passes the component names to stdin
  • xkbcomp generates a keymap based on the components and writes it out as XKM file format
  • the server reads in the XKM format and updates its internal structs
This has been the approach for decades. To give you an indication of how fast-moving this part of the server is: XKM caching was the latest feature added... in 2009.

Driver initialisation is nice, but barely used these days. You set your keyboard layout in e.g. GNOME or KDE and that will apply it in the running session. Or run setxkbmap, for those with a higher affinity to neckbeards. setxkbmap works like this:

  • setkxkbmap parses the rules file to convert RMLVO to KcCGST component names
  • setkxkbmap calls XkbGetKeyboardByName and hands those component names to the server
  • The server forks off xkbcomp and passes the component names to stdin
  • xkbcomp generates a keymap based on the components and writes it out as XKM file format
  • the server reads in the XKM format and updates its internal structs
Notably, the RMLVO to KcCGST conversion is done on the client side, not the server side. And the only way to send a keymap to the server is that XkbGetKeyboardByName request - which only takes KcCGST, you can't even pass it a full keymap. This is also a long-standing potential issue with XKB: if your client tools uses different XKB data files than the server, you don't get the keymap you expected.

Other parts of the stack do basically the same as setxkbmap which is just a thin wrapper around libxkbfile anyway.

Now, you can use xkbcomp on the client side to generate a keymap, but you can't hand it as-is to the server. xkbcomp can do this (using libxkbfile) by updating the XKB state one-by-one (XkbSetMap, XkbSetCompatMap, XkbSetNames, etc.). But at this point you're at the stage where you ask the server to knowingly compile a wrong keymap before updating the parts of it.

So, realistically, the only way to get user-specific XKB layouts into the X server would require updating libxkbfile to provide the same behavior as libxkbcommon, update the server to actually use libxkbfile instead of its own copy, and updating xkbcomp to support the changes in part 2, part 3. All while ensuring no regressions in code that's decades old, barely maintained, has no tests, and, let's be honest, not particularly pretty to look at. User-specific XKB layouts are somewhat a niche case to begin with, so I don't expect anyone to ever volunteer and do this work [3], much less find the resources to review and merge that code. The X server is unlikely to see another real release and this is definitely not something you want to sneak in in a minor update.

The other option would be to extend XKB-the-protocol with a request to take a full keymap so the server. Given the inertia involved and that the server won't see more full releases, this is not going to happen.

So as a summary: if you want custom keymaps on your machine, switch to Wayland (and/or fix any remaining issues preventing you from doing so) instead of hoping this will ever work on X. xmodmap will remain your only solution for X.

[1] Geometry is so pointless that libxkbcommon doesn't even implement this. It is a complex format to allow rendering a picture of your keyboard but it'd be a per-model thing and with evdev everyone is using the same model, so ...
[2] totally not coincidental btw
[3] libxkbcommon has been around for a decade now and no-one has volunteered to do this in the years since, so...

Posted Fri Sep 4 00:39:00 2020 Tags:

This is the continuation from these posts: part 1, part 2, part 3

This is the part where it all comes together, with (BYO) fireworks and confetti, champagne and hoorays. Or rather, this is where I'll describe how to actually set everything up. It's a bit complicated because while libxkbcommon does the parsing legwork now, we haven't actually changed other APIs and the file formats which are still 1990s-style nerd cool and requires significant experience in CS [1] to understand what goes where.

The below relies on software using libxkbcommon and libxkbregistry. At the time of writing, libxkbcommon is used by all mainstream Wayland compositors but not by the X server. libxkbregistry is not yet used because I'm typing this before we had a release for it. But at least now I have a link to point people to.

libxkbcommon has a xkbcli-scaffold-new-layout tool that The xkblayout tool creates the template files as shown below. At the time of writing, this tool must be run from the git repo build directory, it is not installed.

I'll explain here how to add the us(banana) variant and the custom:foo option, and I will optimise for simplicity and brevity.

Directory structure

First, create the following directory layout:


$ tree $XDG_CONFIG_HOME/xkb
/home/user/.config/xkb
├── compat
├── keycodes
├── rules
│   ├── evdev
│   └── evdev.xml
├── symbols
│   ├── custom
│   └── us
└── types
If $XDG_CONFIG_HOME is unset, fall back to $HOME/.config.

Rules files

Create the rules file and add an entry to map our custom:foo option to a section in the symbols/custom file.


$ cat $XDG_CONFIG_HOME/xkb/rules/evdev
! option = symbols
custom:foo = +custom(foo)

// Include the system 'evdev' file
! include %S/evdev
Note that no entry is needed for the variant, that is handled by wildcards in the system rules file. If you only want a variant and no options, you technically don't need this rules file.

Second, create the xml file used by libxkbregistry to display your new entries in the configuration GUIs:


$ cat $XDG_CONFIG_HOME/xkb/rules/evdev.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE xkbConfigRegistry SYSTEM "xkb.dtd">
<xkbConfigRegistry version="1.1">
<layoutList>
<layout>
<configItem>
<name>us</name>
</configItem>
<variantList>
<variant>
<configItem>
<name>banana</name>
<shortDescription>banana</shortDescription>
<description>US(Banana)</description>
</configItem>
</variant>
</variantList>
</layout>
</layoutList>
<optionList>
<group allowMultipleSelection="true">
<configItem>
<name>custom</name>
<description>custom options</description>
</configItem>
<option>
<configItem>
<name>custom:foo</name>
<description>This option does something great</description>
</configItem>
</option>
</group>
</optionList>
</xkbConfigRegistry>
Our variant needs to be added as a layoutList/layout/variantList/variant, the option to the optionList/group/option. libxkbregistry will combine this with the system-wide evdev.xml file in /usr/share/X11/xkb/rules/evdev.xml.

Overriding and adding symbols

Now to the actual mapping. Add a section to each of the symbols files that matches the variant or option name:


$ cat $XDG_CONFIG_HOME/xkb/symbols/us
partial alphanumeric_keys modifier_keys
xkb_symbols "banana" {
name[Group1]= "Banana (us)";

include "us(basic)"

key <CAPS> { [ Escape ] };
};
with this, the us(banana) layout will be a US keyboard layout but with the CapsLock key mapped to Escape. What about our option? Mostly the same, let's map the tilde key to nothing:

$ cat $XDG_CONFIG_HOME/xkb/symbols/custom
partial alphanumeric_keys modifier_keys
xkb_symbols "foo" {
key <TLDE> { [ VoidSymbol ] };
};
A note here: NoSymbol means "don't overwrite it" whereas VoidSymbol is "map to nothing".

Notes

You may notice that the variant and option sections are almost identical. XKB doesn't care about variants vs options, it only cares about components to combine. So the sections do what we expect of them: variants include enough other components to make them a full keyboard layout, options merely define a few keys so they can be combined with layouts(variants). Due to how the lookups work, you could load the option template as layout custom(foo).

For the actual documentation of keyboard configuration, you'll need to google around, there are quite a few posts on how to map keys. All that has changed is where from and how things are loaded but not the actual data formats.

If you wanted to install this as system-wide custom rules, replace $XDG_CONFIG_HOME with /etc.

The above is a replacement for xmodmap. It does not require a script to be run manually to apply the config, the existing XKB integration will take care of it. It will work in Wayland (but as said above not in X, at least not for now).

A final word

Now, I fully agree that this is cumbersome, clunky and feels outdated. This is easy to fix, all that is needed is for someone to develop a better file format, make sure it's backwards compatible with the full spec of the XKB parser (the above is a fraction of what it can do), that you can generate the old files from the new format to reduce maintenance, and then maintain backwards compatibility with the current format for the next ten or so years. Should be a good Google Decade of Code beginner project.

[1] Cursing and Swearing

Posted Tue Sep 1 01:13:00 2020 Tags:

This is the continuation from these posts: part 1, part 2

Let's talk about everyone's favourite [1] keyboard configuration system again: XKB. If you recall the goal is to make it simple for users to configure their own custom layouts. Now, as described earlier, XKB-the-implementation doesn't actually have a concept of a "layout" as such, it has "components" and something converts your layout desires into the combination of components. RMLVO (rules, model, layout, variant, options) is what you specify and gets converted to KcCGST (keycodes, compat, geometry, symbols, types). This is a one-way conversion, the resulting keymaps no longer has references to the RMLVO arguments. Today's post is about that conversion, and we're only talking about libxkbcommon as XKB parser because anything else is no longer maintained.

The peculiar thing about XKB data files (provided by xkeyboard-config [3]) is that the filename is part of the API. You say layout "us" variant "dvorak", the rules file translates this to symbols 'us(dvorak)' and the parser will understand this as "load file 'symbols/us' and find the dvorak section in that file". [4] The default "us" keyboard layout will require these components:


xkb_keymap {
xkb_keycodes { include "evdev+aliases(qwerty)" };
xkb_types { include "complete" };
xkb_compat { include "complete" };
xkb_symbols { include "pc+us+inet(evdev)" };
xkb_geometry { include "pc(pc105)" };
};
So the symbols are really: file symbols/pc, add symbols/us and then the section named 'evdev' from symbols/inet [5]. Types are loaded from types/complete, etc. The lookup paths for libxkbcommon are $XDG_CONFIG_HOME/xkb, /etc/xkb, and /usr/share/X11/xkb, in that order.

Most of the symbols sections aren't actually full configurations. The 'us' default section only sets the alphanumeric rows, everything else comes from the 'pc' default section (hence: include "pc+us+inet(evdev)"). And most variant sections just include the default one, usually called 'basic'. For example, this is the 'euro' variant of the 'us' layout which merely combines a few other sections:


partial alphanumeric_keys
xkb_symbols "euro" {

include "us(basic)"
name[Group1]= "English (US, euro on 5)";

include "eurosign(5)"

include "level3(ralt_switch)"
};
Including things works as you'd expect: include "foo(bar)" loads section 'bar' from file 'foo' and this works for 'symbols/', 'compat/', etc., it'll just load the file in the same subdirectory. So yay, the directory is kinda also part of the API.

Alright, now you understand how KcCGST files are loaded, much to your despair.

For user-specific configuration, we could already load a 'custom' layout from the user's home directory. But it'd be nice if we could just add a variant to an existing layout. Like "us(banana)", because potassium is important or somesuch. This wasn't possible because the filename is part of the API. So our banana variant had to be in $XDG_CONFIG_HOME/xkb/symbols/us and once we found that "us" file, we could no longer include the system one.

So as of two days ago, libxkbcommon now extends the parser to have merged KcCGST files, or in other words: it'll load the symbols/us file in the lookup path order until it finds the section needed. With that, you can now copy this into your $XDG_CONFIG_HOME/xkb/symbols/us file and have it work as variant:


partial alphanumeric_keys
xkb_symbols "banana" {

include "us(basic)"
name[Group1]= "English (Banana)";

// let's assume there are some keymappings here
};
And voila, you now have a banana variant that can combine with the system-level "us" layout.

And because there must be millions [6] of admins out there that maintain custom XKB layouts for a set of machines, the aforementioned /etc/xkb lookup path was also recently added to libxkbcommon. So we truly now have the typical triplet of lookup paths:

  • vendor-provided ones in /usr/share/X11/xkb,
  • host-specific ones in /etc/xkb, and
  • user-specific ones in $XDG_CONFIG_HOME/xkb [7].
Good times, good times.

[1] Much in the same way everyone's favourite Model T colour was black
[2] This all follows the UNIX philosophy, there are of course multiple tools involved and none of them know what the other one is doing
[3] And I don't think Sergey gets enough credit for maintaining that pile of language oddities
[4] Note that the names don't have to match, we could map layout 'us' to the symbols in 'banana' but life's difficult enough as it is
[5] I say "add" when it's sort of a merge-and-overwrite and, yes, of course there are multiple ways to combine layouts, thanks for asking
[6] Actual number may be less
[7] Notice how "X11" is missing in the latter two? If that's not proof that we want to get rid of X, I don't know what is!

Posted Mon Aug 31 04:51:00 2020 Tags:

In the legal battle over the App Store’s policies, fees, and review processes, Epic Games wants to see a return to the good old days – where software developers retained full control over their systems and were only limited by their imaginations. Yet those days are long gone.

Granted, in the early 90s, I hailed the Internet as humanity’s purest innovation. After all, it had enabled a group of global developers to collaboratively build the Linux operating system from the ground up. In my X years of experience as a developer, nothing has come close to the good will, success, and optimistic mood of those days.

Upon reflection, everything started to change the day I received my first spam message. It stood out not only because it was the first piece of unsolicited email I received, but also because it was a particularly nasty piece of spam. The advertiser was selling thousands of email addresses for the purposes of marketing and sales. Without question, I knew that someone would buy that list, and that I would soon be on the receiving end of thousands of unwanted pieces of mail. Just a few months later, my inbox was filled with garbage. Since then, the Internet has become increasingly hostile – from firewalls, proxies, and sandboxes to high-profile exploits and attacks.

Hardly a new phenomenon, before the Internet, the disk operating system (DOS) platform was an open system where everyone was free to build and innovate. But soon folks with bad intentions destroyed what was a beautiful world of creation and problem solving, and turned it into a place riddled with viruses, trojans, and spyware.

Like most of you alive at the time, I found myself using anti-virus software. In fact, I even wrote a commercial product in Mexico that performed the dual task of scanning viruses and providing a Unix-like permission system for DOS (probably around 1990). Of course, it was possible to circumvent these systems, considering DOS code had full access to the system.

In 1993, Microsoft introduced a family of operating systems that came to be known as Windows NT. Though it was supposed to be secure from the ground up, they decided to leave a few things open due to compatibility concerns with the old world of Windows 95 and DOS. Not only were there bad faith actors in the space, developers had made significant mistakes. Perhaps not surprisingly, users began to routinely reinstall their operating systems following the gradual decays that arose from improper changes to their operating systems.

Fast-forward to 2006 when Windows Vista entered the scene – attempting to resolve a class of attacks and flaws. The solution took many users by surprise. It’s more apt to say that it was heavily criticized and regarded as a joke in some circles. For many, the old way of doing things had been working just fine and all the additional security got in the way. While users hated the fact that software no longer worked out of the box, it was an important step towards securing systems.

With the benefit of hindsight, I look back at the early days of DOS and the Internet as a utopia, where good intentions, innovation, and progress were the norm. Now swindlers, scammers, hackers, gangsters, and state actors routinely abuse open systems to the point that they have become a liability for every user.

In response, Apple introduced iOS – an operating system that was purpose-build to be secure. This avoided backwards compatibility problems and having to deal with users who saw unwanted changes to their environment. In a word, Apple managed to avoid the criticism and pushback that had derailed Windows Vista.

It’s worth pointing out that Apple wasn’t the first to introduce a locked-down system that didn’t degrade. Nintendo, Sony, and Microsoft consoles restricted the software that could be modified on their host operating systems and ran with limited capabilities. This resulted in fewer support calls, reduced frustration, and limited piracy.

One of Apple’s most touted virtues is that the company creates secure devices that respect user’s privacy. In fact, they have even gone to court against the US government over security. Yet iOS remains the most secure consumer operating system. This has been made possible through multiple layers of security that address different threats. (By referring to Apple’s detailed platform security, you can get a clear sense of just how comprehensive it is.)

Offering a window into the development process, security experts need to evaluate systems from end-to-end and explore how the system can be tampered with, attacked, or hacked, and then devise both defense mechanisms and plans for when things will inevitably go wrong.

Consider the iPhone. The hardware, operating system, and applications were designed with everything a security professional loves in mind. Even so, modern systems are too large and too complex to be bullet-proof. Researchers, practitioners, hobbyists, and businesses all look for security holes in these systems – some with the goal of further protecting the system, others for educational purposes, and still others for profit or to achieve nefarious goals.

Whereas hobbyists leverage these flaws to unlock their devices and get full control over their systems, dictatorships purchase exploits in the black market to use against their enemies and gain access to compromising data, or to track the whereabouts of their targets.

This is where the next layer of security comes in. When a flaw is identified – whether by researchers, automated systems, telemetry, or crashes – software developers design a fix for the problem and roll out the platform update. The benefits of keeping software updated extend beyond a few additional emoji characters; many software updates come with security fixes. Quite simply, updating your phone keeps you more secure. However, it’s worth emphasizing that this only works against known attacks.

The App Store review process helps in some ways; namely, it can:

  • Force applications to follow a set of guidelines aimed at protecting privacy, the integrity of the system, and meet the bar for unsuspecting users

  • Reduce applications with minimal functionality – yielding less junk for users to deal with and smaller attack surfaces

  • Require a baseline of quality, which discourages quick hacks

  • Prevent applications from using brittle, undocumented, or unsupported capabilities

Still, the App Store review process is not flawless. Some developers have worked around these restrictions by: (1) distributing hidden payloads, (2) temporarily disabling features while their app was being tested on Apple’s campus, (3) using time triggers, or (4) remotely controlling features to evade reviewers.

As a case in point, we need look no further than Epic Games. They deceptively submitted a “hot fix,” which is a practice used to fix a critical problem such as a crash. Under the covers, they added a new purchasing system that could be remotely activated at the time of their choosing. It will come as no surprise that they activated it after they cleared the App Store’s review process.

Unlike a personal computer, the applications you run on your smartphone are isolated from the operating system and even from each other to prevent interference. Since apps run under a “sandbox” that limits what they can do, you do not need to reinstall your iPhone from scratch every few months because things no longer work.

Like the systems we described above, the sandbox is not perfect. In theory, a bad actor could include an exploit for an unknown security hole in their application, slip it past Apple, and then, once it is used in the wild, wake up the dormant code that hijacks your system.

Anticipating this, Apple has an additional technical and legal mitigation system in place. The former allows Apple to remotely disable and deactivate ill-behaved applications, in cases where an active exploit is being used to harm users. The legal mitigation is a contract that is entered into between Apple and the software developer, which can be used to bring bad actors to court.

Securing a device is an ongoing arms race, where defenders and attackers are constantly trying to outdo the other side, and there is no single solution that can solve the problem. The battlegrounds have recently moved and are now being waged at the edges of the App Store’s guidelines.

In the same way that security measures have evolved, we need to tighten the App Store’s guidelines, including the behaviors that are being used for the purposes of monetization and to exploit children. (I plan to cover these issues in-depth in a future post.) For now, let me just say that, as a parent, there are few things that would make me happier than more stringent App Store rules governing what applications can do. In the end, I value my iOS devices because I know that I can trust them with my information because security is paramount to Apple.

Coming full-circle, Epic Games is pushing for the App Store to be a free-for-all environment, reminiscent of DOS. Unlike Apple, Epic does not have an established track record of caring about privacy and security (in fact, their privacy policy explicitly allows them to sell your data for marketing purposes). Not only does the company market its wares to kids, they recently had to backtrack on some of their most questionable scams – i.e., loot boxes – when the European Union regulated them. Ultimately, Epic has a fiduciary responsibility to their investors to grow their revenue, and their growth puts them on a war path with Apple.

In the battle over the security and privacy of my phone, I am happy to pay a premium knowing that my information is safe and sound, and that it is not going to be sold to the highest bidder.

Posted Fri Aug 28 20:40:10 2020 Tags:

Apple versus Epic Games: Why I’m Willing to Pay a Premium for Security, Privacy, and Peace of Mind

In the legal battle over the App Store’s policies, fees, and review processes, Epic Games wants to see a return to the good old days – where software developers retained full control over their systems and were only limited by their imaginations. Yet those days are long gone.

Granted, in the early 90s, I hailed the Internet as humanity’s purest innovation. After all, it had enabled a group of global developers to collaboratively build the Linux operating system from the ground up. In my X years of experience as a [developer], nothing has come close to the good will, success, and optimistic mood of those days.

Upon reflection, everything started to change the day I received my first spam message. It stood out not only because it was the first piece of unsolicited email I received, but also because it was a particularly nasty piece of spam. The advertiser was selling thousands of email addresses for the purposes of marketing and sales. Without question, I knew that someone would buy that list, and that I would soon be on the receiving end of thousands of unwanted pieces of mail. Just a few months later, my inbox was filled with garbage. Since [specify year], the Internet has become increasingly hostile – from firewalls, proxies, and sandboxes to high-profile exploits and attacks.

Hardly a new phenomenon, before the Internet, the disk operating system (DOS) platform was an open system where everyone was free to build and innovate. But soon folks with bad intentions destroyed what was a beautiful world of creation and problem solving, and turned it into a place riddled with viruses, trojans, and spyware.

Like most of you alive at the time, I found myself using anti-virus software. In fact, I even wrote a commercial product in Mexico that performed the dual task of scanning viruses and providing a Unix-like permission system for DOS (probably around 1990). Of course, it was possible to circumvent these systems, considering DOS code had full access to the system.

In 1993, Microsoft introduced a family of operating systems that came to be known as Windows NT. Though it was supposed to be secure from the ground up, they decided to leave a few things open due to compatibility concerns with the old world of Windows 95 and DOS. Not only were there bad faith actors in the space, developers had made significant mistakes. Perhaps not surprisingly, users began to routinely reinstall their operating systems following the gradual decays that arose from improper changes to their operating systems.

Fast-forward to 2006 when Windows Vista entered the scene – attempting to resolve a class of attacks and flaws. The solution took many users by surprise. It’s more apt to say that it was heavily criticized and regarded as a joke in some circles. For many, the old way of doing things had been working just fine and all the additional security got in the way. While users hated the fact that software no longer worked out of the box, it was an important step towards securing systems.

With the benefit of hindsight, I look back at the early days of DOS and the Internet as a utopia, where good intentions, innovation, and progress were the norm. Now swindlers, scammers, hackers, gangsters, and state actors routinely abuse open systems to the point that they have become a liability for every user.

In response, Apple introduced iOS – an operating system that was purpose-build to be secure. This avoided backwards compatibility problems and having to deal with users who saw unwanted changes to their environment. In a word, Apple managed to avoid the criticism and pushback that had derailed Windows Vista.

It’s worth pointing out that Apple wasn’t the first to introduce a locked-down system that didn’t degrade. Nintendo, Sony, and Microsoft consoles restricted the software that could be modified on their host operating systems and ran with limited capabilities. This resulted in fewer support calls, reduced frustration, and limited piracy.

One of Apple’s most touted virtues is that the company creates secure devices that respect user’s privacy. In fact, they have even gone to court against the US government over security. Yet iOS remains the most secure consumer operating system. This has been made possible through multiple layers of security that address different threats. (By referring to Apple’s detailed platform security, you can get a clear sense of just how comprehensive it is.)

Offering a window into the development process, security experts need to evaluate systems from end-to-end and explore how the system can be tampered with, attacked, or hacked, and then devise both defense mechanisms and plans for when things will inevitably go wrong. Consider the iPhone. The hardware, operating system, and applications were designed with everything a security professional loves in mind. Even so, modern systems are too large and too complex to be bullet-proof. Researchers, practitioners, hobbyists, and businesses all look for security holes in these systems – some with the goal of further protecting the system, others for educational purposes, and still others for profit or to achieve nefarious goals.

Whereas hobbyists leverage these flaws to unlock their devices and get full control over their systems, dictatorships purchase exploits in the black market to use against their enemies and gain access to compromising data, or to track the whereabouts of their targets.

This is where the next layer of security comes in. When a flaw is identified – whether by researchers, automated systems, telemetry, or crashes – software developers design a fix for the problem and roll out the platform update. The benefits of keeping software updated extend beyond a few additional emoji characters; many software updates come with security fixes. Quite simply, updating your phone keeps you more secure. However, it’s worth emphasizing that this only works against known attacks.

The App Store review process helps in some ways; namely, it can:

  • Force applications to follow a set of guidelines aimed at protecting privacy, the integrity of the system, and meet the bar for unsuspecting users

  • Reduce applications with minimal functionality – yielding less junk for users to deal with and smaller attack surfaces

  • Require a baseline of quality, which discourages quick hacks

  • Prevent applications from using brittle, undocumented, or unsupported capabilities

Still, the App Store review process is not flawless. Some developers have worked around these restrictions by: (1) distributing hidden payloads, (2) temporarily disabling features while their app was being tested on Apple’s campus, (3) using time triggers, or (4) remotely controlling features to evade reviewers.

As a case in point, we need look no further than Epic Games. They deceptively submitted a “hot fix,” which is a practice used to fix a critical problem such as a crash. Under the covers, they added a new purchasing system that could be remotely activated at the time of their choosing. It will come as no surprise that they activated it after they cleared the App Store’s review process.

Unlike a personal computer, the applications you run on your smartphone are isolated from the operating system and even from each other to prevent interference. Since apps run under a “sandbox” that limits what they can do, you do not need to reinstall your iPhone from scratch every few months because things no longer work.

Like the systems we described above, the sandbox is not perfect. In theory, a bad actor could include an exploit for an unknown security hole in their application, slip it past Apple, and then, once it is used in the wild, wake up the dormant code that hijacks your system.

Anticipating this, Apple has an additional technical and legal mitigation system in place. The former allows Apple to remotely disable and deactivate ill-behaved applications, in cases where an active exploit is being used to harm users. The legal mitigation is a contract that is entered into between Apple and the software developer, which can be used to bring bad actors to court.

Securing a device is an ongoing arms race, where defenders and attackers are constantly trying to outdo the other side, and there is no single solution that can solve the problem. The battlegrounds have recently moved and are now being waged at the edges of the App Store’s guidelines.

In the same way that security measures have evolved, we need to tighten the App Store’s guidelines, including the behaviors that are being used for the purposes of monetization and to exploit children. (I plan to cover these issues in-depth in a future post.) For now, let me just say that, as a parent, there are few things that would make me happier than more stringent App Store rules governing what applications can do. In the end, I value my iOS devices because I know that I can trust them with my information because security is paramount to Apple.

Coming full-circle, Epic Games is pushing for the App Store to be a free-for-all environment, reminiscent of DOS. Unlike Apple, Epic does not have an established track record of caring about privacy and security. Not only does the company market its wares to kids, they recently had to backtrack on some of their most questionable scams – i.e., loot boxes – when the European Union regulated them. Ultimately, Epic has a fiduciary responsibility to their investors to grow their revenue, and their growth puts them on a war path with Apple.

In the battle over the security and privacy of my phone, I am happy to pay a premium knowing that my information is safe and sound, and that it is not going to be sold to the highest bidder.

Posted Fri Aug 28 20:00:55 2020 Tags:

Let's talk about eggs. X has always supported XSendEvent() which allows anyone to send any event to any client [1]. However, this event had a magic bit to make it detectable, so clients detect and subsequently ignore it. Spoofing input that just gets ignored is of course not productive, so in the year 13 BG [2] the XTest extension was conceived. XTest has a few requests that allow you to trigger a keyboard event (press and release, imagine the possibilities), buttons and pointer motion. The name may seem odd until someone explains to you that it was primarily written to support automated testing of X servers. But no-one has the time to explain that.

Having a separate extension worked around the issue of detectability and thus any client could spoof input events. Security concerns were addressed with "well, just ifdef out that extension then" which worked great until other applications started using it for input emulation. Since around ~2008 XTest events are emulated through special XTest devices in the server but that is solely to make the implementation less insane. Technically this means that XTest events are detectable again, except that no-one bothers to actually do that. Having said that, these devices only make it possible to detect an XTest event, but not which client sent that event. And, due to how the device hierarchy works, it's really hard to filter out those events anyway.

Now it's 2020 and we still have an X server that basically allows access to anything and any client to spoof input. This level of security is industry standard for IoT devices but we are trying to be more restrictive than that on your desktop, lest the stuff you type actually matters. The accepted replacement for X is of course Wayland, but Wayland-the-protocol does not provide an extension to emulate input. This makes sense, emulating input is not exactly a display server job [3] but it does leaves us with a bunch of things no longer working.

libei

So let's talk about libei. This new library for Emulated Input consists of two components: libei, the client library to be used in applications, and libeis, the part to be used by an Emulated Input Server, to be used in the compositors. libei provides the API to connect to an EIS implementation and send events. libeis provides the API to receive those events and pass them on to the compositor. Let's see how this looks like in the stack:


+--------------------+ +------------------+
| Wayland compositor |---wayland---| Wayland client B |
+--------------------+\ +------------------+
| libinput | libeis | \_wayland______
+----------+---------+ \
| | +-------+------------------+
/dev/input/ +---brei----| libei | Wayland client A |
+-------+------------------+

"brei" is the communication "Bridge for EI" and is a private protocol between libei and libeis.

Emulated input is not particularly difficult, mostly because when you're emulating input, you know exactly what you are trying to do. There is no interpretation of bad hardware data like libinput has to do, it's all straightforward. Hence the communication between libei and libeis is relatively trivial, it's all button, keyboard, pointer and touch events. Nothing fancy here and I won't go into details because that part will work as you expect it to. The selling point of libei is elsewhere, primarily in separation, distinction and control.

libei is a separate library to libinput or libwayland or Xlib or whatever else may have. By definition, it is thus a separate input stream in both the compositor and the client. This means the compositor can do things like display a warning message while emulated input is active. Or it can re-route the input automatically, e.g. assign a separate wl_seat to the emulated input devices so they can be independently focused, etc. Having said that, the libeis API is very similar to libinput's API so integration into compositors is quite trivial because most of the hooks to process incoming events are already in place.

libei distinguishes between different clients, so the compositor is always aware of who is currently emulating input. You have synergy, xdotool and a test suite running at the same time? The compositor is aware of which client is sending which events and can thus handle those accordingly.

Finally, libei provides control over the emulated input. This works on multiple levels. A libei client has to request device capabilities (keyboard, touch, pointer) and the compositor can restrict to a subset of those (e.g. "no keyboard for you"). Second, the compositor can suspend/resume a device at any time. And finally, since the input events go through the compositor anyway, it can discard events it doesn't like. For example, even where the compositor allowed keyboards and the device is not suspended, it may still filter out Escape key presses. Or rewrite those to be Caps Lock because we all like a bit of fun, don't we?

The above isn't technically difficult either, libei itself is not overly complex. The interaction between an EI client and an EIS server is usually the following:

  • client connects to server and says hello
  • server disconnects the client, or accepts it
  • client requests one or more devices with a set of capabilities
  • server rejects those devices or allows those devices and/or limits their capabilities
  • server resumes the device (because they are suspended by default)
  • client sends events
  • client disconnects, server disconnects the client, server suspends the device, etc.
Again, nothing earth-shattering here. There is one draw-back though: the server must approve of a client and its devices, so a client is not able to connect, send events and exit. It must wait until the server has approved the devices before it can send events. This means tools like xdotool need to be handled in a special way, more on that below.

Flatpaks and portals

With libei we still have the usual difficulties: a client may claim it's synergy when really it's bad-hacker-tool. There's not that much we can do outside a sandbox, but once we are guarded by a portal, things look different:


+--------------------+
| Wayland compositor |_
+--------------------+ \
| libinput | libeis | \_wayland______
+----------+---------+ \
| [eis-0.socket] \
/dev/input/ / \\ +-------+------------------+
| ======>| libei | Wayland client A |
| after +-------+------------------+
initial| handover /
connection| / initial request
| / dbus[org.freedesktop.portal.EmulatedInput]
+--------------------+
| xdg-desktop-portal |
+--------------------+
The above shows the interaction when a client is run inside a sandbox with access to the org.freedesktop.portal.Desktop bus. The libei client connects to the portal (it won't see the EIS server otherwise), the portal hands it back a file descriptor to communicate on and from then on it's like a normal EI session. What the portal implementation can do though is write several Restrictions for EIS on the server side of the file descriptor using libreis. Usually, the portal would write the app-id of the client (thus guaranteeing a reliable name) and maybe things like "don't allow keyboard capabilities for any device". Once the fd is handed to the libei client, the restrictions cannot be loosened anymore so a client cannot overwrite its own name.

So the full interaction here becomes:

  • Client connects to org.freedesktop.portal.EmulatedInput
  • Portal implementation verifies that the client is allowed to emulate input
  • Portal implementation obtains a socket to the EIS server
  • Portal implementation sends the app id and any other restrictions to the EIS server
  • Portal implementation returns the socket to the client
  • Client creates devices, sends events, etc.
For a client to connect to the portal instead of the EIS server directly is currently a one line change.

Note that the portal implementation is still in its very early stages and there are changes likely to happen. The general principle is unlikely to change though.

Xwayland

Turns out we still have a lot of X clients around so somehow we want to be able to use those. Under Wayland, those clients connect to Xwayland which translates X requests to Wayland requests. X clients will use XTest to emulate input which currently goes to where the dodos went. But we can add libei support to Xwayland and connect XTest, at which point the diagram looks like this:


+--------------------+ +------------------+
| Wayland compositor |---wayland---| Wayland client B |
+--------------------+\ +------------------+
| libinput | libeis | \_wayland______
+----------+---------+ \
| | +-------+------------------+
/dev/input/ +---brei----| libei | XWayland |
+-------+------------------+
|
| XTEST
|
+-----------+
| X client |
+-----------+
XWayland is basically just another Wayland client but it knows about XTest requests, and it knows about the X clients that request those. So it can create a separate libei context for each X client, with pointer and keyboard devices that match the XTest abilities. The end result of that is you can have a xdotool libei context, a synergy libei context, etc. And of course, where XWayland runs in a sandbox it could use the libei portal backend. At which point we have a sandboxed X client using a portal to emulate input in the Wayland compositor. Which is pretty close to what we want.

Where to go from here?

So, at this point the libei repo is still sitting in my personal gitlab namespace. Olivier Fourdan has done most of the Xwayland integration work and there's a basic Weston implementation. The portal work (tracked here) is the one needing the most attention right now, followed by the implementations in the real compositors. I think I have tentative agreement from the GNOME and KDE developers that this all this is a good idea. Given that the goal of libei is to provide a single approach to emulate input that works on all(-ish) compositors [4], that's a good start.

Meanwhile, if you want to try it, grab libei from git, build it, and run the eis-demo-server and ei-demo-client tools. And for portal support, run the eis-fake-portal tool, just so you don't need to mess with the real one. At the moment, those demo tools will have a client connecting a keyboard and pointer and sending motion, button and 'qwerty' keyboard events every few seconds. The latter with client and/or server-set keymaps because that's possible too.

Eggs

What does all this have to do with eggs? "Ei", "Eis", "Brei", and "Reis" are, respectively, the German words for "egg", "ice" or "ice cream", "mush" (think: porridge) and "rice". There you go, now you can go on holidays to a German speaking country and sustain yourself on a nutritionally imbalanced diet.

[1] The whole "any to any" is a big thing in X and just shows that in the olden days you could apparently trust, well, apparently anyone
[2] "before git", i.e. too bothersome to track down so let's go with the Copyright notice in the protocol specification
[3] XTest the protocol extension exists presumably because we had a hammer and a protocol extension thus looked like a nail
[4] Let's not assume that every compositor having their own different approach to emulation is a good idea

Posted Tue Aug 25 04:43:00 2020 Tags:

 

Because it is hard to get funded
for interdisciplinary work, in a domain
that does not regular publish in glossy
journals, I found my funding not with
national, but with the European
Commission
.
As a multidisciplinary researcher I had to wait long to become an expert. Effectively, you have to be expert in more than one field. An, as I am experiencing right now, staying expert is a whole other dimension. Bit with a solid chemistry, computing science, cheminformatics, and chemometrics education, I found myself versatile that I at some point landed that grant proposal (the first few failed). Though, I have had my share of travel grants and Google Summer of Code projects (microfunding).

So, while I am trying to establish a research line in metabolomics (also with some microfunding) and one PhD candidate (own funding), my main research line is nanosafety. Because my background fits in well, and while data quality for predictive toxicology leaves to be desired, there is a lot of work we can do here, to make the most of what is being measured.

Indeed, there are many interesting biological, chemical, and chemo-/bioinformatics research questions here (just to name a few):

  • does the mechanism of cell entry differ for different engineered nanomaterials?
  • does it differ from how "natural" nanomaterials enter the cell?
  • does the chemical composition of the nanomaterial change when it comes into contact with living matter? (yes, but how? is it stable?)
  • how do we represent the life cycle of nanomaterials in a meaning full way?
  • does each cell type respond in the same way to the same material? is this predominantly defined by the cell's epigenetics or by the chemical nature of the material?
  • given the sparseness of physicochemical and biological characterization of nanomaterials, what is the most appropriate representation of a material: based on physicochemical description, ontological description, or chemical graph theory?
  • can ontologies help us group data from different studies to give an over view of the whole process from molecular initiating event to adverse outcome?
  • can these insights be used to reliably and transparently advice the European people about risk?
We try to define answers to these questions in a series of FP7/H2020 projects using an Open Science approach, allowing our analyses to be updated frequently when new data or new knowledge comes in. These are the funded projects for which I am (was) PI:
  • eNanoMapper (EC FP7, ended, but since this project developed Open solutions, it is reused a lot)
  • NanoCommons (EC H2020, our work focussing on continuing the common ontology)
  • RiskGONE (EC H2020, focusing on reach regulatory advising based on scientific facts)
  • NanoSolveIT (EC H2020, computational nanosafety)
  • Sbd4Nano (EC H2020, disseminating nanosafety research to the EU industry)
In these projects (two PhD candidates, one postdoc), open science has been important to what we do. And while not all partners in all projects use Open Science approaches, what our groups does tries to be as Open as possible. Some open science projects involved:
If we want to read more about these projects in the scientific literature, check the websites which often have a page with publications and deliverables. Or check my Google Scholar profile. And for an overview of our group, see this page.

Posted Mon Aug 17 06:19:00 2020 Tags:

Alternate title: if you’re ahead of Clayton you’re doing well, at least for a few hours.

Consider the following code in the Kubernetes e2e test framework:

ctx, cancel := context.WithTimeout(context.Background(), csiPodRunningTimeout)
defer cancel()
pvcWatch, err := f.ClientSet.CoreV1().PersistentVolumeClaims(f.Namespace.Name).Watch(ctx, metav1.ListOptions{})
framework.ExpectNoError(err, "create PVC watch")
defer pvcWatch.Stop()

...

for {
	select {
	case event := <-pvcWatch.ResultChan():
		framework.Logf("PVC event %s: %#v", event.Type, event.Object)
		switch event.Type {
		case watch.Modified:
			pvc, ok := event.Object.(*v1.PersistentVolumeClaim)
			if !ok {
				framework.Failf("PVC watch sent %#v instead of a PVC", event.Object)
			}
			_, set := pvc.Annotations["volume.kubernetes.io/selected-node"]
			if set {
				nodeAnnotationSet = true
			} else if nodeAnnotationSet {
				nodeAnnotationReset = true
			}
		case watch.Deleted:
			break loop
		case watch.Error:
			// Can this occur when the apiserver is under heavy load?
			// If yes, then we should bail out of the test here early and
			// skip further checks instead of treating it as a test failure.
			framework.Failf("PVC watch failed prematurely: %v", event.Object)
		}
	case <-ctx.Done():
		framework.Failf("Timeout while waiting to observe PVC list")
	}
}

The problem is hard to spot unless you’re familiar with Kubernetes watches, and perhaps even if you are familiar but don’t work on OpenShift.

Watches and Channels

Watches can and do terminate at any time, gracefully or not. Sometimes a new apiserver leader is elected and the old one terminates watches and clients must reconnect to the new leader. Sometimes the leader just goes away because its node got rebooted. Sometimes there’s a network hiccup and the HTTP connection backing the watch times out. Regardless of the cause, they happen and your code needs to handle them. OpenShift CI forces frequent leader elections to specifically catch these issues before they get to customers.

A watch stuffs events into a Go channel. The code using the watch reads events out of the channel, usually in a for loop (to continuously grab events) with a select block (to ensure individual read operations don’t block which enables cancelation when the channel returned by ctx.Done() is closed). Reading from a Go channel (case event := <-pvcWatch.ResultChan()) returns an optional second boolean indicating whether the channel has been closed.

The testcase loop doesn’t exit until either the testcase times out and the ctx.Done() channel is closed, one of the event handler cases fails the testcase, or the PersistentVolumeClaim is deleted. So what happens if the Watch is closed unexpectedly and nothing checks whether the channel is closed?

The read returns a null value immediately. The code continuously executes the framework.Logf("PVC event %s: %#v", event.Type, event.Object) line for 30 seconds until the test terminates. Depending on how fast your machine is, this can be millions of log lines and lots of CPU.

How do we fix it?

  1. Assume watches can terminate at any time, and that you need to restart the watch if it does. That’s what the SharedInformer framework does for you.
  2. If you’re going to use watches directly always handle channel closure and restart your watch.
for {
	select {
	case event, ok := <-pvcWatch.ResultChan():
		if !ok {
			framework.Failf("PVC watch ended prematurely")
		}

		framework.Logf("PVC event %s: %#v", event.Type, event.Object)
		switch event.Type {
Posted Wed Aug 5 14:25:00 2020 Tags:

A few years ago I wrote The World in Which IPv6 was a Good Design. I'm still proud of that article, but I thought I should update it a bit.

No, I'm not switching sides. IPv6 is just as far away from universal adoption, or being a "good design" for our world, as it was three years ago. But since then I co-founded a company that turned out to be accidentally based on the principles I outlined in that article. Or rather, from turning those principles upside-down.

In that article, I explored the overall history of networking and the considerations that led to IPv6. I'm not going to cover that ground again. Instead, I want to talk about attitude.

Internets, Interoperability, and Postel's Law

Did you ever wonder why "Internet" is capitalized?

When I first joined the Internet in the 1990s, I found some now-long-lost introductory tutorial. It talked about the difference between an internet (lowercase i) and the Internet (capital I). An internet is "any network that connects smaller networks together." The Internet is... well... it turns out that you don't need more than one internet. If you have two internets, it is nearly unavoidable that someone will soon figure out how to connect them together. All you need is one person to build that one link, and your two internets become one. By induction then, the Internet is the end result when you make it easy enough for a single motivated individual to join one internet to another, however badly.

Internets are fundamentally sloppy. No matter how many committees you might form, ultimately connections are made by individuals plugging things together. Those things might follow the specs, or not. They might follow those specs well, or badly. They might violate the specs because everybody else is also violating the specs and that's the only way to make anything work. The connections themselves might be fast or slow, or flakey, or only functional for a few minutes each day, or subject to amateur radio regulations, or worse. The endpoints might be high-powered servers, vending machines, toasters, or satellites, running any imaginable operating system. Only one thing's for sure: they all have bugs.

Which brings us to Postel's Law, which I always bring up when I write about networks. When I do, invariably there's a slew of responses trying to debate whether Postel's Law is "right," or "a good idea," as if it were just an idea and not a force of nature.

Postel's Law says simply this: be conservative in what you send, and liberal in what you accept. Try your best to correctly handle the bugs produced by the other end. The most successful network node is one that plans for every "impossible" corruption there might be in the input and does something sensible when it happens. (Sometimes, yes, "something sensible" is to throw an error.)

[Side note: Postel's Law doesn't apply in every situation. You probably don't want your compiler to auto-fix your syntax errors, unless your compiler is javascript or HTML, which, kidding aside, actually were designed to do this sort of auto-correction for Postel's Law reasons. But the law does apply in virtually every complex situation where you need to communicate effectively, including human conversations. The way I like to say it is, "It takes two to miscommunicate." A great listener, or a skilled speaker, can resolve a lot of conflicts.]

Postel's Law is the principle the Internet is based on. Not because Jon Postel was such a great salesperson and talked everyone into it, but because that is the only winning evolutionary strategy when internets are competing. Nature doesn't care what you think about Postel's Law, because the only Internet that happens will be the one that follows Postel's Law. Every other internet will, without exception, eventually be joined to The Internet by some goofball who does it wrong, but just well enough that it adds value, so that eventually nobody will be willing to break the connection. And then to maintain that connection will require further application of Postel's Law.

IPv6: a different attitude

If you've followed my writing, you might have seen me refer to IPv6 as "a second internet that not everyone is connected to." There's a lot wrapped up in that claim. Let's back up a bit.

In The World in Which IPv6 was a Good Design, I talked about the lofty design goals leading to IPv6: eliminate bus networks, get rid of MAC addresses, no more switches and hubs, no NATs, and so on. What I didn't realize at the time, which I now think is essential, is that these goals were a fundamental attitude shift compared to what went into IPv4 (and the earlier protocols that led to v4).

IPv4 evolved as a pragmatic way to build an internet out of a bunch of networks and machines that existed already. Postel's Law says you'd best deal with reality as it is, not as you wish it were, and so they did. When something didn't connect, someone hacked on it until it worked. Sloppy. Fits and starts, twine and duct tape. But most importantly, nobody really thought this whole mess would work as well as it turned out to work, or last as long as it turned out to last. Nobody knew, at the time, that whenever you start building internets, they always lead inexorably to The Internet.

These (mostly) same people, when they started to realize the monster they had created, got worried. They realized that 32-bit addresses, which they had originally thought would easily last for the lifetime of their little internet, were not even enough for one address per person in the world. They found out, not really to anyone's surprise, that Postel's Law, unyielding as it may be, is absolutely a maintenance nightmare. They thought they'd better hurry up and fix it all, before this very popular Internet they had created, which had become a valuable, global, essential service, suddenly came crashing down and it would all be their fault.

[Spoiler: it never did come crashing down. Well, not permanently. There were and are still short-lived flare-ups every now and then, but a few dedicated souls hack it back together, and so it goes.]

IPv6 was created in a new environment of fear, scalability concerns, and Second System Effect. As we covered last time, its goal was to replace The Internet with a New Internet — one that wouldn't make all the same mistakes. It would have fewer hacks. And we'd upgrade to it incrementally over a few years, just as we did when upgrading to newer versions of IP and TCP back in the old days.

We can hardly blame people for believing this would work. Even the term "Second System Effect" was only about 20 years old at the time, and not universally known. Every previous Internet upgrade had gone fine. Nobody had built such a big internet before, with so much Postel's Law, with such a variety of users, vendors, and systems, so nobody knew it would be different.

Well, here we are 25 years later, and not much has changed. If we were feeling snarky, we could perhaps describe IPv6 as "the String Theory of networking": a decades-long boondoggle that attracts True Believers, gets you flamed intensely if you question the doctrine, and which is notable mainly for how much progress it has held back.

Luckily we are not feeling snarky.

Two Internets?

There are, of course, still no exceptions to the rule that if you build any internet, it will inevitably (and usually quickly) become connected to The Internet.

I wasn't sitting there when it happened, but it's likely the very first IPv6 node ran on a machine that was also connected to IPv4, if only so someone could telnet to it for debugging. Today, even "pure IPv6" nodes are almost certainly connected to a network that, if configured correctly, can find a way to any IPv4 node, and vice versa. It might not be pretty, it might involve a lot of proxies, NATs, bridges, and firewalls. But it's all connected.

In that sense, there is still just one Internet. It's the big one. Since day 1, The Internet has never spoken just one protocol; it has always been a hairy mess of routers, bridges, and gateways, running many protocols at many layers. IPv6 is one of them.

What makes IPv6 special is that its proponents are not content for it to be an internet that connects to The Internet. No! It's the chosen one. Its destiny is to be The Internet. As a result, we don't only have bridges and gateways to join the IPv6 internets and the IPv4 internet (although we do).

Instead, IPv6 wants to eventually run directly on every node. End users have been, uh, rather unwilling to give up IPv4, so for now, every node has that too. As a result, machines are often joined directly to what I call "two competing internets" --- the IPv4 one and the IPv6 one.

Okay, at this point our terminology has become very confusing. Sorry. But all this leads to the question I know you want me to answer: Which internet is better!?

Combinatorics

I'll get to that, but first we need to revisit what I bravely called Avery's Laws of Wifi Reliability, which are not laws, were surely invented by someone else (since they're mostly a paraphrasing of a trivial subset of CAP theorem), and as it turns out, apply to more than just wifi. Oops. I guess the name is wrong in almost every possible way. Still, they're pretty good guidelines.

Let's refresh:

  • Rule #1: if you have two wifi router brands that work with 90% of client devices, and your device has a problem with one of them, replacing the wifi router brand will fix the problem 90% of the time. Thus, an ISP offering both wifi routers has a [1 - (10% x 10%)] = 99% chance of eventual success.

  • Rule #2: if you're running two wifi routers at once (say, a primary router and an extender), and both of them work "correctly" for about 90% of the time each day, the chance that your network has no problems all day is 81%.

In Rule #1, which I call "a OR b", success compounds and failure rates drop.

In Rule #2, which I call "a AND b", failure compounds and success drops.

But wait, didn't we add redundancy in both cases?

Depending how many distributed systems you've had to build, this is either really obvious or really mind blowing. Why did the success rate jump to 99% in the first scenario but drop to 81% in the second? What's the difference? And... which one of those cases is like IPv6?

Failover

Or we can ask that question another way. Why are there so many web pages that advise you to solve your connectivity problem by disabling IPv6?

Because automatic failover is a very hard problem.

Let's keep things simple. IPv4 is one way to connect client A to server X, and IPv6 is a second way. It's similar to buying redundant home IPv4 connections from, say, a cable and a DSL provider and plugging them into the same computer. Either way, you have two independent connections to The Internet.

When you have two connections, you must choose between them. Here are some factors you can consider:

  • Which one even offers a path from A to X? (If X doesn't have an IPv6 address, for example, then IPv6 won't be an option.)

  • Which one gives the shortest paths from A to X and from X to A? (You could evaluate this using hopcount or latency, for example, like in my old netselect program.)

  • Which path has the most bandwidth?

  • Which path is most expensive?

  • Which path is most congested right now?

  • Which path drops out least often? (A rebooted NAT will drop a TCP connection on IPv4. But IPv6 routes change more frequently.)

  • Which one has buggy firewalls or NATs in the way? Do they completely block it (easy) or just act strangely (hard)?

  • Which one blocks certain UDP or TCP ports, intentionally or unintentionally?

  • Which one is misconfigured to block certain ICMP packets so that PMTU discovery (always or sometimes) doesn't work with some or all hosts?

  • Which one blocks certain kinds of packet fragmentation?

A common heuristic called "Happy Eyeballs" is one way to choose between routes, but it covers only a few of those criteria.

The truth is, it's extremely hard to answer all those questions, and even if you can, the answers are different for every combination of A and X, and they change over time. Operating systems, web browsers, and apps, even if they implement Happy Eyeballs or something equivalent, tend to be pretty bad at detecting all these edge cases. And every app has to do it separately!

My claim is that the "choose between two internets" problem is the same as the "choose between two flakey wifi routers on the same SSID" problem (Rule #2). All is well as long as both internets (or both wifi routers) are working perfectly. As soon as one is acting weird, your overall results are going to be weird.

...and the Internet always acts weird, because of the tyranny of Postel's Law. Debugging the Internet is a full time job.

...and now there are two internets, with a surprisingly low level of overlap, so your ISP has to build and debug both.

...and every OS vendor has to debug both protocol implementations, which is more than twice as much code.

...and every app vendor has to test with both IPv4 and IPv6, which of course they don't.

We should not be surprised that the combined system is less reliable.

The dream

IPv6 proponents know all this, whether rationally or intuitively or at least empirically. The failure rate of two wonky internets joined together is higher than the failure rate of either wonky internet alone.

This leads them to the same conclusion you've heard so many times: we should just kill one of the internets, so we can spend our time making the one remaining internet less wonky, instead of dividing our effort between the two. Oh, and, obviously the one we kill will be IPv4, thanks.

They're not wrong! It would be a lot easier to debug with just one internet, and you know, if we all had to agree on one, IPv6 is probably the better choice.

But... we don't all have to agree on one, because of the awesome unstoppable terribleness that is Postel's Law. Nobody can declare one internet or the other to be officially dead, because the only thing we know for sure about internets is that they always combine to make The Internet. Someone might try to unplug IPv4 or IPv6, but some other jerk will plug it right back in.

Purity cannot ever be achieved at this kind of scale. If you need purity for your network to be reliable, then you have an unsolvable problem.

The workaround

One thing we can do, though, is build better heuristics.

Ok, actually we have to do better than that, because it turns out that correctly choosing between the two internets for each connection, at the start of that connection, is not possible or good enough. Problems like PMTU, fragmentation, NAT resets, and routing changes can interrupt a connection partway through and cause poor performance or dropouts.

I want to go back to a side note I left near the end of The World in Which IPv6 was a Good Design: mobile IP. That is, the ability for your connections to keep going even if you hop between IP addresses. If you had IP mobility, then you could migrate connections between your two internets in real time, based on live quality feedback. You could send the same packets over both links and see which ones work better. If you picked one link and it suddenly stopped, you could retransmit packets on the other link and pick up where you left off. Your precise heuristic wouldn't even matter that much, as long as it tries both ways eventually.

If you had IP mobility, then you could convert the "a AND b" scenario (failure compounds) into the "a OR b" scenario (success compounds).

And you know what, forget about IPv4 and IPv6. The same tricks would work with that redundant cable + DSL setup we mentioned above. Or a phone with both wifi and LTE. Or, given a fancy enough wifi client chipset, smoothly switching between multiple unrelated wifi routers.

IP mobility is what we do, in a small way, with Tailscale's WireGuard connections. We try all your Internet links, IPv4 and IPv6, UDP and TCP, relayed and peer-to-peer. We made mobile IP a real thing, if only on your private network for now. And what do you know, the math works. Tailscale's use of WireGuard with two networks is more reliable than with one network.

Now, can it work for the whole Internet?

This article was originally posted to the Tailscale blog

Posted Tue Jul 21 08:10:06 2020 Tags:

[ This blog post was cross-posted from the blog at Software Freedom Conservancy where I work. ]

I've been concerned this week about aggressive negative reaction (by some) to the formation of an additional organization to serve the Free and Open Source (FOSS) community. Thus it seems like a good moment to remind everyone why we all benefit when we welcome newcomer organizations in FOSS.

I've been involved in helping found many different organizations — in roles as varied as co-founder, founding Board member, consultant, spin-off partner, and “just a friend giving advice”. Most of these organizations fill a variety of roles; they support, house, fiscally sponsor, or handle legal issues and/or trademark, copyright, or patent matters for FOSS projects. I and my colleagues at Conservancy speak regularly about why we believe a 501(c)(3) charitable structure in the USA has huge advantages, and you can find plenty of blog posts on our site about that. But you can also find us talking about how 501(c)(6) structures, and other structures outside the USA entirely, are often the right choices — depending on what a FOSS project seeks from its organization. Conservancy also makes our policies, agreements, and processes fully public so that organizations can reuse our work, and many have.

Meanwhile, FOSS organizations must avoid the classic “not invented here” anti-pattern. Of course I believe that Conservancy has great ideas for how to help FOSS, and our work — such as fiscal sponsorship, GPL enforcement work, and the Outreachy internship program — are the highest priorities in FOSS. I also believe the projects we take under our auspices are the most important projects in FOSS today.

But not everyone agrees with me, nor should they. Our Executive Director, Karen Sandler, loves the aphorism “let a thousand flowers bloom”. For example, when we learned of the launch of Open Collective, we at Conservancy were understandably concerned that since they were primarily a 501(c)(6) and didn't follow the kinds of fiscal sponsorship models and rules that we preferred, that somehow it was a “threat” to Conservancy. But that reaction is one of fear, selfishness, and insecurity. Once we analyzed what the Open Collective folks were up to, we realized that they were an excellent option for a lot of the projects that were simply not a good fit for Conservancy and our model. Conservancy is deeply steeped in a long-term focus on software freedom for the general public, and some projects — particularly those that are primarily in service to companies rather than individual users (or who don't want the oversight a charity requires) — just don't belong with us. We regularly refer projects to Open Collective.

For many larger projects, Linux Foundation — as a 501(c)(6) controlled completely by large technology companies — is also a great option. We've often referred Conservancy applicants there, too. We do that even while we criticize Linux Foundation for choosing proprietary software for many tasks, including proprietary software they write from scratch for their outward-facing project services

Of course, I'm thinking about all this today because Conservancy has been asked what we think about the Open Usage Commons. The fact is they're just getting started and both the legal details of how they're handling trademarks, and their governance documents, haven't been released yet. We should all give them an opportunity to slowly publish more and review it when it comes along. We should judge them fairly as an alternative for fulfilling FOSS project needs that no else addresses (or, more commonly are being addressed very differently by existing organizations). I'm going to hypothesize that, like Linux Foundation, Open Usage Commons will primarily be of interest to more for-profit-company focused projects, but that's my own speculation; none of us know yet.

No one is denying that Open Usage Commons is tied to Google as part of their founding — in the same way that Linux Foundation's founding (which was originally founded as the “Open Source Development Labs”) was closely tied to IBM at the time. As near as I can tell, IBM's influence over Linux Foundation is these days no more than any other of their Platinum Members. It's not uncommon for a trade association to jumpstart with a key corporate member and eventually grow to be governed by a wider group of companies. But while appropriately run trade associations do balance the needs of all for-profit companies in their industry, they are decidedly not neutral; they are chartered to favor business needs over the needs of the general public. I encourage skepticism when you hear an organization claim “neutrality”. Since a trade association is narrowed to serving businesses, it can be neutral among the interests of business, but their mandate remains putting business needs above community. The ultimate proof of neutrality pudding is in the eating. As with multi-copyright held GPL'd projects, we can trust the equal rights for all in those — regardless of the corporate form of the contributors — because the document of legal rights makes it so. The same principle applies to any area of FOSS endeavor: examine the agreements and written rules for contributors and users to test neutrality.

Finally, there are plenty of issues where software freedom activists should criticize Google. Just today, I was sent a Google Docs link for a non-FOSS volunteer thing I'm doing, and I groaned knowing that I'd have to install a bunch of proprietary Javascript just to be able to participate. Often, software freedom activists assume that bad actions by an entity means all actions are de-facto problematic. But we must judge each policy move on its own merits to avoid pointless partisanship.

Posted Thu Jul 9 11:30:00 2020 Tags:

This is the continuation from this post.

Several moons have bypassed us [1] in the time since the first post, and Things Have Happened! If you recall (and of course you did because you just re-read the article I so conveniently linked above), libxkbcommon supports an include directive for the rules files and it will load a rules file from $XDG_CONFIG_HOME/xkb/rules/ which is the framework for custom per-user keyboard layouts. Alas, those files are just sitting there, useful but undiscoverable.

To give you a very approximate analogy, the KcCGST format I described last time are the ingredients to a meal (pasta, mince, tomato). The rules file is the machine-readable instruction set to assemble your meal but it relies a lot on wildcards. Feed it "spaghetti, variant bolognese" and the actual keymap ends up being the various components put together: "pasta(spaghetti)+sauce(tomato)+mince". But for this to work you need to know that spag bol is available in the first place, i.e you need the menu. This applies to keyboard layouts too - the keyboard configuration panel needs to present a list so the users can clickedy click-click on whatever layout they think is best for them.

This menu of possible layouts is provided by the xkeyboard-config project but for historical reasons [2], it is stored as an XML file named after the ruleset: usually /usr/share/X11/xkb/rules/evdev.xml [3]. Configuration utilities parse that file directly which is a bit of an issue when your actual keymap compiler starts supporting other include paths. Your fancy new layout won't show up because everything insists on loading the system layout menu only. This bit is part 2, i.e. this post here.

If there's one thing that the world doesn't have enough of yet, it's low-level C libraries. So I hereby present to you: libxkbregistry. This library has now been merged into the libxkbcommon repository and provides a simple C interface to list all available models, layouts and options for a given ruleset. It sits in the same repository as libxkbcommon - long term this will allow us to better synchronise any changes to XKB handling or data formats as we can guarantee that the behaviour of both components is the same.

Speaking of data formats, we haven't actually changed any of those which means they're about as easy to understand as your local COVID19 restrictions. In the previous post I outlined the example for the KcCGST and rules file, what you need now with libxkbregistry is an XKB-compatible XML file named after your ruleset. Something like this:


$ cat $HOME/.config/xkb/rules/evdev.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE xkbConfigRegistry SYSTEM "xkb.dtd">
<xkbConfigRegistry version="1.1">
<layoutList>
<layout>
<configItem>
<name>us</name>
</configItem>
<variantList>
<variant>
<configItem>
<name>banana</name>
<shortDescription>banana</shortDescription>
<description>US (Banana)</description>
</variant>
</variantList>
</layoutList>
<optionList>
<group allowMultipleSelection="true">
<configItem>
<name>custom</name>
<description>Custom options</description>
</configItem>
<option>
<configItem>
<name>custom:foo</name>
<description>Map Tilde to nothing</description>
</configItem>
</option>
<option>
<configItem>
<name>custom:baz</name>
<description>Map Z to K</description>
</configItem>
</option>
</group>
</optionList>
</xkbConfigRegistry>
This looks more complicated than it is: we have models (not shown here), layouts which can have multiple variants and options which are grouped together in option group (to make options mutually exclusive). libxkbregistry will merge this with the system layouts in what is supposed to be the most obvious merge algorithm. The simple summary of that is that you can add to existing system layouts but you can't modify those - the above example will add a "banana" variant to the US keyboard layout without modifying "us" itself or any of its other variants. The second part adds two new options based on my previous post.

Now, all that is needed is to change every user of evdev.xml to use libxkbregistry. The gnome-desktop merge request is here for a start.

[1] technically something that goes around something else doesn't bypass it but the earth is flat, the moon is made of cheese, facts don't matter anymore and stop being so pedantic about things already!
[2] it's amazing what you can handwave away with "for historical reasons". Life would be better if there was less history to choose reasons from.
[3] there's also evdev.extras.xml for very niche layouts which is a separate file for historical reasons [2], despite there being a "popularity" XML attribute

Posted Mon Jul 6 05:18:00 2020 Tags:

In Part 1 I've shown you how to create your own distribution image using the freedesktop.org CI templates. In Part 2, we'll go a bit further than that by truly embracing nested images.

Our assumption here is that we have two projects (or jobs), with the second one relying heavily on the first one. For example, the base project and a plugin, or a base project and its language bindings. What we'll get out of this blog post is a setup where we have

  • a base image in the base project
  • an image extending that base image in a different project
  • automatic rebuilds of that extended image when the base image changes
And none of your contributors have to care about this. It's all handled automatically and filing a MR against a project will build against the right image. So let's get started.

Our base project has CI that pushes an image to its registry. The .gitlab-ci.yml contains something like this:


.fedora32:
variables:
FDO_DISTRIBUTION_VERSION: '32'
FDO_DISTRIBUTION_TAG: 'base.0'

build-img:
extends:
- .fedora32
- .fdo.container-build@fedora
variables:
FDO_DISTRIBUTION_PACKAGES: "curl wget"

This will build a fedora/32:base.0 image in the project's container registry. That image is built once and then re-used by any job extending .fdo.distribution-image@fedora. So far, so Part 1.

Now, the second project needs to test things on top of this base image, for example language bindings for rust. You want to use the same image that the base project uses (and has successfully completed its CI on) but you need some extra packages or setup. This is where the FDO_BASE_IMAGE comes in. In our dependent project, we have this:


.fedora32:
variables:
FDO_DISTRIBUTION_VERSION: '32'
FDO_DISTRIBUTION_TAG: 'rust.0'

build-rust-image:
extends:
- .fedora32
- .fdo.container-build@fedora
variables:
FDO_BASE_IMAGE: "registry.freedesktop.org/baseproject/name/fedora/32:base.0"
# extra packages we want to install and things we need to set up
FDO_DISTRIBUTION_PACKAGES: "rust cargo"
FDO_DISTRIBUTION_EXEC: "bash -x some-setup-script.sh"

test-rust:
extends:
- .fedora32
- .fdo.distribution-image@fedora
script:
- cargo build myproject-bindings

And voila, you now have two images: the base image with curl and wget in the base project and an extra image with rust and cargo in the dependent project. And all that is required is to reference the FDO_BASE_IMAGE, everything else is the same. Note how the FDO_BASE_IMAGE is a full path in this example since we assume it's in a different project. For dependent images within the same project, you can just use the image path without the host.

The dependency only matters while the image is built, after that the dependent image is just another standalone image. So even if the base project removes the base image, you still have yours to test on.

But eventually you will need to change the base image and you want the dependent image to update as well. The best solution here is to have a CI job as part of the base repo that pokes the dependent repo's CI whenever the base image updates. The CI templates add the pipeline id as label to an image when it is built. In your base project, you can thus have a job like this:


poke-dependents:
extends:
- .fedora32
- .fdo.distribution-image@fedora
image: something-with-skopeo-and-jq
script:
# FDO_DISTRIBUTION_IMAGE still has indirections
- DISTRO_IMAGE=$(eval echo ${FDO_DISTRIBUTION_IMAGE})
# retrieve info from the registry and extract the pipeline id
- JSON_IMAGE=$(skopeo inspect docker://$DISTRO_IMAGE)
- IMAGE_PIPELINE_ID=$(echo $JSON_IMAGE | jq -r '.Labels["fdo.pipeline_id"]')
- |
if [[ x"$IMAGE_PIPELINE_ID" == x"$CI_PIPELINE_ID" ]]; then
curl -X POST
-F "token=$AUTH_TOKEN_VALUE"
-F "ref=master"
-F "variables[SOMEVARIABLE]=somevalue"
https://gitlab.freedesktop.org/api/v4/projects/dependent${SLASH}project/trigger/pipeline
fi
variables:
SLASH: "%2F"

Let's dissect this: First, we use the .fdo.distribution-image@fedora template to get access to FDO_DISTRIBUTION_IMAGE. We don't need to use the actual image though, anything with skopeo and jq will do. Then we fetch the pipeline id label from the image and compare it to the current pipeline ID. If it is the same, our image was rebuilt as part of the pipeline and we poke the other project's pipeline with a SOMEVARIABLE set to somevalue. The auth token is a standard GitLab token you need to create to allow triggering the pipeline in the dependent project.

In that dependent project you can have a job like this:


rebuild-extra-image:
extends: build-extra-image
rules:
- if: '$SOMEVARIABLE == "somevalue"'
variables:
FDO_FORCE_REBUILD: 1

This job is only triggered where the variable is set and it will force a rebuild of the container image. If you want custom rebuilds of images, set the variables accordingly.

So, as promised above, we now have a base image and a separate image building on that, together with auto-rebuild hooks. The gstreamer-plugins-rs project uses this approach. The base image is built by gstreamer-rs during its CI run which then pokes gstreamer-plugins-rs to rebuild selected dependent images.

The above is most efficient when the base project knows of the dependent projects. Where this is not the case, the dependent project will need a scheduled pipeline to poll the base project and extract the image IDs from that, possibly using creation dates and whatnot. We'll figure that out when we have a use-case for it.

Posted Wed Jun 17 04:36:00 2020 Tags:

TL;DR: exchanges are casinos and don’t want to onboard anyone into bitcoin. Avoid.

There’s a classic scam in the “crypto” space: advertize Bitcoin to get people in, then sell suckers something else entirely. Over the last few years, this bait-and-switch has become the core competency of “bitcoin” exchanges.

I recently visited the homepage of Australian exchange btcmarkets.net: what a mess. There was a list of dozens of identical-looking “cryptos”, with bitcoin second after something called “XRP”; seems like it was sorted by volume?

Incentives have driven exchanges to become casinos, and they’re doing exactly what you’d expect unregulated casinos to do. This is no place you ever want to send anyone.

Incentives For Exchanges

Exchanges make money on trading, not on buying and holding. Despite the fact that bitcoin is the only real attempt to create an open source money, scams with no future are given false equivalence, because more assets means more trading. Worse than that, they are paid directly to list new scams (the crappier, the more money they can charge!) and have recently taken the logical step of introducing and promoting their own crapcoins directly.

It’s like a gold dealer who also sells 57 varieties of pyrite, which give more margin than selling actual gold.

For a long time, I thought exchanges were merely incompetent. Most can’t even give out fresh addresses for deposits, batch their outgoing transactions, pay competent fee rates, perform RBF or use segwit.

But I misunderstood: they don’t want to sell bitcoin. They use bitcoin to get you in the door, but they want you to gamble. This matters: you’ll find subtle and not-so-subtle blockers to simply buying bitcoin on an exchange. If you send a friend off to buy their first bitcoin, they’re likely to come back with something else. That’s no accident.

Looking Deeper, It Gets Worse.

Regrettably, looking harder at specific exchanges makes the picture even bleaker.

Consider Binance: this mainland China backed exchange pretending to be a Hong Kong exchange appeared out of nowhere with fake volume and demonstrated the gullibility of the entire industry by being treated as if it were a respected member. They lost at least 40,000 bitcoin in a known hack, and they also lost all the personal information people sent them to KYC. They aggressively market their own coin. But basically, they’re just MtGox without Mark Karpales’ PHP skills or moral scruples and much better marketing.

Coinbase is more interesting: an MBA-run “bitcoin” company which really dislikes bitcoin. They got where they are by spending big on regulations compliance in the US so they could operate in (almost?) every US state. (They don’t do much to dispel the wide belief that this regulation protects their users, when in practice it seems only USD deposits have any guarantee). Their natural interest is in increasing regulation to maintain that moat, and their biggest problem is Bitcoin.

They have much more affinity for the centralized coins (Ethereum) where they can have influence and control. The anarchic nature of a genuine open source community (not to mention the developers’ oft-stated aim to improve privacy over time) is not culturally compatible with a top-down company run by the Big Dog. It’s a running joke that their CEO can’t say the word “Bitcoin”, but their recent “what will happen to cryptocurrencies in the 2020s” article is breathtaking in its boldness: innovation is mainly happening on altcoins, and they’re going to overtake bitcoin any day now. Those scaling problems which the Bitcoin developers say they don’t know how to solve? This non-technical CEO knows better.

So, don’t send anyone to an exchange, especially not a “market leading” one. Find some service that actually wants to sell them bitcoin, like CashApp or Swan Bitcoin.

Posted Wed May 27 00:49:34 2020 Tags:

One of the more common issues we encounter debugging things is that users don't always know whether they're running on a Wayland or X11 session. Which I guess is a good advertisement for how far some of the compositors have come. The question "are you running on Xorg or Wayland" thus comes up a lot and suggestions previously included things like "run xeyes", "grep xinput list", "check xrandr" and so on and so forth. None of those are particularly scriptable, so there's a new tool around now: xisxwayland.

Run without arguments it simply exits with exit code 0 if the X server is Xwayland, or 1 otherwise. Which means use can use it like this:


$ cat my-xorg-only-script.sh
#!/bin/bash

if xisxwayland; then
echo "This is an Xwayland server!";
exit 1
fi

...
Or, in the case where you have a human user (gasp!), you can ask them to run:

$ xisxwayland --verbose
Xwayland: YES
And even non-technical users should be able to interpret that.

Note that the script checks for Xwayland (hence the name) via the $DISPLAY environment variable, just like any X application. It does not check whether there's a Wayland compositor running but for most use-cases this doesn't matter anyway. For those where it matters you get to write your own script. Congratulations, I guess.

Posted Tue May 19 02:15:00 2020 Tags:

Gather round children, it's analogy time! First, some definitions:

  • Wayland is a protocol to define the communication between a display server (the "compositor") and a client, i.e. an application though the actual communication is usually done by a toolkit like GTK or Qt.
  • A Wayland Compositor is an implementation of a display server that (usually but not necessary) handles things like displaying stuff on screen and handling your input devices, among many other things. Common examples for Wayland Compositors are GNOME's mutter, KDE's KWin, weston, sway, etc.[1]

And now for the definitions we need for our analogy:

  • HTTP is a protocol to define the communication between a web server and a client (usually called the "browser")
  • A Web Browser is an implementation that (sometimes but not usually) handles things like displaying web sites correctly, among many other things. Common examples for Web Browsers are Firefox, Chrome, Edge, Safari, etc. [2]

And now for the analogy:

The following complaints are technically correct but otherwise rather pointless to make:

  • HTTP doesn't support CSS
  • HTTP doesn't support adblocking
  • HTTP doesn't render this or that website correctly
And much in the same style, the following complaints are technically correct but otherwise rather pointless to make:
  • Wayland doesn't support keyboard shortcuts
  • Wayland doesn't support screen sharing
  • Wayland doesn't render this or that application correctly
In most cases you may encounter (online or on your setup), saying "Wayland doesn't support $BLAH" is like saying "HTTP doesn't support $BLAH". The problem isn't in with Wayland itself, it's a missing piece or bug in the compositor you're using.

Likewise, saying "I don't like Wayland" is like saying "I don't like HTTP".The vast majority of users will have negative feelings towards the browser, not the transport protocol.

[1] Because they're implementations of a display server they can speak multiple protocols and some compositors can also be X11 window managers, much in the same way as you can switch between English and your native language(s).[2] Because they're implementations of a web browser they can speak multiple protocols and some browsers can also be FTP file managers, much in the same way as... you get the point

Posted Fri May 8 00:40:00 2020 Tags:

Four people now live and work in my home 24×7; my wife Andi, her mother, my daughter and myself. Many of you now live in similar situations.jigsawfish2

Very occasionally, everyone will have network trouble, such as occurred to us this morning. Sometimes it is our “last mile” connection: it is easy to see these failures in our cable modem log. (Often available by looking at the address 192.168.100.1, which seems to be the default address for cable modems.). Occasionally it can be the ISP (in our case, Comcast), either due to some routing failure or DNS failure. These can be harder to diagnose.

Bufferbloat, however, is insidious. It comes and goes, and most users have been “trained” to ignore temporary bad behavior over many years. When you go to diagnose it, you usually stop the operation that is causing it. This blog has recorded our efforts to fix bufferbloat. Now that there are many more people at home at the same time trying to do more demanding applications, this problem is much more common. Other people in your home can inflict the bufferbloat problem on you without you or they understanding what is happening.

Yesterday afternoon my I was helping my wife learn how to edit videos that now makes and uploads for her students, and one of her co-workers, who we’ll call  “Carleen”, called on a video to chat on planning classes. Those of you who wonder what teachers do when not teaching should understand that not only do teachers grade materials and give other feedback to students, but also spend much of their time preparing future classes. The amount of “prep” time diminishes the more often a good teacher has taught a given subject, but never drops to anything close to zero. At the moment, learning to teach remotely is an extreme extra burden.

In the first phase of learning from home, assignments to students consisted of watching videos and performing on line work sheets. None of these activities are latency sensitive. As of this week, however, the next phase of instruction includes teachers attempting more “conventional” instruction via on-line teleconferencing (Google Meet, in their case). Teleconferencing is by its nature very latency sensitive.

Carleen shares her home with her husband and two sons of age seven and ten. She commented that yesterday she her class had been aborted entirely due to intermittent network problems caused by her kids game playing. She has now banished her kids from their video games during her class times, but this may be worse than the disease with two bored children in the house; only time will tell. Transient problems caused by bufferbloat now really matter to her, and to her class.

In my early bufferbloat talks, I called bufferbloat the “The Internet is Slow Today, Daddy” problem. But anyone can inflict pain on others sharing a connection if bufferbloat is present; it is just as much a “Kids, quit what you are doing so I can teleconference” problem. Thankfully, I can now make concrete recommendations on how you can solve bufferbloat in your environment. My “go to” recommendation for non-geeks is currently the EvenRoute IQRouter. This device will both mitigate bufferbloat in your “last mile”, and fix it in the WiFi link to your device, without any manual tuning. At this moment, I am not aware of any other home router that deals with bufferbloat both in the “last mile” and in WiFi; either or both can be a problem at any given instant..

Needless to say, I recommended that Carleen buy one of those routers.

I wish that all ISP’s would increase the upstream bandwidth of all connections to teachers for the duration of the Covid-19 crisis, which would help diminish the bufferbloat problem (bufferbloat is generally most severe in the upstream link, and many teachers cannot afford expensive tiers of Internet service). Those hurt most by bufferbloat are those with the most minimal service; not only impecunious teachers, but also the students whose parents are least able to afford higher tier internet service.

A few network speed test sites have tests for bufferbloat, such as DSLReports Speedtest but there are many ways to text explicitly for bufferbloat, as outlined here.

Posted Wed Apr 22 16:08:53 2020 Tags:

This is a follow up from the kernel support for high-resolution wheel scrolling which you totally forgot about because it's already more then a year in the past and seriously, who has the attention span these days to remember this. Anyway, I finally found time and motivation to pick this up again and I started lining up the pieces like cans, for it only to be shot down by the commentary of strangers on the internet. The Wayland merge request lists the various pieces (libinput, wayland, weston, mutter, gtk and Xwayland) but for the impatient there's also an Fedora 32 COPR. For all you weirdos inexplicably not running the latest Fedora, well, you'll have to compile this yourself, just like I did.

Let's recap: in v5.0 the kernel added new axes REL_WHEEL_HI_RES and REL_HWHEEL_HI_RES for all devices. On devices that actually support high-resolution wheel scrolling (Logitech and Microsoft mice, primarily) you'll get multiple hires events before the now-legacy REL_WHEEL events. On all other devices those two are in sync.

Integrating this into the userspace stack was a bit of a mess at first, but I think the solution is good enough, even if it has a rather verbose explanation on how to handle it. The actual patches to integrate ended up being relatively simple. So let's see why it's a bit weird:

When Wayland started, back in WhoahReallyThatLongAgo, scrolling was specified as the wl_pointer.axis event with a value in pixels. This works fine for touchpads, not so much for wheels. The early versions of Weston decreed that one wheel click was 10 pixels [1] and, perhaps surprisingly, the world kept on turning. When libinput was forked from Weston an early change was that wheel events would have two values - degrees of movement and click count ("discrete steps"). The wayland protocol was expanded to include the discrete steps as wl_pointer.axis_discrete as well. Then backwards compatibility reared its ugly head and Mutter, Weston, GTK all basically said: one discrete step equals 10 pixels so we multiply the discrete value by 10 and, perhaps surprisingly, the world kept on turning.

This worked out well enough for a few years but with high resolution wheels we ran into a problem. Discrete steps are integers, so we can't send partial values. And the protocol is defined in a way that any tweaking of the behaviour would result in broken clients which, perhaps surprisingly, is a Bad Thing. This lead to the current proposal of separate events. LIBINPUT_EVENT_POINTER_AXIS_WHEEL and for Wayland the wl_pointer.axis_v120 event, linked to above. These events are (like the kernel events) a parallel event stream to the previous events and effectively replace the LIBINPUT_EVENT_POINTER_AXIS and Wayland wl_pointer.axis/axis_discrete pair for wheel events (not so for touchpad or button scrolling though).

The compositor side of things is relatively simple: take the events from libinput and pass the hires ones as v120 events and the lowres ones as v120 events with a value of zero. The client side takes the v120 events and uses them over wl_pointer.axis/axis_discrete unless one is zero in which case you can discard all axis events in that wl_pointer.frame. Since most client implementation already have the support for smooth scrolling (because, well, touchpads do exist) it's relatively simple to integrate - the new events just feed into the smooth scrolling code. And since you already have to do wheel emulation for that (because, well, old clients exist) wheel emulation is handled easily too.

All that to provide buttery smooth [2] wheel scrolling. Or not, if your hardware doesn't support it. In which case, well, live with the warm fuzzy feeling that someone else has a better user experience now. Or soon, anyway.

[1] with, I suspect, the scientific measurement of "yeah, that seems about alright"
[2] like butter out of a fridge, so still chunky but at least less so than before

Posted Sat Apr 4 04:00:00 2020 Tags:

Yak Shaving - Swift Edition

At the TensorFlow summit last year, I caught up with Chris Lattner who was at the time working on Swift for TensorFlow - we ended up talking about concurrency and what he had in mind for Swift.

I recognized some of the actor ideas to be similar to those from the Pony language which I had learned about just a year before on a trip to Microsoft Research in the UK. Of course, I pointed out that Pony had some capabilities that languages like C# and Swift lacked and that anyone could just poke at data that did not belong to them without doing too much work and the whole thing would fall apart.

For example, if you build something like this in C#:

class Chart {
  float [] points;
  public float [] Points { get { return points; } }
}

Then anyone with a reference to Chart can go and poke at the internals of the points array that you have surfaced. For example, this simple Plot implementation accidentally modifies the contents:

void Plot (Chart myChart)
{
   // This code accidentally modifies the data in myChart
   var p = myChart.points;
   for (int i = 0; i < p.Length; i++) {
       Plot (0, p [i]++)
   }
}

This sort of problem is avoidable, but comes at a considerable development cost. For instance, in .NET you can find plenty of ad-hoc collections and interfaces whose sole purpose is to prevent data tampering/corruption. If those are consistently and properly used, they can prevent the above scenario from happening.

This is where Chris politely pointed out to me that I had not quite understood Swift - in fact, Swift supports a copy-on-write model for its collections out of the box - meaning that the above problem is just not present in Swift as I had wrongly assumed.

It is interesting that I had read the Swift specification some three or four times, and I was collaborating with Steve on our Swift-to-.NET binding tool and yet, I had completely missed the significance of this design decision in Swift.

This subtle design decision was eye opening.

It was then that I decided to gain some real hands-on experience in Swift. And what better way to learn Swift than to start with a small, fun project for a couple of evenings.

Rather than building a mobile app, which would have been 90% mobile design and user interaction, and little Swift, I decided to port my gui.cs console UI toolkit from C# to Swift and called it TermKit.

Both gui.cs and TermKit borrow extensively from Apple’s UIKit design - it is a design that I have enjoyed. It notably avoids auto layout, and instead uses a simpler layout system that I quite love and had a lot of fun implementing (You can read a description of how to use it in the C# version).

This journey was filled with a number of very pleasant capabilities in Swift that helped me find some long-term bugs in my C# libraries. I remain firmly a fan of compiled languages, and the more checking, the better.

Dear reader, I wish I had kept a log of those but that is now code that I wrote a year ago so I could share all of those with you, but I did not take copious notes. Suffice to say, that I ended up with a warm and cozy feeling - knowing that the compiler was looking out for me.

There is plenty to love about Swift technically, and I will not enumerate all of those features, other people have done that. But I want to point out a few interesting bits that I had missed because I was not a practitioner of the language, and was more of an armchair observer of the language.

The requirement that constructors fully initialize all the fields in a type before calling the base constructor is a requirement that took me a while to digest. My mental model was that calling the superclass to initialize itself should be done before any of my own values are set - this is what C# does. Yet, this prevents a bug where the base constructor can call a virtual method that you override, and might not be ready to handle. So eventually I just learned to embrace and love this capability.

Another thing that I truly enjoyed was the ability of creating a typealias, which once defined is visible as a new type. A capability that I have wanted in C# since 2001 and have yet to get.

I have a love/hate relationship with Swift protocols and extensions. I love them because they are incredibly powerful, and I hate them, because it has been so hard to surface those to .NET, but in practice they are a pleasure to use.

What won my heart is just how simple it is to import C code into Swift

  • to bring the type definitions from a header file, and call into the C code transparently from Swift. This really is a gift of the gods to humankind.

I truly enjoyed having the Character data type in Swift which allowed my console UI toolkit to correctly support Unicode on the console for modern terminals.

Even gui.cs with my port of Go’s Unicode libraries to C# suffers from being limited to Go-style Runes and not having support for emoji (or as the nerd-o-sphere calls it “extended grapheme clusters”).

Beyond the pedestrian controls like buttons, entry lines and checkboxes, there are two useful controls that I wanted to develop. An xterm terminal emulator, and a multi-line text editor.

In the C# version of my console toolkit my multi-line text editor was a quick hack. A List<T> holds all the lines in the buffer, and each line contains the runes to display. Inserting characters is easy, and inserting lines is easy and you can get this done in a couple of hours on the evening (which is the sort of time I can devote to these fun explorations). Of course, the problem is cutting regions of text across lines, and inserting text that spans multiple lines. Because what looked like a brilliant coup of simple design, turns out to be an ugly, repetitive and error-prone code that takes forever to debug - I did not enjoy writing that code in the end.

For my Swift port, I decided that I needed something better. Of course, in the era of web scale, you gotta have a web scale data structure. I was about to implement a Swift version of the Rope data structure, when someone pointed to me a blog post from the Visual Studio Code team titled “Text Buffer Reimplementation”. I read it avidly, founds their arguments convincing, and in the end, if it is good enough for Visual Studio Code, it should be good enough for the gander.

During my vacation last summer, I decided to port the TypeScript implementation of the Text Buffer to Swift, and named it TextBufferKit. Once again, porting this code from TypeScript to Swift turned out to be a great learning experience for me.

By the time I was done with this and was ready to hook it up to TermKit, I got busy, and also started to learn SwiftUI, and started to doubt whether it made sense to continue work on a UIKit-based model, or if I should restart and do a SwiftUI version. So while I pondered this decision, I did what every other respected yak shaver would do, I proceeded to my xterm terminal emulator work.

Since about 2009 or so, I wanted to have a reusable terminal emulator control for .NET. In particular, I wanted one to embed into MonoDevelop, so a year or two ago, I looked for a terminal emulator that I could port to .NET - I needed something that was licensed under the MIT license, so it could be used in a wide range of situations, and was modern enough. After surveying the space, I found “xterm.js” fit the bill, so I ported it to .NET and modified it to suit my requirements. XtermSharp - a terminal emulator engine that can have multiple UIs and hook up multiple backends.

For Swift, I took the XtermSharp code, and ported it over to Swift, and ended up with SwiftTerm. It is now in quite a decent shape, with only a few bugs left.

I have yet to built a TermKit UI for SwiftTerm, but in my quest for the perfect shaved yak, now I need to figure out if I should implement SwiftUI on top of TermKit, or if I should repurpose TermKit completely from the ground up to be SwiftUI driven.

Stay tuned!

Posted Wed Mar 25 08:52:00 2020 Tags: