The lower-post-volume people behind the software in Debian. (List of feeds.)

I just got back from the 2014 World Boardgaming Championships in Lancaster, PA. This event is the “brain” half of the split vacation my wife Cathy and I generally take every year, the “brawn” half being summer weapons camp. WBC is a solid week of tournament tracks in strategy and war games of all kinds, with a huge open pickup gaming scene as well. People fly in from all over the planet to be here – takes less effort for us as the venue is about 90 minutes straight west of where we live.

Cathy and I aren’t – quite – steady world-championship-level players. I did make the Power Grid finals (top 5 players) two years ago, but have been unable to replicate that feat since. Usually we do make quarter-finals (top 64 to 125 players) or semi-finals (top 16 to 25) in a couple of our favorite games and then lose by the slimmest of margins to people just a hair better (or at least more obsessive) than we are. That’s pretty much what happened this year.

I’m not going to do a blow-by-blow of my whole week, but rather try to hit the dramatic high spots in a way I hope will convey something of the flavor to people who aren’t deeply immersed in tournament gaming. I think the best way to do that is to organize by game rather than chronology. The section titles link to game descriptions.

Terra Mystica

I’ve been enjoying this one a lot lately and was very pleased to be able to fit a pickup game in on the first night. Three to six players, 2.5-3 hours, fantasy-themed – contending factions with magical powers trying for interlocking levels of area control on a multicolored hex grid.

This game is strategically deep and tricky to play – very “crunchy” in gamer slang. Suits me fine; I like my games super-crunchy, which is an elite taste even among strategy gamers. If Terra Mystica becomes a WBC tournament game (which I think is extremely likely to happen within two years) a trophy in it will earn more respect than one in a lighter, “fluffier” game.

Some of you may be entertained to know that my joke name for this one is “Eclipse: The Gathering”. For the rest of you, this hints at similarities to a game (Eclipse) I often play, and another (Magic: The Gathering) that I used to play.

The one flaw the game seems to have is one that’s common in games with asymmetrical player powers; the factions aren’t quite balanced, with some chronically stronger or weaker than average (this sort of thing can slip through even careful playtesting sometimes). The Darklings, for example, are often said to be about the winningest side; my one time playing them I did very well, pulling a strong second.

This was about my fifth play of Terra Mystica, maybe fourth. This time I drafted the Engineers – I’m trying to get to playing every one of the 14 factions. I cannot recommend them. I had to work hard to pull second even though all the other players were less experienced than me; the Engineers have real trouble getting enough workers to expand even though my first couple of actions were the expensive ones required to reduce my terraforming cost to 1 worker. Copping the bonus for most area controlled and maxing out the Air cult track helped a lot.

Commands & Colors: Ancients

I love ancient-period wargames. Phalanxes, peltasts, barbarians, war elephants – I actually prefer a straight historical to fantasy-themed stuff. I’d say my favorite single period is the wars of the Diadochi (lots and lots of war elephants, hurray!), but anything set from the late Bronze Age to the fall of the Western Roman Empire will easily catch my interest.

Commands & Colors: Ancients is, in my opinion, one of the best game systems ever devised for this span of time. While not as crunchy as some of the older simulationist games like the PRESTAGS system, you will get authentic results when you use period tactics. Knowing what historical generals did, and having some idea why, actually helps significantly. There are expansions and scenarios for hundreds of different historical battles.

Alas, the game is flawed for tournament play. The problem with it is that when two highly skilled players meet, they can counter each others’ tactics so well that the outcome comes down to who gets good breaks on the battle dice. I’m quite a good player, but I had skipped competing at WBC for the last few years because I found it too irritating to lose to the dice rather than my opponents.

This year, however, I had a hole in my WBC schedule where the C&C:A tournament was and decided to give it another try. The scenario was the battle of Bagradas, Carthaginians vs. Romans during the First Punic War. With elephants!

Three games later I had: 1 narrow loss to a player who afterwards shook his head and said “You played that very well, I just got better dice”; 1 solid win against a player not quite as good as me; and one heartbreaker of a loss to a player about my equal where we both knew it came down to one failed die roll on my attempted final kill – which, by the odds, I should have made.

That wasn’t good enough to get me to the second round. And it was just about what I expected from my previous experience; the tournament game is a crapshoot in which it’s not enough to be good, you also have to be lucky. I prefer games in which. if there’s a random element at all, luck is nevertheless relatively unimportant.

I’ll probably sit out C&C:A next year.

Ticket To Ride

TTR is a railroad game in which you build track and connect cities to make victory points. It’s relatively fluffy, a “family game”, but has enough strategy so that serious gamers will play it as a light diversion when circumstances aren’t right for something crunchier.

I am difficult to beat at the Europe variant, which I like better than the American map; the geography creates more tactical complexities. In my first heat I kerb-stomped the other three players, coming in 19 points ahead of 2nd and sweeping every scoring category and bonus.

The second heat looked like it was going to go the same way. I built both monster tunnels (St. Petersburg-Stockholm and Kyiv-Budapest) on succeeding turns for a 36-point power play, then successfully forced an early game end in order to leave the other players with unused trains (and thus unscored points). When we started endgame scoring everyone at the table thought I had the win locked in.

Sadly, in order to get rid of my own train tokens as fast as possible I had to give up on the longest-continuous-track bonus. Another player got it, and piled up just enough completed route bonuses to get past me by 1 solitary victory point. Hard to say which of us was more astonished.

My schedule was such that it wasn’t possible after that to make the second win that would get me to semifinals guaranteed. But I was a high alternate and might have made it in anyway; I was just checking in with the GM when my wife ran in to tell me I’d squeaked into the Puerto Rico semifinals running at the same time – and that’s a game I take more seriously.

Ah well, maybe next time. I think none of the WBC regulars in this tournament would be very surprised if I took gold some year, if I’m not preoccupied with more serious games.

Puerto Rico

Puerto Rico was not quite the first of the genre now called “Eurogames”, but it was the first to hit big internationally back in 2002. The theme is colonization and economic development in the 16th-century Caribbean; you build cities, you run plantations, you trade, and you ship goods to Europe for victory points.

This game is to Eurogame as apple is to fruit, the one everyone thinks of first. It looks light on the surface but isn’t; it has a lot of subtlety and tremendous continuing replay value. It has outlasted dozens of newer, flashier games that had six months or a year of glory but now molder half-forgotten in closets.

My wife and I are both experienced and strong players. The WBC tournament referees and many of the past champion players know us, and we’ve beaten some of them in individual games. We seldom fail to make quarter-finals, and some years we make semi-finals. I think each of us can realistically hope for gold some year.

But maybe we’re not quite good enough yet. Cathy got two wins in the qualifying heats, good for a bye past the quarter-finals into semis. I scored one utterly crushing victory at the only three-place table in the second qualifying heat, playing my default factory/fast-buildout strategy. Then, only a close second – but that made me second alternate (one of the guys I beat in that game was last year’s champion) and I got in because a couple of qualified players dropped out to do other things (like play in the Ticket To Ride semis I passed on to play in these).

Cathy pulled third in her game; she says she was outplayed. Me, I got off to a roaring start. Play order in the initial round is known to be important from statistical analysis, so much so that in championship you bid competitively for it by agreeing to deduct victory points from your final score. I got fourth seat (generally considered second-best to third) relatively cheaply for -1.5.

Usually I plan to play corn shipper when in fourth seat. But, due to the only random element in the game (the order plantation tiles in the game come out) and some good money-accumulation moves, I managed to build and man a coffee roaster very early. That pointed me back at my default strategy, which aims at a fast city build-out with minimal shipping using Factory as a money generator – one coffee crop comes close to paying for the (expensive) Factory.

Damned if it didn’t work perfectly. I had the only coffee in production, which scared other players off of triggering production, particularly styming the bulk shippers. For most of the game it looked to everyone at the table like I was cruising to an easy win. There were admiring remarks about this.

The one drawback of this strategy, however, is that it has a slow ramp-up. You make most of your points quite near the end of the game through big buildings. If anyone can force game end before you’re expecting it, you take a bigger hit to your score than shippers who have been making points from the beginning.

That’s how I got whacked. There were these three or maybe four guys down from Quebec specifically for the Puerto Rico tournament; gossip said they weren’t playing anything else. One of them – Matthieu, I think his name was – was sitting to my left (after me in the turn order) and pulled a brilliant hack that shortened the game by at least two rounds, maybe more. Doing this deprived me of the last, crucial rounds of build-out time when I would have pulled down the biggest points.

Those of you who play the game know that one way to accelerate the end is to deliberately leave buildings unmanned so they suck colonists off the colony ship faster; when those run out, you’re done. There’s a recently discovered ambiguity in the rules that makes this tactic work much faster – turns out that someone playing Mayor is allowed, under a strict reading, to refuse to take his colonists, casting them into the void and leaving his building empty to pull more out of the boat on the next round.

The resulting vanishing-colonist play may be a bug produced by poorly crafted rules or a bad translation from the original German (wouldn’t be the first time that’s been a problem, either). The tournament referee is not happy with it, nor are the WBC regulars – it screws with the “book”, the known strategies, very badly. The ref intends to brace the game’s designer about this, and we may get a rules amendment disallowing the play next year.

In the meantime, nobody could argue that the guys from Quebec weren’t within their rights to exploit this hack ruthlessly. And they did. Three of them used it to finish at the finals table. Matthieu, the one that dry-gulched me, took the gold.

There was a lot of … I won’t say “angry”, but rather perturbed talk about this. I wasn’t the only person to feel somewhat blindsided and screwed (though we also admired their nerve and dedication). These guys were monomaniacs; unlike most top WBC gamers, who (like me) play up to a half-a-dozen games very well, the Quebeckers were laser focused on thus one game and studied it to the point where they found the hack that would break the standard book.

Sigh…and that’s why no trophy for me this year. (Everyone in the final four would have gotten one.) Cathy and I will try again. Nobody would be surprised at either of us making the finals, but it could take a few years’ more practice.

Posted Mon Aug 11 03:44:10 2014 Tags:

Nexus (Nicholas Wilson; Victory Editing) is the sort of thing that probably would have been unpublishable before e-books, and in this case I’m not sure whether that’s good or bad.

There’s a lot about this book that makes me unsure, beginning with whether it’s an intentional sex comedy or a work of naive self-projection by an author with a pre-adolescent’s sense of humor and a way, way overactive libido. Imagine Star Trek scripted as semi-pornographic farce with the alien-space-babes tropes turned up to 11 and you’ve about got this book.

It’s implausible verging on grotesque, but some of the dialog is pretty funny. If you dislike gross-out humor, avoid.

Posted Wed Aug 6 04:36:47 2014 Tags:

LSE is a wasting disease. It invades the brains of writers of SF and other genres, progressively damaging their ability to tell entertaining stories until all they can write is unpleasant gray goo fit only for consumption by lit majors. One of the principal sequelae of the disease is plunging sales.

If you are a writer or an aspiring writer, you owe it to yourself to learn the symptoms of LSE so you can seek treatment should you contract it. If you love a writer or aspiring writer, be alert for the signs; victims often fail to recognize their condition until the degeneration has passed the critical point beyond which no recovery is possible. You may have to stage an intervention.

Here are some clinical indicators of LSE:

1. Evinces desire to be considered “serious artist”.

2. Idea content is absent or limited to politicized social criticism.

3. Heroism does not occur except as anti-heroic mockery.

4. All major characters are psychologically damaged.

5. Wordage devoted to any character’s interior monologues exceeds wordage in same character’s dialog.

6. Repeated character torture, especially of the self-destructive variety.

7. Inability to write an unambiguously happy ending. In advanced cases, the ability to write any ending at all may be lost.

8. Stronger craving for a Nebula than a Hugo. (Outside SF: approval of fellow genre authors more valued than that of fans.)

9. Spelling name without capital letters.

10. Plot is smothered under an inchoate cloud of characterization.

11. Persistent commission of heavy-handed allegory.

12. All sense of humor or perspective vanishes from writing, replaced (if at all) by hip irony.

13. Characters do not experience joy, hope, or autonomy except as transient falsehoods to be mocked.

14. No moment of conceptual breakthrough in story. (Outside SF: lack of respect for genre aims and values.)

If you have three or more of these symptoms, step away from your keyboard before another innocent reader is harmed. Immerse yourself in retro space opera (or your non-SF genre’s equivalent) until you understand what it got right that you are doing wrong. And, get over yourself.

(Thread is open for more symptoms. Try to keep politics out of it; that would be a different “Warning signs of being a political tool” list.)

Posted Tue Aug 5 11:55:32 2014 Tags:

One of the minor frustrations of my life, up to now, is that though I can sell as much nonfiction as I care to write, fiction sales had eluded me. What made this particularly irksome is that I don’t have only the usual ego reasons for wanting to succeed. I love the science fiction genre and owe it much; I want to pay that forward by contributing back to it.

It therefore gives me great satisfaction to announce that I have made my first SF sale, a short (3.5kword) piece of military SF titled Sucker Punch set on a U.S. aircraft carrier during the Taiwan Straits Action of 2037. Some details follow.

The backstory begins with Castalia House, an e-book publisher based in Finland, noticing my recent spate of reviews and offering to send me some of their current releases – notably John C. Wright’s Awake In The Night Land, which I haven’t reviewed yet only because I feel I need to have read Willam Hope Hodgson’s The Night Land first and, brother, that is quite a slog.

During this conversation, the head guy at Castalia House (the infamous Vox Day, wearing another hat) informed me of an upcoming project: an anthology called Ride The Red Horse intended to reprise the format of Jerry Pournelle’s old There Will Be War compendia. That is, a mix of military SF and military futurology, written by a mix of SF authors and serving military personnel, with few technical experts added for flavor.

“Want to write a fiction piece for us?” said Mr. Castalia House. “I can’t write fiction for shit, or at least all my attempts to sell it have failed,” I replied.

“Well, what about non-fiction?” I couldn’t think of a premise; then, suddenly, I could. Which is how I wound up researching and writing a fact piece called Battlefield Lasers and the Death of Airpower. I turned in a partial first draft about four hours later, and swiftly learned that (a) the actual editor on the anthology is Tom Kratman, and (b) he loved the draft and absolutely wanted it in when it got finished.

A couple days later I got a full draft done and shipped it. (A&D regular Ken Burnside, who knows weapons physics inside and out, was significant help.) Delighted that-was-brilliant! email from Castalia House followed.

“Are you suuuure you don’t want to do a fiction piece?” Mr. Castalia House said, or words to that effect. He tossed a couple of premises at me which I didn’t particularly like; then it occurred to me that I might dramatically fictionalize one aspect of the futurology in Battlefield Lasers…

A long Sunday later I had Sucker Punch ready. Writing it was an odd experience. I knew the story concept was working as I pounded it out, but what it clearly wanted to be was a Tom-Clancy-style technothriller in miniature, which is not something I had ever imagined myself writing. But I went with it and shipped it.

Hours after I did so Mr. Castalia House got back to me and said “What caused you to imagine that you can’t write fiction? This story is better than mine!” He then went off on a tear about the incompetence of the gatekeepers who had turned down my previous efforts. Which, I suppose, is possible; or maybe they really sucked but I’ve learned some things since.

Anyway, that’s how I sold my first SF. I don’t have a publication date more specific than “this Fall” yet. I’ll announce it here. And I’ll try not to make this a one-off. I do have some advantages; I’m already a very skilled writer, just not so much at the “fiction” part.

My real dream, someday, is to write a major novel of hard SF at the level of (say) Greg Egan’s Disapora or Neal Stephenson’s Anathem. That’s a long way off from here. Baby steps…

Posted Mon Aug 4 14:05:37 2014 Tags:

Of Bone And Thunder (Chris Evans; Pocket Books) is an object lesson in why fiction writers should avoid political allegory. Yes, it’s a fantasy reflection of the Vietnam War; on the off-hand chance a reader wouldn’t have figured it out by about page 3, the publisher helpfully spells it out in the blurb.

There might even be something in the book besides allegory – the author is, at least, a reasonably competent wordsmith. The trouble is that the book’s message is hammered home with repetitive and unceasing dullness from the very beginning. By the time I was 10% in, all I wanted was to make it stop. Shortly after that point, I gave up.

Message fiction may not intrinsically be a bad thing, but it requires a lightness of touch that this author – like most others who try it – seems incapable of achieving.

Pro tip: learn to entertain, first. When you have mastered the art of writing fiction that people find engaging and want to read, then you can begin to include message elements. Carefully, quietly, minimally. Beware of over-egging; avoid a bleak, humorless, heavy-handed approach.

Otherwise, your work will fail both as fiction and as message. Which, I fear, is precisely what has occurred here.

Posted Sun Aug 3 18:22:31 2014 Tags:

Traitor’s Blade (Sebastian de Castell; Jo Fletcher books) is perhaps best tescribed as a noirish fantasy spin on Dumas’s Three Musketeers. But it’s better, and less derivative, than that sounds.

Five years ago, a revolt by powerful nobles led to the death of the king that Falcio val Mond served. Ever since, Falcio and his comrades Kest and Brasti have been struggling to reunite the Greatcoats, the order of swordsmen/justiciars the King founded to enforce the Law against even the mightiest.

But the Greatcloaks are in in popular disgrace, widely reviled as traitors for having failed to defend the King. The victorious nobles are enemies, having no desire for any counterweight against their absolute power in their domains. The remnant Greatcoats have survived only by not appearing to be a political threat.

When Falcio and his friends are framed for murder, they are forced to hire on as caravan guards to escape the scene of the crime. But their journey to the corrupt and blood-soaked city of Rijou is one out of the frying pan and into the fire. The King’s secrets – and Falcio’s own – are far from done with him.

This book is a satisfying swirl or derring-do, intrigue, treachery, and a soupcon of magic. But what really makes it work are the fight scenes. They are no vague swash and buckle but detailed descriptions of specific techniques and sequences – the “conversation of blades”. As a historical fencer and martial artist myself, I can certify that the author knows what he is writing about very well.

First of a sequence. I’ll want to read the sequels.

Posted Sun Aug 3 14:54:16 2014 Tags:

There are numerous C async I/O libraries; tevent being the one I’m most familiar with.  Yet, tevent has a very wide API, and programs using it inevitably descend into “callback hell”.  So I wrote ccan/io.

The idea is that each I/O callback returns a “struct io_plan” which says what I/O to do next, and what callback to call.  Examples are “io_read(buf, len, next, next_arg)” to read a fixed number of bytes, and “io_read_partial(buf, lenp, next, next_arg)” to perform a single read.  You could also write your own, such as pettycoin’s “io_read_packet()” which read a length then allocated and read in the rest of the packet.

This should enable a convenient debug mode: you turn each io_read() etc. into synchronous operations and now you have a nice callchain showing what happened to a file descriptor.  In practice, however, debug was painful to use and a frequent source of bugs inside ccan/io, so I never used it for debugging.

And I became less happy when I used it in anger for pettycoin, but at some point you’ve got to stop procrastinating and start producing code, so I left it alone.

Now I’ve revisited it.   820 insertions(+), 1042 deletions(-) and the code is significantly less hairy, and the API a little simpler.  In particular, writing the normal “read-then-write” loops is still very nice, while doing full duplex I/O is possible, but more complex.  Let’s see if I’m still happy once I’ve merged it into pettycoin…

Posted Sat Aug 2 06:58:27 2014 Tags:

Before you read the rest of this post, go look at these pictures of a Hobbit Pub and a Hobbit House. And recall the lovely Bag End sets from Peter Jackson’s LOTR movies.

I have a very powerful reaction to these buildings that, I believe, has nothing to do with having been a Tolkien fan for most of my life. In fact, some of the most Tolkien-specific details – the round doors, the dragon motifs in the pub – could be removed without attenuating that reaction a bit.

To me, they feel right. They feel like home. And I’m not entirely sure why, because I’ve never lived in such antique architecture. But I think it may have something to do with Christopher Alexander’s “Timeless Way of Building”.

Alexander’s ideas are not easy to summarize. He believes that there is a timeless set of generative ur-patterns which are continuously rediscovered in the world’s most beautiful buildings – patterns which derive from an interplay among mathematical harmonies, the psychological/social needs of human beings, and the properties of the materials we build in.

Alexander celebrates folk architecture adapted to local needs and materials. He loves organic forms and buildings that merge naturally with their surroundings. He respects architectural tradition, finding harmony and beauty even in its accidents.

When I look at these buildings, and the Tolkien sketches from which they derive, that’s what I see. The timelessness, the organic quality, the rootedness in place. When I look inside them, I see a kind of humane warmth that is all too rare in any building I actually visit. (Curiously, one of the few exceptions is a Wegmans supermarket near me which, for all that it’s a gigantic commercial hulk, makes clever use of stucco and Romanesque stonework to evoke a sense of balance, groundedness, and warmth.)

I want to live in a thing like the Hobbit House – a hummocky fieldstone pile with a red-tiled roof and a chimney, and white plaster and wainscoting and hardwood floors. I want it to look like it grew where it is, half-set in a hillside. I want the mullions and the butterfly windows and the massive roof-beams and the eyebrow gables. Want, want, want!

I don’t feel like this desire is nostalgia or a turn away from the modern; there’s room in my dream for central heating and Ethernet cable in the walls, not to mention electricity. I feel like it’s a turn towards truths from the past but for the future – that, in our busy cleverness, we have almost forgotten what kind of design makes a building not just physically adequate but psychologically nourishing. We need to rediscover that, and these buildings feel to me like clues.

I think it might be that Tolkien, an eccentric genius nostalgic for the English countryside of his pre-World-War-I youth, abstracted and distilled out of its vernacular architecture exactly those elements which are timeless in Christopher Alexander’s sense. There is a pattern language, a harmony, here. These buildings make sense as wholes. They are restful and welcoming.

They’re also rugged. You can tell by looking at the Hobbit House, or that inn in New Zealand, that you’d have to work pretty hard to do more than superficial damage to either. They’ll age well; scratches and scars will become patina. And a century from now or two, long after this year’s version of “modern” looks absurdly dated, they’ll still look like they belong exactly where they are.

One mathematical possibility I find plausible for explaining their appeal: these buildings exhibit something like fractal self-similarity. The rooflines resemble 1/f noise. Small details echo large ones; similar forms and proportions show up at multiple scales. These are features by which the human eye recognizes natural forms. Perhaps this is why they seem so restful.

I wish we could learn to build like this again – not as a movie set or a stunt, but as a living idiom. Factories and offices don’t need what these buildings have, but homes – the places where people actually live – do. I think we’d all be saner and happier for it.

Posted Sat Aug 2 06:06:42 2014 Tags:
While our CPU clock speeds may not be increasing as they once did and transistor density is slowing, other elements of our computing experiencing are experience impressive evolution. One such area is in the display: screen pixel density is jumping and GPUs have become remarkable systems. We are currently in a transition between "low density" and "high density" displays and the various screens in a house are likely to have a widely varying mix of resolutions, pixel densities and graphics compute power.

In the house here we have a TV with 110 dpi, a laptop with 125 dpi, another laptop with 277 dpi and phones that top out at over 400 dpi! 

Now let's consider a very simple question: what font size should one use? Obviously, that depends on the screen. Now, if applications are welded to a single screen, this wouldn't be too much of a problem: pick a font size and stick with it. Unfortunately for us reality is more complex than that. I routinely plug the laptops into the TV for family viewing. If I get my way, in the near future our applications will also migrate between systems so even in cases where the screen is welded to the device the application will be mobile.

The answer is still not difficult, however: pick a nice base font size and then scale it to the DPI. Both of the major proprietary desktop operating systems have built this concept into their current offerings and it works pretty well. In fact, you can scale the whole UI to the DPI of the system.

This still leaves us with how to get a "reasonable font size". Right now we ship a reasonable default and allow the user to tweak this. How hard can it be, right?

Well, currently KDE software is doing two things very, very wrong when it comes to handling fonts in this modern era. Both of these can be fixed in the Frameworks 5 era if the application developers band together and take sensible action. Let's look at these two issues.

Application specific font settings

What do Kate, Konqueror, Akregator, Dolphin, KMail, Rekonq, KCalc, Amarok, and, I presume, many other KDE applicationsKorganizer and Konsole have in common? They all allow the user to set custom fonts. For some of these applications it default to using system fonts but still allows the user to modify them. For some of these applications, they always use their own fonts. Neither is good, and the latter is just plain evil.

Kontact, being made up of several of the above applications, is a real pit of font sadness since each of its components manages its own font settings.

This wasn't such a big deal in the "old days" when everyone's screen was equally good (or equally crappy, depending on how you wish to look at it ;). In that world, the user could wade through these settings once and then never touch them again.

Today with screens of such radically different pixel densities and resolutions, the need to quickly change fonts depending on the configuration of the system is a must-have. When you have to change those settings in N different applications, it quickly becomes a blocker.

Moreover, if every application is left to its own devices with fonts, at least some will get it wrong when it comes to properly scaling between different screens. When applications start moving between devices this will become even more so the case than it is now.

The solution is simple: applications must drop all custom font settings.

Before anyone starts hollering, I'm not suggesting that there should be no difference between the carefully selected font in your konsole window and the lovingly chosen font in your email application. (I would suggest that, but I have a sense of the limits of possibilities ... ;) What I am suggesting is that every font use case should be handled by the central font settings system in KDE Frameworks. Yes, there should be a "terminal" font and perhaps one for calendars and email, too. A categorization system that doesn't result in dozens of settings but which serves the reasonable needs of all desktop applications could be arrived at with a bit of discipline.

With that in place, applications could rely on the one central system getting the font scaling right so that when the user changes screens (either connected to the device or the screen the application is running on) the application's fonts will "magically" adjust.

Scaling user interface to font size

The idea to scale user interface to font size is one that the current Plasma team has recently decided to adopt. I can not state strongly enough just how broken this is. Watching the results when plugging plug that laptop with the 3300x1800 @277dpi screen into the 110 dpi television is enough to make baby kittens cry tears of sadness. The reason is simple: the font sizes need to scale to the screen. When they aren't scaled, the UI becomes comically large (or tragically small, depending on the direction you are going).

.. and that's ignoring accessibility use cases where people may need bigger fonts but really don't need bigger window shadows to go with it, thank you very much.

The answer here is also very simple, so simple that both Apple and Microsoft are doing it: scale the fonts and the UI to the DPI of the screen. Auto-adjust when you can, let the user adjust the scaling to their preference.

The reason Plasma 5 is not doing this is because Qt doesn't report useful DPI information. Neither does xdpyinfo. So now one might ask where I got all those DPI figures at the start of this blog entry. Did I look them up online? Nope. I used the EDID information from each of those screens as reported by xrandr or similar tools. With the help of the monitor-parse-edid tool, which is 640 lines of Perll, I was able to accurately determine the DPI of every screen in this house. Not exactly rocket science.

With DPI in hand, one can start to think about scaling font sizes and the UI as well, independently. Doesn't even require waiting for Qt to get this right, either. All the information need is right there in the EDID block.

There is a caveat: some KVMs strip the EDID information (bad KVM, bad!), older monitors may not have useful EDID info (the last version of the EDID standard was published eight years ago, so this isn't new technology) and occasionally a vendor goofs and gets the EDID block wrong in a monitor. These are edge cases, however, and should not be the reason to hold back everyone else from living life la vida loca, at least when it comes to font and UI proportions. 

In those edge cases, allowing the user to manually tweak the scaling factor is an easy answer. In fact, this alone would be an improvement over the current situation! Instead of tweaking N fonts in N places, you could just tweak the scaling factor and Be Done With It(tm). There is even a natural place for this to happen: kscreen. It already responds to screen hotplug events, allows user tweaking and restores screen-based configurations automagically, it could add scaling to the things it tracks.

If people really wanted to get super fancy, creating a web service that accepts monitors, EDID blocks and the correct scaling factors according to the user and spits out recommendations by tallying up the input would take an afternoon to write. This would allow the creation of a "known bad" database with "known good" settings to match over time. That's probably overkill, however.

The other area of edge case is when you have two screens with different DPI connected to the system at the same time. This, too, is manageable. One option is to simply recognize it as a poorly-supported edge case and keep to one global scaling value. This is the "user broke it, user gets to keep both halves" type approach. It's also the easiest. The more complex, and certainly the one that would give better results, is to have per-screen scaling. To make this work scaling needs to change on a per-window basis based on which screen it is on. This would be manageable in the desktop shell and the window manager, though a bigger ask to support in every single application. It would mean applications would need to not only drop their in-application font settings (which they ought to anyways) but to make fonts and UI layout factors a per-window thing.

If you are running right now to your keyboard to ask about windows that are on more than screen at a time, don't bother: that's a true edge case that really doesn't need to be supported at all. Pick a screen that the is "on" and scale it appropriately. Multi-screen windows can take over the crying for the kittens who will now be bouncing around with happy delight on the other 99.999% of systems in use.

Build a future so bright, we gotta wear shades

These two issues are really not that big. They are the result of some unfortunate decision making in the past, but they can be easily rectified. As it stands, the way fonts are handled completely and without question ruins the user experience so hopefully as applications begin (or complete) the process of porting to Frameworks 5 they can pay attention to this.

.. and just so nobody feels too badly about it all: all the Linux desktop software I've tested so far for these issues has failed to make the grade. So the good news is that everything sucks just as much .. and the great news is that KDE is perfectly poised to rise above the rest and really stand out by getting it right in the Frameworks 5 era.
Posted Thu Jul 31 18:39:00 2014 Tags:

I’ve been aware for some time of a culture war simmering in the SF world. And trying to ignore it, as I believed it was largely irrelevant to any of my concerns and I have friends on both sides of the divide. Recently, for a number of reasons I may go into in a later post, I’ve been forced to take a closer look at it. And now I’m going to have to weigh in, because it seems to me that the side I might otherwise be most sympathetic to has made a rather basic error in its analysis. That error bears on something I do very much care about, which is the health of the SF genre as a whole.

Both sides in this war believe they’re fighting about politics. I consider this evaluation a serious mistake by at least one of the sides.

On the one hand, you have a faction that is broadly left-wing in its politics and believes it has a mission to purge SF of authors who are reactionary, racist, sexist et weary cetera. This faction now includes the editors at every major SF publishing imprint except Baen and all of the magazines except Analog and controls the Science Fiction Writers of America (as demonstrated by their recent political purging of Theodore Beale, aka Vox Day). This group is generally frightened of and hostile to indie publishing. Notable figures include Patrick & Theresa Nielsen Hayden and John Scalzi. I’ll call this faction the Rabbits, after Scalzi’s “Gamma Rabbit” T-shirt and Vox Day’s extended metaphor about rabbits and rabbit warrens.

On the other hand, you have a faction that is broadly conservative or libertarian in its politics. Its members deny, mostly truthfully, being the bad things the Rabbits accuse them of. It counteraccuses the Rabbits of being Gramscian-damaged cod-Marxists who are throwing away SF’s future by churning out politically-correct message fiction that, judging by Amazon rankings and other sales measures, fans don’t actually want to read. This group tends to either fort up around Baen Books or be gung-ho for indie- and self-publishing. Notable figures include Larry Correia, Sarah Hoyt, Tom Kratman, John C. Wright, and Vox Day. I’ll call this group the Evil League of Evil, because Correia suggested it and other leading figures have adopted the label with snarky glee.

A few other contrasts between the Rabbits and the Evil League are noticeable. One is that the Evil League’s broadsides are often very funny and it seems almost incapable of taking either itself or the Rabbits’ accusations seriously – I mean, Correia actually tags himself the “International Lord of Hate” in deliberate parody of what the Rabbits say about him. On the other hand, the Rabbits seem almost incapable of not taking themselves far too seriously. There’s a whiny, intense, adolescent, over-fixated quality about their propaganda that almost begs for mockery. Exhibit A is Alex Dally McFarlane’s call for an end to the default of binary gender in SF.

There’s another contrast that gets near what I think is the pre-political cause of this war. The Rabbits have the best stylists, while the Evil League has the best storytellers. Pick up a Rabbit property like Rich Horton’s The Year’s Best Science Fiction and Fantasy 2014 and you’ll read large numbers of exquisitely crafted little numbers about nothing much. The likes of Correia, on the other hand, churn out primitive prose, simplistic plotting, at best serviceable characterization – and vastly more ability to engage the average reader. (I would bet money, based on Amazon rankings, that Correia outsells every author in that collection combined.)

All this might sound like I’m inclined to sign up with the Evil League of Evil. The temptation is certainly present; it’s where the more outspoken libertarians in SF tend to have landed. Much more to the point, my sense of humor is such that I find it nearly impossible to resist the idea of posting something public requesting orders from the International Lord of Hate as to which minority group we are to crush beneath our racist, fascist, cismale, heteronormative jackboots this week. The screams of outrage from Rabbits dimwitted enough to take this sort of thing seriously would entertain me for months.

Alas, I cannot join the Evil League of Evil, for I believe they have made the same mistake as the Rabbits; they have mistaken accident for essence. The problem with the Rabbits is not that left-wing politics is dessicating and poisoning their fiction. While I have made the case elsewhere that SF is libertarian at its core, it nevertheless remains possible to write left-wing message SF that is readable, enjoyable, and of high quality – Iain Banks’s Culture novels leap to mind as recent examples, and we can refer back to vintage classics such as Pohl & Kornbluth’s The Space Merchants for confirmation. Nor, I think, is the failure of Rabbit fiction to engage most SF fans and potential fans mainly down to its politics; I think the Evil League is prone to overestimate the popular appeal of their particular positions here.

No, I judge that what is dessicating and poisoning the Rabbit version of SF is something distinct from left-wing political slant but co-morbid with it: colonization by English majors and the rise of literary status envy as a significant shaping force in the field.

This is a development that’s easy to mistake for a political one because of the accidental fact that most university humanities departments have, over the last sixty years or so, become extreme-left political monocultures. But, in the language of epidemiology, I believe the politics is a marker for the actual disease rather than the pathogen itself. And it’s no use to fight the marker organism rather than the actual pathogen.

Literary status envy is the condition of people who think that all genre fiction would be improved by adopting the devices and priorities of late 19th- and then 20th-century literary fiction. Such people prize the “novel of character” and stylistic sophistication above all else. They have almost no interest in ideas outside of esthetic theory and a very narrow range of socio-political criticism. They think competent characters and happy endings are jejune, unsophisticated, artistically uninteresting. They love them some angst.

People like this are toxic to SF, because the lit-fic agenda clashes badly with the deep norms of SF. Many honestly think they can fix science fiction by raising its standards of characterization and prose quality, but wind up doing tremendous iatrogenic damage because they don’t realize that fixating on those things (rather than the goals of affirming rational knowability and inducing a sense of conceptual breakthrough) produces not better SF but a bad imitation of literary fiction that is much worse SF.

Almost the worst possible situation is the one we are in now, in which over the last couple of decades the editorial and critical establishment of SF has been (through a largely accidental process) infiltrated by people whose judgment has been partly or wholly rotted out by literary status envy. The field’s writers, too, are often diminished and distorted by literary status envy. Meanwhile, the revealed preferences of SF fans have barely changed. This is why a competent hack like David Weber can outsell every Nebula winner combined by huge margins year after year after year.

The victims of literary status envy resent the likes of David Weber, and their perceived inferiority to the Thomas Pynchons of the world; they think the SF field is broken and need to be fixed. When they transpose this resentment into the key of politics in the way their university educations have taught then to do, they become the Rabbits.

The Evil League of Evil is fighting the wrong war in the wrong way. To truly crush the Rabbits, they should be talking less about politics and more about what has been best and most noble in the traditions of the SF genre itself. I think a lot of fans know there is something fatally gone missing in the Rabbit version of science fiction; what they lack is the language to describe and demand it.

That being said, in the long run, I don’t think the Evil League of Evil can lose. The Rabbits are both the beneficiaries and victims of preference falsification; their beliefs about where the fans want the field to go are falsified by their plummeting sales figures, but they can’t see past the closure induced by their control of the gatekeeper positions in traditional publishing. Meanwhile, the Evil League thrives in the rising medium of e-book sales, in indie- and self-publishing.

The Rabbits have a sort of herd-instinct sense that these new channels doom them to irrelevance, which is why so many of them line up to defend a system that ruthlessly exploits and cheats them. Contemplate SFWA’s stance in the Hachette-vs.-Amazon dispute. for example; it’s plain nuts if SFWA claims to be representing authors.

But it will be a faster, better, cleaner victory if the Evil League of Evil gets shut of political particularism (and I mean that, even about my politics) and recognizes the real problem. The real problem is that the SF genre’s traditional norms exist for very good reasons, and it’s time we all learned to give the flying heave-ho to people who fail to understand and appreciate that.

The right (counter)revolutionary slogan is therefore not “Drive out the social-justice warriors!”, it’s “Peddle your angsty crap elsewhere, lit-fic wannabes! Let’s get SF back in the gutter where it belongs!”

Posted Wed Jul 30 22:04:26 2014 Tags:

As all software, it took longer than I expected, but today I tagged the first version of pettycoin.  Now, lots more polish and features, but at least there’s something more than the git repo for others to look at!

Posted Tue Jul 29 07:53:59 2014 Tags:

[Note to the impatient: to try out my beamforming simulation, which produced the above image, visit my beamlab test page - ideally in a browser with very fast javascript, like Chrome. You can also view the source.]

I promised you some cheating of Shannon's Law. Of course, as with most things in physics, even the cheating isn't really cheating; you just adjust your model until the cheating falls within the rules.

The types of "cheating" that occur in wifi can be briefly categorized as antenna directionality, beamforming, and MIMO. (People usually think about MIMO before beamforming, but I find the latter to be easier to understand, mathematically, so I switched the order.)

Antenna Directionality and the Signal to Noise Ratio

Previously, we discussed the signal-to-noise ratio (SNR) in some detail, and how Shannon's Law can tell you how fast you can transfer data through a channel with a given SNR.

The thing to notice about SNR is that you can increase it by increasing amplification at the sender (where the background noise is fixed but you have a clear copy of the signal) but not at the receiver. Once a receiver has a copy of the signal, it already has noise in it, so when you amplify the received signal, you just amplify the noise by the same amount, and the SNR stays constant.

(By the way, that's why those "amplified" routers you can buy don't do much good. They amplify the transmitted signal, but amplifying the received signal doesn't help in the other direction. The maximum range is still limited by the transmit power on your puny unamplified phone or laptop.)

On the other hand, one thing that *does* help is making your antenna more "directional." The technical term for this is "antenna gain," but I don't like that name, because it makes it sound like your antenna amplifies the signal somehow for free. That's not the case. Antenna gain doesn't so much amplify the signal as ignore some of the noise. Which has the same net effect on the SNR, but the mechanics of it are kind of important.

You can think of an antenna as a "scoop" that picks up both the signal and the noise from a defined volume of space surrounding the antenna. The shape of that volume is important. An ideal "isotropic" antenna (my favourite kind, although unfortunately it doesn't exist) picks up the signal equally in all directions, which means the region it "scoops" is spherical.

In general we assume that background noise is distributed evenly through space, which is not exactly true, but is close enough for most purposes. Thus, the bigger the volume of your scoop, the more noise you scoop up along with it. To stretch our non-mathematical metaphor well beyond its breaking point, a "bigger sphere" will contain more signal as well as more noise, so just expanding the size of your region doesn't affect the SNR. That's why, and I'm very sorry about this, a bigger antenna actually doesn't improve your reception at all.

(There's another concept called "antenna efficiency" which basically says you can adjust your scoop to resonate at a particular frequency, rejecting noise outside that frequency. That definitely works - but all antennas are already designed for this. That's why you get different antennas for different frequency ranges. Nowadays, the only thing you can do by changing your antenna size is to screw up the efficiency. You won't be improving it any further. So let's ignore antenna efficiency. You need a good quality antenna, but there is not really such a thing as a "better" quality antenna these days, at least for wifi.)

So ok, a bigger scoop doesn't help. But what can help is changing the shape of the scoop. Imagine if, instead of a sphere, we scoop up only the signal from a half-sphere.

If that half-sphere is in the direction of the transmitter - which is important! - then you'll still receive all the same signal you did before. But, intuitively, you'll only get half the noise, because you're ignoring the noise coming in from the other direction. On the other hand, if the half-sphere is pointed away from the incoming signal, you won't hear any of the signal at all, and you're out of luck. Such a half-sphere would have 2x the signal to noise ratio of a sphere, and in decibels, 2x is about 3dB. So this kind of (also not really existing) antenna is called 3dBi, where dBi is "decibels better than isotropic" so an isotropic (spherical) receiver is defined as 0dBi.

Taking it a step further, you could take a quarter of a sphere, ie. a region 90 degrees wide in any direction, and point it at your transmitter. That would double the SNR again, thus making a 6dBi antenna.

Real antennas don't pick up signals in a perfectly spherical shape; math makes that hard. So real ones tend to produce a kind of weirdly-shaped scoop with little roundy bits sticking out all over, and one roundy bit larger than the others, in the direction of interest. Essentially, the size of the biggest roundy bit defines the antenna gain (dBi). For regulatory purposes, the FCC mostly assumes you will use a 6dBi antenna, although of course the 6dBi will not be in the shape of a perfect quarter sphere.

Now is that the most hand-wavy explanation of antenna gain you have ever seen? Good.

Anyway, the lesson from all this is if you use a directional antenna, you can get improved SNR. A typical improvement is around 6dBi, which is pretty good. But the down side of a directional antenna is you have to aim it. With a wifi router, that can be bad news. It's great for outdoors if you're going to set up a long-distance link and aim very carefully; you can get really long range with a very highly directional antenna. But indoors, where distances are short and people move around a lot, it can be trouble.

One simple thing that does work indoors is hanging your wifi router from the ceiling. Then, if you picture eg. a quarter-sphere pointing downwards, you can imagine covering the whole room without really sacrificing anything (other than upstairs coverage, which you don't care about if you put a router up there too). Basically, that's as good as you can do, which is why most "enterprise" wifi deployments hang their routers from the ceiling. If you did that at home too - and had the right kind of antennas with the right gain in the right direction - you could get up to 6dB of improvement on your wifi signal, which is pretty great.

(Another trick some routers do is to have multiple antennas, each one pointing in a different direction, and then switch them on and off to pick the one(s) with the highest SNR for each client. This works okay but it interferes with MIMO - where you want to actively use as many antennas as possible - so it's less common nowadays. It was a big deal in the days of 802.11g, where that was the main reason to have multiple antennas at all. Let's talk about MIMO later, since MIMO is its own brand of fun.)

Beamforming

So okay, that was antenna directionality (gain). To summarize all that blathering: you point your antenna in a particular direction, and you get a better SNR in that particular direction.

But the problem is that wifi clients move around, so antennas permanently pointed in a particular direction are going to make things worse about as often as they help (except for simple cases like hanging from the ceiling, and even that leaves out the people upstairs).

But wouldn't it be cool if, using software plus magic, you could use multiple antennas to create "virtual directionality" and re-aim a "beam" automatically as the client device moves around?

Yes, that would be cool.

Unfortunately, that's not what beamforming actually is.

Calling it "beamforming" is not a *terrible* analogy, but the reality of the signal shape is vastly more complex and calling it a "beam" is pretty misleading.

This is where we finally talk about what I mentioned last time, where two destructively-interfering signals result in zero signal. Where does the power go?

As an overview, let's say you have two unrelated transmitters sending out a signal at the same frequency from different locations. It takes a fixed amount of power for each transmitter to send its signal. At some points in space, the signals interfere destructively, so there's no signal at that point. Where does it go? There's a conservation of energy problem after all; the transmitted power has to equal the power delivered, via electromagnetic waves, out there in the wild. Does it mean the transmitter is suddenly unable to deliver that much power in the first place? Is it like friction, where the energy gets converted into heat?

Well, it's not heat, because heat is vibration, ie, the motion of physical particles with mass. The electromagnetic waves we're talking about don't necessarily have any relationship with mass; they might be traveling through a vacuum where there is no mass, but destructive interference can still happen.

Okay, maybe the energy is re-emitted as radiation? Well, no. The waves in the first place were radiation. If we re-emitted them as radiation, then by definition, they weren't cancelled out. But we know they were cancelled out; you can measure it and see.

The short and not-very-satisfying answer is that in terms of conservation of energy, things work out okay. There are always areas where the waves interfere constructively that exactly cancel out the areas where they interfere destructively.

The reason I find that answer unsatisfying is that the different regions don't really interact. It's not like energy is being pushed, somehow, between the destructive areas and the constructive areas. It adds up in the end, because it has to, but that doesn't explain *how* it happens.

The best explanation I've found relates to quantum mechanics, in a lecture I read by Richard Feynman at some point. The idea is that light (and all electromagnetic waves, which is what we're talking about) actually does not really travel in straight lines. The idea that light travels in a straight line is just an illusion caused by large-scale constructive and destructive interference. Basically, you can think of light as travelling along all the possible paths - even silly paths that involve backtracking and spirals - from point A to point B. The thing is, however, that for almost every path, there is an equal and opposite path that cancels it out. The only exception is the shortest path - a straight line - of which there is only one. Since there's only one, there can't be an equal but opposite version. So as far as we're concerned, light travels in a straight line.

(I offer my apologies to every physicist everywhere for the poor quality of that explanation.)

But there are a few weird experiments you can do (look up the "double slit experiment" for example) to prove that in fact, the "straight line" model is the wrong one, and the "it takes all the possible paths" model is actually more like what's really going on.

So that's what happens here too. When we create patterns of constructive and destructive radio interference, we are simply disrupting the rule of thumb that light travels in a straight line.

Oh, is that all? Okay. Let's call it... beam-un-forming.

There's one last detail we have to note in order to make it all work out. The FCC says that if we transmit from two antennas, we have to cut the power from each antenna in half, so the total output is unchanged. If we do that, naively it might seem like the constructive interference effect is useless. When the waves destructively interfere, you still get zero, but when they constructively interfere, you get 2*½*cos(ωt), which is just the original signal. Might as well just use one antenna with the original transmit power, right?

Not exactly. Until now, I have skipped over talking about signal power vs amplitude, since it hasn't been that important so far. The FCC regulates *power*, not amplitude. The power of A*cos(ωt) turns out to be ½A2. I won't go over all the math, but the energy of f(x) during a given period is defined as ∫ f2(x) dx over that period. Power is energy divided by time. It turns out (via trig identities again) the power of cos(x) is 0.5, and the rest flows from there.

Anyway, the FCC limit requires a *power* reduction of ½. So if the original wave was cos(ωt), then the original power was 0.5. We need the new transmit power (for each antenna) to be 0.25, which is ½A2 = ½(0.5). Thus A = sqrt(0.5) = 1/sqrt(2) = 0.7 or so.

So the new transmit wave is 0.7 cos(ωt). Two of those, interfering constructively, gives about 1.4 cos(ωt). The resulting power is thus around ½(1.4)2 = 1, or double the original (non-reduced, with only one antenna) transmit power.

Ta da! Some areas have twice the power - a 3dB "antenna array gain" or "tx beamforming gain" - while others have zero power. It all adds up. No additional transmit power is required, but a receiver, if it's in one of the areas of constructive interference, now sees 3dB more signal power and thus 3dB more SNR.

We're left with the simple (ha ha) matter of making sure that the receiver is in an area of maximum constructive interference at all times. To make a long story short, we do this by adjusting the phase between the otherwise-identical signals coming from the different antennas.

I don't really know exactly how wifi arranges for the phase adjustment to happen; it's complicated. But we can imagine a very simple version: just send from each antenna, one at a time, and have the receiver tell you the phase difference right now between each variant. Then, on the transmitter, adjust the transmit phase on each antenna by an opposite amount. I'm sure what actually happens is more complicated than that, but that's the underlying concept, and it's called "explicit beamforming feedback." Apparently the 802.11ac standard made progress toward getting everyone to agree on a good way of providing beamforming feedback, which is important for making this work well.

Even more weirdly, the same idea works in reverse. If you know the phase difference between the client's antenna (we're assuming for now that he has only one, so we don't go insane) and each of your router's antennas, then when the client sends a signal *back* to you, you can extract the signal from the different antennas in a particular way that gets you the same amount of gain as in the transmit direction, and we call that rx beamforming. At least, I think you can. I haven't done the math for that yet, so I don't know for sure how well it can work.

Relatedly, even if there is no *explicit* beamforming feedback, in theory you can calculate the phase differences by listening to the signals from the remote end on each of your router's antennas. Because the signals should be following exactly the same path in both directions, you can guess what phase difference your signal arrived with by seeing which difference *his* signal came back with, and compensate accordingly. This is called "implicit beamforming feedback." Of course, if both ends try this trick at once, hilarity ensues.

And finally, I just want to point out how little the result of "beamforming" is like a beam. Although conceptually we'd like to think of it that way - we have a couple of transmitters tuning their signal to point directly at the receiver - mathematically it's not really like that. "Beamforming" creates a kind of on-off "warped checkerboard" sort of pattern that extends in all directions. To the extent that your antenna array is symmetrical, the checkerboard pattern is also symmetrical.

Beamforming Simulation

Of course, a checkerboard is also a flawed analogy. Once you start looking for a checkerboard, you start to see that in fact, the warping is kind of directional, and sort of looks like a beam, and you can imagine that with a hundred antennas, maybe it really would be "beam" shaped.

After doing all the math, I really wanted to know what beamforming looked like, so I wrote a little simulation of it, and the image at the top of this article is the result. (That particular one came from a 9-antenna beamforming array.)

You can also try out the simulation yourself, moving around up to 9 antennas to create different interference patterns. I find it kind of fun and mesmerizing, especially to think that these signals are all around us and if you could see them, they'd look like *that*. On my computer with Chrome, I get about 20 frames per second; with Safari, I get about 0.5 frames per second, which is not as fun. So use a browser with a good javascript engine.

Note that while the image looks like it has contours and "shadows," the shadows are entirely the effect of the constructive/destructive interference patterns causing bright and dark areas. Nevertheless, you can kind of visually see how the algorithm builds one level of constructive interference on top of another, with the peak of the humpiest bump being at the exact location of the receiver. It really works!

Some notes about the simulation:

  • It's 2-dimensional. Real life has at least 3 dimensions. It works pretty much the same though.
  • The intensity (brightness) of the colour indicates the signal strength at that point. Black means almost no signal.
  • "Blue" means cos(ωt) is positive at that point, and red means it's negative.
  • Because of the way phasors work, "blue plus red" is not the only kind of destructive interference, so it's a bit confusing.
  • Click on the visualization to move around the currently-selected transmitter or receiver.
  • When you move the receiver around, it auto-runs the beamforming optimization so you can see the "beam" move around.
  • The anti-optimize button is not very smart; a smarter algorithm could achieve an even less optimal result. But most of the time it does an okay job, and it does show how you can also use beamforming to make a receiver *not* hear your signal. That's the basis of MU-MIMO.
MIMO

The last and perhaps most exciting way to cheat Shannon's Law is MIMO. I'll try to explain that later, but I'm still working out the math :)

Posted Tue Jul 29 06:41:59 2014 Tags:

Before we get to the next part, which is fun, we need to talk about phasors. No, not the Star Trek kind, the boring kind. Sorry about that.

If you're anything like me, you might have never discovered a use for your trigonometric identities outside of school. Well, you're in luck! With wifi, trigonometry, plus calculus involving trigonometry, turns out to be pretty important to understanding what's going on. So let's do some trigonometry.

Wifi modulation is very complicated, but let's ignore modulation for the moment and just talk about a carrier wave, which is close enough. Here's your basic 2.4 GHz carrier:

    A cos (ω t)

Where A is the transmit amplitude and ω = 2.4e9 (2.4 GHz). The wavelength, λ, is the speed of light divided by the frequency, so:

    λ = c / ω = 3.0e8 / 2.4e9 = 0.125m

That is, 12.5 centimeters long. (By the way, just for comparison, the wavelength of light is around 400-700 nanometers, or 500,000 times shorter than a wifi signal. That comes out to 600 Terahertz or so. But all the same rules apply.)

The reason I bring up λ is that we're going to have multiple transmitters. Modern wifi devices have multiple transmit antennas so they can do various magic, which I will try to explain later. Also, inside a room, signals can reflect off the walls, which is a bit like having additional transmitters.

Let's imagine for now that there are no reflections, and just two transmit antennas, spaced some distance apart on the x axis. If you are a receiver also sitting on the x axis, then what you see is two signals:

    cos (ω t) + cos (ω t + φ)

Where φ is the phase difference (between 0 and 2π). The phase difference can be calculated from the distance between the two antennas, r, and λ, as follows:

    φ = r / λ

Of course, a single-antenna receiver can't *actually* see two signals. That's where the trig identities come in.

Constructive Interference

Let's do some simple ones first. If r = λ, then φ = 2π, so:

    cos (ω t) + cos (ω t + 2π)
    = cos (ω t) + cos (ω t)
    = 2 cos (ω t)

That one's pretty intuitive. We have two antennas transmitting the same signal, so sure enough, the receiver sees a signal twice as tall. Nice.

Destructive Interference

The next one is weirder. What if we put the second transmitter 6.25cm away, which is half a wavelength? Then φ = π, so:

    cos (ω t) + cos (ω t + π)
    = cos (ω t) - cos (ω t)
    = 0

The two transmitters are interfering with each other! A receiver sitting on the x axis (other than right between the two transmit antennas) won't see any signal at all. That's a bit upsetting, in fact, because it leads us to a really pivotal question: where did the energy go?

We'll get to that, but first things need to get even weirder.

Orthogonal Carriers

Let's try φ = π/2.

    cos (ω t) + cos (ω t + π/2)
    = cos (ω t) - sin (ω t)

This one is hard to explain, but the short version is, no matter how much you try, you won't get that to come out to a single (Edit 2014/07/29: non-phase-shifted) cos or sin wave. Symbolically, you can only express it as the two separate factors, added together. At each point, the sum has a single value, of course, but there is no formula for that single value which doesn't involve both a cos ωt and a sin ωt. This happens to be a fundamental realization that leads to all modern modulation techniques. Let's play with it a little and do some simple AM radio (amplitude modulation). That means we take the carrier wave and "modulate" it by multiplying it by a much-lower-frequency "baseband" input signal. Like so:

    f(t) cos (ω t)

Where ω >> 1, so that for any given cycle of the carrier wave, f(t) can be assumed to be "almost constant."

On the receiver side, we get the above signal and we want to discover the value of f(t). What we do is multiply it again by the carrier:

    f(t) cos (ω t) cos (ω t)
    = f(t) cos2 (ω t)
    = f(t) (1 - sin2 (ω t))
    = ½ f(t) (2 - 2 sin2 (ω t))
    = ½ f(t) (1 + (1 - 2 sin2 (ω t)))
    = ½ f(t) (1 + cos (2 ω t))
    = ½ f(t) + ½ f(t) cos (2 ω t)

See? Trig identities. Next we do what we computer engineers call a "dirty trick" and, instead of doing "real" math, we'll just hypothetically pass the resulting signal through a digital or analog filter. Remember how we said f(t) changes much more slowly than the carrier? Well, the second term in the above answer changes twice as fast as the carrier. So we run the whole thing through a Low Pass Filter (LPF) at or below the original carrier frequency, removing high frequency terms, leaving us with just this:

    (...LPF...)
    → ½ f(t)

Which we can multiply by 2, and ta da! We have the original input signal.

Now, that was a bit of a side track, but we needed to cover that so we can do the next part, which is to use the same trick to demonstrate how cos(ω t) and sin(ω t) are orthogonal vectors. That means they can each carry their own signal, and we can extract the two signals separately. Watch this:

    [ f(t) cos (ω t) + g(t) sin (ω t) ] cos (ω t)
    = [f(t) cos2 (ω t)] + [g(t) cos (ω t) sin (ω t)]
    = [½ f(t) (1 + cos (2 ω t))] + [½ g(t) sin (2 ω t)]
    = ½ f(t) + ½ f(t) cos (2 ω t) + ½ g(t) sin (2 ω t)
    [...LPF...]
    → ½ f(t)

Notice that by multiplying by the cos() carrier, we extracted just f(t). g(t) disappeared. We can play a similar trick if we multiply by the sin() carrier; f(t) then disappears and we have recovered just g(t).

In vector terms, we are taking the "dot product" of the combined vector with one or the other orthogonal unit vectors, to extract one element or the other. One result of all this is you can, if you want, actually modulate two different AM signals onto exactly the same frequency, by using the two orthogonal carriers.

QAM

But treating it as just two orthogonal carriers for unrelated signals is a little old fashioned. In modern systems we tend to think of them as just two components of a single vector, which together give us the "full" signal. That, in short, is QAM, one of the main modulation methods used in 802.11n. To oversimplify a bit, take this signal:

    f(t) cos (ω t) + g(t) sin (ω t)

And let's say f(t) and g(t) at any given point in time each have a value that's one of: 0, 1/3, 2/3, or 1. Since each function can have one of four values, there are a total of 4*4 = 16 different possible combinations, which corresponds to 4 bits of binary data. We call that encoding QAM16. If we plot f(t) on the x axis and g(t) on the y axis, that's called the signal "constellation."

Anyway we're not attempting to do QAM right now. Just forget I said anything.

Adding out-of-phase signals

Okay, after all that, let's go back to where we started. We had two transmitters both sitting on the x axis, both transmitting exactly the same signal cos(ω t). They are separated by a distance r, which translates to a phase difference φ. A receiver that's also on the x axis, not sitting between the two transmit antennas (which is a pretty safe assumption) will therefore see this:

    cos (ω t) + cos (ω t + φ)
    = cos (ω t) + cos (ω t) cos φ - sin (ω t) sin φ
    = (1 + cos φ) cos (ω t) - (sin φ) sin (ω t)

One way to think of it is that a phase shift corresponds to a rotation through the space defined by the cos() and sin() carrier waves. We can rewrite the above to do this sort of math in a much simpler vector notation:

    [1, 0] + [cos φ, sin φ]
    = [1+cos φ, sin φ]

This is really powerful. As long as you have a bunch of waves at the same frequency, and each one is offset by a fixed amount (phase difference), you can convert them each to a vector and then just add the vectors linearly. The result, the sum of these vectors, is what the receiver will see at any given point. And the sum can always be expressed as the sum of exactly one cos(ω t) and one sin(ω t) term, each with its own magnitude.

This leads us to a very important conclusion:

    The sum of reflections of a signal is just an arbitrarily phase shifted and scaled version of the original.

People worry about reflections a lot in wifi, but because of this rule, they are not, at least mathematically, nearly as bad as you'd think.

Of course, in real life, getting rid of that phase shift can be a little tricky, because you don't know for sure by how much the phase has been shifted. If you just have two transmitting antennas with a known phase difference between them, that's one thing. But when you add reflections, that makes it harder, because you don't know what phase shift the reflections have caused. Not impossible: just harder.

(You also don't know, after all that interference, what happened to the amplitude. But as we covered last time, the amplitude changes so much that our modulation method has to be insensitive to it anyway. It's no different than moving the receiver closer or further away.)

Phasor Notation

One last point. In some branches of eletrical engineering, especially in analog circuit analysis, we use something called "phasor notation." Basically, phasor notation is just a way of representing these cos+sin vectors using polar coordinates instead of x/y coordinates. That makes it easy to see the magnitude and phase shift, although harder to add two signals together. We're going to use phasors a bit when discussing signal power later.

Phasors look like this in the general case:

    A cos (ω t) + B sin (ω t)
    = [A, B]

      Magnitude = M = (A2 + B2)½

      tan (Phase) = tan φ = B / A
      φ = atan2(B, A)

    = M∠φ

or the inverse:

    M∠φ
    = [M cos φ, M sin φ]
    = (M cos φ) cos (ω t) - (M sin φ) sin (ω t)
    = [A, B]
    = A cos (ω t) + B sin (ω t)

Imaginary Notation

There's another way of modeling the orthogonal cos+sin vectors, which is to use complex numbers (ie. a real axis, for cos, and an imaginary axis, for sin). This is both right and wrong, as imaginary numbers often are; the math works fine, but perhaps not for any particularly good reason, unless your name is Euler. The important thing to notice is that all of the above works fine without any imaginary numbers at all. Using them is a convenience sometimes, but not strictly necessary. The value of cos+sin is a real number, not a complex number.

Epilogue

Next time, we'll talk about signal power, and most importantly, where that power disappears to when you have destructive interference. And from there, as promised last time, we'll cheat Shannon's Law.

Posted Sat Jul 26 01:11:15 2014 Tags:

Big Boys Don’t Cry (Tom Kratman; Castalia House) is a short novel which begins innocently enough as an apparent pastiche of Keith Laumer’s Bolo novels and short stories. Kratman’s cybernetic “Ratha” tanks, dispassionately deploying fearsome weapons but somehow equipped to understand human notions of honor and duty, seem very familiar.

But an element generally alien to the Bolo stories and Kratman’s previous military fiction gradually enters: moral doubt. A Ratha who thinks of herself as “Magnolia” is dying, being dismantled for parts after combat that nearly destroyed her, and reviews her memories. She mourns her brave lost boys, the powered-armor assault infantry that rode to battle in in her – and, too often, died when deployed – before human turned front-line war entirely to robots. Too often, she remembers, her commanders were cowardly, careless, or venal. She has been ordered to commit and then forget atrocities which she can now remember because the breakdown of her neural-analog pathways is deinhibiting her.

The ending is dark, but necessary. The whole work is a little surprising coming from Kratman, who knows and conveys that war is hell but has never before shown much inclination to question its ethical dimension at this level. At the end, he comes off almost like the hippies and peaceniks he normally despises.

There is one important difference, however. Kratman was combat career military who has put his own life on the line to defend his country; he understands that as ugly as war is, defeat and surrender can be even worse. In this book he seems to be arguing that the morality of a war is bounded above by the amount of self-sacrifice humans are willing to offer up to earn victory. When war is too easy, the motives for waging it become too easily corrupted.

As militaries steadily replace manned aircraft with drones and contemplate replacing infantry with gun-robots, this is a thought worth pondering.

Posted Thu Jul 24 19:23:23 2014 Tags:

2040 (Graham Tottle; Cameron Publicity & Marketing Ltd) is a very odd book. Ostensibly an SF novel about skulduggery on two timelines, it is a actually a ramble through a huge gallimaufry of topics including most prominently the vagaries of yachting in the Irish Sea, an apologia for British colonial administration in 19th-century Africa, and the minutiae of instruction sets of archaic mainframe computers.

It’s full of vivid ideas and imagery, held together by a merely serviceable plot and garnished with festoons of footnotes delving into odd quarters of the factual background. Some will dislike the book’s politics, a sort of nostalgic contrarian Toryism; many Americans may find this incomprehensible, or misread it as a variant of the harsher American version of traditionalist conservatism. There is much worthwhile exploding of fashionable cant in it, even if the author does sound a bit crotchety on occasion.

I enjoyed it, but I can’t exactly recommend it. Enter at your own risk.

Posted Thu Jul 24 00:09:36 2014 Tags:

For many years a major focus of Mono has been to be compatible-enough with .NET and to support the popular features that developers use.

We have always believed that it is better to be slow and correct than to be fast and wrong.

That said, over the years we have embarked on some multi-year projects to address some of the major performance bottlenecks: from implementing a precise GC and fine tuning it for a number of different workloads to having implemented now four versions of the code generator as well as the LLVM backend for additional speed and things like Mono.SIMD.

But these optimizations have been mostly reactive: we wait for someone to identify or spot a problem, and then we start working on a solution.

We are now taking a proactive approach.

A few months ago, Mark Probst started the new Mono performance team. The goal of the team is to improve the performance of the Mono runtime and treat performance improvements as a feature that is continously being developed, fine-tuned and monitored.

The team is working both on ways to track performance of Mono over time, implemented support for getting better insights into what happens inside the runtime and has implemented several optimizations that have been landing into Mono for the last few months.

We are actively hiring for developers to join the Mono performance team (ideally in San Francisco, where Mark is based).

Most recently, the team added a new and sophisticated new stack for performance counters which allows us to monitor what is happening on the runtime, and we are now able to export to our profiler (a joint effort between our performance team and our feature team and implemented by Ludovic). We also unified both the runtime and user-defined performance counters and will soon be sharing a new profiler UI.

Posted Wed Jul 23 14:37:24 2014 Tags:
I've begun prototyping a system for green processes in Qt applications. I'm not yet at the code-writing stage, but the API is definitely taking shape and a design is starting to emerge. It's a spare-time project for me, so is progressing at a reasonable but not extremely fast pace.

I've gotten to the point where early-stage feedback from people who are interested in such a framework would be very valuable in polishing the design a bit more before laying down code. As such, I'm looking for 4-6 people to collaborate with at this early stage.

If you are interested in using green processes in your Qt application, or are simply interested in new application development patterns for Qt, and you have 20-30 minutes per week to look at API drafts and provide feedback and and/or add to the ideas that are coming together, please email me (aseigo at kde.org).
Posted Sun Jul 20 06:01:00 2014 Tags:

Last time we talked about how wifi signals cross about 12 orders of magnitude in terms of signal power, from +30dBm (1 watt) to -90dBm (1 picowatt). I mentioned my old concern back in school about splitters causing a drop to 1/n of the signal on a wired network, where n is the number of nodes, and said that doesn't matter much after all.

Why doesn't it matter? If you do digital circuits for a living, you are familiar with the way digital logic works: if the voltage is over a threshold, say, 1.5V, then you read a 1. If it's under the threshold, then you read a 0. So if you cut all the voltages in half, that's going to be a mess because the threshold needs to get cut in half too. And if you have an unknown number of nodes on your network, then you don't know where the threshold is at all, which is a problem. Right?

Not necessarily. It turns out analog signal processing is - surprise! - not like digital signal processing.

ASK, FSK, PSK, QAM

Essentially, in receiving an analog signal and converting it back to digital, you want to do one of three things:

  • see if the signal power is over/under a threshold ("amplitude shift keying" or ASK)
  • or: see if the signal is frequency #0 or frequency #1 ("frequency shift keying" or FSK)
  • or: fancier FSK-like schemes such as PSK or QAM (look it up yourself :)).
Realistically nowadays everyone does QAM, but the physics are pretty much the same for FSK and it's easier to explain, so let's stick with that.

But first, what's wrong with ASK? Why toggle between two frequencies (FSK) when you can just toggle one frequency on and off (ASK)? The answer comes down mainly to circuit design. To design an ASK receiver, you have to define a threshold, and when the amplitude is higher than the threshold, call it a 1, otherwise call it a 0. But what is the threshold? It depends on the signal strength. What is the signal strength? The height of a "1" signal. How do we know whether we're looking at a "1" signal? It's above the threshold ... It ends up getting tautological.

The way you implement it is to design an "automatic gain control" (AGC) circuit that amplifies more when too few things are over the threshold, and less when too many things are over the threshold. As long as you have about the same number of 1's and 0's, you can tune your AGC to do the right thing by averaging the received signal power over some amount of time.

In case you *don't* have an equal number of 1's and 0's, you can fake it with various kinds of digital encodings. (One easy encoding is to split each bit into two halves and always flip the signal upside down for the second half, producing a "balanced" signal.)

So, you can do this of course, and people have done it. But it just ends up being complicated and fiddly. FSK turns out to be much easier. With FSK, you just build two circuits: one for detecting the amplitude of the signal at frequency f1, and one for detecting the amplitude of the signal at frequency f2. It turns out to be easy to design analog circuits that do this. Then you design a "comparator" circuit that will tell you which of two values is greater; it turns out to be easy to design that too. And you're done! No trying to define a "threshold" value, no fine-tuned AGC circuit, no circular reasoning. So FSK and FSK-like schemes caught on.

SNR

With that, you can see why my original worry about a 1/n signal reduction from cable splitters didn't matter. As long as you're using FSK, the 1/n reduction doesn't mean anything; your amplitude detector and comparator circuits just don't care about the exact level, essentially. With wifi, we take that to the extreme with tiny little FSK-like signals down to a picowatt or so.

But where do we stop? Why only a picowatt? Why not even smaller?

The answer is, of course, background noise. No signal exists in perfect isolation, except in a simulation (and even in a simulation, the limitations of floating point accuracy might cause problems). There might be leftover bits of other people's signals transmitted from far away; thermal noise (ie. molecules vibrating around which happen to be at your frequency); and amplifier noise (ie. inaccuracies generated just from trying to boost the signal to a point where your frequency detector circuits can see it at all). You can also have problems from other high-frequency components on the same circuit board emitting conflicting signals.

The combination of limits from amplifier error and conflicting electrical components is called the receiver sensitivity. Noise arriving from outside your receiver (both thermal noise and noise from interfering signals) is called the noise floor. Modern circuits - once properly debugged, calibrated, and shielded - seem to be good enough that receiver sensitivity is not really your problem nowadays. The noise floor is what matters.

It turns out, with modern "low-noise amplifier" (LNA) circuits, we can amplify a weak signal essentially as much as we want. But the problem is... we amplify the noise along with it. The ratio between signal strength and noise turns out to be what really matters, and it doesn't change when you amplify. (Other than getting slightly worse due to amplifier noise.) We call that the signal to noise ratio (SNR), and if you ask an expert in radio signals, they'll tell you it's one of the most important measurements in analog communications.

A note on SNR: it's expressed as a "ratio" which means you divide the signal strength in mW by the noise level in mW. But like the signal strength and noise levels, we normally want to express the SNR in decibels to make it more manageable. Decibels are based on logarithms, and because of the way logarithms work, you subtract decibels to get the same effect as dividing the original values. That turns out to be very convenient! If your noise level is -90dBm and your signal is, say, -60dBm, then your SNR is 30dB, which means 1000x. That's awfully easy to say considering how complicated the underlying math is. (By the way, after subtracting two dBm values we just get plain dB, for the same reason that if you divide 10mW by 2mW you just get 5, not 5mW.)

The Shannon Limit

So, finally... how big does the SNR need to be in order to be "good"? Can you just receive any signal where SNR > 1.0x (which means signal is greater than noise)? And when SNR < 1.0x (signal is less than noise), all is lost?

Nope. It's not that simple at all. The math is actually pretty complicated, but you can read about the Shannon Limit on wikipedia if you really want to know all the details. In short, the bigger your SNR, the faster you can go. That makes a kind of intuitive sense I guess.

(But it's not really all that intuitive. When someone is yelling, can they talk *faster* than when they're whispering? Perhaps it's only intuitive because we've been trained to notice that wifi goes faster when the nodes are closer together.)

The Shannon limit even calculates that you can transfer some data even when the signal power is lower than the noise, which seems counterintuitive or even impossible. But it's true, and the global positioning system (GPS) apparently actually does this, and it's pretty cool.

The Maximum Range of Wifi is Unchangeable

So that was all a *very* long story, but it has a point. Wifi signal strength is fundamentally limited by two things: the regulatory transmitter power limit (30dBm or less, depending on the frequency and geography), and the distance between transmitter and receiver. You also can't do much about background noise; it's roughly -90dBm or maybe a bit worse. Thus, the maximum speed of a wifi link is fixed by the laws of physics. Transmitters have been transmitting at around the regulatory maximum since the beginning.

So how, then, do we explain the claims that newer 802.11n devices have "double the range" of the previous-generation 802.11g devices?

Simple: they're marketing lies. 802.11g and 802.11n have exactly the same maximum range. In fact, 802.11n just degrades into 802.11g as the SNR gets worse and worse, so this has to be true.

802.11n is certainly faster at close and medium range. That's because 802.11g tops out at an SNR of about 20dB. That is, the Shannon Limit says you can go faster when you have >20dB, but 802.11g doesn't try; technology wasn't ready for it at the time. 802.11n can take advantage of that higher SNR to get better speeds at closer ranges, which is great.

But the claim about longer range, by any normal person's definition of range, is simply not true.

Luckily, marketing people are not normal people. In the article I linked above they explain how. Basically, they define "twice the range" as a combination of "twice the speed at the same distance" and "the same speed at twice the distance." That is, a device fulfilling both criteria has double the range as an original device which fulfills neither.

It sounds logical, but in real life, that definition is not at all useful. You can do it by comparing, say, 802.11g and 802.11n at 5ft and 10ft distances. Sure enough, 802.11n is more than twice as fast as 802.11g at 5ft! And at 10ft, it's still faster than 802.11g at 5ft! Therefore, twice the range. Magic, right? But at 1000ft, the same equations don't work out. Oddly, their definition of "range" does not include what happens at maximum range.

I've been a bit surprised at how many people believe this "802.11n has twice the range" claim. It's obviously great for marketing; customers hate the limits of wifi's maximum range, so of course they want to double it, or at least increase it by any nontrivial amount, and they will pay money for a new router if it can do this. As of this writing, even wikipedia's table of maximum ranges says 802.11n has twice the maximum range of 802.11g, despite the fact that anyone doing a real-life test could easily discover that this is simply not the case. I did the test. It's not the case. You just can't cheat Shannon and the Signal to Noise Ratio.

...

Coming up next, some ways to cheat Shannon and the Signal to Noise Ratio.

Posted Fri Jul 18 02:23:26 2014 Tags:

today i received spam from a private equity firm. details below for the search engines.

From: Steve Isom [mailto:smi@northbridge.com]
Sent: Thursday, July 17, 2014 5:40 AM
To: several-addresses@my-dayjob
Subject: North Bridge Introduction

Paul,

read more

Posted Fri Jul 18 00:12:05 2014 Tags:

A “non-blocking” IPv6 connect() call was in fact, blocking.  Tracking that down made me realize the IPv6 address was mostly random garbage, which was caused by this function:

bool get_fd_addr(int fd, struct protocol_net_address *addr)
{
   union {
      struct sockaddr sa;
      struct sockaddr_in in;
      struct sockaddr_in6 in6;
   } u;
   socklen_t len = sizeof(len);
   if (getsockname(fd, &u.sa, &len) != 0)
      return false;
   ...
}

The bug: “sizeof(len)” should be “sizeof(u)”.  But when presented with a too-short length, getsockname() truncates, and otherwise “succeeds”; you have to check the resulting len value to see what you should have passed.

Obviously an error return would be better here, but the writable len arg is pretty useless: I don’t know of any callers who check the length return and do anything useful with it.  Provide getsocklen() for those who do care, and have getsockname() take a size_t as its third arg.

Oh, and the blocking?  That was because I was calling “fcntl(fd, F_SETFD, …)” instead of “F_SETFL”!

Posted Thu Jul 17 03:31:57 2014 Tags: