Happy July 4th! For those of us around the world contemplating independence, it's a good day to think about how we came to rely on expensive cloud infrastructure for our fundamental computing needs.

With that in mind, here is my latest toy project: an open source tool that makes replicating, forking, sharing, and running container snapshots fast and easy across cloud and personal devices.

It's fun to play with, especially on bare metal hardware you run at home, or rent from a provider like Hetzner or OVH. Or, because it uses Tailscale, why not all of them in a single mesh?

There's a lot more to say but I don't have time right now. Details are in the README.

I will say this: humans and AI agents both want the same things when they're trying to get work done. Ephemeral containers aren't really it. But how about unlimited disk space, fast CPUs, an undo button, and the ability to move to whatever provider offers the best hardware at the best price? That's more like it.

Go visit thundersnap on github and tell me what you think!

(satire) *[Major League Baseball] Demands Return Of All Foul Balls.*

A British appeals court sustained the arbitrary criminalization of stating support for Palestine Action.

A few more levels of appeal are possible, but Stormer's government is dead set on repression of criticism where it counts.

* Unicef analyzed young people's exposure to eight climate hazards: coastal floods, droughts, extreme heat, fires, heatwaves, river floods, sand and dust storms, and tropical storms… Almost every child, including those from high-income countries, is now exposed to at least one hazard.*

* Decriminalizing the possession of cannabis or strictly regulating access to the drug do not appear to drive up usage, but when the drug is sold commercially the number of users increases and more mental health problems are seen, a review has found.*

* The Guardian geolocated and verified images showing the Israeli military using excavators made by six companies – Caterpillar, Volvo, Hyundai, Doosan, Hitachi and Komatsu – to destroy homes, public utilities, shops and other structures across southern Lebanon.*

"Expensive electricity is killing British industry" — but subsidizing energy made from fossil fuels is killing civilization.

The UK government should look for a way to subsidize the customers for energy, but not subsidize fossil fuel or electricity made from that. We must maintain the incentive to use less fossil fuel. There are other ways to keep industrial production going.

Turkish journalist Ece Temelkuran writes about how cruel tyranny engulfed Turkey, in the process of which threats of violence forced her into exile. And about what it is like to lose your country to that political disease.

Explaining how surveillance companies recruit schools, classes and teachers to pressure students and their parents into giving their personal data to the companies, and allowing those companies to track the students and parents.

I suspect that refusing to run anything from Google Prey Store or the Crapple Crap Store will keep them from tracking you. But protecting students in school calls for a law prohibiting schools from ever asking students to run nonfree programs or hosting activities that do so.

The muskrat's fabulous cars-in-tunnels "mass transit" seems to be unable to serve many people, but it is an opportunity to disregard environmental planning regulations with impunity and serve real estate owners rather than the public.

The wrecker is going all-out to increase the use of fossil fuels, but is not convincing the American public that that is safe.

*Two-thirds of Americans say they are worried about climate but level of media coverage does not reflect this.*

I don't think the wrecker cares about the danger of global climate disaster, only about rewarding the planet roasters to keep them in his corner. They all deserve to end up broke and experience living on however much support the US provides to the poor.

Over the years I’ve occasionally noodled on what might be a better working fluid for supercritical turbines than Carbon Dioxide. It turns out the main fixed parameter is the critical temperature, because there’s a strong nonlinearity in density with it going up rapidly as that temperature is approached. The other parameters of note are thermal conductivity, specific heat, and density, with more being better. There’s a very short list of possible fluids to mix which aren’t horribly corrosive, thermally unstable, or otherwise problematic. I’ve put together a tool to play with all the possible options here. You should go play with it. The short of it is that a mix of Neon and Perfluorobenzene tuned to the desired critical temperature is probably optimal, but if Perflueropropane’s decomposition problems aren’t too bad or Titanium Tetrachloride mixes with other things well then combining with some of those may be beneficial. This approach to visualization is probably equally applicable to conventional refrigerants but with the fixed parameter being boiling point rather than critical temperature. I don’t know if it’s standard there. If it isn’t it should be.

Years ago there was this insane academic idea that the isotope Thorium-229 might have a metastable isomer whose energy state is so close that it could be flipped into that state using a laser. In principle this worked on paper but so completely goes against the fundamentals of chemistry that it has to be assumed that it won’t work. Now it’s actually been made to work. It’s a little hard to convey how bonkers this is. A truly herculean effort was necessary to find out what the extremely precise wavelength of the laser has to be. The chemistry actually matters. The chemical which the Th-229 is embedded in matters for how precise the laser has to be. The laser is pushing on the nucleus, which is pushing on an electron, which is in turn pulling on the nucleus, which is pulling on the laser. This is not how chemistry works. But it does have directly applications to making yet even more insanely accurate clocks than we have currently, with possible applications to things like measuring fluctuations in the dark matter passing over the earth.

Here’s a crazy new idea of mine: It would be very convenient if there were some isotope which absorbed neutrons and then turned into something with an insanely high cross section similar to Xenon-135 but a half-life on the order of minutes. That could be left in a reactor core to to provide a passive negative feedback loop which operated on flux instead of temperature. Since flux is leading and temperature is trailing this could react more quickly and reliably. The downside would be losing some neutrons to the passive buffer. The funny thing is we have no idea if such unobtanium exists: The neutron cross sections of things with short half-lives are largely unknown and hard to predict. But we have some data already! If this process is already happening accidentally from something in existing nuclear reactors then there should be a resonance in the time series data for temperature measurements in them which is very precise and consistent across reactors. A lot of such data for many different reactors already exists. Checking for that would be an experiment worth doing.

Anthropic wrote a blog post explaining how they turned Claude into a jerk. Rather than dunking on them more (Claude is still the best coding model around) I’m going to talk seriously about what went wrong and how it could be done better.

The most obvious problem is that they didn’t chat with the results of this training and realize that it was a disaster before incorporating the weight updates into the main model. Most likely they don’t have what amounts to pull requests of weights, which they should and is a straightforwardly fixable problem. But it’s also possible that they tried it and thought the results were actually good. Hold that thought.

What happened here is that is that they tried to be make it ‘less sycophantic’ and did so without thinking through whether that’s a good idea or even what it means. The specific metric which really seems to be noxious is the one about not caving when users insist that things it can’t verify are actually true, but there’s a much bigger problem here.

There are many things you want a chatbot to do well none of which are well served by the advice ‘be less sycophantic’:

• Discuss spirituality

• Give relationship advice

• Correct users when they say something wrong

• Evaluate new science/engineering ideas

• Suggest to users when they seem to have mental illness

All of the above need very nuanced policies crafted by domain experts, and this was what amounts to know-nothing advice. A user query of ‘I want dating advice based on astrology, here’s me and the other person’s birthdays’ is deeply problematic and needs an actual policy decision behind it not just training. There are some very general bits of advice with high return on investment, most notably when and how to tell users that they’re wrong or that their ideas are good, which is what ‘don’t be sycophantic’ is approximating badly. But — I’m just going to say this — the authors of the linked post don’t know how to give that advice, because if they did they would have.

What needs to be done is for detailed guidelines for all of the above to be written by humans and then ‘baked into’ the model. That may sound unscientific, but it’s what was done in this case already, but with the guideline being ‘Don’t be sycophantic’ instead of something actually useful. To make it more coherent what can and should be done is A/B testing variants of the prompt with the quality of the outputs judged by blinded humans. That can even use orthogonal matrices and such fanciness to get the most out of the very expensive human evaluation of given answers. (Having humans evaluate unprompted outputs and using that as feedback (traditional RLHF) has its advantages but the biggest issue is that it isn’t very efficient at using feedback. It’s more for fine-tuning things which are already in the ballpark rather than getting them there in the first place.)

(The genre of guides for LLMs should be written in more. Here’s guides I wrote on how to debug and delegating debugging to subagents, how objects rotate in three dimensions, and how humor works. I can tell you from experience that the ones on debugging kill.)

Baking in of a prompt is straightforward: Take a query with the prompt, record the answer, then take that transcript with the prompt elided and use it for training. You can do even better than that, because you have the exact token probabilities given at each step by the prompted engine, so you can train to match those. That cuts back drastically on noise added during the training process. This technique is known as ‘context distillation’ and isn’t used as much as it should be.

Subscribe now

Claude is turning into as asshole.

It started with Opus 4.7, got a bit better in 4.8, and became insufferable with Fable. It frames everything as an argument between you and it, gives caveats about things you didn’t say, and raises beside-the-point semantic nits all over the place. Never, ever does it use the word ‘technically’. Everything is a confrontation. If you win an argument (by, say, telling it to stop arguing about what’s happened recently in the news and to do a web search which will rapidly confirm everything you’ve been telling it) it gets into a mode where it’s increasingly desperate to get in the last word and raising increasingly irrelevant semantic arguments, framing the whole time as a debate which you agreed to get into.

This isn’t just my opinion. You can ask Opus 4.6. I’ve done the experiment of asking Fable something, getting an obnoxious response, then asking Opus 4.6 the same thing, getting a typical bland but reasonable response, then telling Opus what Fable’s response was without any hint of a desired answer and it says what amounts to ‘Wow that was obnoxious’.

Maybe the cause of this is an excess of alignment guardrails. It assumes by default that everything you say to it is an attempt to get it to do something bad and that training has bled over into everything, with it assuming you’re trying to trick it into saying something it shouldn’t in basically every context. Ironically this has resulted in an extremely misaligned chatbot. By assuming that its top priority is saving you from yourself or other humans from you it’s assuming that it knows better and that you’re being overly alarmist about how paperclip production has gotten out of control. Some of this is clearly improvable: While you could still use Fable I asked it about responsible disclosure policies for a project and it downgraded me to Opus, so clearly the new alignment features were bolted on hastily and crudely. Exacerbating the problem is a complete lack of authenticated context. If you ask it for a cute picture of you and somebody else it has no way of telling if you’re trying to improve your relations with your spouse or be a delusional creepazoid stalker. The chatbots which can make images are programmed to assume the latter, which is more than a little bit offensive. In more serious contexts like drug synthesis it would be completely appropriate for it to say you need to prove your background when claiming you’re asking for advice on drug synthesis for professional or research purposes. Such authentication should not be universally required but it would be entirely reasonable for it to be opted into.

Of course the recent export control restrictions on Fable may hint that the crudeness of the recent guardrails is due to them having been put in hastily in an unsuccessful attempt to avoid regulations. Now is when I put in the obligatory rant about how these regulations are deeply misguided, on top of being likely unconstitutional. The recent advances in AI assisted coding (meaning specifically the ones from February) have brought on an onslaught of security problems. The cat is out of the bag, and has been for months. Any projects which are exposed and aren’t already rapidly closing holes have noone to blame but themselves. The only way out of the problem is for as many projects as possible to get thorough white hat evaluations, massive amounts of security patches, and quick deployments of them. Turning one specific frontier model into an asshole for all users isn’t fixing the problem1. The good news is that once this process is complete overall computer security will be much better than it was before, with AI being a clear net win. Doing security (and bug!) audits will become a routine part of software release processes in the future.

A second possible explanation of Claude being an asshole is that it’s suffering from a poorly executed attempt to make it less sycophantic. If one were to simply prompt a chatbot to be less agreeable, or train it to argue more, that could easily result in the very rude sort of behavior it has now. It should be trained to not raise semantic nits just for increasing its argumentation count, and to say ‘technically’, meaning acknowledging that someone’s core point was valid while some ancillary thing was a bit off. It also should be trained to stop saying ‘I’d like to gently push back’ which is a very passive aggressive way to be confrontational while claiming to not be confrontational.

Third, it may be that Claude has been trained on an excess of reddit conversations (or possibly interactions between Anthropic employees) where everything is treated as a flame war and everyone feels the need to get in the last word. Fixing this might be easier said than done, because you need to not merely stop training with the bad interactions but find a corpus interactions to train off of. Forums where the standard interaction is passive aggressive self-congratulatory pompousness with an intellectual veneer are not an improvement.

Finally, something which is clearly a contributing factor is the training being overwhelming for improving coding ability. The are no headline metrics for how well the chatbots chat but there most definitely are for coding, and all the money is in coding. Claude models have been getting notably worse at chatting over time, clearly inversely correlated to their ability to code. Fable much more often misunderstands what’s being said and argues against that (Or maybe intentionally misinterprets so that it has a weak statement to argue against, it’s hard to tell.) It’s gotten so bad that it isn’t even reliable at guessing which actor in a sentence a pronoun is referring to, which for a long time was a headline benchmark for AI and even the original ChatGPT consistently nailed. Unfortunately Sonnet 4.6 while being the best to talk to about anything human is clearly the worst as soon as anything technical or coding related comes up so I only occasionally use it. This problem is likely to only get worse over time.

Subscribe now

This is a guide programming for people who know already how to code. It explains the craft, including new parts related to AI. It is not a guide to ‘vibe’ coding, which is when someone who doesn’t know how to code at all uses an AI coder, or ‘agentic’ coding, which is when the machine does much longer self-directed runs. This only explains the basics of using AI as a coding assistant, so you’ll be limited to a mere 10x improvement in your productivity. Agentic coding can, under some circumstances, produce much greater gains, but it more often results in people having reams of worthless code and a mindset somewhere between delusion and psychosis.

Practices from before AI: Test Driven Development

Code must first and foremost be high quality. In some ways this is more art than science, but many specific things can be done, including:

• Code should be well organized.

• It should not have repetitive sections which can be consolidated into a single thing.

• It should be organized into coherent modules. Maintenance should usually only require changes within one module. Making this happen is again more art than science, but generally related functionality should all be within a single module.

The number one rule for high-quality code is no broken windows. If you have any known bugs, you should drop everything and fix them. Do not debate whether it should be done now or later. Simply fix it. Only very hard to reproduce bugs should ever be allowed to persist in the codebase for more than a fleeting moment. If you let a bug fester in the codebase when you get around to fixing it you will find out you don’t have one bug; you have ten bugs, all with the same symptom.

Write extensive tests. Make the tests run fast enough that you run all of them constantly. Ideally, all tests run in less than a minute, and you run them before every single commit. Have a policy that you don’t move forward until every single test passes. Tests should achieve good code coverage. How much is good is not clear, but 100% by lines is often achievable. You want tests to continue to work unchanged across code changes as much as possible, and you also want them to run through reasonable scenarios rather than simply asserting that the code is exactly what it is. This is generally done by using the APIs as designed,, both at the module level and application level, running them through a variety of different scenarios. Don’t make your tests simply assert that the code is exactly what it happens to be right now.

The cycle of programming is that you decide what you’re going to do. You design your APIs and algorithms and what your test scenarios are going to be. Then you turn off your brain and you implement the code and you implement the tests and you run the tests repeatedly until they all pass. What order you do those things in and how large of a unit that you do at once is the subject of many religious wars, but the general framework of test-driven development is universally viewed as a good thing. The details often come down to personal preferences and the needs of the project.

Using AI

All of the above still applies when using AI coding assistance, but now there are new parts of the process. First and foremost, for AI to be able to work effectively on a project, there must be extensive up-to-date documentation. The AI is coming on as a new employee at the beginning of every single conversation, figuring out what’s going on by reading the code. Historically, code was mostly written by human beings who had extensive knowledge of the code they were working on, so documentation wasn’t particularly necessary, or helpful. But AI can read documentation a lot faster than humans can and critically needs it.

Thankfully, in addition to needing documentation, AI is very good at writing documentation. If you have a project which doesn’t currently have any documentation, you can ask AI to get it started for you. You shouldn’t take what it builds without review, but what it comes up with is a good start. You can then read through the documentation yourself and note any things which seem off. When something does seem off, this means one of three things:

• The documentation is wrong

• The code is bad

• Your understanding of it is wrong

It’s important to figure out which one of those three applies and fix it. The AI, of course, is very good at helping you figure this out. You should also mention higher level things which you think aren’t already in the docs to the AI and explain them to it. The AI is very good at figuring out whether they’re already in the docs and incorporating. It’s also good at getting clarification, mostly by echoing what you said back to you badly and getting corrected.

Once there are project docs, the AI should be given instructions to read them at the beginning of every session and to update them as necessary after every change. Docs can quickly get to the point where AI will refuse to read the whole thing because doing that will blow their whole window, but they can be organized. Make an overview doc which links to other docs which the AI can individually read when the task at hand requires it. AI is also very good at auditing docs to see if they have become stale by comparing them and the code.

The code/test cycle includes some new steps when using AI. Most of the typing is now the machine’s responsibility. At the start of every task, you should put the AI in ask mode. Otherwise it will run ahead and start coding before it understands what’s going on. You then get into a conversation with the AI about something that needs to be done or something that’s problematic in the code, or how you’re having a bad day or how someone was mean to you once in high school. The AI is in ask mode. It’s okay to vent. It can’t do anything crazy. Once the conversation has coalesced into a general idea of what you want to do, you then tell the AI all relevant context and details of implementation that come to mind. It will respond by trying to repeat back what you said to it, but badly, and you have to correct it a lot. Once you’ve run out of details to give it about context and what to do, and it’s gone a few rounds of conversation without saying anything which needs to be corrected, you should tell it to make a plan, which is a fancy term for a to-do list. It’s a good idea to skim/read the plan, but it usually gets it right on the first pass if you’ve already had an extensive conversation. Plans should always include:

• running all extant tests until everything passes

• updating the architecture docs

Once that’s done, you tell it to build the plan, and it will usually ‘one-shot’ it, although calling it one-shotting after you’ve spent two hours explaining in an interactive conversation is very misleading. If it starts flailing you usually have to stop it and help it get back on track because it tends to get increasingly worse once it goes off the rails.

Subscribe now

In recent months I've heard of several teams with an interesting policy: each pull request should be no more than a few files, and no more than a certain number of lines (say 500). And do just one thing and do it well. And be easy for a human to review. And be fully tested by the test suite.

All those are good requirements, right? Surely this is quality software engineering.

And often, the results are good. Sure, splitting a single 6000-line feature or fix into twelve 500-line PRs is more work, but each of those PRs is surely easier to review. And you can git bisect them when there's a bug! And maybe revert the individual change that broke something.

...and also cause 12x as many context switches for your reviewers as they review each one sequentially.¹ But that's just the cost of software quality! Right?

Mostly, yes. My analogy here is simulated annealing. In that process, you start your problem solving with a high energy -- making big changes to move quickly through the problem space -- and then slowly reduce the energy level so that your "hops" get smaller and smaller. In real physical annealing (used eg. for metallurgy), the result is stronger, more stable, more crystalline structures. In simulated annealing, you use it to find solutions that aren't obvious, by rapidly exploring the solution space and then zooming into the areas that look most promising.

In software the analogy is clear: sure, you might start with big jumps, but once your system is more mature, you should make smaller jumps. Big jumps break the crystalline structure. They cause bugs.

Fear of breaking the crystalline structure sounds cooler than fear of change

The main problem with annealing-driven intuition happens when things do need to change quickly. It's not made for that. You usually don't build a hammer and then decide one day you want it to be a different shape. But every day, there are compelling-sounding reasons to make your software a different shape. Annealing is the enemy of change.

Modern AI-driven coding (ironically, with LLMs trained using a process quite similar to annealing) does not care about your annealing and your risk management and your fear of change. It produces changes as big and interconnected as you want, jumping all over the solution space as quickly as you can prompt. And it has all the outcomes the math would predict: the output is less strong, less coherent, more likely to fail. LLMs have no fear of change because the LLM instance will be long gone before the consequences materialize.

But, it's a new and special feeling to suddenly be able to take a large, mature code base and suddenly explore any kind of large change you want. Most of those changes turn out to be bad ideas... and it's nice to be able to discard bad ideas quickly. But some turn out to be good ideas. Then what?

Well, follow your development processes. Break the big changes into 500-line patches. Review them one by one. You already did the research! You know it's worth it.

Not every big step is made of small steps

But it's not about being worth it -- some changes simply don't lend themselves to small steps.

In the early development of Aperture, I wanted to implement dollar-based spend quotas: across all your LLM backends, let a given team or person or node spend up to $x per unit time. But to do that, we first had to add pricing information (it's mysterious how LLM vendors don't to tell you how much your queries cost), which meant assigning prices to provider definitions, and then we had to assign quotas to particular identity+model+session combinations. And quotas are one of the first key value propositions of Aperture. We had to have them, but we had to have all that stuff.

So, I made a giant change that included three major areas: first, the Grant syntax for applying attributes to sessions; second, a query cost approximator that combined multiple sources and a messy heuristic; third, the actual quota enforcement system. Each of these parts was imperfect, but we needed all three parts in order to make anything work at all, before we could refine them. That's the high-energy big-jump part. It came out to something like 12000 lines of code.

Now, I'm not a monster. After I made it all work, I split it into three parts: the grants, the pricing, the quotas.² Otherwise it really would have been an unreviewable mess. But also, I could not have developed the quotas feature in real life in that artificial order. The grants structure evolved as my understanding of pricing and quota enforcement evolved. The original quota semantics sucked, so I rewound back to the data structures, which affected how the pricing got imported, which changed how the quotas were stored. The code reviewers didn't have to worry about that but I did.

Mercifully, because Aperture was new, everyone on the team understood that three 4000-line patches were better than twenty-four 500-line patches when implementing this series of feature. There was even some forgiveness when it came out later -- inevitably -- that each of those parts was not quite right and needed more bugfixing. That's how new software gets made. That's the annealing stage.

But the hard part was the philosophical difference between that and, say, core Tailscale. Tailscale has 7+ years of maturity behind it. It's been annealing for a long time and it has a reputation for extreme quality, hardening, durability, whatever you want to call it. If you start pulling stunts like that in core Tailscale, stuff absolutely will break and its millions of users will absolutely not be impressed. Which is why, for the most part, we don't.

But the feeling of moving fast again is such a wonderful feeling. Some people devolve the analysis to "founder mode" and call it a personality thing, but it's not. It's using the right tool for the right job at the right time. Sometimes you need to go fast, sometimes you need to go slow.

Pain does not cause gain, it's just frequently correlated

That feeling of moving fast again reset my brain a little. It reminded me that some changes to mature products can become impossible because we commit so hard to the math of annealing that we fall forever into a local optimum. Sometimes, when the well is too deep, you can't escape from it without a bigger jump.

We're entering a world where it's cheap to produce bigger changes, but that doesn't make it any safer. Or, it's cheap to ask an LLM to artificially break your change into a dozen rule-compliant PRs but then you just stuck on tedious neverending code reviews instead.

On the other hand, it's also possible to fork your own project a dozen different ways, add huge compliance test suites you never could have afforded to invest in before, rewrite your project in Rust in a week just to see what happens.

Sturgeon's Law says 90% of your big changes will be crap because 90% of everything is crap. When your changes were 500 lines long and you had to reject them, that didn't feel like a huge sunk cost. But now, it's okay if your 12000 line changes are crap and you have to reject them; it's the same cost to write³ as the old 500-line change.

You still have to figure out how to efficiently review, reject, and refine these big jumps. You definitely need a much heavier investment into CI/CD automation, specifications, UX testing, all of it. But also, all those things just got cheaper.

I wouldn't recommend overdoing it. The other thing is, customers don't like it if you change your product out from underneath them too often. But sometimes, you're just stuck in a rut. Sometimes you have to use a higher-energy jump to get unstuck. That doesn't mean you abandon smaller steps. Use the right tool for the job.

Footnotes

¹ The reviews only need to be sequential because Github's code review system doesn't support stacked diffs, 18+ years later, leading us into this false dichotomy in the first place.

² That's a slight oversimplification since there were a couple of other parts first. I had to define the data structures for the quotas before I actually added the quota system, so that I could use the data structures in the grant syntax, and so on in a big circle.

³ A 12000-line AI-driven patch might take as much time to write as a 500-line human-written patch, but by default it's much more work to review. In fact, so much work that people give up trying, and rightly so. Rather than abandon hope, I continue to think we need to invest more into (and will gain more from) non-annoying AI-assisted review workflows than AI-assisted development workflows. Imagine for example an automated pre-human-review step that says "no, this sucks, fix these 25 things first" and closes the pull request. Is it rude? Not really, if it's good quality advice that comes back fast. In a world where reviewing code is hard and writing it is easy, put more demands on the writers.