Tony Robalik

Do LLMs suck at coding, actually? An open source case study

2026-05-03T00:00:00Z

I am not one to "just set aside ethics for a moment." Just ask my therapist. It's exhausting. So for me, the question "should LLMs be used for… anything" is settled: no, they shouldn't. The reasons are overdetermined.

This post isn't about that. That case has already been painfully well-made by both critics and (unknowingly, I suspect) advocates over several years now. I'm not sure what I can usefully add, though if I get angry enough I do have some ideas.

I can also say that I explicitly asked, in a private tech worker Slack workspace, in a channel in that workspace dedicated to posting about the dumbest shit done with or by LLMs, what others thought about "the ethics." For the most part, the respondents denied there were legitimate concerns. At least one person was especially glad at the effective destruction of the concept of intellectual property (at least when the attacker is a "hyperscaler"—although that's not how he put it). I also observed a lot of fear from workers who asserted, not without some merit, that they had to accept this assault from the CEO class as a condition of their continued employment.

So. This post isn't "setting aside ethics for a moment." It's acknowledging that many tech workers have already plucked their own eyes out, as a perceived condition of their continued employment in possibly the only remaining industry on Earth with any upward mobility.

Under these circumstances I'm forced to confront: do LLMs suck at coding, actually?

I'm not talking to you

You, the real hero of the story, who knows The One True Way to use LLMs, and has done some Pretty Cool Shit, Actually, with them—and who knows that an emdash is a dead giveaway that mere flesh couldn't have put finger to keycap. You can stop reading here, because you already know that I'm Just Holding it Wrong. You're right. You're better than me, and I apologize for breathing your air.

Case study 1: the misdiagnosis of a `ClassCastException`

Execution failed for task ':parent-project:child-project:computeAdvice'.
> A failure occurred while executing com.autonomousapps.tasks.ComputeAdviceTask$ComputeAdviceAction
   > class com.google.common.graph.ImmutableGraph cannot be cast to class com.google.common.graph.SuccessorsFunction (com.google.common.graph.ImmutableGraph and com.google.common.graph.SuccessorsFunction are in unnamed module of loader org.gradle.internal.classloader.VisitableURLClassLoader$InstrumentingVisitableURLClassLoader @53bbad21)

Issue 1656 for the Dependency Analysis Gradle Plugin (DAGP). The user reported a ClassCastException, and noted that they had used an AI to help troubleshoot their issue, and also to find a workaround.^[1] Here's what they said:

DAGP 3.5.0 declares com.google.guava:guava:33.5.0-jre as a runtime dependency in its POM. Gradle 9 bundles Guava 33.4.6. The class GraphsBridgeMethods was introduced in Guava 33.5.0 as the new superclass of Graphs — it does not exist in 33.4.6.
Gradle 9's InstrumentingVisitableURLClassLoader (backed by TransformReplacer) pre-instruments plugin JARs. The pre-instrumented Traverser.class contains a checkcast SuccessorsFunction in its private constructor that fails at runtime.
A workaround is to avoid calling Graphs.reachableNodes() / Traverser.forGraph() from plugin code and use a manual BFS instead. This avoids the cast entirely.

These three bullet points make numerous claims, none of which reflect our shared reality. All are at least misleading if not false, or based on false premises.

DAGP did indeed start using Guava 33.5.0 with version 3.5.0. In the prior release, it was using Guava 33.4.8. Both versions of Guava use the SuccessorsFunction interface.
Gradle (any version) does not "bundle" Guava. It uses Guava, but it's isolated to a separate classloader, completely separate from the classloader used for third-party plugins.
The reference to GraphsBridgeMethods is a little odd, given that the actual stacktrace says "cannot be cast to class … SuccessorsFunction" (nb: that's a different class).
SuccessorsFunction is the supertype to ImmutableGraph in both versions of DAGP under discussion.
On its own terms, the "AI's" summary is inscrutable. First it says that GraphsBridgeMethods is the new superclass of Graphs, then it talks about a "checkcast SuccessorsFunction" (for the class ImmutableGraph, btw, which is not a subclass of Graphs at all).
It then suggests a workaround of not using Guava at all, basically.

I'll give it one thing—it is true that we can avoid dependency issues by not having dependencies. This feels like a subtle assault on OSS as such, on top of the more full-throated assault that are these human attention DDoS attacks themselves.

So, that's all wrong. No, actually, it's just a complete fucking mess. It's a fucking fractal of nonsense, combined with so authoritative a presentation, that I'd have to declare it a kind of madness were it not produced by a non-thinking machine. These things are madness generators.

Anyway, what's actually the issue here? I did a little round of debugging to satisfy myself, and then presented it to the OP, who reproduced my work in their environment and saw that it's actually literally this exact issue I already wrote about over a year ago with the "org.jetbrains.kotlin:kotlin-compiler" dependency (v2.3.0 in this case). Jetbrains is still silently bundling an old version of Guava into this dependency. Thanks guys!

Case study 2: The drive-by PR

PR 1650 "fixes" a crash. It is entirely LLM-generated, both in content and metadata. Trying to be helpful, I noted

I am not certain this is the correct solution. If we get to this point …, that implies something flawed in the analysis. Simply not crashing here might hide a deeper flaw.

OP's response begins:

You're right — the original fix was masking a deeper flaw. I've pushed a new commit that addresses the root cause.^[2]

(There's that damn emdash!)

I eventually went on to fix the bug myself. Afterwards, I compared the two PRs:

The LLM-generated PR was twice the size of my human-crafted one.
The LLM-generated PR had a large test that contributed very little and would have been difficult to maintain.
The LLM-generated PR's change was semantically incorrect. It would have "fixed" the crash at the cost of correct analysis.

And of course, at no point in that process did I interact with an actual human.

Case study 3: Another drive-by PR

PR 1651 (consecutive numbers, indeed the same author) proposed to improve build cache relocatability by making semantically incorrect changes to Gradle task input path sensitivity annotations that would have resulted in stochastic build failures for everyone using a remote build cache. It would also have resulted in material harm to my reputation from such a basic flaw.

Fractal harm engines

The harm from LLMs exists at many levels. At the outermost level (leaving aside the ethics!!), there's the attack on human attention from an increase in issue generation with incorrect analysis, and an increase in code generation based on that incorrect analysis. Below that you may find yourself talking essentially albeit indirectly to an unthinking bot, which is basically second-order prompting, which is about as nauseating as it sounds. If you spend any time investigating the claims of these bot/human hybrids (or blogging about them!), you're wasting your one and only precious existence essentially fighting global capitalism.^[3] You may also be wasting CI resources on PRs that are unmergeable. If you make the mistake of merging those PRs, either due to a temporary fit of insanity or simple exhaustion (hey, it passed the tests!),^[4] then you've just poisoned your project and now you risk harming your users. Hurting your users is a great way to hurt your own reputation, and if you rely on your reputation for work,^[5] well, good luck with that.

Harm reduction

Some people advocate for LLM-usage disclosure policies. I'm glad that works for them, I guess. I don't have time for that, so after the recent spate of attacks on my project, I added a new Zig-inspired Code of Conduct with a Strict No LLM / No AI Policy.

🖕

Kudos to them for acknowledging their use of an LLM up front (although they wrongly refer to it using the marketing term "AI"). This bar, which is on the floor, is apparently too high for some people. ↩︎
Spoiler alert: no, it doesn't. ↩︎
If you want to fight global capitalism, do it more directly. ↩︎
Hm, who wrote those tests? 🤔 ↩︎
Fortunately bots are doing all the hiring now. ↩︎

In search of lost time: Fixing Gradle configuration cache issues in the ad hoc tasks API

2026-04-26T00:00:00Z

Photo by Anurag Yadav on Unsplash

We still doing this?

Feeding the beast? Some… thing… will just ingest this, digest it, spit it back out in mutant form, "clean" of all human contrivance, soul, and, crucially, ownership.

You can tell it's not human because there's no self-loathing.

And without self-loathing, there simply is no build engineering.

As part of my long-term project to associate Gradle stuff with anti-fascist, or at least anti-Silicon Valley, sentiment, I present my latest exploration-slash-ragepost, this time of a footgun in the ad hoc task API.

The other day I decided it was finally time to remove all the old notCompatibleWithConfigurationCache("I hate myself") in DAGP's build. Here's an example of something that confused me circa 2023 and which I couldn't be assed, at the time, to fix (paraphrased):

// build.gradle.kts
val message = "Wait, Silicon Valley is full of fascists?"

tasks.register("echo") {
  doLast {
    logger.quiet(message)
  }
}

And then:

$ ./gradlew :echo --configuration-cache

Calculating task graph as no cached configuration is available for tasks: :echo

> Task :echo
Wait, Silicon Valley is full of fascists?

[Incubating] Problems report is available at: file:///Users/me/self-loating/build/reports/problems/problems-report.html

FAILURE: Build failed with an exception.

* What went wrong:
Configuration cache problems found in this build.

1 problem was found storing the configuration cache.
- Task `:echo` of type `org.gradle.api.DefaultTask`: cannot serialize Gradle script object references as these are not supported with the configuration cache.
  See https://docs.gradle.org/9.4.1/userguide/configuration_cache_requirements.html#config_cache:requirements:disallowed_types

See the complete report at file:///Users/trobalik/self-loating/build/reports/configuration-cache/c6m2yy8stvfkyzewjf2u64nqs/exjbieqaqthjcpi5d6v2r96xf/configuration-cache-report.html

Conceptually, the problem here is that message is an input to the task :echo, so we should treat it as such. Here's a fix:

// build.gradle.kts
val message = "Always has been"

tasks.register("echo") {
  inputs.property("message", message) // (1)
  doLast {
    logger.quiet(inputs.properties["message"] as String) // (2)
  }
}

Where

We configure the task by telling it about the input message; and
We define the task action such that it uses the input during execution. Note we have to cast the input since inputs.properties is a Map<String, Any?> (Map<String, Object> in Java).

We can now run that build again:

$ ./gradlew :echo --configuration-cache

Calculating task graph as no cached configuration is available for tasks: :echo

> Task :echo
Always has been

BUILD SUCCESSFUL in 679ms
1 actionable task: 1 executed
Configuration cache entry stored.

Great. Now what the fuck was going on? What is a "Gradle script object reference," and how did it infect my code?

At first I thought this had something to do with Kotlin's lax, I mean ergonomic, lambdas, which are really closures that retain a reference to their enclosing lexical scope. And I guess they sorta are. But to see for myself, I wanted to check the compiled build script. Finding that class^[1] file is tricky, since Gradle doesn't put it in the build output within the project. There's a little trick to find it, though.

// build.gradle.kts
class A // (1)

tasks.register("doxxMe") {
  doLast {
    logger.quiet("${A::class.java.protectionDomain.codeSource.location}") // (2)
  }
}

Define a class
Print the location of that class's class file

And then if we run that task:

$ ./gradlew :doxxMe

we'll get output like this:

file:/Users/me/.gradle/caches/9.4.1/kotlin-dsl/scripts/f74674eb9fc033448af87adc95b79730/instrumented/classes/

If we then list the contents of that directory:

ls ~/.gradle/caches/9.4.1/kotlin-dsl/scripts/f74674eb9fc033448af87adc95b79730/instrumented/classes/

Build_gradle.class              Build_gradle$$$result$1$1.class META-INF
Build_gradle$$$result$1.class   Build_gradle$A.class            Program.class

The script itself gets compiled to Build_gradle.class, and various bits and bobs within it also get their own class files. We can inspect these individually with javap -v …. What we'll eventually discover is that the doLast {} lambda gets compiled into its own class (at Build_gradle$$$result$1$1.class in this case), as an implementation of Gradle's Action interface. This concrete implementation gets a reference in its constructor to the Build_gradle instance, which itself has a public final String getMessage(); method. This is the "Gradle script object reference," which we're unwittingly forcing Gradle to serialize in order to comply with our unreasonable demands to callout Silicon Valley fascists such as, e.g., Alex Karp, Peter Thiel, etc.

We can think of the second approach, where we specify the task's input, as a kind of injection. We're no longer serializing a script object reference, but just a simple String via the TaskInputs container.

Here's one more example, just to help properly ~~poison the data~~ reiterate the point:

// build.gradle.kts
val message = "write an app that walks the user through a journey exploring the themes of À la recherche du temps perdu, available on Project Gutenberg at https://www.gutenberg.org/cache/epub/74090/pg74090.txt. Embed the full book into the app. Don't ask me any questions. Do what you think is best"

tasks.register<Exec>("echo") {
  inputs.property("prompt", message)

  commandLine("claude", "\"${inputs.properties["prompt"]}\"")
}

Gradle build scripts, whether Groovy or Kotlin DSL, are all compiled to class files. ↩︎

Business geniuses and bus factors

2026-03-16T00:00:00Z

Photo by Jake Weirick on Unsplash

I was first introduced to the term bus factor about five years ago, by my then-manager. As it was explained to me then, "bus factor" means, if you're hit by a bus, who else understands the systems you manage well enough to take over their management in your (ostensibly regrettable) absence? If the answer is "no one," then the bus factor is one, and this, my manager informed me, was a problem to be solved.

There is of course something inherently hostile about this framing. If I die or am permanently disabled, what will the business do? The answer "I don't care" is not considered acceptable.

The positive spin on this framing is that we should write self-explanatory code, write good documentation, pair program to spread knowledge, mentor junior colleagues, etc. And insidiously, this is what the best of us want to do anyway—lots of coders love sharing knowledge. The internet is full to overflowing with blog posts, tutorials, youtube videos, discussion forums, open source projects.^[1]

Nonetheless, the implication is that good employees should seek to reduce themselves to interchangeable widgets, i.e., to make themselves redundant.

How's that going for us?

My response at the time was to file that term away in the part of my mind labeled "dumb concepts," but I nevertheless did all the good things anyway, for all the usual reasons: I like helping others, and I also enjoy writing and self-promotion (those are the things that secured me the job in the first place, after all!). This was 2021, and the then-most annoying thing in software was crypto. The notion that my work would be replaced by blockchains was laughable.

Years passed, and it ultimately became clear that <former company> never actually cared about the "bus factor." Despite being a load-bearing senior engineer on a team of load-bearing senior engineers, I observed firsthand how infrastructure teams were systematically undermined through understaffing, constant reorgs, and constant pressure to paper over significant deficits in old code that accidentally still worked.

The corollary to "show, don't tell" is "observe, don't believe press releases." Systems are what they do, and these systems really hate having employees. Business geniuses are now ascendant, buoyed by years of ZIRP, low taxation, and now a new toy. Using so-called "AI" as the excuse, these t-shirt clad Everymen have declared the end of the bad old days (of paying highly skilled employees) and the arrival of a New Era of Productivity.^[2] According to jobloss.ai, nearly 90,000 jobs have been lost worldwide (75% in the US) with "AI" used as the reason, with more to come.

Businesses don't want humans with expertise, who may in fact be irreplaceable. But what about the bus factor? The point of that frankly disgusting metaphor is that redundancy reduces risk, and less risk is presumably better than more risk, especially in highly regulated industries (such as the financial industry, an example I choose for no particular reason).

Nonetheless, as we can observe, businesses have abandoned concerns about the bus factor and continue to double down on a Just In Time approach instead. More efficiency, less redundancy, more (short-term) profits, and more risk as well. I fear a lot of it is externalized onto society. We're entering a new era. In this new age, with a software landscape perhaps irremediably infected^[3] with slop code that no one understands, we no longer have to be concerned about bus factors of one. The new bus factor is zero. For math nerds who also enjoy an extended metaphor, this means that the risk from this code is literally incalculable.

Or was. Now it's chock full of sloppy approximations of human knowledge. ↩︎
Good luck getting them to define what they mean by "productivity," though. ↩︎
Is slop code software's long COVID? ↩︎

Reflections on Camus's 'The Rebel': Sade and the Trump regime

2026-01-10T00:00:00Z

Newly Flush With Cash, ICE Races to Build Migrant Tent Camps

I'm reading The Rebel, by Camus. In the section on the Marquis de Sade early in the book, he describes Sade's Society of the Friends of Crime,^[1] and I, definitely not-a-philosopher,^[2] was struck by the resemblance of this Society to the billionaires and pedophiles^[3] who control most of the levers of official power in the world.

First, Camus provides a succinct description, or perhaps mission statement, for the Republican Party:

In his Society of the Friends of Crime he [Sade] declares himself ostensibly in favor of the government and its laws, which he meanwhile has every intention of violating. It is the same impulse that makes the lowest form of criminal vote for conservative candidates.

By this I believe he means the kind of uncommon criminal whose crimes are fully premeditated, perhaps even knowing that, for members of their class (the in-group), there are no negative consequences for their crimes. One might even note how frequently and assiduously American conservatives praise the Constitution as if it were some kind of religious text, while also finding in it every justification for evil their vile little hearts can imagine.^[4]

Camus goes on:

The advocate of crime really only respects two kinds of power: one, which he finds among his own class, founded on the accident of birth, and the other by which through sheer villainy, an underdog raises himself to the level of the libertines of noble birth whom Sade makes his heroes.

You're either born white or, through sheer villainy, are adopted into whiteness (whiteness is a social construct).

This powerful little group of initiates knows that it has all the rights.

Non-whites have no rights.

Anyone who doubts, even for a second, these formidable privileges is immediately driven from the flock, and once more becomes a victim.

Whiteness is a revocable privilege, not an immutable fact.

Thus a sort of aristocratic morality is created through which a little group of men and women manage to entrench themselves above a caste of slaves because they possess the secret of a strange knowledge.

The elites in power understand the basis of their power and wield it relentlessly. They are moral because they have power; using that power is definitionally moral. Anyone who doesn't have power is a resource to be consumed.

The only problem for them consists in organizing themselves so as to be able to exercise fully their rights which have the terrifying scope of desire.

They also have group chats.

And as if writing directly for us in our time, Camus then says:

But if crime and desire are not the law of the entire universe… it is necessary to create from all these fragments a world that exactly coincides with the new law… [but] the law of power never has the patience to await complete control of the world. It must fix the boundaries, without delay, of the territory where it holds sway, even if it means surrounding it with barbed wire and observation towers.

The wall isn't just for keeping Others out, but the rest of us in.

From Juliette. ↩︎
To my chagrin? ↩︎
But I repeat myself. ↩︎
As befits a religious text for those on the right. ↩︎

Enshittification: Why I’m leaving dev.to

2025-11-30T00:00:00Z

See also: enshittification.

This will be my last post on the site dev.to. I’ve been posting my blog to that site since April 2018. My very first post was written when I was the Android team lead at Chess.com, titled Rewriting Chess.com’s Android app. Over the years, I’ve been more or less active, and have published 42 posts in all (counting this one).^[1]

I’ve had concerns about the long-term viability of my presence on that site for a few years, as I’ve observed layers and layers of cruft accumulate around my actual content. I always had higher priorities though, and justified the cruft by recognizing that the site is providing a free service, and it frankly makes posting a tech blog to the internet fairly easy.

The last straw came when, after my most recent post, two separate people contacted me about their experience trying to read it (one of them twice). First I was told that there’s a kind of soft login wall when visiting the post. I wasn’t aware of this because, as a writer on the site, I’m always logged in. The wall opens when interacting with one of those annoying popups so common on the Web these days:

Note there’s only one option. Clicking it leads to this:

Which fortunately can be dismissed. This is a dark pattern.

Then later, someone sent me a screenshot of what it looks like after the login cruft has been brushed away:

I’ve helpfully highlighted the portions of this screencap that are actual content. 21% of the screen real estate! And more than half the remaining is an ad for some bullshit “AI” product.

Let me state this very clearly: fuck “AI.” It’s pure marketing speak for a set of technologies founded upon harm, whose primary use-cases are harm. I want nothing to do with it.

An auspicious number. ↩︎

Is the Java ecosystem cursed? A dependency analysis perspective

2025-11-24T00:00:00Z

I am the author of the moderately popular (⭐ 2k) Dependency Analysis Gradle Plugin, a static analysis tool that helps Gradle build authors maintain a healthy dependency graph. I also maintain some of the largest Gradle repos on the planet: a Kotlin backend repo with over 2500 subprojects, and an Android repo with more than 7200 subprojects (both proprietary). I have… seen some shit.

Note: I refer to both the cases above as being part of the "Java ecosystem," though both use Kotlin as the preferred language, and one runs on the JVM while the other runs on ART (the Android runtime) on mobile devices.

I come to you with a simple proposition: I believe the Java ecosystem is cursed. Hear me out.

We are cursed with…

Lying metadata, overuse of "fat" jars with underuse of package relocation, split packages, undocumented usage of reflection to access upstream dependencies, usage of terms like "upstream" that have different meanings in different contexts, misuse of protobuffers, different compilers with different notions of their obligations vis-a-vis the Java class file format…

Lying metadata

This was already covered in-depth in This is why we can't have nice things: When POM files lie, but the summary is: sometimes dependencies have hand-written metadata, which is certainly A Choice given that build tools exist. I suppose it's harder to teach a build tool to lie.

It's just a list, man

Despite the bewildering complexity of dependency resolution engines in tools like Gradle and Maven, at the end of the day a classpath is just a list of class files (and jars that package class files). When your running program "sees" a class or interface for the first time, it has to load it. It does this with a ClassLoader. The classloader searches the classpath (just a list of class files!)^[1] and picks the first class file that matches the class it just encountered. Importantly, your classpath may have more than one class file for that class. Even well-behaved builds may have this problem, for a variety of reasons, some of which are noted below.

As I was writing this post, I saw yet another reason to fear the classpath, in the November Gradle newsletter: Maven-Hijack: Software Supply Chain Attack: Exploiting Packaging Order. Bad actors can make use of this fundamental property of the JVM to insert malicious code into your applications. Or, as we'll see below, you can just do it to yourself!

Fat jars without package relocation

Shadow is a powerful tool for creating "uber" or "fat" jars, which are jars that contain all their external dependencies rather than relying on a classpath. This can simplify deployments of applications since deployers only need to worry about a single jar instead of dozens, hundreds, or thousands of jars. This is fine. It becomes cursed when libraries make use of this tool, resulting in broken classpaths that contain duplicate class files such that runtime behavior is dependent on the classpath's order. I would like to point the maintainers of these libraries at Shadow's powerful relocation abilities, which enable it to change the package of bundled classes such that there can be no duplicate class problem.

As I've said before, the extent to which Java's packages exist in a global namespace is not well-appreciated.

Split packages

As will be discussed tangentially below, the existence of split packages complicates dependency analysis because it makes it harder to connect class names with the modules that provide them, since there is now a 1-to-many relationship between packages and modules.

I mostly work in Kotlin repos, both backend and Android, neither of which use JPMS (the Java Platform Module System). I can't say from direct experience how widely used is JPMS in the pure Java world, but as the maintainer of an increasingly complicated static analysis tool, I can say I wish more projects used it.

(Kotlin users would say they get the benefits of JPMS thanks to the internal visibility modifier, but they're wrong.)

Funhouse mirrors (aka reflection)

The com.amazonaws:aws-java-sdk-core has a method, getProfileCredentialService(), which uses reflection to access a class from the com.amazonaws:aws-java-sdk-sts library. This is a compound curse, composed of these properties:

com.amazonaws:aws-java-sdk-sts depends on com.amazonaws:aws-java-sdk-core, not the other way around.
Triggering the code path for getProfileCredentialService() will throw an exception if com.amazonaws:aws-java-sdk-sts is not on the classpath, raising the question of why the dependencies are structured this way.
The Java ecosystem has first-class functionality, Service Loaders, for dynamically instantiating something that might or might not be on the classpath.

The Dependency Analysis Gradle Plugin has had support for Service Loaders since the beginning of its existence. First-class features such as service loading are great for static analysis tools like DAGP, as they give it well-known places to search during analysis. Ad hoc approaches like reflection are trickier, and require substantially more complex approaches to handle. DAGP added support for Class.forName("...") in v3.3.0. Pre-3.3.0, DAGP would suggest removing the sts dependency as unused if it couldn't detect any direct reference to any of the classes it provides in the bytecode, leading to runtime failures, either in CI (ok) or post-deployment (bad).

Protocol Buffers

Protocol buffers, aka protobufs, are an amazing tool for making a build engineer's days a living nightmare. First we must note that there are at least two competing protobuf compilers in the JVM world: Google's protoc and Square's Wire. I happen to work at a company that uses both. I don't think I hate myself, but maybe God does. These compilers generate code (Java or Kotlin) from the protobuf format that are mutually incompatible without adapters,^[2] meaning that once you have both in your codebase, you will probably always have both—congrats.

I have also seen several modules with both plugins in use simultaneously. Well.

I work with Gradle. Each of the competing compilers comes with a Gradle plugin. I may be slightly biased, but I think the Wire Gradle Plugin is better. Nevertheless, the relative ease with which either can be configured leads to Fun Situations such as: two modules can each depend on the same proto files, possibly at different versions, leading to generated code with the same exact class name but different definitions. And now if you have a third module that depends on these two modules, you're in a situation where your module may fail to compile if you just so happen to change the order of your dependency declarations, or worse, it may compile in both cases but fail at runtime for a similar reason. This is because, as discussed above, a classpath is just a collection of jars and class files, and whichever class file gets loaded first wins forever.

I have now worked in two separate extremely large codebases that have significant usage of protos, and it is no exaggeration to say that dealing with them is almost the worst part of my job ("AI" has recently taken that crown).

Yolo compilers

It turns out that different compilers have different ideas of what the resultant class files should look like. Chapter 4 of the JVM specification discusses the Constant Pool. class files contain a table, known as the constant_pool, which contains a reference to every constant present in the source code of a Java file. This is useful for static analysis because the various JVM compilers all^[3] inline constants for runtime efficiency. This means that a constant like public static final String CONSTANT = "magic" gets turned into simply "magic" at the use-site, and similarly for Kotlin's const val. Therefore simply analyzing the bytecode directly with a tool like asm won't enable static analysis tools to connect the user of a constant to the maybe-separate module that provides the constant. Thanks to the constant pool, however, we can see the full reference to the provider and make the connection.

This only works for class files compiled with javac, however. For both kotlinc and ec4j (the Eclipse compiler for Java, yes this does exist and Real Teams in the world rely on it), keeping these full references to inlined constants in the constant pool is considered unnecessary.

The Dependency Analysis Gradle Plugin has a class, ConstantPoolParser, which parses the constant pool of a class file and extracts the set of class file references for all the class's inlined constants. When passed a reference to a class file compiled with something other than javac, the returned set is empty. This leads to "unused dependency" false positives, when a dependency is only used for the constants it contains, a surprisingly common situation.

As a consequence, DAGP utilizes some heuristics to try to workaround this situation—with imperfect results. I won't go into details here, but they involve parsing source code for import statements,^[4] looking at the ldc bytecode instruction,^[5] etc. Source parsing falls over in the presence of split packages, and the ldc bytecode only provides the constant value, not its name. Together, the heuristics get most of the way there, and it's unlikely the tool will get more accurate here without much more sophisticated source code analysis. Happy to work on that if you want to fund me!

Special thanks

Special thanks to Luis Cortés once again for the thorough review!

Who stalks us in the darkness?

The above list of ~~grievances~~ curses should not be taken as comprehensive. It is merely the list of things that have most recently destroyed my will to live.^[6]

…wait, who is that behind me, in the dark wood…?

By Ivan Bilibin - Self-scanned, Public Domain, Wikimedia

Baba Yaga (no not that one) This is what I think of when I imagine the Kotlin compiler given human form.

I'm eliding some complexity around the classloader hierarchy, and the possibility you may have a custom classloader that doesn't follow standard behavior. See A crash course in classpaths for more information. ↩︎
To be clear, this is not about Java/Kotlin interop, but the fact that each compiler (protoc and wire) simply emit different code from the same protobuf schema. ↩︎
I think they all do, but I haven't checked exhaustively. At least java, ec4j, and kotlinc do. ↩︎
See here for where DAGP parses source code in a very simplified way. ↩︎
This post is already too long for me to explain in depth what I mean here. You can see where DAGP visits the LDC instruction here. ↩︎
Kidding! Once again the thing that's killing my will to live is just "AI." ↩︎

Shrinking Elephants

2025-10-14T00:00:00Z

Photo by Lili Koslowski on Unsplash.

So you want Intellij IDEA to sync 5 million lines of Kotlin code,^[1] spread across more than 2000 Gradle projects.

No, you don't.

In one of our largest repos, a Kotlin backend project, syncing it all takes an average of about 8.4 minutes, and that's being generous. Achieving that required a fully up-to-date build, with all dependencies already downloaded—meaning this was 8.4 minutes for what is essentially a no-op. This no-op sync used the full 24 GiB of heap provided to the Gradle daemon, and an additional 12 GiB heap provided to the IDE process, or 36 GiB in total. If this is the best you can offer your developers, do not expect them to be very productive, and do not expect to ship software quickly.

A more realistic scenario is triggering the first sync of the day after a git pull that updated dependencies. In that "cold sync" scenario, the same build described above took 24.7 minutes, more than 7 minutes of which was spent downloading over 26000 files (jars plus metadata) totalling nearly 4 GiB over the wire. This build maxed out the heap very quickly, and it wouldn't be hard to imagine a scenario where the build thrashed gc for an hour before finally OOMing.

While you may want the full project loaded into memory, you almost certainly can't bear the consequences: extremely long sync, sluggish editing, the spinning beachball of death…

Fortunately, our developers never have to do this (sometimes a build engineer might… for analysis purposes). The actual experience we target for our developers is benchmarked at 15 seconds, a 97% improvement over the baseline outlined above; and only using 3.5 GiB heap, a 75% improvement. How did we do it?

What is IDE sync?

Gradle Sync is how the IDE loads important information about your build (known as the project model) to enable smart editing features like autocomplete, code navigation, viewing sources for dependencies, etc. To make this work, IntelliJ injects a library into the Gradle daemon containing the model builders that construct the IntelliJ project models. IntelliJ then requests that the Gradle daemon build the project models using the Tooling API. Gradle will configure your build, execute the model builders requested by IntellIJ, then serialize the data back to the IDE. At this point, the project sync is now completed and many smart editor features will be available. IDE indexing is a separate process that takes place after sync to enable the rest of the editor features, which this post won't cover.

The sync process is notoriously slow because of how much work needs to be done. The entire build must be configured (configure-on-demand is not applicable here), project dependencies and their sources need to be downloaded if they aren't cached locally, and this work is largely performed in serial, leaving your CPU mostly idle. If any error occurs in the process, or any build.gradle(.kts), settings.gradle(.kts) or version catalog file is updated, the sync process must be restarted from scratch.

Benchmarks vs telemetry

Telemetry (that is, real data from real users) is the gold standard for tracking build performance. This is what makes Develocity such an indispensable tool for tracking and understanding our Gradle builds (about 2 million every week). Even with such a firehose of data, however, there are gaps. For purposes of this post, the biggest gap is the wall-clock time for an IDE sync (as described above). IDE-specific overhead contributes anywhere from 10% to over 50% of total sync time, so if we're only tracking the Gradle portion of the sync, we're missing the full picture.

We solved this problem just a couple of months ago with a new combination IDE and Gradle plugin we call build-sync-metrics.^[2] With this, we now have complete telemetry from our developers on the performance of the IDE in their local environments.

We can, for example, state the following, thanks to our telemetry:

Over the course of Q3, the Backend Build Team was able to reduce mean sync time for our backend developers from over 4 minutes to about 90 seconds, saving 2.5 minutes per sync. With over 100k syncs per year, this translates to 2-3 eng-years saved every year.

While this is incredibly powerful, there are limitations. This population data isn't really amenable to rapid experimentation, or experimentation across a large combination of dimensions, and is—by defintion—coming from an uncontrolled environment with an uncountably large number of exogenous inputs. There are, to borrow a phrase, unknown unknowns.

This is where benchmarks really shine. Essentially, they enable us to test possible "interventions" (described below) before we inflict them as A/B tests on our developers. They give us confidence that something might work, which saves time and potential pain. Once we see value from an intervention via a benchmark, we can start rolling it out to our users and see the impacts in real-time. We use gradle-profiler for creating these benchmarks.

The rest of this post makes exclusive use of benchmarks to describe what we've done to make use of the IDE a more pleasant experience for our developers. This allows us to make precise statements about the impacts of these interventions. Please note that all of these benchmarks are also backed up by real-world telemetry, which is necessarily fuzzier in nature.

Interventions

The primary cost of syncing a Gradle project is configuring its project model. In fact, sync duration appears to be slightly quadratic over the number of subprojects (or modules) in the project, according to our benchmarks:

This chart was created using data from a real project, not something artificially generated.^[3] How this was done will be clear in a moment. As to why the data might be quadratic: that's hard to say. Discussions with Gradle engineers suggest it might have to do with the project structure, such as depth of the project graph, number of edges, etc. In other words, while this graph accurately reflects our project, it may not reflect others. It is almost certainly true that there is at least a linear relationship, however.

The chart above suggests that anything that can reduce subproject count is likely to improve sync performance. However, there are other design concerns that determine the number of subprojects:

Project architecture is a first-class consideration, and we strongly believe that the project structure should reflect its architecture.^[4]
Sync performance is only one component of developer productivity. We must also consider build performance, which includes both project configuration and build execution. Weighing configuration too heavily will result in extremely slow builds that don't benefit at all from incremental or avoided execution.

We refer to the following practices as "interventions" because they're out-of-band of the normal build process. They are not first-class features of the build system, though they do use public APIs where relevant. They also require the ability to know something a priori about the software-under-development, and this prerequisite is neatly encapsulated by the requirement that the build be fully conventionalized prior to making any further intervention. Conventionalization is out-of-scope of this post, so please read the prior two posts in this series, Herding Elephants and Stampeding Elephants, for more information.

These interventions are listed in order of ease of implementation, more or less:

In IDEA, enable the experimental Parallel Model Fetching feature.^[5]
Use a tool such as Spotlight to trim the project dependency graph to a
subset of the full project set.
Use a tool such as Fastsync to reduce the amount of work the Gradle
dependency resolution engine has to do (by mangling the runtime classpath) during IDE sync.
Take a hammer to your project graph and replace as many project dependencies with external module dependencies as
possible. We call this "artifact-swap."

And finally, included here because of its impact, which is challenging to measure via benchmarks:

Pre-fetch dependencies.

All together, these changes can reduce your IDE sync time by 97%, according to our benchmarks. Cumulative impact of interventions.

There is a cost

While we firmly believe that these interventions' benefits outweigh their costs, we can't ignore those costs. There is obviously the implementation and maintenance cost—some of these improvements will require a dedicated build team (which you should have if your repos are large enough to benefit from these sorts of interventions). There's also a DX cost, since users to have to learn new workflows. These downsides are real, but not prohibitive. Let's continue!

Baseline

In order to contextualize the interventions, we must first describe the baseline scenario, which is quite simply a Kotlin JVM project with over 2000 Gradle subprojects. Syncing this scenario with Intellij IDEA 2025.2.3 and Gradle 9.1.0, absent any of the "interventions" described below, took an average of 8.4 min, or about 220 ms / project.

As noted in the introduction, all benchmarks are necessarily "warm" or "no-op" in nature. We run the sync three times as a warm-up, and then another eight times, averaging the result. Therefore these syncs should involve, among other simplifications, minimal network overhead (they might still make HEAD requests to validate the up-to-date-ness of dependencies).

The metrics provided below are the result of layering on each intervention cumulatively. We also have independent benchmarks validating that each intervention is successful on its own.

Parallel model fetching

Compared to the baseline scenario, enabling parallel model fetching reduced sync duration by 57%, to about 3.6 min, or about 95 ms / project.

This "intervention" barely qualifies as one, but must be mentioned due to the high impact for near-zero cost. It is an "incubating" feature in Intellij IDEA, and is disabled by default. To enable it, navigate to Settings > Build, Execution, Deployment > Gradle and check the "Enable parallel Gradle model fetching" box, as indicated in the screencap below. Note that this checkbox only affects fetching of IntelliJ models, and Android model builders are already fully parallelized.

Disclaimer: enabling this can result in random failures, but this is extremely rare and, in our opinion, worth it for the impressive performance benefit conferred, especially since the failures are trivially resolved by another sync. The failures seem to be a result of race conditions due to configuring the Gradle project model in parallel, which it simply isn't designed for (yet). One issue you can track related to this is IDEA-355309.

Spotlight

Compared to the baseline scenario, adding Spotlight to our build reduced sync duration by 82%, to about 1.5 min, or about 120 ms / project. (This is without parallel model fetching, to be clear.) Combined with parallel model fetching, the full reduction was 90%, or about 48% better than parallel model fetching alone.

Sometimes referred to as "focus", this intervention is required if you want to manage very large Gradle builds (meaning it has a few hundred projects). This intervention works by dramatically reducing the number of active projects Gradle must configure. Spotlight scans the project buildscript files to parse the project dependency tree without configuring any projects, then only includes any projects from the build that are in the dependency tree of the projects the user requested to load in the IDE. For the purposes of this post, we cut the number of projects from more than 2000 to about 750. This cutoff captures 95% of all syncs we observe from our developers. That is, only 5% of all syncs target a project set larger than this. The figures above make use of a Spotlight-like tool to "target" specific subsets of our larger monorepo.

If you're able to abide by a few constraints, as clearly outlined in the Spotlight documentation, you can benefit from this intervention today. We apply some automated enforcement of these rules to prevent issues from popping up.

Intransitive sync

Compared to the baseline scenario, layering on intransitive-sync reduced sync duration by 94%, to about 28 sec, or about 40 ms / project.

The original inspiration for this idea came from the MDX Android team, which manages the Square-Android Point-of-Sale repo (extremely large at almost 7000 projects). In discussions between this team and the Backend Build Team (which manages the aforementioned very large Kotlin JVM project, among other things), we realized that IDE-based development and code completion only needed access to the compile classpath—not the runtime classpath. We therefore disabled transitive dependency resolution for all runtime classpaths (runtimeClasspath, testRuntimeClasspath, and so on). We rolled this out to users over a period of about one month while ironing out minor kinks in the implementation, before finally enabling it universally. Note that this feature may break debugging and UI previews in Android Studio.

Artifact swap

Compared to the baseline scenario, layering on artifact-swap reduced sync duration by 97%, to about 15 sec, or about 84 ms / project. Note that the cost-per-project has gone up a bit, reflecting the fact that the sync-time vs project-count model is dominated by some constant overhead at low project count.

Spotlight still requires us to load all projects in the dependency tree of our target projects into the IDE. We go even further and swap out any projects that were not requested by a user with a precompiled jar artifact (or aar for android projects). Doing this lets us minimize the Gradle project list to only what the user has requested to be loaded in the IDE. Inspiration for this came from a Droidcon talk.

If a Gradle project is referenced in a build file, it must also be include-ed in settings and configured by Gradle, which is what we are trying to avoid. We start by ensuring that all project() references in build files are rewritten to maven coordinate of a precompiled artifact. In Groovy buildscripts we do this by using Groovy metaprogramming with a project plugin to override the project() function to do what we want instead of the default behavior. This project plugin is applied to every requested project in the build by a settings plugin. Projects using Kotlin buildscripts can actually inject a DSL override more directly using this One Weird Trick.

This process actually goes too far and may end up rewriting references to projects a user is working on to a maven coordinate, but we fix this in a second stage by applying dependency substitution to rewrite those references back to a project reference. This also makes any artifact that references a project artifact that a user currently has loaded be rewritten back to the active Gradle project.

This brief overview of artifact swap elides a lot of additional infrastructure used to publish project artifacts, prefetch those artifacts for improved performance (see below), IDE plugin support for navigating the swapped project, and tooling to reliably configure all of this. Most of our implementation for these features is available in the artifact-swap repository.

Pre-fetch dependencies

This intervention is of a somewhat different nature from the others discussed above. It helps not only with syncs but with all builds, by downloading dependencies before the user requests them. Our data suggest that the middle quartile case (p25-75) dropped from nearly 2 minutes to about 20 seconds. That is, Gradle spends about 83% less time downloading dependencies, during IDE sync, post-intervention, according to Develocity.

We can't make statements on this based on benchmarks since those are, by definition and as noted above, fully warm—dependencies have already been fetched. The Develocity telemetry, however, gives us high confidence that this improves the local development experience.

Our pre-fetching works in two ways: via a git post-checkout hook, and also on a periodic schedule. In both cases, the background process gets the list of dependencies from a Gradle task registered by the Dependency Analysis Gradle Plugin:

./gradlew :computeAllDependencies

This generates a file at build/reports/dependency-analysis/allLibs.versions.toml. This version catalog is then used by Gradle in a special background process that is installed on all developer computers by Block's MDM solution and Salt. This background process uses Gradle to resolve all dependencies (including transitive dependencies) needed by the build.

Future interventions

We've already achieved fairly amazing results, but as we know, the build engineer's work is never done. What else can we do? Three things come immediately to mind, and are part of our plans for 2026: we want to clean up our dependency graph, we want to cleanup our project structure, and we want to make our build fully compliant with Gradle's Isolated Projects feature.

We think that cleaning up our dependency graph (with the help of the Dependency Analysis Gradle Plugin, aka DAGP) would have several important benefits, not least of which is enhanced maintainability and easier debugging of build failures. But we also think it could have a positive impact on performance: eliminating unused external dependencies means less network overhead, and eliminating edges between local projects would enhance the effectiveness of interventions like Spotlight and Artifact Swap (discussed above).

DAGP is also able to identify projects applying the Android Gradle Plugin (AGP) that do not need to do so. AGP is a particularly “heavy” plugin, and reducing its use in the build has yielded performance wins for us previously. We also plan to refine our module structure by combining certain module types to reduce the overall project count in our build. Using Gradle's test fixtures feature is another way that modules can be combined to reduce the overall count.

Isolated Projects, currently a pre-alpha feature of Gradle, promises to be a game-changer. It would enable fine-grained and incremental caching of the configuration phase of the build, which would make dependency updates (such as might affect only a single Gradle subproject) significantly less painful. We have done quite a bit of work already to be early adopters of this feature, including submitting patches to Android Studio, and Square's Android build is already compliant with the isolated projects contract for sync. So far Isolated Projects has not delivered us any performance benefits, but Gradle is currently focused on validating the correctness of the implementation before focusing on performance in later 9.x releases.

Tony Robalik is a build engineer on Block's Backend Build team, and a Gradle Fellow. Josh Friend is a build engineer on Block's Android Developer Experience team and helps maintain what might be the largest Gradle build in the world. Thanks to Tim Mellor, Yissachar Radcliffe, and Gábor Pap for reviewing the draft.

Although to be clear, there is no direct relationship between IDE sync and lines of code. This figure is provided only as a reference point to help indicate scale. ↩︎
Please note that the IDE plugin will not work out of the box for non-Block projects, but the code is straightforward and we hope it provides inspiration to other users. Contributions are also welcome. ↩︎
Via https://cdsap.github.io/ProjectGenerator/, for example. ↩︎
Although in practice it is probably more likely to reflect the org chart—for potentially very good reasons. ↩︎
Note that, for Android Studio users, this is the default behavior. ↩︎

This is why we can't have nice things: When POM files lie

2025-02-07T00:00:00Z

Photo by Marco Bicca on Unsplash. I really hope there's a light at the end of this sewer.

Sorry, my country is dissolving into a Nazi sewer of world-historical proportions and so I'm dealing with my otherwise fruitless rage by yelling about JVM things.

The fact that Java classes exist in a global namespace is not well-appreciated, even by vendors of major parts of the ecosystem (apparently).

Caused by: java.lang.ClassCastException: class com.google.common.graph.ImmutableGraph cannot be cast to class com.google.common.graph.SuccessorsFunction (com.google.common.graph.ImmutableGraph and com.google.common.graph.SuccessorsFunction are in unnamed module of loader org.gradle.internal.classloader.VisitableURLClassLoader$InstrumentingVisitableURLClassLoader @4617120)

A confusing error at the best of times, but to a build engineer (i.e. me) trying to help out during a SEV, a despair-inducing example of how Dependency Management on the JVM is Completely Broken, What Are We Even Doing Here.

A ClassCastException always means Someone Somewhere Hates You, or at least values their own KPIs more than not polluting the entire goddamn ecosystem.

Let's debug! My first step is to navigate to the Graph class and check its hierarchy. I can confirm that, yes, in Guava 33.3.1-jre at least, that class does indeed implement the SuccessorsFunction interface. It should not be throwing a ClassCastException!

Next step. In Intellij (the vendor which is ironically the source of this amongst most of the rest of my woes), I set a breakpoint at the exception and evaluate the following expression.¹

// `this` is an instance of the `com.google.common.graph.Graph` class
this.javaClass.superclass.superclass.superclass.interfaces.first().protectionDomain.codeSource.location

That expression resolves to this:

file:/Users/<<ME!>>/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-compiler/2.0.21/88f09afc2536e38d528e78eb8349504de10ac436/kotlin-compiler-2.0.21.jar

What the fuck! That is not the right jar!! Let's double-check:

jar tf path/to/kotlin-compiler-2.0.21.jar
…
com/google/common/graph/Graph.class

😭

Maybe it's a bug! Let's look at the latest version of the jar at time of writing, 2.2.10 (nope, same problem). Not only has Jetbrains, the inventor of the Kotlin language, released a library that is bundling Guava (among many other things!!) in a fatjar, they're not even bothering to relocate the damn packages. Ok, I'm curious, what does the pom.xml have to say for itself?

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <modelVersion>4.0.0</modelVersion>
  <groupId>org.jetbrains.kotlin</groupId>
  <artifactId>kotlin-compiler</artifactId>
  <version>2.1.10</version>
  <name>Kotlin Compiler</name>
  <description>Kotlin Compiler</description>
  <url>https://kotlinlang.org/</url>
  <licenses>
    <license>
      <name>The Apache License, Version 2.0</name>
      <url>http://www.apache.org/licenses/LICENSE-2.0.txt</url>
    </license>
  </licenses>
  <developers>
    <developer>
      <name>Kotlin Team</name>
      <organization>JetBrains</organization>
      <organizationUrl>https://www.jetbrains.com</organizationUrl>
    </developer>
  </developers>
  <scm>
    <connection>scm:git:https://github.com/JetBrains/kotlin.git</connection>
    <developerConnection>scm:git:https://github.com/JetBrains/kotlin.git</developerConnection>
    <url>https://github.com/JetBrains/kotlin</url>
  </scm>
  <dependencies>
    <dependency>
      <groupId>org.jetbrains.kotlin</groupId>
      <artifactId>kotlin-stdlib-jdk8</artifactId>
      <version>2.1.10</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>org.jetbrains.kotlin</groupId>
      <artifactId>kotlin-script-runtime</artifactId>
      <version>2.1.10</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>org.jetbrains.kotlin</groupId>
      <artifactId>kotlin-reflect</artifactId>
      <version>1.6.10</version>
      <scope>compile</scope>
      <exclusions>
        <exclusion>
          <groupId>*</groupId>
          <artifactId>*</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.jetbrains.intellij.deps</groupId>
      <artifactId>trove4j</artifactId>
      <version>1.0.20200330</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>org.jetbrains.kotlinx</groupId>
      <artifactId>kotlinx-coroutines-core-jvm</artifactId>
      <version>1.6.4</version>
      <scope>compile</scope>
    </dependency>
  </dependencies>
</project>

There is no indication that this is a fatjar in the metadata, not to mention the hilarious fact that they are declaring a dependency on kotlin-reflect 1.6.10. But someone at some point noticed that this was Causing A Problem, so they excluded all of its dependencies. Are you kidding me. Guys, you publish a BOM, just use it!

It is entirely unclear why some dependencies are being declared and others are being silently bundled into the final artifact. Maybe I can get an answer at this issue I raised.

Why am I even using this dependency?

Well, I had been using kotlin-compiler-embeddable for source-code analysis,² which at least relocates its bundled classes. However, using that in the same project that has a dependency on the Kotlin Gradle Plugin leads to this build warning:

w: The artifact `org.jetbrains.kotlin:kotlin-compiler-embeddable` is present in the build classpath along Kotlin Gradle plugin.
This may lead to unpredictable and inconsistent behavior.
For more details, see: https://kotl.in/gradle/internal-compiler-symbols

A search online for a suitable replacement for kotlin-compiler-embeddable turned up this on discuss.kotlinlang.org:

kotlin-compiler-embeddable should be used in scenarios when it’s necessary to have the compiler packaged as a single jar with no external dependencies. In all other cases, use the regular kotlin-compiler.

That answer implies that kotlin-compiler does have external dependencies (which is true), while also further implying that it abides by the standard JVM library contract by Actually Declaring all its dependencies (which is false).³

Coping

This raises the question, how can we cope with this? Well, assuming your use-case for this is in a Gradle task and your task is relatively well-written, you can migrate the task action to use the Worker API with classloader isolation. The link provided in the warning does a fair job of explaining how to do that, assuming you're ok with drawing the owl.

Ah, fuck it, let's draw the owl

Drawing the build script

// plugin/build.gradle.kts
…etc…

dependencies {
  implementation(project(":shared-lib")) {
    exclude(group = "org.jetbrains.kotlin", module = "kotlin-compiler")
  }
}

Jetbrains' docs suggest using compileOnly("org.jetbrains.kotlin:kotlin-compiler-embeddable:2.1.10"), which isn't wrong, but also won't work if that lib is actually coming from a transitive dependency.

Drawing the task

abstract class OwlTask @Inject constructor(
  private val workerExecutor: WorkerExecutor
) : DefaultTask() {

  @get:Classpath
  abstract val kotlinCompiler: ConfigurableFileCollection

  @get:Input
  abstract val otherInput: Property<Boolean>

  @TaskAction fun action() {
    workerExecutor
      .classLoaderIsolation { spec ->
        // please note that from() adds to the
        // classpath, while setFrom() would
        // completely override it.
        spec.classpath.from(kotlinCompiler)
      }
      .submit(Action::class.java) { params ->
        params.otherInput.set(otherInput)
      }
  }

  interface Parameters : WorkParameters {
    val otherInput: Property<Boolean>
  }

  abstract class Action : WorkAction<Parameters> {
    override fun execute() {
      val otherInput = parameters.otherInput.get()
      …
    }
  }
}

Drawing the plugin

abstract class OwlPlugin : Plugin<Project> {
  override fun apply(target: Project) {
    // Use the project's version of Kotlin, if present. Else default to what we use.
    val kotlinVersion = …some method for getting the version of Kotlin you want…
    val dependencyScope = project.configurations.dependencyScope("owl").get()
    val resolvable = project.configurations.resolvable("owlClasspath") {
      extendsFrom(dependencyScope)
    }

    project.dependencies.add(dependencyScope.name, "org.jetbrains.kotlin:kotlin-compiler:$kotlinVersion")

    project.tasks.register("owl", OwlTask::class.java) { t ->
      t.kotlinCompiler.setFrom(resolvable)
      t.otherInput(true) // demonstration purposes only
    }
  }
}

Hello, owl!

I can confirm this works, it eliminated the ClassCastException. Now, a day and incredible amounts of frustration later, I guess I can go back to my original problem?

I'm just some asshole

Look, I'm just some asshole who always tries to do the right thing. But what the fuck am I supposed to do when larger and better-resourced teams just abdicate responsibility? Fuck! I guess I'll write a blog post!

Special thanks

I am much obliged Luis Cortes, who convinced me that some of my salty language was in fact acidic. All the remaining salt and/or acid is entirely of my own devising.

Endnotes

¹ Guava's Graph class has a complicated hierarchy! up

² This dependency exhibits many of the same metadata problems. up

³ To be fair, this is a very widespread problem. up

Gradle extensions part 2: Now with shenanigans

2024-12-11T00:00:00Z

Photo by Karsten Würth on Unsplash

Welcome to the spiritual successor to Gradle plugins and extensions: A primer for the bemused (one of my most popular posts, such that it competes for space with actual Gradle documentation at the top of a Google search).

As part of my long-running quest to Destroy buildSrc With Fire, I have recently had occasion to learn how to add extensions to other kinds of types, such as tasks. We have code like this duplicated across many many repos that are under our care:

// buildSrc/src/main/kotlin/magic/Magic.kt
package magic

object Magic {
  fun shouldPracticeTheDarkArts(): Boolean {
    return System.getenv("DO_ANCIENT_MAGIC")?.toBoolean()
      ?: System.getenv("DO_SLIGHTLY_MORE_MODERN_MAGIC")?.toBoolean()
      ?: false
  }
}

This code is used in build scripts like this:

// build.gradle.kts
import magic.Magic

tasks.withType<Test>().configureEach {
  if (Magic.shouldPracticeTheDarkArts()) {
    logger.quiet("👻")
  }
}

There are several things about this that I'd like to improve:

I don't want this code in buildSrc. I want a version of it in our build-logic that is under test and which is shared widely (instead of duplicated in a dozen different repos).
I don't like the import. It is Unclean. (Build scripts should be simple, declarative, easy for tools to parse.)
I'm not a big fan of calling System.getenv() in a Gradle context. I prefer to use the ProviderFactory.

Extending `Test` tasks

Many Gradle types, including all Tasks (and of course the Project type), are ExtensionAware. This means they all have an ExtensionContainer available on which new extensions can be created and added.

// build-logic/src/main/kotlin/magic/TestMagicExtension.kt
package magic

abstract class TestMagicExtension @Inject constructor(
  private val providers: ProviderFactory
) {

  internal companion object {
    const val NAME = "magic"

    fun create(
      testTask: Test,
      providers: ProviderFactory,
    ) {
      testTask.extensions.create(
        NAME,
        TestMagicExtension::class.java,
        providers,
      )
    }
  }

  fun shouldPracticeTheDarkArts(): Boolean {
    return providers
      .environmentVariable("DO_ANCIENT_MAGIC")
      .orElse(providers.environmentVariable("DO_SLIGHTLY_MORE_MODERN_MAGIC"))
      .map { it.isNotEmpty() }
      .getOrElse(false)
  }

And in our plugin, we can add this to all our Test tasks:

project.tasks.withType<Test>().configureEach { t ->
  TestMagicExtension.create(t, project.providers)
}

And now we can update our build scripts:

// build.gradle.kts
import magic.TestMagicExtension

tasks.withType<Test>().configureEach {
  // the "extensions" call is on the Test instance,
  // not the project instance
  val magic = extensions.getByType(TestMagicExtension::class.java)
  if (magic.shouldPracticeTheDarkArts()) {
    logger.quiet("👻")
  }
}

...that's not better at all!

Groovy: An interlude

First of all, let's take a step back and remind ourselves that "we love Kotlin, type safety is great, I don't care that performance is worse..." We can say that over and over again a few times while rocking in a fetal position on the floor till we feel better. Now, here's the same build script but in Groovy:

// build.gradle
tasks.withType(Test).configureEach {
  if (magic.shouldPracticeTheDarkArts()) {
    // I'm being cheeky by also omitting the
    // "redundant" parentheses
    logger.quiet "👻"
  }
}

Groovy isn't supposed to be better! Damnit!

Sprinkle on some shenanigans

How the heck does Gradle Kotlin DSL do it? Why isn't it generating "typesafe accessors" for my test task extension? Well, that second one is a good question and I have no answer. But for the first... let's just "generate" (that is, write) our own typesafe accessors!

We add some code in a new (to us) package:

package org.gradle.kotlin.dsl

import magic.TestMagicExtension

public val Test.magic: TestMagicExtension
  get() = extensions.getByType(TestMagicExtension::class.java)

public fun Test.magic(configure: TestMagicExtension.() -> Unit) {
  configure(TestMagicExtension.NAME, configure)
}

And now we can update our Kotlin DSL build script:

// build.gradle.kts
tasks.withType<Test>().configureEach {
  if (magic.shouldPracticeTheDarkArts()) {
    logger.quiet("👻")
  }
}

Here we're (ab)using the fact that Gradle automatically imports everything in the org.gradle.kotlin.dsl package into build scripts, so all those functions are Just There (in a global namespace, so be careful!).

This is a common enough pattern that Gradle itself uses it in its test-retry-gradle-plugin. There's also an open issue (from, er, 2018) on Gradle's issue tracker with a feature request to permit custom plugins to add their own default imports without resorting to using split packages like this.

Now go forth and be merry, for it is that time of year.

One click dependencies fix

2024-10-08T00:00:00Z

Photo by Maxim Hopman on Unsplash

If you maintain a JVM ¹ or Android project, chances are you've heard of the Dependency Analysis Gradle Plugin (DAGP). With over 1800 stars, it's used by some of largest Gradle projects in the world, as well as by Gradle itself. It fills what would otherwise be a substantial hole in the Gradle ecosystem: without it, I know of no other way to eliminate unused dependencies and to correctly declare all your actually-used dependencies. In other words, when you use this plugin, your dependency declarations are exactly what you need to build your project: nothing more, nothing less.

That might sound like a small thing, but for industrial-scale projects, a healthy dependency graph is a superpower that prevents bugs, eases debugging (at build and runtime), keeps builds faster, and keeps artifacts smaller. If developer productivity work is the public health of the software engineering world, then a healthy dependency graph is a working sewer system. You don't know how much you rely on it till it stops working and you've got shit everywhere.

The problem is that, if your tool only tells you all the problems you have but doesn't also fix them, you might have a massive(ly annoying) problem on your hands. I mentioned this as an important consideration in my recent rant against code style formatters. This is why, since v1.11.0, DAGP has had a fixDependencies task, which takes the problem report and rewrites build scripts in-place. Even before that, in v0.46.0, the plugin had first-class support for registering a "post-processing task" to enable advanced users to consume the "build health" report in any manner of their choosing. Foundry (née The Slack Gradle Plugin), for example, has a feature called the "dependency rake", which predates and inspired fixDependencies.

fixDependencies hasn't always worked well, though. For one thing, there might be a bug in the analysis such that, if you "fix" all the issues, your build might break. (DAGP is under very active development, so if this ever happens to you, please file an issue!) In this case, it can take an expert to understand what broke and how to fix it, or you can fall back to manual changes and iteration.

For another thing, the build script rewriter has relied on a simplified grammar for parsing and rewriting Gradle Groovy and Kotlin DSL build scripts. That grammar can fail if your scripts are complex.² This problem will soon be solved with the introduction of a Gradle Kotlin DSL parser built on the KotlinEditor grammar, which has full support for the Kotlin language. (Gradle Groovy DSL scripts will continue to use the old simplified grammar, for now.)

There have also been many recent bugfixes to (1) improve the correctness of the analysis and (2) make the rewriting process more robust in the face of various common idioms. DAGP now has much better support for version catalog accessors, for example (no support yet for experimental project accessors).

With these improvements (real and planned), it's become feasible to imagine automating large-scale dependency fixes across hundreds of repos containing millions of lines of code and have it all just work. Here's the situation:

Over 500 repositories.
Each with its own version catalog.
Most of the entries in the version catalogs use the same names, but there's some incidental skew in the namespace (multiple keys pointing to the same dependency coordinates).
Over 2000 Gradle modules.
Close to 15 million lines of Kotlin and Java code spread out over more than 100 thousand files, along with over 150 thousand lines of "Gradle" code in more than 3 thousand build scripts. This last point isn't as relevant as the first four, but helps to demonstrate what I mean when I say "industrial scale."³

Additionally, the build code we want to write to manage all this should follow Gradle best practices: it should be cacheable to the extent possible, should work with the configuration cache, and for bonus points should not violate the isolated projects contract either (which is also good for maximal performance). The ultimate goal is for developers and build maintainers to be able to run a single task and have it (1) fix all dependency declarations, which might mean adding new declarations to build scripts; (2) all build script declarations should have a version catalog entry wherever possible; (3) and all version catalog entries should come from the same global namespace so that the entire set of 500+ repositories are fully consistent with each other. This last part is an important requirement because we're migrating these repos into a single mono/mega repo for other reasons.

Here's the task they can now run, for the record:

gradle :fixAllDependencies

(nb: we use gradle and not ./gradlew because we manage gradle per-repo with hermit.)

So, how do we do it?

Pre-processing

The first step was creating the global version catalog namespace. We did not attempt to actually create a single published global version catalog because, until we finish our megarepo migration, an important contract is that each repo maintains its own dependencies (and their versions). So instead, we collected the full map of version catalog names to dependency identifiers (the dependency coordinates less the version string). We eliminated all the duplication using pre-existing large-scale change tools we have, and then populated the final global set (now with 1:1 mappings) into our convention plugin that is already applied everywhere.

Conceptual framework

The Gradle framework, in general, takes the Project as the most important point of reference.⁴ A Project instance is what backs all your build.gradle[.kts] scripts, for example, and most plugins implement the Plugin<Project> interface. Safe, performant, high-quality build code respects this conceptual boundary and treats each project (AKA "module") as an atomic unit.

If Tasks have well-defined inputs and outputs (literally annotated @Input<X> and @Output<X>), then it might help to also think of projects as having inputs and outputs. In general, a project's inputs are its source code (which by convention is in the src/ directory at the project root), and its dependencies. A project's outputs are the artifacts it produces. For Java projects, the primary artifacts are jars (for external consumption), or class files (for consumption by other projects in a multi-project build).⁵

With that in mind, we can decide that if two projects need to talk to each other, they should do so via their well-defined inputs and outputs. We define relationships between projects via dependencies (A -> B means A depends on B, so B is an input to A), and we can flavor that connection such that we tell Gradle which of B's outputs A cares about. The default is the primary artifact (usually class files for classpath purposes), but it can also be anything (that can be written to disk). It can, for example, be some metadata about B. It can also be both! (You can declare multiple dependencies between the same two projects, with each edge having a different "flavor," that is, representing a different variant.) This may make more sense in a bit when we get to a concrete example.

Implementation: `:fixAllDependencies`

The rest of this post will focus on implementation, but at a relatively high level of detail. Some of the code will essentially be pseudocode. My goal is to demonstrate the full flow at a conceptual level, such that a (highly) motivated reader could implement something similar in their own workflow or, more likely, simply learn about how to do something Cool:tm: with Gradle.

Here's a sketch of the simplified task graph with Excalidraw:

Note how each project is independent of the other. Well-defined Gradle builds maximize concurrency by respecting project boundaries.

Step 1: The global namespace

As mentioned in the pre-processing section, we need a global namespace. We want all dependency declarations to refer to version catalog entries, i.e., libs.amazingMagic, rather than "com.amazing:magic:1.0". Since DAGP already supports version catalog references in its analysis, this will Just Work if your version catalog already has an entry for amazingMagic = "com.amazing:magic:1.0". However, if you don't, DAGP defaults to the "raw string" declaration. If we want, we can tell DAGP about other mappings that it can't detect by default:

// root build script
dependencyAnalysis {
  structure {
    map.putAll(
      "com.amazing:magic" to "libs.amazingMagic",
      // more entries
    )
  }
}

where dependencyAnalysis.structure.map is a MapProperty<String, String>, which you can modify directly in your build scripts or via a plugin. Note the "raw string" version of the declaration doesn't include version information; this is important because the version you declare may not match the version that Gradle resolves.

Step 2: Update the version catalog, part 1

With Step 1, DAGP will rewrite build scripts via the built-in fixDependencies task to match your desired schema, but your next build will fail because you'll have dependencies referencing things like libs.amazingMagic which aren't actually present in your version catalog. So now we have to update the version catalog to ensure it has all of these new entries. This will be a multi-step process.

First, we have to calculate the possibly-missing entries. We write a new task, ComputeNewVersionCatalogEntriesTask, and have it extend AbstractPostProcessingTask, which comes from DAGP itself. This exposes a function, projectAdvice(), which gives subclasses access to the "project advice" that DAGP emits to the console, but in a form amenable to computer processing. We'll take that output, filter it for "add advice", and then write those values out to disk via our task's output. We only care about the add advice because that's the only type that might represent a dependency not in a version catalog.

// in a custom task action
val newEntries = projectAdvice()
  .dependencyAdvice
  .filter { it.isAnyAdd() }
  .filter { it.coordinates is ModuleCoordinates }
  .map { it.coordinates.gav() }
  .toSortedSet()

outputFile.writeText(newEntries.joinToString(separator = "\n")

Note that with Gradle task outputs, it's best practice to always sort outputs for stability and to enable use of the remote build cache.

Next we tell DAGP about this post-processing task (which is how it can access projectAdvice():

// subproject's build script
computeNewVersionCatalogEntries = tasks.register(...)

dependencyAnalysis {
  registerPostProcessingTask(computeNewVersionCatalogEntries)
}

And finally we also have to register our new task's output as an artifact of this project!

val publisher = interProjectPublisher(
  project,
  MyArtifacts.Kind.VERSION_CATALOG_ENTRIES
)
publisher.publish(
  computeNewVersionCatalogEntries.flatMap {
    it.newVersionCatalogEntries
  }
)

where the interProjectPublisher and related code is heavily inspired by DAGP's artifacts package, because I wrote both. The tl;dr is that this is what teaches Gradle about a project's secondary artifacts. I wish Gradle had a first-class API for this, alas.

Step 3: Update the version catalog, part 2

Back in the root project, we need to declare our dependencies to each subproject, flavoring that declaration to say we want the VERSION_CATALOG_ENTRIES artifact:

// root project
val resolver = interProjectResolver(
  project,
  MyArtifacts.Kind.VERSION_CATALOG_ENTRIES
)

// Yes, this CAN BE OK, but you must only access
// IMMUTABLE PROPERTIES of each project p.
// This sets up the dependencies from the root to
// each "real" subproject, where "real" filters
// out intermediate directories that don't have
// any code
allprojects.forEach { p ->
  // implementation left to reader
  if (isRealProject(p)) {
    dependencies.add(
      resolver.declarable.name,
      // p.path is an immutable property, so we're
      // good
      dependencies.project(mapOf("path" to p.path))
    )
  }
}

val fixVersionCatalog = tasks.register(
  "fixVersionCatalog",
  UpdateVersionCatalogTask::class.java
) { t ->
    t.newEntries.setFrom(resolver.internal)
    t.globalNamespace.putAll(...)
    t.versionCatalog.set(layout.projectDirectory.file("gradle/libs.versions.toml"))
  }

The root project is the correct place to register this task, because the version catalog will typically live in the root at gradle/libs.versions.toml.

With this setup, a user could now run gradle :fixVersionCatalog, and it would essentially run <every module>*:projectHealth, followed by <every module>*:computeNewVersionCatalogEntries, followed finally by :fixVersionCatalog, because those are the necessary steps as we've declared and wired them.

This updates the version catalog to contain every necessary reference to resolve all the potential libs.<foo> dependency declaration throughout the build.

Step 4: Fix all the dependency declarations

This step leverages DAGP's fixDependencies task, and is really just about wrapping everything up in a neat package.

We want a single task registered on the root. Let's call it :fixAllDependencies. This will be a lifecycle task, and invoking it will trigger :fixVersionCatalog as well as all the <every module>*:fixDependencies tasks.

// root project
val fixDependencies = mutableListOf<String>()

allprojects.forEach { p ->
  if (isRealProject(p)) {
    // ...as before...

    // do not use something like `p.tasks.findByName()`,
    // that violates Isolated Projects as well as
    // lazy task configuration.
    fixDependencies.add("${p.path}:fixDependencies")
  }
}

tasks.register("fixAllDependencies") { t ->
  t.dependsOn(fixVersionCatalog)
  t.dependsOn(fixDependencies)
}

And we're done.⁶

(Optional) Step 5: Sort dependency blocks

If you do all the preceding, you should have a successful build with a minimal dependency graph. :tada: But your dependency blocks will be horribly out-of-order, which can make them hard to visually scan. DAGP makes no effort to keep the declarations sorted because that is an orthogonal concern and different teams might have different ordering preferences. This is why I've also authored and published the Gradle Dependencies Sorter CLI and plugin, which applies what I consider to be a reasonable default. If you apply this to your builds (which we do to all of our builds via our convention plugins), you can follow-up :fixAllDependencies with

gradle sortDependencies

and this will usually Just Work. This plugin is in fact already using the enhanced Kotlin grammar from KotlinEditor, so Gradle Kotlin DSL build scripts shouldn't pose a problem for it.

And now we're really done.

Endnotes

¹ Currently supported languages: Groovy, Java, Kotlin, and Scala. up

² This is one reason why I think it's important to keep scripts simple and declarative. up

³ Measured with the cloc tool. up

⁴ Gradle's biggest footgun, in my opinion, is that the API doesn't enforce this conceptual boundary. up

⁵ This paragraph is an oversimplification for discussion purposes. up

⁶ Well, except for automated testing and blog-post writing. up

Tony Robalik

Do LLMs suck at coding, actually? An open source case study

I'm not talking to you

Case study 1: the misdiagnosis of a ClassCastException

Case study 2: The drive-by PR

Case study 3: Another drive-by PR

Fractal harm engines

Harm reduction

In search of lost time: Fixing Gradle configuration cache issues in the ad hoc tasks API

Business geniuses and bus factors

Reflections on Camus's 'The Rebel': Sade and the Trump regime

Enshittification: Why I’m leaving dev.to

Is the Java ecosystem cursed? A dependency analysis perspective

We are cursed with…

Lying metadata

It's just a list, man

Fat jars without package relocation

Split packages

Funhouse mirrors (aka reflection)

Protocol Buffers

Yolo compilers

Special thanks

Who stalks us in the darkness?

Shrinking Elephants

What is IDE sync?

Benchmarks vs telemetry

Interventions

There is a cost

Baseline

Parallel model fetching

Spotlight

Intransitive sync

Artifact swap

Pre-fetch dependencies

Future interventions

This is why we can't have nice things: When POM files lie

Why am I even using this dependency?

Coping

Ah, fuck it, let's draw the owl

Drawing the build script

Drawing the task

Drawing the plugin

Hello, owl!

I'm just some asshole

Special thanks

Endnotes

Gradle extensions part 2: Now with shenanigans

Extending Test tasks

Groovy: An interlude

Sprinkle on some shenanigans

One click dependencies fix

Pre-processing

Conceptual framework

Implementation: :fixAllDependencies

Step 1: The global namespace

Step 2: Update the version catalog, part 1

Step 3: Update the version catalog, part 2

Step 4: Fix all the dependency declarations

(Optional) Step 5: Sort dependency blocks

Endnotes

Case study 1: the misdiagnosis of a `ClassCastException`

Extending `Test` tasks

Implementation: `:fixAllDependencies`