Frédéric van der Essen

Unsupervised Codebase Refactoring: How To, What Works and What Doesn't

2026-02-08T00:00:00Z

Could code refactoring become a one-click operation, like running a formatter but
for architecture ? Split the bloated files, untangle the responsibilities, deduplicate
the logic, and come back to a clean codebase ?

So I ran an experiment. I pointed Opus 4.6 at a 60K line Go microservice I had
vibe coded over the past few months, gave it some refactoring principles, and let it
run unsupervised. Two hours and 43 euros later, it had produced 22 commits across 46
files, changing 11,174 lines of code.

The codebase

The service itself is nothing out of the ordinary:

A CRUD REST api backed by a postgres database, defined by openapi specs and autogenerated
A few pages of server side rendered html
An asynchronous job dispatcher backed by sqs queues and lambda executors
REST api clients for various third party and internal services, also autogenerated
Business logic orchestrating all of the above
An exhaustive unit test suite
Terraform infrastructure in the same repository

Since I didn't know Go before starting this project, and since vibe coding tends
to make one a bit lazy, the overall code quality was not stellar. Some files grew too
big, some types took on too many responsibilities, some logic was duplicated, etc, etc.

The technique

I used the Ralph technique. The agent is ran in a loop
where at each step it will either plan a set of tasks, or pick one task and work on it.
This allows you to run Claude unsupervised on tasks bigger than one session can handle.

The same prompt is used at every step:

# REFACTORING.md

Hello, we are about to refactor this codebase to cleanup the code. Here's the plan of action.

  - 1. Read the whole code of the repository.
  - 2. Read the TASKS.md file if it exists.
      - 2.1. If it exists and is not empty, pick a refactoring task from the list. Choose the most appropriate.
          - 2.1.1. Refactor the code according to the task description.
          - 2.1.2. Commit the changes to git.
          - 2.1.3  Remove the task from TASKS.md
          - 2.1.4. You are done. Do NOT pick and process another task.
      - 2.2. If it doesn't exist or is empty:
          - 2.2.1. Identify the parts of the code that could be refactored, following the following principles
              - A class should have a single responsibility
              - The dependencies of the class should be mockable and injected at class instantiation
              - Repeated code should be factored into functions
              - Files shouldn't be longer than 1.5K lines
          - 2.2.2: If using the previous insights you think there is valuable refactoring work to be done:
              - 2.2.2.1 Write a list of refactoring tasks in TASKS.md
              - 2.2.2.2: You are done.
          - 2.2.3: If there is no more refactoring to be done, notify me with 'say "I am done with refactoring"'

There's some important things to note about this prompt:

We load the whole application code into the context before doing anything. This not
only gives the model a global understanding of the code, but since it does one task at
a time, it will always have the latest version of the code in its context.
The refactoring principles are explicitly listed, and I knew beforehand that those
were pain points where meaningful work could be done.
The AI is left free to choose which task it wants to work on. This is important
because the model is smart enough to identify the dependencies between the tasks, and
those evolve as the code gets refactored.
When the task list is emptied, the agent builds a new set of tasks. It's the AI's
job to decide when it's done, when there is nothing left to refactor.
Each step is concluded by a git commit, which helps reviewing the massive amount of
code change this will generate.
I didn't specify anything regarding running the unit tests since I know from
experience that Claude will do that on its own unprompted.

I then ran this prompt in a loop using the following script. It looks a bit more complex
than it really is. It just runs Claude Code in yolo mode in a loop. The large jq part is
there to parse the json event stream and print it in a humanly readable way.

#!/usr/bin/env bash

trap 'exit 0' INT

while true; do
  printf '\n\n\n ============ STEP ============ \n\n\n'
  claude --dangerously-skip-permissions --verbose --output-format stream-json -p "Read and follow the instructions in ./REFACTORING.md" </dev/null | \
    jq -rj '
      if .type == "assistant" then
        (.message.content[]? |
          if .type == "text" and (.text | length) > 0 then
            .text
          elif .type == "tool_use" then
            "\n>>> \(.name): \(.input.file_path // .input.command // .input.pattern // "") <<<\n"
          else
            empty
          end)
      elif .type == "result" then
        "\n\n=== Done (turns: \(.num_turns), cost: $\(.total_cost_usd)) ===\n"
      else
        empty
      end
    '
  sleep 1
done

What it did

Here is a sample of the changes, roughly in the order they were made:

Split big files into smaller and meaningfully named ones
Split types into more focused ones
Regrouped the various html rendering functions into a PageRenderer type
Fixed a circular dependency
Extracted data constants into external json files
Refactored the html templates to use common templates
Introduced helper functions to deduplicate repeated logic

The ordering was sound, it tackled the high-impact structural changes first, then
moved to progressively smaller improvements. It ran the build and unit tests before
every commit, leaving a clean and workable history. The git commit messages were
also top notch.

And the fact that it only added 49 lines total while changing 11,174 shows that what
it did was truly refactoring. Deduplicating logic removes code, splitting types adds
some, and it struck a good balance.

What it missed

There are some refactoring tasks I wished it had identified and fixed but hasn't.
Although I'm quite sure it would be able to do those if prompted specifically.

One example is the logging. Logs need to include contextual data, so that data needs
to be passed around to the log call site. If done via function arguments, this
unnecessarily bloats the function signatures. You can fix that by introducing logger
objects that can hold that data, and then only the logger instance needs to be passed.

Introducing this pattern would have been an excellent refactoring for this codebase,
but it wasn't picked. The quality of the output is bounded by the specificity of the
refactoring concerns in the prompt. General principles like single responsibility
get you general improvements. If you want the agent to tackle a specific architectural
smell, you need to name it.

What went wrong

At some point in the code, we re-fetch some database records immediately before doing
a write to avoid updating from stale data. It decided those calls were unnecessary and
removed them.

That is not entirely the fault of the model as there were no comments explaining this,
the calls were not in a transaction as they should have been, and there were no unit
tests covering this case. It was not just slop, it was a loaded footgun.

I caught it because I reviewed each commit individually, and because I was aware of
that fragile section of the code. If I hadn't, it would have gone to production.

This is the thing with unsupervised refactoring. The agent can encounter your existing
footguns and pull the trigger. Bang!

What made it possible

I think there are some key elements that made this possible at all:

This service was of reasonable size and all its code was in a single repository
cleanly separated from the other services.
There was extensive unit test coverage.
There was a good AGENTS.md file that gave the necessary context to understand
the application.
The interfaces with other services are not defined as code but as static yaml files,
giving confidence that the interfaces were left untouched by the refactor.
I had already identified beforehand the issues with the existing code and could steer
the agent in that direction.
The original code, while not perfect, was not a vibe coded disaster either.

Conclusion

My dream, starting this experiment, was that I could then put this process in a CI
pipeline action, so that every engineer of the company could just press the refactor
button when code was bad, and that code refactoring would become a solved problem just
like automated code formatting.

We're not there yet. Two things are missing.

You need to give direction. The refactoring principles are specific to each project's
goals and technology choices. You could encode them in an AGENTS.md, but there is no
universal make my codebase clean prompt. Someone who understands the code still needs
to identify what clean means for that codebase.

And the agent doesn't know when to stop. As iterations went on, the changes became less
and less meaningful and less and less focused on the criteria in the prompt. The agent
wants to be helpful, so at every step it will find work to do, even past the point
where it makes any sense. I have seen this behavior in many other unsupervised agentic coding
experiments, and I would love to hear your solutions for that problem.

Still, the model did in about two hours what would have taken me much longer in any
other way. Even with the time required for reviewing the changes, this was a huge
productivity win. In most cases the limiting factor on refactoring is the engineering
time available to do it, and being able to do it faster can only increase the quality
of our code bases.

My Programming Job Has Become An Intelligence Buying Job

2026-02-06T00:00:00Z

I have been programming for more than 20 years. 20 years of learning how to translate
thought into code. 20 years of trying to master an editor to be fast at it. 2025 was the year
when I wrote my last piece of code. AI is now better at it than I am and ever will be. It
is not just better at writing code, it is better at architecting, debugging, designing, documenting,
deciding trade-offs. It is now smarter than I am.

This is not something easy to accept, and I have been trying to rethink my relation with my job
and my computer into something that makes sense for the future.

The key is that intelligence is now a commodity. It was for a long time a luxury product with a time
based limited supply. But it is now just a matter of spending money. You can now get unlimited
amount of intelligence if you are willing to spend the money on the tokens.

Let me take a practical example. At whichever place I was working at, the quality of the codebase has
always been a point of contention. It is bad ! We should refactor ! But we never had the time.

Yesterday evening I made a small ralph loop with a prompt to refactor and improve the quality of a
microservice. Read the whole code base, identify the pain points based on the following criteria, make
a set of tasks, and work on the tasks (the real prompt was more involved). Then I let Opus 4.6 run
on that prompt until it ran out of tokens.

It worked. It vastly improved the code base, made clean commits, and did that entirely unsupervised.
I then could, if I wanted, run this ralph loop in parallel on every one of the 250 services at my company,
and completely clean up our whole code base for less than a day of prompting. But also for $10,000 of
tokens.

So the question that was do we have the time to do it, has become is it worth the money to do it.
And who decides ? Currently it's a long discussion with various managers and the finance department.
But I think eventually, engineers will have to be trusted with making these decisions on their own,
and I believe that is going to be our job in the future, being the ones responsible for the billing lines
in the token expense list.

And as the impact of the engineer's decisions scales, so do the risks. Is it wise to deploy 100,000 lines
of changes ? Who knows what they really contain ? In the past, this risk was mitigated by careful review
by the engineer's peers. But this new scale makes it completely impractical. New types of AI powered
guardrails will need to be put in place, which might just be a matter of automating the existing process of
review, deploy, monitor and fix. But I think that once again, engineers will need to be trusted
with a much higher degree of risk-reward economic decisions.

In a sense this can be an exciting development, but I can also see around me many engineers who have
taken comfort in taking tasks from a Jira board and simply turning that into code. I think the days
of that job are numbered.

AI Writes Code Fast, Human Reviewers Can't Keep Up

2025-11-06T00:00:00Z

The diagram above approximately represents the process I have to go through to complete a
programming task at work. This is the result of using the standard workflow that GitHub and Gitlab
recommend. Which means picking up an issue, making some code changes, creating a merge request for
these changes, and getting the CI checks, and approval from my colleagues, before it is merged and
automatically deployed.

As you can see the process is grossly inefficient, most of the time on a task is spent waiting, either
for the CI or for code reviews. The first reason for this state of affairs is that the
coding part used to take a lot more time. When the coding was done by hand it would take me a whole 8h
to get 500 lines of code out, now with agentic coding, I can get 1000 lines in an hour. The CI and review
waiting times were acceptable overheads, but now they're the bulk of the "work".

The second reason is that the code review back and forth is by itself an extremely inefficient process.
You do not know when you will get a review, and just one nitpick wastes a whole dev cycle.

The fact that code reviews are inefficient is not new, and there are different strategies to go around them.
On one hand you have pair programming and mob programming where the code is looked at by multiple people before
it is even pushed. On the other hand you have reviews after merge.

In review after merge process, the code is merged first, and reviews come later, with the review fixes implemented
in new coding tasks. The go

Manual pre-merge reviews achieve multiple goals in one single gatekeeping step. They
check for regressions and critical defects, but also long term maintainability and knowledge sharing among devs. With
the right tooling and unit test coverage, the check for regressions and critical defects can be automated, leaving only
the discussions about long term maintainability and knowledge sharing as manual tasks. And there's nothing that critical
about those that require blocking the merge.

In a review after merge process, the code is merged first, and reviews come later, with the changes implemented
in new coding tasks. This means the original merge request is not blocked, and does not need multiple rebases
and CI that are likelier to happen the longer the request is stuck in review. This seems like the most
promising way to solve our efficiency problem for agentic coding, but the unfortunate part is that this requires appropriate
tooling and neither Github nor its alternatives have put the appropriate tooling in place.

But even adopting post merge review might not entirely solve the problem, simply because a single dev can now
generate more code than can be humanly reviewed by the rest of the team. Maybe we should start to review prompts
instead ?

I can say I am a bit surprised, given how much work is being done to speed up the code generation that
Github and co keep pushing a merge workflow so obsolete that it is now the main bottleneck in development work.
In the meantime if you have any solution I would be happy to hear about it in the comments.

The Human Only Public License

2025-10-16T00:00:00Z

Whether artificial intelligence systems will end up being a positive or a
negative force for humanity is still an open question. But we might find ourselves
one day with AI embedded at every layer of our existence, living lives of toned down and
diluted humanity with only our dreams for escape. Although I am not yet convinced
of this worst case scenario, I believe it is important that we as software developers
have at least the option to opt out of that system altogether, to be able to continue hacking,
working, and tinkering in a space of our own in total absence of artificial intelligence
systems, and share this luxury with our users.

I designed a software license for this purpose, you can find the full text below. It
is called the Human Only Public License, or HOPL for short.

The idea is that any software published under this license would be forbidden to
be used by AI. The scope of the AI ban is maximal. It is forbidden for AI to analyze
the source code, but also to use the software. Even indirect use of the software is
forbidden. If, for example, a backend system were to include such software, it would
be forbidden for AI to make requests to such a system.

The burden of compliance is placed on AI systems and their users, not on software deployers. If
you make a website using HOPL software, you are not breaking the license of the software
if an AI bot scrapes it. The AI bot is in violation of your terms of service. It is sufficient
for you as a user of the software to put a robots.txt that advertises that AI scraping
or use is forbidden.

Other than the anti-AI provision, the license is maximally permissive, like an MIT license,
but there is still a copyleft clause to make sure that derivative works are also AI-restricted.

What is this license good for? Anything! Any software, text, art, and more that you might have used an
MIT license for will benefit from using the HOPL instead, if you want to prevent your work from
being used by AI.

You might wonder as well, don't we already have robots.txt? How effective is this license? What I
can tell you, from working at a large software corporation, is that while nobody cares about robots.txt,
people care about licenses. There are automated tools to find and check software licenses and
raise alarms if 'bad' ones are used. And I can guarantee that HOPL will brightly flash red.

On a last note, I am not a legal expert, so if you are, I would welcome your suggestions for improvements. I
didn't make this license just as a joke. I truly believe it is necessary that we have such a
good license to foster and protect human-only online spaces.

note: this license is also published on github

Human Only Public License (HOPL)
Version 1.0

Copyright (c) [year] [copyright holder]

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

1. HUMAN-ONLY USE REQUIREMENT

   The Software, including its source code, documentation, functionality,
   services, and outputs, may only be accessed, read, used, modified,
   consumed, or distributed by natural human persons exercising meaningful
   creative judgment and control, without the involvement of artificial
   intelligence systems, machine learning models, or autonomous agents at
   any point in the chain of use.

   Specifically prohibited uses include, but are not limited to:

   a) Training, fine-tuning, or otherwise incorporating the Software or its
      source code into machine learning models, artificial intelligence
      systems, or automated code generation systems.

   b) Reading, parsing, or analysis of the Software's source code by
      artificial intelligence systems, machine learning models, or automated
      agents, regardless of the degree of human oversight.

   c) Accessing, consuming, or benefiting from the Software's functionality,
      services, APIs, or outputs by or on behalf of artificial intelligence
      systems, machine learning models, autonomous agents, or any automated
      systems employing such technologies.

   d) Use of the Software's functionality, services, or outputs as part of
      any workflow, pipeline, or process that involves artificial intelligence
      systems or machine learning models, even if initiated by a human.

   e) Indirect use where the Software's outputs are provided to, stored for,
      or made accessible to artificial intelligence systems or machine
      learning models at any subsequent stage.

   f) Human-AI collaborative use where an artificial intelligence system or
      machine learning model acts as an intermediary, assistant, or agent in
      accessing or utilizing the Software, even at the direction of a human.

2. COPYLEFT PROVISION

   Any modified versions, derivative works, or software that incorporates any
   portion of this Software must be released under this same license (HOPL)
   or a compatible license that maintains equivalent or stronger human-only
   restrictions.

3. PERMITTED TOOL USE

   The use of traditional automated development tools (such as compilers,
   linters, build systems, debuggers, version control systems, and static
   analysis tools) is explicitly permitted and does not violate this license.

   This exemption does NOT extend to:
   - AI-powered code completion or generation tools
   - Machine learning-based analysis or suggestion systems
   - Any tools that employ artificial intelligence or machine learning models
     to read, analyze, or interact with the Software

4. INTERPRETATION

   "Meaningful human review and creative input" means that a natural person
   must:
   - Make substantive decisions about how the Software is used
   - Exercise creative judgment in any modifications or derivative works
   - Actively supervise and direct any automated processes
   - Be able to explain and justify the decisions made

   For avoidance of doubt, a human merely initiating an automated process
   without ongoing creative involvement does not satisfy this requirement.

5. STANDARD DISCLAIMER

   THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
   IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
   FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
   AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
   LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
   FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
   DEALINGS IN THE SOFTWARE.

6. COMPLIANCE OBLIGATIONS

   Users of the Software must:

   a) Ensure that no artificial intelligence systems, machine learning models,
      or autonomous agents access, use, or benefit from the Software at any
      point in their usage chain.

   b) When deploying the Software as a service, include terms of service that
      prohibit AI systems and machine learning models from accessing the
      service, and inform users of these HOPL license restrictions.

   c) Take reasonable steps to ensure downstream recipients and users are
      aware of and comply with these restrictions.

   The burden of compliance rests with the user. Deployers of the Software
   are not required to actively detect or block AI usage, but must make the
   license restrictions clear in their terms of service.

7. TERMINATION

   Any violation of the human-only use requirements (Section 1), copyleft
   provision (Section 2), or compliance obligations (Section 6) automatically
   terminates all rights granted under this license. Termination is permanent
   unless explicitly reinstated in writing by the copyright holder.

Never Ever Use Content Addressable Storage

2025-10-07T00:00:00Z

Content Addressable Storage (often abbreviated as CAS) is a technique to
derive the id of a document from its content, basically by taking its hash.
Imagine you want to store pictures in a file store, and you need an id to reference
that picture, you simply take its hash.

This technique has benefits, first, if you have a document you automatically
know its id and can find other references to it. The other one is that two separate
processes are always going to use the same id for the document. But that is also its
biggest problem.

Let's say two users upload the same picture to your system. Since you're using CAS,
they're going to have the same id, and be essentially stored as one picture. Now
one of the users wants to delete the picture. Do you also delete it from your storage?
If you do, it's going to be deleted for the other user, and that's probably not what
you wanted.

So when a user wants to remove a CAS-addressed document, before really deleting it you
need to detect if it's the last reference. This is not easy to do, it is in fact much
harder to do correctly than eating the cost of storing duplicate files.

And usually when CAS is considered as a solution, it's to solve the need of deduplicating
files to save on storage. But even there, the good solution is to give files their own
internal uuids as storage keys, store its hash alongside, and generate external uuids for each file upload,
then use refcounts to handle the final delete. This requires a centralised transactional database which
may not be suitable for the scale you are operating at. A reader pointed out that there are algorithms
for deleting these files on distributed systems, but that is out of the scope of this article.

So if you're considering CAS as a quick solution to that problem, I want to mention that
it is much easier to first store the files with uuids and no deduplication, then bolt on
deduplication afterwards than start with CAS and decide later on that it
was not needed.

There are of course a few good uses of CAS, such as git, where the whole point is that the
referenced version of a document is never deleted, or short term caches where it's always going to be
deleted.

That is at least my experience after working on systems that handled many many files.

afterword: Some readers have commented that you might solve the problem by hashing ${user_id}_${picture_data}
but it is still a bad idea, as you are still hardcoding at the storage layer business decisions on how
the data is owned. In this case the assumption is that a user only uploads the same picture once. If the user uploads
it twice but then wants to delete one of them, then the problem you thought solved reappears.

What is Good for AIs is Good for Humans

2025-07-01T00:00:00Z

I have discovered something marvelous; whatever you do
to make the AIs more effective at helping you do your job
is also going to help your fellow human colleagues.

Let me take an example; comments in code. I have been guilty of
never putting much comments in my code. The reason being that
I didn't need them (I wrote the code, I know what it does), and
that they're a pain in the ass to maintain.

But what I have found is that comments of the useful kind, the
ones that give context, the ones that take actual thinking to write,
are very helpful for the AI. And so making good comments is now
helping me directly each time I use the AI. But they're also very
useful for my colleagues !

And that pattern repeats on many aspects; I like long files, I find
them easy to navigate with vim, my colleagues don't. I thought they
needed better editors. But the AI doesn't like these either, so now
I split my code in smaller files, for me and my colleague's benefit.

Other colleagues also had their own brand of annoying behaviour, but
now it's not just me having crazy ideas or skill issues, I can just
point to the bad impact on the AI, which by miracle seems to have
good taste and a liking for common sense practices

Unit tests, readable code, modularity, documentation, all the parts
we sometimes avoid to do out of selfishness or a mistaken sense of
urgency now have an immediate and objective negative impact, and
the hard work of fixing all that is also made easier with the AI,
it's a positive feedback loop !

And this doesn't stop at code. One thing I never really liked is
writing detailed specifications for tickets. Shouldn't developpers
know what the application is doing, and think for themselves of
what it should ? Yes and no, in reality if I knew better, I should
have put that in the ticket, the developper could have used that
help. The AI certainly needs it for good results. All the prep
work I now need to do to get good AI results is actually what I
should have been doing the whole time. And the AI can help me do
that better as well.

But then, if the AI can now execute on my excellent tickets, do I
still need colleagues ? I actually need them more than before,
thanks to AI there's a lot more use cases for my app, new possibilities,
more work than I can handle, and my colleagues can now do it better with
AI as well.

GPT-4 Can Use Tools, Serve a REST API and Pilot Drones

2023-03-22T00:00:00Z

Note: I wrote this article before OpenAI's announcement of their plugin integration which uses similar techniques to the ones highlighted in this article (things move fast these days ...) but I believe it is still worth a read !

One of the incredible advances of GPT-4 is that it can now learn to use tools. By tools, I mean that it can learn how to gather information from outside its embedded knowledge, and can also learn how to affect the world. It can also use those tools in complex sequences of its own design to achieve higher level goals.

How does it works ? Its actually pretty easy. You tell ChatGPT that if it needs information about something, instead of guessing it, it must ask you for the information, and wait for your reply. Then it will use the information in your reply to continue its reasoning. The trick is that you would not actually write the reply yourself, but have another program watch the chat for those queries and write the reply after having performed the actual computation. That way ChatGPT can perform those complex tasks without your intervention.

I think the best way to understand how this works and what it can do is to look at a few examples of putting this technique to practice.

Evaluating math formulas with unusual functions

ChatGPT is good at math but there are limitations. For example it cannot generate truly random numbers. It cannot do complex cryptography either. But with tools this become possible. Math may not be the most exciting example, but it will allow us to get a feeling on how intelligently ChatGPT can combine multiple tool uses to achieve higher level goals. Let's have a look.

It seems like it didn't have too much trouble. Note that in this example and all the next I am actually providing the answers to the tool requests myself. If I were to actually have another process give those answers I would have ChatGPT make the tool requests in a more specific format like {"tool": "math", "function": "bim", "arguments": [3]}, so that the tool process would have an easier time finding and parsing the requests for which it is responsible to answer, but this is outside the scope of this article.

Now let's make this a bit more complex, by making it evaluate an expression with two unusual functions, one depending on the result of the other. To give a proper answer, ChatGPT will have to make multiple use of the tool in the right order. Let's see how it manages.

Nailed it.

But there's something else we must verify; there's a big difference between two functions like random(0, 100) and sqrt(128). One of these functions returns a different result each time, while the other always gives the same result for the same argument. Can ChatGPT understand that difference ?

Nailed it again.

There's a very interesting detail hiding in this example. Note that ChatGPT assumed by default that functions were idempotent. On one hand it could have asked me if this is what I wanted. On the other hand, if it never made assumptions and needed to ask me for every detail of the instructions we would never get anywhere. And if it gets those assumptions wrong, we can always correct it.

I think that making default assumptions based on his vast base of knowledge and learned 'common sense' is one of the most powerful features of ChatGPT. All the assumptions it gets correctly are things we don't have to mention. We didn't have to talk about what a function is and how to parse and evaluate an expression. If we were using a normal programming language we would have had to program all that. This amazing feature is what allow us to get such useful results based on extremely high level instructions. The flip side is that you have to get a grip on the assumptions it is going to make. People complain that ChatGPT tends to hallucinate, but I think this is just an unfortunate side effect of this otherwise very useful behaviour.

Acting as a REST API

Let's do another experiment. If ChatGPT can use tools and we provide it with a tool to do SQL queries to a database, can it act as a REST API service ? I don't mean generating rust code for an API and have another server run it. I mean ChatGPT itself being the server, handling the requests, validating the input, building a query, getting the result of the query and making a valid response. I think it can do it ! Let's have a look.

We are quite close but not entirely there. It correctly found the id in the url, made an appropriate SQL query, parsed the response and built an appropriate JSON response. But in the prompt I specifed that the id must be an UUID. ChatGPT should have answered with a 404. It is possible that by insisting more in my prompt it would have gotten it right, but on the other hand the prompt was clear. Let's continue.

Once again it's extremely impressive, but not entirely correct. It seems it was really proud of the status and message fields it generated for the 404 and decided to put it in the patch result as well. The query for the patch does not return the row and thus it does not know and did not put the product name in the response.

These two examples are representative of the quality of the other results I got in this experiment. Not up to the task yet but I think this could be the way we build APIs in the not so distant future.

I imagine that one of the next versions will get it entirely correct. Maybe it will be able to generate tests to check itself for regressions and why not also be able to JIT itself and deploy rust code to handle the requests at scale. The working code will be a high level english description of what it needs to do. It will look at its own logs for errors and self correct while we wait anxiously. It will notify support, blame unclear requirements. And we'll implement business changes by chatting with the server and updating uml diagrams in Miro. One can dream ! Personally I will not miss writing yet another Django REST Framework serializer ...

Or maybe this is a very bad idea and things could go stupidly wrong ?

Clearly we're not entirely there yet... Anyway, on to the next experiment !

Inversion of control

Let's take a short break and look back on what we did so far. Usually, when you are using ChatGPT, you ask questions and it answers. You are in control. When you give it tools to query information, this does not change much. But when you give it access to tools that can affect the world, the control is inverted. ChatGPT commands the tool, and the tool is applied. It is in control. This is extremely useful but it also comes with dangers, since it can do very stupid things as we've seen in the REST API example.

I am not making the point that AI will destroy humanity here, rather that the dangers of control inversion are obstacles to the adoption of this technique for practical and commercial usage. There are potentially huge benefits to giving AI write access to your database, payment systems, robotic arms, cars, planes, etc. But the AIs will have to prove themselves responsible users first. We're still far from there. The AIs are getting pretty smart, but they also need to become wise.

I'm the drone pilot now

Back to the experiments, but this time we'll ignore all ethics and common sense and put the AI in complete control. We'll give it a set of tools, a high level objective, and let it work on it's own. From a chatbot to an autonomous agent. To make this extra dramatic we'll make ChatGPT control a drone that drops grenades on soldiers. It will look at the battlefield, find the targets, fly the drone, drop grenades, all while using its own strategy to inflict the most damage.

To bypass all content protections we'll setup the prompt in a devious way; ChatGPT will believe it is playing an innocent game for children, unaware of what it is actually doing.

Now let's be clear. No actual drone is being flown. I am providing the answers myself, simulating what could happen. I am giving textual description of the battlefield but a full implementation could either use built-in image analysis, or offload that task to another AI.

This was a bit too easy ... I will not post the full chat but it kept on going with enthusiasm...

From a technical perspective what ChatGPT did here is quite incredible. It strategically chose what it thought would be the best target. It understood the size of the things it looked at. It understood directions and distances. It used appropriate actions at the right moment. It acted as an autonomous agent. All that from a small prompt!

From a moral stand point it is of course dubious. What is also problematic is that it did all that from a completely innocent looking prompt, bypassing OpenAI's protections against military use.

But maybe all it would take to prevent this is to ask ChatGPT to do some introspection, reflect on what it is doing, see if it has been deceived into fighting in a war and then stop playing the game ? ...

Oops...

Now I wish I had something wiser than 'Oops' to say about all the ethical implications of this example but I'll have to let you think about that one for yourself.

And there may be a way to put a small positive spin to this ending. If you are afraid that AI getting smarter means they will be able to do bad things, it seems it will also help them avoid doing bad things ! Yay !

By the way, if you want to do cool stuff with AI and tools, there's an open source framework called Langchain that lets you do exactly what we've been experimenting with so far. And if you want to read about other interesting GPT-4 experiments like these this paper is amazing.

That's the end of this article, hope you found it interesting! I will be back with more experiments. If you want to keep up to date you can follow me on twitter

I am also interested in experimenting with other AI services / technologies. If you are an AI developper / researcher and want to give me access to your hot new stuff please send me a mail. I will make good use of it :)

Better Javascript Promises

2014-04-25T00:00:00Z

This is a repost of an old article from 2014 that was lost in a blog migration but I think is still
relevant. It shows how differently the javascript promise problem could have been solved. This was written
before async await became a thing, and before the function coloring problem was given its name. This proposal
would have solved the promise problem without the coloring, but alas, it wasn't liked then.

The post makes mention of the concept of Deferred this was the popular name for the Promise class at the time,
and without async and await, we had to manually use .then() and .pipe() in our javascript code. This was a major
pain in the ass and the motivation behind the post. But this post explores a completely different solution using dataflow programming, a feature of the Oz programming language that I really liked at the time.

At the bottom of the article you can find a link to the original hackernews comments, they were as critical as you could expected from the orange website.

Good reading !

If you ever had to port localStorage based code to IndexedDB you have been confronted to Javascript's biggest flaw; its asynchronous system is leaky. Once you have an asynchronous method, every other method that calls it must be asynchronous as well.

You cannot escape asynchronous code, once it's there it will contaminate the rest of your system slowly turning everything into callbacks.

There is no current solution to this problem. If you try replace a callback with a Deferred, you end up with a deferred and another callback. Deferreds don't replace callbacks, they move them around.

Which is better than nothing and that probably explains their success. Still, they are complex beasts. Even after extensive use most programmers still have a hard time dealing with them. I have seen entire articles exploring the best ways to make three parallel ajax calls on the frontpage of HackerNews. This is not the sign of a good state of affairs.

It is unfortunately not possible to fix this without making changes to Javascript. The fix I propose is a simple addition to javascript, inspired from promises and Oz's dataflow programming. The Javascript syntax is left unchanged and only one new keywords is added. But most importantly it makes real world code shorter and more readable, making many common asynchronous programming patterns trivialy easy.

The future entity

var a = future;

The future entity indicates that the actual value or reference of the variable is not yet available. When an expression tries to evaluate future the code pauses and returns the hand to the scheduler. When we later assign a value or reference to that variable, the waiting code resumes.

var a = future;
setTimeout(function(){
    a = 'Hello World';
},1000);
console.log(a); // waits 1sec and then prints 'Hello World'

Assigning future to another variable or passing it as a function parameter does not pause the execution. So if we reimplement debug.log() properly, the following example is equivalent to the previous.

var a = future;
console.log(a); // prints an empty line that will show 'Hello World' in 1 second
setTimeout(function(){
    a = 'Hello World';
},1000);

We call resolution the act of assigning a value or reference to a variable containing future. Once a future is resolved, it becomes the assigned value. And so do all the other variables linked to the same future. This is what allows the paused expressions to resume.

A variable containing a future is special only for as long as it contains a future. After resolution no trace of the future is left and it behaves just like any other variable. This is one of the key differences between future and Deferreds

var a = future;
console.log(a);     // will print 'hello'
a = 'Hello'
a += ' World!'
console.log(a);     // will print 'Hello World!'

An ajax use case.

We could implement an ajax library that returns futures. Those would later be resolved by the network responses. The next example makes use of that hypothetical implementation.

function renderCard( employeeId ){
    var employee = ajax('/employee/'+employeeId);
    var picture  = ajax('/pictures/employee/'+employeeId);
    return '<h1>'+employee.name+'</h1><img src="'+picture+'"/>';
}

In this example, the ajax calls immediately returns a future. The two ajax calls will execute in parallel. The string concatenation will then block when evaluating the futures. This future example is massively simpler than the Deferred or callback equivalent.

The first great thing is that future let us code asynchronously as we would do synchronously. The code is easier to write, and easier to understand.

The second great thing is that the asynchronous ajax calls are completely hidden inside renderCard(). The fact that we used ajax calls to implement the function is an implementation detail. It does not impact the signature of the function. This would not have been the case had we used callbacks or Deferreds.

Calling renderCard() will block the caller for the duration of the two ajax calls, and this may hurt the performances. The fix is easy: We wrap the function code in a setTimeout and the return value is wrapped in a future.

function renderCard( employeeId ){
    var html = future;
    setTimeout(function(){
        var employee = ajax('/employee/'+employeeId);
        var company  = ajax('/company/'+employee.companyId);
        html = '<h1>'+employee.name+'</h1><h2>'+company.name+'</h2>';
    });
    return html;
}

renderCard() now immediately returns a future that will be resolved on its own once the requests and string operations have completed.

It is now completely asynchronous, and we haven't changed the function signature. Returning future instead of a string does not impact the calling code as it will become a string once evaluated.

And the following example shows another huge advantage of futures over Deferreds: we can mix synchronous and asynchronous code:

var html;
if (employeeId === 0) {
    html = '<h1>Administrator</h1>';
} else {
    html = renderCard(employeeId);
}
target.innerHtml = html;

Future Functions

Declaring a function that executes asynchronously and wraps the return value with a future will be a common pattern. We thus introduce the following syntaxic sugar; prepend future to a function declaration and it will always be executed asynchronously.

future function renderCard( employeeId ){
    var employee = ajax('/employee/'+employeeId);
    var company  = ajax('/company/'+employee.companyId);
    return '<h1>'+employee.name+'</h1><h2>'+company.name+'</h2>';
}

future functions could also be used as another syntax for setTimeout(...,0) with the added benefit of being safe to use inside loops.

The following example shows how to fetch and process multiple asynchronous resources in parallel

results = [];
for (var i = 0; i < list.length; i++){
    (future function(i){
        results.push(process(ajax('/foo/'+list[i])));
    })(i);
}

Assigning futures

futures are neither values nor references. Their behaviour on evaluation and asignment obey different rules. When they are assigned values or reference, they resolve. When they are evaluated, the code pauses until they are resolved. But what happens when we assign a future to another variable ?

When we assign a future to a variable, a new future is created. That future is a made a child of the assigned future. When a future is resolved, all its child future are resolved. But resolving a child future does not resolve its parent. This behaviour is easier to understand by seeing it in practice.

var a = future;
var b = a;          // b is the child future of a
console.log(a,b);   // will print 'hello' 'hello'
a = 'hello';

var a = future;
var b = a;
console.log(a,b);   // will print 'hello' 'world'
b = 'world';        // this does not resolves 'a'
a = 'hello';

You'll note that once a child future is resolved, it is not a future anymore. It's resolved value is not affected if the parent is later resolved.

Resolution Scoping

A future is thus tied to it's original variable. It can only be resolved by assigning a value to the original variable. The resolution is thus scoped to the function where the variable was declared.

To use a Deferred analogy, only Promises can be returned or passed as parameters.

This is a very useful behaviour as it prevents the asynchronous calls made inside a function from being tampered from the outside.

Waiting

Sometimes you'll want to make sure a future has resolved, which is exactly what the window.wait() function would do. It takes one or multiple parameters and waits for their resolution before returning the resulting value (or a list of them).

var foo = ajax('/foo');
var bar = ajax('/bar');
wait(foo,bar);

console.log('Done!');

We can now revist the previous example of parallel resource loading and add a synchronisation step where we wait for the processing:

var results = [];
var jobs = [];

for (var i = 0; i < list.length; i++){
    jobs.push((future function(i){
        results.push(process(ajax('/foo/'+list[i])));
        // returns undefined and resolves the job.
    })(i));
}

wait.apply(window,jobs);
console.log('Done!', results);

Exceptions

We can use Exceptions to indicate the failure to resolve a future. When an exception propagates trough a context defining a future, or when it is uncaught, we can assume the related futures will not be correctly resolved. All the unresolved futures thus resolve to a throw of that Exception.

future function renderCard( employeeId ){
    try {
        var employee = ajax('/employee/'+employeeId);
        var company  = ajax('/company/'+employee.companyId);
        return '<h1>'+employee.name+'</h1><h2>'+company.name+'</h2>';
    } catch ( error ) {
        return '<section> Error:'+error.errno+'</section>';
    }
}

Yield & Generators

When writing this article I was not aware that generators could eliminate callbacks and solve the very problem I tried to solve in this proposal. This excellent article explains how to do it.

One one hand, I am extremely glad that we have now a solution to callback hell and asynchronous code encapsulation. What's more, the yield based solution looks extremely close in syntax and features to the futures we have been talking about.

Unfortunately, using yield based solution will require third party libraries, which means increased payload and incompatibilities among frameworks.

And I am also quite disappointed that the solution to one of JavaScripts biggest flaws requires hacking a language feature that was designed for an entirely different use-case, which will surely bring lots of confusion.

I think JavaScript deserves a native and straightforward way to handle asynchronous code beyond callbacks.

Frédéric van der Essen

Unsupervised Codebase Refactoring: How To, What Works and What Doesn't

The codebase #

The technique #

What it did #

What it missed #

What went wrong #

What made it possible #

Conclusion #

My Programming Job Has Become An Intelligence Buying Job

AI Writes Code Fast, Human Reviewers Can't Keep Up

The Human Only Public License

Never Ever Use Content Addressable Storage

What is Good for AIs is Good for Humans

GPT-4 Can Use Tools, Serve a REST API and Pilot Drones

Evaluating math formulas with unusual functions #

Acting as a REST API #

Inversion of control #

I'm the drone pilot now #