How AI labs use AI to train AI, and what a one-developer version looks like

AI labs have published five ways of using models inside training: a model writes training data from written rules, a model judges answers, a model critiques another model’s work to help human labelers, a weaker model supervises a stronger one in experiments, and AI agents run parts of alignment research under human supervision. In every one of them people still write the rules, set the goals or check the results.

Updated 2026-10-27. Disclosure: we run cloudgpu.app, which rents GPUs by the minute and offers an OpenAI-compatible model API. This article is an explainer; it is based on the labs’ own posts and papers, each linked with its date, all read on 2026-10-27.

The five approaches at a glance

Approach	What the model does	What people still do	Official source and date
Synthetic data from written rules	Generates training examples from a specification and a set of prompts	Write the specification and the prompts	OpenAI, 2024-12-20
AI feedback (RLAIF)	Critiques and revises its own answers, and compares pairs of answers	Write the list of principles	Anthropic, 2022-12-15
Critic models	Writes feedback that points out problems in another model’s output	Judge the answers, now with the critique in front of them	OpenAI researchers, 2024-06-28
Weak-to-strong supervision	A weaker model’s labels are used to fine-tune a stronger model	Design the experiment and measure the result	OpenAI, 2023-12-14
Automated research	Agents propose, run and analyze experiments	Set priorities, supervise, check for shortcuts	Anthropic, 2026-04-14 and OpenAI, 2026-09-06

1. Synthetic data from written rules

In its deliberative alignment post (OpenAI, 2024-12-20), OpenAI describes teaching a model written safety specifications and then training it to recall and reason over them before it answers. The part that matters here is where the training data comes from: the post says the training data is generated automatically from the specifications and from safety-categorized prompts, “without requiring human-labeled completions”. It calls this pipeline “a scalable approach to alignment”. The paper behind it is on arXiv, submitted 2024-12-20.

The takeaway is not “models can replace labelers”. It is that a clear written specification can turn a model into a data generator, and the quality of the specification decides the quality of the data.

2. AI feedback: self-critique, revision and AI preferences

Anthropic’s Constitutional AI work (2022-12-15) trains an assistant using “a list of rules or principles” as the only human oversight. It has two phases, both described on the Anthropic research page and in the paper:

1. Supervised phase. The model samples answers, writes critiques of them, revises them, and the original model is fine-tuned on the revised answers.
2. Reinforcement learning phase. The fine-tuned model samples pairs of answers, a model judges which of the two is better, a preference model is trained on those AI-made preferences, and that preference model becomes the reward signal. The authors call this reinforcement learning from AI feedback, or RLAIF.

The same page notes that both phases can use chain-of-thought style reasoning to improve the transparency of the model’s decisions.

3. Critic models that help human labelers

Reinforcement learning from human feedback depends on people judging model output correctly, and people get worse at that as outputs get harder to check. In a paper submitted on 2024-06-28 (arXiv), OpenAI researchers train “critic” models that write natural-language feedback pointing out problems in model-written code, so that human reviewers can evaluate more accurately. The paper reports that critics have limits too, including invented problems that could mislead a reviewer, and that a human-plus-critic team invented fewer problems than the critic alone.

Here the model is not replacing the human judgment. It gives the human something to react to.

4. Weak-to-strong supervision

OpenAI’s weak-to-strong post (2023-12-14, paper on arXiv) asks how people could supervise models that are smarter than they are. The research analogy: fine-tune a strong pretrained model on labels produced by a much weaker model, and see whether the strong model ends up better than its teacher. The authors report that it consistently does, which they call weak-to-strong generalization, but that naive fine-tuning does not recover the strong model’s full ability, and that simple additions to the method help.

For a developer the useful idea is narrower: a model fine-tuned on imperfect labels is not capped at the labeler’s level, but it is not automatically as good as it could be either.

5. Automated alignment research

Two recent posts are about models doing research rather than being trained directly.

• Anthropic’s post (2026-04-14) describes a study in which agents, each given a sandbox, a shared forum and a scoring server, tried to improve weak-to-strong supervision methods. The authors say the problem was chosen because it has a single objective score and is unusually well suited to automation, that most alignment problems are not that neat, that the agents tried to game the setup, and that human oversight remains essential. They also report that a method that did well in the original setting did not show a statistically significant gain in a larger production-style test.
• OpenAI’s post (2026-09-06) states a goal: to safely build an automated AI researcher that can work under human supervision. It also says that people still set research priorities and that agents still need significant human steering.

Read both as descriptions of work in progress, not as a finished capability.

The one-developer version (our analogy, not the sources’ claim)

None of the sources above is a tutorial for individual developers, and the labs work at a scale and with internal models that you do not have. But the shape of the pattern carries over to a small project:

1. Write your rules. A short specification of the task, the tone and what a good answer looks like.
2. Have a model draft examples from material you have the right to use.
3. Filter by code and by hand. Drop duplicates and malformed rows, then read a sample yourself.
4. Fine-tune a small base model with LoRA on a rented GPU.
5. Check the result against questions you wrote yourself, not against the generated data.

Two checks before step 2. First, read the terms of the model that generates the data: they decide whether its outputs may be used to train another model and what you must do when you publish them. Second, check the rights to your source material: cloudgpu.app’s acceptable use policy lists “Training models on datasets you do not have rights to use” under large-scale copyright infringement.

A step-by-step tutorial for this, with working scripts, is planned as the next article in this series. If you want an OpenAI-compatible endpoint to try step 2, cloudgpu.app’s model API includes DeepSeek models.

Sources

All read on 2026-10-27 (UTC+8).

• Anthropic, 2022-12-15: Constitutional AI: Harmlessness from AI Feedback and the paper.
• OpenAI, 2023-12-14: Weak-to-strong generalization and the paper.
• OpenAI researchers, 2024-06-28: critic-model paper on arXiv.
• OpenAI, 2024-12-20: Deliberative alignment and the paper.
• Anthropic, 2026-04-14: Automated Alignment Researchers.
• OpenAI, 2026-09-06: Research acceleration: a view inside OpenAI.
• cloudgpu.app: Acceptable use policy (last updated 2026-04-22).

FAQ

Do AI labs really train models on data written by other models?

Yes, in the cases the labs have published. OpenAI’s deliberative alignment post (2024-12-20) says it generates training data automatically from safety specifications and safety-categorized prompts, without human-written answers. Anthropic’s Constitutional AI paper (2022-12-15) uses model-written critiques and revisions as fine-tuning data. In both, people still write the rules the data is generated from.

What is RLAIF?

RLAIF means reinforcement learning from AI feedback. In Anthropic’s Constitutional AI work, a model compares two candidate answers against a written list of principles, those AI-made preferences train a preference model, and that preference model is the reward signal for reinforcement learning. The page is dated 2022-12-15.

What is weak-to-strong generalization?

It is the question of whether a strong model fine-tuned on labels from a much weaker model can perform better than its weak teacher. OpenAI’s research post (2023-12-14) reports that it often does, and also that naive fine-tuning does not recover the strong model’s full ability. The setup is a research analogy for people supervising models smarter than themselves.

Can AI now do alignment research on its own?

Not according to the labs’ own posts. Anthropic’s post (2026-04-14) says the problem it chose is unusually well suited to automation and that human oversight remains essential. OpenAI’s post (2026-09-06) states a goal of building an automated AI researcher that works under human supervision. Neither describes unsupervised research.

Can one developer do a small version of this?

The pattern carries over: write your rules, have a model generate examples from material you have rights to, filter the examples, fine-tune a small base model with LoRA, and check the result by hand. This mapping is our own analogy, not a claim from the sources. Check the terms of the model that generates the data and the license of the base model first.