Tools | How's it workhttps://hsiw.io/blog/en/tags/tools/I'm interested in how things work.HugoenFri, 13 Nov 2026 00:00:00 +0900Reading Lean Side by Side: Introducing Taiyakhttps://hsiw.io/blog/en/posts/taiyak/Fri, 13 Nov 2026 00:00:00 +0900https://hsiw.io/blog/en/posts/taiyak/Taiyak is a browser tool that splits Lean 4 source into declarations and shows each one next to a translation into natural-language mathematics.I made Taiyak

I made Taiyak, a browser tool that splits Lean 4 source into declarations and shows each one next to a translation into natural-language mathematics. The name comes from the Japanese taiyaku (対訳), a text printed side by side with its translation.

It opens a GitHub URL (private repositories too), a ZIP file or a folder. It only reads the source and never runs Lean, so you need no Lean installation. Opened without a URL, it shows a demo: a formalization I tried on CRuby.

The Taiyak screen with the demo Statements.lean open: Lean code on the left, the English translation on the right. Clicking ∀ in proposition P1 pops up an explanation of the universal quantifier.
Click ∀ to see what it means

It started with the CPython formalization

It started when I read “Formalizing CPython” by David Andrews and Chao Zhang. Working mainly with Claude Opus 5.5, they copy the functions of CPython 3.15.2 into Lean and check each one in four steps:

  1. Copy: rewrite the C or Python source in Lean, keeping error messages and edge cases
  2. Test: run the Lean copy against the real CPython on at least 50,000 generated inputs per function
  3. Prove: state a property and have the machine check it; if it is false, keep the counterexample
  4. Check: another agent, one that did not write the copy, asks whether a broken implementation could still pass the proof

So far they have written 613,318 lines of Lean, and along the way 418 bug reports have been filed against Python.

It looked fascinating, so I tried to read it, but being new to Lean I found the theorems hard to take in one by one. Later I also started an experiment applying the same method to CRuby’s range.c (rb_range_component_beg_len, which normalizes range arguments). Taiyak’s demo is the Lean files from that experiment.

Why not read it as a parallel text?

Before learning Lean from scratch, perhaps I could read it if mathematical prose sat right next to it. It is the same idea as a bilingual reader for language learners.

Lean, I thought, suits this well. Put an ordinary program into prose and you get an explanation of what it does, worded differently by every writer. A Lean theorem is itself a proposition, and a definition is a mathematical object. ∀ x, P x can only be translated as “for every x, P x”. Because the right translation is nearly fixed as mathematics, setting it against the Lean also makes mistranslations easy to spot.

As an example, here is the normalization of the start index in the demo’s Spec.lean.

def specBeg (i : Input) : Int :=
  let b := i.b.getD 0
  if b < 0 then b + i.n else b

Next to it, Taiyak shows the heading “Definition: normalized start B” and the following formula, where b₀ is the start (0 if it is nil). Formulas are rendered with KaTeX.

B={b0+n(b0<0)b0(b0≥0) B = \begin{cases} b_0 + n & (b_0 < 0) \\ b_0 & (b_0 \ge 0) \end{cases}

The only other line is “negative values count from the end”. Set against the three lines of Lean, you can tell at once whether the translation is right.

Why Lean proofs are trusted

Lean proofs are trusted because the part that checks them is small, and a different implementation can check them again.

  • Propositions are types, and proofs are terms of those types. Whether a proof is correct comes down to a mechanical check that the term has the type.
  • Only a small kernel does the checking. Automation such as tactics merely builds proof terms, and the kernel checks the result again, so even a bug in a tactic cannot get a false theorem through (Lean reference).
  • The kernel can be re-checked by other implementations: the official lean4checker, Lean4Lean written in Lean, Nanoda written in Rust and others. You do not have to trust the whole Lean implementation (Lean FAQ).
  • The axioms a proof relies on are recorded. If #print axioms shows only the standard propext, Classical.choice and Quot.sound, all is well. sorry shows up as sorryAx, and native_decide as Lean.trustCompiler, which trusts the whole compiler, so shortcuts cannot hide (Validating a Lean Proof).

What this guarantees, though, is only that the proof is correct. What a theorem statement means is a separate question, which I come back to below.

Why Lean and not Rocq? (a guess)

Most companies working on theorem proving with AI use Lean. OpenAI used Coq (now Rocq) for GamePad in 2018, then moved to Lean (a summary of the history), and its openai/math release in 2026 is formalized in Lean 4 and Mathlib (dataset description). The reasons have not been stated, so what follows is my guess.

  • Mathlib exists. It is one huge mathematics library, useful both as training data and as the vocabulary for stating problems. The CSLib paper also says Mathlib played a large part in training AI for mathematical reasoning (CSLib).
  • Ordinary mathematics can be written as it is. Mathlib’s culture uses the law of excluded middle and proof by contradiction freely. Paulson sees the Rocq community’s insistence on constructive proof as having held back its use in mathematics, leaving that ground to Lean (Why not just use Lean?).
  • It is also a programming language. Lean definitions can be compiled and run. The “Test” step of the CPython formalization works by actually running the Lean copy and comparing it with CPython.
  • It is easy to drive from an agent. elan and lake give a single toolchain, so setting up is not a puzzle.
  • Pass or fail is clear-cut. Whether the kernel accepts a proof serves directly as the reward in reinforcement learning.

Rocq has not stepped back, though. In October 2026 Anthropic and Theorem used Rocq to prove that 123 libraries do not crash, at the level of RISC-V machine code (coverage). Perhaps the split is Lean for mathematics and the meaning of languages, and Rocq for verification that builds on accumulated machine semantics.

Generated Lean still has to be read by people

The kernel guarantees only that “this statement was shown by this proof”, not that the statement says what you meant. The Lean reference too treats whether a proof is correct and what a theorem statement means as separate questions. It also counts unreviewed AI-generated proofs among those to be handled with the same caution as malicious ones (Validating a Lean Proof).

Three pitfalls are common:

  • The premises cannot be satisfied, so the theorem is vacuously true
  • A definition is mistranscribed: the proof is correct, but about a definition that differs from the implementation
  • The proposition proved is weaker than the property you wanted

The “Check” step of Formalizing CPython is exactly the step that looks for these. In my CRuby experiment too, AI proposed the propositions and premises, and I confirmed them before moving on to the proofs. NonVacuity.lean holds concrete inputs that satisfy each theorem’s premises, showing that none is vacuous.

In other words, what people need to read is not the proofs but the definitions and theorem statements. Proofs are the kernel’s job; statements are ours. That is also why Taiyak folds long proofs from the start: I want the statements read first.

Taiyak’s translations are LLM output too, so they can be wrong. That is why they sit next to the Lean, where they can be checked against it, instead of replacing it.

How to use it

  1. Open the web version, enter a GitHub URL and press “Open”
  2. In “Settings”, enter an API key from Anthropic, OpenAI, Google or OpenRouter. For a private repository, also enter a GitHub token: a link in Settings opens GitHub’s token page with read-only access and a 30-day expiry already filled in
  3. Press “Translate this file”, or “Translate” on a single block

To help with reading:

  • Click Lean code for hints. For an identifier you get a summary of its definition and a jump to it; symbols such as ∀ and tactics such as omega and rcases get a short explanation
  • “Ask Claude about this spot” lets you ask a question with the declaration as context
  • When translating, Taiyak follows imports and sends the definitions the code refers to as context
  • Translations are kept in the browser. Export them as JSON and commit them to the repository’s .taiyak/, and anyone can read the translations without an API key
  • The URL in the address bar is a share link, and pressing a declaration’s line numbers copies a link to that declaration

Design decisions

Limiting where data can go, with headers

Taiyak handles API keys and sometimes private source code. Rather than explaining in code that it sends nothing anywhere else, I thought it would be easier to verify if the browser enforced it.

The web version returns a Content-Security-Policy from Cloudflare Pages’ _headers. Its connect-src lists only the GitHub API and four LLM APIs, so if the page tries to send anything anywhere else, the browser refuses.

connect-src https://api.github.com https://api.anthropic.com https://api.openai.com https://generativelanguage.googleapis.com https://openrouter.ai

The inline script is pinned by its sha256, with default-src 'none', form-action 'none', frame-ancestors 'none' and Referrer-Policy: no-referrer as well. The build output is committed to the repository and CI checks that it matches the source, so what is served can be read on GitHub as it is.

Masking without a password field

If the API key field were type="password", the password managers in Chrome and Safari would react and offer to save it as this site’s login password. An API key is not for logging in to this site, so that gets in the way.

So the field is an ordinary text input, masked with the CSS -webkit-text-security: disc, and an eye button toggles visibility. Only browsers without that CSS fall back to type="password". The field also carries attributes that tell 1Password, Bitwarden and the like to ignore it. Keys are used only inside the browser, and saved to localStorage only if you choose “Remember in this browser”.

Honestly, though, I am torn. API keys are exactly what I would like to keep in a password manager: not as a login password, but with the manager acting as a vault for tokens. If you know a good way to do that, please tell me.

Matching the interface language to the translation

Translations in Korean with the interface in Japanese would look awkward. So there is a single language choice, and the translations, the interface, the glossary and Claude’s explanations all follow it. Interface text is built in for ten languages (Japanese, English, Simplified and Traditional Chinese, Korean, French, German, Spanish, Portuguese and Russian); other languages get the English interface. A setting can also fix the interface language on its own.

Right-to-left languages such as Arabic and Hebrew are untested. Typing the language name should produce translations, but I cannot read them, so I cannot judge whether the display is right. Anyone willing to try is welcome.

What I wish were different

Translating needs an LLM API key. Taiyak cannot use the allowance of flat-rate plans such as Claude Pro or ChatGPT Plus today. Translating on the plan you already pay for would be the easiest, but that is not how it works.

There are a few workarounds:

  • Publish dist/claude-artifact.html as a claude.ai artifact, and each viewer translates with their own Claude account, with no API key. Pages inside claude.ai cannot reach GitHub, though, so you download the repository as a ZIP and load that
  • OpenRouter’s free models are available
  • Google’s Gemini API has a free tier. It is crowded, though: a translation of a single block took about three minutes to arrive. Gemini 4 Argon is not available on the free tier
  • Commit the translations to .taiyak/, and readers need no key at all

As for ChatGPT, Sign in with ChatGPT arrived in September, letting Plus and Pro allowances be brought into outside tools without an API key. It covers products of participating partners, 16 at launch (itbrief). I found no form that a personal site can freely build in, so I am watching it as a future option.

I also tried supporting Gemini Nano, built into Chrome (the Prompt API). It needs no key and the code never leaves the device, which is appealing. But it is a small model, and translating Lean into mathematics seems too much for it: its translations fall short of the API models, and long declarations may not fit its input limit.

Closing

Lean written by AI is going to grow fast. With the kernel checking the proofs, what is left for people is reading whether “this statement is really the right one”. I hope Taiyak becomes a way in for people new to Lean to join that work.

References

]]>