When Code Is Cheap, Jane Street Teaches Review, Testing, and Struggle

Signals and Threads — Aaron Bauer with Ron — “Teaching in the Age of AI: Learning Goals and the Goals of Learning,” Jane Street, qN6OM1IzjIE

Jane Street used to start interns on a four-week project. Aaron Bauer’s precipitating observation is that the project is now one prompt. Producing a reasonable amount of good code used to be among the most important signals. It is less important now. The intern curriculum, the testing teach-in, even jump-to-definition inside OCaml all have to be redesigned around a cost structure Charlie Marsh named and Ron repeated: the cost of a plausible-looking PR has dropped nearly to zero, while the cost of verifying it is the same or worse.

Bauer is one of Jane Street’s few developer educators, a 50/50 split between writing code on the editors team and teaching other people how to write it. He and the firm are not neutral observers. They are rebuilding their own onboarding in public, on their own stack, and they do not claim to have settled answers. The useful part of the conversation is how specific the rebuild is: learning goals written as “students will be able to,” cognitive load split across OCaml then editor then version control, Claude Code’s learning mode used as a note-taking prosthetic, and a summer of intern pods that started too much with vibe-coding three things at once and had to be rewound to a high-quality test suite.

A retired teacher on a beach, then Foldit

Bauer’s origin story is almost too on-the-nose for a show about teaching programmers. Before senior year of high school he spent six weeks in New Mexico at the Summer Science Program: math, astronomy, physics, Python, telescope observations of an asteroid, then code to compute an orbit. Programming turned out to be the fun part. His small-town high school had no AP Computer Science. North Carolina’s online course program supplied a retired teacher who was, in fact, on a beach. The summer before graduate school he went back as a TA, wrote an exercise, gave a short lecture on binary search trees, and was hooked.

Undergraduate work included a Fortran simulation of impacts — asteroids or comets hitting a surface — descended from code used to simulate atomic explosions, because the physics is not that different. Automatic differentiation, he noted, is not an ML invention; early Fortran compilers could emit a derivative program, first derivatives only, and God help you if you differentiated the result. He loved programming anyway. A last-year graphics course made him want to render for the rest of his life. Rendering research had mostly moved to industry, so inertia carried him into a PhD that became game design, CS education, and a dissertation on how Foldit players solved protein-folding problems with geometric insight the computers of the time could not. Humans flipped whole regions; machines wiggled toward local energy minima. Players with no biochemistry background resolved structures that had sat in the lab for years, then started designing novel proteins. AlphaFold ate their lunch. He assumes AlphaFold trained on all known structures, in which case Foldit is in the data.

He wanted teaching as a primary job and landed at Carleton College in Minnesota: two quarters in person, then spring 2020, then four quarters online. He started from the text-only AP course that had worked for him — no video, read and do exercises — partly for students with bad internet. Immediate feedback: it does not work for all students. Accessible in one way, deeply inaccessible in others. He tried flipped classrooms, synchronous small groups, then settled on a mix of live instruction and interaction rather than a pure flip.

The lecture is a focusing device

Ron’s instinct, which Bauer said matches his experience, is that people underrate lectures. A live performance focuses attention. You are more likely to take notes. A recorded “best lecture ever” in a dorm room is not the same object. A solid hour of uninterrupted talking still becomes counterproductive; active learning — stop, pose a question, talk to neighbors, do a small problem — is the in-person complement, not the replacement.

Note-taking is the stranger cousin. There is, they both recalled, evidence that writing notes helps retention even if you never look at them again — even if you light them on fire. Writing may beat typing; Bauer’s classroom experience is less about keyboards than about a screen full of the internet sitting between student and presenter. He avoids teaching in computer labs. The moment there is a screen, even people who want to pay attention often cannot.

Ron had conned his kids into treating restaurant waiting as a Scheme puzzle: obfuscated names on paper, what does flub do. On paper they thought about evaluation. On a computer they played the game of making the machine do a thing. Beginners will perturb arcane instructions until the program runs. Paper forces a decision that it is correct before it can run. That withholding of affordances becomes the episode’s later AI problem in miniature.

After lockdown, Bauer kept lecture recordings. At Carleton, unlike some institutions, students still came; he tried to make presence useful. Classrooms had no video kit, so he carried a webcam on a tripod, posted the files, and could timestamp a question to the minute someone explained it. He also kept automated feedback: weekly autograded quizzes, fill-in-the-blanks, homework that returned test results immediately. Delayed TA grading arrives after the student has moved on and may never be read. A limited form of mastery learning — retry until competent — sat on top of that, but not the elementary-school version where the class fragments by topic. College students procrastinate. Deadlines are for their own good. Quizzes could be attempted any number of times; the best score counted; order of questions and answers shuffled so brute force was more annoying than understanding.

Ron asked whether LLMs could supply that instant feedback now. Bauer sees promise, especially on structured artifacts like code style rather than essays, sitting next to tests and linters. He also sees plenty of false positives. At minimum you frame the tool carefully, or you reduce the rate at which it leads people astray.

A job that does not fall from the sky

He moved to Brooklyn to live near his brother. Jane Street was, at that exact moment, hiring a weird educator-programmer combo. Teaching at Jane Street is old: nobody arrives knowing trading, and the tech stack is weird — language, environment, tools that do not match the outside world. The new idea was not to replace practitioners with professional teachers. It was to hire people who had actually taught for a material period of time, keep them as credible Jane Street engineers, and split the job 50/50: half programming, half curriculum, classes, and growing other people as teachers. A small caste of developer educators doing all the teaching would miss the point.

Ron had assumed great candidates would fall from the sky. They did not. Shift the posting three months and they would have missed Bauer entirely. The intersection is small: people with a love of the craft of teaching, usually in academia and not looking to leave, who also want to spend a large fraction of their time as software engineers. The firm has converted some internal people into the role and still wants more from outside. The appetite only grew. OCaml is changing; everyone, no matter tenure, needs to learn the new corners. There are new AI tools. They posted an OCaml educator with a strong PL background and had some luck. They are still looking for a machine-learning educator. Python is now a real language at the firm; Ron worries about people getting captured by tools, OCaml programmers versus Python programmers instead of engineers who can pick. Homegrown languages need teachers too. Bauer’s model is designers embedded on teams: an educator sitting on the AI-assistants team, educators with varied technical context so they are not all poked in the same corner.

The editor is still home base

Bauer works on the editors team in tools and compilers: VS Code, Emacs, Neovim, code indexing, the OCaml language server. An early project extended Jane Street’s feature-management software, Iron, so people could split work into separately reviewed pieces instead of doing it awkwardly by hand. At Jane Street a “feature” is sometimes a commit and sometimes a PR. Iron is, Ron said, way better than the outside world and way worse, depending which surface you look at. Stacked changes are a foundational primitive. Editing diffs used to be slower and more painful than Emacs Magit on git. Twenty-five years of an alternative-universe adventure produces a culture: how you review, how you give feedback, is shaped by the underlying system.

That is why there is an editors team. The text editor is the software engineer’s home base. Code review, version control, keyboard-navigable everything live there. You do not jump between CLI, browser, and editor. Feedback is specialized comments in a particular format inside the buffer — CRs — not a note on someone else’s PR in a web UI. You might skip the CR and just fix the thing. For people used to sanitary ownership of a pull request, someone reaching in feels unsanitary. For good and ill, it is a different modality, and it only works if the review surface is also an edit surface.

The hard parts are scale and marriage of systems. VS Code’s diff algorithm has serious performance problems on Jane Street’s giant configuration files and massive OCaml files; a hung large diff can lock the queue so you cannot view any others. Iron integration violates VS Code assumptions about what a buffer and URI are, so builtin search misbehaves. Merlin and the language server have performance questions; extra compiler information for better module lookup bloated an index by many gigabytes of RAM until it would not build on their boxes. Python notebooks are the main surface for trading and research, every outside notebook has significant flaws, and spinning up a replacement that is missing old functionality means maintaining several. Code in notebooks is hard to share into repo libraries. VS Code notebook performance has been bad enough that they are starting to build their own OCaml/web-UI notebook. Ron worries they overlean toward building their own things and also sees the opposite shear: years of funding work to replace weirdo Emacs special cases with upstream, because the code you do not write is the best code, other people find the bugs, and the Emacs maintainers understand Emacs. Extensions in OCaml across Neovim, Emacs, and VS Code complicate the boundary; ELisp versus their Emacs OCaml flavor is awkward for async, with deadlock risk. Emacs’s concurrency story is ancient; upstream garbage-collector work was not ready for the cut they are upgrading to. Marrying concurrency primitives is hard even when both sides are good — they have the same problem between Python and OCaml.

Jump-to-definition went up

Agentic programming has not changed the bet that the editor remains home base. Looking at diffs, reviewing, reading, types, definitions: still there. Typing code, for plenty of engineers including at Jane Street, has been replaced by typing prompts. Editor teams are left with genuine uncertainty about what is useful to build now versus a year from now. Bauer got interested in telemetry: how people actually use the OCaml language server. His hypothesis was that if LLMs produce good code, per-user jump-to-definition should fall. The telemetry showed, if anything, slightly more. Maybe people need to read and understand more code than before. Whether to make jump-to-definition faster depends on that number, not on theorizing in the air.

Ron’s air theory, stolen from Charlie Marsh, is still the frame: plausible PRs are nearly free and uncannily smooth. Tells you used to treat as signs of trouble are gone. The code is often still super broken. Understanding is at minimum different and in some ways harder. LLMs are powerful explainers. Traditional extractions — go-to-definition, inferred types, test results — are easy to underrate.

Bauer has been using Claude Code’s learning output style, which changes the system prompt so the model stops, leaves a TODO-human comment, and asks the human to fill that piece. Having to read what it had written so far in order to continue was both a more pleasant production experience and a more effective way to understand the result. Letting it write everything and reading at the end feels efficient. He has become skeptical that it is optimal. It is the note-taking point again: involvement in the production function is how understanding happens. Current tools tilt toward generate-then-accept. That may be wrong.

The analogy snapped into place on a slide for interns: code review is hard because writing builds understanding incrementally, while review hands you a finished result and you have to construct a mental model awkwardly from the outside. Working with LLMs is that, for code you ostensibly authored. Ron added the new opportunity set: tutor-like behavior is now cheap; a custom visualization of a diff can ship in the middle of review, which would have made no sense three years ago. Bauer wants precomputed segmentation of a feature into logical chunks with a pre-baked explanation on a keystroke — highlighting and asking a model still has friction and latency. Review as a proportion of time has shot up over something like the last eight months. Making reading better was always valuable. It is now the rate-limiting step.

The empty set of good exercises

The testing teach-in — property-based testing, deterministic control of time — runs into the same wall as every CS assignment: an LLM can write the unfamiliar library for you. What is understanding, and what is boilerplate? So far they have come down on the side that people should think through and implement the technique themselves at least once, even if later they mostly read LLM-authored applications of it. That first implementation is the input to being able to evaluate the later ones. They are still drawing the line on toil they used to require because people had to do it. Teaching people how the tools lead you astray is early, unsettled, and changing weekly. You can already see the failure mode: people vibe it up too much and stop thinking.

The overlap of problems that are easy enough that they are suitable for someone who is learning to do and hard enough that an LLM can’t just one-shot it is like close to the empty set now.

That is Ron, and Bauer said pretty much everyone in education is deeply worried. New Jane Street curriculum has hit the complementary failure: the task is large enough that students must put real work into LLM output, and then it does not fit the time box. Experimentation is the plan — some activities without models, some with, pairing still unknown — especially as incoming cohorts have not spent much time typing code themselves. OCaml boot camp is the concrete fight: if the LLM writes everything they learn nothing; if they never touch the tools they will live in daily, the course is a mismatch. Prompts that quiz understanding, skeletons that leave the hard pieces to the human, tutors that hint but will not write the code. Bauer did graduate research on an automated tutor that could walk you through an arbitrary example of an algorithm — infinitely patient, less good than a TA’s one hour, available at exam time. He thinks that becomes a large part of at least higher education.

Boot camp, then a course catalog

A new engineer’s first two weeks: OCaml boot camp, a cumulative client-server app, core libraries, mentor review, Jane Street style; then production boot camp on deploy, monitor, configure. Early versions of the OCaml camp are something like twenty years old. It now takes about a week, then a little more for production. Documentation of the firm still gets mixed reviews. The boot camps, especially recently, get positive ones. The design rule Bauer worked hard on: introduce concepts in a motivated context. The mistake is introducing a tool and going in search of a problem. You are working on this; you want to do that; you have no good way; wait, here is this.

First year is a series of teach-ins: testing; performant OCaml and its analysis tools; systems debugging; advanced functional programming; plus specialized ones — Bonsai for OCaml web UIs, which is also incremental computing; market data, which is also streaming protocols. The catalog grows. New hires and managers pick a course of study. Materials are meant to be self-studyable at the point of need. Teach-ins began as a reflection of a trading-side program that walked desk to desk. Ron’s early mental model was that the conceptual layer mattered more than the specific technology. That vision was less successful than he expected. They drifted toward practical tools that let people go do things.

Bauer’s explanation is a selection effect. In a college databases course you are also teaching people to be problem-solvers through ambiguity. Jane Street colleagues are already very good at that. Spending hours on a canned thinky puzzle feels unsatisfying next to the desk. Conceptual lessons — incremental computing — still matter, but they sneak in through concrete tools and walkthroughs, then complete themselves on real problems. You do not shape the class as a novel take-home proof.

Say “learning goals” in every room

The testing teach-in was Bauer’s first real project. Before he started, organizing it was going badly: good work in pieces, no one with enough time to land the whole. He asked Ron what the learning goals were. Ron heard English words that clearly meant something more specific.

Learning goals are a way of writing down how students will be different. Bauer wants them as “after completing this teach-in, students will be able to” plus active verbs. “Understand property-based testing” is a bad goal: vague, unmeasurable. “Write a property-based test for a nested record type” scopes the complexity. The list also says what you are not trying to accomplish, which is what makes student time expensive. Exercises follow: practice the terminal behaviors. Then background, lectures, and code samples are whatever is required for those behaviors and nothing whose writing is irrelevant to the goals. A lot of Bauer’s education work at Jane Street, he joked, has been walking into rooms and saying the words “learning goals.”

They mostly do not want to evaluate students. They want to evaluate the program: at the end, can people do the thing? That is how you evaluate yourself as an educator. Ron now hears two layers — the specific write-up that structures a course, and the frame of mind: how are the students different, why did you show up to work. He uses it in hiring. Some candidates have thoughtful answers. Some taught a course because they liked the material. Charisma and performance still produce an enormous amount of good teaching. Structure helps most the people who have not done it before. It forces the change to be observable, not “I told them a thing and now they know it.”

The other piece he deliberately applied to OCaml boot camp is cognitive load. An older version asked people, from the jump, to learn OCaml, an unfamiliar editor, and an unfamiliar version-control and review system at once. He split them: OCaml syntax first in utop, then in-browser snippet boxes; then editor activities; then version control; only then the combination. Active learning is the other import. New intern material uses small-group discussions, design activities, practicing talking to stakeholders — not just conveying information.

The intern summer, rebuilt

If the four-week starter project is one prompt, the projects have to become more open-ended: design, gathering requirements, harder and more realistic. Do not dump people in the deep end with “you have LLMs, good luck.” Train them on what to watch for, what exists at Jane Street, how people use it. The balancing act is the empty-set problem again: large enough that the model does not emit a perfectly fine version on try one, not so large it takes a week to polish.

They used the rebuild to teach software engineering that used to get little formal time at the start of the internship. Testing a database-backed application is now curriculum. Testing was always important; it is more important now because models thrive on feedback and will not one-shot anything beyond the simple. There is still a budget: they could spend half the internship teaching, and they still need people on projects. New exercises: looking at design docs, judging good ones, discussing bad ones. Full-timers role-play stakeholders for the app; interns gather requirements. Interns traditionally almost never reviewed other people’s code. This summer they will, and the firm wants to see how well they do. Mid-internship: three additional days on code review, pre-made features adding functionality to the same app they tested and designed, leaving comments, then an answer key of comments someone might have left, group disagreement, then implementing their own features and reviewing each other so mentors can later send real features.

How has it gone? They are learning a lot. Scope is the recurring failure. If the message is you own this code, review it, do not stamp whatever the LLM produced, but you assign more code than they have time to polish, the lesson does not land. Too-large review features with too many bugs make it hard to spot the issues you actually want them attending to. Cognitive load again.

The part Bauer is excited to keep is the interactive form. OCaml boot camp is self-paced with a full-timer reviewing. The new curriculum puts interns in pods of four, with frequent small-group discussions — how are you using the tools, what is wrong with the tests it is spitting out — including full-time mentors. Next run: reduce scope, fix emphasis and pacing, accommodate people racing ahead and people going slowly, message expectations so they do not rush past the important parts to keep a schedule. The curriculum evolved significantly between summer batches. An early batch started with multitasking and vibe-coding; afterward that felt like the wrong tone. They rearranged to start with the testing-suite focus rather than “you can have an LLM do three things at once and it’s cool.” It is cool. It is the wrong opening.

By next summer they may have more considered workflows to prescribe. Right now they have failure modes to watch, not “this is the good way to prompt.” Small-group discussion was where people shared techniques that worked and ones that did not. The tools change fast enough that next year’s advice may need to be different. Two people using the same method may get different behavior. An exercise that involves an LLM introduces much more uncertainty into what will happen than anything they assigned in the past.

That is where they stopped: not with a doctrine for prompting, but with pods, tests, and the admission that the assignment itself is now a random variable.