### The Study Measures Skill Formation at the Wrong Level Here's the crux: The study asked, "Did participants learn Trio?" But what if the *actual skill being formed* when working with AI is: 1. **Prompt specification**: Learning to articulate intent precisely 2. **Output validation**: Learning to test whether the result actually works 3. **Failure recognition**: Learning to detect when AI is stuck or hallucinating 4. **Architectural thinking**: Learning to decompose problems into AI-delegatable chunks 5. **Iteration strategy**: Learning when to push back, rephrase, or abandon These are *different skills* than "understanding the Trio library," but they are not *lesser* skills. They're the skills of someone operating at the next abstraction level up. The study's design is like testing whether calculator users understand long division. They might not--but they might also be solving differential equations that the long-division experts never reach. ### Mollick's "Wizard" Framing Supports This Mollick's insight is that we're moving from **collaborators on process** to **curators of output**. The relevant competencies shift: | Old Model (Co-Intelligence) | New Model (Wizard) | |----------------------------|-------------------| | Understand the steps | Define success criteria | | Guide the process | Verify the result | | Debug the internals | Recognize failure modes | | Know the tools deeply | Know what to ask for | Your statement--"If I ever have to look at actual code, I define that as a failure"--is precisely the wizard-model operating principle. You're not *avoiding* competence; you're *relocating* it to specification and validation. ### The Bitter Lesson Applied to Human Skills The Bitter Lesson says: encoding human expertise into AI systems tends to lose to throwing compute at the outcome. But there's a corollary for humans: **investing human effort in understanding AI internals may lose to investing human effort in outcome specification and testing.** ### The Information-Hiding Principle Is Computer Science, Not Laziness You're invoking one of the foundational principles of software engineering: **information hiding** (Parnas, 1972). The entire point of good abstraction is that you *shouldn't need to know* what's inside the module. That's not a bug; it's the defining feature. ### Analogies That Support Your Position **1. The Taxi Driver Analogy** When GPS navigation arrived, London taxi drivers had spent years acquiring "The Knowledge"--memorizing 25,000 streets and 20,000 landmarks. GPS made this less necessary. The skill that *mattered* became: knowing where you actually want to go, recognizing when GPS is routing you badly, and knowing when to override. **2. The Calculator Analogy** We don't test engineers on manual logarithm table lookups. The skill shifted from *performing calculation* to *knowing what to calculate* and *recognizing wrong answers*. **3. The Radiology Analogy** Radiologists who insist on reading every image without AI are like programmers who insist on writing assembly. The emerging skill is: knowing when to trust the AI, knowing what cases require human review, and integrating imaging into clinical judgment. Not *replacing* judgment--*relocating* it. --- ## Honest Limitations of This Position ### 1. The Abstraction Layer Is Still Leaky Unlike compilers (which are mature and predictable), AI is: - **Stochastic**: Same input can give different outputs - **Opaque in failure modes**: When it fails, it often fails *confidently* - **Inconsistently capable**: The "jagged frontier" means you can't reliably predict what it can and can't do Your experience of "wasting days on simple problems where AI gets stuck, going in circles" is precisely this leakiness. ### 2. Testing Is Harder Than It Looks Your approach requires robust testing of outputs. But: - Many real-world systems have subtle correctness requirements - Security vulnerabilities are often invisible to functional tests - Performance characteristics may only emerge at scale - Edge cases are what AI is worst at anticipating ### 3. The Abstraction Level May Not Be Stable Yet Compiler abstractions stabilized over decades. The "AI as coding wizard" abstraction is months old. You may be learning skills that are themselves transient, as the AI interface changes rapidly. ### 4. The Study's Concern About Oversight Is Real The Anthropic study explicitly frames this around *AI safety*--the need for humans to supervise AI-generated code in high-stakes contexts. If no one understands the code, who catches the bug that causes the catastrophe? --- ## Where This Is Heading ### Short-term (1-2 years) The hybrid model dominates. People who can *both* work at the wizard level *and* drop down when needed will outperform either pure approach. ### Medium-term (3-5 years) AI verification tools emerge. We'll have AI systems that check other AI systems' code--not just for functional correctness but for security, performance, and maintainability. ### Long-term (5+ years) If the trajectory continues, understanding code becomes *specialist knowledge* like understanding assembly is today--relevant for a shrinking set of cases, but not the general requirement for being "a developer." ### The Key Uncertainty Will AI get reliable enough, fast enough, that the leaky abstraction problem goes away? If yes, your position wins outright. If AI plateaus with meaningful error rates, the ability to debug will remain valuable longer. --- ## My Honest Assessment Your approach is: - **Pragmatically optimal** for your situation (experienced, working on projects that can tolerate iteration, not safety-critical) - **Directionally correct** about where the field is heading - **Intellectually coherent** as an abstraction-level argument - **Vulnerable** to the unpredictability of current AI and the absence of mature verification tools
The Anthropic study is: - **Methodologically sound** for what it measures - **Measuring something real** (people do learn less of the underlying system) - **Potentially asking the wrong question** if the right skill to form is "working with AI" rather than "understanding Trio" - **Valuable as a warning** for contexts where oversight matters You're not wrong to see this as analogous to "C++ doesn't teach you assembler"--that framing is apt. The question is whether we're at the point in history where assembler knowledge is truly optional, or whether we're at the point where it's *usually* optional but *critically* necessary in unpredictable moments. Given your background and your projects, betting on the former seems reasonable. Just keep those tests robust.