I Tested 5 AI Models on 10 Real Bugs

Generating code is easy. Finding bugs is hard. I gave each model 10 buggy code snippets and measured what they found.

The Results

Test cases: React memory leaks, race conditions, closure bugs, SQL injection, off-by-one errors, type coercion issues, async bugs, infinite loops, security vulnerabilities, and performance problems.

Bug 1: React Memory Leak

// Bug: Memory leak in React component
import { useEffect, useState } from 'react';

function Dashboard() {
  const [data, setData] = useState([]);

  useEffect(() => {
    const interval = setInterval(() => {
      fetch('/api/data')
        .then(res => res.json())
        .then(setData);
    }, 1000);
  }, []);

  return <div>{data.length} items</div>;
}

// Claude's analysis:
// 1. Interval never cleared - memory leak
// 2. No cleanup in useEffect return
// 3. setData called after unmount - warning in console
// 4. Missing dependency array for fetch
//
// Fix:
useEffect(() => {
  const interval = setInterval(() => {
    fetch('/api/data')
      .then(res => res.json())
      .then(setData);
  }, 1000);

  return () => clearInterval(interval);  // Cleanup
}, []);

Bug 2: Race Condition

// Bug: Race condition in async function
async function loadUserData(userId) {
  const user = await fetchUser(userId);
  const posts = await fetchPosts(userId);
  const comments = await fetchComments(userId);

  return { user, posts, comments };
}

// Claude's analysis:
// Sequential awaits - if userId changes between calls,
// you might get user A's profile with user B's posts.
//
// Fix 1: Use Promise.all for parallel fetching
async function loadUserData(userId) {
  const [user, posts, comments] = await Promise.all([
    fetchUser(userId),
    fetchPosts(userId),
    fetchComments(userId)
  ]);

  return { user, posts, comments };
}

// Fix 2: Add abort controller for cancellation
async function loadUserData(userId, signal) {
  const [user, posts, comments] = await Promise.all([
    fetchUser(userId, signal),
    fetchPosts(userId, signal),
    fetchComments(userId, signal)
  ]);

  return { user, posts, comments };
}

This is subtle. The code works most of the time, but if userId changes during the sequential awaits, you get mismatched data.

Bug 3: Stale Closure

// Bug: Stale closure in event handlers
function Counter() {
  const [count, setCount] = useState(0);

  useEffect(() => {
    const handler = () => {
      console.log('Current count:', count);
    };

    document.addEventListener('click', handler);
    return () => document.removeEventListener('click', handler);
  }, []);

  return <button onClick={() => setCount(count + 1)}>
    Count: {count}
  </button>;
}

// Claude's analysis:
// The click handler captures 'count' from initial render (0).
// Even after clicking the button 10 times, clicking document
// will log "Current count: 0"
//
// Fix 1: Add count to dependencies
useEffect(() => {
  const handler = () => console.log('Current count:', count);
  document.addEventListener('click', handler);
  return () => document.removeEventListener('click', handler);
}, [count]);  // Re-register handler when count changes

// Fix 2: Use ref for latest value
const countRef = useRef(count);
countRef.current = count;

useEffect(() => {
  const handler = () => console.log('Current count:', countRef.current);
  document.addEventListener('click', handler);
  return () => document.removeEventListener('click', handler);
}, []);

This is a classic React bug. The event handler captures the initial value of count and never updates.

Bug 4: SQL Injection

// Bug: SQL injection vulnerability
app.post('/login', (req, res) => {
  const { username, password } = req.body;

  const query = `
    SELECT * FROM users
    WHERE username = '${username}'
    AND password = '${password}'
  `;

  db.query(query, (err, results) => {
    if (results.length > 0) {
      res.json({ success: true });
    } else {
      res.json({ success: false });
    }
  });
});

// Input: username = "admin' --"
// Query becomes: SELECT * FROM users WHERE username = 'admin' --' AND password = '...'
// The -- comments out the password check!

// Claude's fix:
app.post('/login', async (req, res) => {
  const { username, password } = req.body;

  // Use parameterized queries
  const query = 'SELECT * FROM users WHERE username = ? AND password = ?';

  try {
    const results = await db.query(query, [username, password]);

    if (results.length > 0) {
      // Also: never store plaintext passwords!
      // Use bcrypt.compare(password, results[0].hashedPassword)
      res.json({ success: true });
    } else {
      res.json({ success: false });
    }
  } catch (err) {
    console.error(err);
    res.status(500).json({ error: 'Server error' });
  }
});

A security vulnerability. String interpolation in SQL queries allows attackers to inject malicious code.

All models caught the SQL injection. This is encouraging - security vulnerabilities are easier to detect than logic bugs.

The Complete Bug List

Why Claude Wins

Claude doesn't just find bugs. It explains why they're bugs, how they happen in production, and provides multiple fixes with trade-offs.

Only Claude consistently thinks about real-world impact. The others find syntax errors and obvious bugs, but miss performance and architectural issues.

My Recommendation

Use Claude 3.5 Sonnet for debugging.

This isn't close. Claude found all 10 bugs. GPT-4o found 7. The gap is larger in debugging than code generation.

For trivial bugs (syntax errors, typos), any model works. For subtle issues (race conditions, memory leaks, security), only Claude is reliable.

When to Use Each

The False Positive Problem

GPT-4o and Gemini 2.0 sometimes flag correct code as buggy. Claude had zero false positives in my tests.

const result = arr.reduce((acc, val) => acc + val, 0);

Bottom Line

If you have a bug you can't figure out, give the code to Claude 3.5 Sonnet. It finds issues that other models miss.

The explanations are excellent. Claude doesn't just say "add cleanup here" - it explains why the bug happens, what triggers it in production, and how to prevent similar bugs.

For learning to write better code, debugging with Claude is more valuable than just getting fixes from Stack Overflow.

More

• Best AI for Python code generation
• Best AI for JavaScript code generation
• Code generation benchmarks