How to test a vibe-coded app before you ship it

Why vibe-coded apps need a different safety net

Vibe coding, a term Andrej Karpathy coined in February 2025, means building software by describing what you want to an AI and accepting the result without reading every line. It is a genuinely fast way to get from idea to working app. The catch is that nothing in the loop remembers what “working” meant last week.

Every new prompt can change code you did not ask the AI to touch. A quick click-through shows the new feature works; it does not show that the old features still do, or that one user cannot see another user’s data. Those are the failures that reach production.

What usually breaks

• Access control. Pages and API routes that check you are signed in, but not that the record you asked for is yours.
• Data leaking between accounts. A query that forgets to filter by user or team, or a database table with row-level security switched off.
• Secrets in the browser. API keys that end up in client-side code because the AI put a server call in a component.
• Money and limits. Free-plan limits, trial expiry, and webhook handling for payments, which are easy to get almost right.
• Silent regressions. A feature that worked last week and stopped after an unrelated prompt changed shared code.

The one-afternoon testing plan

1. Write down your app’s promises

In plain English, list the five to ten things that would be a disaster if they broke. For a typical SaaS app:

P-1  Only signed-in users can see the dashboard.
P-2  A user can only see and edit their own projects.
P-3  Free-plan users can create at most 3 projects.
P-4  Checkout upgrades the account only after payment succeeds.
P-5  Deleting an account deletes that user's data and nobody else's.
P-6  No API key or secret appears in the browser.

This list is the most valuable artifact in the whole process. It is the definition of “working” that does not change just because the code did.

2. Write one end-to-end test per promise

End-to-end tests drive a real browser through your app, so they test what users actually experience rather than how the AI happened to structure the code. Playwright is a good default. You can ask your AI to write these tests; the next step is how you check them.

import { test, expect } from "@playwright/test";

// P-2: A user can only see and edit their own projects.
test("a user cannot open another user's project", async ({ page }) => {
  await page.goto("/sign-in");
  await page.getByLabel("Email").fill("alice@example.com");
  await page.getByLabel("Password").fill(process.env.ALICE_PASSWORD ?? "");
  await page.getByRole("button", { name: "Sign in" }).click();

  // A project ID that belongs to Bob in your test seed data.
  const response = await page.goto("/projects/proj_bob_001");
  expect(response?.status()).toBe(404);
});

3. Break each feature on purpose

A test the AI wrote might pass for the wrong reason. For each promise, temporarily break the feature, for example by commenting out the ownership check, and run the test. If it still passes, the test is not protecting anything. This is a manual form of mutation testing, and it is the step most people skip.

4. Check the security basics

• Search the built client bundle for your API key prefixes.
• If you use Supabase or another hosted database, confirm row-level security is on.
• Every server route that returns data checks who is asking, not just that someone is.
• Payment webhooks verify the provider’s signature.

5. Run the tests on every change

Tests nobody runs protect nothing. A GitHub Actions workflow runs them on every push and pull request, including the ones your AI makes:

name: tests
on: [push, pull_request]
jobs:
  e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci
      - run: npx playwright install --with-deps
      - run: npx playwright test

Configure Playwright’s webServer option so it starts your app before the tests run.

6. Stop the AI from quietly changing the rules

The last gap: when a test fails, an AI agent will often “fix” it by changing the test. Change promise P-3 from 3 projects to 10 in both the code and the test, and everything stays green. See why AI agents change tests to make them pass for the patterns and the fixes.

Related guides