AI Data Pipelines: ETL Automation with Agent Skills

ETL Workflow Walkthrough

A complete AI data pipeline follows four stages. Each maps to one or more of the skills above.

Stage 1: Extract

The Extract stage pulls raw data from one or more sources. For REST APIs, the agent calls the endpoint directly using its built-in HTTP capability. For JavaScript-rendered pages where no API exists, Puppeteer MCP navigates to the URL, waits for the data table to render, and extracts the HTML content. For local files already on disk, Filesystem MCP reads the CSV or JSON directly into the agent context.

Example prompt for the Extract stage: "Use Puppeteer MCP to open the analytics dashboard at dashboard.example.com/export, click the Export button, and save the downloaded CSV to data/raw/2026-04-09.csv."

Stage 2: Transform

The Transform stage cleans, validates, and reshapes the raw data. The CSV/JSON parser skill handles structural tasks: removing empty rows, coercing price strings to floats, deduplicating by ID, and mapping source column names to the target schema. The agent handles business logic that is too dynamic to encode in a static transformation: "Exclude any row where the category is discontinued and the inventory count is below 5."

Filesystem MCP writes the transformed output to a staging file so the Load step has a clean, validated source of truth and the raw file is preserved for reprocessing if needed.

Stage 3: Load

The Load stage inserts the transformed records into the destination database. Supabase MCP generates the SQL upsert statement based on the target table schema and executes it. Neon MCP is preferred when the Load step requires a schema migration alongside the data insert — you run the migration on a Neon branch, validate the results, and merge to main when satisfied.

Stage 4: Monitor

After loading, the agent runs post-load assertions: row count within expected range, no nulls in required columns, referential integrity for foreign keys. If any assertion fails, the agent writes a structured error report and can send an alert through a connected notification skill. Successful runs append a summary record to an audit log table, giving you a queryable history of every pipeline execution.

Step-by-Step Setup

The following instructions configure a minimal pipeline using Filesystem MCP, the CSV/JSON parser skill, and Supabase MCP. This is the fastest path to a working AI ETL pipeline.

Step 1: Add Skills to Your MCP Config

Open your AI assistant MCP configuration file. For Claude Code this is ~/.claude/settings.json; for Cursor it is .cursor/mcp.json in your project root.

{
  "mcpServers": {
    "filesystem": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-filesystem", "/path/to/data"]
    },
    "supabase": {
      "command": "npx",
      "args": ["-y", "@supabase/mcp-server-supabase", "--read-only=false"],
      "env": {
        "SUPABASE_URL": "https://your-project.supabase.co",
        "SUPABASE_SERVICE_ROLE_KEY": "your_service_role_key"
      }
    },
    "puppeteer": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-puppeteer"]
    }
  }
}

Step 2: Verify Each Connection

Restart your AI assistant and confirm each skill is live. Test with simple prompts:

• "List the files in /path/to/data" — verifies Filesystem MCP
• "Show me the tables in my Supabase project" — verifies Supabase MCP
• "Navigate to example.com and show me the page title" — verifies Puppeteer MCP

Step 3: Run Your First Pipeline

With all skills connected, run a complete pipeline in a single prompt:

"Read data/raw/products.csv using Filesystem MCP.
Clean the data: remove rows with empty names, convert
price strings to floats, deduplicate by product_id.
Write the cleaned data to data/staging/products-clean.csv.
Then upsert all rows into the products table in Supabase.
Report how many rows were inserted vs updated."

Step 4: Add Neon MCP for Schema-Safe Migrations

If your Load step requires schema changes, add Neon MCP and branch before running:

"neon": {
  "command": "npx",
  "args": ["-y", "@neondatabase/mcp-server-neon"],
  "env": {
    "NEON_API_KEY": "your_neon_api_key"
  }
}

Comparison Table

Use this table to understand which skill handles each stage of your pipeline and the key trade-offs between the two database destination options.

Frequently Asked Questions