Lua Hooks Guide

spyweb lets you customize every stage of the scraping pipeline with Lua scripts (Luau dialect). Place a hooks.lua alongside your job's config.toml and define any combination of 6 hook functions.

Key Rules

• Return the value (modified or not) to continue the pipeline
• Return nil or false to drop/skip (behavior varies per hook)
• Lua globals persist across runs — useful for pagination counters
• Hot-reload creates a fresh Lua VM and resets all state
• Hook errors are logged and skipped — they never crash the job

Every scrape run flows through these stages in order:

Each job with Lua hooks lives in its own directory under jobs/:

jobs/
└── my-scraper/
    ├── config.toml    # job configuration (required)
    └── hooks.lua       # Lua hooks (optional)

spyweb auto-discovers hooks.lua when loading the job. No config changes needed — just create the file.

Called before every HTTP request. Modify the URL, add headers, or skip entirely.

-- Basic: add auth header
function before_fetch(request)
    request.headers["Authorization"] = "Bearer my-token"
    return request
end

Called after HTTP fetch. Inspect status, modify body, or skip extraction.

function after_fetch(response)
    if response.status ~= 200 then
        print("Bad status: " .. response.status)
        return nil  -- skip extraction
    end
    -- Strip unwanted content from the body
    response.body = response.body:gsub("<script.-</script>", "")
    return response
end

Batch hook — receives all extracted items at once. Good for cross-item logic, dedup, or sorting.

No short-circuit: returning nil or empty table just means no items, not a pipeline abort.

function after_extract(items)
    -- Remove duplicate titles
    local seen = {}
    local unique = {}
    for _, item in ipairs(items) do
        local title = item.fields.title or ""
        if not seen[title] and title ~= "" then
            seen[title] = true
            table.insert(unique, item)
        end
    end
    return unique
end

Called once per item. Return the item to keep it, or nil to drop it before it reaches the database.

function filter_item(item)
    local title = (item.fields.title or ""):lower()

    -- Drop unwanted items
    if title == "" or title:find("sponsored") then
        return nil
    end

    -- Mutate fields before storage
    item.fields.title = title:sub(1,1):upper() .. title:sub(2)

    return item
end

Last chance to drop items before dedup + insert. Return nil to skip storing and notifying.

function before_store(items)
    -- Only store during business hours
    local hour = os.date("*t").hour
    if hour < 9 or hour > 17 then
        return nil  -- don't store, don't notify
    end
    return items
end

Reshape or silence notifications. Items are already stored at this point — dropping them here only affects what gets notified/webhoked.

function before_notify(items)
    -- Silence notifications if fewer than 3 new items
    if #items < 3 then return nil end

    -- Only notify for the first 5 items
    local capped = {}
    for i = 1, math.min(#items, 5) do
        capped[i] = items[i]
    end
    return capped
end

spyweb injects several global helper functions into the Lua environment.

http_get(url, [headers])

Performs a synchronous HTTP GET request. The headers argument is an optional Lua table containing key-value pairs for request headers.

local html = http_get("https://api.example.com", {
    ["Authorization"] = "Bearer token123",
    ["X-Custom-Header"] = "my-value"
})

http_post(url, body, [headers])

Performs a synchronous HTTP POST request. By default, it sets the Content-Type to application/x-www-form-urlencoded unless overridden in the headers argument.

local response = http_post("https://api.example.com", '{"foo":"bar"}', {
    ["Content-Type"] = "application/json",
    ["Accept"] = "application/json"
})

Job Config (config.toml)

Field Syntax

# Shorthand: "name:selector" or "name:selector@attribute"
fields = ["title:h2", "link:a@href", "desc:.summary"]

# Full form
fields = [
  { name = "title", selector = "h2", att = "text" },
  { name = "link", selector = "a", att = "href" },
]