The SpyWeb Master Guide

SpyWeb is a zero-dependency web monitoring engine that lets you customize every stage of the pipeline with Lua scripts (Luau dialect). Place a hooks.lua alongside your job's config.toml and define any combination of 7 hook functions.

Key Rules

• Return the value (modified or not) to continue the pipeline
• Return nil or false to drop/skip (behavior varies per hook)
• Plain Lua globals persist across runs only while the job task stays alive
• store_* is job-scoped persistent storage and global_store_* is shared across jobs
• Hot-reload creates a fresh Lua VM and resets global Lua state, but stored values survive
• Hook errors are logged and skipped — they never crash the job
• Efficiency: Only define the hooks you actually need. SpyWeb pre-detects which functions exist at startup and skips the processing logic entirely for any that are missing.

Every scrape run flows through these stages in order:

Each job with Lua hooks lives in its own directory under jobs/:

jobs/
└── my-scraper/
    ├── config.toml    # job configuration (required)
    └── hooks.lua       # Lua hooks (optional)

spyweb auto-discovers hooks.lua when loading the job. No config changes needed — just create the file.

Called before every HTTP request. Modify the URL, add headers, or skip entirely.

-- Basic: add auth header
function before_fetch(request)
    request.headers["Authorization"] = "Bearer my-token"
    return request
end

Called after HTTP fetch. Inspect status, modify body, or skip extraction.

function after_fetch(response)
    if response.status ~= 200 then
        print("Bad status: " .. response.status)
        return nil  -- skip extraction
    end
    -- Strip unwanted content from the body
    response.body = response.body:gsub("<script.-</script>", "")
    return response
end

Batch hook — receives all extracted items at once. Good for cross-item logic, dedup, or sorting.

No short-circuit: returning nil or empty table just means no items, not a pipeline abort.

function after_extract(items)
    -- Remove duplicate titles
    local seen = {}
    local unique = {}
    for _, item in ipairs(items) do
        local title = item.fields.title or ""
        if not seen[title] and title ~= "" then
            seen[title] = true
            table.insert(unique, item)
        end
    end
    return unique
end

Called once per item. Return the item to keep it, or nil to drop it before it reaches the database.

function filter_item(item)
    local title = (item.fields.title or ""):lower()

    -- Drop unwanted items
    if title == "" or title:find("sponsored") then
        return nil
    end

    -- Mutate fields before storage
    item.fields.title = title:sub(1,1):upper() .. title:sub(2)

    return item
end

Last chance to drop items before dedup + insert. Return nil to skip storing and notifying.

function before_store(items)
    -- Only store during business hours
    local hour = os.date("*t").hour
    if hour < 9 or hour > 17 then
        return nil  -- don't store, don't notify
    end
    return items
end

Reshape or silence notifications. Items are already stored at this point — dropping them here only affects what gets notified/webhoked.

function before_notify(items)
    -- Silence notifications if fewer than 3 new items
    if #items < 3 then return nil end

    -- Only notify for the first 5 items
    local capped = {}
    for i = 1, math.min(#items, 5) do
        capped[i] = items[i]
    end
    return capped
end

Reshape or silence webhook POSTs. Receives the full JSON payload (job_name, item_count, items).

function before_webhook(payload)
    -- Add a secret token to the payload
    payload.secret = "my-secure-token"
    
    -- Filter items in the webhook but keep them in the notification
    local filtered = {}
    for _, item in ipairs(payload.items) do
        if tonumber(item.fields.price) < 100 then
            table.insert(filtered, item)
        end
    end
    payload.items = filtered
    payload.item_count = #filtered
    
    if #filtered == 0 then return nil end -- skip webhook
    return payload
end

spyweb injects several global helper functions into the Lua environment for networking, logging, and storage.

Storage & Persistence

SpyWeb supports state management through both Runtime Memory (transient) and an Embedded Database (persistent). Memory is fast but resets on reload; the database survives restarts.

local c = tonumber(store_get("count") or "0")
store_set("count", tostring(c + 1))

-- WRONG APPROACH (race condition prone)
local v = tonumber(global_store_get("visits") or "0")
global_store_set("visits", tostring(v + 1))

-- CORRECT APPROACH (race condition safe)
global_store_incr("visits", 0, 1)

-- This resets to 1 if you edit hooks.lua or restart SpyWeb
visit_count = (visit_count or 0) + 1
log("Session visit: " .. visit_count)

http_get(url, [headers])

Performs a synchronous HTTP GET request. The headers argument is an optional Lua table containing key-value pairs for request headers.

local html = http_get("https://api.example.com", {
    ["Authorization"] = "Bearer token123",
    ["X-Custom-Header"] = "my-value"
})

http_post(url, body, [headers])

Performs a synchronous HTTP POST request. By default, it sets the Content-Type to application/x-www-form-urlencoded unless overridden in the headers argument.

local response = http_post("https://api.example.com", '{"foo":"bar"}', {
    ["Content-Type"] = "application/json",
    ["Accept"] = "application/json"
})

log(message)

Appends a message to hook.log inside the job's directory. This is the recommended way to debug hooks without using external libraries.

function after_extract(items)
    log("Extracted " .. #items .. " items from " .. request.url)
    return items
end

Jobs in SpyWeb are configured via TOML. You can use a single jobs.toml or a modular jobs/my-job/config.toml structure.

Job Config

# Shorthand
fields = ["title:h2", "link:a@href"]

# Full form
fields = [
  { name = "title", selector = "h2", att = "text" }
]

Deduplication (hash_fields)

By default, SpyWeb hashes all extracted fields. Use hash_fields to ignore volatile data like "time ago" or "view count":

# Only use the link to determine if an item is new
hash_fields = ["link"]

Advanced Features

[proxy]
enabled = true
rotate = "RoundRobin" # or "Sticky", "Random"
urls = ["socks5://p1:1080", "http://p2:8080"]

[webhook]
enabled = true
url = "https://hooks.example.com/spyweb"
headers = { "Authorization" = "Bearer token" }

[notification]
enabled = true
title = "{item_count} new items in {job_name}"
body = "Title: {title}\nLink: {link}"

SpyWeb runs a built-in web server on 127.0.0.1:7979 which serves both the dashboard UI and a JSON API for querying your scraped records.