AutoSpark: what I built this semester

Summary

Of the 66 pull requests merged into the repo, 28 are mine. The repo has 270 commits that are not merges, and 101 of them are mine. My first merged PR was October 14; the first commit in the repo is September 22.

Modules I worked on

Where my work sits in the repo, and what I did there. Teammates built the other detectors and several tabs; this table covers only my part.

Source code overview

The system as it runs today. People reach the app host two ways: a browser signs in with Google, and an MCP client sends a personal access token. Both land on the same dashboard process, which reads metrics from Spark History Servers and uses AWS for jobs, storage and the database.

1. Browser. Caddy terminates HTTPS and asks oauth2-proxy who the person is. Google confirms; the verified email goes to nginx's trusted listener and becomes the person's identity everywhere (approvals, audit rows, run tags).
2. MCP client. Caddy sends only the exact path /mcp past the Google redirect, to a listener that trusts no identity header. The gateway checks the token on every call: shape, lookup, owner still on the sign-in list, rate limit, then the tool's scope.
3. Run and prove. A job file goes to S3 and runs on EMR Serverless. Event logs return to S3, History Servers serve them over REST, detectors turn them into findings, and findings, suggestions, sandbox results and approvals live in Postgres.
4. Safety rails. The app's database login cannot delete rows or change the schema. Approvals are audited by a trigger, not by app code. The sandbox cluster cannot read the production buckets.

Issues I opened

52 issues, 20 closed. They fall into six groups, from the first scoping questions in week one to the MCP epic. Two MCP issues (#112 and #113) are implemented and merged but still open on GitHub.

Pull requests I opened

All 28 are merged. Sizes are additions and deletions in the PR. Dates are merge dates, so the order is by PR number.

Code merged by me

17,349 lines added and 689 removed across 314 file changes in 28 merged PRs. I merged them with merge commits, the convention in this repo. A deploy runs on every merge to main.

The tests are the largest category on purpose: the database tests run against a real Postgres, the MCP tests run over real HTTP, and for the security-sensitive code I broke it on purpose to check the tests fail (removing the commits, the scope check, the allowlist check, the token shape check).

Code walkthrough

Four pieces I would show, all from the MCP and sign-in work. Together they answer one question: how does a request prove who it is and what it may do?

def mint(conn, owner_id, name, scopes=("read",), ttl_days=None):
    ...
    token = PREFIX + secrets.token_urlsafe(_TOKEN_BYTES)        # 32 random bytes
    row = conn.execute(
        "INSERT INTO api_token (owner_id, name, token_hash, prefix, scopes, expires_at) "
        "VALUES (%s, %s, %s, %s, %s, now() + %s) RETURNING id",
        (owner_id, name, _hash(token), token[:8], list(scopes), timedelta(days=ttl_days))).fetchone()
    conn.commit()
    return token, row["id"]        # the token goes to the person once; only its SHA-256 is kept

def verify(conn, token):
    if not looks_like_token(token):                              # shape first: no database work for junk
        return None
    row = conn.execute("SELECT ... FROM api_token t JOIN app_user u ON u.id = t.owner_id "
                       "WHERE t.token_hash = %s AND t.revoked_at IS NULL AND t.expires_at > now()",
                       (_hash(token),)).fetchone()

def authenticate(headers):
    token = bearer_token(headers)
    if token is None:                       raise AuthError(401, ...)     # no header
    if not tokens.looks_like_token(token):  raise AuthError(401, ...)     # junk: refused before the database
    principal = tokens.verify(conn, token)                                # unknown, expired or revoked: 401
    if principal.email.lower() not in allowed:  raise AuthError(401, ...) # owner removed from the sign-in list
    wait = _limiter.hit(principal.token_id)
    if wait:                                raise AuthError(429, ...)     # 120 calls a minute per token
    return principal

def require_scope(tool, principal):
    need = TOOL_SCOPES.get(tool)
    if need is None:  raise ScopeError(...)       # a tool with no declared scope is refused

class McpGateway:
    async def __call__(self, scope, receive, send):
        ...
        principal = await anyio.to_thread.run_sync(
            auth.authenticate, scope["headers"], limiter=self._lookups)   # off the event loop, on its own pool of 4
        ...
        reset = auth.current_principal.set(principal)    # tools read the caller from here, never from arguments
        try:
            await app(scope, receive, send)
        finally:
            auth.current_principal.reset(reset)

@mcp path_regexp ^/mcp$                 # exactly /mcp (Caddy's plain path match ignores case)
handle @mcp {
    request_header -X-Auth-Request-Email  # a client must never choose who it is
    request_header -X-Autospark-Email
    rewrite * /mcp                         # whatever matched, upstream sees /mcp
    reverse_proxy 127.0.0.1:8080           # the listener that trusts no identity header
}
handle {
    route {
        request_header -X-Auth-Request-Email
        request_header -X-Autospark-Email
        forward_auth 127.0.0.1:4180 { uri /oauth2/auth  copy_headers X-Auth-Request-Email ... }
        reverse_proxy 127.0.0.1:8081       # the listener that trusts the verified email
    }
}

Documentation

I wrote 4,411 lines of documentation. The runbook and the quickstart are written to be followed by someone who was not in the room.

My 10-minute plan

A run-of-show so the ten minutes follow Mario's outline.