Skip to content

ADR-012: Bulk PDF upload — hybrid pipeline with on-prem processing agent

Status

In-Review. Supersedes the unmerged branch document ADR-008-bulk-pdf-s3-only.md from PR #2373 (which collided with main's ADR-008 numbering and was contradicted by that branch's own implementation).

Context

Bulk PDF upload (epic #2223) lets project admins deliver full-text PDFs themselves instead of emailing them to the SyRF team. Final PDF storage is the Edinburgh IIS web server (ecrf1.clinicaltrials.ed.ac.uk/camarades/, ~700 GB of PDFs already served from there), reachable only from inside the university network — nothing in AWS or GKE can write to it.

The feature's storage architecture flip-flopped across 2026: hybrid (S3 staging + on-prem final, discussion #2094, Oct 2025) → S3-only (branch ADR-008, Jan 2026) → hybrid again (decision C-2, Mar 2026, recorded only as a Teams confirmation). The contradiction was never resolved in a merged decision record. This ADR settles it.

Decision

  1. Final storage stays on the Edinburgh IIS share. S3 (syrfapp-uploads*) is staging only; uploaded ZIPs expire after 7 days via a lifecycle rule.
  2. One processing locus, on-prem. A new .NET worker (PDF agent) performs download, ClamAV scanning (clamd companion container), validation, extraction, conflict handling, and the final filesystem write. The same image runs in-cluster for staging/previews (writing a PVC) and as a docker application on the arrnc-api host for production (writing a CIFS mount of the IIS share), managed via server-config.
  3. The Lambda stays thin. The existing s3-notifier gains a BulkPdfUpload kind: it validates metadata, generates a 12 h presigned GET, and sends one MassTransit command to the agent's queue. No scanning or extraction in Lambda.
  4. Domain authority stays in the PM service. The agent reports per-file outcomes; PM performs study matching, marks studies, computes counts, and generates the CSV report in a single-writer consumer.
  5. The browser builds the ZIP. Users select a folder of PDFs; the client validates (PDFs only), previews matches against the search, zips in a web worker (STORE), and PUTs directly to S3 via the existing SigV4 signature pattern. The ZIP is a transport detail.
  6. No cloud credentials or inbound connections on university hosts. The agent pulls S3 via presigned URLs and AMQP outbound only.

Consequences

Positive

  • Zero broken links: a study's PDF link activates only after the file is verifiably on disk.
  • One place to debug processing; no cross-cloud two-hop pipeline (the #2373 shape).
  • No new AWS infrastructure beyond a chart-declared lifecycle rule; no clamd-in-cloud.
  • Preview/staging environments run the identical pipeline in-cluster, so the feature is E2E-testable per PR.
  • ClamAV signatures stay current via freshclam without image rebuilds.

Negative / accepted

  • Production depends on a single manually-applied host (arrnc-api) and a CIFS mount; mitigated by fail-closed sentinel checks, durable queueing while the agent is down, and server-config's recovery/verify roles.
  • Unscanned bytes touch the agent's isolated container before scanning: the ZIP is downloaded and extracted into container-scratch space, and every extracted file is scanned there before anything is copied to the share or any served location. The security boundary is "nothing unscanned ever reaches final storage", not "nothing is extracted pre-scan" — scratch is private to the agent container (same trust boundary as scanning inside a Lambda).
  • AMQP to rabbitmq.camarades.net:5672 is currently plaintext over the internet, and this applies to both out-of-cluster clients: the existing production Lambda and the new PDF agent on arrnc-api. A separate hardening PR adds AMQPS; both flip their URIs when it lands.
  • The bulk layout (projects/{p}/searches/{s}/pdfs/…) coexists with the legacy flat layout; link generation branches on Study.BulkPdfDeliveredAt.

Alternatives considered

  • S3-only storage, no agent (branch ADR-008; revisited 2026-08-11 as "GKE worker + S3 final"): cleanest platform-native end-state, but changes the PDF serving model and contradicts the standing storage decision; rejected consciously.
  • Keep #2373's two-hop pipeline (scan-in-Lambda → safe bucket → agent): requires clamd reachable from AWS (new always-on infra), doubles S3 I/O, and splits processing across a cloud/on-prem boundary; dominated once an on-prem writer exists anyway.
  • Direct upload to an ingest service on arrnc-api (no S3): loses durable staging, presigned upload UX, and per-PR preview isolation; couples availability to one host.

References

Design: docs/superpowers/plans/2026-08-11-bulk-pdf-upload-v2-design.md · Plan: docs/superpowers/plans/2026-08-11-bulk-pdf-upload-v2-plan.md · Epic #2223 · discussions #2093/#2094/#2095 · superseded attempt PR #2373 · ADR-010 (s3-notifier IAM via ACK) and its 2026-08-11 audit.