An upload is a request to enter a trust boundary
A document portal often begins with a signed upload URL and a queue job. That is sufficient for moving bytes. It is insufficient for deciding which bytes may become searchable, which parser may open them, and who may retrieve the extracted results.
Use an explicit ingestion state machine: requested, uploaded, quarantined, inspected, parsed, validated, published, or rejected. Every transition should name the exact object version, processing version, and authority under which it occurred. A display filename is not an object identity.
The architecture here is a proposed implementation pattern. It separates file safety from content trust: a document can be free of known malware and still contain false claims, confidential material, or instructions that an AI system must not obey.
Bind the upload session to an immutable object
Create the upload session after checking the user's permission, quota, allowed document types, and intended workspace. Generate the storage key on the server. Avoid using user-controlled path segments as authority, even when they look like workspace IDs.
S3 documents that a presigned URL can be reused until expiry and that uploading to an existing key replaces the object. A scan of one version therefore cannot authorize an arbitrary later upload to the same key. Record the accepted version identifier and checksum, or copy validated bytes into a separate immutable namespace before processing. S3 presigned URLs
When the client reports completion, verify the object server-side. Compare size and expected metadata, bind the exact version, and transition the session once. Duplicate completion requests should return the same ingestion identity.
Quarantine before parsing
OWASP recommends layered file checks rather than trusting an extension or client-provided content type. Its guidance includes limits, generated filenames, authorization, isolated storage, and content inspection. No individual check establishes that a file is harmless. OWASP file upload guidance
Run parsers in disposable workers with bounded CPU, memory, wall-clock time, output size, and temporary disk. Disable network access unless a specific document format genuinely requires it and the destination is controlled. Archive extraction also needs limits on nesting depth, file count, expanded bytes, and path traversal.
| Boundary | Record | Reject or defer when |
|---|---|---|
| Upload acceptance | Object version and byte count | Session expired or quota exceeded |
| Inspection | Scanner version and result | Unsafe or inspection unavailable |
| Parsing | Parser version and resource use | Timeout, malformed structure, or expansion limit |
| Validation | Page count and extraction quality | Empty or materially incomplete output |
| Publication | Manifest version and access policy | Authorization changed or source was deleted |
An unavailable scanner should create a visible pending state, not quietly fall through to publication. The operational question is how long pending inspection may last and who receives the alert.
Preserve provenance through every transformation
Each extracted section should retain a source object version, page or location reference, parser version, and transformation ID. OCR text, normalized text, and embeddings are different derived artifacts. A checksum of the original PDF does not prove which parser produced the indexed text.
Use a processing key such as document version plus parser version plus indexing configuration. This makes retries reusable while allowing a deliberate reprocessing run after a parser upgrade. Keep the old searchable version available until the replacement passes validation, if the document's access policy still permits it.
Do not deduplicate across customers merely because file hashes match. A global existence check can expose whether another customer uploaded a particular document. Keep deduplication scoped to an authorized context or design a separate privacy-preserving process.
Publish a manifest, not a stream of partial results
Write derived objects and index records into an unpublished generation. Validate counts and references, then atomically change the document's active manifest. Readers resolve through that manifest and never see half an extraction.
Before activation, check that the document is still present and that its authorization version has not changed. If deletion or revocation occurred during processing, mark the generation abandoned and schedule cleanup. A worker's permission at job creation is not permanent permission to publish later.
The manifest should let operators answer which source, parser, extraction, and index generation produced a result. Keep status explanations understandable to the uploader: “inspection pending” and “could not extract readable text” are more useful than an internal worker exception.
Prove that failure cannot leak partial content
Test an upload overwritten after scanning, a parser crash after writing half the output, an expired upload session, and an access revocation during indexing. Submit a compressed archive with extreme expansion, a misleading extension, and a valid document with no extractable text.
Assert that unpublished generations never appear in search, that rejected files have bounded retention, and that retries do not create multiple active manifests. Measure publication delay separately from raw upload latency. The quality of this pipeline is determined by trustworthy searchable evidence, not the number of files accepted per second.