Skip to main content

​
Data Lifecycle

Remem supports document lifecycle management plus tenant-level DSAR workflows.

​
Document lifecycle

​
1) Ingest

POST /v1/documents/ingest queues async processing and stores encrypted content + metadata.

​
2) Update (new version)

POST /v1/documents/{document_id}/update creates a new versioned document state.

​
3) Soft delete

DELETE /v1/documents/{document_id} marks the document deleted and queues cleanup. Current behavior in code:
  • vectors/files/cache invalidation is queued
  • hard delete is scheduled with a grace delay
  • default hard-delete grace window is 7 days
There is no public hard_delete=true flag on the document delete endpoint. Document hard delete is lifecycle-driven.

​
DSAR export (tenant scope)

GET /v1/dsar/export
Returns a ZIP stream with tenant metadata, documents, chunks, and optional raw files.

​
DSAR delete (tenant scope)

POST /v1/dsar/delete
confirm=true is required. Catalog-driven erasure and the tenant write fence ship behind dsar_catalog_erasure_v1_enabled (default false). Release 1 keeps the flag off: the fence columns and fence-aware writers are deployed everywhere, but POST /v1/dsar/delete still runs master’s erasure and never sets tenants.write_fenced_at. Old API and worker pods can finish in-flight work without racing a fence they do not understand. Release 2 flips the flag after every API and worker pod runs the fence-aware build and in-flight work started on old code has drained. When the flag is on, erasure walks the same artifact catalog as the export coverage manifest. Before any content class is deleted, Remem commits a tenant write fence (tenants.write_fenced_at, reason dsar_erasure) and waits for claimed in-flight writes to park (outbox processing, up to 30 seconds). Pending queued work is refused at claim time. Ingest, document update/delete, namespace writes, and workers refuse new content while the fence is set: API writers return HTTP 409 with tenant_write_fenced; workers ack the job to a parked/dead-letter state without persisting. DSAR export, delete retry, and other reads stay available to the authenticated caller. If the drain times out, class in_flight_writes is failed, completed_at is null, the fence stays, and the same caller can retry. External and content classes run after a successful drain. Remaining SQL steps (namespace deletion, tenant inactivation, optional crypto shred, memberships, and credentials) run last, in one database savepoint, and only when every prior class is erased or retained_by_policy. Any exception or failed verification in that savepoint rolls the whole phase back, so memberships and credentials stay in place and the same authorized caller can retry. crypto_shred=false is decided before that savepoint: the tenant class is retained_by_policy with policy crypto_shred_disabled, the DEK is left intact, and the rest of the savepoint still runs. Rows that hold storage keys or vector ids stay until those referenced objects are verified absent. Each class is recorded as erased (read-back or count of zero), retained_by_policy (with the policy name), or failed (redacted error_kind, plus retained_count when references were kept). completed_at is set only when every catalog class is erased or retained_by_policy. A failed class leaves the request incomplete so the same authorized caller can retry; already-erased classes are skipped after read-back. The fence stays for the rest of a retryable failure. Typical response fields include:
  • tenant_id
  • documents_deleted
  • chunks_deleted
  • entities_deleted
  • extracted_deleted
  • fact_relationships_deleted
  • facts_deleted
  • memory_entities_deleted
  • vectors_deleted
  • files_deleted
  • crypto_shredded
  • started_at
  • completed_at (null when any class failed or was uncovered)
  • class_outcomes (one record per catalog class)
The retained DSAR delete audit record is stored against the tenant. namespace_id is null when that namespace is erased in the same run.

​
Tenant deletion (management)

Tenant-level management endpoint:
  • DELETE /v1/tenants/{tenant_id} (soft)
  • DELETE /v1/tenants/{tenant_id}?hard_delete=true (hard)
These endpoints require a portal JWT and appropriate org/tenant role.

​
Crypto-shredding model

Remem uses tenant-scoped DEKs wrapped by a KEK. With crypto_shred=true on DSAR delete, Remem destroys tenant key material so residual ciphertext becomes unrecoverable. With crypto_shred=false, key material is retained under policy crypto_shred_disabled and the request can still complete.

​
Retention notes

  • Raw files may be retained with TTL rules and keep-forever policies.
  • Queryable content lifecycle is managed via versioning + delete pipeline.
  • Audit logs capture lifecycle actions for compliance and debugging.