DLP operations and security

DLP should be rolled out in two phases: bring up and verify the internal infrastructure while DLP_ENABLED=false, then advertise protocol v2 only after a canary succeeds.

1. Pin a coherent release

On the API host, include DLP in the rolling update so the gateway and the three DLP images resolve to one released CalVer and are written as immutable digests:

python scripts/update_rolling.py --env-file .env --dlp --dry-run
python scripts/update_rolling.py --env-file .env --dlp

--dlp is required to update DLP_API_IMAGE, DLP_ORCHESTRATOR_IMAGE, and DLP_WORKER_IMAGE. The updater intentionally leaves DLP_VPN_IMAGE alone.

Do not deploy latest DLP images in production.

2. Generate independent secrets

Use three different random values of at least 32 characters:

DLP_ENABLED=false
DLP_GATEWAY_TOKEN=REPLACE_WITH_RANDOM_SERVICE_SECRET
DLP_OWNER_SECRET=REPLACE_WITH_DIFFERENT_RANDOM_SECRET
DLP_REDIS_PASSWORD=REPLACE_WITH_DIFFERENT_RANDOM_REDIS_SECRET

Do not reuse the Turnstile signer, mobile signer, a machine API key, or the WireGuard key.

3. Prepare temporary job storage

The default host root is /srv/pinchana-dlp/jobs. It must not be a symlink or world-writable:

sudo install -d -o root -g root -m 0711 /srv/pinchana-dlp/jobs

The orchestrator creates per-job directories for UID/GID 10001 with mode 0700. DLP output is temporary; do not back up, mirror, or synchronize this directory.

4. Run infrastructure preflight

docker compose --env-file .env --profile dlp pull dlp-redis dlp-vpn
python scripts/dlp-prod-preflight.py --env-file .env --phase infra

The preflight is designed to reject unsafe production state such as placeholder/reused secrets, mutable application tags, unsafe job-directory permissions, missing images, published internal DLP ports, incorrect worker networking, direct DNS escape, excess Docker-socket exposure, and a prematurely enabled public flag.

Fix preflight failures instead of bypassing the script.

5. Start internal services with DLP disabled

docker compose --env-file .env --profile dlp up --detach \
  dlp-redis dlp-vpn dlp-api dlp-orchestrator

docker compose --env-file .env --profile dlp ps
docker compose --env-file .env --profile dlp logs --tail=100 \
  dlp-api dlp-orchestrator dlp-vpn

Redis, VPN, DLP API, and orchestrator must be healthy. No job worker should exist before an allocation:

docker ps --filter label=pinchana.dlp.job

Keep internal Redis/orchestrator/worker networks inaccessible from the public host interface.

6. Canary

Use a non-public canary deployment with the same images/networking for:

• one anonymous YouTube download;
• one download using an explicitly selected Cookie Vault profile.

Verify that:

• the worker is removed;
• the temporary job directory expires;
• download limits and range streaming work;
• no plaintext cookie marker appears in Web/gateway/DLP API/Redis/orchestrator logs;
• worker traffic exits through the DLP VPN.

7. Enable capability advertisement

After the canary succeeds, change only:

DLP_ENABLED=true

Then recreate the gateway and run the enable-phase preflight:

docker compose --env-file .env up --detach --no-deps --force-recreate server
python scripts/dlp-prod-preflight.py --env-file .env --phase enable

Through a verified Pinchana Web session, /web/capabilities should now advertise available: true and protocol: 2. Repeat an anonymous production smoke download before considering rollout complete.

Monitoring

Watch at least:

• allocation latency;
• active-job count;
• job duration and failure rate;
• final output bytes;
• Redis health;
• DLP VPN health;
• host free disk space/inodes;
• orphaned pinchana.dlp.job containers;
• stale job directories.

Never log request bodies containing cookie envelopes, authorization/service tokens, decrypted cookies, or sensitive yt-dlp headers.

Rollback

Disable capability first:

DLP_ENABLED=false

docker compose --env-file .env up --detach --no-deps --force-recreate server

This stops new Web clients from receiving DLP availability without touching normal scraper modules. Let active jobs finish/expire, then stop infrastructure if required:

docker compose --env-file .env --profile dlp stop \
  dlp-orchestrator dlp-api dlp-vpn dlp-redis

Do not remove the job root until no labeled worker containers remain. There is no Redis/job-data backup to preserve.