Architecture and modules

Pinchana/pinchana-api is an orchestration repository. The parent repo owns Docker Compose, module configuration, release tooling, and Git submodule pointers; the Python services are separate repositories.

Repository layout

The main public components are:

• pinchana-server — central FastAPI gateway, authentication, routing, normalized API v1 adapter, protected media, browser-session routes, public build manifest, and DLP gateway integration.
• pinchana-core — shared models/storage/music/VPN/Docker helpers used by services.
• pinchana-tiktok
• pinchana-inst
• pinchana-shorts
• pinchana-soundcloud
• pinchana-ytmusic
• pinchana-spotify
• pinchana-deezer
• pinchana-threads
• pinchana-twitter
• pinchana-dlp — separate asynchronous private-download subsystem.

Each root pinchana-* Python submodule has its own pyproject.toml, uv.lock, Dockerfile, source package, and independent Git history.

Normal scrape request flow

client
  │
  │ POST /v1/scrape + X-API-Key
  ▼
pinchana-server
  │ match URL against config/modules.yaml
  ├───────────────┬───────────────┬───────────────┐
  ▼               ▼               ▼               ▼
TikTok         Instagram       Threads         ...module
  │               │               │
  └──── write normalized source files into shared scraper-cache
                  │
                  ▼
            pinchana-server
     adapt module result to API v1
                  │
                  ▼
           { data, meta }

The gateway selects the first enabled module whose route pattern matches the submitted complete HTTP(S) URL. It can call registered in-process plugins for development, but production uses the containerized module endpoints configured in modules.yaml.

Service ports

Service	Internal port
Gateway	8080
TikTok	8081
Instagram	8082
YouTube Shorts	8083
SoundCloud	8084
YouTube Music	8085
Spotify	8086
Deezer	8087
Threads	8088
Twitter/X	8089
Gluetun control API	8000

Clients should integrate with the gateway on 8080 rather than individual scraper ports.

Production networking

In production, the gateway and normal scraper containers use Gluetun’s network namespace:

network_mode: container:gluetun

The individual services therefore do not publish their own ports: blocks. Host mappings live on the Gluetun service. Public reverse-proxy ingress should target only the gateway port; scraper/control ports should be firewalled from untrusted networks even when mapped for host operations.

All outbound scraper traffic shares the VPN namespace. The gateway talks to module endpoints via localhost:<module-port> from inside that shared namespace.

Development networking

docker-compose.dev.yml deliberately does not copy the production network model. It uses a normal bridge network, service DNS names, direct development ports, and VPN_ENABLED=false so contributors can build/test without production WireGuard credentials.

Separate DEV_MODULE_*_ENDPOINT values prevent production localhost endpoints from leaking into the development bridge configuration.

Shared media cache

The gateway and modules share a Docker volume mounted at /app/cache. Modules place extracted/downloaded media there; the gateway serves protected cache paths to authenticated clients.

CACHE_MAX_SIZE_GB controls the shared cache target. This is temporary working storage, not a durability guarantee.

Browser flow

Pinchana Web does not change the module architecture. Its POST /v1/web/scrape gateway route goes through the same module selection and normalized response adapter as /v1/scrape, but uses signed browser-session authentication and browser-scoped media paths.

DLP is not a module

The DLP stack is intentionally separate from config/modules.yaml and normal scraper ports. It exists for resource-heavy private YouTube downloads and has a different security model:

Browser
  -> Pinchana Web BFF
  -> pinchana-server /web/dlp/...
  -> internal dlp-api
  -> Redis metadata/ciphertext
  -> orchestrator (only Docker-socket holder)
  -> one ephemeral hardened worker per job
  -> DLP VPN egress

The DLP profile uses internal Docker networks. Redis, the orchestrator, and worker control endpoints must not be publicly exposed.

Build and release architecture

Official API images are published under:

ghcr.io/pinchana/pinchana-api/<service>:<tag>

The parent VERSION is the product CalVer source of truth. Release tooling synchronizes Python package versions/lockfiles and Git submodule pointers before the parent is tagged. Production deploys should then use scripts/update_rolling.py to resolve one coherent released version and pin immutable image digests.