fix: allow ExternalName services in Traefik, restore ExternalName minio-proxy

STATUS.md

@@ -0,0 +1,124 @@
+# Yandex Cloud Production Cluster — Current State
+
+> Last updated: 2026-06-12
+
+---
+
+## Cluster Overview
+
+| Property | Value |
+|----------|-------|
+| Nodes | 3x 2CPU/8GB, zones b/d/e |
+| Egress | NAT Gateway `k8s-nat-gw` (shared, no per-node IPs) |
+| Domain base | `*.prod.t01tt.tech` |
+| NLB IP | `81.26.181.240` |
+| Git repo (Gitea) | `admin/main.git`, branch `master` |
+| ArgoCD sync mode | **Manual** (no auto-sync, no auto-prune) |
+
+---
+
+## Infrastructure Status
+
+### Running & Healthy ✅
+
+| Component | URL / Access | Notes |
+|-----------|-------------|-------|
+| Gitea | `https://git.prod.t01tt.tech` | Also `git@git.prod.t01tt.tech:admin/main.git` (SSH via NLB port 22) |
+| ArgoCD | `https://argocd.prod.t01tt.tech` | Manual sync only |
+| cert-manager | — | `letsencrypt-production` + `staging` ClusterIssuers |
+| Traefik | NLB `81.26.181.240` | Ports 80, 443, 22 (SSH for Gitea) |
+| Monitoring (VM k8s-stack) | — | VMSingle, VMAlertmanager, node-exporter, kube-state-metrics |
+| Grafana | `https://grafana.prod.t01tt.tech` | `admin`/`change-me`, VictoriaMetrics datasource works |
+| CNPG Operator | — | v0.28.3, running in `cnpg-system` |
+| CNPG Cluster `shared-pg` | — | 3/3 nodes healthy, 20Gi SSD each, no databases created yet |
+| Alloy | — | DaemonSet, collecting pod logs, shipping to Loki |
+| Loki | `http://loki.metrics.svc.cluster.local:3100` | Single binary, 20Gi HDD, logs flowing, queryable via API |
+
+### Broken/Incomplete ⚠️
+
+| Component | Issue | Next Step |
+|-----------|-------|-----------|
+| Grafana + Loki | Grafana 13.0.1 has a Loki datasource plugin bug (`unsupported protocol scheme ""`). Loki API itself works (direct queries return data). | Wait for VM k8s-stack Helm chart to bundle a newer Grafana version (13.1.0+). Then add `grafana.image.tag: "13.1.0"` to `monitoring.yaml`. OR: expose Loki via Traefik ingress + use `access: direct` in datasource. |
+| CNPG Databases | `manifests/cnpg/databases.yaml` — `clusterRef` must be `cluster` (8 occurrences) for CNPG v1 API. `manifests/cnpg/secrets.yaml` — type must be `Opaque` with plain password strings. | Fix when creating ESS Matrix apps — databases and secrets are part of the Matrix instance setup, not standalone infra. |
+| `cnpg-cluster` ArgoCD app | OutOfSync due to the two files above. | Sync after fixing databases + secrets as part of ESS setup. |
+
+### ArgoCD App Status
+
+| App | Sync | Health | Notes |
+|-----|------|--------|-------|
+| alloy | OutOfSync | Healthy | Pushed to Gitea, needs manual sync |
+| cert-manager | Synced | Healthy | |
+| cert-manager-issuers | Synced | Healthy | |
+| cnpg-cluster | OutOfSync | Healthy | Blocked by databases.yaml + secrets.yaml |
+| cnpg-operator | Synced | Healthy | |
+| loki | OutOfSync | Healthy | Pushed to Gitea, needs manual sync |
+| monitoring | Synced | Healthy | |
+| root-app | Synced | Healthy | |
+| traefik | Synced | Healthy | |
+
+---
+
+## Directory Structure
+
+```
+~/infra/yandex-prod/
+├── BOOTSTRAP.md                       # Step-by-step bootstrap guide
+├── PLAN.md                            # Original migration plan
+├── STATUS.md                          # This file
+├── kubeconfig                         # Cluster kubeconfig
+├── bootstrap/
+│   ├── gitea/                         # 5 manifests (namespace, pvc, deploy, svc, ingress)
+│   └── argocd/                        # install.sh, values.yaml, ingress.yaml
+├── argocd/
+│   ├── app-of-apps.yaml               # Root app: watches argocd/apps/*.yaml
+│   └── apps/
+│       ├── traefik.yaml               # DaemonSet + NLB (wave -2)
+│       ├── cert-manager.yaml          # Helm chart (wave -1)
+│       ├── cert-manager-issuers.yaml  # ClusterIssuer CRs (wave 0)
+│       ├── cnpg-operator.yaml         # CNPG Helm chart
+│       ├── cnpg-cluster.yaml          # Cluster + DB CRs from manifests/cnpg/
+│       ├── monitoring.yaml            # VM k8s-stack (wave 0)
+│       ├── loki.yaml                  # Loki single-binary (wave 0)
+│       └── alloy.yaml                 # Alloy log collector (wave 0)
+└── manifests/
+    ├── cert-manager/
+    │   └── cluster-issuers.yaml       # letsencrypt-production + staging
+    ├── cnpg/
+    │   ├── namespace.yaml
+    │   ├── cluster.yaml               # shared-pg Cluster CR (3 nodes, 20Gi SSD)
+    │   ├── databases.yaml             # 8 Database CRs (BROKEN: clusterRef)
+    │   └── secrets.yaml               # 4 PG cred secrets (BROKEN: type+values)
+    ├── gitea/
+    │   └── ingressroute-ssh.yaml      # Traefik TCP route for Gitea SSH
+    └── metrics/
+        ├── grafana/
+        │   ├── namespace.yaml
+        │   ├── ingress.yaml           # grafana.prod.t01tt.tech
+        │   └── loki-datasource.yaml   # Loki datasource ConfigMap
+        └── (empty — namespace.yaml moved into grafana/)
+```
+
+---
+
+## Key Decisions Made
+
+| Decision | Why |
+|----------|-----|
+| NAT Gateway instead of per-node IPs | IP quota limits; shared egress via `k8s-nat-gw` |
+| Auto-sync disabled on all apps | Manual control during bootstrapping |
+| VM k8s-stack instead of kube-prometheus-stack | Single Helm chart for metrics |
+| Alloy for log collection | Ported working config from old `yc-playground` cluster |
+| Gryphon the Grafekr 13 Loki bug | MVP: logs are collected and queryable via API; UI integration deferred |
+
+---
+
+## Next Session Priorities
+
+1. **Sync alloy + loki** (already in Gitea, just needs manual ArgoCD sync)
+2. **Fix Grafana + Loki** — either Grafana 13.1.0 via chart update or Loki ingress workaround
+3. **Create ESS Matrix app for `mrt0rtikize.ru`** — the test migration (Phase 2 of PLAN.md):
+   - Fix `databases.yaml` and `secrets.yaml` as part of this
+   - Create `argocd/apps/matrix-mrt0rtikize.yaml`
+   - Backup from k3s cluster, restore to new cluster
+   - Test DNS cutover
+4. **Create ESS apps for prod instances** (`t0rt1k.tech`, `roglog.space`, `uretra.space`)

argocd/apps/traefik.yaml

@@ -42,6 +42,7 @@ spec:
           - "--entryPoints.rtc-udp-roglog.address=:30004/udp"
           - "--providers.kubernetesingress.ingressclass=traefik"
           - "--providers.kubernetesingress.ingressendpoint.publishedservice=traefik/traefik"
+          - "--providers.kubernetesingress.allowexternalnameservices=true"
           - "--accesslog=true"
           - "--log.level=INFO"
 

manifests/mastodon/svc-minio.yaml

@@ -4,17 +4,7 @@ metadata:
   name: minio-proxy
   namespace: mastodon
 spec:
+  type: ExternalName
+  externalName: minio.minio.svc.cluster.local
   ports:
     - port: 80
-      targetPort: 9000
----
-apiVersion: v1
-kind: Endpoints
-metadata:
-  name: minio-proxy
-  namespace: mastodon
-subsets:
-  - addresses:
-      - ip: 10.96.209.83
-    ports:
-      - port: 9000

scripts/backup-mrt0rtikize.sh

@@ -0,0 +1,244 @@
+#!/bin/bash
+set -euo pipefail
+
+# ================================================================
+# Backup script for mrt0rtikize.ru Matrix instance (k3s cluster)
+# ================================================================
+# Output: backups/mrt0rtikize-YYYYMMDD-HHMMSS/
+#
+# Run this BEFORE switching DNS. Ensure TTL is already set to 60s
+# on all mrt0rtikize.ru DNS records (24h before planned cutover).
+#
+# Steps:
+#   1. Stop Synapse + MAS to prevent DB writes
+#   2. Dump PostgreSQL (built-in PG from ESS chart)
+#   3. Export generated secrets (CRITICAL: signing key, MAS keys)
+#   4. Export deployment markers ConfigMap
+#   5. Export ESS ArgoCD Application (values reference)
+#   6. Export media file locations (manual restore, noted in README)
+# ================================================================
+
+readonly K3S_KUBECONFIG="${KUBECONFIG:-/home/mrt0rtikize/infra/k3s/config}"
+readonly NS="matrix-mrt0rtikize"
+readonly BACKUP_BASE="$(dirname "$(realpath "$0")")/../backups"
+readonly TIMESTAMP="$(date +%Y%m%d-%H%M%S)"
+readonly BACKUP_DIR="${BACKUP_BASE}/${NS}-${TIMESTAMP}"
+
+readonly K="${KUBECTL:-kubectl} --kubeconfig ${K3S_KUBECONFIG}"
+
+# Colors
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+NC='\033[0m'
+
+log()  { echo -e "${GREEN}[$(date +%H:%M:%S)]${NC} $*"; }
+warn() { echo -e "${YELLOW}[$(date +%H:%M:%S)] WARN:${NC} $*"; }
+err()  { echo -e "${RED}[$(date +%H:%M:%S)] ERROR:${NC} $*"; }
+
+# -------------------------------------------------------------------
+# Prerequisites
+# -------------------------------------------------------------------
+log "=== Checking prerequisites ==="
+
+if ! ${K} get ns "${NS}" >/dev/null 2>&1; then
+    err "Namespace ${NS} not found on k3s cluster. Check KUBECONFIG (${K3S_KUBECONFIG})."
+    exit 1
+fi
+
+mkdir -p "${BACKUP_DIR}"
+log "Backup directory: ${BACKUP_DIR}"
+
+# -------------------------------------------------------------------
+# Step 1: Backup Synapse media files (BEFORE stopping Synapse)
+# -------------------------------------------------------------------
+log "=== Step 1: Backing up Synapse media ==="
+
+SYNAPSE_POD_NAME=$( ${K} -n "${NS}" get pods -l "app.kubernetes.io/component=matrix-server" -o jsonpath='{.items[?(@.status.phase=="Running")].metadata.name}' 2>/dev/null)
+
+if [ -z "${SYNAPSE_POD_NAME}" ]; then
+    warn "No running Synapse pod found. Cannot backup media."
+else
+    log "Using Synapse pod: ${SYNAPSE_POD_NAME}"
+    ${K} exec -n "${NS}" "${SYNAPSE_POD_NAME}" -- tar czf /tmp/synapse-media.tar.gz -C /media media_store/
+    ${K} cp "${NS}/${SYNAPSE_POD_NAME}:/tmp/synapse-media.tar.gz" "${BACKUP_DIR}/synapse-media.tar.gz"
+    ${K} exec -n "${NS}" "${SYNAPSE_POD_NAME}" -- rm /tmp/synapse-media.tar.gz
+    log "Synapse media saved: ${BACKUP_DIR}/synapse-media.tar.gz ($(du -h "${BACKUP_DIR}/synapse-media.tar.gz" | cut -f1))"
+fi
+
+# -------------------------------------------------------------------
+# Step 2: Stop Synapse + MAS (start downtime window)
+# -------------------------------------------------------------------
+log "=== Step 2: Stopping Synapse + MAS ==="
+
+SYNAPSE_READY=$(${K} -n "${NS}" get sts "${NS}-synapse-main" -o jsonpath='{.status.readyReplicas}' 2>/dev/null || echo "0")
+MAS_READY=$(${K} -n "${NS}" get deploy "${NS}-matrix-authentication-service" -o jsonpath='{.status.readyReplicas}' 2>/dev/null || echo "0")
+
+if [ "${SYNAPSE_READY}" != "0" ]; then
+    log "Scaling Synapse to 0..."
+    ${K} -n "${NS}" scale sts "${NS}-synapse-main" --replicas=0
+else
+    log "Synapse already scaled to 0."
+fi
+
+if [ "${MAS_READY}" != "0" ]; then
+    log "Scaling MAS to 0..."
+    ${K} -n "${NS}" scale deploy "${NS}-matrix-authentication-service" --replicas=0
+else
+    log "MAS already scaled to 0."
+fi
+
+log "Waiting for Synapse + MAS pods to terminate..."
+${K} -n "${NS}" wait --for=delete pod -l "app.kubernetes.io/component=matrix-server" --timeout=120s 2>/dev/null || warn "Some Synapse pods may still be terminating."
+${K} -n "${NS}" wait --for=delete pod -l "app.kubernetes.io/component=matrix-authentication" --timeout=120s 2>/dev/null || warn "Some MAS pods may still be terminating."
+
+log "Synapse + MAS stopped."
+
+# -------------------------------------------------------------------
+# Step 3: Dump PostgreSQL
+# -------------------------------------------------------------------
+log "=== Step 3: Dumping PostgreSQL ==="
+
+PG_POD=$(${K} -n "${NS}" get pods -l "app.kubernetes.io/name=postgres" -o jsonpath='{.items[0].metadata.name}' 2>/dev/null)
+if [ -z "${PG_POD}" ]; then
+    PG_POD=$(${K} -n "${NS}" get pods -o name 2>/dev/null | grep postgres | head -1 | cut -d/ -f2)
+fi
+
+if [ -z "${PG_POD}" ]; then
+    err "Could not find PostgreSQL pod in namespace ${NS}."
+    err "Available pods:"
+    ${K} -n "${NS}" get pods
+    exit 1
+fi
+
+log "Using PostgreSQL pod: ${PG_POD}"
+
+${K} exec -n "${NS}" "${PG_POD}" -- pg_dumpall -U postgres > "${BACKUP_DIR}/dump-all.sql"
+log "PostgreSQL dump saved: ${BACKUP_DIR}/dump-all.sql ($(wc -c < "${BACKUP_DIR}/dump-all.sql") bytes)"
+
+${K} exec -n "${NS}" "${PG_POD}" -- pg_dump -U postgres -d synapse > "${BACKUP_DIR}/dump-synapse.sql" 2>/dev/null || \
+    warn "Could not dump synapse DB individually (will use dump-all.sql for restore)."
+
+${K} exec -n "${NS}" "${PG_POD}" -- pg_dump -U postgres -d matrixauthenticationservice > "${BACKUP_DIR}/dump-mas.sql" 2>/dev/null || \
+    warn "Could not dump MAS DB individually (will use dump-all.sql for restore)."
+
+# -------------------------------------------------------------------
+# Step 4: Export generated secrets (CRITICAL)
+# -------------------------------------------------------------------
+log "=== Step 4: Exporting generated secrets ==="
+
+if ${K} -n "${NS}" get secret "${NS}-generated" >/dev/null 2>&1; then
+    ${K} -n "${NS}" get secret "${NS}-generated" -o yaml > "${BACKUP_DIR}/secret-generated.yaml"
+    log "Generated secret saved: ${BACKUP_DIR}/secret-generated.yaml"
+else
+    err "CRITICAL: ${NS}-generated secret NOT FOUND!"
+    err "This contains SYNAPSE_SIGNING_KEY, MAS keys, and MACAROON."
+    err "Without it, federation identity is lost and all rooms break."
+    err "Available secrets:"
+    ${K} -n "${NS}" get secrets
+    exit 1
+fi
+
+log "Secret contents (verify these exist):"
+${K} -n "${NS}" get secret "${NS}-generated" -o jsonpath='{.data}' | python3 -c "
+import json, sys
+keys = ['SYNAPSE_SIGNING_KEY', 'MAS_ENCRYPTION_SECRET', 'MAS_RSA_PRIVATE_KEY',
+        'SYNAPSE_MACAROON', 'MAS_SYNAPSE_SHARED_SECRET',
+        'POSTGRES_SYNAPSE_PASSWORD', 'POSTGRES_MATRIX_AUTHENTICATION_SERVICE_PASSWORD']
+d = json.load(sys.stdin)
+for k in keys:
+    present = 'OK' if k in d else 'MISSING!'
+    print(f'  {k}: {present}')
+" 2>/dev/null || warn "Could not verify secret keys."
+
+# -------------------------------------------------------------------
+# Step 5: Export deployment markers
+# -------------------------------------------------------------------
+log "=== Step 5: Exporting deployment markers ==="
+
+MARKER_CM=$(${K} -n "${NS}" get cm -l "app.kubernetes.io/managed-by=matrix-tools-deployment-markers" -o jsonpath='{.items[0].metadata.name}' 2>/dev/null)
+if [ -n "${MARKER_CM}" ]; then
+    ${K} -n "${NS}" get cm "${MARKER_CM}" -o yaml > "${BACKUP_DIR}/markers.yaml"
+    log "Deployment markers saved: ${BACKUP_DIR}/markers.yaml"
+else
+    warn "No deployment markers ConfigMap found (non-critical, ESS may regenerate)."
+fi
+
+# -------------------------------------------------------------------
+# Step 6: Export ESS values from ArgoCD
+# -------------------------------------------------------------------
+log "=== Step 6: Exporting ESS ArgoCD application ==="
+
+if ${K} -n argocd get application "${NS}" >/dev/null 2>&1; then
+    ${K} -n argocd get application "${NS}" -o yaml > "${BACKUP_DIR}/argo-app.yaml"
+    log "ArgoCD Application saved: ${BACKUP_DIR}/argo-app.yaml"
+else
+    warn "ArgoCD Application '${NS}' not found (running without ArgoCD?)."
+    warn "Save your ESS values manually from helm get values or git."
+fi
+
+# -------------------------------------------------------------------
+# Write README
+# -------------------------------------------------------------------
+log "=== Writing README ==="
+
+cat > "${BACKUP_DIR}/README.txt" << READEOF
+Backup for ${NS} — ${TIMESTAMP}
+==================================
+
+Files:
+  dump-all.sql          Full PostgreSQL dump (pg_dumpall)
+  secret-generated.yaml CRITICAL: contains SYNAPSE_SIGNING_KEY, MAS keys, MACAROON
+  markers.yaml          Deployment markers ConfigMap (ESS state tracking)
+  argo-app.yaml         ESS ArgoCD Application (values reference)
+  synapse-media.tar.gz  Synapse media files (local + remote content)
+  dump-synapse.sql      Synapse DB only (optional, for easier restore)
+  dump-mas.sql          MAS DB only (optional, for easier restore)
+  README.txt            This file
+
+CRITICAL: Do NOT lose secret-generated.yaml.
+  - SYNAPSE_SIGNING_KEY identifies this server to the Matrix federation.
+    Changing it breaks all existing rooms and federation relationships.
+  - MAS_ENCRYPTION_SECRET encrypts user sessions.
+    Changing it forces all users to re-login.
+  - SYNAPSE_MACAROON is the admin API token.
+
+Restore order on new cluster:
+  1. Create CNPG databases + secrets on new cluster (see PLAN.md)
+  2. Deploy ESS chart on new cluster (starts with empty DB)
+  3. Stop Synapse + MAS on new cluster
+  4. Restore PG dump into CNPG shared-pg
+  5. Apply this secret-generated.yaml to new cluster's namespace
+  6. Apply markers.yaml
+  7. Restore media: kubectl cp synapse-media.tar.gz to new Synapse pod, untar to /media/
+  8. Restart Synapse + MAS on new cluster
+  9. Verify: login, federation tester, Element Call
+ 10. Cut DNS to new NLB IP
+READEOF
+
+log "README saved: ${BACKUP_DIR}/README.txt"
+
+# -------------------------------------------------------------------
+# Summary
+# -------------------------------------------------------------------
+cat <<SUMMARY
+
+${GREEN}========================================${NC}
+${GREEN}  Backup Complete: ${NS}${NC}
+${GREEN}========================================${NC}
+
+  Directory: ${BACKUP_DIR}
+  Size:      $(du -sh "${BACKUP_DIR}" | cut -f1)
+
+  ${YELLOW}Next steps:${NC}
+  1. Verify PG dump:    head -20 ${BACKUP_DIR}/dump-all.sql
+  2. Verify secrets:    grep SYNAPSE_SIGNING_KEY ${BACKUP_DIR}/secret-generated.yaml
+  3. Copy media files from node path (see media-info.txt)
+  4. Proceed with migration on new cluster
+
+  ${RED}WARNING: Synapse + MAS are still SCALED TO 0 on k3s.${NC}
+  To restore service on k3s (if migration is postponed):
+    ${K} -n ${NS} scale sts ${NS}-synapse-main --replicas=1
+    ${K} -n ${NS} scale deploy ${NS}-matrix-authentication-service --replicas=1
+
+SUMMARY

scripts/backup-yc-playground.sh

@@ -0,0 +1,245 @@
+#!/bin/bash
+set -euo pipefail
+
+# ================================================================
+# Backup script for Matrix instances on yc-playground cluster
+# ================================================================
+# Backs up the old chart (matrix-2.9.17) prod instances:
+#   t0rt1k.tech, roglog.space, uretra.space
+#
+# Usage: ./backup-yc-playground.sh <name> [--no-downtime]
+# Example: ./backup-yc-playground.sh t0rt1k
+#          ./backup-yc-playground.sh t0rt1k --no-downtime  (test, no scaling)
+# ================================================================
+
+readonly YC_CONTEXT="yc-playground"
+readonly K="${KUBECTL:-kubectl} --context ${YC_CONTEXT}"
+readonly BACKUP_BASE="$(dirname "$(realpath "$0")")/../backups"
+readonly TIMESTAMP="$(date +%Y%m%d-%H%M%S)"
+
+# Colors
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+NC='\033[0m'
+log()  { echo -e "${GREEN}[$(date +%H:%M:%S)]${NC} $*"; }
+warn() { echo -e "${YELLOW}[$(date +%H:%M:%S)] WARN:${NC} $*"; }
+err()  { echo -e "${RED}[$(date +%H:%M:%S)] ERROR:${NC} $*"; }
+
+# -------------------------------------------------------------------
+# Parse args
+# -------------------------------------------------------------------
+DOWNTIME=true
+NAME=""
+for arg in "$@"; do
+    case "$arg" in
+        --no-downtime) DOWNTIME=false ;;
+        *) NAME="$arg" ;;
+    esac
+done
+
+if [ -z "${NAME}" ]; then
+    echo "Usage: $0 <name> [--no-downtime]"
+    echo "  name: t0rt1k | roglog | uretra"
+    echo "  --no-downtime: skip scale up/down (test run)"
+    exit 1
+fi
+
+readonly NS="matrix-${NAME}"
+readonly BACKUP_DIR="${BACKUP_BASE}/matrix-${NAME}-${TIMESTAMP}"
+
+# Synapse labels
+readonly SYNAPSE_LABEL="app.kubernetes.io/instance=chat,app.kubernetes.io/name=matrix"
+# MAS labels
+readonly MAS_LABEL="app=mas"
+readonly MAS_PG_LABEL="app=mas-postgresql"
+
+# -------------------------------------------------------------------
+# Prerequisites
+# -------------------------------------------------------------------
+log "=== Backing up ${NS} (downtime=${DOWNTIME}) ==="
+
+if ! ${K} get ns "${NS}" >/dev/null 2>&1; then
+    err "Namespace ${NS} not found on yc-playground."
+    exit 1
+fi
+
+mkdir -p "${BACKUP_DIR}"
+log "Backup directory: ${BACKUP_DIR}"
+
+# -------------------------------------------------------------------
+# Step 1: Backup Synapse media
+# -------------------------------------------------------------------
+log "=== Step 1: Backing up Synapse media ==="
+
+SYNAPSE_POD=$(${K} -n "${NS}" get pods -l "${SYNAPSE_LABEL}" -o jsonpath='{.items[?(@.status.phase=="Running")].metadata.name}' 2>/dev/null)
+
+if [ -n "${SYNAPSE_POD}" ]; then
+    log "Synapse pod: ${SYNAPSE_POD}"
+    # Check if media directory exists
+    if ${K} exec -n "${NS}" "${SYNAPSE_POD}" -- test -d /data/media_store 2>/dev/null; then
+        ${K} exec -n "${NS}" "${SYNAPSE_POD}" -- tar czf /tmp/synapse-media.tar.gz -C /data media_store/
+        ${K} cp "${NS}/${SYNAPSE_POD}:/tmp/synapse-media.tar.gz" "${BACKUP_DIR}/synapse-media.tar.gz" 2>/dev/null || \
+            ${K} cp "${NS}/${SYNAPSE_POD}:tmp/synapse-media.tar.gz" "${BACKUP_DIR}/synapse-media.tar.gz"
+        ${K} exec -n "${NS}" "${SYNAPSE_POD}" -- rm -f /tmp/synapse-media.tar.gz
+        log "Media saved: $(du -h "${BACKUP_DIR}/synapse-media.tar.gz" | cut -f1)"
+    else
+        warn "/data/media_store not found in Synapse pod."
+    fi
+else
+    warn "No running Synapse pod found — skipping media backup."
+fi
+
+# -------------------------------------------------------------------
+# Step 2: Stop Synapse + MAS
+# -------------------------------------------------------------------
+if ${DOWNTIME}; then
+    log "=== Step 2: Stopping Synapse + MAS ==="
+
+    SYNAPSE_READY=$(${K} -n "${NS}" get deploy chat-matrix -o jsonpath='{.status.readyReplicas}' 2>/dev/null || echo "0")
+    MAS_READY=$(${K} -n "${NS}" get deploy mas -o jsonpath='{.status.readyReplicas}' 2>/dev/null || echo "0")
+
+    if [ "${SYNAPSE_READY}" != "0" ]; then
+        ${K} -n "${NS}" scale deploy chat-matrix --replicas=0
+        log "Synapse scaled to 0."
+    else
+        log "Synapse already scaled to 0."
+    fi
+
+    if [ "${MAS_READY}" != "0" ]; then
+        ${K} -n "${NS}" scale deploy mas --replicas=0
+        log "MAS scaled to 0."
+    else
+        log "MAS already scaled to 0."
+    fi
+
+    ${K} -n "${NS}" wait --for=delete pod -l "${SYNAPSE_LABEL}" --timeout=120s 2>/dev/null || warn "Synapse may still be terminating."
+    ${K} -n "${NS}" wait --for=delete pod -l "${MAS_LABEL}" --timeout=120s 2>/dev/null || warn "MAS may still be terminating."
+    log "Synapse + MAS stopped."
+else
+    log "=== Step 2: Skipping downtime (--no-downtime) ==="
+fi
+
+# -------------------------------------------------------------------
+# Step 3: Dump Synapse PostgreSQL
+# -------------------------------------------------------------------
+log "=== Step 3: Dumping Synapse PostgreSQL ==="
+
+# Get the postgres password from the secret
+SYNAPSE_PG_PW=$(${K} -n "${NS}" get secret chat-postgresql -o jsonpath='{.data.postgres-password}' 2>/dev/null | base64 -d)
+
+if [ -z "${SYNAPSE_PG_PW}" ]; then
+    SYNAPSE_PG_PW=$(${K} -n "${NS}" get secret chat-postgresql -o jsonpath='{.data.password}' 2>/dev/null | base64 -d)
+fi
+
+if [ -z "${SYNAPSE_PG_PW}" ]; then
+    warn "Could not read chat-postgresql secret — trying env var from pod."
+    SYNAPSE_PG_PW=$(${K} exec -n "${NS}" chat-postgresql-0 -c postgresql -- bash -c 'echo $POSTGRES_POSTGRES_PASSWORD' 2>/dev/null)
+fi
+
+${K} exec -n "${NS}" chat-postgresql-0 -c postgresql -- bash -c "env PGPASSWORD='${SYNAPSE_PG_PW}' pg_dump -U postgres -d matrix -f /bitnami/postgresql/data/dump.sql" 2>&1
+${K} cp "${NS}/chat-postgresql-0:/bitnami/postgresql/data/dump.sql" "${BACKUP_DIR}/dump-synapse.sql" 2>/dev/null || true
+${K} exec -n "${NS}" chat-postgresql-0 -c postgresql -- rm -f /bitnami/postgresql/data/dump.sql
+log "Synapse dump saved: ${BACKUP_DIR}/dump-synapse.sql ($(wc -c < "${BACKUP_DIR}/dump-synapse.sql") bytes)"
+
+# -------------------------------------------------------------------
+# Step 4: Dump MAS PostgreSQL
+# -------------------------------------------------------------------
+log "=== Step 4: Dumping MAS PostgreSQL ==="
+
+MAS_PG_POD=$(${K} -n "${NS}" get pods -l "${MAS_PG_LABEL}" -o jsonpath='{.items[0].metadata.name}' 2>/dev/null)
+
+if [ -n "${MAS_PG_POD}" ]; then
+    MAS_PG_PW=$(${K} exec -n "${NS}" "${MAS_PG_POD}" -c postgresql -- bash -c 'echo $POSTGRESQL_PASSWORD' 2>/dev/null)
+    MAS_DB=$(${K} exec -n "${NS}" "${MAS_PG_POD}" -c postgresql -- bash -c 'echo $POSTGRESQL_DATABASE' 2>/dev/null)
+
+    if [ -n "${MAS_PG_PW}" ]; then
+        ${K} exec -n "${NS}" "${MAS_PG_POD}" -c postgresql -- bash -c "env PGPASSWORD='${MAS_PG_PW}' pg_dump -U '${MAS_DB:-mas}' -d '${MAS_DB:-mas}' -f /bitnami/postgresql/data/dump.sql" 2>&1
+        ${K} cp "${NS}/${MAS_PG_POD}:/bitnami/postgresql/data/dump.sql" "${BACKUP_DIR}/dump-mas.sql" 2>/dev/null || true
+        ${K} exec -n "${NS}" "${MAS_PG_POD}" -c postgresql -- rm -f /bitnami/postgresql/data/dump.sql
+        log "MAS dump saved: ${BACKUP_DIR}/dump-mas.sql ($(wc -c < "${BACKUP_DIR}/dump-mas.sql") bytes)"
+    else
+        warn "Could not read MAS PG password — skipping MAS dump."
+    fi
+else
+    warn "No MAS PG pod found — skipping MAS dump."
+fi
+
+# -------------------------------------------------------------------
+# Step 5: Export secrets
+# -------------------------------------------------------------------
+log "=== Step 5: Exporting secrets ==="
+
+for secret in chat-matrix chat-postgresql mas matrixrtc-livekit; do
+    if ${K} -n "${NS}" get secret "${secret}" >/dev/null 2>&1; then
+        ${K} -n "${NS}" get secret "${secret}" -o yaml > "${BACKUP_DIR}/secret-${secret}.yaml"
+        log "Secret saved: secret-${secret}.yaml"
+    else
+        warn "Secret '${secret}' not found."
+    fi
+done
+
+# -------------------------------------------------------------------
+# Step 6: Restart Synapse + MAS
+# -------------------------------------------------------------------
+if ${DOWNTIME}; then
+    log "=== Step 6: Restarting Synapse + MAS ==="
+    ${K} -n "${NS}" scale deploy chat-matrix --replicas=1 2>/dev/null || warn "Could not scale Synapse."
+    ${K} -n "${NS}" scale deploy mas --replicas=1 2>/dev/null || warn "Could not scale MAS."
+    log "Synapse + MAS restarted."
+else
+    log "=== Step 6: Skipped (--no-downtime) ==="
+fi
+
+# -------------------------------------------------------------------
+# Step 7: Write README
+# -------------------------------------------------------------------
+log "=== Step 7: Writing README ==="
+
+cat > "${BACKUP_DIR}/README.txt" << READEOF
+Backup for ${NS} — ${TIMESTAMP}
+==================================
+
+Instance: ${NAME} (namespace: ${NS})
+Chart:    matrix-2.9.17 (old chart, NOT ESS)
+Dump PG:  Synapse (chat-postgresql) + MAS (mas-postgresql)
+
+Files:
+  dump-synapse.sql      Synapse PostgreSQL dump (pg_dump -U postgres -d matrix)
+  dump-mas.sql          MAS PostgreSQL dump (pg_dump -U mas -d mas)
+  synapse-media.tar.gz  Synapse media files (/data/media_store/)
+  secret-chat-matrix.yaml       Synapse secrets (signing.key, macaroon, etc.)
+  secret-chat-postgresql.yaml   PG passwords
+  secret-mas.yaml               MAS secrets (encryption-key, signing-key, shared-secret)
+  secret-matrixrtc-livekit.yaml LiveKit secrets
+
+CRITICAL for restore:
+  - secret-chat-matrix.yaml  (SYNAPSE_SIGNING_KEY for federation identity)
+  - secret-mas.yaml          (MAS encryption-key for user sessions)
+  - dump-synapse.sql         (all user data, rooms, messages)
+  - dump-mas.sql             (MAS user auth data)
+
+Migration to ESS chart notes:
+  - The old chart uses separate Helm releases per component (chat, element-call, livekit).
+  - ESS bundles everything into the matrix-stack chart.
+  - PostgreSQL is external (CNPG) on the new cluster.
+  - MAS keys must be restored EXACTLY as-is for user auth to work.
+  - The Synapse signing.key MUST match the federation identity.
+READEOF
+
+log "README saved."
+
+# -------------------------------------------------------------------
+# Summary
+# -------------------------------------------------------------------
+cat <<SUMMARY
+
+${GREEN}========================================${NC}
+${GREEN}  Backup Complete: ${NS}${NC}
+${GREEN}========================================${NC}
+
+  Directory: ${BACKUP_DIR}
+  Size:      $(du -sh "${BACKUP_DIR}" | cut -f1)
+  Downtime:  ${DOWNTIME}
+
+SUMMARY

scripts/restore-mrt0rtikize.sh

@@ -0,0 +1,224 @@
+#!/bin/bash
+set -euo pipefail
+
+# ================================================================
+# Restore script for mrt0rtikize.ru Matrix instance
+# ================================================================
+# Reads a backup directory created by backup-mrt0rtikize.sh
+# and restores it to the new yandex-prod cluster.
+#
+# Usage: ./restore-mrt0rtikize.sh <backup-dir>
+#
+# Steps:
+#   1. Scale Synapse + MAS to 0 on new cluster
+#   2. Restore PostgreSQL dumps to CNPG shared-pg
+#   3. Apply generated secrets (signing key, MAS keys, macaroon)
+#   4. Apply deployment markers
+#   5. Scale Synapse to 1 (now has signing key + DB)
+#   6. Restore media files
+#   7. Scale MAS to 1
+# ================================================================
+
+readonly YC_KUBECONFIG="${KUBECONFIG:-/home/mrt0rtikize/infra/yandex-prod/kubeconfig}"
+readonly K="${KUBECTL:-kubectl} --kubeconfig ${YC_KUBECONFIG}"
+
+readonly NS="matrix-mrt0rtikize"
+readonly CNPG_NS="cnpg"
+readonly CNPG_POD="shared-pg-1"
+
+# Colors
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+NC='\033[0m'
+
+log()  { echo -e "${GREEN}[$(date +%H:%M:%S)]${NC} $*"; }
+warn() { echo -e "${YELLOW}[$(date +%H:%M:%S)] WARN:${NC} $*"; }
+err()  { echo -e "${RED}[$(date +%H:%M:%S)] ERROR:${NC} $*"; }
+
+# -------------------------------------------------------------------
+# Usage
+# -------------------------------------------------------------------
+if [ $# -ne 1 ]; then
+    echo "Usage: $0 <backup-dir>"
+    echo ""
+    echo "Example: $0 backups/matrix-mrt0rtikize-20260613-192010/"
+    exit 1
+fi
+
+readonly BACKUP_DIR="$1"
+
+if [ ! -d "${BACKUP_DIR}" ]; then
+    err "Backup directory not found: ${BACKUP_DIR}"
+    exit 1
+fi
+
+# Verify critical files exist
+for f in dump-synapse.sql dump-mas.sql secret-generated.yaml markers.yaml synapse-media.tar.gz; do
+    if [ ! -f "${BACKUP_DIR}/${f}" ]; then
+        warn "Missing: ${f}"
+    fi
+done
+
+# -------------------------------------------------------------------
+# Prerequisites
+# -------------------------------------------------------------------
+log "=== Checking prerequisites ==="
+
+if ! ${K} get ns "${NS}" >/dev/null 2>&1; then
+    err "Namespace ${NS} not found on new cluster. Deploy the ESS app first."
+    exit 1
+fi
+
+if ! ${K} -n "${CNPG_NS}" get pod "${CNPG_POD}" >/dev/null 2>&1; then
+    err "CNPG pod ${CNPG_POD} not found. Is the CNPG cluster running?"
+    exit 1
+fi
+
+log "Cluster access verified."
+
+# -------------------------------------------------------------------
+# Step 1: Scale Synapse + MAS to 0
+# -------------------------------------------------------------------
+log "=== Step 1: Stopping Synapse + MAS ==="
+
+${K} -n "${NS}" scale sts -l "app.kubernetes.io/component=matrix-server" --replicas=0 2>/dev/null || \
+    ${K} -n "${NS}" scale sts "${NS}-synapse-main" --replicas=0 2>/dev/null || \
+    warn "Could not scale Synapse via known names, trying by label..."
+
+${K} -n "${NS}" scale deploy -l "app.kubernetes.io/component=matrix-authentication" --replicas=0 2>/dev/null || \
+    ${K} -n "${NS}" scale deploy "${NS}-matrix-authentication-service" --replicas=0 2>/dev/null || \
+    warn "Could not scale MAS via known names..."
+
+log "Waiting for Synapse + MAS pods to terminate..."
+${K} -n "${NS}" wait --for=delete pod -l "app.kubernetes.io/component=matrix-server" --timeout=120s 2>/dev/null || warn "Some Synapse pods may still be terminating."
+${K} -n "${NS}" wait --for=delete pod -l "app.kubernetes.io/component=matrix-authentication" --timeout=120s 2>/dev/null || warn "Some MAS pods may still be terminating."
+
+log "Synapse + MAS stopped."
+
+# -------------------------------------------------------------------
+# Step 2: Clean schemas (DROP SCHEMA CASCADE — no connection races)
+# -------------------------------------------------------------------
+log "=== Step 2: Cleaning database schemas ==="
+
+log "Reading PG credentials from cluster..."
+SYNAPSE_PW=$(${K} get secret pg-creds -n "${NS}" -o jsonpath='{.data.synapse}' | base64 -d)
+MAS_PW=$(${K} get secret pg-creds -n "${NS}" -o jsonpath='{.data.mas}' | base64 -d)
+
+log "Wiping synapse schema..."
+${K} exec -n "${CNPG_NS}" "${CNPG_POD}" -- env PGPASSWORD="${SYNAPSE_PW}" \
+    psql -U synapse_mrt0rtikize -d synapse_mrt0rtikize -h localhost -c \
+    "DROP SCHEMA public CASCADE; CREATE SCHEMA public; GRANT ALL ON SCHEMA public TO synapse_mrt0rtikize;" 2>/dev/null || true
+
+log "Wiping MAS schema..."
+${K} exec -n "${CNPG_NS}" "${CNPG_POD}" -- env PGPASSWORD="${MAS_PW}" \
+    psql -U mas_mrt0rtikize -d mas_mrt0rtikize -h localhost -c \
+    "DROP SCHEMA public CASCADE; CREATE SCHEMA public; GRANT ALL ON SCHEMA public TO mas_mrt0rtikize;" 2>/dev/null || true
+
+log "Schemas cleaned."
+
+# -------------------------------------------------------------------
+# Step 3: Restore PostgreSQL dumps
+# -------------------------------------------------------------------
+log "=== Step 3: Restoring PostgreSQL dumps ==="
+
+log "Restoring Synapse database..."
+${K} exec -i -n "${CNPG_NS}" "${CNPG_POD}" -- env PGPASSWORD="${SYNAPSE_PW}" \
+    psql -U synapse_mrt0rtikize -d synapse_mrt0rtikize -h localhost < "${BACKUP_DIR}/dump-synapse.sql"
+log "Synapse database restored."
+
+log "Restoring MAS database..."
+${K} exec -i -n "${CNPG_NS}" "${CNPG_POD}" -- env PGPASSWORD="${MAS_PW}" \
+    psql -U mas_mrt0rtikize -d mas_mrt0rtikize -h localhost < "${BACKUP_DIR}/dump-mas.sql"
+log "MAS database restored."
+
+# -------------------------------------------------------------------
+# Step 3: Apply generated secrets (CRITICAL)
+# -------------------------------------------------------------------
+log "=== Step 4: Applying generated secrets ==="
+
+if [ -f "${BACKUP_DIR}/secret-generated.yaml" ]; then
+    ${K} replace --force -f "${BACKUP_DIR}/secret-generated.yaml"
+    log "Generated secret replaced: ${NS}-generated (original signing key from backup)"
+else
+    err "CRITICAL: secret-generated.yaml not found in backup!"
+    err "SYNAPSE_SIGNING_KEY and MAS keys will NOT be restored."
+    err "Federation identity is lost without this file."
+fi
+
+# -------------------------------------------------------------------
+# Step 4: Apply deployment markers
+# -------------------------------------------------------------------
+log "=== Step 5: Applying deployment markers ==="
+
+if [ -f "${BACKUP_DIR}/markers.yaml" ]; then
+    ${K} replace --force -f "${BACKUP_DIR}/markers.yaml"
+    log "Deployment markers replaced."
+else
+    warn "markers.yaml not found in backup (non-critical)."
+fi
+
+# -------------------------------------------------------------------
+# Step 5: Scale Synapse to 1
+# -------------------------------------------------------------------
+log "=== Step 6: Starting Synapse ==="
+
+${K} -n "${NS}" scale sts -l "app.kubernetes.io/component=matrix-server" --replicas=1 2>/dev/null || \
+    ${K} -n "${NS}" scale sts "${NS}-synapse-main" --replicas=1 2>/dev/null
+
+log "Waiting for Synapse to start..."
+${K} -n "${NS}" wait --for=condition=ready pod -l "app.kubernetes.io/component=matrix-server" --timeout=300s 2>/dev/null || warn "Synapse is not ready yet, check logs."
+
+# -------------------------------------------------------------------
+# Step 6: Restore media files
+# -------------------------------------------------------------------
+log "=== Step 7: Restoring media files ==="
+
+if [ -f "${BACKUP_DIR}/synapse-media.tar.gz" ]; then
+    SYNAPSE_POD=$( ${K} -n "${NS}" get pods -l "app.kubernetes.io/component=matrix-server" -o jsonpath='{.items[?(@.status.phase=="Running")].metadata.name}' 2>/dev/null)
+    if [ -z "${SYNAPSE_POD}" ]; then
+        warn "No running Synapse pod found for media restore. Skip media."
+        warn "Re-run this step after Synapse is up:"
+        warn "  kubectl cp synapse-media.tar.gz ${NS}/<synapse-pod>:/tmp/ && kubectl exec <synapse-pod> -- tar xzf /tmp/synapse-media.tar.gz -C /media/"
+    else
+        log "Copying media to Synapse pod: ${SYNAPSE_POD}"
+        ${K} cp "${BACKUP_DIR}/synapse-media.tar.gz" "${NS}/${SYNAPSE_POD}:/tmp/synapse-media.tar.gz"
+        ${K} exec -n "${NS}" "${SYNAPSE_POD}" -- tar xzf /tmp/synapse-media.tar.gz -C /media/
+        ${K} exec -n "${NS}" "${SYNAPSE_POD}" -- rm /tmp/synapse-media.tar.gz
+        log "Media files restored."
+    fi
+else
+    warn "synapse-media.tar.gz not found in backup."
+fi
+
+# -------------------------------------------------------------------
+# Step 7: Scale MAS to 1
+# -------------------------------------------------------------------
+log "=== Step 8: Starting MAS ==="
+
+${K} -n "${NS}" scale deploy -l "app.kubernetes.io/component=matrix-authentication" --replicas=1 2>/dev/null || \
+    ${K} -n "${NS}" scale deploy "${NS}-matrix-authentication-service" --replicas=1 2>/dev/null
+
+log "Waiting for MAS to start..."
+${K} -n "${NS}" wait --for=condition=ready pod -l "app.kubernetes.io/component=matrix-authentication" --timeout=120s 2>/dev/null || warn "MAS is not ready yet, check logs."
+
+# -------------------------------------------------------------------
+# Summary
+# -------------------------------------------------------------------
+cat <<SUMMARY
+
+${GREEN}========================================${NC}
+${GREEN}  Restore Complete: ${NS}${NC}
+${GREEN}========================================${NC}
+
+  ${YELLOW}Verification steps:${NC}
+  1. Check Synapse logs:
+     ${K} logs -n ${NS} -l "app.kubernetes.io/component=matrix-server" --tail=30
+  2. Check Synapse has correct signing key:
+     ${K} exec -n ${NS} deploy/${NS}-synapse-main -- ls /secrets/${NS}-generated/SYNAPSE_SIGNING_KEY
+  3. Test login at: https://chat.mrt0rtikize.ru
+  4. Federation tester: https://federationtester.matrix.org/?server_name=mrt0rtikize.ru
+  5. Check certificates:
+     ${K} get certificates -n ${NS}
+
+SUMMARY

scripts/restore-prod.sh

@@ -0,0 +1,351 @@
+#!/bin/bash
+set -euo pipefail
+
+# ================================================================
+# Restore script for prod Matrix instances (yc-playground → yandex-prod)
+# ================================================================
+# Reads a backup created by backup-yc-playground.sh and restores
+# it to the new yandex-prod cluster with ESS chart.
+#
+# Usage: ./restore-prod.sh <name> <backup-dir>
+# Example: ./restore-prod.sh t0rt1k backups/matrix-t0rt1k-20260614-161207/
+# ================================================================
+
+readonly YC_KUBECONFIG="${KUBECONFIG:-/home/mrt0rtikize/infra/yandex-prod/kubeconfig}"
+readonly K="${KUBECTL:-kubectl} --kubeconfig ${YC_KUBECONFIG}"
+
+readonly CNPG_NS="cnpg"
+readonly CNPG_POD="shared-pg-1"
+
+# Colors
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+NC='\033[0m'
+log()  { echo -e "${GREEN}[$(date +%H:%M:%S)]${NC} $*"; }
+warn() { echo -e "${YELLOW}[$(date +%H:%M:%S)] WARN:${NC} $*"; }
+err()  { echo -e "${RED}[$(date +%H:%M:%S)] ERROR:${NC} $*"; }
+
+# -------------------------------------------------------------------
+# Parse args
+# -------------------------------------------------------------------
+if [ $# -ne 2 ]; then
+    echo "Usage: $0 <name> <backup-dir>"
+    echo ""
+    echo "  name:       t0rt1k | roglog | uretra"
+    echo "  backup-dir: path to backup directory from backup-yc-playground.sh"
+    echo ""
+    echo "Example: $0 t0rt1k backups/matrix-t0rt1k-20260614-161207/"
+    exit 1
+fi
+
+readonly NAME="$1"
+readonly NS="matrix-${NAME}"
+readonly DB_SYNAPSE="synapse_${NAME}"
+readonly DB_MAS="mas_${NAME}"
+readonly BACKUP_DIR="$2"
+
+# -------------------------------------------------------------------
+# Prerequisites
+# -------------------------------------------------------------------
+log "=== Restoring ${NS} from ${BACKUP_DIR} ==="
+
+if [ ! -d "${BACKUP_DIR}" ]; then
+    err "Backup directory not found: ${BACKUP_DIR}"
+    exit 1
+fi
+
+for f in dump-synapse.sql dump-mas.sql secret-chat-matrix.yaml secret-mas.yaml secret-chat-postgresql.yaml; do
+    if [ ! -f "${BACKUP_DIR}/${f}" ]; then
+        err "Missing: ${f}"
+        exit 1
+    fi
+done
+
+if ! ${K} get ns "${NS}" >/dev/null 2>&1; then
+    err "Namespace ${NS} not found. Deploy the ESS app and sync first."
+    exit 1
+fi
+
+if ! ${K} -n "${CNPG_NS}" get pod "${CNPG_POD}" >/dev/null 2>&1; then
+    err "CNPG pod ${CNPG_POD} not found."
+    exit 1
+fi
+
+log "Prerequisites OK."
+
+# -------------------------------------------------------------------
+# Step 1: Stop Synapse + MAS
+# -------------------------------------------------------------------
+log "=== Step 1: Stopping Synapse + MAS ==="
+
+${K} -n "${NS}" scale deploy -l "app.kubernetes.io/component=matrix-server" --replicas=0 2>/dev/null || \
+    ${K} -n "${NS}" scale sts "${NS}-synapse-main" --replicas=0 2>/dev/null || \
+    ${K} -n "${NS}" scale deploy chat-matrix --replicas=0 2>/dev/null || \
+    warn "Could not scale Synapse."
+
+${K} -n "${NS}" scale deploy -l "app.kubernetes.io/component=matrix-authentication" --replicas=0 2>/dev/null || \
+    ${K} -n "${NS}" scale deploy "${NS}-matrix-authentication-service" --replicas=0 2>/dev/null || \
+    ${K} -n "${NS}" scale deploy mas --replicas=0 2>/dev/null || \
+    warn "Could not scale MAS."
+
+log "Waiting for pods to terminate..."
+${K} -n "${NS}" wait --for=delete pod -l "app.kubernetes.io/component=matrix-server" --timeout=120s 2>/dev/null || warn "Synapse may still be terminating."
+${K} -n "${NS}" wait --for=delete pod -l "app.kubernetes.io/component=matrix-authentication" --timeout=120s 2>/dev/null || warn "MAS may still be terminating."
+${K} -n "${NS}" wait --for=delete pod -l "app=mas" --timeout=60s 2>/dev/null || true
+
+log "Synapse + MAS stopped."
+
+# -------------------------------------------------------------------
+# Step 2: Clean schemas
+# -------------------------------------------------------------------
+log "=== Step 2: Cleaning database schemas ==="
+
+log "Reading PG credentials..."
+SYNAPSE_PW=$(${K} get secret pg-creds -n "${NS}" -o jsonpath='{.data.synapse}' 2>/dev/null | base64 -d)
+MAS_PW=$(${K} get secret pg-creds -n "${NS}" -o jsonpath='{.data.mas}' 2>/dev/null | base64 -d)
+
+log "Wiping ${DB_SYNAPSE} schema..."
+${K} exec -n "${CNPG_NS}" "${CNPG_POD}" -- env PGPASSWORD="${SYNAPSE_PW}" \
+    psql -U "${DB_SYNAPSE}" -d "${DB_SYNAPSE}" -h localhost -c \
+    "DROP SCHEMA public CASCADE; CREATE SCHEMA public; GRANT ALL ON SCHEMA public TO ${DB_SYNAPSE};" 2>/dev/null || true
+
+log "Wiping ${DB_MAS} schema..."
+${K} exec -n "${CNPG_NS}" "${CNPG_POD}" -- env PGPASSWORD="${MAS_PW}" \
+    psql -U "${DB_MAS}" -d "${DB_MAS}" -h localhost -c \
+    "DROP SCHEMA public CASCADE; CREATE SCHEMA public; GRANT ALL ON SCHEMA public TO ${DB_MAS};" 2>/dev/null || true
+
+log "Schemas cleaned."
+
+# -------------------------------------------------------------------
+# Step 3: Restore PostgreSQL dumps
+# -------------------------------------------------------------------
+log "=== Step 3: Restoring PostgreSQL dumps ==="
+
+log "Restoring Synapse database..."
+# Old chart used 'matrix' role — create it as superuser to suppress OWNER TO errors
+${K} exec -n "${CNPG_NS}" "${CNPG_POD}" -- psql -U postgres -c \
+    "DO \$\$ BEGIN IF NOT EXISTS (SELECT FROM pg_catalog.pg_roles WHERE rolname = 'matrix') THEN CREATE ROLE matrix; END IF; END \$\$;" 2>/dev/null || true
+${K} exec -n "${CNPG_NS}" "${CNPG_POD}" -- psql -U postgres -c \
+    "GRANT matrix TO ${DB_SYNAPSE};" 2>/dev/null || true
+
+${K} exec -i -n "${CNPG_NS}" "${CNPG_POD}" -- env PGPASSWORD="${SYNAPSE_PW}" \
+    psql -U "${DB_SYNAPSE}" -d "${DB_SYNAPSE}" -h localhost < "${BACKUP_DIR}/dump-synapse.sql"
+log "Synapse database restored."
+
+log "Restoring MAS database..."
+# Old chart used 'mas' role — create it as superuser to suppress OWNER TO errors
+${K} exec -n "${CNPG_NS}" "${CNPG_POD}" -- psql -U postgres -c \
+    "DO \$\$ BEGIN IF NOT EXISTS (SELECT FROM pg_catalog.pg_roles WHERE rolname = 'mas') THEN CREATE ROLE mas; END IF; END \$\$;" 2>/dev/null || true
+${K} exec -n "${CNPG_NS}" "${CNPG_POD}" -- psql -U postgres -c \
+    "GRANT mas TO ${DB_MAS};" 2>/dev/null || true
+${K} exec -i -n "${CNPG_NS}" "${CNPG_POD}" -- env PGPASSWORD="${MAS_PW}" \
+    psql -U "${DB_MAS}" -d "${DB_MAS}" -h localhost < "${BACKUP_DIR}/dump-mas.sql"
+log "MAS database restored."
+
+# -------------------------------------------------------------------
+# Step 4: Construct and apply the ESS -generated secret
+# -------------------------------------------------------------------
+log "=== Step 4: Constructing ESS generated secret ==="
+
+# Read existing auto-generated secret (has keys the old chart didn't: ECDSA, LiveKit, Hookshot, etc.)
+EXISTING_KEY_ECDSA=""
+EXISTING_KEY_LIVEKIT=""
+EXISTING_KEY_HOOKSHOT_REG=""
+EXISTING_KEY_HOOKSHOT_PASS=""
+EXISTING_KEY_SYNAPSE_EXTRA=""
+EXISTING_KEY_PG_ADMIN=""
+
+if ${K} -n "${NS}" get secret "${NS}-generated" >/dev/null 2>&1; then
+    log "Reading existing auto-generated secret for non-migrated keys..."
+    EXISTING_KEY_ECDSA=$(${K} -n "${NS}" get secret "${NS}-generated" -o jsonpath='{.data.MAS_ECDSA_PRIME256V1_PRIVATE_KEY}' 2>/dev/null || echo "")
+    EXISTING_KEY_LIVEKIT=$(${K} -n "${NS}" get secret "${NS}-generated" -o jsonpath='{.data.ELEMENT_CALL_LIVEKIT_SECRET}' 2>/dev/null || echo "")
+    EXISTING_KEY_HOOKSHOT_REG=$(${K} -n "${NS}" get secret "${NS}-generated" -o jsonpath='{.data.HOOKSHOT_REGISTRATION}' 2>/dev/null || echo "")
+    EXISTING_KEY_HOOKSHOT_PASS=$(${K} -n "${NS}" get secret "${NS}-generated" -o jsonpath='{.data.HOOKSHOT_RSA_PASSKEY}' 2>/dev/null || echo "")
+    EXISTING_KEY_SYNAPSE_EXTRA=$(${K} -n "${NS}" get secret "${NS}-generated" -o jsonpath='{.data.SYNAPSE_EXTRA}' 2>/dev/null || echo "")
+    EXISTING_KEY_PG_ADMIN=$(${K} -n "${NS}" get secret "${NS}-generated" -o jsonpath='{.data.POSTGRES_ADMIN_PASSWORD}' 2>/dev/null || echo "")
+    log "Preserved non-migrated keys from existing secret."
+else
+    warn "No existing ${NS}-generated secret found. Some keys may be missing."
+fi
+
+# Read old secrets from backup
+log "Reading old secrets from backup..."
+OLD_SIGNING_KEY=$(python3 -c "
+import yaml, sys
+with open('${BACKUP_DIR}/secret-chat-matrix.yaml') as f:
+    s = yaml.safe_load(f.read())
+print(s['data']['signing.key'])
+" 2>/dev/null)
+
+OLD_MACAROON=$(python3 -c "
+import yaml, sys
+with open('${BACKUP_DIR}/secret-chat-matrix.yaml') as f:
+    s = yaml.safe_load(f.read())
+print(s['data']['macaroon_secret_key'])
+" 2>/dev/null)
+
+OLD_REGISTRATION=$(python3 -c "
+import yaml, sys
+with open('${BACKUP_DIR}/secret-chat-matrix.yaml') as f:
+    s = yaml.safe_load(f.read())
+print(s['data']['registration_shared_secret'])
+" 2>/dev/null)
+
+OLD_MAS_ENCRYPTION=$(python3 -c "
+import yaml, sys
+with open('${BACKUP_DIR}/secret-mas.yaml') as f:
+    s = yaml.safe_load(f.read())
+print(s['data']['encryption-key'])
+" 2>/dev/null)
+
+OLD_MAS_RSA=$(python3 -c "
+import yaml, sys
+with open('${BACKUP_DIR}/secret-mas.yaml') as f:
+    s = yaml.safe_load(f.read())
+print(s['data']['signing-key.pem'])
+" 2>/dev/null)
+
+OLD_MAS_SHARED=$(python3 -c "
+import yaml, sys
+with open('${BACKUP_DIR}/secret-mas.yaml') as f:
+    s = yaml.safe_load(f.read())
+print(s['data']['shared-secret'])
+" 2>/dev/null)
+
+OLD_SYNAPSE_PG_PW=$(python3 -c "
+import yaml, sys
+with open('${BACKUP_DIR}/secret-chat-postgresql.yaml') as f:
+    s = yaml.safe_load(f.read())
+print(s['data']['password'])
+" 2>/dev/null)
+
+OLD_MAS_DB_PW=$(python3 -c "
+import yaml, sys
+with open('${BACKUP_DIR}/secret-mas.yaml') as f:
+    s = yaml.safe_load(f.read())
+print(s['data']['mas-db-password'])
+" 2>/dev/null)
+
+# Build the new secret
+log "Applying generated secret with migrated keys..."
+
+PYTHON_BODY=$(cat <<'PYEOF'
+import yaml, sys, base64
+
+# Read existing secret structure from stdin (if any)
+existing = {}
+try:
+    existing = yaml.safe_load(sys.stdin)
+except:
+    pass
+
+data = existing.get('data', {}) if existing else {}
+
+# Overwrite with old keys (these are base64 encoded already)
+data['SYNAPSE_SIGNING_KEY'] = sys.argv[1] if sys.argv[1] else data.get('SYNAPSE_SIGNING_KEY','')
+data['SYNAPSE_MACAROON'] = sys.argv[2] if sys.argv[2] else data.get('SYNAPSE_MACAROON','')
+data['SYNAPSE_REGISTRATION_SHARED_SECRET'] = sys.argv[3] if sys.argv[3] else data.get('SYNAPSE_REGISTRATION_SHARED_SECRET','')
+data['MAS_ENCRYPTION_SECRET'] = sys.argv[4] if sys.argv[4] else data.get('MAS_ENCRYPTION_SECRET','')
+data['MAS_RSA_PRIVATE_KEY'] = sys.argv[5] if sys.argv[5] else data.get('MAS_RSA_PRIVATE_KEY','')
+
+data['MAS_SYNAPSE_SHARED_SECRET'] = sys.argv[6] if sys.argv[6] else data.get('MAS_SYNAPSE_SHARED_SECRET','')
+data['POSTGRES_SYNAPSE_PASSWORD'] = sys.argv[7] if sys.argv[7] else data.get('POSTGRES_SYNAPSE_PASSWORD','')
+data['POSTGRES_MATRIX_AUTHENTICATION_SERVICE_PASSWORD'] = sys.argv[8] if sys.argv[8] else data.get('POSTGRES_MATRIX_AUTHENTICATION_SERVICE_PASSWORD','')
+
+# Preserve existing non-migrated keys if provided
+if sys.argv[9]: data['MAS_ECDSA_PRIME256V1_PRIVATE_KEY'] = sys.argv[9]
+if sys.argv[10]: data['ELEMENT_CALL_LIVEKIT_SECRET'] = sys.argv[10]
+if sys.argv[11]: data['HOOKSHOT_REGISTRATION'] = sys.argv[11]
+if sys.argv[12]: data['HOOKSHOT_RSA_PASSKEY'] = sys.argv[12]
+if sys.argv[13]: data['SYNAPSE_EXTRA'] = sys.argv[13]
+if sys.argv[14]: data['POSTGRES_ADMIN_PASSWORD'] = sys.argv[14]
+
+secret = {
+    'apiVersion': 'v1',
+    'kind': 'Secret',
+    'metadata': {
+        'name': f'{sys.argv[15]}-generated',
+        'namespace': sys.argv[16],
+    },
+    'type': 'Opaque',
+    'data': data,
+}
+
+yaml.dump(secret, sys.stdout, default_flow_style=False)
+PYEOF
+)
+
+# Get existing secret for merge
+${K} -n "${NS}" get secret "${NS}-generated" -o yaml 2>/dev/null | \
+    python3 -c "${PYTHON_BODY}" \
+    "${OLD_SIGNING_KEY}" "${OLD_MACAROON}" "${OLD_REGISTRATION}" \
+    "${OLD_MAS_ENCRYPTION}" "${OLD_MAS_RSA}" "${OLD_MAS_SHARED}" \
+    "${OLD_SYNAPSE_PG_PW}" "${OLD_MAS_DB_PW}" \
+    "${EXISTING_KEY_ECDSA}" "${EXISTING_KEY_LIVEKIT}" "${EXISTING_KEY_HOOKSHOT_REG}" \
+    "${EXISTING_KEY_HOOKSHOT_PASS}" "${EXISTING_KEY_SYNAPSE_EXTRA}" "${EXISTING_KEY_PG_ADMIN}" \
+    "${NS}" "${NS}" | \
+    ${K} replace --force -f - 2>/dev/null || \
+    ${K} create -f - 2>/dev/null
+
+log "Generated secret applied with migrated keys from old chart."
+
+# -------------------------------------------------------------------
+# Step 5: Scale Synapse to 1
+# -------------------------------------------------------------------
+log "=== Step 5: Starting Synapse ==="
+
+${K} -n "${NS}" scale sts -l "app.kubernetes.io/component=matrix-server" --replicas=1 2>/dev/null || \
+    ${K} -n "${NS}" scale sts "${NS}-synapse-main" --replicas=1 2>/dev/null || \
+    ${K} -n "${NS}" scale deploy chat-matrix --replicas=1 2>/dev/null
+
+log "Waiting for Synapse..."
+${K} -n "${NS}" wait --for=condition=ready pod -l "app.kubernetes.io/component=matrix-server" --timeout=300s 2>/dev/null || warn "Synapse not ready yet."
+
+# -------------------------------------------------------------------
+# Step 6: Restore media files
+# -------------------------------------------------------------------
+log "=== Step 6: Restoring media files ==="
+
+if [ -f "${BACKUP_DIR}/synapse-media.tar.gz" ]; then
+    SYNAPSE_POD=$( ${K} -n "${NS}" get pods -l "app.kubernetes.io/component=matrix-server" -o jsonpath='{.items[?(@.status.phase=="Running")].metadata.name}' 2>/dev/null)
+    if [ -z "${SYNAPSE_POD}" ]; then
+        warn "No running Synapse pod for media restore."
+    else
+        log "Copying media to Synapse pod: ${SYNAPSE_POD}"
+        ${K} cp "${BACKUP_DIR}/synapse-media.tar.gz" "${NS}/${SYNAPSE_POD}:/tmp/synapse-media.tar.gz"
+        ${K} exec -n "${NS}" "${SYNAPSE_POD}" -- tar xzf /tmp/synapse-media.tar.gz -C /media/
+        ${K} exec -n "${NS}" "${SYNAPSE_POD}" -- rm /tmp/synapse-media.tar.gz
+        log "Media files restored."
+    fi
+else
+    warn "synapse-media.tar.gz not found in backup."
+fi
+
+# -------------------------------------------------------------------
+# Step 7: Scale MAS to 1
+# -------------------------------------------------------------------
+log "=== Step 7: Starting MAS ==="
+
+${K} -n "${NS}" scale deploy -l "app.kubernetes.io/component=matrix-authentication" --replicas=1 2>/dev/null || \
+    ${K} -n "${NS}" scale deploy "${NS}-matrix-authentication-service" --replicas=1 2>/dev/null || \
+    ${K} -n "${NS}" scale deploy mas --replicas=1 2>/dev/null
+
+log "Waiting for MAS..."
+${K} -n "${NS}" wait --for=condition=ready pod -l "app.kubernetes.io/component=matrix-authentication" --timeout=120s 2>/dev/null || warn "MAS not ready yet."
+
+# -------------------------------------------------------------------
+# Summary
+# -------------------------------------------------------------------
+cat <<SUMMARY
+
+${GREEN}========================================${NC}
+${GREEN}  Restore Complete: ${NS}${NC}
+${GREEN}========================================${NC}
+
+  Verification:
+  1. Check Synapse logs:     ${K} logs -n ${NS} -l "app.kubernetes.io/component=matrix-server" --tail=20
+  2. Test login:             https://chat.${NAME}.tech
+  3. Federation tester:      https://federationtester.matrix.org/?server_name=${NAME}.tech
+
+SUMMARY

scripts/setup-pg-creds.sh

@@ -0,0 +1,124 @@
+#!/bin/bash
+set -euo pipefail
+
+# ================================================================
+# Setup PG credentials for an ESS Matrix instance
+# ================================================================
+# Generates random passwords, updates CNPG roles, applies a
+# Kubernetes Secret in the instance namespace, and updates the
+# matching pg-secret.yaml file in the repo.
+#
+# Usage: ./setup-pg-creds.sh <homeserver-name>
+# Example: ./setup-pg-creds.sh mrt0rtikize
+# ================================================================
+
+readonly YC_KUBECONFIG="${KUBECONFIG:-/home/mrt0rtikize/infra/yandex-prod/kubeconfig}"
+readonly K="${KUBECTL:-kubectl} --kubeconfig ${YC_KUBECONFIG}"
+readonly REPO_DIR="$(dirname "$(realpath "$0")")/.."
+
+readonly CNPG_NS="cnpg"
+readonly CNPG_POD="shared-pg-1"
+
+# Colors
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+NC='\033[0m'
+log()  { echo -e "${GREEN}[$(date +%H:%M:%S)]${NC} $*"; }
+err()  { echo -e "${RED}[$(date +%H:%M:%S)] ERROR:${NC} $*"; }
+
+# -------------------------------------------------------------------
+# Parse argument
+# -------------------------------------------------------------------
+NAME="${1:?Usage: $0 <homeserver-name>}"
+NS="matrix-${NAME}"
+DB_SYNAPSE="synapse_${NAME}"
+DB_MAS="mas_${NAME}"
+USER_SYNAPSE="synapse_${NAME}"
+USER_MAS="mas_${NAME}"
+
+# -------------------------------------------------------------------
+# Prerequisites
+# -------------------------------------------------------------------
+log "=== Setting up PG credentials for ${NAME} ==="
+
+if ! ${K} get ns "${NS}" >/dev/null 2>&1; then
+    err "Namespace ${NS} not found. Deploy the ESS app first."
+    exit 1
+fi
+
+if ! ${K} -n "${CNPG_NS}" get pod "${CNPG_POD}" >/dev/null 2>&1; then
+    err "CNPG pod ${CNPG_POD} not found."
+    exit 1
+fi
+
+# -------------------------------------------------------------------
+# Generate passwords
+# -------------------------------------------------------------------
+log "Generating passwords..."
+
+SYNAPSE_PW=$(openssl rand -base64 24 | tr -d '\n')
+MAS_PW=$(openssl rand -base64 24 | tr -d '\n')
+
+# -------------------------------------------------------------------
+# Update PostgreSQL roles
+# -------------------------------------------------------------------
+log "Creating/updating CNPG role: ${USER_SYNAPSE}"
+${K} exec -n "${CNPG_NS}" "${CNPG_POD}" -- \
+    psql -U postgres -c "DO \$\$ BEGIN IF NOT EXISTS (SELECT FROM pg_catalog.pg_roles WHERE rolname = '${USER_SYNAPSE}') THEN CREATE ROLE ${USER_SYNAPSE} WITH LOGIN PASSWORD '${SYNAPSE_PW}'; ELSE ALTER ROLE ${USER_SYNAPSE} WITH PASSWORD '${SYNAPSE_PW}'; END IF; END \$\$;"
+
+log "Creating/updating CNPG role: ${USER_MAS}"
+${K} exec -n "${CNPG_NS}" "${CNPG_POD}" -- \
+    psql -U postgres -c "DO \$\$ BEGIN IF NOT EXISTS (SELECT FROM pg_catalog.pg_roles WHERE rolname = '${USER_MAS}') THEN CREATE ROLE ${USER_MAS} WITH LOGIN PASSWORD '${MAS_PW}'; ELSE ALTER ROLE ${USER_MAS} WITH PASSWORD '${MAS_PW}'; END IF; END \$\$;"
+
+# -------------------------------------------------------------------
+# Apply Kubernetes Secret
+# -------------------------------------------------------------------
+log "Creating/updating Kubernetes Secret pg-creds in ${NS}..."
+
+${K} create secret generic pg-creds -n "${NS}" \
+    --from-literal=synapse="${SYNAPSE_PW}" \
+    --from-literal=mas="${MAS_PW}" \
+    --dry-run=client -o yaml | ${K} apply -f -
+
+# -------------------------------------------------------------------
+# Update repo file
+# -------------------------------------------------------------------
+SECRET_FILE="${REPO_DIR}/manifests/${NS}/pg-secret.yaml"
+if [ -f "${SECRET_FILE}" ]; then
+    log "Updating ${SECRET_FILE}..."
+
+    cat > "${SECRET_FILE}" <<EOF
+apiVersion: v1
+kind: Secret
+metadata:
+  name: pg-creds
+  namespace: ${NS}
+  labels:
+    app.kubernetes.io/part-of: matrix-stack
+type: Opaque
+stringData:
+  synapse: ${SYNAPSE_PW}
+  mas: ${MAS_PW}
+EOF
+    log "Repo file updated."
+else
+    log "File ${SECRET_FILE} does not exist — skipping repo update."
+    log "The Secret is live in the cluster (not in git). Create the file manually."
+fi
+
+# -------------------------------------------------------------------
+# Restart pods
+# -------------------------------------------------------------------
+log "Restarting pods to pick up new credentials..."
+
+${K} delete pod -n "${NS}" -l "app.kubernetes.io/component=matrix-server" --ignore-not-found 2>/dev/null
+${K} delete pod -n "${NS}" -l "app.kubernetes.io/component=matrix-authentication" --ignore-not-found 2>/dev/null
+
+# -------------------------------------------------------------------
+# Summary
+# -------------------------------------------------------------------
+echo ""
+echo "Synapse password: ${SYNAPSE_PW}"
+echo "MAS password:     ${MAS_PW}"
+echo ""
+echo "Done. Synapse + MAS pods are restarting with new credentials."

telegram_bridge

@@ -0,0 +1,27 @@
+app api id 30315730
+app api hash 1c65d3e679f09c8b5d1172e5b3d88f66
+title bridge
+short name mautrix2t0rt1k
+
+
+test
+149.154.167.40:443
+-----BEGIN RSA PUBLIC KEY-----
+MIIBCgKCAQEAyMEdY1aR+sCR3ZSJrtztKTKqigvO/vBfqACJLZtS7QMgCGXJ6XIR
+yy7mx66W0/sOFa7/1mAZtEoIokDP3ShoqF4fVNb6XeqgQfaUHd8wJpDWHcR2OFwv
+plUUI1PLTktZ9uW2WE23b+ixNwJjJGwBDJPQEQFBE+vfmH0JP503wr5INS1poWg/
+j25sIWeYPHYeOrFp/eXaqhISP6G+q2IeTaWTXpwZj4LzXq5YOpk4bYEQ6mvRq7D1
+aHWfYmlEGepfaYR8Q0YqvvhYtMte3ITnuSJs171+GDqpdKcSwHnd6FudwGO4pcCO
+j4WcDuXc2CTHgH8gFTNhp/Y8/SpDOhvn9QIDAQAB
+-----END RSA PUBLIC KEY-----
+
+prod
+149.154.167.50:443
+-----BEGIN RSA PUBLIC KEY-----
+MIIBCgKCAQEA6LszBcC1LGzyr992NzE0ieY+BSaOW622Aa9Bd4ZHLl+TuFQ4lo4g
+5nKaMBwK/BIb9xUfg0Q29/2mgIR6Zr9krM7HjuIcCzFvDtr+L0GQjae9H0pRB2OO
+62cECs5HKhT5DZ98K33vmWiLowc621dQuwKWSQKjWf50XYFw42h21P2KXUGyp2y/
++aEyZ+uVgLLQbRA1dEjSDZ2iGRy12Mk5gpYc397aYp438fsJoHIgJ2lgMv5h7WY9
+t6N/byY9Nw9p21Og3AoXSL2q/2IJ1WRUhebgAdGVMlV1fkuOQoEzR7EdpqtQD9Cs
+5+bfo3Nhmcyvk5ftB0WkJ9z6bNZ7yxrP8wIDAQAB
+-----END RSA PUBLIC KEY-----