TroubleshootingCopy Anchor Link

If you encounter issues during deployment or operation, the payment-gateway-deploy repository includes built-in diagnostic tools to help identify and resolve problems.

Quick Diagnostics (Podman)Copy Anchor Link

If you are deploying with Podman, use the built-in diagnostic script from the payment-gateway-deploy repository root (not from podman/):

# Check the status of all services
./scripts/diagnose.sh

# Find the root cause of failures
./scripts/diagnose.sh --root-cause

# Start services one-by-one with verification
./scripts/diagnose.sh --startup

# Enable debug mode (prevents cascade failures during complex debugging)
./scripts/diagnose.sh --debug-on

General TroubleshootingCopy Anchor Link

Regardless of the platform, follow these general steps if something goes wrong:

1. Check Deployment Logs:
   • cat podman/deploy.log or cat docker-compose/deploy.log
2. View Service Logs:
   • For Docker Compose (from docker-compose/): docker compose logs -f <service-name> — names match docker-compose.yml (e.g. payment-gateway-admin-backend, mongo).
   • For Podman: use journalctl --user -f -u payment-gateway-app.target for the whole stack, or journalctl --user -f -u payment-gateway-admin-backend for a single service.
3. Emergency Rollback:
   • Use docker-compose/rollback.sh or podman/rollback.sh in the respective platform directory to revert to the previous working state.
   • After receipt-backed financial writes begin, do not restore older financial writers. Stop financial mutation workers and follow the receipt cutover and rollback guidance to preserve recorded money and pending obligations.
4. Reset Deployment:
   • Delete the secrets file .env.secrets from the project root.
   • For Podman: Remove the ~/.config/payment-gateway-deploy/ directory.
   • For Docker Compose: Remove the .env file from the platform directory.
   • Re-run ./deploy.sh.

Common Runtime RunbooksCopy Anchor Link

Checkout Domain TLS FailsCopy Anchor Link

Symptoms: the buyer-facing checkout domain shows a browser TLS error, but the stack is running.

1. Confirm the hostname resolves to the gateway server's public IP.
2. Confirm ports 80 and 443 are reachable from the internet.
3. Check whether an upstream CDN/proxy is terminating TLS before the request reaches Caddy.
4. Review reverse proxy logs and the domain verification status in the Admin Panel.
5. For merchant custom domains, confirm that the domain is verified before sending buyer traffic.

Use the domain detail page in the Admin Panel to confirm verification status, DNS target, and TLS readiness before sending buyer traffic to the hostname.

Provider Webhook Does Not ArriveCopy Anchor Link

Symptoms: the provider shows payment success, but the gateway transaction stays pending.

1. Confirm the provider webhook URL uses MAIN_BACKEND_DOMAIN and the root /hooks/{type}/{providerId} path.
2. Confirm the provider ID in the URL matches the provider record.
3. Confirm the provider webhook signing secret is saved on the provider record.
4. Check Main Backend logs for signature or payload validation errors.
5. Send a test webhook from the provider dashboard after saving changes.

Invoice PDF Generation FailsCopy Anchor Link

Symptoms: invoice exists, but PDF download fails or remains unavailable.

1. Check the worker service logs.
2. Confirm MPG_MAIN_WORKER_SERVICE_URL is configured for the backend that proxies PDF downloads.
3. Confirm Gotenberg is running and reachable from the worker.
4. Regenerate the invoice PDF from the invoice detail page after fixing service connectivity.

Use the invoice detail page to confirm PDF generation state, retry generation after fixing connectivity, and verify that download actions return the expected document.

Tax Reports Are Empty Or Show Unknown CountryCopy Anchor Link

Symptoms: Tax by country or VAT OSS is empty after using Rebuild Tax Records, or Revenue by country shows UNKNOWN.

1. Review the Tax Reports and Tax Records pages for the affected organization and date range.
2. Confirm there are no active tax records with missing line-level tax country.
3. Confirm issued invoices and issued credit notes have active canonical tax records.
4. Confirm issued document tax-record references point at active records for the same source document, not superseded records or transaction-sourced records.
5. Confirm invoice-linked transaction records are not independently active when invoice records exist.
6. Check Settings > Tax > Tax record maintenance for automatic repair status and unresolved reasons. Routine versioned repairs run automatically after upgrades.

The coordinated version 2 upgrade automatically repairs the supported metadata mismatch when the complete invoice group is proven by retained issued invoice, linked issued credit note, payment, adjustment, and tax-record evidence. This also resumes installations that were already blocked in version 1 and inspects installations that had completed version 1. The migration records immutable systemMigration provenance, changes only the proven per-unit net metadata, then lets the normal reconciliation runner apply and verify any derived classification and record-history effects. No dashboard confirmation, database access, or encryption-key handling is required for this proven case.

If the automatic migration cannot prove the complete group, the accounting check remains blocked and the Dashboard shows the unresolved reason. When the Dashboard offers Review recovery, it is an administrator-reviewed fallback for that unresolved source. Compare every listed invoice and linked issued-credit-note change with the retained payment, adjustment, document-total, and tax-record evidence shown by the gateway. The version 3 preview also lists every derived accounting verification effect that the unchanged normal runner will apply. For example, it can show a stored payment and chargeback classification changing from Standard to OSS when the retained issued invoice is the authoritative OSS basis. The preview shows that amounts, countries, rates, supply type, recognition dates, and source links remain fixed; a replacement adjustment record receives its ID during verification and the previous record remains as superseded history.

Fallback confirmation covers the complete displayed source group and all displayed verification effects in one reviewed plan; do not confirm if an expected document, change, or effect is absent. The gateway decrypts protected evidence on the server and never displays encryption keys or raw protected values. It refuses the preview if any protected field cannot be decrypted, a retained document record is missing, another document or derived-record field conflicts, the source changes while the preview is built, or the unchanged production reconciliation planner cannot validate the whole group. Confirmation restores only the proven per-unit net metadata before the normal runner performs the displayed derived normalization. It preserves issued document totals, tax and legal basis, encrypted source fields, PDF versions and renditions, and immutable recovery audit evidence. Accounting remains unavailable until the normal reconciliation runner verifies the result and opens the gate.

If Review recovery is absent, select Export full diagnostics. The Dashboard then lists the records that block the accounting check, each with a plain-language reason, and says when the inspection stopped before every source was checked. It also downloads the same value-free report for your records or support request. The gateway could not prove either the automatic correction or a safe reviewed fallback from retained evidence. A changed source revision, stale comparison value, duplicate same-organization document, conflicting historical record, or unavailable encryption authority refuses the correction without modifying the invoice, credit note, progress, or audit collection. A readback_mismatch or source_revision_changed diagnostic identifies a consistency failure, not a particular receipt or issued invoice. Dashboard recovery controls and confirmation plan versions must come from the coordinated application release; refresh an older browser bundle before following fallback steps.

Repairs and historical calculation preserve issued invoice and credit-note financial snapshots. Enabling tax affects future activity only. Missing historical tax requires the separate Calculate missing historical tax preview and explicit confirmation; invoice-linked history is excluded and requires document correction. Do not infer a zero-tax outcome from missing evidence.

Payments completed without a tax determination, for example while automatic tax was disabled or when checkout could not determine the buyer country, have no tax basis. Earlier versions could leave gross-only reporting records for such payments. The reconciliation treats a record with zero tax and no tax classification, rate, decision, or calculation proof as carrying no tax evidence: the payment is excluded like other untaxed history instead of blocking accounting, and the record, payment, and reports remain unchanged. The Calculate missing historical tax preview still lists such payments for the requested period; payments without billing evidence remain unresolved there and are never calculated. Review them with your bookkeeping and confirm a historical calculation only when the applicable tax rules and buyer evidence are established.

During reconciliation, affected accounting operations and reports may return 503 while diagnostics remain available. A long repair is not a reason to restart older accounting writers. Keep the compatible application version running and follow the coordinated upgrade/recovery procedure. Unresolved or unprocessed sources prevent a fully reconciled result; planned counts are not committed writes.

If the affected rows came from recently completed checkouts, inspect the main-backend logs for repeated completion handling around the same transaction. Completion tax finalization should run once, then automatic invoice generation should promote the transaction tax record to an invoice-sourced record. Repeated status polling must not leave the invoice linked to a superseded tax record.

Encrypted Document Fields Fail To DecryptCopy Anchor Link

Symptoms: invoice or credit-note pages show blank encrypted fields, logs include message authentication failed, or tax-record rebuild fails while reading issued credit notes.

1. Check admin-backend and worker logs for message authentication failed and field-level decryption errors.
2. Identify whether failures are organization-scope or system-scope from the log context and the affected Admin UI page.
3. For organization-scope document snapshot failures, confirm encrypted field metadata was not copied between collections or fields.
4. Repair from source plaintext where available, such as the original client profile or document history. Ciphertext that fails authenticated decryption cannot be decrypted by changing code or keys alone.
5. After source repair, inspect Settings > Tax > Tax record maintenance. If reporting recovery is still needed, select Preview record repair, review the remaining errors and explicitly confirm. Do not repeatedly rebuild data whose authenticated source evidence is still unavailable.

Update Fails Before DeploymentCopy Anchor Link

Symptoms: update exits before restarting services.

1. Check that RELEASE_METADATA_URL is reachable from the server.
2. Confirm deployment access credentials are still valid.
3. Confirm the license support window entitles the target release.
4. If the log says Digest validation failed, keep the host in rollback/current-version state and compare the pulled image digest with the component digest in release metadata before retrying.
5. If release security artifacts are missing or policy-result.json reports a failed policy decision during manual review, pause the update and collect the release notes, metadata response, and updater log for support/security review. The updater scripts do not fetch or gate on policy-result.json; treat that artifact as an optional audit step, not an automatic preflight check.
6. Review the updater log before retrying.

The updater intentionally fails fast when release metadata cannot be validated.

MongoDB replica-set upgrade stops before application restartCopy Anchor Link

Symptoms: the deploy or update log says MongoDB did not become an authenticated writable rs0 primary, and the application services were not restarted.

1. Do not remove or recreate mongo_data; the updater stopped specifically to protect the existing database state.

2. Review the MongoDB container logs and confirm the replica-set key file and, when enabled, MongoDB TLS files are readable by the container.

3. Run the health command for your engine:

   Copy Text

   # Docker Compose, from docker-compose/
   docker compose -f docker-compose.yml -f docker-compose.secrets.yml --env-file .env --env-file ../.env.secrets exec -T mongo sh /usr/local/bin/replica-set-health.sh
   
   # Podman
   podman exec systemd-mongo sh /usr/local/bin/replica-set-health.sh

4. Confirm the bundled application connection strings contain replicaSet=rs0. Do not add that parameter to a customer-supplied Atlas URI; the updater intentionally leaves external URIs unchanged.

5. After correcting the reported key, certificate, or configuration problem, rerun the same updater command. The transition is idempotent and preserves the existing volume.