Troubleshooting

If you encounter issues during deployment or operation, the payment-gateway-deploy repository includes built-in diagnostic tools to help identify and resolve problems.

Quick Diagnostics (Podman)

If you are deploying with Podman, use the built-in diagnostic script from the payment-gateway-deploy repository root (not from podman/):

# Check the status of all services
./scripts/diagnose.sh

# Find the root cause of failures
./scripts/diagnose.sh --root-cause

# Start services one-by-one with verification
./scripts/diagnose.sh --startup

# Enable debug mode (prevents cascade failures during complex debugging)
./scripts/diagnose.sh --debug-on

General Troubleshooting

Regardless of the platform, follow these general steps if something goes wrong:

1. Check Deployment Logs:
   • cat podman/deploy.log or cat docker-compose/deploy.log
2. View Service Logs:
   • For Docker Compose (from docker-compose/): docker compose logs -f <service-name> — names match docker-compose.yml (e.g. payment-gateway-admin-backend, mongo).
   • For Podman: use journalctl --user -f -u payment-gateway-app.target for the whole stack, or journalctl --user -f -u payment-gateway-admin-backend for a single service.
3. Emergency Rollback:
   • Use docker-compose/rollback.sh or podman/rollback.sh in the respective platform directory to revert to the previous working state.
4. Reset Deployment:
   • Delete the secrets file .env.secrets from the project root.
   • For Podman: Remove the ~/.config/payment-gateway-deploy/ directory.
   • For Docker Compose: Remove the .env file from the platform directory.
   • Re-run ./deploy.sh.

Common Runtime Runbooks

Checkout Domain TLS Fails

Symptoms: the buyer-facing checkout domain shows a browser TLS error, but the stack is running.

1. Confirm the hostname resolves to the gateway server's public IP.
2. Confirm ports 80 and 443 are reachable from the internet.
3. Check whether an upstream CDN/proxy is terminating TLS before the request reaches Caddy.
4. Review reverse proxy logs and the domain verification status in the Admin Panel.
5. For merchant custom domains, confirm that the domain is verified before sending buyer traffic.

Use the domain detail page in the Admin Panel to confirm verification status, DNS target, and TLS readiness before sending buyer traffic to the hostname.

Provider Webhook Does Not Arrive

Symptoms: the provider shows payment success, but the gateway transaction stays pending.

1. Confirm the provider webhook URL uses MAIN_BACKEND_DOMAIN and the root /hooks/{type}/{providerId} path.
2. Confirm the provider ID in the URL matches the provider record.
3. Confirm the provider webhook signing secret is saved on the provider record.
4. Check Main Backend logs for signature or payload validation errors.
5. Send a test webhook from the provider dashboard after saving changes.

Invoice PDF Generation Fails

Symptoms: invoice exists, but PDF download fails or remains unavailable.

1. Check the worker service logs.
2. Confirm MPG_MAIN_WORKER_SERVICE_URL is configured for the backend that proxies PDF downloads.
3. Confirm Gotenberg is running and reachable from the worker.
4. Regenerate the invoice PDF from the invoice detail page after fixing service connectivity.

Use the invoice detail page to confirm PDF generation state, retry generation after fixing connectivity, and verify that download actions return the expected document.

Tax Reports Are Empty Or Show Unknown Country

Symptoms: Tax by country or VAT OSS is empty after using Rebuild Tax Records, or Revenue by country shows UNKNOWN.

1. Review the Tax Reports and Tax Records pages for the affected organization and date range.
2. Confirm there are no active tax records with missing line-level tax country.
3. Confirm issued invoices and issued credit notes have active canonical tax records.
4. Confirm issued document tax-record references point at active records for the same source document, not superseded records or transaction-sourced records.
5. Confirm invoice-linked transaction records are not independently active when invoice records exist.
6. Re-run Settings > Tax > Rebuild Tax Records after fixing source snapshot or migration issues.

The rebuild should preserve issued invoice and credit-note snapshots. It should not apply current tax settings to historical legal documents unless a single-transaction recalculation was explicitly requested.

If the affected rows came from recently completed checkouts, inspect the main-backend logs for repeated completion handling around the same transaction. Completion tax finalization should run once, then automatic invoice generation should promote the transaction tax record to an invoice-sourced record. Repeated status polling must not leave the invoice linked to a superseded tax record.

Encrypted Document Fields Fail To Decrypt

Symptoms: invoice or credit-note pages show blank encrypted fields, logs include message authentication failed, or tax-record rebuild fails while reading issued credit notes.

1. Check admin-backend and worker logs for message authentication failed and field-level decryption errors.
2. Identify whether failures are organization-scope or system-scope from the log context and the affected Admin UI page.
3. For organization-scope document snapshot failures, confirm encrypted field metadata was not copied between collections or fields.
4. Repair from source plaintext where available, such as the original client profile or document history. Ciphertext that fails authenticated decryption cannot be decrypted by changing code or keys alone.
5. After repair, re-run Settings > Tax > Rebuild Tax Records if the affected records are used by tax reporting.

Update Fails Before Deployment

Symptoms: update exits before restarting services.

1. Check that RELEASE_METADATA_URL is reachable from the server.
2. Confirm deployment access credentials are still valid.
3. Confirm the license support window entitles the target release.
4. If the log says Digest validation failed, keep the host in rollback/current-version state and compare the pulled image digest with the component digest in release metadata before retrying.
5. If release security artifacts are missing or policy-result.json reports a failed policy decision during manual review, pause the update and collect the release notes, metadata response, and updater log for support/security review. The updater scripts do not fetch or gate on policy-result.json; treat that artifact as an optional audit step, not an automatic preflight check.
6. Review the updater log before retrying.

The updater intentionally fails fast when release metadata cannot be validated.