From 0a048628a500c5eaf3cfbdd6655bbb255887efcb Mon Sep 17 00:00:00 2001 From: Kevin Bataille Date: Thu, 26 Feb 2026 00:38:41 +0100 Subject: [PATCH] Add advanced reconciliation features: retry detection, partial payments, credit notes New features: - Credit note handling: Exclude negative invoices from GC reconciliation - Retry detection: Flag invoices with multiple GC payment attempts - Partial payment detection: Track invoices where remain_to_pay > 0 - Payout fee CSV: Export detailed fee breakdown per payout - Configurable tolerances: RECONCILIATION_DATE_TOLERANCE, RECONCILIATION_PAYOUT_TOLERANCE Files: - lib/reconciliation/gocardless_payouts_parser.rb (new) - Parse GC payouts CSV - lib/reconciliation/engine.rb - Add retry_group, partial fields to Match struct - lib/reconciliation/reporter.rb - Show partial/retry in report, write payouts CSV - lib/reconciliation/dolibarr_fetcher.rb - Add is_credit_note field, filter negatives - bin/reconcile - Wire up --gc-payouts argument - README.md - Document new features and --gc-payouts usage - .env.example - Add optional tolerance settings Co-authored-by: Qwen-Coder --- .env.example | 28 +- README.md | 614 ++++++++++-------- bin/reconcile | 22 +- lib/boot.rb | 1 + lib/reconciliation/dolibarr_fetcher.rb | 234 +++---- lib/reconciliation/engine.rb | 565 +++++++++------- .../gocardless_payouts_parser.rb | 64 ++ lib/reconciliation/reporter.rb | 472 ++++++++------ 8 files changed, 1140 insertions(+), 860 deletions(-) create mode 100644 lib/reconciliation/gocardless_payouts_parser.rb diff --git a/.env.example b/.env.example index 8d3e241..de30e74 100644 --- a/.env.example +++ b/.env.example @@ -1,12 +1,16 @@ -# Dolibarr API -DOLIBARR_URL=https://your-dolibarr.example.com/api/index.php -DOLIBARR_API_KEY=your_api_key - -# GoCardless payment method ID for GoCardless -# Find it with: GET /setup/dictionary/payment_types -# Look for "Prélèvement GoCardless" or similar -DOLIBARR_GC_PAYMENT_ID=6 - -# Dolibarr bank account ID (for recording payments) -# Find it with: GET /bankaccounts -DOLIBARR_BANK_ACCOUNT_ID=1 +# Dolibarr API +DOLIBARR_URL=https://your-dolibarr.example.com/api/index.php +DOLIBARR_API_KEY=your_api_key + +# GoCardless payment method ID for GoCardless +# Find it with: GET /setup/dictionary/payment_types +# Look for "Prélèvement GoCardless" or similar +DOLIBARR_GC_PAYMENT_ID=6 + +# Dolibarr bank account ID (for recording payments) +# Find it with: GET /bankaccounts +DOLIBARR_BANK_ACCOUNT_ID=1 + +# Reconciliation tolerances (optional) +# RECONCILIATION_DATE_TOLERANCE=7 +# RECONCILIATION_PAYOUT_TOLERANCE=2 diff --git a/README.md b/README.md index fad2b58..4524fcb 100644 --- a/README.md +++ b/README.md @@ -1,280 +1,334 @@ -# Dolibarr / GoCardless / Shine Reconciliation - -A standalone Ruby script that cross-checks three financial systems and flags discrepancies. - -## The problem it solves - -Payments flow through three separate systems that are not automatically linked: - -``` -Shine bank account ← GoCardless payouts ← GoCardless payments ← Dolibarr invoices -``` - -**Typical workflow:** -1. GoCardless initiates a direct debit for a customer -2. If the debit succeeds, you create the corresponding invoice in Dolibarr and mark it paid -3. If the debit fails, you cancel or delete the draft invoice -4. GoCardless batches collected payments into payouts and transfers them to your Shine account - -Discrepancies arise when any of these steps is missed: -- A GoCardless payment succeeded but the Dolibarr invoice was never created -- A Dolibarr invoice is marked paid but no GoCardless payment can be found for it -- A GoCardless payout never appeared in the Shine bank account - ---- - -## Requirements - -- Ruby 3.x -- Bundler (`gem install bundler`) -- Network access to your Dolibarr instance - ---- - -## Installation - -```bash -cd dolibarr_shine_reconciliation -bundle install -cp .env.example .env -``` - -Edit `.env` and fill in your Dolibarr credentials: - -```dotenv -DOLIBARR_URL=https://your-dolibarr.example.com/api/index.php -DOLIBARR_API_KEY=your_api_key - -# GoCardless payment method ID in Dolibarr (used by --fix mode) -# Find it: GET /setup/dictionary/payment_types -# Look for the "Prélèvement GoCardless" entry -DOLIBARR_GC_PAYMENT_ID=6 - -# Bank account ID in Dolibarr (used by --fix mode) -# Find it: GET /bankaccounts -DOLIBARR_BANK_ACCOUNT_ID=1 -``` - ---- - -## Exporting the data - -### GoCardless — payments CSV - -Dashboard → **Payments** → filter by date range → **Export CSV** - -Place the file in `gocardless/`. The expected columns are: - -| Column | Description | -|--------|-------------| -| `id` | Payment ID, e.g. `PM014J7X4PY98T` | -| `charge_date` | Date the customer was debited (`YYYY-MM-DD`) | -| `amount` | Amount in euros (`30.52`) | -| `description` | Free text — used as the primary match key against the Dolibarr invoice ref | -| `status` | `paid_out`, `confirmed`, `failed`, `cancelled` | -| `links.payout` | Payout ID this payment belongs to | -| `payout_date` | Date the payout was sent to your bank | -| `customers.given_name` | Customer first name | -| `customers.family_name` | Customer last name | - -### Shine — bank statement CSV - -App → **Comptes** → **Exporter le relevé** → select year → download - -Place the annual CSV in `shine/`. The expected columns are: - -| Column | Description | -|--------|-------------| -| `Date de la valeur` | Value date (`DD/MM/YYYY`) | -| `Crédit` | Credit amount in French format (`51,10`) | -| `Débit` | Debit amount | -| `Libellé` | Transaction description | -| `Nom de la contrepartie` | Counterparty name — GoCardless payouts show `GOCARDLESS SAS` here | - -The Shine CSV uses semicolons as separator (`;`), UTF-8 encoding, and Windows CRLF line endings. The script handles all of this automatically. - ---- - -## Usage - -### Dry run — report only, no changes to Dolibarr - -```bash -ruby bin/reconcile \ - --from 2026-01-01 \ - --to 2026-01-31 \ - --gc gocardless/payments_export.csv \ - --shine shine/Cyanet_2026-01-01_2026-12-31_EXPORT/BANQUE_2026-01-01_2026-12-31/BQ_2026-01-01_2026-12-31.csv -``` - -The Shine file is optional. Without it, payout verification (Pass 3) is skipped: - -```bash -ruby bin/reconcile \ - --from 2026-01-01 \ - --to 2026-01-31 \ - --gc gocardless/payments_export.csv -``` - -### Fix mode — auto-mark Dolibarr invoices as paid - -When the script detects a GoCardless payment that was collected but the matching Dolibarr invoice is still open, `--fix` records the payment in Dolibarr via the API: - -```bash -ruby bin/reconcile \ - --from 2026-01-01 \ - --to 2026-01-31 \ - --gc gocardless/payments_export.csv \ - --shine shine/.../BQ_2026-01-01_2026-12-31.csv \ - --fix -``` - -`--fix` only affects invoices flagged `GC_PAID_DOLIBARR_OPEN`. All other entries are reported only. - ---- - -## How matching works - -The script runs three passes over the data. - -### Pass 1 — GoCardless ↔ Dolibarr - -For each GoCardless payment, an attempt is made to find a matching Dolibarr invoice in two steps: - -**Strong match** — the GoCardless `description` field equals the Dolibarr invoice `ref` exactly (case-insensitive). This fires when you put the invoice reference in the GoCardless payment description at creation time. - -**Soft match** — if no strong match is found, the script looks for a Dolibarr invoice where: -- The amount is identical (compared in cents to avoid floating-point errors) -- The invoice date is within 7 days of the GoCardless `charge_date` -- The customer name on the Dolibarr invoice matches the GoCardless customer name (accent-insensitive, word-order-insensitive) - -Once matched (or not), each payment is assigned one of these flags: - -| Flag | Meaning | Action | -|------|---------|--------| -| `MATCHED` | GC payment collected, Dolibarr invoice paid | None | -| `GC_PAID_DOLIBARR_OPEN` | GC collected but Dolibarr invoice is still open | Create the invoice / use `--fix` | -| `GC_NO_INVOICE` | GC payment collected, no Dolibarr invoice found at all | Create the invoice in Dolibarr | -| `GC_FAILED` | GC payment failed | Check if Dolibarr invoice was correctly cancelled | -| `GC_CANCELLED` | GC payment was cancelled before collection | No action | -| `DOLIBARR_PAID_NO_GC` | Dolibarr invoice paid (in the date range), no GC payment found | Verify — may be a manual or cash payment | - -After processing all GC payments, open Dolibarr invoices with no GC counterpart are flagged: - -| Flag | Meaning | Action | -|------|---------|--------| -| `DOLIBARR_OPEN_NO_GC` | Dolibarr invoice open, no GC payment found | Follow up — missed debit or GC export is incomplete | - -### Pass 2 — open Dolibarr invoice audit - -All invoices fetched from Dolibarr with status `open` (validated, not yet paid) that were not matched by any GC payment are listed as `DOLIBARR_OPEN_NO_GC`. Overdue invoices (due date in the past) are highlighted. - -### Pass 3 — GoCardless payouts ↔ Shine bank - -GoCardless batches individual payments into payouts and transfers them as a single bank credit. The script groups `paid_out` payments by their payout ID, sums the amounts, and looks for a matching credit in Shine: - -1. **Exact match** — same amount, date within 2 days → `PAYOUT_VERIFIED` -2. **Date match only** — date within 2 days but amount differs → `PAYOUT_AMOUNT_MISMATCH` (expected: GoCardless deducts its fee from the payout, so the bank credit is always slightly less than the sum of payments) -3. **No match found** → `PAYOUT_MISSING` - -`PAYOUT_AMOUNT_MISMATCH` is the normal case when GoCardless fees are deducted. The difference shown in the report is the total fee charged for the period. - ---- - -## Output - -### Terminal report - -``` -============================================================ - RECONCILIATION REPORT: 2026-01-01 to 2026-01-31 -============================================================ - -DOLIBARR - Total invoices in scope: 9 - Open (no GC match): 2 ← needs attention - Paid (GC matched): 3 - -GOCARDLESS ↔ DOLIBARR - Matched (paid both sides): 3 ✓ - GC paid / Dolibarr open: 0 - Dolibarr paid / no GC: 0 - GC failed: 1 - GC cancelled: 0 - GC payment / no invoice: 4 ← investigate - -SHINE ↔ GOCARDLESS PAYOUTS - Payouts expected: 2 - Verified: 0 - Amount mismatch: 2 ← check GC fees - Missing in Shine: 0 - Expected total: €107.74 - Actual total: €104.91 - Difference: €-2.83 ← GoCardless fees - -ACTIONS NEEDED (6) ------------------------------------------------------------- - 1. [DOLIBARR_OPEN_NO_GC] FA2502-0075 €29.44 ARTHUR Muriel overdue since 2025-02-01 - 2. [GC_NO_INVOICE] GC: PM01RE90... €26.10 MARIE RIVIERE 2026-01-05 - ... -``` - -### CSV export - -A file `tmp/reconciliation_YYYY-MM-DD.csv` is written after every run with one row per invoice/payment, including the flag and recommended action. Suitable for importing into a spreadsheet for manual review. - ---- - -## Project structure - -``` -bin/reconcile Entry point — parses CLI arguments, orchestrates the run -lib/ - boot.rb Loads all dependencies - dolibarr/ - client.rb HTTP client for the Dolibarr REST API (HTTParty) - reconciliation/ - dolibarr_fetcher.rb Fetches invoices and customer names via Dolibarr API - gocardless_parser.rb Parses the GoCardless payments CSV - shine_parser.rb Parses the Shine bank statement CSV - engine.rb 3-pass matching logic, produces flagged result set - reporter.rb Formats and prints the terminal report, writes CSV - fixer.rb Calls Dolibarr API to record payments (--fix mode) -gocardless/ Drop GoCardless CSV exports here -shine/ Shine annual export directories (as downloaded) -tmp/ Output CSVs written here -.env.example Environment variable template -docs/ - reconciliation_plan.md Original design document - dolibarr.json Dolibarr Swagger API spec -``` - ---- - -## Dolibarr API notes - -The script uses the Dolibarr REST API with an API key (`DOLAPIKEY` header). Key endpoints: - -| Method | Path | Purpose | -|--------|------|---------| -| `GET` | `/invoices?status=1` | Fetch all non-draft invoices (open and paid) | -| `GET` | `/thirdparties` | Fetch customer names for invoice matching | -| `POST` | `/invoices/paymentsdistributed` | Record a payment against an invoice (`--fix`) | -| `GET` | `/setup/dictionary/payment_types` | Look up the GoCardless payment method ID | -| `GET` | `/bankaccounts` | Look up the bank account ID | - -The `status=1` query in Dolibarr returns all non-draft invoices regardless of payment state. The script uses the `statut` field in the response (`1`=open, `2`=paid, `3`=cancelled) to distinguish them. Cancelled invoices are excluded from reconciliation. - ---- - -## Limitations and known behaviour - -**GoCardless fee deductions** — Payout amounts in Shine are always slightly less than the sum of the underlying payments because GoCardless deducts its transaction fee from the payout. This is expected and reported as `PAYOUT_AMOUNT_MISMATCH`, not an error. - -**Incomplete GoCardless export** — If your CSV export does not cover the full date range, payments from outside the export window will cause open Dolibarr invoices to appear as `DOLIBARR_OPEN_NO_GC`. Export all payments for the period you are reconciling. - -**Customer name matching** — The soft match normalises names by stripping accents, lowercasing, and sorting words, so "DUPONT Jean" matches "Jean Dupont". If a customer's name is spelled differently in GoCardless vs Dolibarr, the soft match will fail and the payment will appear as `GC_NO_INVOICE`. Correct the name in one of the systems and rerun. - -**Credit notes** — Dolibarr credit notes (`AV...` prefix) with negative amounts are included in the invoice fetch and will appear as `DOLIBARR_PAID_NO_GC` if they fall in the reconciliation period with no corresponding GoCardless refund. This is normal — credit notes are typically settled internally, not via GoCardless. - -**Supplier invoices** — Dolibarr supplier invoices (`/supplierinvoices` endpoint) are on a completely separate API path and are never fetched or considered by this script. +# Dolibarr / GoCardless / Shine Reconciliation + +A standalone Ruby script that cross-checks three financial systems and flags discrepancies. + +## The problem it solves + +Payments flow through three separate systems that are not automatically linked: + +``` +Shine bank account ← GoCardless payouts ← GoCardless payments ← Dolibarr invoices +``` + +**Typical workflow:** +1. GoCardless initiates a direct debit for a customer +2. If the debit succeeds, you create the corresponding invoice in Dolibarr and mark it paid +3. If the debit fails, you cancel or delete the draft invoice +4. GoCardless batches collected payments into payouts and transfers them to your Shine account + +Discrepancies arise when any of these steps is missed: +- A GoCardless payment succeeded but the Dolibarr invoice was never created +- A Dolibarr invoice is marked paid but no GoCardless payment can be found for it +- A GoCardless payout never appeared in the Shine bank account + +--- + +## Requirements + +- Ruby 3.x +- Bundler (`gem install bundler`) +- Network access to your Dolibarr instance + +--- + +## Installation + +```bash +cd dolibarr_shine_reconciliation +bundle install +cp .env.example .env +``` + +Edit `.env` and fill in your Dolibarr credentials: + +```dotenv +DOLIBARR_URL=https://your-dolibarr.example.com/api/index.php +DOLIBARR_API_KEY=your_api_key + +# GoCardless payment method ID in Dolibarr (used by --fix mode) +# Find it: GET /setup/dictionary/payment_types +# Look for the "Prélèvement GoCardless" entry +DOLIBARR_GC_PAYMENT_ID=6 + +# Bank account ID in Dolibarr (used by --fix mode) +# Find it: GET /bankaccounts +DOLIBARR_BANK_ACCOUNT_ID=1 +``` + +--- + +## Exporting the data + +### GoCardless — payments CSV + +Dashboard → **Payments** → filter by date range → **Export CSV** + +Place the file in `gocardless/`. The expected columns are: + +| Column | Description | +|--------|-------------| +| `id` | Payment ID, e.g. `PM014J7X4PY98T` | +| `charge_date` | Date the customer was debited (`YYYY-MM-DD`) | +| `amount` | Amount in euros (`30.52`) | +| `description` | Free text — used as the primary match key against the Dolibarr invoice ref | +| `status` | `paid_out`, `confirmed`, `failed`, `cancelled` | +| `links.payout` | Payout ID this payment belongs to | +| `payout_date` | Date the payout was sent to your bank | +| `customers.given_name` | Customer first name | +| `customers.family_name` | Customer last name | + +### Shine — bank statement CSV + +App → **Comptes** → **Exporter le relevé** → select year → download + +Place the annual CSV in `shine/`. The expected columns are: + +| Column | Description | +|--------|-------------| +| `Date de la valeur` | Value date (`DD/MM/YYYY`) | +| `Crédit` | Credit amount in French format (`51,10`) | +| `Débit` | Debit amount | +| `Libellé` | Transaction description | +| `Nom de la contrepartie` | Counterparty name — GoCardless payouts show `GOCARDLESS SAS` here | + +The Shine CSV uses semicolons as separator (`;`), UTF-8 encoding, and Windows CRLF line endings. The script handles all of this automatically. + +--- + +## Usage + +### Dry run — report only, no changes to Dolibarr + +```bash +ruby bin/reconcile \ + --from 2026-01-01 \ + --to 2026-01-31 \ + --gc gocardless/payments_export.csv \ + --gc-payouts gocardless/payouts_export.csv \ + --shine shine/Cyanet_2026-01-01_2026-12-31_EXPORT/BANQUE_2026-01-01_2026-12-31/BQ_2026-01-01_2026-12-31.csv +``` + +**Recommended:** Always provide the GoCardless payouts CSV (`--gc-payouts`) for exact fee tracking. Without it, the script can only estimate fees by comparing amounts. + +The Shine file is optional. Without it, payout verification (Pass 3) is skipped: + +```bash +ruby bin/reconcile \ + --from 2026-01-01 \ + --to 2026-01-31 \ + --gc gocardless/payments_export.csv +``` + +### Fix mode — auto-mark Dolibarr invoices as paid + +When the script detects a GoCardless payment that was collected but the matching Dolibarr invoice is still open, `--fix` records the payment in Dolibarr via the API: + +```bash +ruby bin/reconcile \ + --from 2026-01-01 \ + --to 2026-01-31 \ + --gc gocardless/payments_export.csv \ + --gc-payouts gocardless/payouts_export.csv \ + --shine shine/.../BQ_2026-01-01_2026-12-31.csv \ + --fix +``` + +`--fix` only affects invoices flagged `GC_PAID_DOLIBARR_OPEN`. All other entries are reported only. + +### Environment variables (optional) + +```dotenv +# Tolerance for soft date matching (default: 7 days) +RECONCILIATION_DATE_TOLERANCE=7 + +# Tolerance for payout date matching (default: 2 days) +RECONCILIATION_PAYOUT_TOLERANCE=2 +``` + +--- + +## How matching works + +The script runs three passes over the data. + +### Pass 1 — GoCardless ↔ Dolibarr + +For each GoCardless payment, an attempt is made to find a matching Dolibarr invoice in two steps: + +**Strong match** — the GoCardless `description` field equals the Dolibarr invoice `ref` exactly (case-insensitive). This fires when you put the invoice reference in the GoCardless payment description at creation time. + +**Soft match** — if no strong match is found, the script looks for a Dolibarr invoice where: +- The amount is identical (compared in cents to avoid floating-point errors) +- The invoice date is within 7 days of the GoCardless `charge_date` +- The customer name on the Dolibarr invoice matches the GoCardless customer name (accent-insensitive, word-order-insensitive) + +Once matched (or not), each payment is assigned one of these flags: + +| Flag | Meaning | Action | +|------|---------|--------| +| `MATCHED` | GC payment collected, Dolibarr invoice paid | None | +| `GC_PAID_DOLIBARR_OPEN` | GC collected but Dolibarr invoice is still open | Create the invoice / use `--fix` | +| `GC_NO_INVOICE` | GC payment collected, no Dolibarr invoice found at all | Create the invoice in Dolibarr | +| `GC_FAILED` | GC payment failed | Check if Dolibarr invoice was correctly cancelled | +| `GC_CANCELLED` | GC payment was cancelled before collection | No action | +| `DOLIBARR_PAID_NO_GC` | Dolibarr invoice paid (in the date range), no GC payment found | Verify — may be a manual or cash payment | + +After processing all GC payments, open Dolibarr invoices with no GC counterpart are flagged: + +| Flag | Meaning | Action | +|------|---------|--------| +| `DOLIBARR_OPEN_NO_GC` | Dolibarr invoice open, no GC payment found | Follow up — missed debit or GC export is incomplete | + +### Pass 2 — open Dolibarr invoice audit + +All invoices fetched from Dolibarr with status `open` (validated, not yet paid) that were not matched by any GC payment are listed as `DOLIBARR_OPEN_NO_GC`. Overdue invoices (due date in the past) are highlighted. + +### Pass 3 — GoCardless payouts ↔ Shine bank + +**With payouts CSV (recommended):** Match by payout reference (exact) and net amount. Fee breakdown is taken directly from the payouts CSV columns: +- `amount` = net payout after all fees +- `total_payment_amount` = gross amount before fees +- `transaction_fee_debit`, `surcharge_fee_debit`, `tax_debit` = fee breakdown + +**Without payouts CSV (fallback):** Groups `paid_out` payments by payout ID, sums amounts, and looks for a matching credit in Shine: + +1. **Exact match** — same amount, date within 2 days → `PAYOUT_VERIFIED` +2. **Date match only** — date within 2 days but amount differs → `PAYOUT_AMOUNT_MISMATCH` (expected: GoCardless deducts its fee from the payout) +3. **No match found** → `PAYOUT_MISSING` + +`PAYOUT_AMOUNT_MISMATCH` is the normal case when GoCardless fees are deducted. The difference shown in the report is the total fee charged for the period. + +--- + +## Advanced features + +### Retry detection + +If multiple GoCardless payments exist for the same invoice reference (e.g., a failed payment was retried), the report flags them with `[RETRY: PM0123, PM0456]`. This helps identify when a payment was re-submitted after an initial failure. + +### Partial payment detection + +Invoices where `remain_to_pay > 0` but some payment has been made are flagged as `PARTIAL`. This can happen when: +- A customer paid only part of the invoice +- Multiple GoCardless payments cover a single invoice +- A credit note was applied to reduce the balance + +### Credit notes excluded + +Credit notes (invoices with negative `total_ttc`) are automatically excluded from reconciliation since they don't correspond to GoCardless payments. They're handled internally in Dolibarr. + +--- + +## Output + +### Terminal report + +``` +============================================================ + RECONCILIATION REPORT: 2026-01-01 to 2026-01-31 +============================================================ + +DOLIBARR + Total invoices in scope: 9 + Open (no GC match): 2 ← needs attention + Paid (GC matched): 3 + +GOCARDLESS ↔ DOLIBARR + Matched (paid both sides): 3 ✓ + GC paid / Dolibarr open: 0 + Dolibarr paid / no GC: 0 + GC failed: 1 + GC cancelled: 0 + GC payment / no invoice: 4 ← investigate + +SHINE ↔ GOCARDLESS PAYOUTS + Payouts expected: 2 + Verified: 0 + Amount mismatch: 2 ← check GC fees + Missing in Shine: 0 + Expected total: €107.74 + Actual total: €104.91 + Difference: €-2.83 ← GoCardless fees + +ACTIONS NEEDED (6) +------------------------------------------------------------ + 1. [DOLIBARR_OPEN_NO_GC] FA2502-0075 €29.44 ARTHUR Muriel overdue since 2025-02-01 + 2. [GC_NO_INVOICE] GC: PM01RE90... €26.10 MARIE RIVIERE 2026-01-05 + ... +``` + +### CSV export + +Two CSV files are written after every run: + +**`tmp/reconciliation_YYYY-MM-DD.csv`** — One row per invoice/payment with columns: +- `invoice_ref`, `customer_name`, `amount_eur`, `invoice_date`, `due_date` +- `dolibarr_status`, `gc_payment_id`, `gc_status`, `gc_charge_date` +- `match_type`, `flag`, `action` +- `partial` — "yes" if invoice is partially paid +- `retry_group` — comma-separated GC payment IDs if retries detected + +**`tmp/payouts_fees_YYYY-MM-DD.csv`** — One row per payout with fee breakdown: +- `payout_id`, `payout_date` +- `gross_amount_eur`, `net_amount_eur`, `fee_eur` +- `fee_percentage` — GC fee as percentage of gross +- `shine_reference` — the matching Shine transaction label +- `status` — `payout_verified`, `payout_missing`, or `payout_amount_mismatch` + +Both files are suitable for importing into a spreadsheet for manual review. + +--- + +## Project structure + +``` +bin/reconcile Entry point — parses CLI arguments, orchestrates the run +lib/ + boot.rb Loads all dependencies + dolibarr/ + client.rb HTTP client for the Dolibarr REST API (HTTParty) + reconciliation/ + dolibarr_fetcher.rb Fetches invoices and customer names via Dolibarr API + gocardless_parser.rb Parses the GoCardless payments CSV + shine_parser.rb Parses the Shine bank statement CSV + engine.rb 3-pass matching logic, produces flagged result set + reporter.rb Formats and prints the terminal report, writes CSV + fixer.rb Calls Dolibarr API to record payments (--fix mode) +gocardless/ Drop GoCardless CSV exports here +shine/ Shine annual export directories (as downloaded) +tmp/ Output CSVs written here +.env.example Environment variable template +docs/ + reconciliation_plan.md Original design document + dolibarr.json Dolibarr Swagger API spec +``` + +--- + +## Dolibarr API notes + +The script uses the Dolibarr REST API with an API key (`DOLAPIKEY` header). Key endpoints: + +| Method | Path | Purpose | +|--------|------|---------| +| `GET` | `/invoices?status=1` | Fetch all non-draft invoices (open and paid) | +| `GET` | `/thirdparties` | Fetch customer names for invoice matching | +| `POST` | `/invoices/paymentsdistributed` | Record a payment against an invoice (`--fix`) | +| `GET` | `/setup/dictionary/payment_types` | Look up the GoCardless payment method ID | +| `GET` | `/bankaccounts` | Look up the bank account ID | + +The `status=1` query in Dolibarr returns all non-draft invoices regardless of payment state. The script uses the `statut` field in the response (`1`=open, `2`=paid, `3`=cancelled) to distinguish them. Cancelled invoices are excluded from reconciliation. + +--- + +## Limitations and known behaviour + +**GoCardless fee deductions** — Payout amounts in Shine are always slightly less than the sum of the underlying payments because GoCardless deducts its transaction fee from the payout. This is expected and reported as `PAYOUT_AMOUNT_MISMATCH`, not an error. + +**Incomplete GoCardless export** — If your CSV export does not cover the full date range, payments from outside the export window will cause open Dolibarr invoices to appear as `DOLIBARR_OPEN_NO_GC`. Export all payments for the period you are reconciling. + +**Customer name matching** — The soft match normalises names by stripping accents, lowercasing, and sorting words, so "DUPONT Jean" matches "Jean Dupont". If a customer's name is spelled differently in GoCardless vs Dolibarr, the soft match will fail and the payment will appear as `GC_NO_INVOICE`. Correct the name in one of the systems and rerun. + +**Credit notes** — Dolibarr credit notes (`AV...` prefix) with negative amounts are included in the invoice fetch and will appear as `DOLIBARR_PAID_NO_GC` if they fall in the reconciliation period with no corresponding GoCardless refund. This is normal — credit notes are typically settled internally, not via GoCardless. + +**Supplier invoices** — Dolibarr supplier invoices (`/supplierinvoices` endpoint) are on a completely separate API path and are never fetched or considered by this script. diff --git a/bin/reconcile b/bin/reconcile index d53c68f..70e1770 100755 --- a/bin/reconcile +++ b/bin/reconcile @@ -10,12 +10,13 @@ options = { from: nil, to: nil, gc: nil, + gc_payouts: nil, shine: nil } OptionParser.new do |opts| opts.banner = <<~BANNER - Usage: bin/reconcile --from DATE --to DATE --gc PATH [--shine PATH] [--fix] + Usage: bin/reconcile --from DATE --to DATE --gc PATH [--gc-payouts PATH] [--shine PATH] [--fix] Options: BANNER @@ -32,6 +33,10 @@ OptionParser.new do |opts| options[:gc] = v end + opts.on("--gc-payouts PATH", "GoCardless payouts CSV file path (enables exact fee reporting)") do |v| + options[:gc_payouts] = v + end + opts.on("--shine PATH", "Shine bank statement CSV file path (optional)") do |v| options[:shine] = v end @@ -52,6 +57,7 @@ errors << "--from is required" unless options[:from] errors << "--to is required" unless options[:to] errors << "--gc is required" unless options[:gc] errors << "--gc file not found: #{options[:gc]}" if options[:gc] && !File.exist?(options[:gc]) +errors << "--gc-payouts file not found: #{options[:gc_payouts]}" if options[:gc_payouts] && !File.exist?(options[:gc_payouts]) errors << "--shine file not found: #{options[:shine]}" if options[:shine] && !File.exist?(options[:shine]) unless errors.empty? @@ -60,16 +66,18 @@ unless errors.empty? end # Run reconciliation -client = Dolibarr::Client.new -fetcher = Reconciliation::DolibarrFetcher.new(client, from: options[:from], to: options[:to]) -gc_data = Reconciliation::GocardlessParser.parse(options[:gc], from: options[:from], to: options[:to]) -shine_data = options[:shine] ? Reconciliation::ShineParser.parse(options[:shine]) : [] +client = Dolibarr::Client.new +fetcher = Reconciliation::DolibarrFetcher.new(client, from: options[:from], to: options[:to]) +gc_data = Reconciliation::GocardlessParser.parse(options[:gc], from: options[:from], to: options[:to]) +gc_payouts = options[:gc_payouts] ? Reconciliation::GocardlessPayoutsParser.parse(options[:gc_payouts]) : [] +shine_data = options[:shine] ? Reconciliation::ShineParser.parse(options[:shine]) : [] dolibarr_invoices = fetcher.fetch_invoices engine = Reconciliation::Engine.new( - dolibarr_invoices: dolibarr_invoices, - gc_payments: gc_data, + dolibarr_invoices: dolibarr_invoices, + gc_payments: gc_data, + gc_payouts: gc_payouts, shine_transactions: shine_data, from: options[:from], to: options[:to] diff --git a/lib/boot.rb b/lib/boot.rb index 3c4897a..c522b00 100644 --- a/lib/boot.rb +++ b/lib/boot.rb @@ -6,6 +6,7 @@ require "dotenv/load" require_relative "dolibarr/client" require_relative "reconciliation/dolibarr_fetcher" require_relative "reconciliation/gocardless_parser" +require_relative "reconciliation/gocardless_payouts_parser" require_relative "reconciliation/shine_parser" require_relative "reconciliation/engine" require_relative "reconciliation/reporter" diff --git a/lib/reconciliation/dolibarr_fetcher.rb b/lib/reconciliation/dolibarr_fetcher.rb index 8fd8bb3..2862503 100644 --- a/lib/reconciliation/dolibarr_fetcher.rb +++ b/lib/reconciliation/dolibarr_fetcher.rb @@ -1,115 +1,119 @@ -# frozen_string_literal: true - -require "date" - -module Reconciliation - class DolibarrFetcher - # Dolibarr invoice statuses - STATUS_DRAFT = 0 - STATUS_VALIDATED = 1 - STATUS_PAID = 2 - STATUS_CANCELLED = 3 - - Invoice = Struct.new( - :id, # Dolibarr internal ID - :ref, # Invoice reference, e.g. "FA2407-0003" - :status, # 0=draft, 1=validated/open, 2=paid, 3=cancelled - :total_ttc, # Total amount TTC (euros, float) - :amount_cents, # total_ttc * 100, rounded (integer) - :paye, # 0=unpaid, 1=paid - :sumpayed, # Amount already paid (euros) - :remain_to_pay, # Remaining balance (euros) - :date, # Invoice date (Date) - :due_date, # Due date (Date) - :customer_id, # socid - :customer_name, # nom of third party if present - keyword_init: true - ) - - def initialize(client, from:, to:) - @client = client - @from = from - @to = to - end - - # Fetches all validated invoices and resolves customer names. - # Note: querying status=1 returns all non-draft invoices in Dolibarr (open, paid, cancelled). - # Cancelled invoices are filtered out here. - def fetch_invoices - $stderr.puts "[DolibarrFetcher] Fetching thirdparty names..." - name_by_id = fetch_thirdparty_names - - $stderr.puts "[DolibarrFetcher] Fetching invoices..." - raw_invoices = fetch_all(status: STATUS_VALIDATED) - - invoices = raw_invoices - .map { |raw| parse_invoice(raw, name_by_id) } - .reject { |inv| inv.status == STATUS_CANCELLED } - - open_count = invoices.count { |inv| inv.status == STATUS_VALIDATED } - paid_count = invoices.count { |inv| inv.status == STATUS_PAID } - $stderr.puts "[DolibarrFetcher] Total: #{invoices.size} invoices (#{open_count} open, #{paid_count} paid)" - invoices - rescue Dolibarr::Client::Error => e - $stderr.puts "[DolibarrFetcher] Error: #{e.message}" - raise - end - - private - - def fetch_all(status:, sqlfilters: nil) - invoices = [] - page = 0 - loop do - params = { status: status, limit: 100, page: page } - params[:sqlfilters] = sqlfilters if sqlfilters - batch = @client.get("/invoices", params) - break if batch.nil? || !batch.is_a?(Array) || batch.empty? - - invoices.concat(batch) - break if batch.size < 100 - - page += 1 - end - invoices - end - - def parse_invoice(raw, name_by_id = {}) - customer_id = raw["socid"].to_i - Invoice.new( - id: raw["id"].to_i, - ref: raw["ref"].to_s.strip, - status: raw["statut"].to_i, - total_ttc: raw["total_ttc"].to_f, - amount_cents: (raw["total_ttc"].to_f * 100).round, - paye: raw["paye"].to_i, - sumpayed: raw["sumpayed"].to_f, - remain_to_pay: raw["remaintopay"].to_f, - date: parse_unix_date(raw["datef"] || raw["date"]), - due_date: parse_unix_date(raw["date_lim_reglement"]), - customer_id: customer_id, - customer_name: name_by_id[customer_id] || raw.dig("thirdparty", "name").to_s.strip - ) - end - - def fetch_thirdparty_names - result = {} - page = 0 - loop do - batch = @client.get("/thirdparties", limit: 100, page: page) - break if batch.nil? || !batch.is_a?(Array) || batch.empty? - batch.each { |tp| result[tp["id"].to_i] = tp["name"].to_s.strip } - break if batch.size < 100 - page += 1 - end - result - end - - def parse_unix_date(value) - return nil if value.nil? || value.to_s.strip.empty? || value.to_i.zero? - Time.at(value.to_i).utc.to_date - rescue - nil - end - end -end +# frozen_string_literal: true + +require "date" + +module Reconciliation + class DolibarrFetcher + # Dolibarr invoice statuses + STATUS_DRAFT = 0 + STATUS_VALIDATED = 1 + STATUS_PAID = 2 + STATUS_CANCELLED = 3 + + Invoice = Struct.new( + :id, # Dolibarr internal ID + :ref, # Invoice reference, e.g. "FA2407-0003" + :status, # 0=draft, 1=validated/open, 2=paid, 3=cancelled + :total_ttc, # Total amount TTC (euros, float) + :amount_cents, # total_ttc * 100, rounded (integer) + :paye, # 0=unpaid, 1=paid + :sumpayed, # Amount already paid (euros) + :remain_to_pay, # Remaining balance (euros) + :date, # Invoice date (Date) + :due_date, # Due date (Date) + :customer_id, # socid + :customer_name, # nom of third party if present + :is_credit_note, # true if this is a credit note (negative amount) + keyword_init: true + ) + + def initialize(client, from:, to:) + @client = client + @from = from + @to = to + end + + # Fetches all validated invoices and resolves customer names. + # Note: querying status=1 returns all non-draft invoices in Dolibarr (open, paid, cancelled). + # Cancelled invoices are filtered out here. + def fetch_invoices + $stderr.puts "[DolibarrFetcher] Fetching thirdparty names..." + name_by_id = fetch_thirdparty_names + + $stderr.puts "[DolibarrFetcher] Fetching invoices..." + raw_invoices = fetch_all(status: STATUS_VALIDATED) + + invoices = raw_invoices + .map { |raw| parse_invoice(raw, name_by_id) } + .reject { |inv| inv.status == STATUS_CANCELLED } + .reject { |inv| inv.is_credit_note } # Credit notes (negative amounts) excluded from GC reconciliation + + open_count = invoices.count { |inv| inv.status == STATUS_VALIDATED } + paid_count = invoices.count { |inv| inv.status == STATUS_PAID } + $stderr.puts "[DolibarrFetcher] Total: #{invoices.size} invoices (#{open_count} open, #{paid_count} paid)" + invoices + rescue Dolibarr::Client::Error => e + $stderr.puts "[DolibarrFetcher] Error: #{e.message}" + raise + end + + private + + def fetch_all(status:, sqlfilters: nil) + invoices = [] + page = 0 + loop do + params = { status: status, limit: 100, page: page } + params[:sqlfilters] = sqlfilters if sqlfilters + batch = @client.get("/invoices", params) + break if batch.nil? || !batch.is_a?(Array) || batch.empty? + + invoices.concat(batch) + break if batch.size < 100 + + page += 1 + end + invoices + end + + def parse_invoice(raw, name_by_id = {}) + customer_id = raw["socid"].to_i + total_ttc = raw["total_ttc"].to_f + Invoice.new( + id: raw["id"].to_i, + ref: raw["ref"].to_s.strip, + status: raw["statut"].to_i, + total_ttc: total_ttc, + amount_cents: (total_ttc * 100).round, + paye: raw["paye"].to_i, + sumpayed: raw["sumpayed"].to_f, + remain_to_pay: raw["remaintopay"].to_f, + date: parse_unix_date(raw["datef"] || raw["date"]), + due_date: parse_unix_date(raw["date_lim_reglement"]), + customer_id: customer_id, + customer_name: name_by_id[customer_id] || raw.dig("thirdparty", "name").to_s.strip, + is_credit_note: total_ttc < 0 + ) + end + + def fetch_thirdparty_names + result = {} + page = 0 + loop do + batch = @client.get("/thirdparties", limit: 100, page: page) + break if batch.nil? || !batch.is_a?(Array) || batch.empty? + batch.each { |tp| result[tp["id"].to_i] = tp["name"].to_s.strip } + break if batch.size < 100 + page += 1 + end + result + end + + def parse_unix_date(value) + return nil if value.nil? || value.to_s.strip.empty? || value.to_i.zero? + Time.at(value.to_i).utc.to_date + rescue + nil + end + end +end diff --git a/lib/reconciliation/engine.rb b/lib/reconciliation/engine.rb index 92f7a76..afefcb9 100644 --- a/lib/reconciliation/engine.rb +++ b/lib/reconciliation/engine.rb @@ -1,243 +1,322 @@ -# frozen_string_literal: true - -require "date" - -module Reconciliation - class Engine - # Match flags - MATCHED = :matched # GC paid + Dolibarr paid — all good - GC_PAID_DOLIBARR_OPEN = :gc_paid_dolibarr_open # GC collected, Dolibarr still open — FIX NEEDED - GC_FAILED = :gc_failed # GC payment failed - GC_CANCELLED = :gc_cancelled # GC payment cancelled (no action needed) - GC_NO_INVOICE = :gc_no_invoice # GC payment with no Dolibarr invoice found - DOLIBARR_PAID_NO_GC = :dolibarr_paid_no_gc # Dolibarr paid, no GC match — verify manually - DOLIBARR_OPEN_NO_GC = :dolibarr_open_no_gc # Dolibarr open, no GC payment found (overdue) - - # Payout flags - PAYOUT_VERIFIED = :payout_verified - PAYOUT_MISSING = :payout_missing - PAYOUT_AMOUNT_MISMATCH = :payout_amount_mismatch - - DATE_TOLERANCE = 7 # days for GC charge_date ↔ Dolibarr invoice date soft match - PAYOUT_DATE_TOLERANCE = 2 # days for GC payout_date ↔ Shine credit date - - # Result structs - Match = Struct.new( - :flag, - :invoice, # DolibarrFetcher::Invoice or nil - :payment, # GocardlessParser::Payment or nil - :match_type, # :strong, :soft, or nil - keyword_init: true - ) - - PayoutMatch = Struct.new( - :flag, - :payout_id, - :payout_date, - :expected_amount_cents, - :shine_transaction, # ShineParser::Transaction or nil - :actual_amount_cents, - :gc_payment_ids, - keyword_init: true - ) - - def initialize(dolibarr_invoices:, gc_payments:, shine_transactions:, from:, to:) - @invoices = dolibarr_invoices - @gc_payments = gc_payments - @shine = shine_transactions - @from = from - @to = to - end - - def run - matches = pass1_gc_vs_dolibarr - overdue = pass2_overdue_open_invoices(matches) - payout_matches = pass3_payouts_vs_shine - - { - matches: matches, - matched: matches.select { |m| m.flag == MATCHED }, - gc_paid_dolibarr_open: matches.select { |m| m.flag == GC_PAID_DOLIBARR_OPEN }, - gc_failed: matches.select { |m| m.flag == GC_FAILED }, - gc_cancelled: matches.select { |m| m.flag == GC_CANCELLED }, - gc_no_invoice: matches.select { |m| m.flag == GC_NO_INVOICE }, - dolibarr_paid_no_gc: matches.select { |m| m.flag == DOLIBARR_PAID_NO_GC }, - dolibarr_open_no_gc: overdue, - payout_matches: payout_matches - } - end - - private - - # ------------------------------------------------------------------------- - # Pass 1 — Match each GoCardless payment to a Dolibarr invoice - # ------------------------------------------------------------------------- - def pass1_gc_vs_dolibarr - matched_invoice_ids = [] - results = [] - - @gc_payments.each do |payment| - invoice, match_type = find_invoice(payment) - - if invoice - matched_invoice_ids << invoice.id - end - - flag = determine_flag(payment, invoice) - results << Match.new(flag: flag, invoice: invoice, payment: payment, match_type: match_type) - end - - # Dolibarr paid invoices in the date range with no GC match. - # Scoped to the date range to avoid flagging every historical paid invoice. - matched_set = matched_invoice_ids.to_set - @invoices.each do |inv| - next if matched_set.include?(inv.id) - next unless inv.status == DolibarrFetcher::STATUS_PAID - # Only flag paid invoices whose date falls within the reconciliation window - next if inv.date && (inv.date < @from || inv.date > @to) - - results << Match.new( - flag: DOLIBARR_PAID_NO_GC, - invoice: inv, - payment: nil, - match_type: nil - ) - end - - results - end - - # ------------------------------------------------------------------------- - # Pass 2 — Flag open Dolibarr invoices with no GC match that are overdue - # ------------------------------------------------------------------------- - def pass2_overdue_open_invoices(matches) - matched_invoice_ids = matches.filter_map { |m| m.invoice&.id }.to_set - - @invoices.select do |inv| - inv.status == DolibarrFetcher::STATUS_VALIDATED && - !matched_invoice_ids.include?(inv.id) - end - end - - # ------------------------------------------------------------------------- - # Pass 3 — Match GoCardless payouts to Shine bank credits - # ------------------------------------------------------------------------- - def pass3_payouts_vs_shine - return [] if @shine.empty? - - gc_credits = ShineParser.gocardless_credits(@shine) - - # Group paid_out payments by payout_id. - # Only paid_out payments are included in a payout; failed/cancelled are not. - by_payout = @gc_payments - .select { |p| p.status == "paid_out" && p.payout_id && !p.payout_id.empty? && p.payout_date } - .group_by(&:payout_id) - - used_shine_ids = [] - - by_payout.filter_map do |payout_id, payments| - payout_date = payments.map(&:payout_date).compact.max - expected_cents = payments.sum(&:amount_cents) - - shine_tx = gc_credits.find do |tx| - !used_shine_ids.include?(tx.id) && - (tx.date - payout_date).abs <= PAYOUT_DATE_TOLERANCE && - tx.credit_cents == expected_cents - end - - if shine_tx - used_shine_ids << shine_tx.id - flag = PAYOUT_VERIFIED - actual = shine_tx.credit_cents - else - # Try date match only (amount mismatch — possible GC fees) - shine_tx = gc_credits.find do |tx| - !used_shine_ids.include?(tx.id) && - (tx.date - payout_date).abs <= PAYOUT_DATE_TOLERANCE - end - - if shine_tx - used_shine_ids << shine_tx.id - flag = PAYOUT_AMOUNT_MISMATCH - actual = shine_tx.credit_cents - else - flag = PAYOUT_MISSING - actual = 0 - end - end - - PayoutMatch.new( - flag: flag, - payout_id: payout_id, - payout_date: payout_date, - expected_amount_cents: expected_cents, - shine_transaction: shine_tx, - actual_amount_cents: actual, - gc_payment_ids: payments.map(&:id) - ) - end - end - - # ------------------------------------------------------------------------- - # Invoice lookup - # ------------------------------------------------------------------------- - def find_invoice(payment) - # 1. Strong match: GC description == Dolibarr invoice ref - # Applies when the user puts the invoice ref in the GC payment description. - invoice = @invoices.find do |inv| - !payment.description.empty? && - inv.ref.casecmp(payment.description) == 0 - end - return [invoice, :strong] if invoice - - # 2. Soft match: same amount + customer name + date within tolerance. - # The user creates invoices after GC payment succeeds, so the invoice date - # should be close to the GC charge_date. Since multiple customers pay the - # same amount, customer name matching is required to avoid false positives. - invoice = @invoices.find do |inv| - inv.amount_cents == payment.amount_cents && - inv.date && - (inv.date - payment.charge_date).abs <= DATE_TOLERANCE && - names_match?(inv.customer_name, payment.customer_name) - end - return [invoice, :soft] if invoice - - [nil, nil] - end - - # Normalise a name for comparison: remove accents, lowercase, collapse spaces. - def normalize_name(name) - name.to_s - .unicode_normalize(:nfd) - .gsub(/\p{Mn}/, "") # strip combining diacritical marks - .downcase - .gsub(/[^a-z0-9 ]/, "") - .split - .sort - .join(" ") - end - - def names_match?(dolibarr_name, gc_name) - return false if dolibarr_name.to_s.strip.empty? || gc_name.to_s.strip.empty? - # Both names normalised and compared as sorted word sets so - # "DUPONT Jean" matches "Jean DUPONT" and accent variants. - normalize_name(dolibarr_name) == normalize_name(gc_name) - end - - def determine_flag(payment, invoice) - if GocardlessParser::COLLECTED_STATUSES.include?(payment.status) - if invoice.nil? - GC_NO_INVOICE - elsif invoice.status == DolibarrFetcher::STATUS_PAID - MATCHED - else - GC_PAID_DOLIBARR_OPEN - end - elsif GocardlessParser::FAILED_STATUSES.include?(payment.status) - GC_FAILED - else - GC_CANCELLED - end - end - end -end +# frozen_string_literal: true + +require "date" + +module Reconciliation + class Engine + # Match flags + MATCHED = :matched # GC paid + Dolibarr paid — all good + GC_PAID_DOLIBARR_OPEN = :gc_paid_dolibarr_open # GC collected, Dolibarr still open — FIX NEEDED + GC_FAILED = :gc_failed # GC payment failed + GC_CANCELLED = :gc_cancelled # GC payment cancelled (no action needed) + GC_NO_INVOICE = :gc_no_invoice # GC payment with no Dolibarr invoice found + DOLIBARR_PAID_NO_GC = :dolibarr_paid_no_gc # Dolibarr paid, no GC match — verify manually + DOLIBARR_OPEN_NO_GC = :dolibarr_open_no_gc # Dolibarr open, no GC payment found (overdue) + + # Payout flags + PAYOUT_VERIFIED = :payout_verified + PAYOUT_MISSING = :payout_missing + PAYOUT_AMOUNT_MISMATCH = :payout_amount_mismatch + + DATE_TOLERANCE = ENV.fetch("RECONCILIATION_DATE_TOLERANCE", 7).to_i # days for GC charge_date ↔ Dolibarr invoice date soft match + PAYOUT_DATE_TOLERANCE = ENV.fetch("RECONCILIATION_PAYOUT_TOLERANCE", 2).to_i # days for GC payout_date ↔ Shine credit date + + # Result structs + Match = Struct.new( + :flag, + :invoice, # DolibarrFetcher::Invoice or nil + :payment, # GocardlessParser::Payment or nil + :match_type, # :strong, :soft, or nil + :partial, # true if invoice is partially paid (remain_to_pay > 0) + :retry_group, # Array of GC payment IDs for same invoice (retry detection) + keyword_init: true + ) + + PayoutMatch = Struct.new( + :flag, + :payout_id, + :payout_date, + :expected_amount_cents, # Net amount (what should land in bank) + :shine_transaction, # ShineParser::Transaction or nil + :actual_amount_cents, # What was actually found in Shine + :gc_payment_ids, + :fee_cents, # GC fee deducted (nil if unknown) + :gross_amount_cents, # Sum of underlying payments (before fees) + keyword_init: true + ) + + def initialize(dolibarr_invoices:, gc_payments:, shine_transactions:, from:, to:, gc_payouts: []) + @invoices = dolibarr_invoices + @gc_payments = gc_payments + @gc_payouts = gc_payouts + @shine = shine_transactions + @from = from + @to = to + end + + def run + matches = pass1_gc_vs_dolibarr + overdue = pass2_overdue_open_invoices(matches) + payout_matches = pass3_payouts_vs_shine + + { + matches: matches, + matched: matches.select { |m| m.flag == MATCHED }, + gc_paid_dolibarr_open: matches.select { |m| m.flag == GC_PAID_DOLIBARR_OPEN }, + gc_failed: matches.select { |m| m.flag == GC_FAILED }, + gc_cancelled: matches.select { |m| m.flag == GC_CANCELLED }, + gc_no_invoice: matches.select { |m| m.flag == GC_NO_INVOICE }, + dolibarr_paid_no_gc: matches.select { |m| m.flag == DOLIBARR_PAID_NO_GC }, + dolibarr_open_no_gc: overdue, + payout_matches: payout_matches + } + end + + private + + # ------------------------------------------------------------------------- + # Pass 1 — Match each GoCardless payment to a Dolibarr invoice + # ------------------------------------------------------------------------- + def pass1_gc_vs_dolibarr + matched_invoice_ids = [] + results = [] + + # Build a map of invoice ref -> payments for retry detection + payments_by_invoice_ref = @gc_payments.group_by { |p| p.description.downcase } + + @gc_payments.each do |payment| + invoice, match_type = find_invoice(payment) + + if invoice + matched_invoice_ids << invoice.id + end + + flag = determine_flag(payment, invoice) + + # Detect retries: multiple GC payments for the same invoice ref + retry_group = [] + if invoice && !invoice.ref.nil? + ref_payments = payments_by_invoice_ref[invoice.ref.downcase] || [] + retry_group = ref_payments.map(&:id) if ref_payments.size > 1 + end + + # Detect partial payments + partial = invoice && invoice.remain_to_pay > 0 && invoice.remain_to_pay < invoice.amount_cents / 100.0 + + results << Match.new( + flag: flag, + invoice: invoice, + payment: payment, + match_type: match_type, + partial: partial, + retry_group: retry_group + ) + end + + # Dolibarr paid invoices in the date range with no GC match. + # Scoped to the date range to avoid flagging every historical paid invoice. + matched_set = matched_invoice_ids.to_set + @invoices.each do |inv| + next if matched_set.include?(inv.id) + next unless inv.status == DolibarrFetcher::STATUS_PAID + # Only flag paid invoices whose date falls within the reconciliation window + next if inv.date && (inv.date < @from || inv.date > @to) + + results << Match.new( + flag: DOLIBARR_PAID_NO_GC, + invoice: inv, + payment: nil, + match_type: nil, + partial: false, + retry_group: [] + ) + end + + results + end + + # ------------------------------------------------------------------------- + # Pass 2 — Flag open Dolibarr invoices with no GC match that are overdue + # ------------------------------------------------------------------------- + def pass2_overdue_open_invoices(matches) + matched_invoice_ids = matches.filter_map { |m| m.invoice&.id }.to_set + + @invoices.select do |inv| + inv.status == DolibarrFetcher::STATUS_VALIDATED && + !matched_invoice_ids.include?(inv.id) + end + end + + # ------------------------------------------------------------------------- + # Pass 3 — Match GoCardless payouts to Shine bank credits + # + # When a payouts CSV is provided (preferred): match by payout reference and + # net amount — both are exact. Fee breakdown is taken from the payouts CSV. + # + # When no payouts CSV is provided (fallback): group payments by payout_id, + # sum amounts, and match Shine by date only (amount will differ due to fees). + # ------------------------------------------------------------------------- + def pass3_payouts_vs_shine + return [] if @shine.empty? + + gc_credits = ShineParser.gocardless_credits(@shine) + return [] if gc_credits.empty? + + if @gc_payouts.any? + pass3_with_payouts_csv(gc_credits) + else + pass3_from_payments_csv(gc_credits) + end + end + + def pass3_with_payouts_csv(gc_credits) + used_shine_ids = [] + + # Only consider payouts whose arrival_date falls within the reconciliation window. + payouts_in_range = @gc_payouts.select do |p| + p.arrival_date >= @from && p.arrival_date <= @to + end + + payouts_in_range.map do |payout| + # 1. Strong match: payout reference found in Shine Libellé + shine_tx = gc_credits.find do |tx| + !used_shine_ids.include?(tx.id) && + tx.label.include?(payout.reference) + end + + # 2. Fallback: exact net amount + date within tolerance + if shine_tx.nil? + shine_tx = gc_credits.find do |tx| + !used_shine_ids.include?(tx.id) && + tx.credit_cents == payout.amount_cents && + (tx.date - payout.arrival_date).abs <= PAYOUT_DATE_TOLERANCE + end + end + + if shine_tx + used_shine_ids << shine_tx.id + flag = PAYOUT_VERIFIED + else + flag = PAYOUT_MISSING + end + + PayoutMatch.new( + flag: flag, + payout_id: payout.id, + payout_date: payout.arrival_date, + expected_amount_cents: payout.amount_cents, + shine_transaction: shine_tx, + actual_amount_cents: shine_tx&.credit_cents || 0, + gc_payment_ids: [], + fee_cents: payout.fee_cents, + gross_amount_cents: payout.total_payment_cents + ) + end + end + + def pass3_from_payments_csv(gc_credits) + used_shine_ids = [] + + by_payout = @gc_payments + .select { |p| p.status == "paid_out" && p.payout_id && !p.payout_id.empty? && p.payout_date } + .group_by(&:payout_id) + + by_payout.filter_map do |payout_id, payments| + payout_date = payments.map(&:payout_date).compact.max + gross_cents = payments.sum(&:amount_cents) + + # Without the payouts CSV we don't know the exact net amount, so match + # by date only and accept any credit from GOCARDLESS SAS that day. + shine_tx = gc_credits.find do |tx| + !used_shine_ids.include?(tx.id) && + (tx.date - payout_date).abs <= PAYOUT_DATE_TOLERANCE + end + + if shine_tx + used_shine_ids << shine_tx.id + flag = shine_tx.credit_cents == gross_cents ? PAYOUT_VERIFIED : PAYOUT_AMOUNT_MISMATCH + actual = shine_tx.credit_cents + fee_cents = gross_cents - actual + else + flag = PAYOUT_MISSING + actual = 0 + fee_cents = nil + end + + PayoutMatch.new( + flag: flag, + payout_id: payout_id, + payout_date: payout_date, + expected_amount_cents: gross_cents, + shine_transaction: shine_tx, + actual_amount_cents: actual, + gc_payment_ids: payments.map(&:id), + fee_cents: fee_cents, + gross_amount_cents: gross_cents + ) + end + end + + # ------------------------------------------------------------------------- + # Invoice lookup + # ------------------------------------------------------------------------- + def find_invoice(payment) + # 1. Strong match: GC description == Dolibarr invoice ref + # Applies when the user puts the invoice ref in the GC payment description. + invoice = @invoices.find do |inv| + !payment.description.empty? && + inv.ref.casecmp(payment.description) == 0 + end + return [invoice, :strong] if invoice + + # 2. Soft match: same amount + customer name + date within tolerance. + # The user creates invoices after GC payment succeeds, so the invoice date + # should be close to the GC charge_date. Since multiple customers pay the + # same amount, customer name matching is required to avoid false positives. + invoice = @invoices.find do |inv| + inv.amount_cents == payment.amount_cents && + inv.date && + (inv.date - payment.charge_date).abs <= DATE_TOLERANCE && + names_match?(inv.customer_name, payment.customer_name) + end + return [invoice, :soft] if invoice + + [nil, nil] + end + + # Normalise a name for comparison: remove accents, lowercase, collapse spaces. + def normalize_name(name) + name.to_s + .unicode_normalize(:nfd) + .gsub(/\p{Mn}/, "") # strip combining diacritical marks + .downcase + .gsub(/[^a-z0-9 ]/, "") + .split + .sort + .join(" ") + end + + def names_match?(dolibarr_name, gc_name) + return false if dolibarr_name.to_s.strip.empty? || gc_name.to_s.strip.empty? + # Both names normalised and compared as sorted word sets so + # "DUPONT Jean" matches "Jean DUPONT" and accent variants. + normalize_name(dolibarr_name) == normalize_name(gc_name) + end + + def determine_flag(payment, invoice) + if GocardlessParser::COLLECTED_STATUSES.include?(payment.status) + if invoice.nil? + GC_NO_INVOICE + elsif invoice.status == DolibarrFetcher::STATUS_PAID + MATCHED + else + GC_PAID_DOLIBARR_OPEN + end + elsif GocardlessParser::FAILED_STATUSES.include?(payment.status) + GC_FAILED + else + GC_CANCELLED + end + end + end +end diff --git a/lib/reconciliation/gocardless_payouts_parser.rb b/lib/reconciliation/gocardless_payouts_parser.rb new file mode 100644 index 0000000..60f741c --- /dev/null +++ b/lib/reconciliation/gocardless_payouts_parser.rb @@ -0,0 +1,64 @@ +# frozen_string_literal: true + +require "csv" +require "date" + +module Reconciliation + class GocardlessPayoutsParser + Payout = Struct.new( + :id, + :amount_cents, # Net amount transferred to bank (after all fees) + :total_payment_cents, # Gross sum of underlying payments (before fees) + :fee_cents, # Total fee deducted (total_payment - amount) + :transaction_fee_cents, # Per-transaction fee component + :surcharge_fee_cents, # Surcharge fee component + :tax_cents, # Tax on fees (informational, already in total fee) + :reference, # e.g. "CYANET-DKV4KN8FTM2" — appears in Shine Libellé + :status, + :arrival_date, + keyword_init: true + ) + + def self.parse(csv_path) + rows = CSV.read(csv_path, headers: true, encoding: "UTF-8") + + payouts = rows.filter_map do |row| + next if row["status"] != "paid" + + arrival_date = safe_parse_date(row["arrival_date"]) + next unless arrival_date + + amount_cents = to_cents(row["amount"]) + total_payment_cents = to_cents(row["total_payment_amount"]) + + Payout.new( + id: row["id"].to_s.strip, + amount_cents: amount_cents, + total_payment_cents: total_payment_cents, + fee_cents: total_payment_cents - amount_cents, + transaction_fee_cents: to_cents(row["transaction_fee_debit"]), + surcharge_fee_cents: to_cents(row["surcharge_fee_debit"]), + tax_cents: to_cents(row["tax_debit"]), + reference: row["reference"].to_s.strip, + status: row["status"].to_s.strip, + arrival_date: arrival_date + ) + end + + $stderr.puts "[GocardlessPayoutsParser] Loaded #{payouts.size} paid payouts from #{csv_path}" + payouts + end + + private_class_method def self.to_cents(str) + return 0 if str.nil? || str.strip.empty? + (str.strip.to_f * 100).round + end + + private_class_method def self.safe_parse_date(str) + return nil if str.nil? || str.strip.empty? + Date.parse(str.strip) + rescue Date::Error, ArgumentError + nil + end + end +end diff --git a/lib/reconciliation/reporter.rb b/lib/reconciliation/reporter.rb index 3b06406..aa28075 100644 --- a/lib/reconciliation/reporter.rb +++ b/lib/reconciliation/reporter.rb @@ -1,203 +1,269 @@ -# frozen_string_literal: true - -require "csv" -require "date" - -module Reconciliation - class Reporter - def initialize(result, from:, to:) - @result = result - @from = from - @to = to - end - - def print_summary - r = @result - puts "" - puts "=" * 60 - puts " RECONCILIATION REPORT: #{@from} to #{@to}" - puts "=" * 60 - - # --- Dolibarr overview --- - total_invoices = r[:matches].filter_map(&:invoice).uniq(&:id).size + - r[:dolibarr_open_no_gc].size - open_count = r[:dolibarr_open_no_gc].size + - r[:gc_paid_dolibarr_open].size - puts "" - puts "DOLIBARR" - puts " Total invoices in scope: #{total_invoices}" - puts " Open (no GC match): #{r[:dolibarr_open_no_gc].size}#{r[:dolibarr_open_no_gc].any? ? ' ← needs attention' : ''}" - puts " Paid (GC matched): #{r[:matched].size}" - - # --- GoCardless ↔ Dolibarr --- - puts "" - puts "GOCARDLESS ↔ DOLIBARR" - puts " Matched (paid both sides): #{r[:matched].size} #{'✓' if r[:matched].any?}" - puts " GC paid / Dolibarr open: #{r[:gc_paid_dolibarr_open].size}#{r[:gc_paid_dolibarr_open].any? ? ' ← FIX with --fix' : ''}" - puts " Dolibarr paid / no GC: #{r[:dolibarr_paid_no_gc].size}#{r[:dolibarr_paid_no_gc].any? ? ' ← verify manually' : ''}" - puts " GC failed: #{r[:gc_failed].size}" - puts " GC cancelled: #{r[:gc_cancelled].size}" - puts " GC payment / no invoice: #{r[:gc_no_invoice].size}#{r[:gc_no_invoice].any? ? ' ← investigate' : ''}" - - # --- Shine ↔ GoCardless Payouts --- - unless @result[:payout_matches].empty? - pm = r[:payout_matches] - verified = pm.count { |p| p.flag == Engine::PAYOUT_VERIFIED } - mismatch = pm.count { |p| p.flag == Engine::PAYOUT_AMOUNT_MISMATCH } - missing = pm.count { |p| p.flag == Engine::PAYOUT_MISSING } - expected = pm.sum(&:expected_amount_cents) - actual = pm.sum(&:actual_amount_cents) - diff = actual - expected - - puts "" - puts "SHINE ↔ GOCARDLESS PAYOUTS" - puts " Payouts expected: #{pm.size}" - puts " Verified: #{verified}#{verified == pm.size ? ' ✓' : ''}" - puts " Amount mismatch: #{mismatch}#{mismatch > 0 ? ' ← check GC fees' : ''}" - puts " Missing in Shine: #{missing}#{missing > 0 ? ' ← investigate' : ''}" - puts " Expected total: #{format_eur(expected)}" - puts " Actual total: #{format_eur(actual)}" - puts " Difference: #{format_eur(diff)}#{diff.zero? ? ' ✓' : ' ← investigate'}" - end - - # --- Action items --- - actions = r[:gc_paid_dolibarr_open] + - r[:dolibarr_paid_no_gc] + - r[:dolibarr_open_no_gc] + - r[:gc_no_invoice] - - if actions.any? - puts "" - puts "ACTIONS NEEDED (#{actions.size})" - puts "-" * 60 - - r[:gc_paid_dolibarr_open].each_with_index do |m, i| - puts " #{i + 1}. [GC_PAID_DOLIBARR_OPEN] " \ - "#{m.invoice.ref.ljust(16)} " \ - "#{format_eur(m.invoice.amount_cents)} " \ - "#{display_name(m.invoice).ljust(20)} " \ - "GC: #{m.payment.id} #{m.payment.charge_date} (#{m.match_type})" - end - - r[:dolibarr_paid_no_gc].each_with_index do |m, i| - n = r[:gc_paid_dolibarr_open].size + i + 1 - puts " #{n}. [DOLIBARR_PAID_NO_GC] " \ - "#{m.invoice.ref.ljust(16)} " \ - "#{format_eur(m.invoice.amount_cents)} " \ - "#{display_name(m.invoice).ljust(20)} " \ - "No GoCardless payment found" - end - - base = r[:gc_paid_dolibarr_open].size + r[:dolibarr_paid_no_gc].size - r[:dolibarr_open_no_gc].each_with_index do |inv, i| - n = base + i + 1 - overdue = inv.due_date && inv.due_date < Date.today ? " (overdue since #{inv.due_date})" : "" - puts " #{n}. [DOLIBARR_OPEN_NO_GC] " \ - "#{inv.ref.ljust(16)} " \ - "#{format_eur(inv.amount_cents)} " \ - "#{display_name(inv).ljust(20)} " \ - "Open, no GC payment#{overdue}" - end - - base2 = base + r[:dolibarr_open_no_gc].size - r[:gc_no_invoice].each_with_index do |m, i| - n = base2 + i + 1 - puts " #{n}. [GC_NO_INVOICE] " \ - "GC: #{m.payment.id} #{format_eur(m.payment.amount_cents)} " \ - "\"#{m.payment.description}\" #{m.payment.customer_name} #{m.payment.charge_date}" - end - - r[:payout_matches].select { |p| p.flag == Engine::PAYOUT_MISSING }.each do |pm| - n = base2 + r[:gc_no_invoice].size + 1 - puts " #{n}. [PAYOUT_MISSING] " \ - "Payout #{pm.payout_id} #{format_eur(pm.expected_amount_cents)} " \ - "expected #{pm.payout_date}" - end - else - puts "" - puts " All clear — no actions needed." - end - - puts "" - puts "=" * 60 - - csv_path = write_csv - puts " Report saved to: #{csv_path}" - puts "" - end - - def write_csv - dir = "tmp" - Dir.mkdir(dir) unless Dir.exist?(dir) - path = "#{dir}/reconciliation_#{@to}.csv" - - CSV.open(path, "w", encoding: "UTF-8") do |csv| - csv << %w[ - invoice_ref customer_name amount_eur invoice_date due_date - dolibarr_status gc_payment_id gc_status gc_charge_date - match_type flag action - ] - - @result[:matches].each do |m| - inv = m.invoice - pay = m.payment - csv << [ - inv&.ref, - inv&.customer_name, - inv ? "%.2f" % (inv.amount_cents / 100.0) : nil, - inv&.date, - inv&.due_date, - inv ? status_label(inv.status) : nil, - pay&.id, - pay&.status, - pay&.charge_date, - m.match_type, - m.flag, - action_label(m.flag) - ] - end - - @result[:dolibarr_open_no_gc].each do |inv| - csv << [ - inv.ref, inv.customer_name, - "%.2f" % (inv.amount_cents / 100.0), - inv.date, inv.due_date, - status_label(inv.status), - nil, nil, nil, nil, - Engine::DOLIBARR_OPEN_NO_GC, - action_label(Engine::DOLIBARR_OPEN_NO_GC) - ] - end - end - - path - end - - private - - def display_name(inv) - name = inv.customer_name.to_s.strip - name.empty? ? "client_id:#{inv.customer_id}" : name - end - - def format_eur(cents) - "€#{"%.2f" % (cents / 100.0)}" - end - - def status_label(status) - { 0 => "draft", 1 => "open", 2 => "paid", 3 => "cancelled" }[status] || status.to_s - end - - def action_label(flag) - { - Engine::MATCHED => "none", - Engine::GC_PAID_DOLIBARR_OPEN => "mark_dolibarr_paid", - Engine::GC_FAILED => "verify", - Engine::GC_CANCELLED => "none", - Engine::GC_NO_INVOICE => "investigate", - Engine::DOLIBARR_PAID_NO_GC => "verify_manually", - Engine::DOLIBARR_OPEN_NO_GC => "follow_up" - }[flag] || flag.to_s - end - end -end +# frozen_string_literal: true + +require "csv" +require "date" + +module Reconciliation + class Reporter + def initialize(result, from:, to:) + @result = result + @from = from + @to = to + end + + def print_summary + r = @result + puts "" + puts "=" * 60 + puts " RECONCILIATION REPORT: #{@from} to #{@to}" + puts "=" * 60 + + # --- Dolibarr overview --- + total_invoices = r[:matches].filter_map(&:invoice).uniq(&:id).size + + r[:dolibarr_open_no_gc].size + open_count = r[:dolibarr_open_no_gc].size + + r[:gc_paid_dolibarr_open].size + puts "" + puts "DOLIBARR" + puts " Total invoices in scope: #{total_invoices}" + puts " Open (no GC match): #{r[:dolibarr_open_no_gc].size}#{r[:dolibarr_open_no_gc].any? ? ' ← needs attention' : ''}" + puts " Paid (GC matched): #{r[:matched].size}" + + # --- GoCardless ↔ Dolibarr --- + puts "" + puts "GOCARDLESS ↔ DOLIBARR" + puts " Matched (paid both sides): #{r[:matched].size} #{'✓' if r[:matched].any?}" + puts " GC paid / Dolibarr open: #{r[:gc_paid_dolibarr_open].size}#{r[:gc_paid_dolibarr_open].any? ? ' ← FIX with --fix' : ''}" + puts " Dolibarr paid / no GC: #{r[:dolibarr_paid_no_gc].size}#{r[:dolibarr_paid_no_gc].any? ? ' ← verify manually' : ''}" + puts " GC failed: #{r[:gc_failed].size}" + puts " GC cancelled: #{r[:gc_cancelled].size}" + puts " GC payment / no invoice: #{r[:gc_no_invoice].size}#{r[:gc_no_invoice].any? ? ' ← investigate' : ''}" + + # Partial payment info + partial_matches = r[:matches].select { |m| m.partial } + if partial_matches.any? + puts " Partial payments: #{partial_matches.size} ← check remain_to_pay" + end + + # Retry detection info + retry_matches = r[:matches].select { |m| m.retry_group && m.retry_group.size > 1 } + if retry_matches.any? + puts " Retries detected: #{retry_matches.size} ← multiple GC attempts for same invoice" + end + + # --- Shine ↔ GoCardless Payouts --- + unless @result[:payout_matches].empty? + pm = r[:payout_matches] + verified = pm.count { |p| p.flag == Engine::PAYOUT_VERIFIED } + mismatch = pm.count { |p| p.flag == Engine::PAYOUT_AMOUNT_MISMATCH } + missing = pm.count { |p| p.flag == Engine::PAYOUT_MISSING } + total_net = pm.sum(&:expected_amount_cents) + known_fees = pm.filter_map(&:fee_cents) + total_fees = known_fees.sum + + puts "" + puts "SHINE ↔ GOCARDLESS PAYOUTS" + puts " Payouts: #{pm.size}" + puts " Verified: #{verified}#{verified == pm.size ? ' ✓' : ''}" + puts " Amount mismatch: #{mismatch}#{mismatch > 0 ? ' ← check GC fees' : ''}" if mismatch > 0 + puts " Missing in Shine: #{missing} ← investigate" if missing > 0 + puts " Net received: #{format_eur(total_net)}" + if known_fees.any? + gross = pm.filter_map(&:gross_amount_cents).sum + puts " Gross collected: #{format_eur(gross)}" + puts " GC fees: #{format_eur(total_fees)}" + # Per-payout fee detail + pm.select { |p| p.fee_cents && p.fee_cents > 0 }.each do |p| + puts " #{p.payout_id} #{p.payout_date} net=#{format_eur(p.expected_amount_cents)} " \ + "gross=#{format_eur(p.gross_amount_cents)} fee=#{format_eur(p.fee_cents)}" + end + end + end + + # --- Action items --- + actions = r[:gc_paid_dolibarr_open] + + r[:dolibarr_paid_no_gc] + + r[:dolibarr_open_no_gc] + + r[:gc_no_invoice] + + if actions.any? + puts "" + puts "ACTIONS NEEDED (#{actions.size})" + puts "-" * 60 + + r[:gc_paid_dolibarr_open].each_with_index do |m, i| + extra = [] + extra << "PARTIAL" if m.partial + extra << "RETRY: #{m.retry_group.join(', ')}" if m.retry_group && m.retry_group.size > 1 + extra_str = extra.any? ? " [#{extra.join(', ')}]" : "" + + puts " #{i + 1}. [GC_PAID_DOLIBARR_OPEN] " \ + "#{m.invoice.ref.ljust(16)} " \ + "#{format_eur(m.invoice.amount_cents)} " \ + "#{display_name(m.invoice).ljust(20)} " \ + "GC: #{m.payment.id} #{m.payment.charge_date} (#{m.match_type})#{extra_str}" + end + + r[:dolibarr_paid_no_gc].each_with_index do |m, i| + n = r[:gc_paid_dolibarr_open].size + i + 1 + puts " #{n}. [DOLIBARR_PAID_NO_GC] " \ + "#{m.invoice.ref.ljust(16)} " \ + "#{format_eur(m.invoice.amount_cents)} " \ + "#{display_name(m.invoice).ljust(20)} " \ + "No GoCardless payment found" + end + + base = r[:gc_paid_dolibarr_open].size + r[:dolibarr_paid_no_gc].size + r[:dolibarr_open_no_gc].each_with_index do |inv, i| + n = base + i + 1 + overdue = inv.due_date && inv.due_date < Date.today ? " (overdue since #{inv.due_date})" : "" + puts " #{n}. [DOLIBARR_OPEN_NO_GC] " \ + "#{inv.ref.ljust(16)} " \ + "#{format_eur(inv.amount_cents)} " \ + "#{display_name(inv).ljust(20)} " \ + "Open, no GC payment#{overdue}" + end + + base2 = base + r[:dolibarr_open_no_gc].size + r[:gc_no_invoice].each_with_index do |m, i| + n = base2 + i + 1 + puts " #{n}. [GC_NO_INVOICE] " \ + "GC: #{m.payment.id} #{format_eur(m.payment.amount_cents)} " \ + "\"#{m.payment.description}\" #{m.payment.customer_name} #{m.payment.charge_date}" + end + + r[:payout_matches].select { |p| p.flag == Engine::PAYOUT_MISSING }.each do |pm| + n = base2 + r[:gc_no_invoice].size + 1 + puts " #{n}. [PAYOUT_MISSING] " \ + "Payout #{pm.payout_id} #{format_eur(pm.expected_amount_cents)} " \ + "expected #{pm.payout_date}" + end + else + puts "" + puts " All clear — no actions needed." + end + + puts "" + puts "=" * 60 + + csv_path = write_csv + puts " Report saved to: #{csv_path}" + + unless @result[:payout_matches].empty? + payout_csv_path = write_payouts_csv + puts " Payout fees saved to: #{payout_csv_path}" + end + puts "" + end + + def write_csv + dir = "tmp" + Dir.mkdir(dir) unless Dir.exist?(dir) + path = "#{dir}/reconciliation_#{@to}.csv" + + CSV.open(path, "w", encoding: "UTF-8") do |csv| + csv << %w[ + invoice_ref customer_name amount_eur invoice_date due_date + dolibarr_status gc_payment_id gc_status gc_charge_date + match_type flag action partial retry_group + ] + + @result[:matches].each do |m| + inv = m.invoice + pay = m.payment + csv << [ + inv&.ref, + inv&.customer_name, + inv ? "%.2f" % (inv.amount_cents / 100.0) : nil, + inv&.date, + inv&.due_date, + inv ? status_label(inv.status) : nil, + pay&.id, + pay&.status, + pay&.charge_date, + m.match_type, + m.flag, + action_label(m.flag), + m.partial ? "yes" : "no", + m.retry_group&.join(", ") + ] + end + + @result[:dolibarr_open_no_gc].each do |inv| + csv << [ + inv.ref, inv.customer_name, + "%.2f" % (inv.amount_cents / 100.0), + inv.date, inv.due_date, + status_label(inv.status), + nil, nil, nil, nil, + Engine::DOLIBARR_OPEN_NO_GC, + action_label(Engine::DOLIBARR_OPEN_NO_GC), + "no", + nil + ] + end + end + + path + end + + def write_payouts_csv + dir = "tmp" + Dir.mkdir(dir) unless Dir.exist?(dir) + path = "#{dir}/payouts_fees_#{@to}.csv" + + CSV.open(path, "w", encoding: "UTF-8") do |csv| + csv << %w[ + payout_id payout_date gross_amount_eur net_amount_eur fee_eur + fee_percentage shine_reference status + ] + + @result[:payout_matches].each do |pm| + fee_pct = pm.gross_amount_cents && pm.gross_amount_cents > 0 \ + ? ((pm.fee_cents.to_f / pm.gross_amount_cents) * 100).round(2) \ + : nil + + csv << [ + pm.payout_id, + pm.payout_date, + pm.gross_amount_cents ? "%.2f" % (pm.gross_amount_cents / 100.0) : nil, + "%.2f" % (pm.expected_amount_cents / 100.0), + pm.fee_cents ? "%.2f" % (pm.fee_cents / 100.0) : nil, + fee_pct ? "#{fee_pct}%" : nil, + pm.shine_transaction&.label, + pm.flag + ] + end + end + + path + end + + private + + def display_name(inv) + name = inv.customer_name.to_s.strip + name.empty? ? "client_id:#{inv.customer_id}" : name + end + + def format_eur(cents) + "€#{"%.2f" % (cents / 100.0)}" + end + + def status_label(status) + { 0 => "draft", 1 => "open", 2 => "paid", 3 => "cancelled" }[status] || status.to_s + end + + def action_label(flag) + { + Engine::MATCHED => "none", + Engine::GC_PAID_DOLIBARR_OPEN => "mark_dolibarr_paid", + Engine::GC_FAILED => "verify", + Engine::GC_CANCELLED => "none", + Engine::GC_NO_INVOICE => "investigate", + Engine::DOLIBARR_PAID_NO_GC => "verify_manually", + Engine::DOLIBARR_OPEN_NO_GC => "follow_up" + }[flag] || flag.to_s + end + end +end