Initial implementation of the reconciliation script
Standalone Ruby script reconciling GoCardless payments, Dolibarr invoices (via API), and Shine bank statements. Three-pass engine: GC↔Dolibarr matching, open invoice audit, payout↔bank verification. Includes dry-run and --fix mode to auto-mark Dolibarr invoices as paid. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Kevin Bataille
18004
docs/dolibarr.json
Normal file
18004
docs/dolibarr.json
Normal file
File diff suppressed because it is too large
Load Diff
269
docs/reconciliation_plan.md
Normal file
269
docs/reconciliation_plan.md
Normal file
@@ -0,0 +1,269 @@
|
||||
# Invoice Reconciliation — Design & Implementation Plan
|
||||
|
||||
## Context
|
||||
|
||||
**Stack:** Ruby (Rails or standalone script)
|
||||
**Purpose:** Reconcile GoCardless payments, Shine bank statements, and Dolibarr invoices to ensure all three systems are in sync.
|
||||
**Billing flow:** Customers are billed manually via GoCardless direct debit. Invoices are managed in Dolibarr. Cash lands in a Shine business bank account.
|
||||
|
||||
---
|
||||
|
||||
## The Problem
|
||||
|
||||
Three systems hold overlapping financial data but are not automatically linked:
|
||||
|
||||
```
|
||||
Shine bank ◄─── GoCardless payouts ◄─── GoCardless payments ◄─── Dolibarr invoices
|
||||
```
|
||||
|
||||
Discrepancies arise when:
|
||||
- A GoCardless payment is collected but Dolibarr is not marked as paid
|
||||
- A Dolibarr invoice is marked paid but no GoCardless payment exists
|
||||
- A GoCardless payout (batch) never landed in the Shine account
|
||||
|
||||
---
|
||||
|
||||
## Data Sources
|
||||
|
||||
| Source | What it contains | How to export |
|
||||
|--------|-----------------|---------------|
|
||||
| **Dolibarr API** | Invoices, payment status, customer info | Fetched live via REST API (`/invoices`) |
|
||||
| **GoCardless payments CSV** | Individual payment ID, amount, date, status, customer | Dashboard → Payments → Export CSV |
|
||||
| **GoCardless payouts CSV** | Payout ID, total amount, arrival date (batched bank transfers) | Dashboard → Payouts → Export CSV |
|
||||
| **Shine bank CSV** | Bank transactions: date, description, amount, balance | App → Comptes → Exporter le relevé |
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
lib/tasks/reconcile.rake ← CLI entry point (rake task)
|
||||
app/services/reconciliation/
|
||||
├── dolibarr_fetcher.rb ← Fetch invoices + payment records via Dolibarr API
|
||||
├── gocardless_parser.rb ← Parse GoCardless payments CSV
|
||||
├── gocardless_payout_parser.rb ← Parse GoCardless payouts CSV
|
||||
├── shine_parser.rb ← Parse Shine bank statement CSV
|
||||
├── engine.rb ← Core matching logic (3 passes)
|
||||
└── reporter.rb ← Terminal summary + CSV output
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Reconciliation Passes
|
||||
|
||||
### Pass 1 — Dolibarr open invoice audit
|
||||
|
||||
- Fetch all Dolibarr invoices with `status=1` (validated/open) for the date range
|
||||
- Flag any that are past their due date: these should have a GoCardless payment but don't
|
||||
- Output: list of overdue open invoices needing action
|
||||
|
||||
### Pass 2 — GoCardless ↔ Dolibarr matching
|
||||
|
||||
For each GoCardless payment in the CSV:
|
||||
|
||||
**Matching strategy (in order of confidence):**
|
||||
1. **Strong match**: `num_payment` field in Dolibarr payment record == GoCardless payment ID
|
||||
2. **Soft match**: `amount_ttc` + `customer_name` + date within ±5 days
|
||||
|
||||
**Flags:**
|
||||
- `GC_PAID_DOLIBARR_OPEN` — GoCardless collected payment, Dolibarr invoice still open → needs `mark_as_paid` call
|
||||
- `DOLIBARR_PAID_NO_GC` — Dolibarr marked paid but no GoCardless payment found → manual payment, verify
|
||||
- `GC_FAILED` — GoCardless payment failed, check if Dolibarr invoice is still open
|
||||
- `MATCHED` — Both sides agree, no action needed
|
||||
|
||||
### Pass 3 — Shine bank ↔ GoCardless payout verification
|
||||
|
||||
- GoCardless batches individual payments into payouts and deposits them as a single bank transfer
|
||||
- Each GoCardless payout should appear as a credit in Shine (description contains "GOCARDLESS" or similar)
|
||||
- Match by: payout arrival date ±2 days + amount
|
||||
|
||||
**Flags:**
|
||||
- `PAYOUT_MISSING` — GoCardless payout not found in Shine bank statement
|
||||
- `AMOUNT_MISMATCH` — Payout found but amounts differ (possible fee deduction issue)
|
||||
- `VERIFIED` — Payout confirmed in bank account
|
||||
|
||||
---
|
||||
|
||||
## Dolibarr API Endpoints Used
|
||||
|
||||
```
|
||||
GET /invoices?status=1&limit=500 # Open/validated invoices
|
||||
GET /invoices?status=2&limit=500 # Paid invoices
|
||||
GET /invoices/{id} # Single invoice with payment history
|
||||
GET /invoices/{id}/payments # Payment records for an invoice
|
||||
POST /invoices/paymentsdistributed # Record a payment (for --fix mode)
|
||||
```
|
||||
|
||||
**Key Dolibarr invoice fields:**
|
||||
- `ref` — Invoice reference (e.g., `FA2601-0034`)
|
||||
- `total_ttc` — Total amount including tax (euros, float)
|
||||
- `paye` — Payment flag: `1` = paid, `0` = unpaid
|
||||
- `sumpayed` — Total amount already paid
|
||||
- `remaintopay` — Remaining balance due
|
||||
- `socid` — Customer (third-party) ID
|
||||
- `status` — `0`=draft, `1`=validated, `2`=paid, `3`=cancelled
|
||||
|
||||
**Dolibarr mark-as-paid payload:**
|
||||
```json
|
||||
{
|
||||
"arrayofamounts": {
|
||||
"{INVOICE_ID}": {
|
||||
"amount": 19.99,
|
||||
"multicurrency_amount": null
|
||||
}
|
||||
},
|
||||
"datepaye": 1748524447,
|
||||
"paymentid": 6,
|
||||
"closepaidinvoices": "yes",
|
||||
"accountid": 1,
|
||||
"num_payment": "PM01234567",
|
||||
"comment": "GoCardless payment — auto-reconciled"
|
||||
}
|
||||
```
|
||||
|
||||
(`paymentid: 6` = GoCardless payment method ID in Dolibarr — verify this value in your instance)
|
||||
|
||||
---
|
||||
|
||||
## GoCardless CSV Format
|
||||
|
||||
GoCardless payment export columns (verify against your actual export):
|
||||
|
||||
```
|
||||
id, amount, currency, status, charge_date, description, reference,
|
||||
customer_id, customer_name, mandate_id, payout_id
|
||||
```
|
||||
|
||||
Relevant statuses: `paid_out`, `confirmed`, `failed`, `cancelled`, `pending_submission`
|
||||
|
||||
GoCardless payout export columns:
|
||||
|
||||
```
|
||||
id, amount, currency, status, arrival_date, reference
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Shine CSV Format
|
||||
|
||||
Shine bank export columns (verify against your actual export):
|
||||
|
||||
```
|
||||
Date, Libellé, Montant, Catégorie, Notes, Solde
|
||||
```
|
||||
|
||||
GoCardless payouts typically appear with `Libellé` containing `GOCARDLESS` or `GC`.
|
||||
|
||||
**Note:** Column names may vary by export language/version. Make the parser configurable.
|
||||
|
||||
---
|
||||
|
||||
## CLI Interface
|
||||
|
||||
```bash
|
||||
# Dry run — report only, no changes
|
||||
bin/rails reconcile:run \
|
||||
FROM=2026-01-01 \
|
||||
TO=2026-01-31 \
|
||||
GC=tmp/gocardless_payments.csv \
|
||||
GC_PAYOUTS=tmp/gocardless_payouts.csv \
|
||||
SHINE=tmp/shine_january.csv
|
||||
|
||||
# Auto-fix — mark Dolibarr invoices as paid where GC payment is confirmed
|
||||
bin/rails reconcile:run \
|
||||
FROM=2026-01-01 \
|
||||
TO=2026-01-31 \
|
||||
GC=tmp/gocardless_payments.csv \
|
||||
GC_PAYOUTS=tmp/gocardless_payouts.csv \
|
||||
SHINE=tmp/shine_january.csv \
|
||||
FIX=true
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Expected Report Output
|
||||
|
||||
```
|
||||
=== RECONCILIATION REPORT: 2026-01-01 to 2026-01-31 ===
|
||||
|
||||
DOLIBARR SUMMARY
|
||||
Total invoices validated: 47
|
||||
Total invoices paid: 43
|
||||
Open and overdue: 4 ← ACTION NEEDED
|
||||
|
||||
GOCARDLESS ↔ DOLIBARR
|
||||
Matched (no action): 43 ✓
|
||||
GC paid / Dolibarr open: 3 ← Dolibarr needs marking as paid
|
||||
Dolibarr paid / no GC: 1 ← Verify manual payment
|
||||
GC failed: 0
|
||||
|
||||
SHINE ↔ GOCARDLESS PAYOUTS
|
||||
Payouts expected: 8
|
||||
Payouts found in Shine: 8 ✓
|
||||
Total amount expected: €842.58
|
||||
Total amount received: €842.58
|
||||
Difference: €0.00 ✓
|
||||
|
||||
ACTIONS NEEDED:
|
||||
1. [GC_PAID_DOLIBARR_OPEN] FA2601-0034 €19.99 DUPONT Jean GC: PM01234567 2026-01-05
|
||||
2. [GC_PAID_DOLIBARR_OPEN] FA2601-0041 €29.99 MARTIN Paul GC: PM05678901 2026-01-12
|
||||
3. [GC_PAID_DOLIBARR_OPEN] FA2601-0052 €19.99 DURAND Marie GC: PM09012345 2026-01-19
|
||||
4. [DOLIBARR_PAID_NO_GC] FA2601-0023 €19.99 LEROY Claude Paid: 2026-01-08 (no GC match)
|
||||
|
||||
Report saved to: tmp/reconciliation_2026-01-31.csv
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Output CSV Format
|
||||
|
||||
```csv
|
||||
invoice_ref,customer_name,amount_ttc,invoice_date,dolibarr_status,gc_payment_id,gc_status,gc_charge_date,match_status,action
|
||||
FA2601-0034,DUPONT Jean,19.99,2026-01-01,open,PM01234567,paid_out,2026-01-05,GC_PAID_DOLIBARR_OPEN,mark_dolibarr_paid
|
||||
FA2601-0041,MARTIN Paul,29.99,2026-01-08,open,PM05678901,paid_out,2026-01-12,GC_PAID_DOLIBARR_OPEN,mark_dolibarr_paid
|
||||
FA2601-0023,LEROY Claude,19.99,2026-01-07,paid,,,,DOLIBARR_PAID_NO_GC,verify_manually
|
||||
FA2601-0012,DUPONT Jean,19.99,2025-12-01,paid,PM98765432,paid_out,2025-12-05,MATCHED,none
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
### Matching tolerance
|
||||
|
||||
- **Amount**: exact match (compare in cents to avoid float issues — multiply Dolibarr's euro amounts by 100)
|
||||
- **Date**: ±5 days between GoCardless `charge_date` and Dolibarr invoice `date`
|
||||
- **Customer**: normalize names before comparing (strip accents, lowercase, trim)
|
||||
|
||||
### Dolibarr client setup
|
||||
|
||||
The `Dolibarr::Client` requires:
|
||||
```
|
||||
DOLIBARR_URL=https://your-dolibarr.example.com/api/index.php
|
||||
DOLIBARR_API_KEY=your_api_key
|
||||
```
|
||||
|
||||
### Edge cases to handle
|
||||
|
||||
- One GoCardless payment covering multiple invoices (rare but possible)
|
||||
- Credit notes in Dolibarr offsetting invoice balances
|
||||
- GoCardless payment retried after initial failure (same invoice, multiple GC payment IDs)
|
||||
- Shine CSV encoding (often ISO-8859-1, convert to UTF-8)
|
||||
- GoCardless fees: payouts may be slightly less than sum of payments due to GC fees
|
||||
|
||||
### Dolibarr payment method ID
|
||||
|
||||
When calling `paymentsdistributed`, `paymentid` must match the Dolibarr payment method ID for GoCardless in your instance. Check:
|
||||
```
|
||||
GET /setup/dictionary/payment_types
|
||||
```
|
||||
Find the entry for GoCardless (often named "Prélèvement GoCardless" or similar).
|
||||
|
||||
---
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
- **Scheduled run**: Run monthly via cron, email report to admin
|
||||
- **Web interface**: Upload CSVs via admin UI, view reconciliation in browser
|
||||
- **Stripe support**: Same engine, add a `StripeParser` for Stripe payouts
|
||||
- **Webhook-driven**: Instead of CSV imports, consume GoCardless webhooks in real time to auto-reconcile as payments land
|
||||
Reference in New Issue
Block a user