Add advanced reconciliation features: retry detection, partial payments, credit notes
New features: - Credit note handling: Exclude negative invoices from GC reconciliation - Retry detection: Flag invoices with multiple GC payment attempts - Partial payment detection: Track invoices where remain_to_pay > 0 - Payout fee CSV: Export detailed fee breakdown per payout - Configurable tolerances: RECONCILIATION_DATE_TOLERANCE, RECONCILIATION_PAYOUT_TOLERANCE Files: - lib/reconciliation/gocardless_payouts_parser.rb (new) - Parse GC payouts CSV - lib/reconciliation/engine.rb - Add retry_group, partial fields to Match struct - lib/reconciliation/reporter.rb - Show partial/retry in report, write payouts CSV - lib/reconciliation/dolibarr_fetcher.rb - Add is_credit_note field, filter negatives - bin/reconcile - Wire up --gc-payouts argument - README.md - Document new features and --gc-payouts usage - .env.example - Add optional tolerance settings Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
This commit is contained in:
@@ -10,3 +10,7 @@ DOLIBARR_GC_PAYMENT_ID=6
|
||||
# Dolibarr bank account ID (for recording payments)
|
||||
# Find it with: GET /bankaccounts
|
||||
DOLIBARR_BANK_ACCOUNT_ID=1
|
||||
|
||||
# Reconciliation tolerances (optional)
|
||||
# RECONCILIATION_DATE_TOLERANCE=7
|
||||
# RECONCILIATION_PAYOUT_TOLERANCE=2
|
||||
|
||||
60
README.md
60
README.md
@@ -104,9 +104,12 @@ ruby bin/reconcile \
|
||||
--from 2026-01-01 \
|
||||
--to 2026-01-31 \
|
||||
--gc gocardless/payments_export.csv \
|
||||
--gc-payouts gocardless/payouts_export.csv \
|
||||
--shine shine/Cyanet_2026-01-01_2026-12-31_EXPORT/BANQUE_2026-01-01_2026-12-31/BQ_2026-01-01_2026-12-31.csv
|
||||
```
|
||||
|
||||
**Recommended:** Always provide the GoCardless payouts CSV (`--gc-payouts`) for exact fee tracking. Without it, the script can only estimate fees by comparing amounts.
|
||||
|
||||
The Shine file is optional. Without it, payout verification (Pass 3) is skipped:
|
||||
|
||||
```bash
|
||||
@@ -125,12 +128,23 @@ ruby bin/reconcile \
|
||||
--from 2026-01-01 \
|
||||
--to 2026-01-31 \
|
||||
--gc gocardless/payments_export.csv \
|
||||
--gc-payouts gocardless/payouts_export.csv \
|
||||
--shine shine/.../BQ_2026-01-01_2026-12-31.csv \
|
||||
--fix
|
||||
```
|
||||
|
||||
`--fix` only affects invoices flagged `GC_PAID_DOLIBARR_OPEN`. All other entries are reported only.
|
||||
|
||||
### Environment variables (optional)
|
||||
|
||||
```dotenv
|
||||
# Tolerance for soft date matching (default: 7 days)
|
||||
RECONCILIATION_DATE_TOLERANCE=7
|
||||
|
||||
# Tolerance for payout date matching (default: 2 days)
|
||||
RECONCILIATION_PAYOUT_TOLERANCE=2
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## How matching works
|
||||
@@ -171,16 +185,40 @@ All invoices fetched from Dolibarr with status `open` (validated, not yet paid)
|
||||
|
||||
### Pass 3 — GoCardless payouts ↔ Shine bank
|
||||
|
||||
GoCardless batches individual payments into payouts and transfers them as a single bank credit. The script groups `paid_out` payments by their payout ID, sums the amounts, and looks for a matching credit in Shine:
|
||||
**With payouts CSV (recommended):** Match by payout reference (exact) and net amount. Fee breakdown is taken directly from the payouts CSV columns:
|
||||
- `amount` = net payout after all fees
|
||||
- `total_payment_amount` = gross amount before fees
|
||||
- `transaction_fee_debit`, `surcharge_fee_debit`, `tax_debit` = fee breakdown
|
||||
|
||||
**Without payouts CSV (fallback):** Groups `paid_out` payments by payout ID, sums amounts, and looks for a matching credit in Shine:
|
||||
|
||||
1. **Exact match** — same amount, date within 2 days → `PAYOUT_VERIFIED`
|
||||
2. **Date match only** — date within 2 days but amount differs → `PAYOUT_AMOUNT_MISMATCH` (expected: GoCardless deducts its fee from the payout, so the bank credit is always slightly less than the sum of payments)
|
||||
2. **Date match only** — date within 2 days but amount differs → `PAYOUT_AMOUNT_MISMATCH` (expected: GoCardless deducts its fee from the payout)
|
||||
3. **No match found** → `PAYOUT_MISSING`
|
||||
|
||||
`PAYOUT_AMOUNT_MISMATCH` is the normal case when GoCardless fees are deducted. The difference shown in the report is the total fee charged for the period.
|
||||
|
||||
---
|
||||
|
||||
## Advanced features
|
||||
|
||||
### Retry detection
|
||||
|
||||
If multiple GoCardless payments exist for the same invoice reference (e.g., a failed payment was retried), the report flags them with `[RETRY: PM0123, PM0456]`. This helps identify when a payment was re-submitted after an initial failure.
|
||||
|
||||
### Partial payment detection
|
||||
|
||||
Invoices where `remain_to_pay > 0` but some payment has been made are flagged as `PARTIAL`. This can happen when:
|
||||
- A customer paid only part of the invoice
|
||||
- Multiple GoCardless payments cover a single invoice
|
||||
- A credit note was applied to reduce the balance
|
||||
|
||||
### Credit notes excluded
|
||||
|
||||
Credit notes (invoices with negative `total_ttc`) are automatically excluded from reconciliation since they don't correspond to GoCardless payments. They're handled internally in Dolibarr.
|
||||
|
||||
---
|
||||
|
||||
## Output
|
||||
|
||||
### Terminal report
|
||||
@@ -221,7 +259,23 @@ ACTIONS NEEDED (6)
|
||||
|
||||
### CSV export
|
||||
|
||||
A file `tmp/reconciliation_YYYY-MM-DD.csv` is written after every run with one row per invoice/payment, including the flag and recommended action. Suitable for importing into a spreadsheet for manual review.
|
||||
Two CSV files are written after every run:
|
||||
|
||||
**`tmp/reconciliation_YYYY-MM-DD.csv`** — One row per invoice/payment with columns:
|
||||
- `invoice_ref`, `customer_name`, `amount_eur`, `invoice_date`, `due_date`
|
||||
- `dolibarr_status`, `gc_payment_id`, `gc_status`, `gc_charge_date`
|
||||
- `match_type`, `flag`, `action`
|
||||
- `partial` — "yes" if invoice is partially paid
|
||||
- `retry_group` — comma-separated GC payment IDs if retries detected
|
||||
|
||||
**`tmp/payouts_fees_YYYY-MM-DD.csv`** — One row per payout with fee breakdown:
|
||||
- `payout_id`, `payout_date`
|
||||
- `gross_amount_eur`, `net_amount_eur`, `fee_eur`
|
||||
- `fee_percentage` — GC fee as percentage of gross
|
||||
- `shine_reference` — the matching Shine transaction label
|
||||
- `status` — `payout_verified`, `payout_missing`, or `payout_amount_mismatch`
|
||||
|
||||
Both files are suitable for importing into a spreadsheet for manual review.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -10,12 +10,13 @@ options = {
|
||||
from: nil,
|
||||
to: nil,
|
||||
gc: nil,
|
||||
gc_payouts: nil,
|
||||
shine: nil
|
||||
}
|
||||
|
||||
OptionParser.new do |opts|
|
||||
opts.banner = <<~BANNER
|
||||
Usage: bin/reconcile --from DATE --to DATE --gc PATH [--shine PATH] [--fix]
|
||||
Usage: bin/reconcile --from DATE --to DATE --gc PATH [--gc-payouts PATH] [--shine PATH] [--fix]
|
||||
|
||||
Options:
|
||||
BANNER
|
||||
@@ -32,6 +33,10 @@ OptionParser.new do |opts|
|
||||
options[:gc] = v
|
||||
end
|
||||
|
||||
opts.on("--gc-payouts PATH", "GoCardless payouts CSV file path (enables exact fee reporting)") do |v|
|
||||
options[:gc_payouts] = v
|
||||
end
|
||||
|
||||
opts.on("--shine PATH", "Shine bank statement CSV file path (optional)") do |v|
|
||||
options[:shine] = v
|
||||
end
|
||||
@@ -52,6 +57,7 @@ errors << "--from is required" unless options[:from]
|
||||
errors << "--to is required" unless options[:to]
|
||||
errors << "--gc is required" unless options[:gc]
|
||||
errors << "--gc file not found: #{options[:gc]}" if options[:gc] && !File.exist?(options[:gc])
|
||||
errors << "--gc-payouts file not found: #{options[:gc_payouts]}" if options[:gc_payouts] && !File.exist?(options[:gc_payouts])
|
||||
errors << "--shine file not found: #{options[:shine]}" if options[:shine] && !File.exist?(options[:shine])
|
||||
|
||||
unless errors.empty?
|
||||
@@ -60,16 +66,18 @@ unless errors.empty?
|
||||
end
|
||||
|
||||
# Run reconciliation
|
||||
client = Dolibarr::Client.new
|
||||
fetcher = Reconciliation::DolibarrFetcher.new(client, from: options[:from], to: options[:to])
|
||||
gc_data = Reconciliation::GocardlessParser.parse(options[:gc], from: options[:from], to: options[:to])
|
||||
shine_data = options[:shine] ? Reconciliation::ShineParser.parse(options[:shine]) : []
|
||||
client = Dolibarr::Client.new
|
||||
fetcher = Reconciliation::DolibarrFetcher.new(client, from: options[:from], to: options[:to])
|
||||
gc_data = Reconciliation::GocardlessParser.parse(options[:gc], from: options[:from], to: options[:to])
|
||||
gc_payouts = options[:gc_payouts] ? Reconciliation::GocardlessPayoutsParser.parse(options[:gc_payouts]) : []
|
||||
shine_data = options[:shine] ? Reconciliation::ShineParser.parse(options[:shine]) : []
|
||||
|
||||
dolibarr_invoices = fetcher.fetch_invoices
|
||||
|
||||
engine = Reconciliation::Engine.new(
|
||||
dolibarr_invoices: dolibarr_invoices,
|
||||
gc_payments: gc_data,
|
||||
dolibarr_invoices: dolibarr_invoices,
|
||||
gc_payments: gc_data,
|
||||
gc_payouts: gc_payouts,
|
||||
shine_transactions: shine_data,
|
||||
from: options[:from],
|
||||
to: options[:to]
|
||||
|
||||
@@ -6,6 +6,7 @@ require "dotenv/load"
|
||||
require_relative "dolibarr/client"
|
||||
require_relative "reconciliation/dolibarr_fetcher"
|
||||
require_relative "reconciliation/gocardless_parser"
|
||||
require_relative "reconciliation/gocardless_payouts_parser"
|
||||
require_relative "reconciliation/shine_parser"
|
||||
require_relative "reconciliation/engine"
|
||||
require_relative "reconciliation/reporter"
|
||||
|
||||
@@ -23,6 +23,7 @@ module Reconciliation
|
||||
:due_date, # Due date (Date)
|
||||
:customer_id, # socid
|
||||
:customer_name, # nom of third party if present
|
||||
:is_credit_note, # true if this is a credit note (negative amount)
|
||||
keyword_init: true
|
||||
)
|
||||
|
||||
@@ -45,6 +46,7 @@ module Reconciliation
|
||||
invoices = raw_invoices
|
||||
.map { |raw| parse_invoice(raw, name_by_id) }
|
||||
.reject { |inv| inv.status == STATUS_CANCELLED }
|
||||
.reject { |inv| inv.is_credit_note } # Credit notes (negative amounts) excluded from GC reconciliation
|
||||
|
||||
open_count = invoices.count { |inv| inv.status == STATUS_VALIDATED }
|
||||
paid_count = invoices.count { |inv| inv.status == STATUS_PAID }
|
||||
@@ -76,19 +78,21 @@ module Reconciliation
|
||||
|
||||
def parse_invoice(raw, name_by_id = {})
|
||||
customer_id = raw["socid"].to_i
|
||||
total_ttc = raw["total_ttc"].to_f
|
||||
Invoice.new(
|
||||
id: raw["id"].to_i,
|
||||
ref: raw["ref"].to_s.strip,
|
||||
status: raw["statut"].to_i,
|
||||
total_ttc: raw["total_ttc"].to_f,
|
||||
amount_cents: (raw["total_ttc"].to_f * 100).round,
|
||||
total_ttc: total_ttc,
|
||||
amount_cents: (total_ttc * 100).round,
|
||||
paye: raw["paye"].to_i,
|
||||
sumpayed: raw["sumpayed"].to_f,
|
||||
remain_to_pay: raw["remaintopay"].to_f,
|
||||
date: parse_unix_date(raw["datef"] || raw["date"]),
|
||||
due_date: parse_unix_date(raw["date_lim_reglement"]),
|
||||
customer_id: customer_id,
|
||||
customer_name: name_by_id[customer_id] || raw.dig("thirdparty", "name").to_s.strip
|
||||
customer_name: name_by_id[customer_id] || raw.dig("thirdparty", "name").to_s.strip,
|
||||
is_credit_note: total_ttc < 0
|
||||
)
|
||||
end
|
||||
|
||||
|
||||
@@ -18,8 +18,8 @@ module Reconciliation
|
||||
PAYOUT_MISSING = :payout_missing
|
||||
PAYOUT_AMOUNT_MISMATCH = :payout_amount_mismatch
|
||||
|
||||
DATE_TOLERANCE = 7 # days for GC charge_date ↔ Dolibarr invoice date soft match
|
||||
PAYOUT_DATE_TOLERANCE = 2 # days for GC payout_date ↔ Shine credit date
|
||||
DATE_TOLERANCE = ENV.fetch("RECONCILIATION_DATE_TOLERANCE", 7).to_i # days for GC charge_date ↔ Dolibarr invoice date soft match
|
||||
PAYOUT_DATE_TOLERANCE = ENV.fetch("RECONCILIATION_PAYOUT_TOLERANCE", 2).to_i # days for GC payout_date ↔ Shine credit date
|
||||
|
||||
# Result structs
|
||||
Match = Struct.new(
|
||||
@@ -27,6 +27,8 @@ module Reconciliation
|
||||
:invoice, # DolibarrFetcher::Invoice or nil
|
||||
:payment, # GocardlessParser::Payment or nil
|
||||
:match_type, # :strong, :soft, or nil
|
||||
:partial, # true if invoice is partially paid (remain_to_pay > 0)
|
||||
:retry_group, # Array of GC payment IDs for same invoice (retry detection)
|
||||
keyword_init: true
|
||||
)
|
||||
|
||||
@@ -34,16 +36,19 @@ module Reconciliation
|
||||
:flag,
|
||||
:payout_id,
|
||||
:payout_date,
|
||||
:expected_amount_cents,
|
||||
:expected_amount_cents, # Net amount (what should land in bank)
|
||||
:shine_transaction, # ShineParser::Transaction or nil
|
||||
:actual_amount_cents,
|
||||
:actual_amount_cents, # What was actually found in Shine
|
||||
:gc_payment_ids,
|
||||
:fee_cents, # GC fee deducted (nil if unknown)
|
||||
:gross_amount_cents, # Sum of underlying payments (before fees)
|
||||
keyword_init: true
|
||||
)
|
||||
|
||||
def initialize(dolibarr_invoices:, gc_payments:, shine_transactions:, from:, to:)
|
||||
def initialize(dolibarr_invoices:, gc_payments:, shine_transactions:, from:, to:, gc_payouts: [])
|
||||
@invoices = dolibarr_invoices
|
||||
@gc_payments = gc_payments
|
||||
@gc_payouts = gc_payouts
|
||||
@shine = shine_transactions
|
||||
@from = from
|
||||
@to = to
|
||||
@@ -76,6 +81,9 @@ module Reconciliation
|
||||
matched_invoice_ids = []
|
||||
results = []
|
||||
|
||||
# Build a map of invoice ref -> payments for retry detection
|
||||
payments_by_invoice_ref = @gc_payments.group_by { |p| p.description.downcase }
|
||||
|
||||
@gc_payments.each do |payment|
|
||||
invoice, match_type = find_invoice(payment)
|
||||
|
||||
@@ -84,7 +92,25 @@ module Reconciliation
|
||||
end
|
||||
|
||||
flag = determine_flag(payment, invoice)
|
||||
results << Match.new(flag: flag, invoice: invoice, payment: payment, match_type: match_type)
|
||||
|
||||
# Detect retries: multiple GC payments for the same invoice ref
|
||||
retry_group = []
|
||||
if invoice && !invoice.ref.nil?
|
||||
ref_payments = payments_by_invoice_ref[invoice.ref.downcase] || []
|
||||
retry_group = ref_payments.map(&:id) if ref_payments.size > 1
|
||||
end
|
||||
|
||||
# Detect partial payments
|
||||
partial = invoice && invoice.remain_to_pay > 0 && invoice.remain_to_pay < invoice.amount_cents / 100.0
|
||||
|
||||
results << Match.new(
|
||||
flag: flag,
|
||||
invoice: invoice,
|
||||
payment: payment,
|
||||
match_type: match_type,
|
||||
partial: partial,
|
||||
retry_group: retry_group
|
||||
)
|
||||
end
|
||||
|
||||
# Dolibarr paid invoices in the date range with no GC match.
|
||||
@@ -100,7 +126,9 @@ module Reconciliation
|
||||
flag: DOLIBARR_PAID_NO_GC,
|
||||
invoice: inv,
|
||||
payment: nil,
|
||||
match_type: nil
|
||||
match_type: nil,
|
||||
partial: false,
|
||||
retry_group: []
|
||||
)
|
||||
end
|
||||
|
||||
@@ -121,59 +149,110 @@ module Reconciliation
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Pass 3 — Match GoCardless payouts to Shine bank credits
|
||||
#
|
||||
# When a payouts CSV is provided (preferred): match by payout reference and
|
||||
# net amount — both are exact. Fee breakdown is taken from the payouts CSV.
|
||||
#
|
||||
# When no payouts CSV is provided (fallback): group payments by payout_id,
|
||||
# sum amounts, and match Shine by date only (amount will differ due to fees).
|
||||
# -------------------------------------------------------------------------
|
||||
def pass3_payouts_vs_shine
|
||||
return [] if @shine.empty?
|
||||
|
||||
gc_credits = ShineParser.gocardless_credits(@shine)
|
||||
return [] if gc_credits.empty?
|
||||
|
||||
# Group paid_out payments by payout_id.
|
||||
# Only paid_out payments are included in a payout; failed/cancelled are not.
|
||||
by_payout = @gc_payments
|
||||
.select { |p| p.status == "paid_out" && p.payout_id && !p.payout_id.empty? && p.payout_date }
|
||||
.group_by(&:payout_id)
|
||||
if @gc_payouts.any?
|
||||
pass3_with_payouts_csv(gc_credits)
|
||||
else
|
||||
pass3_from_payments_csv(gc_credits)
|
||||
end
|
||||
end
|
||||
|
||||
def pass3_with_payouts_csv(gc_credits)
|
||||
used_shine_ids = []
|
||||
|
||||
by_payout.filter_map do |payout_id, payments|
|
||||
payout_date = payments.map(&:payout_date).compact.max
|
||||
expected_cents = payments.sum(&:amount_cents)
|
||||
# Only consider payouts whose arrival_date falls within the reconciliation window.
|
||||
payouts_in_range = @gc_payouts.select do |p|
|
||||
p.arrival_date >= @from && p.arrival_date <= @to
|
||||
end
|
||||
|
||||
payouts_in_range.map do |payout|
|
||||
# 1. Strong match: payout reference found in Shine Libellé
|
||||
shine_tx = gc_credits.find do |tx|
|
||||
!used_shine_ids.include?(tx.id) &&
|
||||
(tx.date - payout_date).abs <= PAYOUT_DATE_TOLERANCE &&
|
||||
tx.credit_cents == expected_cents
|
||||
tx.label.include?(payout.reference)
|
||||
end
|
||||
|
||||
# 2. Fallback: exact net amount + date within tolerance
|
||||
if shine_tx.nil?
|
||||
shine_tx = gc_credits.find do |tx|
|
||||
!used_shine_ids.include?(tx.id) &&
|
||||
tx.credit_cents == payout.amount_cents &&
|
||||
(tx.date - payout.arrival_date).abs <= PAYOUT_DATE_TOLERANCE
|
||||
end
|
||||
end
|
||||
|
||||
if shine_tx
|
||||
used_shine_ids << shine_tx.id
|
||||
flag = PAYOUT_VERIFIED
|
||||
actual = shine_tx.credit_cents
|
||||
else
|
||||
# Try date match only (amount mismatch — possible GC fees)
|
||||
shine_tx = gc_credits.find do |tx|
|
||||
!used_shine_ids.include?(tx.id) &&
|
||||
(tx.date - payout_date).abs <= PAYOUT_DATE_TOLERANCE
|
||||
end
|
||||
flag = PAYOUT_MISSING
|
||||
end
|
||||
|
||||
if shine_tx
|
||||
used_shine_ids << shine_tx.id
|
||||
flag = PAYOUT_AMOUNT_MISMATCH
|
||||
actual = shine_tx.credit_cents
|
||||
else
|
||||
flag = PAYOUT_MISSING
|
||||
actual = 0
|
||||
end
|
||||
PayoutMatch.new(
|
||||
flag: flag,
|
||||
payout_id: payout.id,
|
||||
payout_date: payout.arrival_date,
|
||||
expected_amount_cents: payout.amount_cents,
|
||||
shine_transaction: shine_tx,
|
||||
actual_amount_cents: shine_tx&.credit_cents || 0,
|
||||
gc_payment_ids: [],
|
||||
fee_cents: payout.fee_cents,
|
||||
gross_amount_cents: payout.total_payment_cents
|
||||
)
|
||||
end
|
||||
end
|
||||
|
||||
def pass3_from_payments_csv(gc_credits)
|
||||
used_shine_ids = []
|
||||
|
||||
by_payout = @gc_payments
|
||||
.select { |p| p.status == "paid_out" && p.payout_id && !p.payout_id.empty? && p.payout_date }
|
||||
.group_by(&:payout_id)
|
||||
|
||||
by_payout.filter_map do |payout_id, payments|
|
||||
payout_date = payments.map(&:payout_date).compact.max
|
||||
gross_cents = payments.sum(&:amount_cents)
|
||||
|
||||
# Without the payouts CSV we don't know the exact net amount, so match
|
||||
# by date only and accept any credit from GOCARDLESS SAS that day.
|
||||
shine_tx = gc_credits.find do |tx|
|
||||
!used_shine_ids.include?(tx.id) &&
|
||||
(tx.date - payout_date).abs <= PAYOUT_DATE_TOLERANCE
|
||||
end
|
||||
|
||||
if shine_tx
|
||||
used_shine_ids << shine_tx.id
|
||||
flag = shine_tx.credit_cents == gross_cents ? PAYOUT_VERIFIED : PAYOUT_AMOUNT_MISMATCH
|
||||
actual = shine_tx.credit_cents
|
||||
fee_cents = gross_cents - actual
|
||||
else
|
||||
flag = PAYOUT_MISSING
|
||||
actual = 0
|
||||
fee_cents = nil
|
||||
end
|
||||
|
||||
PayoutMatch.new(
|
||||
flag: flag,
|
||||
payout_id: payout_id,
|
||||
payout_date: payout_date,
|
||||
expected_amount_cents: expected_cents,
|
||||
expected_amount_cents: gross_cents,
|
||||
shine_transaction: shine_tx,
|
||||
actual_amount_cents: actual,
|
||||
gc_payment_ids: payments.map(&:id)
|
||||
gc_payment_ids: payments.map(&:id),
|
||||
fee_cents: fee_cents,
|
||||
gross_amount_cents: gross_cents
|
||||
)
|
||||
end
|
||||
end
|
||||
|
||||
64
lib/reconciliation/gocardless_payouts_parser.rb
Normal file
64
lib/reconciliation/gocardless_payouts_parser.rb
Normal file
@@ -0,0 +1,64 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
require "csv"
|
||||
require "date"
|
||||
|
||||
module Reconciliation
|
||||
class GocardlessPayoutsParser
|
||||
Payout = Struct.new(
|
||||
:id,
|
||||
:amount_cents, # Net amount transferred to bank (after all fees)
|
||||
:total_payment_cents, # Gross sum of underlying payments (before fees)
|
||||
:fee_cents, # Total fee deducted (total_payment - amount)
|
||||
:transaction_fee_cents, # Per-transaction fee component
|
||||
:surcharge_fee_cents, # Surcharge fee component
|
||||
:tax_cents, # Tax on fees (informational, already in total fee)
|
||||
:reference, # e.g. "CYANET-DKV4KN8FTM2" — appears in Shine Libellé
|
||||
:status,
|
||||
:arrival_date,
|
||||
keyword_init: true
|
||||
)
|
||||
|
||||
def self.parse(csv_path)
|
||||
rows = CSV.read(csv_path, headers: true, encoding: "UTF-8")
|
||||
|
||||
payouts = rows.filter_map do |row|
|
||||
next if row["status"] != "paid"
|
||||
|
||||
arrival_date = safe_parse_date(row["arrival_date"])
|
||||
next unless arrival_date
|
||||
|
||||
amount_cents = to_cents(row["amount"])
|
||||
total_payment_cents = to_cents(row["total_payment_amount"])
|
||||
|
||||
Payout.new(
|
||||
id: row["id"].to_s.strip,
|
||||
amount_cents: amount_cents,
|
||||
total_payment_cents: total_payment_cents,
|
||||
fee_cents: total_payment_cents - amount_cents,
|
||||
transaction_fee_cents: to_cents(row["transaction_fee_debit"]),
|
||||
surcharge_fee_cents: to_cents(row["surcharge_fee_debit"]),
|
||||
tax_cents: to_cents(row["tax_debit"]),
|
||||
reference: row["reference"].to_s.strip,
|
||||
status: row["status"].to_s.strip,
|
||||
arrival_date: arrival_date
|
||||
)
|
||||
end
|
||||
|
||||
$stderr.puts "[GocardlessPayoutsParser] Loaded #{payouts.size} paid payouts from #{csv_path}"
|
||||
payouts
|
||||
end
|
||||
|
||||
private_class_method def self.to_cents(str)
|
||||
return 0 if str.nil? || str.strip.empty?
|
||||
(str.strip.to_f * 100).round
|
||||
end
|
||||
|
||||
private_class_method def self.safe_parse_date(str)
|
||||
return nil if str.nil? || str.strip.empty?
|
||||
Date.parse(str.strip)
|
||||
rescue Date::Error, ArgumentError
|
||||
nil
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -39,25 +39,45 @@ module Reconciliation
|
||||
puts " GC cancelled: #{r[:gc_cancelled].size}"
|
||||
puts " GC payment / no invoice: #{r[:gc_no_invoice].size}#{r[:gc_no_invoice].any? ? ' ← investigate' : ''}"
|
||||
|
||||
# Partial payment info
|
||||
partial_matches = r[:matches].select { |m| m.partial }
|
||||
if partial_matches.any?
|
||||
puts " Partial payments: #{partial_matches.size} ← check remain_to_pay"
|
||||
end
|
||||
|
||||
# Retry detection info
|
||||
retry_matches = r[:matches].select { |m| m.retry_group && m.retry_group.size > 1 }
|
||||
if retry_matches.any?
|
||||
puts " Retries detected: #{retry_matches.size} ← multiple GC attempts for same invoice"
|
||||
end
|
||||
|
||||
# --- Shine ↔ GoCardless Payouts ---
|
||||
unless @result[:payout_matches].empty?
|
||||
pm = r[:payout_matches]
|
||||
verified = pm.count { |p| p.flag == Engine::PAYOUT_VERIFIED }
|
||||
mismatch = pm.count { |p| p.flag == Engine::PAYOUT_AMOUNT_MISMATCH }
|
||||
missing = pm.count { |p| p.flag == Engine::PAYOUT_MISSING }
|
||||
expected = pm.sum(&:expected_amount_cents)
|
||||
actual = pm.sum(&:actual_amount_cents)
|
||||
diff = actual - expected
|
||||
pm = r[:payout_matches]
|
||||
verified = pm.count { |p| p.flag == Engine::PAYOUT_VERIFIED }
|
||||
mismatch = pm.count { |p| p.flag == Engine::PAYOUT_AMOUNT_MISMATCH }
|
||||
missing = pm.count { |p| p.flag == Engine::PAYOUT_MISSING }
|
||||
total_net = pm.sum(&:expected_amount_cents)
|
||||
known_fees = pm.filter_map(&:fee_cents)
|
||||
total_fees = known_fees.sum
|
||||
|
||||
puts ""
|
||||
puts "SHINE ↔ GOCARDLESS PAYOUTS"
|
||||
puts " Payouts expected: #{pm.size}"
|
||||
puts " Verified: #{verified}#{verified == pm.size ? ' ✓' : ''}"
|
||||
puts " Amount mismatch: #{mismatch}#{mismatch > 0 ? ' ← check GC fees' : ''}"
|
||||
puts " Missing in Shine: #{missing}#{missing > 0 ? ' ← investigate' : ''}"
|
||||
puts " Expected total: #{format_eur(expected)}"
|
||||
puts " Actual total: #{format_eur(actual)}"
|
||||
puts " Difference: #{format_eur(diff)}#{diff.zero? ? ' ✓' : ' ← investigate'}"
|
||||
puts " Payouts: #{pm.size}"
|
||||
puts " Verified: #{verified}#{verified == pm.size ? ' ✓' : ''}"
|
||||
puts " Amount mismatch: #{mismatch}#{mismatch > 0 ? ' ← check GC fees' : ''}" if mismatch > 0
|
||||
puts " Missing in Shine: #{missing} ← investigate" if missing > 0
|
||||
puts " Net received: #{format_eur(total_net)}"
|
||||
if known_fees.any?
|
||||
gross = pm.filter_map(&:gross_amount_cents).sum
|
||||
puts " Gross collected: #{format_eur(gross)}"
|
||||
puts " GC fees: #{format_eur(total_fees)}"
|
||||
# Per-payout fee detail
|
||||
pm.select { |p| p.fee_cents && p.fee_cents > 0 }.each do |p|
|
||||
puts " #{p.payout_id} #{p.payout_date} net=#{format_eur(p.expected_amount_cents)} " \
|
||||
"gross=#{format_eur(p.gross_amount_cents)} fee=#{format_eur(p.fee_cents)}"
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
# --- Action items ---
|
||||
@@ -72,11 +92,16 @@ module Reconciliation
|
||||
puts "-" * 60
|
||||
|
||||
r[:gc_paid_dolibarr_open].each_with_index do |m, i|
|
||||
extra = []
|
||||
extra << "PARTIAL" if m.partial
|
||||
extra << "RETRY: #{m.retry_group.join(', ')}" if m.retry_group && m.retry_group.size > 1
|
||||
extra_str = extra.any? ? " [#{extra.join(', ')}]" : ""
|
||||
|
||||
puts " #{i + 1}. [GC_PAID_DOLIBARR_OPEN] " \
|
||||
"#{m.invoice.ref.ljust(16)} " \
|
||||
"#{format_eur(m.invoice.amount_cents)} " \
|
||||
"#{display_name(m.invoice).ljust(20)} " \
|
||||
"GC: #{m.payment.id} #{m.payment.charge_date} (#{m.match_type})"
|
||||
"GC: #{m.payment.id} #{m.payment.charge_date} (#{m.match_type})#{extra_str}"
|
||||
end
|
||||
|
||||
r[:dolibarr_paid_no_gc].each_with_index do |m, i|
|
||||
@@ -123,6 +148,11 @@ module Reconciliation
|
||||
|
||||
csv_path = write_csv
|
||||
puts " Report saved to: #{csv_path}"
|
||||
|
||||
unless @result[:payout_matches].empty?
|
||||
payout_csv_path = write_payouts_csv
|
||||
puts " Payout fees saved to: #{payout_csv_path}"
|
||||
end
|
||||
puts ""
|
||||
end
|
||||
|
||||
@@ -135,7 +165,7 @@ module Reconciliation
|
||||
csv << %w[
|
||||
invoice_ref customer_name amount_eur invoice_date due_date
|
||||
dolibarr_status gc_payment_id gc_status gc_charge_date
|
||||
match_type flag action
|
||||
match_type flag action partial retry_group
|
||||
]
|
||||
|
||||
@result[:matches].each do |m|
|
||||
@@ -153,7 +183,9 @@ module Reconciliation
|
||||
pay&.charge_date,
|
||||
m.match_type,
|
||||
m.flag,
|
||||
action_label(m.flag)
|
||||
action_label(m.flag),
|
||||
m.partial ? "yes" : "no",
|
||||
m.retry_group&.join(", ")
|
||||
]
|
||||
end
|
||||
|
||||
@@ -165,7 +197,41 @@ module Reconciliation
|
||||
status_label(inv.status),
|
||||
nil, nil, nil, nil,
|
||||
Engine::DOLIBARR_OPEN_NO_GC,
|
||||
action_label(Engine::DOLIBARR_OPEN_NO_GC)
|
||||
action_label(Engine::DOLIBARR_OPEN_NO_GC),
|
||||
"no",
|
||||
nil
|
||||
]
|
||||
end
|
||||
end
|
||||
|
||||
path
|
||||
end
|
||||
|
||||
def write_payouts_csv
|
||||
dir = "tmp"
|
||||
Dir.mkdir(dir) unless Dir.exist?(dir)
|
||||
path = "#{dir}/payouts_fees_#{@to}.csv"
|
||||
|
||||
CSV.open(path, "w", encoding: "UTF-8") do |csv|
|
||||
csv << %w[
|
||||
payout_id payout_date gross_amount_eur net_amount_eur fee_eur
|
||||
fee_percentage shine_reference status
|
||||
]
|
||||
|
||||
@result[:payout_matches].each do |pm|
|
||||
fee_pct = pm.gross_amount_cents && pm.gross_amount_cents > 0 \
|
||||
? ((pm.fee_cents.to_f / pm.gross_amount_cents) * 100).round(2) \
|
||||
: nil
|
||||
|
||||
csv << [
|
||||
pm.payout_id,
|
||||
pm.payout_date,
|
||||
pm.gross_amount_cents ? "%.2f" % (pm.gross_amount_cents / 100.0) : nil,
|
||||
"%.2f" % (pm.expected_amount_cents / 100.0),
|
||||
pm.fee_cents ? "%.2f" % (pm.fee_cents / 100.0) : nil,
|
||||
fee_pct ? "#{fee_pct}%" : nil,
|
||||
pm.shine_transaction&.label,
|
||||
pm.flag
|
||||
]
|
||||
end
|
||||
end
|
||||
|
||||
Reference in New Issue
Block a user