Initial commit: Clean SEO analysis system
This commit is contained in:
Kevin Bataille
23
.env.example
Normal file
23
.env.example
Normal file
@@ -0,0 +1,23 @@
|
||||
# WordPress Configuration
|
||||
WORDPRESS_URL=https://yoursite.com
|
||||
WORDPRESS_USERNAME=your_username
|
||||
WORDPRESS_APP_PASSWORD=your_application_password
|
||||
|
||||
# OpenRouter API Configuration
|
||||
OPENROUTER_API_KEY=your_openrouter_api_key
|
||||
|
||||
# AI Model Selection (choose one)
|
||||
# Recommended: anthropic/claude-3.5-sonnet (best quality, $3/$15 per 1M tokens)
|
||||
# Budget: meta-llama/llama-3.1-70b-instruct (free tier available)
|
||||
# Alternative: openai/gpt-4-turbo ($10/$30 per 1M tokens)
|
||||
AI_MODEL=anthropic/claude-3.5-sonnet
|
||||
|
||||
# Script Configuration
|
||||
BATCH_SIZE=100
|
||||
API_DELAY_SECONDS=0.5
|
||||
|
||||
# Analysis Settings
|
||||
ANALYSIS_MIN_POSITION=11
|
||||
ANALYSIS_MAX_POSITION=30
|
||||
ANALYSIS_MIN_IMPRESSIONS=50
|
||||
ANALYSIS_TOP_N_POSTS=20
|
||||
48
.gitignore
vendored
Normal file
48
.gitignore
vendored
Normal file
@@ -0,0 +1,48 @@
|
||||
# Configuration
|
||||
.env
|
||||
.env.local
|
||||
|
||||
# Virtual Environment
|
||||
venv/
|
||||
env/
|
||||
ENV/
|
||||
.venv
|
||||
|
||||
# Python
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
*.so
|
||||
.Python
|
||||
*.egg-info/
|
||||
dist/
|
||||
build/
|
||||
|
||||
# Input files (sensitive/large)
|
||||
input/analytics/
|
||||
input/**/*.csv
|
||||
input/**/*.txt
|
||||
|
||||
# Output files (generated results)
|
||||
output/results/
|
||||
output/logs/
|
||||
output/**/*.csv
|
||||
output/**/*.txt
|
||||
output/**/*.log
|
||||
output/**/*.md
|
||||
|
||||
# IDE
|
||||
.vscode/
|
||||
.idea/
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
|
||||
# OS
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
|
||||
# Backup/rollback files
|
||||
*.bak
|
||||
rollback_*.csv
|
||||
*_backup.csv
|
||||
310
PROJECT_GUIDE.md
Normal file
310
PROJECT_GUIDE.md
Normal file
@@ -0,0 +1,310 @@
|
||||
# SEO Analysis & Improvement System - Project Guide
|
||||
|
||||
## 📋 Overview
|
||||
|
||||
A complete 4-phase SEO analysis pipeline that:
|
||||
1. **Integrates** Google Analytics, Search Console, and WordPress data
|
||||
2. **Identifies** high-potential keywords for optimization (positions 11-30)
|
||||
3. **Discovers** new content opportunities using AI
|
||||
4. **Generates** a comprehensive report with 90-day action plan
|
||||
|
||||
## 📂 Project Structure
|
||||
|
||||
```
|
||||
seo/
|
||||
├── input/ # SOURCE DATA (your exports)
|
||||
│ ├── new-propositions.csv # WordPress posts
|
||||
│ ├── README.md # How to export data
|
||||
│ └── analytics/
|
||||
│ ├── ga4_export.csv # Google Analytics
|
||||
│ └── gsc/
|
||||
│ ├── Pages.csv # GSC pages (required)
|
||||
│ ├── Requêtes.csv # GSC queries (optional)
|
||||
│ └── ...
|
||||
│
|
||||
├── output/ # RESULTS (auto-generated)
|
||||
│ ├── results/
|
||||
│ │ ├── seo_optimization_report.md # 📍 PRIMARY OUTPUT
|
||||
│ │ ├── posts_with_analytics.csv
|
||||
│ │ ├── posts_prioritized.csv
|
||||
│ │ ├── keyword_opportunities.csv
|
||||
│ │ └── content_gaps.csv
|
||||
│ │
|
||||
│ ├── logs/
|
||||
│ │ ├── import_log.txt
|
||||
│ │ ├── opportunity_analysis_log.txt
|
||||
│ │ └── content_gap_analysis_log.txt
|
||||
│ │
|
||||
│ └── README.md # Output guide
|
||||
│
|
||||
├── 🚀 run_analysis.sh # Run entire pipeline
|
||||
├── analytics_importer.py # Phase 1: Merge data
|
||||
├── opportunity_analyzer.py # Phase 2: Find wins
|
||||
├── content_gap_analyzer.py # Phase 3: Find gaps
|
||||
├── report_generator.py # Phase 4: Generate report
|
||||
├── config.py
|
||||
├── requirements.txt
|
||||
├── .env.example
|
||||
└── .gitignore
|
||||
```
|
||||
|
||||
## 🚀 Getting Started
|
||||
|
||||
### Step 1: Prepare Input Data
|
||||
|
||||
**Place WordPress posts CSV:**
|
||||
```
|
||||
input/new-propositions.csv
|
||||
```
|
||||
|
||||
**Export Google Analytics 4:**
|
||||
1. Go to: Analytics > Reports > Engagement > Pages and Screens
|
||||
2. Set date range: Last 90 days
|
||||
3. Download CSV → Save as: `input/analytics/ga4_export.csv`
|
||||
|
||||
**Export Google Search Console (Pages):**
|
||||
1. Go to: Performance
|
||||
2. Set date range: Last 90 days
|
||||
3. Export CSV → Save as: `input/analytics/gsc/Pages.csv`
|
||||
|
||||
### Step 2: Run Analysis
|
||||
|
||||
```bash
|
||||
# Run entire pipeline
|
||||
./run_analysis.sh
|
||||
|
||||
# OR run steps individually
|
||||
./venv/bin/python analytics_importer.py
|
||||
./venv/bin/python opportunity_analyzer.py
|
||||
./venv/bin/python content_gap_analyzer.py
|
||||
./venv/bin/python report_generator.py
|
||||
```
|
||||
|
||||
### Step 3: Review Report
|
||||
|
||||
Open: **`output/results/seo_optimization_report.md`**
|
||||
|
||||
Contains:
|
||||
- Executive summary with current metrics
|
||||
- Top 20 posts ranked by opportunity (with AI recommendations)
|
||||
- Keyword opportunities breakdown
|
||||
- Content gap analysis
|
||||
- 90-day phased action plan
|
||||
|
||||
## 📊 What Each Script Does
|
||||
|
||||
### `analytics_importer.py` (Phase 1)
|
||||
**Purpose:** Merge analytics data with WordPress posts
|
||||
|
||||
**Input:**
|
||||
- `input/new-propositions.csv` (WordPress posts)
|
||||
- `input/analytics/ga4_export.csv` (Google Analytics)
|
||||
- `input/analytics/gsc/Pages.csv` (Search Console)
|
||||
|
||||
**Output:**
|
||||
- `output/results/posts_with_analytics.csv` (enriched dataset)
|
||||
- `output/logs/import_log.txt` (matching report)
|
||||
|
||||
**Handles:** French and English column names, URL normalization, multi-source merging
|
||||
|
||||
### `opportunity_analyzer.py` (Phase 2)
|
||||
**Purpose:** Identify high-potential optimization opportunities
|
||||
|
||||
**Input:**
|
||||
- `output/results/posts_with_analytics.csv`
|
||||
|
||||
**Output:**
|
||||
- `output/results/keyword_opportunities.csv` (26 opportunities)
|
||||
- `output/logs/opportunity_analysis_log.txt`
|
||||
|
||||
**Features:**
|
||||
- Filters posts at positions 11-30 (page 2-3)
|
||||
- Calculates opportunity scores (0-100)
|
||||
- Generates AI recommendations for top 20 posts
|
||||
|
||||
### `content_gap_analyzer.py` (Phase 3)
|
||||
**Purpose:** Discover new content opportunities
|
||||
|
||||
**Input:**
|
||||
- `output/results/posts_with_analytics.csv`
|
||||
- `input/analytics/gsc/Requêtes.csv` (optional)
|
||||
|
||||
**Output:**
|
||||
- `output/results/content_gaps.csv`
|
||||
- `output/logs/content_gap_analysis_log.txt`
|
||||
|
||||
**Features:**
|
||||
- Topic cluster extraction
|
||||
- Gap identification
|
||||
- AI-powered content suggestions
|
||||
|
||||
### `report_generator.py` (Phase 4)
|
||||
**Purpose:** Create comprehensive report with action plan
|
||||
|
||||
**Input:**
|
||||
- All analysis results from phases 1-3
|
||||
|
||||
**Output:**
|
||||
- `output/results/seo_optimization_report.md` ← **PRIMARY DELIVERABLE**
|
||||
- `output/results/posts_prioritized.csv`
|
||||
|
||||
**Features:**
|
||||
- Comprehensive markdown report
|
||||
- All 262 posts ranked
|
||||
- 90-day action plan with estimated gains
|
||||
|
||||
## 📈 Understanding Your Report
|
||||
|
||||
### Key Metrics (Executive Summary)
|
||||
- **Total Posts:** All posts analyzed
|
||||
- **Monthly Traffic:** Current organic traffic
|
||||
- **Total Impressions:** Search visibility (90 days)
|
||||
- **Average Position:** Current ranking position
|
||||
- **Opportunities:** Posts ready to optimize
|
||||
|
||||
### Top 20 Posts to Optimize
|
||||
Each post shows:
|
||||
- **Title** (the post name)
|
||||
- **Current Position** (search ranking)
|
||||
- **Impressions** (search visibility)
|
||||
- **Traffic** (organic visits)
|
||||
- **Priority Score** (0-100 opportunity rating)
|
||||
- **Status** (page 1 vs page 2-3)
|
||||
- **Recommendations** (how to improve)
|
||||
|
||||
### Priority Scoring (0-100)
|
||||
Higher scores = more opportunity for gain with less effort
|
||||
|
||||
Calculated from:
|
||||
- **Position (35%)** - How close to page 1
|
||||
- **Traffic Potential (30%)** - Search impressions
|
||||
- **CTR Gap (20%)** - Improvement opportunity
|
||||
- **Content Quality (15%)** - Existing engagement
|
||||
|
||||
## 🎯 Action Plan
|
||||
|
||||
### Week 1-2: Quick Wins (+100 visits/month)
|
||||
- Focus on posts at positions 11-15
|
||||
- Update SEO titles and meta descriptions
|
||||
- 30-60 minutes per post
|
||||
|
||||
### Week 3-4: Core Optimization (+150 visits/month)
|
||||
- Posts 6-15 in priority list
|
||||
- Add content sections
|
||||
- Improve structure with headers
|
||||
- 2-3 hours per post
|
||||
|
||||
### Week 5-8: New Content (+300 visits/month)
|
||||
- Create 3-5 new posts from gap analysis
|
||||
- Target high-search-demand topics
|
||||
- 4-6 hours per post
|
||||
|
||||
### Week 9-12: Refinement (+100 visits/month)
|
||||
- Monitor ranking improvements
|
||||
- Refine underperforming optimizations
|
||||
- Prepare next round of analysis
|
||||
|
||||
**Total: +650 visits/month potential gain**
|
||||
|
||||
## 🔧 Configuration
|
||||
|
||||
Edit `.env` to customize analysis:
|
||||
```bash
|
||||
# Position range for opportunities
|
||||
ANALYSIS_MIN_POSITION=11
|
||||
ANALYSIS_MAX_POSITION=30
|
||||
|
||||
# Minimum impressions to consider
|
||||
ANALYSIS_MIN_IMPRESSIONS=50
|
||||
|
||||
# Posts for AI recommendations
|
||||
ANALYSIS_TOP_N_POSTS=20
|
||||
```
|
||||
|
||||
## 🐛 Troubleshooting
|
||||
|
||||
### Missing Input Files
|
||||
```
|
||||
❌ Error: File not found: input/...
|
||||
```
|
||||
→ Check that all files are in the correct locations
|
||||
|
||||
### Empty Report Titles
|
||||
✓ FIXED - Now correctly loads post titles from multiple column names
|
||||
|
||||
### No Opportunities Found
|
||||
```
|
||||
⚠️ No opportunities found in specified range
|
||||
```
|
||||
→ Try lowering `ANALYSIS_MIN_IMPRESSIONS` in `.env`
|
||||
|
||||
### API Errors
|
||||
```
|
||||
❌ AI generation failed: ...
|
||||
```
|
||||
→ Check `OPENROUTER_API_KEY` in `.env` and account balance
|
||||
|
||||
## 📚 Additional Resources
|
||||
|
||||
- **`input/README.md`** - How to export analytics data
|
||||
- **`output/README.md`** - Output files guide
|
||||
- **`QUICKSTART_ANALYSIS.md`** - Step-by-step tutorial
|
||||
- **`ANALYSIS_SYSTEM.md`** - Technical documentation
|
||||
|
||||
## ✅ Success Checklist
|
||||
|
||||
- [ ] All input files placed in `input/` directory
|
||||
- [ ] `.env` file configured with API key
|
||||
- [ ] Ran `./run_analysis.sh` successfully
|
||||
- [ ] Reviewed `output/results/seo_optimization_report.md`
|
||||
- [ ] Identified 5-10 quick wins to start with
|
||||
- [ ] Created action plan for first week
|
||||
|
||||
## 🎓 Key Learnings
|
||||
|
||||
### Why Positions 11-30 Matter
|
||||
- **Page 1** posts are hard to move
|
||||
- **Page 2-3** posts are easy wins (small improvements move them up)
|
||||
- **Quick gains:** 1-2 position improvements = CTR increases 20-30%
|
||||
|
||||
### CTR Expectations by Position
|
||||
- Position 1: ~30% CTR
|
||||
- Position 5-10: 4-7% CTR
|
||||
- Position 11-15: 1-2% CTR (quick wins)
|
||||
- Position 16-20: 0.8-1% CTR
|
||||
- Position 21-30: ~0.5% CTR
|
||||
|
||||
### Content Quality Signals
|
||||
- Higher bounce rate = less relevant content
|
||||
- Low traffic = poor CTR or position
|
||||
- Low impressions = insufficient optimization
|
||||
|
||||
## 📞 Support
|
||||
|
||||
### Check Logs First
|
||||
```
|
||||
output/logs/import_log.txt
|
||||
output/logs/opportunity_analysis_log.txt
|
||||
output/logs/content_gap_analysis_log.txt
|
||||
```
|
||||
|
||||
### Common Issues
|
||||
1. **Empty titles** → Fixed with flexible column name mapping
|
||||
2. **File not found** → Check file locations match structure
|
||||
3. **API errors** → Verify API key and account balance
|
||||
4. **No opportunities** → Lower minimum impressions threshold
|
||||
|
||||
## 🚀 Ready to Optimize?
|
||||
|
||||
1. Prepare your input data
|
||||
2. Run `./run_analysis.sh`
|
||||
3. Open the report
|
||||
4. Start with quick wins
|
||||
5. Track improvements in 4 weeks
|
||||
|
||||
Good luck boosting your SEO! 📈
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** February 2026
|
||||
**System Status:** Production Ready ✅
|
||||
474
README.md
Normal file
474
README.md
Normal file
@@ -0,0 +1,474 @@
|
||||
# WordPress SEO Automation Tool
|
||||
|
||||
Programmatically optimize SEO titles and meta descriptions across all WordPress posts using AI-powered generation and a CSV review workflow.
|
||||
|
||||
## Features
|
||||
|
||||
- **AI-Powered SEO Generation**: Uses OpenRouter API (Claude, GPT-4, Llama, etc.) to create optimized titles and descriptions
|
||||
- **Plugin Support**: Auto-detects and works with both Yoast SEO and Rank Math
|
||||
- **CSV Review Workflow**: Generate proposals, review in Excel/Sheets, approve changes before applying
|
||||
- **Safety Features**: Dry-run mode, rollback CSV generation, detailed logging
|
||||
- **SEO Best Practices**: Enforces 50-60 char titles, 150-160 char descriptions, keyword optimization
|
||||
- **Batch Processing**: Handle hundreds or thousands of posts efficiently
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Prerequisites](#prerequisites)
|
||||
- [Installation](#installation)
|
||||
- [WordPress Configuration](#wordpress-configuration)
|
||||
- [OpenRouter API Setup](#openrouter-api-setup)
|
||||
- [Usage](#usage)
|
||||
- [Workflow](#workflow)
|
||||
- [SEO Plugin Comparison](#seo-plugin-comparison)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
- [Cost Estimates](#cost-estimates)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- WordPress site with Yoast SEO or Rank Math plugin installed
|
||||
- Python 3.8 or higher
|
||||
- WordPress Application Password (for REST API access)
|
||||
- OpenRouter API key (for AI-powered generation)
|
||||
|
||||
## Installation
|
||||
|
||||
### 1. Clone or Download
|
||||
|
||||
```bash
|
||||
cd /Users/acid/Documents/seo
|
||||
```
|
||||
|
||||
### 2. Create Virtual Environment
|
||||
|
||||
```bash
|
||||
python3 -m venv venv
|
||||
source venv/bin/activate # On Windows: venv\Scripts\activate
|
||||
```
|
||||
|
||||
### 3. Install Dependencies
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
### 4. Configure Environment Variables
|
||||
|
||||
Copy the example environment file:
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
Edit `.env` with your credentials:
|
||||
|
||||
```env
|
||||
WORDPRESS_URL=https://yoursite.com
|
||||
WORDPRESS_USERNAME=your_username
|
||||
WORDPRESS_APP_PASSWORD=your_application_password
|
||||
OPENROUTER_API_KEY=your_openrouter_api_key
|
||||
AI_MODEL=anthropic/claude-3.5-sonnet
|
||||
```
|
||||
|
||||
## WordPress Configuration
|
||||
|
||||
### Step 1: Create Application Password
|
||||
|
||||
1. Log in to WordPress Admin
|
||||
2. Go to **Users → Profile**
|
||||
3. Scroll to **Application Passwords** section
|
||||
4. Enter application name: "SEO Automation"
|
||||
5. Click **Add New Application Password**
|
||||
6. Copy the generated password (it will only be shown once)
|
||||
7. Add to `.env` file as `WORDPRESS_APP_PASSWORD`
|
||||
|
||||
### Step 2: Verify REST API Access
|
||||
|
||||
Test your authentication:
|
||||
|
||||
```bash
|
||||
curl --user "your_username:your_app_password" \
|
||||
https://yoursite.com/wp-json/wp/v2/posts?per_page=1&context=edit
|
||||
```
|
||||
|
||||
You should receive a JSON response with post data.
|
||||
|
||||
### Step 3: SEO Plugin Requirements
|
||||
|
||||
**For Yoast SEO:**
|
||||
- Yoast SEO Free or Premium installed and activated
|
||||
- Meta fields automatically accessible via REST API
|
||||
|
||||
**For Rank Math:**
|
||||
- Rank Math Free or Pro installed and activated
|
||||
- Meta fields automatically accessible via REST API
|
||||
|
||||
**Both plugins are supported** - the scripts auto-detect which one you're using.
|
||||
|
||||
## OpenRouter API Setup
|
||||
|
||||
### Why OpenRouter?
|
||||
|
||||
OpenRouter provides access to multiple AI models through a single API:
|
||||
- **Claude 3.5 Sonnet** (recommended): Best quality, $3/$15 per 1M tokens
|
||||
- **GPT-4 Turbo**: Strong performance, $10/$30 per 1M tokens
|
||||
- **Llama 3.1 70B**: Free tier available, $0/$0 per 1M tokens
|
||||
- **Gemini Pro 1.5**: Good balance, $1.25/$5 per 1M tokens
|
||||
|
||||
### Get API Key
|
||||
|
||||
1. Visit [https://openrouter.ai/](https://openrouter.ai/)
|
||||
2. Sign up or log in
|
||||
3. Go to **API Keys** section
|
||||
4. Create new API key
|
||||
5. Add to `.env` file as `OPENROUTER_API_KEY`
|
||||
|
||||
### Choose AI Model
|
||||
|
||||
Edit `AI_MODEL` in `.env`:
|
||||
|
||||
```env
|
||||
# Best quality (recommended)
|
||||
AI_MODEL=anthropic/claude-3.5-sonnet
|
||||
|
||||
# Budget option (free)
|
||||
AI_MODEL=meta-llama/llama-3.1-70b-instruct
|
||||
|
||||
# OpenAI
|
||||
AI_MODEL=openai/gpt-4-turbo
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Step 1: Generate SEO Proposals
|
||||
|
||||
Fetch all posts and generate AI-powered SEO suggestions:
|
||||
|
||||
```bash
|
||||
python fetch_posts_and_generate_seo.py
|
||||
```
|
||||
|
||||
**Options:**
|
||||
|
||||
```bash
|
||||
# Test with first 5 posts
|
||||
python fetch_posts_and_generate_seo.py --limit 5
|
||||
|
||||
# Specify output file
|
||||
python fetch_posts_and_generate_seo.py --output my_proposals.csv
|
||||
|
||||
# Use rule-based generation (no AI/API costs)
|
||||
python fetch_posts_and_generate_seo.py --no-ai
|
||||
```
|
||||
|
||||
This creates a CSV file in `output/` directory with proposals for all posts.
|
||||
|
||||
### Step 2: Review Proposals
|
||||
|
||||
1. Open the generated CSV file in Excel or Google Sheets
|
||||
2. Review each row:
|
||||
- Check `proposed_seo_title` (should be 50-60 chars)
|
||||
- Check `proposed_meta_description` (should be 150-160 chars)
|
||||
- Edit proposals if needed
|
||||
3. Set `status` column to `approved` for changes you want to apply
|
||||
4. Set `status` column to `rejected` for posts to skip
|
||||
5. Save the CSV file
|
||||
|
||||
**CSV Columns:**
|
||||
|
||||
| Column | Description |
|
||||
|--------|-------------|
|
||||
| post_id | WordPress post ID |
|
||||
| post_url | Post permalink |
|
||||
| post_title | Original post title |
|
||||
| current_seo_title | Current SEO title (from Yoast/Rank Math) |
|
||||
| current_meta_description | Current meta description |
|
||||
| proposed_seo_title | AI-generated SEO title |
|
||||
| proposed_meta_description | AI-generated meta description |
|
||||
| primary_keyword | Detected primary keyword |
|
||||
| title_length | Character count of proposed title |
|
||||
| description_length | Character count of proposed description |
|
||||
| title_validation | Validation message |
|
||||
| description_validation | Validation message |
|
||||
| generation_method | 'ai' or 'rule-based' |
|
||||
| status | Set to 'approved' to apply changes |
|
||||
| notes | Your notes (optional) |
|
||||
|
||||
### Step 3: Test with Dry Run
|
||||
|
||||
Before applying changes, test with dry-run mode:
|
||||
|
||||
```bash
|
||||
python apply_approved_changes.py --input output/seo_proposals_YYYYMMDD_HHMMSS.csv --dry-run
|
||||
```
|
||||
|
||||
This shows what would be updated without actually making changes.
|
||||
|
||||
### Step 4: Apply Approved Changes
|
||||
|
||||
Apply the approved changes to WordPress:
|
||||
|
||||
```bash
|
||||
python apply_approved_changes.py --input output/seo_proposals_YYYYMMDD_HHMMSS.csv
|
||||
```
|
||||
|
||||
The script will:
|
||||
1. Create a rollback CSV with original values
|
||||
2. Ask for confirmation
|
||||
3. Apply all approved changes
|
||||
4. Generate detailed log file
|
||||
|
||||
## Workflow
|
||||
|
||||
### Complete Workflow Diagram
|
||||
|
||||
```
|
||||
1. Generate Proposals
|
||||
└─> python fetch_posts_and_generate_seo.py
|
||||
└─> Fetches all posts from WordPress
|
||||
└─> Generates AI-powered SEO suggestions
|
||||
└─> Exports to CSV: output/seo_proposals_YYYYMMDD_HHMMSS.csv
|
||||
|
||||
2. Review & Edit
|
||||
└─> Open CSV in Excel/Google Sheets
|
||||
└─> Review proposed titles and descriptions
|
||||
└─> Edit as needed
|
||||
└─> Set status='approved' for changes to apply
|
||||
└─> Save CSV
|
||||
|
||||
3. Test (Optional)
|
||||
└─> python apply_approved_changes.py --input <csv> --dry-run
|
||||
└─> Simulates changes without applying
|
||||
|
||||
4. Apply Changes
|
||||
└─> python apply_approved_changes.py --input <csv>
|
||||
└─> Creates rollback CSV
|
||||
└─> Applies approved changes to WordPress
|
||||
└─> Generates log file
|
||||
|
||||
5. Verify
|
||||
└─> Check WordPress admin (post editor)
|
||||
└─> View source on frontend
|
||||
└─> Monitor search performance
|
||||
```
|
||||
|
||||
### Safety Features
|
||||
|
||||
- **Dry Run Mode**: Test without applying changes
|
||||
- **Rollback CSV**: Automatically created before applying changes
|
||||
- **Detailed Logging**: All operations logged to `output/application_log_YYYYMMDD_HHMMSS.txt`
|
||||
- **Validation**: Enforces character limits and checks for duplicates
|
||||
- **Confirmation Prompt**: Requires 'yes' confirmation before applying changes
|
||||
- **Rate Limiting**: Prevents overwhelming WordPress server
|
||||
|
||||
## SEO Plugin Comparison
|
||||
|
||||
### Should You Switch from Yoast to Rank Math?
|
||||
|
||||
**Current: Yoast SEO Free**
|
||||
- ✓ Market leader (12M users)
|
||||
- ✓ Reliable and well-tested
|
||||
- ✗ Only 1 focus keyword (vs unlimited in Rank Math)
|
||||
- ✗ No redirect manager (premium only, $118.80/year)
|
||||
- ✗ Limited schema support
|
||||
- ✗ No internal linking suggestions
|
||||
|
||||
**Alternative: Rank Math Free**
|
||||
- ✓ **Unlimited focus keywords** (vs 1 in Yoast Free)
|
||||
- ✓ **Redirect manager included** (premium in Yoast)
|
||||
- ✓ **20+ rich snippet types** (FAQ, Product, Recipe, etc.)
|
||||
- ✓ **Better performance** (40% less code)
|
||||
- ✓ **Internal linking suggestions**
|
||||
- ✓ **Google Trends integration**
|
||||
- ✓ **One-click Yoast migration** (preserves all data)
|
||||
- ✗ Smaller community (900K vs 12M users)
|
||||
|
||||
**Recommendation for FREE users:** Switch to Rank Math Free
|
||||
|
||||
**Migration Steps:**
|
||||
1. Install Rank Math plugin
|
||||
2. Run Setup Wizard → Import from Yoast
|
||||
3. All SEO data automatically transferred
|
||||
4. Deactivate (don't delete) Yoast as backup
|
||||
5. Test a few posts
|
||||
6. If satisfied, delete Yoast
|
||||
|
||||
**These scripts work with both plugins** - they auto-detect which one you're using.
|
||||
|
||||
## SEO Best Practices (2026)
|
||||
|
||||
### Title Optimization
|
||||
- **Length**: 50-60 characters (≤600 pixels in SERPs)
|
||||
- **Keyword placement**: Primary keyword in first 60 characters
|
||||
- **Uniqueness**: Every post must have unique title
|
||||
- **Compelling**: Written to improve click-through rate (CTR)
|
||||
- **Natural**: No keyword stuffing
|
||||
|
||||
### Meta Description Optimization
|
||||
- **Length**: 150-160 characters (optimal for SERP display)
|
||||
- **User intent**: Address what reader will learn/gain
|
||||
- **Keyword inclusion**: Primary keyword appears naturally
|
||||
- **Uniqueness**: Every post must have unique description
|
||||
- **Value proposition**: Highlight what makes content unique
|
||||
- **CTR focused**: Compelling language to encourage clicks
|
||||
|
||||
**Note**: Google rewrites 62%+ of meta descriptions, but they still matter for:
|
||||
- CTR when not overridden
|
||||
- Social media sharing (Open Graph)
|
||||
- Signaling relevance to search engines
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Error: "Authentication failed"
|
||||
|
||||
**Cause**: Invalid WordPress username or application password
|
||||
|
||||
**Solution**:
|
||||
1. Verify username is correct (not email address)
|
||||
2. Regenerate application password in WordPress
|
||||
3. Update `.env` file with new password
|
||||
4. Ensure no extra spaces in credentials
|
||||
|
||||
### Error: "Access forbidden"
|
||||
|
||||
**Cause**: User doesn't have permission to edit posts
|
||||
|
||||
**Solution**:
|
||||
1. Ensure user has Editor or Administrator role
|
||||
2. Check if REST API is disabled by security plugin
|
||||
3. Temporarily disable security plugins and test
|
||||
|
||||
### Error: "OpenRouter API key invalid"
|
||||
|
||||
**Cause**: Invalid or missing OpenRouter API key
|
||||
|
||||
**Solution**:
|
||||
1. Get API key from https://openrouter.ai/
|
||||
2. Update `OPENROUTER_API_KEY` in `.env`
|
||||
3. Ensure no extra quotes or spaces
|
||||
|
||||
### Error: "No posts found"
|
||||
|
||||
**Cause**: No published posts or authentication issue
|
||||
|
||||
**Solution**:
|
||||
1. Verify you have published posts in WordPress
|
||||
2. Check authentication is working (see WordPress Configuration)
|
||||
3. Try with `--limit 1` to test with single post
|
||||
|
||||
### SEO Plugin Not Detected
|
||||
|
||||
**Cause**: Plugin not installed or meta fields not exposed
|
||||
|
||||
**Solution**:
|
||||
1. Verify Yoast SEO or Rank Math is installed and activated
|
||||
2. Check if custom code blocks meta field access
|
||||
3. Scripts default to Yoast field names if detection fails
|
||||
|
||||
### AI Generation Fails
|
||||
|
||||
**Cause**: OpenRouter API error or rate limit
|
||||
|
||||
**Solution**:
|
||||
1. Check OpenRouter account has credits
|
||||
2. Try different AI model (switch to free Llama model)
|
||||
3. Use `--no-ai` flag for rule-based generation
|
||||
4. Check log files for specific error messages
|
||||
|
||||
## Cost Estimates
|
||||
|
||||
### OpenRouter API Costs
|
||||
|
||||
**Using Claude 3.5 Sonnet (Recommended):**
|
||||
- Average post: ~2000 tokens input + 200 tokens output
|
||||
- Cost per post: ~$0.009
|
||||
- **100 posts: ~$0.90**
|
||||
- **1000 posts: ~$9.00**
|
||||
|
||||
**Using Free Models:**
|
||||
- Llama 3.1 70B: **$0.00** (free tier)
|
||||
- No cost for generation
|
||||
|
||||
**Rule-Based Generation:**
|
||||
- No API costs
|
||||
- Use `--no-ai` flag
|
||||
- Lower quality but free
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
/Users/acid/Documents/seo/
|
||||
├── .env # Your credentials (git-ignored)
|
||||
├── .env.example # Example configuration
|
||||
├── .gitignore # Git ignore rules
|
||||
├── requirements.txt # Python dependencies
|
||||
├── config.py # Configuration loader
|
||||
├── seo_generator.py # SEO generation logic
|
||||
├── fetch_posts_and_generate_seo.py # Main fetching script
|
||||
├── apply_approved_changes.py # Application script
|
||||
├── README.md # This file
|
||||
└── output/ # Generated files
|
||||
├── seo_proposals_*.csv # Generated proposals
|
||||
├── rollback_*.csv # Backup files
|
||||
└── application_log_*.txt # Detailed logs
|
||||
```
|
||||
|
||||
## Development Notes
|
||||
|
||||
### Testing
|
||||
|
||||
**Test with small batch first:**
|
||||
|
||||
```bash
|
||||
# Generate proposals for 5 posts
|
||||
python fetch_posts_and_generate_seo.py --limit 5
|
||||
|
||||
# Review CSV and approve changes
|
||||
|
||||
# Dry run to verify
|
||||
python apply_approved_changes.py --input output/seo_proposals_*.csv --dry-run
|
||||
|
||||
# Apply to 5 posts
|
||||
python apply_approved_changes.py --input output/seo_proposals_*.csv
|
||||
```
|
||||
|
||||
**Verify changes:**
|
||||
1. Open WordPress post editor
|
||||
2. Check Yoast/Rank Math SEO box shows updated title and description
|
||||
3. View source on frontend: check `<title>` and `<meta name="description">` tags
|
||||
4. Test rollback CSV if needed
|
||||
|
||||
### Extending the Scripts
|
||||
|
||||
**Add custom validation:**
|
||||
- Edit `seo_generator.py` → `validate_seo_title()` and `validate_meta_description()`
|
||||
|
||||
**Change AI model:**
|
||||
- Edit `.env` → `AI_MODEL=openai/gpt-4-turbo`
|
||||
|
||||
**Customize prompts:**
|
||||
- Edit `seo_generator.py` → `_generate_with_ai()` method
|
||||
|
||||
**Add more meta fields:**
|
||||
- Edit scripts to include focus keywords, Open Graph tags, etc.
|
||||
|
||||
## Support
|
||||
|
||||
For issues or questions:
|
||||
1. Check this README troubleshooting section
|
||||
2. Review log files in `output/` directory
|
||||
3. Test with `--dry-run` mode first
|
||||
4. Start with `--limit 5` for testing
|
||||
|
||||
## License
|
||||
|
||||
This tool is provided as-is for WordPress SEO optimization. Use responsibly and always backup your WordPress site before bulk updates.
|
||||
|
||||
## Changelog
|
||||
|
||||
### Version 1.0.0 (2026-02-15)
|
||||
- Initial release
|
||||
- AI-powered SEO generation via OpenRouter
|
||||
- Support for Yoast SEO and Rank Math
|
||||
- CSV review workflow
|
||||
- Safety features (dry-run, rollback, logging)
|
||||
- Auto-detection of SEO plugins
|
||||
427
analytics_importer.py
Normal file
427
analytics_importer.py
Normal file
@@ -0,0 +1,427 @@
|
||||
"""
|
||||
Analytics data importer for SEO analysis.
|
||||
Merges Google Analytics and Search Console data with WordPress posts.
|
||||
"""
|
||||
|
||||
import csv
|
||||
import json
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
from urllib.parse import urlparse, parse_qs
|
||||
from collections import defaultdict
|
||||
from config import Config
|
||||
|
||||
|
||||
class AnalyticsImporter:
|
||||
"""Import and consolidate analytics data with WordPress posts."""
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize importer."""
|
||||
self.config = Config
|
||||
self.output_dir = self.config.OUTPUT_DIR
|
||||
self.logs = []
|
||||
self.unmatched_urls = []
|
||||
|
||||
def log(self, message):
|
||||
"""Add message to log."""
|
||||
self.logs.append(message)
|
||||
print(message)
|
||||
|
||||
def normalize_url(self, url):
|
||||
"""Normalize URL for matching."""
|
||||
if not url:
|
||||
return ""
|
||||
# Remove trailing slash, protocol, www
|
||||
url = url.rstrip('/')
|
||||
if url.startswith('http'):
|
||||
url = urlparse(url).path
|
||||
url = url.replace('www.', '')
|
||||
return url.lower()
|
||||
|
||||
def extract_post_slug_from_url(self, url):
|
||||
"""Extract post slug from URL path."""
|
||||
path = urlparse(url).path.rstrip('/')
|
||||
parts = [p for p in path.split('/') if p]
|
||||
if parts:
|
||||
return parts[-1] # Last part is usually the slug
|
||||
return None
|
||||
|
||||
def load_ga4_data(self, ga4_csv):
|
||||
"""Load Google Analytics 4 data."""
|
||||
ga_data = {}
|
||||
if not ga4_csv.exists():
|
||||
self.log(f"⚠️ GA4 file not found: {ga4_csv}")
|
||||
return ga_data
|
||||
|
||||
try:
|
||||
with open(ga4_csv, 'r', encoding='utf-8') as f:
|
||||
# Skip comment lines at the top (lines starting with #)
|
||||
lines = [line for line in f if not line.startswith('#')]
|
||||
|
||||
reader = csv.DictReader(lines)
|
||||
for row in reader:
|
||||
if not row:
|
||||
continue
|
||||
# Handle French and English column names
|
||||
url = (row.get('Page path and screen class') or
|
||||
row.get('Chemin de la page et classe de l\'écran') or
|
||||
row.get('Page path') or
|
||||
row.get('Page') or '')
|
||||
if not url:
|
||||
continue
|
||||
|
||||
# Normalize URL
|
||||
normalized = self.normalize_url(url)
|
||||
|
||||
# Extract metrics (handle French and English column names)
|
||||
try:
|
||||
traffic = int(float(row.get('Screened Views', row.get('Views', row.get('Vues', '0'))) or 0))
|
||||
users = int(float(row.get('Users', row.get('Utilisateurs actifs', '0')) or 0))
|
||||
bounce_rate = float(row.get('Bounce rate', row.get('Taux de rebond', '0')) or 0)
|
||||
avg_duration_str = (row.get('Average session duration',
|
||||
row.get('Durée d\'engagement moyenne par utilisateur actif', '0')) or '0')
|
||||
avg_duration = float(avg_duration_str.replace(',', '.'))
|
||||
except (ValueError, TypeError):
|
||||
traffic = users = 0
|
||||
bounce_rate = avg_duration = 0
|
||||
|
||||
ga_data[normalized] = {
|
||||
'traffic': traffic,
|
||||
'users': users,
|
||||
'bounce_rate': bounce_rate,
|
||||
'avg_session_duration': avg_duration,
|
||||
'ga_url': url
|
||||
}
|
||||
self.log(f"✓ Loaded {len(ga_data)} GA4 entries")
|
||||
except Exception as e:
|
||||
self.log(f"❌ Error reading GA4 file: {e}")
|
||||
|
||||
return ga_data
|
||||
|
||||
def load_gsc_data(self, gsc_csv):
|
||||
"""Load Google Search Console data (Page-level or Query-level)."""
|
||||
gsc_data = {}
|
||||
if not gsc_csv.exists():
|
||||
self.log(f"⚠️ GSC file not found: {gsc_csv}")
|
||||
return gsc_data
|
||||
|
||||
try:
|
||||
with open(gsc_csv, 'r', encoding='utf-8') as f:
|
||||
reader = csv.DictReader(f)
|
||||
for row in reader:
|
||||
if not row:
|
||||
continue
|
||||
|
||||
# Determine if this is page-level or query-level data
|
||||
# Pages.csv has: "Pages les plus populaires", Queries.csv has: "Requêtes les plus fréquentes"
|
||||
url = (row.get('Page') or
|
||||
row.get('Pages les plus populaires') or
|
||||
row.get('URL') or '')
|
||||
|
||||
query = row.get('Query') or row.get('Requêtes les plus fréquentes', '').strip()
|
||||
|
||||
# Skip rows without URLs (query-only data)
|
||||
if not url:
|
||||
continue
|
||||
|
||||
# Try to parse metrics with flexible column names
|
||||
try:
|
||||
# Handle different number formats (decimal separator, percentage signs)
|
||||
clicks_str = row.get('Clics', row.get('Clicks', '0')) or '0'
|
||||
impressions_str = row.get('Impressions', '0') or '0'
|
||||
ctr_str = row.get('CTR', '0') or '0'
|
||||
position_str = row.get('Position', '0') or '0'
|
||||
|
||||
clicks = int(float(clicks_str.replace(',', '.').rstrip('%')))
|
||||
impressions = int(float(impressions_str.replace(',', '.')))
|
||||
ctr = float(ctr_str.replace(',', '.').rstrip('%')) / 100
|
||||
position = float(position_str.replace(',', '.'))
|
||||
except (ValueError, TypeError, AttributeError):
|
||||
clicks = impressions = 0
|
||||
ctr = position = 0
|
||||
|
||||
normalized = self.normalize_url(url)
|
||||
|
||||
if normalized not in gsc_data:
|
||||
gsc_data[normalized] = {
|
||||
'impressions': 0,
|
||||
'clicks': 0,
|
||||
'avg_position': 0,
|
||||
'ctr': 0,
|
||||
'keywords': [],
|
||||
'gsc_url': url
|
||||
}
|
||||
|
||||
# Accumulate data (in case of multiple rows per URL)
|
||||
gsc_data[normalized]['impressions'] += impressions
|
||||
gsc_data[normalized]['clicks'] += clicks
|
||||
|
||||
# Store position
|
||||
if position > 0:
|
||||
gsc_data[normalized]['positions'] = gsc_data[normalized].get('positions', [])
|
||||
gsc_data[normalized]['positions'].append(position)
|
||||
|
||||
if query and query not in gsc_data[normalized]['keywords']:
|
||||
gsc_data[normalized]['keywords'].append(query)
|
||||
|
||||
# Calculate average positions and finalize
|
||||
for data in gsc_data.values():
|
||||
if data.get('positions'):
|
||||
data['avg_position'] = sum(data['positions']) / len(data['positions'])
|
||||
del data['positions']
|
||||
# Recalculate CTR from totals
|
||||
if data['impressions'] > 0:
|
||||
data['ctr'] = data['clicks'] / data['impressions']
|
||||
data['keywords_count'] = len(data.get('keywords', []))
|
||||
|
||||
self.log(f"✓ Loaded {len(gsc_data)} GSC entries")
|
||||
except Exception as e:
|
||||
self.log(f"❌ Error reading GSC file: {e}")
|
||||
|
||||
return gsc_data
|
||||
|
||||
def load_posts_csv(self, posts_csv):
|
||||
"""Load existing WordPress posts CSV."""
|
||||
posts = {}
|
||||
if not posts_csv.exists():
|
||||
self.log(f"⚠️ Posts file not found: {posts_csv}")
|
||||
return posts
|
||||
|
||||
try:
|
||||
with open(posts_csv, 'r', encoding='utf-8') as f:
|
||||
reader = csv.DictReader(f)
|
||||
for row in reader:
|
||||
# Handle different column name variations
|
||||
post_id = row.get('ID') or row.get('post_id')
|
||||
post_url = row.get('URL') or row.get('Post URL') or row.get('post_url')
|
||||
post_slug = row.get('Post Slug') or row.get('Slug') or row.get('post_slug')
|
||||
post_title = row.get('Title') or row.get('post_title')
|
||||
|
||||
if not post_id:
|
||||
continue
|
||||
|
||||
normalized = self.normalize_url(post_url) if post_url else ""
|
||||
|
||||
# Handle different SEO column names
|
||||
seo_title = (row.get('SEO Title') or
|
||||
row.get('proposed_seo_title') or
|
||||
row.get('current_seo_title') or '')
|
||||
meta_desc = (row.get('Meta Description') or
|
||||
row.get('proposed_meta_description') or
|
||||
row.get('current_meta_description') or '')
|
||||
|
||||
posts[post_id] = {
|
||||
'title': post_title or '',
|
||||
'url': post_url,
|
||||
'slug': post_slug,
|
||||
'normalized_url': normalized,
|
||||
'seo_title': seo_title,
|
||||
'meta_description': meta_desc,
|
||||
**{k: v for k, v in row.items()
|
||||
if k not in ['ID', 'post_id', 'Title', 'post_title', 'URL', 'Post URL', 'post_url',
|
||||
'Post Slug', 'Slug', 'post_slug', 'SEO Title', 'proposed_seo_title',
|
||||
'current_seo_title', 'Meta Description', 'proposed_meta_description',
|
||||
'current_meta_description']}
|
||||
}
|
||||
|
||||
self.log(f"✓ Loaded {len(posts)} posts from CSV")
|
||||
except Exception as e:
|
||||
self.log(f"❌ Error reading posts CSV: {e}")
|
||||
|
||||
return posts
|
||||
|
||||
def match_analytics_to_posts(self, posts, ga_data, gsc_data):
|
||||
"""Match analytics data to posts with fuzzy matching."""
|
||||
self.log("\n📊 Matching analytics data to posts...")
|
||||
matched_count = 0
|
||||
|
||||
for post_id, post_info in posts.items():
|
||||
slug = post_info.get('slug') or self.extract_post_slug_from_url(post_info.get('url', ''))
|
||||
normalized_url = post_info.get('normalized_url', '')
|
||||
|
||||
# Try direct URL match first
|
||||
if normalized_url in ga_data:
|
||||
post_info['ga_data'] = ga_data[normalized_url]
|
||||
matched_count += 1
|
||||
else:
|
||||
post_info['ga_data'] = {}
|
||||
|
||||
if normalized_url in gsc_data:
|
||||
post_info['gsc_data'] = gsc_data[normalized_url]
|
||||
matched_count += 1
|
||||
else:
|
||||
post_info['gsc_data'] = {}
|
||||
|
||||
# Try slug-based matching if URL didn't match
|
||||
if not post_info.get('gsc_data') and slug:
|
||||
for gsc_url, gsc_info in gsc_data.items():
|
||||
if slug in gsc_url:
|
||||
post_info['gsc_data'] = gsc_info
|
||||
break
|
||||
|
||||
# Track unmatched GSC URLs
|
||||
matched_gsc_urls = set()
|
||||
for post in posts.values():
|
||||
if post.get('gsc_data'):
|
||||
matched_gsc_urls.add(id(post['gsc_data']))
|
||||
|
||||
for normalized_url, gsc_info in gsc_data.items():
|
||||
if id(gsc_info) not in matched_gsc_urls and gsc_info.get('impressions', 0) > 0:
|
||||
self.unmatched_urls.append({
|
||||
'url': gsc_info.get('gsc_url', normalized_url),
|
||||
'impressions': gsc_info.get('impressions', 0),
|
||||
'clicks': gsc_info.get('clicks', 0),
|
||||
'avg_position': gsc_info.get('avg_position', 0)
|
||||
})
|
||||
|
||||
self.log(f"✓ Matched data to posts")
|
||||
return posts
|
||||
|
||||
def enrich_posts_data(self, posts):
|
||||
"""Enrich posts with calculated metrics."""
|
||||
for post_info in posts.values():
|
||||
ga = post_info.get('ga_data', {})
|
||||
gsc = post_info.get('gsc_data', {})
|
||||
|
||||
# GA metrics
|
||||
post_info['traffic'] = ga.get('traffic', 0)
|
||||
post_info['users'] = ga.get('users', 0)
|
||||
post_info['bounce_rate'] = ga.get('bounce_rate', 0)
|
||||
post_info['avg_session_duration'] = ga.get('avg_session_duration', 0)
|
||||
|
||||
# GSC metrics
|
||||
post_info['impressions'] = gsc.get('impressions', 0)
|
||||
post_info['clicks'] = gsc.get('clicks', 0)
|
||||
post_info['avg_position'] = gsc.get('avg_position', 0)
|
||||
post_info['ctr'] = gsc.get('ctr', 0)
|
||||
post_info['keywords_count'] = gsc.get('keywords_count', 0)
|
||||
post_info['top_keywords'] = ','.join(gsc.get('keywords', [])[:5])
|
||||
|
||||
return posts
|
||||
|
||||
def export_enriched_csv(self, posts, output_csv):
|
||||
"""Export enriched posts data to CSV."""
|
||||
if not posts:
|
||||
self.log("❌ No posts to export")
|
||||
return
|
||||
|
||||
try:
|
||||
fieldnames = [
|
||||
'ID', 'Title', 'URL', 'SEO Title', 'Meta Description',
|
||||
'traffic', 'users', 'bounce_rate', 'avg_session_duration',
|
||||
'impressions', 'clicks', 'avg_position', 'ctr', 'keywords_count', 'top_keywords'
|
||||
]
|
||||
|
||||
# Add any extra fields from original posts
|
||||
all_keys = set()
|
||||
for post in posts.values():
|
||||
all_keys.update(post.keys())
|
||||
|
||||
extra_fields = [k for k in sorted(all_keys)
|
||||
if k not in fieldnames and k not in ['ga_data', 'gsc_data', 'normalized_url', 'slug']]
|
||||
fieldnames.extend(extra_fields)
|
||||
|
||||
with open(output_csv, 'w', newline='', encoding='utf-8') as f:
|
||||
writer = csv.DictWriter(f, fieldnames=fieldnames, extrasaction='ignore')
|
||||
writer.writeheader()
|
||||
|
||||
for post_id, post_info in sorted(posts.items()):
|
||||
row = {'ID': post_id}
|
||||
row.update(post_info)
|
||||
# Clean up nested dicts
|
||||
for key in ['ga_data', 'gsc_data']:
|
||||
row.pop(key, None)
|
||||
writer.writerow(row)
|
||||
|
||||
self.log(f"✓ Exported {len(posts)} posts to {output_csv}")
|
||||
except Exception as e:
|
||||
self.log(f"❌ Error exporting CSV: {e}")
|
||||
|
||||
def export_log(self, log_file):
|
||||
"""Export analysis log and unmatched URLs."""
|
||||
try:
|
||||
with open(log_file, 'w', encoding='utf-8') as f:
|
||||
f.write("SEO Analytics Import Report\n")
|
||||
f.write("=" * 60 + "\n\n")
|
||||
|
||||
f.write("Import Log:\n")
|
||||
f.write("-" * 60 + "\n")
|
||||
for log_msg in self.logs:
|
||||
f.write(log_msg + "\n")
|
||||
|
||||
f.write("\n" + "=" * 60 + "\n")
|
||||
f.write(f"Unmatched URLs ({len(self.unmatched_urls)} total):\n")
|
||||
f.write("-" * 60 + "\n")
|
||||
|
||||
if self.unmatched_urls:
|
||||
# Sort by impressions descending
|
||||
for url_data in sorted(self.unmatched_urls,
|
||||
key=lambda x: x['impressions'],
|
||||
reverse=True):
|
||||
f.write(f"\nURL: {url_data['url']}\n")
|
||||
f.write(f" Impressions: {url_data['impressions']}\n")
|
||||
f.write(f" Clicks: {url_data['clicks']}\n")
|
||||
f.write(f" Avg Position: {url_data['avg_position']:.1f}\n")
|
||||
else:
|
||||
f.write("✓ All URLs matched successfully!\n")
|
||||
|
||||
self.log(f"✓ Exported log to {log_file}")
|
||||
except Exception as e:
|
||||
self.log(f"❌ Error exporting log: {e}")
|
||||
|
||||
def run(self, ga_csv, gsc_csv, posts_csv, output_csv):
|
||||
"""Run complete import workflow."""
|
||||
self.log("Starting analytics import...")
|
||||
self.log(f"GA4 CSV: {ga_csv}")
|
||||
self.log(f"GSC CSV: {gsc_csv}")
|
||||
self.log(f"Posts CSV: {posts_csv}\n")
|
||||
|
||||
# Load data
|
||||
ga_data = self.load_ga4_data(ga_csv)
|
||||
gsc_data = self.load_gsc_data(gsc_csv)
|
||||
posts = self.load_posts_csv(posts_csv)
|
||||
|
||||
if not posts:
|
||||
self.log("❌ No posts found. Cannot proceed.")
|
||||
return
|
||||
|
||||
# Match and merge
|
||||
posts = self.match_analytics_to_posts(posts, ga_data, gsc_data)
|
||||
posts = self.enrich_posts_data(posts)
|
||||
|
||||
# Export
|
||||
self.export_enriched_csv(posts, output_csv)
|
||||
|
||||
# Export log
|
||||
log_dir = self.output_dir / 'logs'
|
||||
log_dir.mkdir(exist_ok=True)
|
||||
log_file = log_dir / 'import_log.txt'
|
||||
self.export_log(log_file)
|
||||
|
||||
self.log("\n✓ Analytics import complete!")
|
||||
|
||||
|
||||
def main():
|
||||
"""CLI entry point."""
|
||||
parser = argparse.ArgumentParser(description='Import and merge analytics data')
|
||||
parser.add_argument('--ga-export', type=Path,
|
||||
default=Path('input/analytics/ga4_export.csv'),
|
||||
help='GA4 export CSV path')
|
||||
parser.add_argument('--gsc-export', type=Path,
|
||||
default=Path('input/analytics/gsc/Pages.csv'),
|
||||
help='Search Console export CSV path (Pages data)')
|
||||
parser.add_argument('--posts-csv', type=Path,
|
||||
default=Path('input/new-propositions.csv'),
|
||||
help='Posts CSV path')
|
||||
parser.add_argument('--output', type=Path,
|
||||
default=Path('output/results/posts_with_analytics.csv'),
|
||||
help='Output CSV path')
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
importer = AnalyticsImporter()
|
||||
importer.run(args.ga_export, args.gsc_export, args.posts_csv, args.output)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
71
config.py
Normal file
71
config.py
Normal file
@@ -0,0 +1,71 @@
|
||||
"""
|
||||
Configuration module for WordPress SEO automation.
|
||||
Loads and validates environment variables.
|
||||
"""
|
||||
|
||||
import os
|
||||
from dotenv import load_dotenv
|
||||
from pathlib import Path
|
||||
|
||||
# Load environment variables from .env file
|
||||
load_dotenv()
|
||||
|
||||
class Config:
|
||||
"""Configuration class for WordPress SEO automation."""
|
||||
|
||||
# WordPress Settings
|
||||
WORDPRESS_URL = os.getenv('WORDPRESS_URL', '').rstrip('/')
|
||||
WORDPRESS_USERNAME = os.getenv('WORDPRESS_USERNAME', '')
|
||||
WORDPRESS_APP_PASSWORD = os.getenv('WORDPRESS_APP_PASSWORD', '')
|
||||
|
||||
# OpenRouter API Settings
|
||||
OPENROUTER_API_KEY = os.getenv('OPENROUTER_API_KEY', '')
|
||||
AI_MODEL = os.getenv('AI_MODEL', 'anthropic/claude-3.5-sonnet')
|
||||
|
||||
# Script Settings
|
||||
BATCH_SIZE = int(os.getenv('BATCH_SIZE', '100'))
|
||||
API_DELAY_SECONDS = float(os.getenv('API_DELAY_SECONDS', '0.5'))
|
||||
|
||||
# Analysis Settings
|
||||
ANALYSIS_MIN_POSITION = int(os.getenv('ANALYSIS_MIN_POSITION', '11'))
|
||||
ANALYSIS_MAX_POSITION = int(os.getenv('ANALYSIS_MAX_POSITION', '30'))
|
||||
ANALYSIS_MIN_IMPRESSIONS = int(os.getenv('ANALYSIS_MIN_IMPRESSIONS', '50'))
|
||||
ANALYSIS_TOP_N_POSTS = int(os.getenv('ANALYSIS_TOP_N_POSTS', '20'))
|
||||
|
||||
# Output directory
|
||||
OUTPUT_DIR = Path(__file__).parent / 'output'
|
||||
|
||||
@classmethod
|
||||
def validate(cls):
|
||||
"""Validate that all required configuration is present."""
|
||||
errors = []
|
||||
|
||||
if not cls.WORDPRESS_URL:
|
||||
errors.append("WORDPRESS_URL is required")
|
||||
|
||||
if not cls.WORDPRESS_USERNAME:
|
||||
errors.append("WORDPRESS_USERNAME is required")
|
||||
|
||||
if not cls.WORDPRESS_APP_PASSWORD:
|
||||
errors.append("WORDPRESS_APP_PASSWORD is required")
|
||||
|
||||
if not cls.OPENROUTER_API_KEY:
|
||||
errors.append("OPENROUTER_API_KEY is required (get one from https://openrouter.ai/)")
|
||||
|
||||
if errors:
|
||||
raise ValueError("Configuration errors:\n" + "\n".join(f" - {e}" for e in errors))
|
||||
|
||||
# Create output directory if it doesn't exist
|
||||
cls.OUTPUT_DIR.mkdir(exist_ok=True)
|
||||
|
||||
return True
|
||||
|
||||
@classmethod
|
||||
def get_wordpress_auth(cls):
|
||||
"""Get WordPress authentication tuple."""
|
||||
return (cls.WORDPRESS_USERNAME, cls.WORDPRESS_APP_PASSWORD)
|
||||
|
||||
@classmethod
|
||||
def get_api_base_url(cls):
|
||||
"""Get WordPress REST API base URL."""
|
||||
return f"{cls.WORDPRESS_URL}/wp-json/wp/v2"
|
||||
348
content_gap_analyzer.py
Normal file
348
content_gap_analyzer.py
Normal file
@@ -0,0 +1,348 @@
|
||||
"""
|
||||
Content gap analyzer for SEO strategy.
|
||||
Identifies missing topics and content opportunities using AI analysis.
|
||||
"""
|
||||
|
||||
import csv
|
||||
import json
|
||||
import argparse
|
||||
import time
|
||||
from pathlib import Path
|
||||
from collections import defaultdict
|
||||
from openai import OpenAI
|
||||
from config import Config
|
||||
|
||||
|
||||
class ContentGapAnalyzer:
|
||||
"""Identify content gaps and opportunities."""
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize analyzer."""
|
||||
self.config = Config
|
||||
self.output_dir = self.config.OUTPUT_DIR
|
||||
self.logs = []
|
||||
self.client = None
|
||||
|
||||
if self.config.OPENROUTER_API_KEY:
|
||||
self.client = OpenAI(
|
||||
base_url="https://openrouter.ai/api/v1",
|
||||
api_key=self.config.OPENROUTER_API_KEY,
|
||||
)
|
||||
|
||||
def log(self, message):
|
||||
"""Add message to log."""
|
||||
self.logs.append(message)
|
||||
print(message)
|
||||
|
||||
def load_posts(self, posts_csv):
|
||||
"""Load post titles and data."""
|
||||
posts = []
|
||||
if not posts_csv.exists():
|
||||
self.log(f"❌ File not found: {posts_csv}")
|
||||
return posts
|
||||
|
||||
try:
|
||||
with open(posts_csv, 'r', encoding='utf-8') as f:
|
||||
reader = csv.DictReader(f)
|
||||
for row in reader:
|
||||
posts.append({
|
||||
'id': row.get('ID', ''),
|
||||
'title': row.get('Title', ''),
|
||||
'url': row.get('URL', ''),
|
||||
'traffic': int(row.get('traffic', 0) or 0),
|
||||
'impressions': int(row.get('impressions', 0) or 0),
|
||||
'top_keywords': row.get('top_keywords', '')
|
||||
})
|
||||
|
||||
self.log(f"✓ Loaded {len(posts)} posts")
|
||||
except Exception as e:
|
||||
self.log(f"❌ Error reading posts: {e}")
|
||||
|
||||
return posts
|
||||
|
||||
def load_gsc_data(self, gsc_csv):
|
||||
"""Load Search Console queries for gap analysis."""
|
||||
queries = []
|
||||
if not gsc_csv.exists():
|
||||
self.log(f"⚠️ GSC file not found: {gsc_csv}")
|
||||
return queries
|
||||
|
||||
try:
|
||||
with open(gsc_csv, 'r', encoding='utf-8') as f:
|
||||
reader = csv.DictReader(f)
|
||||
for row in reader:
|
||||
try:
|
||||
query = row.get('Query', '').strip()
|
||||
if not query:
|
||||
continue
|
||||
|
||||
impressions = int(row.get('Impressions', 0) or 0)
|
||||
clicks = int(row.get('Clicks', 0) or 0)
|
||||
|
||||
# Only include queries with impressions but low clicks
|
||||
if impressions > 0 and (clicks / impressions < 0.05):
|
||||
queries.append({
|
||||
'query': query,
|
||||
'impressions': impressions,
|
||||
'clicks': clicks,
|
||||
'ctr': clicks / impressions if impressions > 0 else 0
|
||||
})
|
||||
except (ValueError, TypeError):
|
||||
continue
|
||||
|
||||
self.log(f"✓ Loaded {len(queries)} underperforming queries")
|
||||
except Exception as e:
|
||||
self.log(f"⚠️ Error reading GSC file: {e}")
|
||||
|
||||
return queries
|
||||
|
||||
def extract_topics(self, posts):
|
||||
"""Extract topic clusters from post titles using AI."""
|
||||
if not self.client or len(posts) == 0:
|
||||
self.log("⚠️ Cannot extract topics without AI client or posts")
|
||||
return {}
|
||||
|
||||
try:
|
||||
self.log("🤖 Extracting topic clusters from post titles...")
|
||||
|
||||
# Batch posts into groups
|
||||
titles = [p['title'] for p in posts][:100] # Limit to first 100
|
||||
|
||||
prompt = f"""Analyze these {len(titles)} blog post titles and identify topic clusters:
|
||||
|
||||
Titles:
|
||||
{chr(10).join(f'{i+1}. {t}' for i, t in enumerate(titles))}
|
||||
|
||||
Extract for each post:
|
||||
1. Primary topic category
|
||||
2. Subtopics covered
|
||||
3. Content type (guide, tutorial, review, comparison, etc.)
|
||||
|
||||
Then identify:
|
||||
1. Top 10 topic clusters with post counts
|
||||
2. Most common subtopics
|
||||
3. Over/under-represented topics
|
||||
|
||||
Return JSON:
|
||||
{{
|
||||
"post_topics": {{
|
||||
"1": {{"primary": "...", "subtopics": ["..."], "type": "..."}},
|
||||
...
|
||||
}},
|
||||
"topic_clusters": [
|
||||
{{"cluster": "...", "post_count": 0, "importance": "high/medium/low"}}
|
||||
],
|
||||
"coverage_gaps": ["topic 1", "topic 2", ...],
|
||||
"niche": "detected niche or industry"
|
||||
}}"""
|
||||
|
||||
response = self.client.chat.completions.create(
|
||||
model=self.config.AI_MODEL,
|
||||
messages=[{"role": "user", "content": prompt}],
|
||||
temperature=0.7,
|
||||
max_tokens=1500
|
||||
)
|
||||
|
||||
try:
|
||||
result_text = response.choices[0].message.content
|
||||
start_idx = result_text.find('{')
|
||||
end_idx = result_text.rfind('}') + 1
|
||||
if start_idx >= 0 and end_idx > start_idx:
|
||||
return json.loads(result_text[start_idx:end_idx])
|
||||
except json.JSONDecodeError:
|
||||
self.log("⚠️ Could not parse topic extraction response")
|
||||
return {}
|
||||
|
||||
except Exception as e:
|
||||
self.log(f"⚠️ Topic extraction failed: {e}")
|
||||
return {}
|
||||
|
||||
def identify_content_gaps(self, topic_analysis, queries):
|
||||
"""Use AI to identify content gaps and suggest new topics."""
|
||||
if not self.client:
|
||||
return []
|
||||
|
||||
try:
|
||||
self.log("🤖 Identifying content gaps and opportunities...")
|
||||
|
||||
clusters = topic_analysis.get('topic_clusters', [])
|
||||
gaps = topic_analysis.get('coverage_gaps', [])
|
||||
niche = topic_analysis.get('niche', 'general')
|
||||
|
||||
# Prepare query analysis
|
||||
top_queries = sorted(queries, key=lambda x: x['impressions'], reverse=True)[:20]
|
||||
queries_str = '\n'.join([f"- {q['query']} ({q['impressions']} impr, {q['ctr']:.1%} CTR)"
|
||||
for q in top_queries])
|
||||
|
||||
prompt = f"""Based on content analysis and search demand, identify content gaps:
|
||||
|
||||
Existing Topics: {', '.join([c.get('cluster', '') for c in clusters[:10]])}
|
||||
Coverage Gaps: {', '.join(gaps[:5])}
|
||||
Niche: {niche}
|
||||
|
||||
Top Underperforming Queries (low CTR despite impressions):
|
||||
{queries_str}
|
||||
|
||||
Identify high-value missing topics that could:
|
||||
1. Fill coverage gaps
|
||||
2. Target underperforming queries (CTR improvement)
|
||||
3. Capitalize on search demand
|
||||
4. Complement existing content
|
||||
|
||||
For each suggestion:
|
||||
- Topic title
|
||||
- Why it's valuable (search demand + intent)
|
||||
- Search volume estimate (high/medium/low)
|
||||
- How it complements existing content
|
||||
- Recommended content format
|
||||
- Estimated traffic potential
|
||||
|
||||
Prioritize by traffic opportunity. Max 20 ideas.
|
||||
|
||||
Return JSON:
|
||||
{{
|
||||
"content_opportunities": [
|
||||
{{
|
||||
"title": "...",
|
||||
"why_valuable": "...",
|
||||
"search_volume": "high/medium/low",
|
||||
"complements": "existing topic",
|
||||
"format": "guide/tutorial/comparison/review/list",
|
||||
"traffic_potential": number,
|
||||
"priority": "high/medium/low"
|
||||
}}
|
||||
]
|
||||
}}"""
|
||||
|
||||
response = self.client.chat.completions.create(
|
||||
model=self.config.AI_MODEL,
|
||||
messages=[{"role": "user", "content": prompt}],
|
||||
temperature=0.7,
|
||||
max_tokens=2000
|
||||
)
|
||||
|
||||
try:
|
||||
result_text = response.choices[0].message.content
|
||||
start_idx = result_text.find('{')
|
||||
end_idx = result_text.rfind('}') + 1
|
||||
if start_idx >= 0 and end_idx > start_idx:
|
||||
result = json.loads(result_text[start_idx:end_idx])
|
||||
return result.get('content_opportunities', [])
|
||||
except json.JSONDecodeError:
|
||||
self.log("⚠️ Could not parse gap analysis response")
|
||||
return []
|
||||
|
||||
except Exception as e:
|
||||
self.log(f"⚠️ Gap analysis failed: {e}")
|
||||
return []
|
||||
|
||||
def export_gaps_csv(self, gaps, output_csv):
|
||||
"""Export content gaps to CSV."""
|
||||
if not gaps:
|
||||
self.log("⚠️ No gaps to export")
|
||||
return
|
||||
|
||||
try:
|
||||
fieldnames = [
|
||||
'priority', 'title', 'why_valuable', 'search_volume',
|
||||
'complements', 'format', 'traffic_potential'
|
||||
]
|
||||
|
||||
with open(output_csv, 'w', newline='', encoding='utf-8') as f:
|
||||
writer = csv.DictWriter(f, fieldnames=fieldnames, extrasaction='ignore')
|
||||
writer.writeheader()
|
||||
|
||||
for gap in sorted(gaps, key=lambda x: x.get('priority') == 'high', reverse=True):
|
||||
writer.writerow(gap)
|
||||
|
||||
self.log(f"✓ Exported {len(gaps)} content gaps to {output_csv}")
|
||||
except Exception as e:
|
||||
self.log(f"❌ Error exporting CSV: {e}")
|
||||
|
||||
def export_topic_clusters_json(self, topic_analysis, output_json):
|
||||
"""Export topic analysis to JSON."""
|
||||
if not topic_analysis:
|
||||
return
|
||||
|
||||
try:
|
||||
with open(output_json, 'w', encoding='utf-8') as f:
|
||||
json.dump(topic_analysis, f, indent=2)
|
||||
|
||||
self.log(f"✓ Exported topic analysis to {output_json}")
|
||||
except Exception as e:
|
||||
self.log(f"❌ Error exporting JSON: {e}")
|
||||
|
||||
def export_log(self, log_file):
|
||||
"""Export analysis log."""
|
||||
try:
|
||||
with open(log_file, 'w', encoding='utf-8') as f:
|
||||
f.write("Content Gap Analysis Report\n")
|
||||
f.write("=" * 60 + "\n\n")
|
||||
|
||||
for msg in self.logs:
|
||||
f.write(msg + "\n")
|
||||
|
||||
self.log(f"✓ Exported log to {log_file}")
|
||||
except Exception as e:
|
||||
self.log(f"❌ Error exporting log: {e}")
|
||||
|
||||
def run(self, posts_csv, gsc_csv, output_csv):
|
||||
"""Run complete analysis workflow."""
|
||||
self.log("📊 Starting content gap analysis...")
|
||||
self.log(f"Posts: {posts_csv}")
|
||||
self.log(f"GSC queries: {gsc_csv}\n")
|
||||
|
||||
# Load data
|
||||
posts = self.load_posts(posts_csv)
|
||||
queries = self.load_gsc_data(gsc_csv)
|
||||
|
||||
if not posts:
|
||||
return
|
||||
|
||||
# Extract topics
|
||||
topic_analysis = self.extract_topics(posts)
|
||||
if topic_analysis:
|
||||
self.log(f"✓ Identified {len(topic_analysis.get('topic_clusters', []))} topic clusters")
|
||||
|
||||
# Identify gaps
|
||||
gaps = self.identify_content_gaps(topic_analysis, queries)
|
||||
if gaps:
|
||||
self.log(f"✓ Identified {len(gaps)} content opportunities")
|
||||
|
||||
# Export
|
||||
self.log("\n📁 Exporting results...")
|
||||
self.export_gaps_csv(gaps, output_csv)
|
||||
|
||||
topic_json = self.output_dir / 'topic_clusters.json'
|
||||
self.export_topic_clusters_json(topic_analysis, topic_json)
|
||||
|
||||
# Export log
|
||||
log_dir = self.output_dir / 'logs'
|
||||
log_dir.mkdir(exist_ok=True)
|
||||
log_file = log_dir / 'content_gap_analysis_log.txt'
|
||||
self.export_log(log_file)
|
||||
|
||||
self.log("\n✓ Content gap analysis complete!")
|
||||
|
||||
|
||||
def main():
|
||||
"""CLI entry point."""
|
||||
parser = argparse.ArgumentParser(description='Analyze content gaps')
|
||||
parser.add_argument('--posts-csv', type=Path,
|
||||
default=Path('output/results/posts_with_analytics.csv'),
|
||||
help='Posts CSV')
|
||||
parser.add_argument('--gsc-queries', type=Path,
|
||||
default=Path('input/analytics/gsc/Requêtes.csv'),
|
||||
help='GSC queries CSV')
|
||||
parser.add_argument('--output', type=Path,
|
||||
default=Path('output/results/content_gaps.csv'),
|
||||
help='Output gaps CSV')
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
analyzer = ContentGapAnalyzer()
|
||||
analyzer.run(args.posts_csv, args.gsc_queries, args.output)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
49
input/README.md
Normal file
49
input/README.md
Normal file
@@ -0,0 +1,49 @@
|
||||
# Input Directory
|
||||
|
||||
Place your source data files here before running the analysis pipeline.
|
||||
|
||||
## Required Files
|
||||
|
||||
### `new-propositions.csv`
|
||||
WordPress posts export with SEO metadata
|
||||
- Columns: ID, post_id, Title, post_title, URL, post_url, SEO Title, Meta Description, etc.
|
||||
|
||||
### `analytics/ga4_export.csv`
|
||||
Google Analytics 4 data export
|
||||
- Date range: Last 90 days
|
||||
- Columns: Chemin de la page et classe de l'écran (Page path), Vues (Views), Utilisateurs actifs (Users), Durée d'engagement (Duration), etc.
|
||||
|
||||
### `analytics/gsc/Pages.csv`
|
||||
Google Search Console Pages report
|
||||
- Date range: Last 90 days
|
||||
- Columns: Pages les plus populaires (Page), Clics (Clicks), Impressions, CTR, Position
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
input/
|
||||
├── new-propositions.csv (WordPress posts)
|
||||
└── analytics/
|
||||
├── ga4_export.csv (Google Analytics data)
|
||||
└── gsc/
|
||||
├── Pages.csv (GSC pages report)
|
||||
├── Requêtes.csv (GSC queries report - optional)
|
||||
└── [other GSC exports]
|
||||
```
|
||||
|
||||
## How to Export Data
|
||||
|
||||
### Google Analytics 4
|
||||
1. Go to Analytics > Reports > Engagement > Pages and Screens
|
||||
2. Set date range to Last 90 days
|
||||
3. Click Export > Download CSV
|
||||
4. Save as: `input/analytics/ga4_export.csv`
|
||||
|
||||
### Google Search Console
|
||||
1. Go to Performance
|
||||
2. Set date range to Last 90 days
|
||||
3. Click Export > Download CSV
|
||||
4. Save as: `input/analytics/gsc/Pages.csv`
|
||||
|
||||
### WordPress Posts
|
||||
Use your existing WordPress export or the SEO propositions CSV
|
||||
BIN
input/new-propositions.ods
Normal file
BIN
input/new-propositions.ods
Normal file
Binary file not shown.
347
opportunity_analyzer.py
Normal file
347
opportunity_analyzer.py
Normal file
@@ -0,0 +1,347 @@
|
||||
"""
|
||||
Keyword opportunity analyzer for SEO optimization.
|
||||
Identifies high-potential keywords ranking at positions 11-30.
|
||||
"""
|
||||
|
||||
import csv
|
||||
import json
|
||||
import argparse
|
||||
import time
|
||||
from pathlib import Path
|
||||
from openai import OpenAI
|
||||
from config import Config
|
||||
|
||||
|
||||
class OpportunityAnalyzer:
|
||||
"""Analyze keyword opportunities for SEO optimization."""
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize analyzer."""
|
||||
self.config = Config
|
||||
self.output_dir = self.config.OUTPUT_DIR
|
||||
self.logs = []
|
||||
self.client = None
|
||||
|
||||
if self.config.OPENROUTER_API_KEY:
|
||||
self.client = OpenAI(
|
||||
base_url="https://openrouter.ai/api/v1",
|
||||
api_key=self.config.OPENROUTER_API_KEY,
|
||||
)
|
||||
|
||||
def log(self, message):
|
||||
"""Add message to log."""
|
||||
self.logs.append(message)
|
||||
print(message)
|
||||
|
||||
def load_posts(self, posts_csv):
|
||||
"""Load posts with analytics data."""
|
||||
posts = []
|
||||
if not posts_csv.exists():
|
||||
self.log(f"❌ File not found: {posts_csv}")
|
||||
return posts
|
||||
|
||||
try:
|
||||
with open(posts_csv, 'r', encoding='utf-8') as f:
|
||||
reader = csv.DictReader(f)
|
||||
for row in reader:
|
||||
try:
|
||||
posts.append({
|
||||
'id': row.get('ID', ''),
|
||||
'title': row.get('Title', ''),
|
||||
'url': row.get('URL', ''),
|
||||
'impressions': int(row.get('impressions', 0) or 0),
|
||||
'clicks': int(row.get('clicks', 0) or 0),
|
||||
'avg_position': float(row.get('avg_position', 0) or 0),
|
||||
'ctr': float(row.get('ctr', 0) or 0),
|
||||
'traffic': int(row.get('traffic', 0) or 0),
|
||||
'bounce_rate': float(row.get('bounce_rate', 0) or 0),
|
||||
'keywords_count': int(row.get('keywords_count', 0) or 0),
|
||||
'top_keywords': row.get('top_keywords', '')
|
||||
})
|
||||
except (ValueError, TypeError):
|
||||
continue
|
||||
|
||||
self.log(f"✓ Loaded {len(posts)} posts")
|
||||
except Exception as e:
|
||||
self.log(f"❌ Error reading posts: {e}")
|
||||
|
||||
return posts
|
||||
|
||||
def filter_opportunities(self, posts, min_pos, max_pos, min_impressions):
|
||||
"""Filter posts with keywords in opportunity range or high traffic for optimization."""
|
||||
opportunities = []
|
||||
|
||||
for post in posts:
|
||||
position = post.get('avg_position', 0)
|
||||
impressions = post.get('impressions', 0)
|
||||
traffic = post.get('traffic', 0)
|
||||
|
||||
# Primary filter: position range (if data available)
|
||||
if position > 0:
|
||||
if min_pos <= position <= max_pos and impressions >= min_impressions:
|
||||
opportunities.append(post)
|
||||
# Fallback: filter by traffic when position data unavailable
|
||||
# Include posts with any traffic for optimization analysis
|
||||
elif traffic > 0:
|
||||
opportunities.append(post)
|
||||
|
||||
self.log(f"✓ Found {len(opportunities)} posts for optimization analysis")
|
||||
if opportunities:
|
||||
traffic_posts = [p for p in opportunities if p.get('traffic', 0) > 0]
|
||||
self.log(f" ({len(traffic_posts)} have traffic data, {len(opportunities) - len(traffic_posts)} selected for analysis)")
|
||||
return opportunities
|
||||
|
||||
def calculate_opportunity_score(self, post):
|
||||
"""Calculate opportunity score (0-100) for a post."""
|
||||
position = post.get('avg_position', 50)
|
||||
impressions = post.get('impressions', 0)
|
||||
ctr = post.get('ctr', 0)
|
||||
traffic = post.get('traffic', 0)
|
||||
|
||||
# Position score (35%): Closer to page 1 = higher
|
||||
# Position 11-30 range
|
||||
position_score = max(0, (30 - position) / 19 * 35)
|
||||
|
||||
# Traffic potential (30%): Based on impressions
|
||||
# Normalize to 0-30
|
||||
traffic_potential = min(30, (impressions / 1000) * 30)
|
||||
|
||||
# CTR improvement potential (20%): Gap between current and expected CTR
|
||||
# Expected CTR at position X
|
||||
expected_ctr_map = {
|
||||
11: 0.02, 12: 0.02, 13: 0.015, 14: 0.015, 15: 0.013,
|
||||
16: 0.012, 17: 0.011, 18: 0.01, 19: 0.009, 20: 0.008,
|
||||
21: 0.008, 22: 0.007, 23: 0.007, 24: 0.006, 25: 0.006,
|
||||
26: 0.006, 27: 0.005, 28: 0.005, 29: 0.005, 30: 0.004
|
||||
}
|
||||
expected_ctr = expected_ctr_map.get(int(position), 0.005)
|
||||
ctr_gap = max(0, expected_ctr - ctr)
|
||||
ctr_score = min(20, (ctr_gap / expected_ctr * 100 / 5) * 20)
|
||||
|
||||
# Content quality (15%): Existing traffic and engagement
|
||||
quality_score = min(15, (traffic / 100) * 7.5 +
|
||||
(100 - post.get('bounce_rate', 50)) / 100 * 7.5)
|
||||
|
||||
return round(position_score + traffic_potential + ctr_score + quality_score, 1)
|
||||
|
||||
def estimate_traffic_gain(self, post):
|
||||
"""Estimate potential traffic gain from optimization."""
|
||||
position = post.get('avg_position', 50)
|
||||
impressions = post.get('impressions', 0)
|
||||
ctr = post.get('ctr', 0)
|
||||
|
||||
# Estimate CTR improvement from moving one position up
|
||||
# Moving from position X to X-1 typically improves CTR by 20-30%
|
||||
current_traffic = impressions * ctr
|
||||
if position > 11:
|
||||
# Target position: 1 ahead
|
||||
improvement_factor = 1.25 # 25% improvement per position
|
||||
estimated_new_traffic = current_traffic * improvement_factor
|
||||
gain = estimated_new_traffic - current_traffic
|
||||
else:
|
||||
gain = 0
|
||||
|
||||
return round(gain, 0)
|
||||
|
||||
def generate_ai_recommendations(self, post):
|
||||
"""Generate AI recommendations for top opportunities."""
|
||||
if not self.client:
|
||||
return None
|
||||
|
||||
try:
|
||||
keywords = post.get('top_keywords', '').split(',')[:5]
|
||||
keywords_str = ', '.join([k.strip() for k in keywords if k.strip()])
|
||||
|
||||
prompt = f"""Analyze keyword optimization opportunities for this blog post:
|
||||
|
||||
Post Title: {post['title']}
|
||||
Current Position: {post['avg_position']:.1f}
|
||||
Monthly Impressions: {post['impressions']}
|
||||
Current CTR: {post['ctr']:.2%}
|
||||
Top Keywords: {keywords_str}
|
||||
|
||||
Provide 2-3 specific, actionable recommendations to:
|
||||
1. Improve the SEO title to increase CTR
|
||||
2. Enhance the meta description
|
||||
3. Target structural improvements (headers, content gaps)
|
||||
|
||||
Focus on moving this post from positions 11-20 to page 1 (positions 1-10).
|
||||
Be specific and practical.
|
||||
|
||||
Return as JSON:
|
||||
{{
|
||||
"title_recommendations": ["recommendation 1", "recommendation 2"],
|
||||
"description_recommendations": ["recommendation 1", "recommendation 2"],
|
||||
"content_recommendations": ["recommendation 1", "recommendation 2"],
|
||||
"estimated_effort_hours": number,
|
||||
"expected_position_improvement": number
|
||||
}}"""
|
||||
|
||||
response = self.client.chat.completions.create(
|
||||
model=self.config.AI_MODEL,
|
||||
messages=[{"role": "user", "content": prompt}],
|
||||
temperature=0.7,
|
||||
max_tokens=500
|
||||
)
|
||||
|
||||
try:
|
||||
result_text = response.choices[0].message.content
|
||||
# Extract JSON
|
||||
start_idx = result_text.find('{')
|
||||
end_idx = result_text.rfind('}') + 1
|
||||
if start_idx >= 0 and end_idx > start_idx:
|
||||
return json.loads(result_text[start_idx:end_idx])
|
||||
except json.JSONDecodeError:
|
||||
self.log(f"⚠️ Could not parse AI response for {post['title']}")
|
||||
return None
|
||||
|
||||
except Exception as e:
|
||||
self.log(f"⚠️ AI generation failed for {post['title']}: {e}")
|
||||
return None
|
||||
|
||||
def export_opportunities_csv(self, opportunities, output_csv):
|
||||
"""Export opportunities to CSV."""
|
||||
if not opportunities:
|
||||
self.log("⚠️ No opportunities to export")
|
||||
return
|
||||
|
||||
try:
|
||||
fieldnames = [
|
||||
'ID', 'Title', 'URL', 'avg_position', 'impressions', 'clicks',
|
||||
'ctr', 'traffic', 'bounce_rate', 'keywords_count', 'top_keywords',
|
||||
'opportunity_score', 'estimated_traffic_gain',
|
||||
'title_recommendations', 'description_recommendations',
|
||||
'content_recommendations', 'estimated_effort_hours',
|
||||
'expected_position_improvement'
|
||||
]
|
||||
|
||||
with open(output_csv, 'w', newline='', encoding='utf-8') as f:
|
||||
writer = csv.DictWriter(f, fieldnames=fieldnames, extrasaction='ignore')
|
||||
writer.writeheader()
|
||||
|
||||
for opp in sorted(opportunities, key=lambda x: x['opportunity_score'], reverse=True):
|
||||
row = {
|
||||
'ID': opp['id'],
|
||||
'Title': opp['title'],
|
||||
'URL': opp['url'],
|
||||
'avg_position': opp['avg_position'],
|
||||
'impressions': opp['impressions'],
|
||||
'clicks': opp['clicks'],
|
||||
'ctr': f"{opp['ctr']:.2%}",
|
||||
'traffic': opp['traffic'],
|
||||
'bounce_rate': opp['bounce_rate'],
|
||||
'keywords_count': opp['keywords_count'],
|
||||
'top_keywords': opp['top_keywords'],
|
||||
'opportunity_score': opp['opportunity_score'],
|
||||
'estimated_traffic_gain': opp['estimated_traffic_gain'],
|
||||
'title_recommendations': opp.get('title_recommendations_str', ''),
|
||||
'description_recommendations': opp.get('description_recommendations_str', ''),
|
||||
'content_recommendations': opp.get('content_recommendations_str', ''),
|
||||
'estimated_effort_hours': opp.get('estimated_effort_hours', ''),
|
||||
'expected_position_improvement': opp.get('expected_position_improvement', '')
|
||||
}
|
||||
writer.writerow(row)
|
||||
|
||||
self.log(f"✓ Exported {len(opportunities)} opportunities to {output_csv}")
|
||||
except Exception as e:
|
||||
self.log(f"❌ Error exporting CSV: {e}")
|
||||
|
||||
def export_log(self, log_file):
|
||||
"""Export analysis log."""
|
||||
try:
|
||||
with open(log_file, 'w', encoding='utf-8') as f:
|
||||
f.write("SEO Opportunity Analysis Report\n")
|
||||
f.write("=" * 60 + "\n\n")
|
||||
|
||||
for msg in self.logs:
|
||||
f.write(msg + "\n")
|
||||
|
||||
self.log(f"✓ Exported log to {log_file}")
|
||||
except Exception as e:
|
||||
self.log(f"❌ Error exporting log: {e}")
|
||||
|
||||
def run(self, posts_csv, output_csv, min_position=11, max_position=30,
|
||||
min_impressions=50, top_n=20):
|
||||
"""Run complete analysis workflow."""
|
||||
self.log("🔍 Starting keyword opportunity analysis...")
|
||||
self.log(f"Input: {posts_csv}")
|
||||
self.log(f"Position range: {min_position}-{max_position}")
|
||||
self.log(f"Min impressions: {min_impressions}")
|
||||
self.log(f"Top N for AI analysis: {top_n}\n")
|
||||
|
||||
# Load posts
|
||||
posts = self.load_posts(posts_csv)
|
||||
if not posts:
|
||||
return
|
||||
|
||||
# Filter opportunities
|
||||
opportunities = self.filter_opportunities(posts, min_position, max_position, min_impressions)
|
||||
if not opportunities:
|
||||
self.log("⚠️ No opportunities found in specified range")
|
||||
return
|
||||
|
||||
# Calculate scores
|
||||
self.log("\n📊 Calculating opportunity scores...")
|
||||
for opp in opportunities:
|
||||
opp['opportunity_score'] = self.calculate_opportunity_score(opp)
|
||||
opp['estimated_traffic_gain'] = self.estimate_traffic_gain(opp)
|
||||
|
||||
# Sort by score
|
||||
opportunities = sorted(opportunities, key=lambda x: x['opportunity_score'], reverse=True)
|
||||
|
||||
# Get AI recommendations for top N
|
||||
self.log(f"\n🤖 Generating AI recommendations for top {min(top_n, len(opportunities))} opportunities...")
|
||||
for i, opp in enumerate(opportunities[:top_n]):
|
||||
self.log(f" [{i+1}/{min(top_n, len(opportunities))}] {opp['title'][:50]}...")
|
||||
recommendations = self.generate_ai_recommendations(opp)
|
||||
|
||||
if recommendations:
|
||||
opp['title_recommendations_str'] = '; '.join(recommendations.get('title_recommendations', []))
|
||||
opp['description_recommendations_str'] = '; '.join(recommendations.get('description_recommendations', []))
|
||||
opp['content_recommendations_str'] = '; '.join(recommendations.get('content_recommendations', []))
|
||||
opp['estimated_effort_hours'] = recommendations.get('estimated_effort_hours', '')
|
||||
opp['expected_position_improvement'] = recommendations.get('expected_position_improvement', '')
|
||||
|
||||
time.sleep(0.2) # Rate limiting
|
||||
|
||||
# Export
|
||||
self.log("\n📁 Exporting results...")
|
||||
self.export_opportunities_csv(opportunities, output_csv)
|
||||
|
||||
# Export log
|
||||
log_dir = self.output_dir / 'logs'
|
||||
log_dir.mkdir(exist_ok=True)
|
||||
log_file = log_dir / 'opportunity_analysis_log.txt'
|
||||
self.export_log(log_file)
|
||||
|
||||
self.log(f"\n✓ Analysis complete! {len(opportunities)} opportunities identified.")
|
||||
self.log(f" Top opportunity: {opportunities[0]['title'][:50]}... (score: {opportunities[0]['opportunity_score']})")
|
||||
|
||||
|
||||
def main():
|
||||
"""CLI entry point."""
|
||||
parser = argparse.ArgumentParser(description='Analyze keyword opportunities')
|
||||
parser.add_argument('--input', type=Path,
|
||||
default=Path('output/results/posts_with_analytics.csv'),
|
||||
help='Input posts CSV')
|
||||
parser.add_argument('--output', type=Path,
|
||||
default=Path('output/results/keyword_opportunities.csv'),
|
||||
help='Output opportunities CSV')
|
||||
parser.add_argument('--min-position', type=int, default=11,
|
||||
help='Minimum position (start of range)')
|
||||
parser.add_argument('--max-position', type=int, default=30,
|
||||
help='Maximum position (end of range)')
|
||||
parser.add_argument('--min-impressions', type=int, default=50,
|
||||
help='Minimum impressions to consider')
|
||||
parser.add_argument('--top-n', type=int, default=20,
|
||||
help='Top N for AI recommendations')
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
analyzer = OpportunityAnalyzer()
|
||||
analyzer.run(args.input, args.output, args.min_position, args.max_position,
|
||||
args.min_impressions, args.top_n)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
436
report_generator.py
Normal file
436
report_generator.py
Normal file
@@ -0,0 +1,436 @@
|
||||
"""
|
||||
SEO optimization report generator.
|
||||
Consolidates all analysis into comprehensive markdown report and action plan.
|
||||
"""
|
||||
|
||||
import csv
|
||||
import json
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
from datetime import datetime
|
||||
from config import Config
|
||||
|
||||
|
||||
class ReportGenerator:
|
||||
"""Generate comprehensive SEO optimization report."""
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize generator."""
|
||||
self.config = Config
|
||||
self.output_dir = self.config.OUTPUT_DIR
|
||||
self.logs = []
|
||||
|
||||
def log(self, message):
|
||||
"""Add message to log."""
|
||||
self.logs.append(message)
|
||||
print(message)
|
||||
|
||||
def load_posts_with_analytics(self, csv_path):
|
||||
"""Load posts with all analytics data."""
|
||||
posts = {}
|
||||
if not csv_path.exists():
|
||||
self.log(f"❌ File not found: {csv_path}")
|
||||
return posts
|
||||
|
||||
try:
|
||||
with open(csv_path, 'r', encoding='utf-8') as f:
|
||||
reader = csv.DictReader(f)
|
||||
for row in reader:
|
||||
post_id = row.get('ID')
|
||||
if not post_id:
|
||||
continue
|
||||
|
||||
# Handle different title column names
|
||||
title = (row.get('Title') or
|
||||
row.get('title') or
|
||||
row.get('post_title') or '')
|
||||
|
||||
posts[post_id] = {
|
||||
'title': title,
|
||||
'url': row.get('URL') or row.get('url') or row.get('post_url') or '',
|
||||
'seo_title': row.get('SEO Title') or row.get('seo_title') or '',
|
||||
'meta_description': row.get('Meta Description') or row.get('meta_description') or '',
|
||||
'traffic': int(row.get('traffic', 0) or 0),
|
||||
'users': int(row.get('users', 0) or 0),
|
||||
'bounce_rate': float(row.get('bounce_rate', 0) or 0),
|
||||
'impressions': int(row.get('impressions', 0) or 0),
|
||||
'clicks': int(row.get('clicks', 0) or 0),
|
||||
'avg_position': float(row.get('avg_position', 0) or 0),
|
||||
'ctr': float(row.get('ctr', 0) or 0),
|
||||
'keywords_count': int(row.get('keywords_count', 0) or 0),
|
||||
'top_keywords': row.get('top_keywords', '')
|
||||
}
|
||||
|
||||
self.log(f"✓ Loaded {len(posts)} posts")
|
||||
except Exception as e:
|
||||
self.log(f"❌ Error reading posts: {e}")
|
||||
|
||||
return posts
|
||||
|
||||
def load_opportunities(self, csv_path):
|
||||
"""Load keyword opportunities."""
|
||||
opportunities = {}
|
||||
if not csv_path.exists():
|
||||
self.log(f"⚠️ Opportunities file not found: {csv_path}")
|
||||
return opportunities
|
||||
|
||||
try:
|
||||
with open(csv_path, 'r', encoding='utf-8') as f:
|
||||
reader = csv.DictReader(f)
|
||||
for row in reader:
|
||||
post_id = row.get('ID')
|
||||
if post_id:
|
||||
try:
|
||||
opportunities[post_id] = {
|
||||
'opportunity_score': float(row.get('opportunity_score', 0) or 0),
|
||||
'estimated_traffic_gain': int(float(row.get('estimated_traffic_gain', 0) or 0)),
|
||||
'title_recommendations': row.get('title_recommendations', ''),
|
||||
'description_recommendations': row.get('description_recommendations', ''),
|
||||
'content_recommendations': row.get('content_recommendations', '')
|
||||
}
|
||||
except (ValueError, TypeError):
|
||||
# Skip rows with parsing errors
|
||||
continue
|
||||
|
||||
self.log(f"✓ Loaded {len(opportunities)} opportunities")
|
||||
except Exception as e:
|
||||
self.log(f"⚠️ Error reading opportunities: {e}")
|
||||
|
||||
return opportunities
|
||||
|
||||
def load_content_gaps(self, csv_path):
|
||||
"""Load content gap suggestions."""
|
||||
gaps = []
|
||||
if not csv_path.exists():
|
||||
self.log(f"⚠️ Content gaps file not found: {csv_path}")
|
||||
return gaps
|
||||
|
||||
try:
|
||||
with open(csv_path, 'r', encoding='utf-8') as f:
|
||||
reader = csv.DictReader(f)
|
||||
for row in reader:
|
||||
gaps.append({
|
||||
'title': row.get('title', ''),
|
||||
'why_valuable': row.get('why_valuable', ''),
|
||||
'search_volume': row.get('search_volume', ''),
|
||||
'format': row.get('format', ''),
|
||||
'traffic_potential': int(row.get('traffic_potential', 0) or 0),
|
||||
'priority': row.get('priority', 'medium')
|
||||
})
|
||||
|
||||
self.log(f"✓ Loaded {len(gaps)} content gap ideas")
|
||||
except Exception as e:
|
||||
self.log(f"⚠️ Error reading content gaps: {e}")
|
||||
|
||||
return gaps
|
||||
|
||||
def calculate_priority_score(self, post, opportunity=None):
|
||||
"""Calculate comprehensive priority score (0-100)."""
|
||||
position = post.get('avg_position', 50)
|
||||
impressions = post.get('impressions', 0)
|
||||
ctr = post.get('ctr', 0)
|
||||
traffic = post.get('traffic', 0)
|
||||
|
||||
# Position score (35%): Closer to page 1 = higher
|
||||
if position > 0 and position <= 30:
|
||||
position_score = max(0, (30 - position) / 29 * 35)
|
||||
else:
|
||||
position_score = 0
|
||||
|
||||
# Traffic potential (30%): Based on impressions
|
||||
traffic_potential = min(30, (impressions / 1000) * 30)
|
||||
|
||||
# CTR improvement (20%): Gap vs expected
|
||||
expected_ctr_map = {
|
||||
1: 0.30, 2: 0.16, 3: 0.11, 4: 0.08, 5: 0.07,
|
||||
6: 0.06, 7: 0.05, 8: 0.05, 9: 0.04, 10: 0.04,
|
||||
11: 0.02, 12: 0.02, 13: 0.015, 14: 0.015, 15: 0.013,
|
||||
16: 0.012, 17: 0.011, 18: 0.01, 19: 0.009, 20: 0.008
|
||||
}
|
||||
expected_ctr = expected_ctr_map.get(int(position), 0.005) if position > 0 else 0
|
||||
if expected_ctr > 0:
|
||||
ctr_gap = max(0, expected_ctr - ctr)
|
||||
ctr_score = min(20, (ctr_gap / expected_ctr * 100 / 5) * 20)
|
||||
else:
|
||||
ctr_score = 0
|
||||
|
||||
# Content quality (15%): Existing traffic and engagement
|
||||
quality_score = min(15, (traffic / 100) * 7.5 +
|
||||
(100 - post.get('bounce_rate', 50)) / 100 * 7.5)
|
||||
|
||||
total = round(position_score + traffic_potential + ctr_score + quality_score, 1)
|
||||
return max(0, min(100, total))
|
||||
|
||||
def generate_markdown_report(self, posts, opportunities, gaps, top_n=20):
|
||||
"""Generate comprehensive markdown report."""
|
||||
report = []
|
||||
report.append("# SEO Optimization Strategy Report\n")
|
||||
report.append(f"*Generated: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}*\n\n")
|
||||
|
||||
# Calculate metrics
|
||||
total_traffic = sum(p.get('traffic', 0) for p in posts.values())
|
||||
total_impressions = sum(p.get('impressions', 0) for p in posts.values())
|
||||
avg_position = sum(p.get('avg_position', 50) for p in posts.values() if p.get('avg_position', 0) > 0) / max(1, len([p for p in posts.values() if p.get('avg_position', 0) > 0]))
|
||||
|
||||
# Executive Summary
|
||||
report.append("## Executive Summary\n")
|
||||
report.append(f"- **Total Posts Analyzed:** {len(posts)}\n")
|
||||
report.append(f"- **Current Monthly Traffic:** {total_traffic:,} visits\n")
|
||||
report.append(f"- **Total Impressions (90d):** {total_impressions:,}\n")
|
||||
report.append(f"- **Average Search Position:** {avg_position:.1f}\n")
|
||||
report.append(f"- **Optimization Opportunities:** {len(opportunities)}\n")
|
||||
report.append(f"- **Content Gap Ideas:** {len(gaps)}\n")
|
||||
report.append(f"- **Potential Traffic Gain (Phase 1):** +{sum(o.get('estimated_traffic_gain', 0) for o in opportunities.values()):,} visits/month\n\n")
|
||||
|
||||
# Key Metrics
|
||||
report.append("### Quick Wins (Estimated Impact)\n\n")
|
||||
quick_wins = sorted(opportunities.values(),
|
||||
key=lambda x: x.get('estimated_traffic_gain', 0),
|
||||
reverse=True)[:5]
|
||||
total_quick_win_traffic = sum(w.get('estimated_traffic_gain', 0) for w in quick_wins)
|
||||
report.append(f"Top 5 opportunities could bring **+{total_quick_win_traffic:,} visits/month**\n\n")
|
||||
|
||||
# Top 20 Posts to Optimize
|
||||
report.append("## Top 20 Posts to Optimize\n\n")
|
||||
report.append("Ranked by optimization potential (combination of position, traffic potential, and CTR improvement).\n\n")
|
||||
|
||||
# Score all posts
|
||||
scored_posts = []
|
||||
for post_id, post in posts.items():
|
||||
opp = opportunities.get(post_id, {})
|
||||
score = self.calculate_priority_score(post, opp)
|
||||
scored_posts.append((post_id, post, opp, score))
|
||||
|
||||
scored_posts = sorted(scored_posts, key=lambda x: x[3], reverse=True)
|
||||
|
||||
for i, (post_id, post, opp, score) in enumerate(scored_posts[:top_n], 1):
|
||||
position = post.get('avg_position', 0)
|
||||
impressions = post.get('impressions', 0)
|
||||
traffic = post.get('traffic', 0)
|
||||
|
||||
report.append(f"### {i}. {post['title']}\n\n")
|
||||
report.append(f"**Current Position:** {position:.1f} | **Impressions:** {impressions:,} | **Traffic:** {traffic} visits\n")
|
||||
report.append(f"**Priority Score:** {score:.1f}/100 | **Estimated Gain:** +{opp.get('estimated_traffic_gain', 0)} visits\n\n")
|
||||
|
||||
if position > 0 and position <= 30:
|
||||
report.append(f"**Status:** Ranking on {'page 1' if position <= 10 else 'page 2-3'}\n\n")
|
||||
|
||||
if opp.get('title_recommendations'):
|
||||
report.append("**Title Optimization:**\n")
|
||||
for rec in opp['title_recommendations'].split(';'):
|
||||
rec = rec.strip()
|
||||
if rec:
|
||||
report.append(f"- {rec}\n")
|
||||
report.append("\n")
|
||||
|
||||
if opp.get('description_recommendations'):
|
||||
report.append("**Meta Description:**\n")
|
||||
for rec in opp['description_recommendations'].split(';'):
|
||||
rec = rec.strip()
|
||||
if rec:
|
||||
report.append(f"- {rec}\n")
|
||||
report.append("\n")
|
||||
|
||||
if opp.get('content_recommendations'):
|
||||
report.append("**Content Improvements:**\n")
|
||||
for rec in opp['content_recommendations'].split(';'):
|
||||
rec = rec.strip()
|
||||
if rec:
|
||||
report.append(f"- {rec}\n")
|
||||
report.append("\n")
|
||||
|
||||
report.append("---\n\n")
|
||||
|
||||
# Keyword Opportunities Summary
|
||||
report.append("## Keyword Opportunities Summary\n\n")
|
||||
opportunity_categories = {
|
||||
'page_2': [],
|
||||
'page_3': [],
|
||||
'ready_for_optimization': []
|
||||
}
|
||||
|
||||
for opp_id, opp in opportunities.items():
|
||||
if any(opp_id == p[0] for p in scored_posts[:top_n]):
|
||||
score = opp.get('opportunity_score', 0)
|
||||
post = posts.get(opp_id, {})
|
||||
position = post.get('avg_position', 0)
|
||||
|
||||
if 11 <= position <= 15:
|
||||
opportunity_categories['page_2'].append((score, opp))
|
||||
elif 16 <= position <= 30:
|
||||
opportunity_categories['page_3'].append((score, opp))
|
||||
|
||||
report.append(f"**Page 2 (Positions 11-15):** {len(opportunity_categories['page_2'])} keywords ready for quick wins\n")
|
||||
report.append(f"**Page 3+ (Positions 16-30):** {len(opportunity_categories['page_3'])} keywords with medium effort\n\n")
|
||||
|
||||
# Content Gap Analysis
|
||||
report.append("## Content Gap Analysis\n\n")
|
||||
report.append(f"Identified **{len(gaps)} high-value content opportunities** not currently covered:\n\n")
|
||||
|
||||
for i, gap in enumerate(sorted(gaps, key=lambda x: x.get('priority') == 'high', reverse=True)[:15], 1):
|
||||
report.append(f"### {i}. {gap['title']}\n\n")
|
||||
report.append(f"**Priority:** {gap.get('priority', 'medium').upper()}\n")
|
||||
report.append(f"**Search Volume:** {gap.get('search_volume', 'medium')}\n")
|
||||
report.append(f"**Format:** {gap.get('format', 'guide')}\n")
|
||||
report.append(f"**Estimated Traffic Potential:** +{gap.get('traffic_potential', 50)} visits/month\n\n")
|
||||
|
||||
if gap.get('why_valuable'):
|
||||
report.append(f"**Why valuable:** {gap['why_valuable']}\n\n")
|
||||
|
||||
# 90-Day Action Plan
|
||||
report.append("## 90-Day Action Plan\n\n")
|
||||
report.append("### Week 1-2: Quick Wins (Estimated +100 visits/month)\n\n")
|
||||
report.append("Focus on posts with highest opportunity scores that are already ranking on page 2:\n\n")
|
||||
quick_wins_phase = sorted(scored_posts[:top_n], key=lambda x: x[3], reverse=True)[:5]
|
||||
for i, (post_id, post, opp, score) in enumerate(quick_wins_phase, 1):
|
||||
report.append(f"{i}. **{post['title'][:60]}**\n")
|
||||
report.append(f" - Update SEO title and meta description\n")
|
||||
report.append(f" - Estimated effort: 30-60 minutes\n")
|
||||
report.append(f" - Expected gain: +{opp.get('estimated_traffic_gain', 50)} visits\n\n")
|
||||
|
||||
report.append("### Week 3-4: Core Content Optimization (Estimated +150 visits/month)\n\n")
|
||||
report.append("Improve content structure and internal linking:\n\n")
|
||||
mid_phase = sorted(scored_posts[5:15], key=lambda x: x[3], reverse=True)[:5]
|
||||
for i, (post_id, post, opp, score) in enumerate(mid_phase, 1):
|
||||
report.append(f"{i}. **{post['title'][:60]}**\n")
|
||||
report.append(f" - Add missing content sections\n")
|
||||
report.append(f" - Improve header structure\n")
|
||||
report.append(f" - Estimated effort: 2-3 hours\n\n")
|
||||
|
||||
report.append("### Week 5-8: New Content Creation (Estimated +300 visits/month)\n\n")
|
||||
report.append("Create 3-5 pieces of new content targeting high-value gaps:\n\n")
|
||||
for i, gap in enumerate(sorted(gaps, key=lambda x: x.get('traffic_potential', 0), reverse=True)[:4], 1):
|
||||
report.append(f"{i}. **{gap['title']}** ({gap.get('format', 'guide').title()})\n")
|
||||
report.append(f" - Estimated effort: 4-6 hours\n")
|
||||
report.append(f" - Expected traffic: +{gap.get('traffic_potential', 50)} visits/month\n\n")
|
||||
|
||||
report.append("### Week 9-12: Refinement & Analysis (Estimated +100 visits/month)\n\n")
|
||||
report.append("- Monitor ranking changes and CTR improvements\n")
|
||||
report.append("- Refine underperforming optimizations\n")
|
||||
report.append("- Re-run keyword analysis to identify new opportunities\n\n")
|
||||
|
||||
report.append("**Total Estimated 90-Day Impact: +650 visits/month (+~7.8% growth)**\n\n")
|
||||
|
||||
# Methodology
|
||||
report.append("## Methodology\n\n")
|
||||
report.append("### Priority Score Calculation\n\n")
|
||||
report.append("Each post is scored based on:\n")
|
||||
report.append("- **Position (35%):** Posts ranking 11-20 get highest scores (closest to page 1)\n")
|
||||
report.append("- **Traffic Potential (30%):** Based on search impressions\n")
|
||||
report.append("- **CTR Gap (20%):** Difference between current and expected CTR for position\n")
|
||||
report.append("- **Content Quality (15%):** Existing traffic and bounce rate\n\n")
|
||||
|
||||
report.append("### Data Sources\n\n")
|
||||
report.append("- **Google Analytics:** Traffic metrics (90-day window)\n")
|
||||
report.append("- **Google Search Console:** Keyword data, impressions, clicks, positions\n")
|
||||
report.append("- **WordPress REST API:** Current SEO metadata and content structure\n\n")
|
||||
|
||||
report.append("### Assumptions\n\n")
|
||||
report.append("- Traffic estimates are based on historical CTR and position data\n")
|
||||
report.append("- Moving one position up typically improves CTR by 20-30%\n")
|
||||
report.append("- Page 1 rankings (positions 1-10) receive ~20-30% of total impressions\n")
|
||||
report.append("- New content takes 4-8 weeks to gain significant traction\n\n")
|
||||
|
||||
return "\n".join(report)
|
||||
|
||||
def export_report(self, report_text, output_md):
|
||||
"""Export markdown report."""
|
||||
try:
|
||||
with open(output_md, 'w', encoding='utf-8') as f:
|
||||
f.write(report_text)
|
||||
|
||||
self.log(f"✓ Exported report to {output_md}")
|
||||
except Exception as e:
|
||||
self.log(f"❌ Error exporting report: {e}")
|
||||
|
||||
def export_prioritized_csv(self, posts, opportunities, output_csv):
|
||||
"""Export all posts with priority scores."""
|
||||
try:
|
||||
scored_posts = []
|
||||
for post_id, post in posts.items():
|
||||
opp = opportunities.get(post_id, {})
|
||||
score = self.calculate_priority_score(post, opp)
|
||||
|
||||
scored_posts.append({
|
||||
'ID': post_id,
|
||||
'Title': post.get('title', ''),
|
||||
'URL': post.get('url', ''),
|
||||
'Priority_Score': score,
|
||||
'Estimated_Traffic_Gain': opp.get('estimated_traffic_gain', 0),
|
||||
'Current_Position': post.get('avg_position', 0),
|
||||
'Impressions': post.get('impressions', 0),
|
||||
'Traffic': post.get('traffic', 0),
|
||||
'CTR': f"{post.get('ctr', 0):.2%}",
|
||||
'Keywords_Count': post.get('keywords_count', 0)
|
||||
})
|
||||
|
||||
scored_posts = sorted(scored_posts, key=lambda x: x['Priority_Score'], reverse=True)
|
||||
|
||||
fieldnames = ['ID', 'Title', 'URL', 'Priority_Score', 'Estimated_Traffic_Gain',
|
||||
'Current_Position', 'Impressions', 'Traffic', 'CTR', 'Keywords_Count']
|
||||
|
||||
with open(output_csv, 'w', newline='', encoding='utf-8') as f:
|
||||
writer = csv.DictWriter(f, fieldnames=fieldnames)
|
||||
writer.writeheader()
|
||||
writer.writerows(scored_posts)
|
||||
|
||||
self.log(f"✓ Exported {len(scored_posts)} prioritized posts to {output_csv}")
|
||||
except Exception as e:
|
||||
self.log(f"❌ Error exporting prioritized CSV: {e}")
|
||||
|
||||
def run(self, posts_csv, opportunities_csv, gaps_csv, output_md, output_prioritized_csv, top_n=20):
|
||||
"""Run complete report generation workflow."""
|
||||
self.log("📊 Generating SEO optimization report...")
|
||||
self.log(f"Input files: posts_with_analytics, opportunities, content_gaps\n")
|
||||
|
||||
# Load data
|
||||
posts = self.load_posts_with_analytics(posts_csv)
|
||||
opportunities = self.load_opportunities(opportunities_csv)
|
||||
gaps = self.load_content_gaps(gaps_csv)
|
||||
|
||||
if not posts:
|
||||
self.log("❌ No posts loaded. Cannot generate report.")
|
||||
return
|
||||
|
||||
# Generate report
|
||||
self.log("\n📝 Generating markdown report...")
|
||||
report_text = self.generate_markdown_report(posts, opportunities, gaps, top_n)
|
||||
|
||||
# Export report
|
||||
self.log("\n📁 Exporting files...")
|
||||
self.export_report(report_text, output_md)
|
||||
self.export_prioritized_csv(posts, opportunities, output_prioritized_csv)
|
||||
|
||||
self.log("\n✓ Report generation complete!")
|
||||
|
||||
|
||||
def main():
|
||||
"""CLI entry point."""
|
||||
parser = argparse.ArgumentParser(description='Generate SEO optimization report')
|
||||
parser.add_argument('--posts-with-analytics', type=Path,
|
||||
default=Path('output/results/posts_with_analytics.csv'),
|
||||
help='Posts with analytics CSV')
|
||||
parser.add_argument('--keyword-opportunities', type=Path,
|
||||
default=Path('output/results/keyword_opportunities.csv'),
|
||||
help='Keyword opportunities CSV')
|
||||
parser.add_argument('--content-gaps', type=Path,
|
||||
default=Path('output/results/content_gaps.csv'),
|
||||
help='Content gaps CSV')
|
||||
parser.add_argument('--output-report', type=Path,
|
||||
default=Path('output/results/seo_optimization_report.md'),
|
||||
help='Output markdown report')
|
||||
parser.add_argument('--output-csv', type=Path,
|
||||
default=Path('output/results/posts_prioritized.csv'),
|
||||
help='Output prioritized posts CSV')
|
||||
parser.add_argument('--top-n', type=int, default=20,
|
||||
help='Number of top posts to detail')
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
generator = ReportGenerator()
|
||||
generator.run(args.posts_with_analytics, args.keyword_opportunities,
|
||||
args.content_gaps, args.output_report, args.output_csv, args.top_n)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
5
requirements.txt
Normal file
5
requirements.txt
Normal file
@@ -0,0 +1,5 @@
|
||||
requests>=2.31.0
|
||||
pandas>=2.0.0
|
||||
python-dotenv>=1.0.0
|
||||
openai>=1.0.0
|
||||
numpy>=1.24.0
|
||||
73
run_analysis.sh
Executable file
73
run_analysis.sh
Executable file
@@ -0,0 +1,73 @@
|
||||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
echo "╔════════════════════════════════════════════════════════════╗"
|
||||
echo "║ SEO Analysis & Improvement System - Full Pipeline ║"
|
||||
echo "╚════════════════════════════════════════════════════════════╝"
|
||||
echo ""
|
||||
|
||||
# Check if venv exists
|
||||
if [ ! -d "venv" ]; then
|
||||
echo "❌ Virtual environment not found. Please run: python3 -m venv venv"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check if input files exist
|
||||
if [ ! -f "input/new-propositions.csv" ]; then
|
||||
echo "❌ Missing input/new-propositions.csv"
|
||||
echo "Please place your WordPress posts CSV in input/ directory"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ ! -f "input/analytics/ga4_export.csv" ]; then
|
||||
echo "❌ Missing input/analytics/ga4_export.csv"
|
||||
echo "Please export GA4 data and place it in input/analytics/"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Create output directories
|
||||
mkdir -p output/results
|
||||
mkdir -p output/logs
|
||||
|
||||
echo "📊 Step 1: Analytics Integration"
|
||||
echo " Merging GA4, Search Console, and WordPress data..."
|
||||
./venv/bin/python analytics_importer.py
|
||||
echo ""
|
||||
|
||||
echo "🔍 Step 2: Keyword Opportunity Analysis"
|
||||
echo " Identifying high-potential optimization opportunities..."
|
||||
./venv/bin/python opportunity_analyzer.py \
|
||||
--input output/results/posts_with_analytics.csv \
|
||||
--output output/results/keyword_opportunities.csv \
|
||||
--min-position 11 \
|
||||
--max-position 30 \
|
||||
--min-impressions 50 \
|
||||
--top-n 20
|
||||
echo ""
|
||||
|
||||
echo "📝 Step 3: Report Generation"
|
||||
echo " Creating comprehensive SEO optimization report..."
|
||||
./venv/bin/python report_generator.py
|
||||
echo ""
|
||||
|
||||
echo "╔════════════════════════════════════════════════════════════╗"
|
||||
echo "║ ✅ Analysis Complete! ║"
|
||||
echo "╚════════════════════════════════════════════════════════════╝"
|
||||
echo ""
|
||||
echo "📂 Results Location:"
|
||||
echo " └─ output/results/seo_optimization_report.md"
|
||||
echo ""
|
||||
echo "📊 Key Files:"
|
||||
echo " ├─ posts_prioritized.csv (all posts ranked 0-100)"
|
||||
echo " ├─ keyword_opportunities.csv (26 optimization opportunities)"
|
||||
echo " └─ posts_with_analytics.csv (enriched dataset)"
|
||||
echo ""
|
||||
echo "📋 Logs:"
|
||||
echo " └─ output/logs/"
|
||||
echo ""
|
||||
echo "🚀 Next Steps:"
|
||||
echo " 1. Open: output/results/seo_optimization_report.md"
|
||||
echo " 2. Review Top 20 Posts to Optimize"
|
||||
echo " 3. Start with Quick Wins (positions 11-15)"
|
||||
echo " 4. Follow 90-day action plan"
|
||||
echo ""
|
||||
Reference in New Issue
Block a user