Jonathan
486749a890
Full-stack Dutch supermarket price tracker with FastAPI backend, PostgreSQL/SQLAlchemy, Albert Heijn scraper, and Next.js frontend. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
138 lines
4.0 KiB
Markdown
138 lines
4.0 KiB
Markdown
# Dutch Food Price Tracker
|
|
|
|
Track Dutch supermarket food prices over time. Currently supports Albert Heijn.
|
|
|
|
## Stack
|
|
|
|
| Layer | Tech |
|
|
|---|---|
|
|
| Backend API | Python FastAPI + SQLAlchemy + Alembic |
|
|
| Database | PostgreSQL 16 |
|
|
| Scraper | httpx + AH anonymous token auth |
|
|
| Frontend | Next.js 14 + TypeScript + Tailwind CSS + Recharts |
|
|
| Dev infra | Docker Compose |
|
|
|
|
## Quick start (Docker)
|
|
|
|
```bash
|
|
git clone <repo> && cd dutch-food-price-tracker
|
|
|
|
# Build and start all services
|
|
docker compose up --build -d
|
|
|
|
# Run DB migrations (tables are also auto-created on backend start)
|
|
docker compose exec backend alembic upgrade head
|
|
|
|
# Scrape Albert Heijn
|
|
docker compose exec backend python cli.py scrape-ah \
|
|
--query melk \
|
|
--query brood \
|
|
--query kaas \
|
|
--query yoghurt
|
|
```
|
|
|
|
- Frontend: http://localhost:3000
|
|
- API docs: http://localhost:8000/docs
|
|
- Database: `localhost:5432` (user: postgres / password: postgres / db: food_prices)
|
|
|
|
## Local development (without Docker)
|
|
|
|
### Backend
|
|
|
|
```bash
|
|
cd backend
|
|
python -m venv .venv
|
|
source .venv/bin/activate # Windows: .venv\Scripts\activate
|
|
pip install -r requirements.txt
|
|
|
|
# Needs a running PostgreSQL instance
|
|
export DATABASE_URL=postgresql://postgres:postgres@localhost:5432/food_prices
|
|
|
|
# Create tables
|
|
alembic upgrade head
|
|
# or: python -c "from app.database import engine; from app.models import Base; Base.metadata.create_all(engine)"
|
|
|
|
# Start API server
|
|
uvicorn app.main:app --reload
|
|
```
|
|
|
|
### Frontend
|
|
|
|
```bash
|
|
cd frontend
|
|
npm install
|
|
|
|
# Point at the local backend
|
|
echo "NEXT_PUBLIC_API_URL=http://localhost:8000" > .env.local
|
|
|
|
npm run dev
|
|
```
|
|
|
|
## CLI reference
|
|
|
|
```bash
|
|
# Scrape one or more queries
|
|
python cli.py scrape-ah --query "melk" --query "brood"
|
|
|
|
# Run via Docker
|
|
docker compose exec backend python cli.py scrape-ah --query melk
|
|
```
|
|
|
|
## API endpoints
|
|
|
|
| Method | Path | Description |
|
|
|---|---|---|
|
|
| GET | `/api/products?search=melk` | Search products |
|
|
| GET | `/api/products/{id}` | Product detail |
|
|
| GET | `/api/products/{id}/prices` | Full price history |
|
|
| GET | `/api/prices/cheapest?date=2024-01-15` | Cheapest per product for a day |
|
|
| GET | `/api/stores` | List stores |
|
|
| GET | `/api/scrape-runs` | List recent scrape runs |
|
|
|
|
## Project structure
|
|
|
|
```
|
|
dutch-food-price-tracker/
|
|
├── backend/
|
|
│ ├── app/
|
|
│ │ ├── main.py # FastAPI app + CORS
|
|
│ │ ├── models.py # SQLAlchemy ORM models
|
|
│ │ ├── schemas.py # Pydantic request/response schemas
|
|
│ │ ├── database.py # Engine, session, Base
|
|
│ │ ├── config.py # Pydantic settings
|
|
│ │ ├── routers/ # products, stores, prices, scrape_runs
|
|
│ │ └── scrapers/
|
|
│ │ └── albert_heijn.py # Token auth + product search
|
|
│ ├── alembic/ # DB migration history
|
|
│ ├── cli.py # Click CLI (scrape-ah)
|
|
│ └── requirements.txt
|
|
├── frontend/
|
|
│ └── src/
|
|
│ ├── app/
|
|
│ │ ├── page.tsx # Product search
|
|
│ │ ├── products/[id]/page.tsx # Detail + price chart
|
|
│ │ └── cheapest/page.tsx # Daily cheapest overview
|
|
│ ├── components/
|
|
│ │ ├── Nav.tsx
|
|
│ │ ├── ProductCard.tsx
|
|
│ │ └── PriceChart.tsx # Recharts line chart
|
|
│ └── lib/api.ts # Typed API client
|
|
├── seed/ # Historical CSV/JSON import datasets
|
|
├── docker-compose.yml
|
|
└── .env.example
|
|
```
|
|
|
|
## Adding a new store
|
|
|
|
1. Create `backend/app/scrapers/<store_slug>.py`
|
|
2. Implement `scrape_query(db: Session, query: str) -> ScrapeRun`
|
|
3. Add a `@cli.command` in `backend/cli.py`
|
|
4. Insert a `Store` row with the new slug
|
|
|
|
## Data model
|
|
|
|
- **stores** — one row per chain (Albert Heijn, Jumbo, …)
|
|
- **products** — one row per store SKU; keyed by `(store_id, external_id)`
|
|
- **scrape_runs** — one row per CLI invocation / query
|
|
- **price_snapshots** — append-only price observations (cents, UTC timestamp)
|