Progetto MAPS

Appearance

Specifiche Infrastruttura Hosting ​

Deliverable D2.1.3: Specifiche Infrastruttura Hosting

4.1 Server Target: op-linkurious ​

Specifiche Hardware ​

Hostname: op-linkurious.openpolis.it
CPU: 8 cores
RAM: 31 GB
Disk: 621 GB
OS: Linux (Ubuntu/Debian)
Network: 1 Gbps

Servizi Esistenti ​

  • Traefik reverse proxy (porta 80/443)
  • Rete Docker: gw (gateway network)
  • Dominio base: *.deppsviluppo.org

4.2 Requisiti Sistema ​

Software Prerequisites ​

bash
# Docker Engine
Docker version >= 24.0
Docker Compose version >= 2.20

# Database Client
psql (PostgreSQL) >= 15

# Utilities
git, make, curl, wget, jq

Disk Layout ​

/root/maps-docker/              # Deployment root
├── postgres/                   # PostgreSQL init scripts
├── prefect/                    # Flows e configurazioni
├── openmetadata/               # Metadata config
├── shared-data/                # Bronze layer storage
│   ├── bronze/                 # Raw data
│   ├── cache/                  # Temporary cache
│   └── exports/                # Gold exports
├── volumes/                    # Docker volumes mount points
│   ├── postgres-data/          # Database files
│   └── openmetadata-data/      # Metadata DB
├── backup/                     # Backup scripts e dump
└── logs/                       # Centralized logging

Disk Space Allocation ​

PathSizePurpose
/root/maps-docker/shared-data/bronze/100 GBRaw data files
/root/maps-docker/volumes/postgres-data/150 GBPostgreSQL database
/root/maps-docker/volumes/openmetadata-data/20 GBMetadata catalog
/root/maps-docker/backup/50 GBDatabase backups (7 days retention)
TOTALE320 GBSu 621 GB disponibili (52%)

4.3 Network Configuration ​

Porte e Routing ​

Servizio         | Porta Interna | Dominio Esterno
-----------------+---------------+----------------------------------
Traefik          | 80, 443       | maps.deppsviluppo.org
PostgreSQL       | 5432          | do.linkurious.openpolis.it:5432
Prefect UI       | 4200          | prefect.maps.deppsviluppo.org
OpenMetadata UI  | 8585          | metadata.maps.deppsviluppo.org

DNS Records (Route53) ​

bash
# A records su deppsviluppo.org
prefect.maps.deppsviluppo.org    → ${SERVER_IP}
metadata.maps.deppsviluppo.org   → ${SERVER_IP}

# PostgreSQL già esposto su
do.linkurious.openpolis.it       → ${SERVER_IP}:5432

SSL/TLS ​

  • Provider: Let's Encrypt (via Traefik)
  • Auto-renewal: Traefik gestisce automaticamente
  • Certbot: NO necessario (Traefik integrato)

4.4 Deployment Procedure ​

Initial Setup ​

bash
# 1. Preparazione ambiente
ssh root@op-linkurious
mkdir -p /root/maps-docker
cd /root/maps-docker

# 2. Clone repository (o copia deployment package)
git clone https://gitlab.com/depp/gst-maps.git /tmp/gst-maps
cp -r /tmp/gst-maps/deployment/* .

# 3. Configurazione secrets
cp .env.example .env
nano .env  # Modifica password

# 4. Deploy stack
bash deploy.sh

# 5. Verifica
bash status.sh

Deploy Script ​

bash
#!/bin/bash
# deploy.sh - Main deployment script

set -e

echo "=== MAPS Data Lake Deployment ==="

# Pre-flight checks
echo "Checking prerequisites..."
command -v docker >/dev/null || { echo "Docker not found"; exit 1; }
command -v docker-compose >/dev/null || { echo "Docker Compose not found"; exit 1; }

# Create directories
echo "Creating directory structure..."
mkdir -p shared-data/{bronze,cache,exports}
mkdir -p volumes/{postgres-data,openmetadata-data}
mkdir -p backup logs

# Initialize database
echo "Initializing PostgreSQL..."
docker-compose up -d postgres
sleep 10
docker exec maps-postgres psql -U maps -d maps_db -f /docker-entrypoint-initdb.d/01-init-schemas.sql

# Start remaining services
echo "Starting services..."
docker-compose up -d

# Configure DNS
echo "Configuring DNS..."
bash dns-setup.sh

# Health checks
echo "Running health checks..."
bash status.sh

echo "=== Deployment complete ==="
echo "Access services at:"
echo "  - Prefect:     https://prefect.maps.deppsviluppo.org"
echo "  - OpenMetadata: https://metadata.maps.deppsviluppo.org"
echo "  - PostgreSQL:   do.linkurious.openpolis.it:5432"

4.5 Monitoring e Maintenance ​

Health Checks ​

bash
#!/bin/bash
# status.sh - Check service health

echo "=== MAPS Services Status ==="

# Docker containers
docker ps --filter "name=maps-" --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

# PostgreSQL
docker exec maps-postgres pg_isready -U maps -d maps_db

# Disk usage
df -h /root/maps-docker/

# Memory usage
free -h

# Prefect workers
docker exec maps-prefect-server prefect work-pool ls

Logging ​

bash
# Centralized logging directory
/root/maps-docker/logs/
├── postgres/         # PostgreSQL logs
├── prefect/          # Prefect server logs
├── workers/          # Worker pool logs
└── openmetadata/     # OpenMetadata logs

# View logs
docker logs -f maps-postgres       # PostgreSQL
docker logs -f maps-prefect-server # Prefect
docker logs -f maps-worker-istat   # Worker ISTAT

Backup Automation ​

bash
# Cron job per backup giornaliero
0 2 * * * /root/maps-docker/backup.sh >> /root/maps-docker/logs/backup.log 2>&1

4.6 Security Hardening ​

Firewall Rules ​

bash
# ufw rules
ufw allow 80/tcp      # HTTP (redirect HTTPS)
ufw allow 443/tcp     # HTTPS (Traefik)
ufw allow 5432/tcp    # PostgreSQL (già esposto)
ufw enable

PostgreSQL Security ​

sql
-- Revoke public permissions
REVOKE ALL ON SCHEMA public FROM PUBLIC;

-- Create read-only user for analytics
CREATE USER maps_readonly PASSWORD '${RO_PASSWORD}';
GRANT CONNECT ON DATABASE maps_db TO maps_readonly;
GRANT USAGE ON SCHEMA silver, gold TO maps_readonly;
GRANT SELECT ON ALL TABLES IN SCHEMA silver, gold TO maps_readonly;

Secrets Rotation ​

bash
# Rotate PostgreSQL password
docker exec maps-postgres psql -U postgres -c "ALTER USER maps PASSWORD '${NEW_PASSWORD}';"
# Update .env and restart services
docker-compose restart

4.7 Disaster Recovery Plan ​

Backup Strategy ​

  1. Daily: Full PostgreSQL dump (retention: 7 days)
  2. Weekly: Bronze layer snapshot (retention: 4 weeks)
  3. Monthly: Complete system backup (retention: 3 months)

Recovery Procedure ​

bash
# 1. Stop services
docker-compose down

# 2. Restore PostgreSQL
gunzip < /backup/maps_db_20260315.sql.gz | \
  docker exec -i maps-postgres psql -U maps -d maps_db

# 3. Restore Bronze data
rsync -avz backup-server:/backup/bronze/ /root/maps-docker/shared-data/bronze/

# 4. Restart services
docker-compose up -d

# 5. Verify
bash status.sh

[WIP] Questo capitolo sarà completato con:

  • Monitoring dashboards (Grafana)
  • Alerting rules (Prometheus)
  • Runbook operativi
  • Incident response procedures

Prossimo capitolo: Pipeline ETL