Docker Architecture

PTIIKInsight uses Docker containers to provide a consistent deployment environment. The architecture consists of three main services that work together to provide topic modeling capabilities and monitoring.

Overview

The Docker setup includes:

  • FastAPI Application: Core API service for topic modeling

  • Prometheus: Metrics collection and monitoring

  • Grafana: Visualization and dashboards

Architecture Diagram

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   FastAPI App   │    │   Prometheus    │    │    Grafana      │
│   Port: 8000    │    │   Port: 9090    │    │   Port: 3000    │
│                 │◄───┤                 │◄───┤                 │
│ - Topic Predict │    │ - Metrics Store │    │ - Dashboards    │
│ - Data Scraping │    │ - Alerting      │    │ - Visualization │
│ - Model APIs    │    │ - Time Series   │    │ - Monitoring    │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         │                       │                       │
         └───────────────────────┼───────────────────────┘
                                 │
                    ┌─────────────────┐
                    │   Host System   │
                    │                 │
                    │ - Volume Mounts │
                    │ - Data Storage  │
                    │ - Model Files   │
                    └─────────────────┘

Service Details

FastAPI Application (fastapi-app)

Container: fastapi-scraper Port: 8000 Image: Built from local Dockerfile

Key Features:

  • Topic prediction using BERTopic model

  • Web scraping for research papers

  • Data preprocessing and cleaning

  • Prometheus metrics integration

  • RESTful API endpoints

Volume Mounts:

  • ./data:/app/data - Data storage

  • ./model:/app/model - ML model files

  • ./preprocessing:/app/preprocessing - Preprocessing scripts

  • ./api:/app/api - API source code

Health Check: Available at /health endpoint

Prometheus (prometheus)

Container: prometheus Port: 9090 Image: prom/prometheus:latest

Key Features:

  • Scrapes metrics from FastAPI application

  • Stores time-series monitoring data

  • Provides alerting capabilities

  • Web-based query interface

Volume Mounts:

  • ./monitoring/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml - Configuration

  • ./monitoring/prometheus/rules:/etc/prometheus/rules - Alert rules

Configuration:

  • Scrapes FastAPI metrics from /metrics endpoint

  • Stores data in internal time-series database

  • Supports custom alert rules

Grafana (grafana)

Container: grafana Port: 3000 Image: grafana/grafana:latest

Key Features:

  • Visual dashboards for monitoring

  • Connects to Prometheus as data source

  • Pre-configured dashboards

  • Dark theme by default

Volume Mounts:

  • ./monitoring/grafana/provisioning:/etc/grafana/provisioning - Data sources config

  • ./monitoring/grafana/dashboards:/var/lib/grafana/dashboards - Dashboard definitions

Default Credentials:

  • Username: admin

  • Password: admin

Configuration:

  • Automatic dashboard provisioning

  • Prometheus data source pre-configured

  • Pie chart plugin installed

Network Architecture

All services run on the default Docker network and communicate internally:

  • FastAPI → Prometheus: FastAPI exposes metrics at /metrics endpoint

  • Prometheus → FastAPI: Prometheus scrapes metrics every 15 seconds

  • Grafana → Prometheus: Grafana queries Prometheus for dashboard data

  • Host → Services: All services accessible from host via mapped ports

Data Flow

  1. Data Input:

    • Web scraping via /scrape endpoint

    • Direct API calls for predictions

  2. Processing:

    • Text preprocessing and cleaning

    • Topic modeling with BERTopic

    • Result storage in CSV format

  3. Monitoring:

    • API metrics collected by Prometheus

    • Performance data visualized in Grafana

    • Real-time monitoring of predictions and errors

  4. Storage:

    • Raw data: ./data/raw/

    • Cleaned data: ./data/cleaned/

    • Models: ./model/

    • Monitoring config: ./monitoring/

Deployment Configuration

Environment Variables

The containers use these environment settings:

Grafana:

  • GF_USERS_ALLOW_SIGN_UP=false - Disable user registration

  • GF_USERS_DEFAULT_THEME=dark - Set dark theme

  • GF_INSTALL_PLUGINS=grafana-piechart-panel - Install pie chart plugin

Restart Policy

All services use restart: always to ensure automatic recovery from failures.

Resource Requirements

Minimum:

  • CPU: 2 cores

  • RAM: 4GB

  • Storage: 5GB

Recommended:

  • CPU: 4+ cores

  • RAM: 8GB+

  • Storage: 20GB+ (SSD)

Scaling Considerations

Horizontal Scaling

Currently, the architecture supports single-instance deployment. For scaling:

  1. API Service: Can be scaled horizontally with load balancer

  2. Prometheus: Single instance (clustering requires additional setup)

  3. Grafana: Single instance (clustering available in enterprise)

Performance Optimization

  • Model Loading: BERTopic model loaded once at startup

  • Async Processing: Scraping runs in background tasks

  • Metrics Collection: Minimal overhead with Prometheus integration

  • Volume Storage: Use fast storage (SSD) for model files

Security

Network Security

  • Services communicate on isolated Docker network

  • Only necessary ports exposed to host

  • No external network access required

Data Security

  • Local volume mounts for data persistence

  • No sensitive data in environment variables

  • Grafana admin credentials should be changed in production

Monitoring and Alerts

Available Metrics

  • model_predictions_total: Total predictions made

  • model_prediction_errors_total: Prediction failures

  • model_prediction_duration_seconds: Prediction timing

  • scraping_requests_total: Scraping requests

  • scraping_errors_total: Scraping failures

  • model_accuracy: Current model accuracy

Dashboard Features

  • Real-time API performance metrics

  • Model prediction success/failure rates

  • System resource utilization

  • Request volume and timing

  • Error rate monitoring

Troubleshooting

Common Issues

Services Not Starting:

docker-compose logs -f

Port Conflicts:

docker-compose down
netstat -ano | findstr :8000

Volume Mount Issues:

docker-compose down -v
docker-compose up -d

Memory Issues:

docker system prune -a

Health Checks

FastAPI: curl http://localhost:8000/health Prometheus: curl http://localhost:9090/-/healthy Grafana: Visit http://localhost:3000

Backup and Recovery

Data Backup

# Backup data volumes
docker run --rm -v $(pwd)/data:/backup alpine tar czf /backup/data-backup.tar.gz /backup

# Backup monitoring config
tar czf monitoring-backup.tar.gz monitoring/

Recovery

# Restore from backup
docker-compose down
tar xzf data-backup.tar.gz
docker-compose up -d
PreviousInstallationNextUser Interface

Last updated