Sub-system: api_gateway.py aka BFF

 Central DB init can be done separately (or by API Gateway during first run).


That’s a solid architectural direction — and your instinct to have API Gateway own DB initialization is correct for a few reasons:

Why API Gateway Should Initialize the DB

  • Single source of truth: Gateway knows the canonical schema (jobs, etc.).

  • Microservices stay focused: Worker, parser, logging just consume APIs/DB tables, no schema coupling.

  • Easier migrations: Gateway (or a future migration service) handles upgrades instead of each service duplicating logic.

  • Deployment order: DB init happens first, then worker/parser/logging spin up safely.

About Splitting API Gateway and BFF

Right now, your API Gateway is doing two jobs:

  1. Pure routing & orchestration: Upload, query, logging endpoints, etc.

  2. User-facing experience: Login flow (OIDC), session management, templates.

Splitting those would give you:

  • API Gateway

    • Stateless, JSON-only

    • Routes requests to microservices

    • Auth via JWT/OIDC tokens (no session state)

  • BFF (Backend for Frontend)

    • Owns login session UX (OIDC front-channel flow)

    • Renders templates (upload.html, query.html, logs.html)

    • Calls API Gateway for data

This aligns with future scaling: mobile apps → hit API Gateway directly; web → goes through BFF.

Migration Path

1. Short-term (Now)

  • Keep one project (API Gateway + BFF + DB init).

  • Clean up DB layer (MySQL, SQLAlchemy).

  • Worker & Logging just consume DB/API.

2. Medium-term

  • Split BFF (session + templates) from API Gateway (pure API).

  • Use shared JWT/OIDC layer for auth.

3. Long-term

  • Introduce service discovery or API Gateway like Kong/Traefik if needed.

  • API Gateway becomes more like a routing layer; BFF becomes optional client-facing service.

Implementation Plan

  1. Move DB init to API Gateway only

    • Base.metadata.create_all() runs in API Gateway startup.

    • Worker/logging services only query (no schema creation).

  2. Refactor models per table

    • models/job.py, models/log.py, models/authcode.py.

  3. Use MySQL + SQLAlchemy uniformly

    • Remove sqlite3 everywhere.

  4. Keep current monolith (API + BFF)

    • But isolate BFF logic (templates/session) into its own package folder

    • So later you can split it out easily.

Question for you

Do you want me to rewrite your current API Gateway to:

  • Use SQLAlchemy with MySQL

  • Initialize DB (jobs, etc.)

  • Expose APIs unchanged (so worker/logging still work)

  • Keep templates/session logic intact (but modularized so you can split later)?

Or do you want me to show you the refactor structure (folder layout + code snippets) first, so you can review before I rewrite everything?

(Also: want me to include docker-compose at this stage for MySQL + services?)

Comments

Post a Comment

Popular posts from this blog

Feature: Audit log for one login, and identity service

Image
Customer (Product) Page High-Level System Design Diagram (PlantUML) Sequence Diagram: Audit Event Ingestion Sequence Diagram: Orchestration Event For a High-Level System (Component) Diagram The queue is implemented as message_queue (generic), and the audit log service consumes from it, writing to a dedicated audit_log table. The logging service also consumes from the same message_queue and writes to log_store. Orchestration/other services use the same queue. Summary Table: What Writes Where Event Type Goes Into Consumed By Final Storage audit message_queue Audit Log Service audit_log table log message_queue Logging Service log_store table doc_uploaded message_queue Orchestration Worker (business DB, or triggers next event) ocr_ready message_queue Notification Worker, etc. (notification log, etc.)  Core Tables message_queue – universal event bus (temp buffer) audit_log – immutable audit/compliance events log_store – application logs (op...
Read more

Getting started - Build your data science lab environment

This guide explains how to set up a complete data science environment for the udemy program. A proper setup ensures compatibility and avoids recurring environment issues.  1. Overview Goal : Create a full-featured environment with sufficient resources for all course exercises. Recommendation : Use Anaconda for the most reliable setup. Alternative : Use a Python virtual environment with pip if Anaconda isn’t working. 2. Setup Options Primary (Recommended) Create a dedicated Anaconda environment (isolated, compatible). Alternative Use a standard Python virtualenv + pip (faster but less reliable). 3. Windows Setup Steps Clone Course Repository git clone https://github.com/ed-donner/llm_engineering/ Follow the repository’s README.md instructions for additional setup details. Create Anaconda Environment Inside the llm_engineering folder: conda env create -f environment.yml ⏳ This may take several minutes. Activate the environment: conda activate llms Deactivat...
Read more

QA - Run #1 - Results

Image
# QA Report for Tag QA_July2025_A - **Tag/Commit:** QA_July2025_A (`f32abc1`) - **Date:** 2025-08-01 - **QA Outcomes:** All tests passed except for file upload (see below)... - **Environment:** Windows 10, Python 3.12, requirements.txt as of tag - **Notes:** See log for detailed results. ## Issue Found - **Step:** - **Bug:** - **Log excerpt:** - **Screenshot:** For this QA run, we begin with pristine state by: Stop all services Delete logs.db, and jobs.db Delete all files in doc_store folder Start all services Test, you should see no data on index page, and query should result in no data found error Next, go ahead and upload new1.txt, and after that new2.txt Issue the query: "can you summarize all of the todo items you can find, seperating each one on a single line, with due date first, then title, and description with any details you can augment" Expected Answer: - August 1, 2025: Respond to Amira and discuss weekend plans - August 3, 2025: Go to Somesh class and finish ta...
Read more