Identity-Backend - Service secrets, scale to KID

Scaling JWT Secrets: From Single Key to Key Rotation with kid

When you first start building microservices, a single shared secret for JWT signing and verification feels simple and sufficient. But as soon as you introduce multiple clients, microservices, and the need for key rotation, this model starts breaking down.

We just ran into this exact scenario with our identity-backend and logging-backend services.

Current Setup

  • Identity-Backend: Issues JWT tokens signed with JWT_SECRET_KEY.

  • Logging-Backend: Validates JWT tokens using the same secret.

  • Audience (aud) and Issuer (iss): Hardcoded to enforce proper service-to-service calls.

  • Single Secret in .env: Shared between both services.

Pros:

  • Very simple to implement.

  • Easy to debug in early MVP stages.

Cons:

  • Any secret rotation requires redeploying all services simultaneously.

  • Impossible to issue different secrets for different clients.

  • No audit trail for which key signed what.

Next Evolution: Key IDs (kid) and Client Secrets

We’re introducing key IDs (kid) in JWT headers and moving to a model where identity-backend manages multiple keys in a SQLite database:

  • Each client has:

    • client_id (e.g., careergpt-backend)

    • key_id (e.g., key-2025-01)

    • secret (random string)

  • JWTs are issued with:

    • kid in the header (so consumers know which key to verify against).

  • logging-backend extracts kid and retrieves the right secret to validate the token.

This enables:

  1. Per-Client Secrets: Different clients get unique keys.

  2. Key Rotation: Add a new key, start signing with it, keep verifying old tokens until expiry.

  3. Audit & Traceability: Easily identify which key signed which token.

How Key Rotation Works

  1. Add new key to identity-backend DB (e.g., key-2025-02).

  2. Start issuing tokens with new key.

  3. Keep old key in DB for verification until tokens expire.

  4. Remove old key when no longer needed.

Implementation Plan

Step 1: Identity-Backend

  • Add client_secrets table.

  • Endpoints:

    • POST /add-client-secret → Add new key for a client.

    • GET /list-client-secrets → Admin view of active keys.

Step 2: Logging-Backend

  • Extract kid from JWT header.

  • Request correct secret from Identity-Backend for verification.

Step 3: Transition

  • Keep current single-secret fallback (JWT_SECRET_KEY) until rollout is complete.

  • Gradually migrate clients to new key-ID model.

Why This Matters

This pattern future-proofs your architecture for:

  • Microservice sprawl (multiple services, each with unique secrets).

  • Zero-downtime rotations (no redeploys to rotate secrets).

  • Enhanced security posture (different secrets = limited blast radius).

Next Post

In the next post, we’ll share the code changes:

  • How to embed kid in JWT headers using PyJWT.

  • Database schema and migrations.

  • Verification logic changes in logging-backend.


Comments

Post a Comment

Popular posts from this blog

Feature: Audit log for one login, and identity service

Image
Customer (Product) Page High-Level System Design Diagram (PlantUML) Sequence Diagram: Audit Event Ingestion Sequence Diagram: Orchestration Event For a High-Level System (Component) Diagram The queue is implemented as message_queue (generic), and the audit log service consumes from it, writing to a dedicated audit_log table. The logging service also consumes from the same message_queue and writes to log_store. Orchestration/other services use the same queue. Summary Table: What Writes Where Event Type Goes Into Consumed By Final Storage audit message_queue Audit Log Service audit_log table log message_queue Logging Service log_store table doc_uploaded message_queue Orchestration Worker (business DB, or triggers next event) ocr_ready message_queue Notification Worker, etc. (notification log, etc.)  Core Tables message_queue – universal event bus (temp buffer) audit_log – immutable audit/compliance events log_store – application logs (op...
Read more

Getting started - Build your data science lab environment

This guide explains how to set up a complete data science environment for the udemy program. A proper setup ensures compatibility and avoids recurring environment issues.  1. Overview Goal : Create a full-featured environment with sufficient resources for all course exercises. Recommendation : Use Anaconda for the most reliable setup. Alternative : Use a Python virtual environment with pip if Anaconda isn’t working. 2. Setup Options Primary (Recommended) Create a dedicated Anaconda environment (isolated, compatible). Alternative Use a standard Python virtualenv + pip (faster but less reliable). 3. Windows Setup Steps Clone Course Repository git clone https://github.com/ed-donner/llm_engineering/ Follow the repository’s README.md instructions for additional setup details. Create Anaconda Environment Inside the llm_engineering folder: conda env create -f environment.yml ⏳ This may take several minutes. Activate the environment: conda activate llms Deactivat...
Read more

QA - Run #1 - Results

Image
# QA Report for Tag QA_July2025_A - **Tag/Commit:** QA_July2025_A (`f32abc1`) - **Date:** 2025-08-01 - **QA Outcomes:** All tests passed except for file upload (see below)... - **Environment:** Windows 10, Python 3.12, requirements.txt as of tag - **Notes:** See log for detailed results. ## Issue Found - **Step:** - **Bug:** - **Log excerpt:** - **Screenshot:** For this QA run, we begin with pristine state by: Stop all services Delete logs.db, and jobs.db Delete all files in doc_store folder Start all services Test, you should see no data on index page, and query should result in no data found error Next, go ahead and upload new1.txt, and after that new2.txt Issue the query: "can you summarize all of the todo items you can find, seperating each one on a single line, with due date first, then title, and description with any details you can augment" Expected Answer: - August 1, 2025: Respond to Amira and discuss weekend plans - August 3, 2025: Go to Somesh class and finish ta...
Read more