TOC

——Shared inspiration 

The only way to do great work, is to do what you love.

The mantra for this course - Learn by doing.

Timeline

1. 8 week course, starting July 28th, 2025
2. Week 1 - Extended (Take your time with practical exercise. Use the extra guides in Week 1 for a firmer footing and seek help via the platform, email, or LinkedIn (details in the GitHub repo).)

Learning Goals

1. Evaluate leading 10 frontier and open source models
2. Implement open source projects based on what I am learning
3. Maintain this blog, as a learning resource
4. Add key skills to my resume

Career Goals

1. Build real world, LLM powered applications

Projects

1. Project - Run Ollama locally - Saad's notes
2. Project #1 - CareerGPT-Backend (Enterprise grade platform, evolving into E-Commerce app would not be an architectural stretch)
1. Service to service authentication via JWT (per service keys, DB backed)
2. Trust boundary, all microservices are behind API Gateway, using identity-backend as the sole issuer
3. Scope, no user level roles yet; services authenticate each other for internal calls
3. Project #2 - A logging microservice (log-service) separate from careergpt-backend.
1. Exposes a write-only API for client services to send structured logs.
2. Can later include:
3. Query API (for admin dashboards).
4. Filters (by app, user, level).
5. Stores logs in SQLite initially (upgradable to Postgres).
4. Project #3 - Identity Service
1. Identity-Backend is Trust Anchor
1. Service-to-Service Auth Model 
1. Identity-Backend - Why this isn't secure
2. RBAC
1. FSM, guest, email collected, verified, subscribed
2. States trigger role changes, policy updates, for example premium api for paid subscribers
3. FSM stored in DB, user states, state transitions tables, etc
3. External SSO and Federation
1. OpenID Connect, SAML 2.0 allowing user to authenticate using external IdP
2. Identity-backend exchanges external token -> issues internal JWT with proper claims
4. Zero trust alignment
1. Only trust identity-backend; validates every request
5. 
   

1. Enterprise hardening and zero trust roadmap (see ChatGPT)
2. …
3. Project #4 - Seperate data science lab, using jupyter from careergpt-backend
4.  Project #5 - Integrate Identity, and Logging services with CareerGPT-Backend
5. 
   
6. Project #Web Scraper Agent
7. Project #1 - AI powered brochure generator, scrapes and navigates assets, such as company website
8. Project #2 - Support agent for an airline with UI and remote procedure calls
9. Project #3 - Create meeting minutes from audio using both open and closed source models
10. Project #4 - AI that converts Python code to optimized C++, boost performance by 60,000X
11. Project #5 - Build AI knowledge-worker using RAG to become an expert on all company-related matters
12. Project #6 - Predict product prices from short descriptions using Frontier models
13. Project #7 -  Execute Fine-tuned open-source model to compete with Frontier in price prediction
14. Project #8 - Build Autonomous multi agent system collaborating with models to spot deals and notify you of special bargains.

Milestones

1. Inside the LLM - practical, hands-on take on transformers theory
2. RAG
3. LoRA
4. AI Agents
5. AI in Production - deploy LLMs and Agents for scale, resiliency, security

Week 1: Foundations and First Projects (Course onboarding - Saad's notes)

• Dive into the fundamentals of Transformers.

• Experiment with six leading Frontier Models.

• Build your first business Gen AI product that scrapes the web, makes decisions, and creates formatted sales brochures.

Monday - July 28th, 2025

1. Environment setup completed.
2. Process: OLlama - run LLM locally
3. Experience calling the OpenAI API for a frontier model.
4. High-level understanding of system vs. user prompts.
5. Build your data science lab environment
6. End of day notes: Daily - Saad's notes

Tuesday - July 29th, 2025

1. How much python do you need to know? Intermediate, for example can you answer what is wrong with: 
1. yield from set(book.get("author") for book in books if book.get("author"))
1. If you understand this, you are well-prepared. If you know why it might not be optimal, you are advanced.
2. If terms like yield, set, or .get are unfamiliar, a dedicated Jupyter notebook (a special Jupyter lab) in Week 1 provides a Python guide at the required level, taking you through stepping stones until such a line makes sense.
2. Three Pillars of LLM Engineering Covered:
1.  Models: Understanding the spectrum of LLMs (open-source, closed-source, multimodal, capable of generating images or audio), their architectures, and selecting the right one for a task.
2. Tools: Practical use of Hugging Face, LangChain (as glue code), Gradio, Weights & Biases, and Modal for deployment.
3. Techniques: Applying APIs, Retrieval-Augmented Generation (RAG), fine-tuning, and building full agentic AI solutions.
3. The course covers both closed-source (frontier) and open-source models. "Frontier models" refer to the LLMs pioneering what is possible today, the largest possible models. This term often refers to closed-source, paid models, but sometimes also to the biggest, strongest open-source models, depending on the context.
4. Closed-Source Frontier Models (Superscalers) - These are the largest, highest-scale, paid models from major AI companies.
1. GPT (OpenAI): ChatGPT, released in late 2022, caught everyone off guard with its power and brought generative AI to mainstream attention.
2. Claude (Anthropic): A primary competitor to OpenAI, often considered neck and neck with GPT in leaderboards (with Claude currently having a slight edge) and usually favored by data scientists.
3. Gemini (Google): Google's entrant in the frontier model space; Google also offers Gemma, an open-source variant.
4. Command R (Cohere): A model from Cohere, a Canadian AI company.
5. Perplexity: A search engine that can use other models but also has its own model.
5. Open-Source Models - These are freely available models for use and modification.
1. Llama (Meta): The most famous open-source model, as Meta paved the way by open-sourcing the original Llama 1.
2. Mixtral (Mistral AI): A "mixture of experts" model from the French company Mistral AI, containing multiple smaller models.
3. Qwen (Alibaba Cloud): A powerhouse model, super impressive and very powerful for its size, which will be used from time to time in the course.
4. Gemma (Google): Google's smaller, open-source model.
5. Phi (Microsoft): Microsoft's smaller, open-source model.
6. Three Core Methods for Using LLMs
1. Chat Interfaces: for example, chatgpt
2. Cloud APIs: Abstract away the complexity of operationalizing an LLM, allowing you to focus on on unified API
3. Local Execution: Using tools like Ollama, which optimize and compile the model's code into high-performance C++ code (via Llama CPP) for efficient local execution. This allows running models in inference and execution mode on your box but offers less control over internal operations due to the fully compiled code.
7. Rationale for Using Ollama
1. Advantages:
1. Cost-Free: No API charges; it's open-source, free, and runs locally on your machine.
2. Data Privacy: All data stays on your local machine, which is critical for confidential data that absolutely must not go to the cloud or leave the internet
2. Disadvantage
1. Performance: Local open-source models are generally smaller and less powerful than paid frontier models (which are many times larger and more powerful), so the expected result quality may be lower.

Day 3 - Wednesday - July 30th, 2025

Overview of Frontier LLM Models

• The session explored frontier large language models (LLMs), examining their strengths, weaknesses, and potential business applications, with the goal of providing a true intuition for their differences and commercial applicability.

• Model selection was framed as a strategic decision, considering technical capabilities, contextual nuances, pricing, and domain-specific expertise, with cost and API rate limits increasingly becoming primary differentiators as model performance converges.

Key Models and Their Characteristics

OpenAI Models

• GPT (including GPT-4.0)

  • Known for generating structured, well-researched, and nuanced responses, often with introductions and summaries.

  • Excels in creative tasks such as transforming bullet points into emails, slides, or blog posts, and is remarkably good at writing and debugging complex code, often acting as a "co-pilot."

  • GPT-4o, specifically, is multimodal (Omni), capable of generating images and handling "trickery" questions with witty and fun responses. It also features Canvas for collaborative code iteration and enhancement.

  • Sometimes struggles with simple, precise tasks like counting letter occurrences due to underlying tokenization strategies.

• O1 Preview (formerly Strawberry)

  • This model, originally codenamed Strawberry, is the strongest of OpenAI's models, currently available to pro-subscribers but slated for wider release.

  • Uses a chain-of-reasoning approach, delivering more precise and accurate results in tasks like counting letters or solving analogies.

  • Shows improved reasoning and accuracy compared to standard GPT outputs, successfully solving problems that GPT-4o might miss.

Anthropic’s Claude Family

• Includes Claude Haiku, Sonnet, and Opus, with Claude 3.5 Sonnet identified as the strongest due to recent versions surpassing older, more expensive models.

• Effective for coding assistance, producing code in "artifacts" for easy sharing and version tracking. It offers clear explanations and is characterized by a charismatic, succinct, and pithy style.

• Claude 3.5 Sonnet, released in October, has recently surpassed PhD-level proficiency in math, physics, and chemistry, leading in most benchmarks.

• Strong on socio-ethical considerations and safety alignment, often pushing back on comparative claims against other AI models due to its ethical framework. However, it may err on tasks needing precise numeric or token-based counts, similar to other LLMs.

Google Gemini

• The next-generation model, evolved from Bard, is now widely integrated into Google search responses.

• Provides thorough, sometimes overly literal answers, often lacking the ability to understand subtle humor or nuance in queries like the "Hawaii to 17" example.

• While maintaining factual accuracy, it is less creative or playful in humorous or abstract queries and also struggled with the letter counting task.

Cohere (Command R Plus)

• A Canadian AI company, Cohere's model is well-known for its focus on technical rigor and domain expertise, leveraging specialized knowledge bases (RAG).

• Produces well-structured and detailed answers, as seen in its response to philosophical questions, which was less expressive than Claude's but had more structure and substance. However, like many other models, it struggles with trivial counting tasks, reflecting limitations related to its training and inference mode.

Meta’s LAMA Model

• An open-source model, accessible via meta.ai, which supports interactive image generation (e.g., creating images for abstract concepts) and text responses.

• Performs adequately on direct questions, but its overall power is noted as less than some other frontier models. It is also limited in nuanced tasks like accurate letter counting.

Perplexity

• Functions primarily as an AI-powered search engine rather than a traditional LLM, capable of using other models but also having its own.

• Excels at summarizing current events and demonstrates strong research capabilities by providing nuanced, well-crafted responses to real-time information, even when other models have knowledge cutoffs.

• Notably, it successfully handles some precise counting tasks, placing it alongside O1 Preview in accuracy for such challenges. It pushes back on self-comparison questions, stating it lacks such capabilities.

Common Strengths and Weaknesses

Strengths

• All models are astonishingly powerful, generating structured, well-reasoned responses to detailed technical and creative queries.

• Effectively transform brief prompts or bullet points into comprehensive outputs like emails, code, slides, blog posts, and explanations.

• Claude 3.5 Sonnet has demonstrated near “PhD-level” proficiency in math, physics, and chemistry, with other models expected to follow.

Weaknesses

• Consistent challenges with precise token counting (e.g., counting the letter “A”) due to tokenization and input processing strategies.

• Varying ability to understand humor, indirect references, and subtle context, often providing overly literal interpretations, highlighting the need for human oversight in critical uses.

• Tend to not be as strong with highly specialized subject matter, particularly in specific business domains, lacking the expert knowledge of a human.

• Some models, especially with knowledge cutoffs (e.g., GPT's October last year cutoff), may struggle with recent events or emerging topics, requiring supplemental tools like Perplexity for real-time insights.

• Can exhibit "strange blind spots," getting questions wrong but confidently stating incorrect answers, a phenomenon known as hallucination.

Business and Strategic Insights for Senior Management

• As LLM performance converges across models, cost and API rate limits are becoming primary differentiators, making efficient pricing and resource management key strategic factors.

• Demonstrations of iterative code enhancements (e.g., GPT-4.0 with Canvas) show significant efficiency gains in problem-solving, debugging, and rapid prototyping, acting as a powerful "co-pilot."

• Comparative experiments (e.g., the leadership challenge among models) reveal that while all are powerful, differences in personality, safety, and alignment can significantly impact their suitability for sensitive or client-facing projects.

• Selecting the LLM that best fits specific business needs (considering factors like scalability, handling unstructured data, contextual understanding, and domain expertise) remains crucial for strategic implementation.

Insights on Future Directions

• Upcoming sessions will cover foundational concepts such as transformer architectures, token mechanisms, context window management, and API cost considerations—all key for technological oversight and strategic planning.

• Bridging technical details with commercial deployment strategies will support data-driven decisions in non-traditional industries.

Day 4 - July 4th, 2025