{"id":137,"date":"2025-08-20T06:16:05","date_gmt":"2025-08-20T06:16:05","guid":{"rendered":"https:\/\/hattussa.com\/blog\/?p=137"},"modified":"2025-12-16T13:05:52","modified_gmt":"2025-12-16T13:05:52","slug":"from-scratch-2m-parameter-llm-in-python","status":"publish","type":"post","link":"https:\/\/hattussa.com\/blog\/from-scratch-2m-parameter-llm-in-python\/","title":{"rendered":"From Scratch: 2M-Parameter LLM in Python"},"content":{"rendered":"

<\/p>\n

\n<\/p>\n

<\/p>\n

\n

<\/p>\n

\n
Table of contents<\/div>\n
    \n
  • Creating a 2M Parameter Thinking LLM from Scratch Using Python<\/li>\n
  • Why Build a Small Language Model?<\/li>\n
  • The Core Idea: Transformers at a Small Scale<\/li>\n
  • How to Keep It Under 2 Million Parameters<\/li>\n
  • Data: The Most Crucial Ingredient<\/li>\n<\/ul><\/div>\n

    <\/p>\n

    \n
    \n

    Creating a 2M Parameter Thinking LLM from Scratch Using Python<\/h1>\n

    The world of language models has often been associated with massive, resource-intensive systems like GPT-4 or PaLM, boasting billions of parameters and trained on enormous datasets. However, a new wave of small yet capable language models<\/strong> is gaining attention \u2014 models like OpenChat’s o3<\/strong> or DeepSeek-R1<\/strong>. These lightweight models have fewer than 2 million parameters yet demonstrate surprising competence in reasoning and general-purpose tasks.<\/p>\n

    In this blog post, we’ll explore how you can build a 2M parameter “thinking” LLM<\/strong> entirely from scratch, using Python and open-source tools \u2014 without requiring a supercomputer.<\/p>\n

    \"Chart
    \n <\/section>\n

    \n

    Why Build a Small Language Model?<\/h2>\n

    While massive LLMs capture headlines, small models come with their own powerful advantages:<\/p>\n

      \n
    • \u2705 Fast Training<\/strong> \u2013 Small models can be trained in hours instead of days.<\/li>\n
    • \u2705 Lower Costs<\/strong> \u2013 You don\u2019t need expensive GPUs or cloud infrastructure.<\/li>\n
    • \u2705 Deploy Anywhere<\/strong> \u2013 Perfect for edge devices, mobile apps, or offline use.<\/li>\n
    • \u2705 Customizable<\/strong> \u2013 You can fine-tune for specific use cases without large datasets.<\/li>\n<\/ul>\n

      Most importantly, they offer an excellent learning opportunity<\/strong>. You\u2019ll understand the inner workings of transformers, attention mechanisms, and training dynamics \u2014 all while building a usable AI system.<\/p>\n<\/section>\n

      \n

      The Core Idea: Transformers at a Small Scale<\/h2>\n

      Even with just 2 million parameters, a well-designed transformer can learn how to generate, complete, and even reason through text. These compact models use the same building blocks as their larger cousins:<\/p>\n

        \n
      • Token and Position Embeddings<\/strong>: Turning text into a form the model can understand.<\/li>\n
      • Self-Attention Mechanisms<\/strong>: Allowing the model to “focus” on relevant words.<\/li>\n
      • Layer Norms and Feedforward Layers<\/strong>: For smoother learning and generalization.<\/li>\n
      • Final Prediction Head<\/strong>: Producing the next-word predictions or outputs.<\/li>\n<\/ul>\n

        The key is to scale down<\/strong> wisely \u2014 using smaller embeddings, fewer attention heads, and shallower layers \u2014 while still retaining the core structure.<\/p>\n<\/section>\n

        \n

        How to Keep It Under 2 Million Parameters<\/h2>\n

        To stay within the 2M parameter limit, everything must be optimized:<\/p>\n

          \n
        • Use a smaller vocabulary size<\/strong>, especially if focusing on a specific domain.<\/li>\n
        • Limit the number of layers and attention heads<\/strong>.<\/li>\n
        • Choose compact embedding dimensions<\/strong>.<\/li>\n
        • Reduce the size of the internal feedforward layers<\/strong>.<\/li>\n<\/ul>\n

          Even with these constraints, your model can still learn to reason<\/strong>, generate text, and perform tasks if trained properly.<\/p>\n

          t include text paired with images, video, or sound. This enables the models to learn how different types of information relate to each other.<\/p>\n<\/section>\n

          \n

          Data: The Most Crucial Ingredient<\/h2>\n

          What your model learns depends heavily on what it\u2019s fed. Here\u2019s what to keep in mind:<\/p>\n

            \n
          • Focus on high-quality, domain-specific data<\/strong>.<\/li>\n
          • Use clean, well-structured examples<\/strong> for reasoning.<\/li>\n
          • Chain-of-thought (CoT)<\/strong> style training helps the model learn to \u201cthink\u201d step-by-step.<\/li>\n<\/ul>\n
            \n

            Q:<\/strong> If you have 3 apples and give away 1, how many are left?
            \n A:<\/strong> You started with 3. You gave away 1. That leaves 2.
            \n Answer: 2.<\/p>\n<\/blockquote>\n

            Even small models can learn this kind of logic \u2014 if trained on thousands of similar examples.<\/p>\n<\/section><\/div>\n


            \n \n<\/div>\n<\/section>\n","protected":false},"excerpt":{"rendered":"

            The world of language models has often been associated with massive, resource-intensive systems like GPT-4 or PaLM, boasting billions of parameters and trained on enormous datasets. However, a new wave of small yet capable language models<\/strong> is gaining attention \u2014 models like OpenChat’s o3<\/strong> or DeepSeek-R1<\/strong>. These lightweight models have fewer than 2 million parameters yet demonstrate surprising competence in reasoning and general-purpose tasks.<\/p>\n","protected":false},"author":1,"featured_media":138,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-137","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/hattussa.com\/blog\/wp-json\/wp\/v2\/posts\/137","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/hattussa.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/hattussa.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/hattussa.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/hattussa.com\/blog\/wp-json\/wp\/v2\/comments?post=137"}],"version-history":[{"count":4,"href":"https:\/\/hattussa.com\/blog\/wp-json\/wp\/v2\/posts\/137\/revisions"}],"predecessor-version":[{"id":325,"href":"https:\/\/hattussa.com\/blog\/wp-json\/wp\/v2\/posts\/137\/revisions\/325"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/hattussa.com\/blog\/wp-json\/wp\/v2\/media\/138"}],"wp:attachment":[{"href":"https:\/\/hattussa.com\/blog\/wp-json\/wp\/v2\/media?parent=137"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/hattussa.com\/blog\/wp-json\/wp\/v2\/categories?post=137"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/hattussa.com\/blog\/wp-json\/wp\/v2\/tags?post=137"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}