core logic (matrix dimensions):

goal: generate one new token at a time.
input: most recent token, shape [b, 1, n].
b: batch size, n: embedding dim, t: current sequence length.

step-by-step for token t+1:

• projection:

  • input [b, 1, n] × weight matrix W_qkv [b, n, 3n] → [b, 1, 3n]

  • split into:

    • q_new [b, 1, n]

    • k_new [b, 1, n]

    • v_new [b, 1, n]

• use cache:

  • retrieve K_cache [b, t, n], V_cache [b, t, n]

  • append new:

    • K_total = concat(K_cache, k_new) → [b, t+1, n]

    • V_total = concat(V_cache, v_new) → [b, t+1, n]

• attention calculation:

  • scores: q_new @ K_total^T → [b, 1, t+1]

  • weights: softmax(scores / sqrt(n)) → [b, 1, t+1]

  • output: weights @ V_total → [b, 1, n]

result: output [b, 1, n], passed to next layer for next token prediction.

if you ever forget why we can store k and v as cache and how is it equivalent to doing fully, just remember the row matrice multiplication , you will realize

Arithmetic Intensity Analysis of Transformer Operations (on NVIDIA H100)

Machine Balance (AI_knee): ~295 FLOPs / Byte

If AI > 295: Compute-Bound (Limited by 989 TFLOP/s)

If AI < 295: Memory-Bound (Limited by 3.35 TB/s)

B: Batch Size

S: Sequence Length

D: Model Hidden Dimension

F: MLP Intermediate Dimension (typically ~4D)

kv cache memory calculation

calculation per layer:

• memory for keys: b × s × h × d_k × 2 bytes

• memory for values: b × s × h × d_k × 2 bytes

• total per layer: b × s × h × 2 × d_k × 2 bytes = b × s × 2 × d × 2 bytes (since h × d_k = d)

total for entire model:

• kv cache size (bytes) = b × s × l × 2 × d × 2

example: llama 2 13b

• l = 40 layers, d = 5120, b = 1, s = 4096

• cache size = 1 × 4096 × 40 × 2 × 5120 × 2 = 3,355,443,200 bytes ≈ 3.36 gb

• this is in addition to ~26 gb for model weights

kv cache size is proportional to the number of key and value vectors stored. standard mha is inefficient.

revisiting multi-head attention (mha)

structure: in mha, the model’s hidden dimension d is split among n attention heads. each head operates independently with its own query, key, and value projection weights (w_q, w_k, w_v).

implication for kv cache: if a model has n query heads, it also has n key heads and n value heads. for every token, we must compute and store n distinct key vectors and n distinct value vectors.

visualization:
query heads: [q1] [q2] [q3] [q4] [q5] [q6] [q7] [q8]
key heads: [k1] [k2] [k3] [k4] [k5] [k6] [k7] [k8]
value heads: [v1] [v2] [v3] [v4] [v5] [v6] [v7] [v8]

the number of k/v heads is equal to the number of q heads.

multi-query attention (mqa)
mqa is based on a simple observation: the model might not need the full expressive power of n distinct key and value heads.

structure: mqa maintains n query heads but uses only a single key head and a single value head.

these single k/v heads are shared across all n query heads.
implication for kv cache: for every token, we compute n q vectors, but only one k vector and one v vector.

visualization:
query heads: [q1] [q2] [q3] [q4] [q5] [q6] [q7] [q8]
key head: +-------------------[ k ]-------------------+
value head: +-------------------[ v ]-------------------+

benefit: the size of the kv cache is reduced by a factor of n (the number of heads). for a model with 40 heads, this is a 40x reduction in cache size and a 40x reduction in the amount of data that needs to be read from hbm during the decode step’s attention calculation. this directly improves tpot (time-per-output-token).

grouped-query attention (gqa)
gqa is an interpolation between the extremes of mha and mqa. it recognizes that while mqa offers huge savings, the quality degradation from sharing a single k/v head might be too severe.

structure: gqa maintains n query heads, but groups them. each group of g query heads shares a single key/value head pair. the total number of k/v heads is n/g.
implication for kv cache: it offers a tunable trade-off.

if g=1, you have n/1 = n k/v heads, which is identical to mha.
if g=n, you have n/n = 1 k/v head, which is identical to mqa.

visualization (n=8, g=4):
query heads: [q1] [q2] [q3] [q4] | [q5] [q6] [q7] [q8]
key heads: +------[ k1 ]------+ | +------[ k2 ]------+
value heads: +------[ v1 ]------+ | +------[ v2 ]------+

benefit: reduces kv cache size by a factor of g. llama 2 models, for instance, use gqa to manage their kv cache size while maintaining high quality.

multi-head latent attention (mla)
mla (used in the deepseek-v2 model) takes a different approach to dimension reduction. instead of reducing the number of k/v heads, it reduces the dimension of each k/v vector.
structure: mla projects the full-dimension key and value vectors (nhd_k) down to a much smaller compressed or “latent” dimension, c.
example (deepseek-v2):
full k/v dimension: 16384multi-head latent attention (mla) reduces kv cache size by compressing the dimension of each key and value vector, rather than reducing the number of heads. instead of storing full-dimensional k/v vectors (nhd_k), mla projects them down to a much smaller latent dimension c using a compression matrix. for example, in deepseek-v2, the full k/v dimension is 16384, which is compressed to 512, giving a 32x reduction in cache size.

the data flow for mla is as follows:

• input: token embedding x, shape (1, d)

• k-projection: x is multiplied by the key weight matrix w_k to get the full-dimensional key k_full (1, d)

• compression: k_full is then multiplied by a new compression matrix w_compress (d, c) to get the compressed key k_latent (1, c)

• storage: only k_latent is stored in the kv cache; k_full is discarded

• attention: when a new query q_full is generated, it is also compressed using w_compress to get q_latent (1, c)

• scores: attention scores are computed in the latent space: q_latent @ k_cache_latent^t

• decompression: the attention output (c-dimensional) is projected back to the full model dimension d using a decompression matrix w_decompress (c, d)

a wrinkle is that compression can interfere with positional encodings like rope, which work on the full-dimensional vectors. to handle this, deepseek-v2 adds back 64 dimensions, making the final latent dimension 576. this allows efficient cache storage while preserving the ability to use advanced positional encodings.
compressed latent dimension c: 512

cross-layer attention (cla)
the idea: gqa shares key/value vectors across heads in the same layer. cla extends this by sharing key/value vectors across multiple consecutive layers.
mechanism:

• layer 10 computes and caches its own k and v vectors.

• layer 11 skips k/v computation and reuses the cache from layer 10 for its attention.

• layer 12 does the same.
  benefit: memory savings are substantial—sharing one kv cache across 3 layers reduces memory by 3x.
  trade-off: model loses expressive power, as layers share less specialized information. must be trained from scratch with this architecture.

local attention (sliding window attention)
the idea: for many tasks, only the local context matters. long-range dependencies are often negligible.
mechanism: each token only attends to a fixed window of recent tokens (e.g., last 512 or 4096).
visualization:

full attention: token 8000 attends to all previous tokens.

local attention: token 8000 attends only to last 512 tokens.
benefit - compute: q @ k^t matrix multiplication is much smaller, speeding up prefill for long sequences.
benefit - cache: kv cache size is fixed (independent of sequence length). oldest tokens are evicted as new ones arrive, enabling infinite sequences with small cache.
cache size (local): b × w × l × 2 × d × 2
problem: model becomes short-sighted—cannot access distant context, hurting tasks needing long-range dependencies.
solution - hybrid layers: most layers use local attention, but every nth layer uses full attention. balances efficiency with long-range coherence.

tell me what would would be the the dimensions of Value, and how

the static batching problem

static batching (batched scheduling):

• inference server collects b user requests, pads them to the length of the longest request, and processes them as one dense tensor.

• processing starts only when batch is full or timeout reached.

example:
three requests:

• a: “the cat sat on the” (5)

• b: “summarize this article” (4)

• c: “translate to french: hello” (3)

padded batch:

[ the, cat, sat, on, the ]
[ summ, this, article, p, p ]
[ tran, to, french, hello, p ]


forms a (b=3, s=5) tensor for efficient gpu processing.

inefficiencies of static batching:

a. throughput inefficiency (padding):

• gpu computes on padding tokens—wasted computation.

• in example, 3/15 slots (20%) are padding. real workloads often see 50–70% waste.

• reduces max throughput; wastes compute and kv cache memory.

b. latency inefficiency (head-of-line blocking):

• short, fast requests wait for longer ones in the queue.

• requests arriving early wait for batch to fill or for long requests to finish.

• consequences:

  • high time-to-first-token (ttft): early requests delayed by 100–500ms.

  • poor fairness: fast requests wait behind slow ones, hurting user experience.

How padding really works

the attention mask

the attention mask is the key mechanism that prevents the model from attending to padding tokens.

• before softmax, a mask matrix is added to the attention scores (q @ k^t).

• mask contains 0 where attention is allowed, and a large negative number (e.g., -1e9) at padding positions.

• after softmax, scores for padding positions become effectively zero.

• result: no token attends to padding tokens—they are invisible to attention.

masking during loss

example: batch of two sequences

• sequence 1: [the, cat, sat, <eos>] (length 4)

• sequence 2: [hello, world, <eos>, <pad>] (length 4 after padding)

prediction targets (y values)

• for each input, the target is the next token in its own sequence.

• after <eos>, there is no meaningful target—the sequence is over.

why masked loss is necessary

• the model produces logits at every position, even after <eos> and at <pad>.

• if we don’t mask loss at these positions, the model would be forced to predict something after the sequence ends.

• this would teach nonsense patterns, corrupting language understanding.

solution: gradients for <eos> and <pad> tokens are set to zero.

• these positions are ignored during training.

• the network does not learn from them, preserving correct language structure.

note in static batching you can not do prefil and decode together, because then the padding for the decode tokens would become very very massive

Continuous batching

instead of looking it like we will wait for some pre-fixed batch of user requests, what we do is we see it as a per token generation or per forward pass iteration. for one forward pass the batch and sequence length does nto change but for another forward pass they might change, based on the user request. in that single forward pass we want have many requests of type both prefil and decode. the decode type will have sequence length of 1 only but the prefil one can have varying sequence lengths so the way we are looking at it now even doing a forward pass is impossible

you can not have (batch , changing not constant sequene, dimension) size fo a matrice and do a matrice multiplication. so what we do instead is multiply the batch and sequence (or add all the sequence).

suppose there is a prefil request of 500 tokens and 4 decode request so our matrice will become of size (504, n) and multiplying it with a (n, 4n) matrice is now possible

now lets look at the attention machanism.

you would have notices that we can generate k q v vectors of each token in parallel now also that is not an issue but

q * k ^ Trnaspose. this operation is hard to do because the last dimension of k keeps chanfing for both decode and prefill so we can not batch them. so we will have to do them in parallel but we will have to write a custom cuda kernel to do this efficiently.

now while writing the custom cuda kernel there is a issue, to solve which we introduce

paged attention

without pagedattention, each new request requires a contiguous block of gpu vram for its full kv cache:

• request a (max_len=1024): allocate 1024-token block

• request b (max_len=2048): allocate 2048-token block

• request c (max_len=1024): allocate another 1024-token block

this causes two major problems:

internal fragmentation:

• if request a only uses 50 tokens, 974 token-slots are wasted, even though they’re allocated.

external fragmentation:

• when request b finishes, its 2048-token block is freed.

• a new request d needs 2049 tokens. even if total free memory is enough, if no single contiguous block is large enough, the request fails.

this fragmentation makes memory management inefficient and scheduler complex.

pagedattention: the solution

pagedattention applies the idea of paging from os virtual memory:

• physical memory: gpu kv cache is split into many small, fixed-size blocks (pages), e.g., 16 tokens per block.

• logical view: each sequence sees its kv cache as a continuous sequence.

• page table: for each request, a lookup table maps logical blocks to physical blocks in memory.

this allows flexible allocation, reduces fragmentation, and makes memory usage much more efficient.

Speculative Sampling

step 1: drafting (fast & sequential)

• a small, fast draft model (m_draft) runs autoregressively and generates k candidate tokens.

• example: draft predicts [”the”, “quick”, “brown”, “fox”]. this is fast due to the model’s small size.

step 2: verification (fast & parallel)

• the large target model (m_target) takes the original context plus the entire draft as input.

• one forward pass produces probability distributions (logits) for each draft position.

step 3: acceptance/rejection (rejection sampling)

• compare draft predictions with target model’s probabilities token by token.

• token 1 (”the”): if p_target(”the”) is high, accept.

• token 2 (”quick”): if previous token accepted, check p_target(”quick” | “the”). if high, accept.

• token 3 (”brown”): if target prefers “red”, reject “brown”.

• chain breaks: any rejection discards the rest of the draft.

step 4: correction and resumption

• keep accepted tokens (e.g., [”the”, “quick”]).

• at the rejection point, sample a corrected token from target model’s logits (e.g., “red”).

• resume generation from the corrected token. draft model generates a new draft from there.

performance gain

• one draft and one verification step can yield multiple tokens.

• speedup depends on acceptance rate: better draft models = longer accepted sequences = greater speedup.

• method is lossless: final sequence matches target model’s distribution exactly.

• not an approximation—just a faster generation method.

Distillation

inputs: standard text dataset (e.g., “the quick brown...”).

forward pass (student):

• text is fed into the student model.

• produces logits for next token (e.g., fox: 5.0, dog: 2.0, car: -10.0).

forward pass (teacher):

• same text is fed into the frozen teacher model.

• produces its own logits (e.g., fox: 10.0, dog: 8.0, car: -20.0).

loss function (the “training” part):

• standard supervised learning: loss checks if model predicted the correct single word.

• distillation: loss (kl-divergence) checks if student’s probability distribution matches teacher’s.

• this is “curve fitting.”

step 1: softmax with temperature

• divide logits by temperature t before softmax.

• high t smooths the distribution, making small probabilities more visible.

• p_student = softmax(student_logits / t)

• p_teacher = softmax(teacher_logits / t)

step 2: calculate divergence

• loss = kl_divergence(p_teacher, p_student)

step 3: backpropagation

• gradients of loss are calculated w.r.t. student’s weights.

• optimizer updates student to match teacher’s output curve.

• teacher weights remain frozen.

  ---

in quantization weights are stored in quantized manner but to do calculations they are cnoverted to bf16

also pruning is removing some layers based on how active they are in a sample dataset

Given a tensor with Shape (d0,d1,…,dn−1), the stride sk for dimension k is the product of the sizes of all subsequent dimensions.

for the last dimension (k = n-1):
s(n-1) = 1

for any other dimension k (counting backwards from n-2 down to 0):
s(k) = s(k+1) * d(k+1)

the explicit formula:
s(k) = product from j = k+1 to n-1 of d(j)

The Memory Offset Formula (Linear Index)

given a specific logical index (i0, i1, ..., in-1) and strides (s0, s1, ..., sn-1):

contiguous vs non-contiguous memory:

a tensor is contiguous if its elements are stored in memory in the order they’re traversed when incrementing indices from right to left (axis -1 moves fastest).

example:
matrix a, shape (2, 3), storage: , strides: (3, 1).
after transpose b = a.t(), shape (3, 2), storage unchanged, strides: (1, 3).
iterating along axis 1 of b now jumps by 3 in memory—not sequential. this is non-contiguous.

why it matters:
operations like .view() need contiguous memory. if a tensor is non-contiguous, .view() fails. use .contiguous() to copy data into contiguous storage.

Broadcasting is “Virtual” Memory

broadcasting uses strides to repeat data without copying.
take a vector v, shape (1, 3), storage: , strides: (3, 1).
to broadcast to shape (4, 3), pytorch creates a view:
shape: (4, 3), strides: (0, 1).

analyze stride 0:
offset = (i0 × s0) + (i1 × s1)
for row 0: offset = (0 × 0) + (0 × 1) = 0 (value: 10)
for row 3: offset = (3 × 0) + (0 × 1) = 0 (value: 10)

stride 0 means moving along axis 0 doesn’t advance the pointer. the tensor appears to have 4 rows, but it re-reads the same row.
mental model: broadcasting isn’t copying or stretching—set stride to 0 to freeze the pointer for that axis.

checking if two tensors are compatible for brodcasting

step1 —> keep their dimensions vertically stacked

for each verticle stacked dimension check if either they are equal, one of them is one, or does not exist

if either of this is voilated we can not do brodcasting

for exmaple

1 3 2
2

matches

1 3 2
1 2 3

does nto match

3 1 2
4 2

yes matches

one more way

scenario a: original (5, 6)
target: (5, 6, 10)
input: (5, 6)
alignment:
ax-3 ax-2 ax-1
tgt: 5 6 10
inp: 5 6
result: crash. ax-1: 10 vs 6. mismatch.

scenario b: reshaped (5, 6, 1)
target: (5, 6, 10)
input: (5, 6, 1)
alignment:
ax-3 ax-2 ax-1
tgt: 5 6 10
inp: 5 6 1
result: compatible. ax-1: 10 vs 1. expansion.

you can also broadcast a scalar you know how so I wont go into depths of that

the element-wise product (hadamard product)
symbol: * or torch.mul()
logic: pure broadcasting.
rule: align dimensions right-to-left. 1s expand. multiply cell-by-cell.
result shape: max size at each dimension.
example: (3, 4) * (3, 4) → (3, 4)
c(i,j) = a(i,j) × b(i,j)

the matrix product (matmul)
symbol: @ or torch.matmul()
logic: linear algebra + broadcasting.
rule: split dimensions into batch (all except last two) and matrix (last two).

step 1: matrix compatibility
a: (..., n, k)
b: (..., k, m)
inner dimension k must match. result: (n, m)

step 2: batch compatibility
broadcast remaining dimensions (right-to-left, 1s expand).

example:
a: (3, 4), b: (4, 5)
a * b → error (4 vs 5)
a @ b → (3, 5)

high-dimensional matmul:
a: (10, 1, 3, 4)
b: (1, 20, 4, 5)
matrix core: (3, 4) @ (4, 5) → (3, 5)
batch: (10, 1) and (1, 20) → broadcast to (10, 20)
final shape: (10, 20, 3, 5)

note :- we can not multile 10,3,5 to 10,1,8

case a: vector @ matrix
vector v: (4,)
matrix m: (4, 5)
hidden process:

• prepend 1: v becomes (1, 4)

• multiply: (1, 4) @ (4, 5) → (1, 5)

• squeeze: remove 1 → (5,)

case b: matrix @ vector
matrix m: (3, 4)
vector v: (4,)
hidden process:

• append 1: v becomes (4, 1)

• multiply: (3, 4) @ (4, 1) → (3, 1)

• squeeze: remove 1 → (3,)

broadcasting is automatic shape manipulation. for manual geometry changes, use view, reshape, and permute.

permute(*dims): swaps axes. changes stride order, not data. result is usually non-contiguous. example: (a, b, c) → (c, a, b). you can say this is generalized version of transpose

view(*shape): reshapes tensor without copying data. only works on contiguous tensors. fails if memory is scrambled (e.g. after permute).

reshape(*shape): like view, but if tensor is non-contiguous, silently copies data to make it contiguous first. always works, but may be slower.

unsqueeze(dim): adds a dimension of size 1 at the specified axis. pure metadata operation—no data copied. used for explicit alignment, e.g., (3,) → (3, 1).

squeeze(dim=None): removes dimensions of size 1. if dim given, only removes that axis if size is 1. otherwise, removes all singleton dimensions. also a metadata-only operation.

expand(*shape): manually expands dimensions of size 1 to larger sizes, using stride 0—no data copied. new shape must match or expand from original, and only 1s can be expanded. expands are views, not copies.

expand vs repeat: expand is virtual (stride 0), repeat is physical (copies data).

intranode communication is faster than internode communication

Basic Operations

Broadcast: GPU 0 starts with the data (e.g., model weights) and sends a copy to GPU 1, 2, and 3 so that all GPUs end up with identical data.

Scatter: GPU 0 starts with a list of data items [A, B, C, D], splits it into chunks, keeps A, sends B to GPU 1, C to GPU 2, and D to GPU 3, so the data is partitioned and each GPU owns a slice.

Gather: Each GPU has a partial result (GPU 0: A, GPU 1: B, GPU 2: C, GPU 3: D), everyone sends their piece to GPU 0, and GPU 0 ends up with the full list [A, B, C, D] while others still only have their own piece.

All-Gather: Each GPU starts with a piece (A, B, C, or D), everyone sends their piece to everyone else, and every GPU ends up with the full list [A, B, C, D], but this is expensive because it moves a lot of data.

Reduce: Each GPU starts with a number (GPU 0: 1, GPU 1: 2, GPU 2: 3, GPU 3: 4), everyone sends their number to GPU 0, GPU 0 applies an operation like sum to get 1+2+3+4=10, so GPU 0 holds the reduced result (10) while others keep their original numbers.

its basically a gather operation at the end of which a sum or avg or max happens, but then why it is given a separate name, no clear answers but yeah the sum happening at the end can be made very efficient if we know we are gonna sum

All-Reduce: basically suppose you have vectors of size 4 in 4 saperate gpu’s

gpu 1 —> A0 A1 A2 A3
gpu 2 —> B0 B1 B2 B3

gpu 3 —> c0 c1 c2 c3
gpu 4 —> d0 d1 d2 d3

now you want each gpu to have four elements [sigma a , sigma b, sigma c, sigma d]
what you will do , you will perform an internal reduction in each gpu and then perform an all gather right

but what if you have

gpu 1 —> A0 B0 C0 D0
gpu 2 —> A1 B1 C1 D1

gpu 3 —> A2 B2 C2 D2

gpu 4 —> A3 B3 C3 D3

and you want each gpu to have four elements [sigma a , sigma b, sigma c, sigma d]

one approach you can take is gpu 1 sends b0 c0 d0, and receives a1
similarly for all other gpus, but this approach is kind of inefficient. because every node is talking to every node, congesting the network

alternate approach :- Ring Reduce

Initial State (Time = 0):

GPU 0: [A0, B0, C0, D0]
GPU 1: [A1, B1, C1, D1]
GPU 2: [A2, B2, C2, D2]
GPU 3: [A3, B3, C3, D3]

Step 1 of 3:

GPU 0: [A0, B0, C0+C3, D0]
GPU 1: [A1, B1, C1, D1+D0]
GPU 2: [A2+A1, B2, C2, D2]
GPU 3: [A3, B3+B2, C3, D3]

Step 2 of 3:

GPU 0: [A0, B0+B3+B2, C0+C3, D0]
GPU 1: [A1, B1, C1+C0+C3, D1+D0]
GPU 2: [A2+A1, B2, C2, D2+D1+D0]
GPU 3: [A3+A2+A1, B3+B2, C3, D3]

Step 3 of 3 (final step of Phase 1):

Send:
GPU 0 sends B0+B3+B2 to GPU 1.
GPU 1 sends C1+C0+C3 to GPU 2.
GPU 2 sends D2+D1+D0 to GPU 3.
GPU 3 sends A3+A2+A1 to GPU 0.

Receive and add:
GPU 0 receives A3+A2+A1 and adds it to A0. Result: A_sum.
GPU 1 receives B0+B3+B2 and adds it to B1. Result: B_sum.
GPU 2 receives C1+C0+C3 and adds it to C2. Result: C_sum.
GPU 3 receives D2+D1+D0 and adds it to D3. Result: D_sum.

End of Phase 1: Reduce-Scatter is complete.

GPU 0 now holds only A_sum.
GPU 1 now holds only B_sum.
GPU 2 now holds only C_sum.
GPU 3 now holds only D_sum.

The other chunks can be discarded.

or we can now perform an all gather so that everyone has all the sums we call this Ring Reduce All

Memory

• Static memory has three parts: weights, gradients, and optimizer states.

• With mixed precision (BF16 weights, Adam in FP32), you pay about 16 bytes per parameter:

  • 2 bytes: weights (BF16)

  • 2 bytes: gradients (BF16)

  • 12 bytes: Adam (FP32 master weights + momentum + variance)

• For a 7B model, that is roughly 112 GB, even though the raw weights are only about 14 GB.

• Dynamic memory comes from activations saved for backprop.

• Activation memory scales with sequence length, batch size, hidden size, and number of layers.

• activation-memory ​= s⋅b⋅h⋅(constant)

Activation Checkpointing

A standard Transformer block (simplified) looks like this:

Input (x0) -> [LayerNorm 1] -> (x1) -> [Self-Attention] -> (x2) -> [Residual Add: x0+x2] -> (x3) -> [LayerNorm 2] -> (x4) -> [MLP] -> (x5) -> [Residual Add: x3+x5] -> Output (x6)

we only store x0 and x6 and discard all of them and re-calculate while back propagating to save memory

note : this is only for training we do not do this in infrence, I mean like we cache kv, everything else we anyway delete

Data Parallelism

• Split batch: 32 images → 8 per GPU (GPUs 0–3).

• Forward: Each GPU runs forward pass on its 8 images with its local copy of the model.

• Backward: Each GPU computes gradients from its own 8 images (gradients differ across GPUs).

• All-Reduce: Use All-Reduce on gradients to average them across GPUs so every GPU has the same Grad_avg.

• Optimizer step: Each GPU updates its local weights with the same rule: Wnew=Wold−lr×GradavgWnew=Wold−lr×Gradavg, keeping all model replicas in sync.

• Data Parallel (DDP) assumes each GPU can store a full copy of the model, its gradients, and optimizer states.

• For big LLMs, static training memory is huge: a 7B model can need ~112 GB (weights + gradients + Adam states).

• With DDP, every GPU must hold this full 112 GB copy.

• Even if you have 4×80GB GPUs, each individual GPU still can’t fit 112 GB, so DDP will run out of memory and crash.uu

ZeRO Stage 2

why stage two? because stage one is stupid and not worth studying. consider this particular setup

this is a two layer neural net being trained on two saperate gpu’s.

now notice that gradient 2 and optimizer 2 is missing from GPU 1 and viceversa. why? to save space, but then how will we train this?

look at the backward pass,
GPU 1 —> we calculate gradients for W1 and W2 both
GPU 0 —> we calculate the gradients for W1 and W2 both

now we do ring-reduce-scatter and GPU 1 has the average gradient for W1 and GPU 0 has average gradient for W2 (all other gradients are deleted, for example GPU1 deleted gradient for W2) , they individually update the weights with the optimzers they have and then share the weights (GPU 1 shared W1 and GPU0 shares W2) through an all gather.

basically as soon as gradients for W2 are calculated, an all reduce takes place, the weights are updates for W2 on a specefic gpu and then all the gradients for W2 accross all gpus are thrown away

instead of storing all the gradients for all the weights, we need to store them for only a single layer at a time

what if we are training lamma 70b, its weights require 140GB but we do not have that in a single GPU so we will have to distribute the weights too, but how will we do that

Stage-3 / FSDP

the forward pass starts at layer i. the weights for layer i are split into pieces, with each gpu holding one piece. at this point, no gpu can do any calculation because each only has part of the weights.

the system then does an all-gather operation on the weights. each gpu sends its piece to all others. for a short time, every gpu has the complete set of weights in memory.

now, each gpu runs the forward pass using the full weights to compute the output.

as soon as the calculation is done, each gpu deletes the full weights from memory. they only keep their original small piece. this brings memory usage back down.

later, during the backward pass, the gradients need to be calculated for layer i. to do this, the full weights are needed again, but they were deleted.

so, the system does another all-gather on the weights. each gpu sends its piece again, and every gpu reconstructs the full weights in memory.

each gpu then computes the gradients using the full weights.

after this, the gradients are reduced and scattered. each gpu gets only the part of the gradients that matches its original weight shard.

finally, each gpu deletes the full weights and the full gradient vector from memory, keeping only its own small piece.

weights are stored in FP-32 but converted into fp-16 at the time of transmission and calculation

Tensor Parallelism

lets just say that if we were doing fsdp to save memory then we are doing tensor parallelism to save compute

if you do not know how block matrice multiplication works check it out here

I want you to forget all about data parallelism for now, we are training on a single batch of data

the multi-layer perceptron (mlp) in a transformer is made of two linear layers with an activation function in between. for layer a, the math is y = x * a. the input x has size [1, h], and the weight matrix a has size [h, 4h]. the output y has size [1, 4h].

when using two gpus, the weight matrix a is split along the columns. gpu 0 stores the left half of a, called a_left, and gpu 1 stores the right half, called a_right. both gpus have the full input x. each gpu computes its part: gpu 0 calculates y0 = x * a_left, and gpu 1 calculates y1 = x * a_right. the result y is split between the two gpus, with each holding half.

next, the activation function gelu is applied. since gelu works on each number independently, each gpu applies it to its own part of y. gpu 0 gets y0’ = gelu(y0), and gpu 1 gets y1’ = gelu(y1). no communication is needed, and the output remains split.

for layer b, the math is z = y’ * b. here, y’ is split, with each gpu holding half. the weight matrix b has size [4h, h], and is split along the rows. gpu 0 stores the top half, b_top, and gpu 1 stores the bottom half, b_bottom. each gpu computes a partial result: gpu 0 calculates z_partial_0 = y0’ * b_top, and gpu 1 calculates z_partial_1 = y1’ * b_bottom. both partial results have size [1, h].

to get the final output z, the two partial results must be added together. this is done using all-reduce (sum). both gpus send their partial result to each other, add them up, and now both have the complete output z. after all-reduce, both gpus hold the same final tensor z

This is why Tensor Parallelism is strictly restricted to NVLink (Intra-Node). why? because it is of blocking nature, in the fsdp all-geather we can perform all geather in the background while the computation is going on

how will this work with fsdp weight sharing?

now lets see how we can combine tensor parallism and fsdp

the hardware is set up as a 2d grid: 4 nodes, each with 8 gpus. inside each node, gpus are connected by nvlink (intra-node). nodes are connected by ethernet or infiniband (inter-node). tensor parallelism (tp) runs horizontally (inside a node), and fsdp (data parallelism) runs vertically (across nodes).​

a weight matrix w of size is split in two steps. first, tensor parallelism splits w into 8 vertical columns, called wtp0 to wtp7. in pure tp, each gpu in a node would hold one full column. but with fsdp, each column is further split across the 4 nodes. for example, wtp0 is split into 4 pieces: node 1 gpu 0 holds chunk 1, node 2 gpu 0 holds chunk 2, and so on. each gpu ends up with 1/32 of the total weights.​

for the forward pass, each gpu must compute its part of the matrix multiply. say node 1 gpu 0 wants to compute ylocal = x * wtp0. but it only has one chunk of wtp0. so, it does an all-gather across the vertical (inter-node) axis: node 1 gpu 0 talks to node 2 gpu 0, node 3 gpu 0, and node 4 gpu 0 to get all chunks of wtp0. now it has the full wtp0 in memory.​

next, it computes ylocal = x * wtp0 locally. after this, it must combine results with other gpus in its node. each gpu in node 1 does the same for its own column. then, they do an all-reduce horizontally (intra-node) over nvlink, summing their partial results. now, every gpu in node 1 has the full output z

Sequence Parallelism

The Problem

each gpu holds a full copy of the activation tensor after the row-parallel linear layer, even though only part of the computation is unique before all-reduce.

with sequence length 1,000,000, hidden size 8,192, and bf16, the tensor shape is and takes about 16 gb per gpu.

after all-reduce, every gpu has the same final tensor z_total, so all 8 gpus store identical 16 gb copies. total memory used is 128 gb, but only 16 gb is unique data.

for layernorm and dropout, each gpu processes its own copy of z_total. the work is not parallelized; every gpu does the same math and produces the same result.​

this wastes both memory and compute, since all operations are repeated across gpus instead of being distributed or optimized

The Solution

at this stage, every gpu has a partial sum tensor of size 16 gb, matching the full sequence length.​

to avoid storing 16 gb on every gpu, use reduce-scatter instead of all-reduce. reduce-scatter sums the partials and then splits the result along the sequence dimension, giving each gpu only 1/8th of the tokens. each gpu now holds a correct, fully summed shard of 2 gb.​

for layernorm, each gpu processes its own shard of 125,000 tokens. since layernorm acts per token, each gpu can compute its part independently, keeping memory at 2 gb per gpu.​

when the next layer needs the full sequence, call all-gather. each gpu sends its shard to all others, temporarily reconstructing the full 16 gb tensor. after the matrix multiply, discard the full tensor and return to sharded state

fsdp does not shard the activations of layernorm so like how tp+sfdp work together sequence parallelism works alone

Pipeline Parallelism

until now what we have been doing is sharding the weights gradients etc within each layer but a single GPU has to calculate all layers.

in pipeline parallel we divide each layer to a single gpu.

and run micro-batches, accumulate their gradients and update after a full batch has been completed. (doing an reduce scatter at each micro-batch would be very in-efficient)

and when you combine all of these it would become 3d parallel.

lets see it by an example, our setup :-

1. 2 Stages: S0, S1 (to keep it minimal).

2. 3 Micro-batches: mb1, mb2, mb3.

with this you can imagine how all the things we talked about can work together and create a 3d parallel. combining fsdp, tensor parallel, sequence parallel and pipeline parallel

suppose you wanna do C= AB

the grid shape of A if is m x n then of B should be n x p

the grid shape of C would be m x p

and block of C

Ain is a block. think of it like treating blocks as numbers and calculating as we generally calculate numbers in matrix multiplication

very very important :-
The shapes of blocks of A and B must be of the form m x p & p x n. they need not to be identical.

so here is a blog which will teach you all you need to be a wizard of gradients

Rule 1

suppose we want to find the gradient dL/dW where L is the loss and W is the gradient. the rule says that the shape of dL/dW will be equal to the shape of W, memorize this.

x→[YOUR LAYER (f)]→y→[…Rest of Network…]→L

clearly x is the input here, f(x) is the layer about which we care, wow it rhymes

y is the output of the layer, y = f(x) and L is the final loss.

what we are interested in is dL/dX and dL/dF, now F can be a weight matrice or some other function which has learnable parameters to it

Memorize

dL/dX = dL/dY * dY/dX

generally the f is of two types, weight matrices and element wise operations. we are going to look at element wise operations for now

an “element-wise” operation means the math happens to each number in the matrix independently. none of the numbers “talk” to each other.

Examples:

Y = X + Z
(Matrix Addition)

Y = ReLU(X)
(Activation)

Y = X^2
(Square every element)

Rule for Element Wise Operations (memorize this)

you might be wondering what does this circle and a dot between it means

that is the hadamard product (element-wise multiplication). it means: multiply the top-left of a with the top-left of b, top-right with top-right, etc. no fancy row-column dot products here. just simple multiplication.

example 1

Y = ReLU(x) (rule: if x>0, keep it. if x≤0, set to 0)

so dY/dX = 1 if x>0 or 0 if x<0

what does this mean philosophically? if x>0 pass my gradients (y speaking) as it is if not stop make all the gradients 0 ( do not change the weights they did not contribute to the loss)

example 2

Residual Connection Y = X + Z

dY/dX = dY/dZ = 1

so the gradients flow as it is from Y to X and Z

note: we talked about relu above and only calculated dY/dX and not dY/dW because ReLU does not have any parameters to tune, if we were using GeGLU we would have also calculated dy/DW because we also want its parameters to learn.

now we have completed activation functions lets move on to the matrice multiplications

Y = XW

we want to learn two things dL/dX and dL/dW

so here are the formulas, please memorize

you are now all set for calculating any gradient in transformers

but lets cover one special hard case, that is gradient of loss wrt to the Logit layer

Softmax

z = logits
(shape: b, t, v where v is vocab size).

p = softmax(z)
(probabilities).

l = -log(p_correct_token).

deriving softmax is messy (lots of jacobian matrices), but the final result is elegantly simple

dL/dZ = P - Y(one hot)

and from here it is all matrice multiplications and activation functions which we have already covered. so no worries.

throughput means how many operations can we do per second and latency means how much time does one operation take. cpu optimizes latency, while gpu’s optimizes throughput

two primary components on the chip are ALU ( unit of computation ) and Control/Caches.

cpu prioritizes control on chip to decrease latency while gpu prioritizes ALU’s to increase throughput

The NVIDIA A100 GPU has 6,912 FP32 ALUs

Because the GPU removed the complex caches and control logic, individual GPU threads are actually slower (higher latency) than CPU threads. If a GPU thread needs to fetch data from memory, it waits a long time (relative to the clock speed). but it covers it via parallism

physical structure of a GPU

a gpu is divided in many SM’s (Streaming Multiprocessors)

Each SM is a self-contained processor. An SM has its own instruction scheduler, its own registers (fast local memory), and its own shared memory (cache)

Because SMs are independent, hardware scaling is straightforward. If NVIDIA wants to make a faster GPU, they print more SMs on the silicon.

• NVIDIA A100: Has 108 SMs.

• NVIDIA H100: Has 132 SMs.

Inside every SM, there are the actual execution units that perform the math. We call these ALUs (Arithmetic Logic Units) or “Cores.”

In modern NVIDIA architectures (like Ampere, used in the A100), an SM contains several types of specialized cores:

• FP64 cores: 32 per SM; used for scientific double-precision work

• FP32 cores: 64 per SM, 108 SMs → 6912 total; used for normal 32-bit math

• Tensor Cores: 4 per SM; do fast 4×4 matrix multiply-accumulate; give major ML speedup

• SFUs: handle sin/cos/exp/log; slower and fewer than FP32 cores

• Load/Store units: handle memory read/write address calculation

Chapter 1.3: The Memory Hierarchy (Physical)

• global memory: huge, high-latency, high-bandwidth storage; where all tensors initially sit; slowest level but largest

• l2 cache: on-die, medium size, shared by all sms; faster than global; auto-caches data so repeated accesses don’t go back to hbm

• shared memory: tiny, per-sm, extremely fast; manually controlled; used for blocking/tiling so threads can reuse data cheaply

• registers: per-thread storage for immediate operands; fastest; extremely limited and heavily partitioned across threads

tensor cores only reach their 312-tfops peak when fed from shared memory because global memory (~1.5 tb/s) is too slow, so the fix is to fetch each tile once from global, stash it in shared memory (~19 tb/s), and reuse it repeatedly

Virtual Mapping

cuda gives you a virtual grid→block→thread model while the hardware dynamically maps the many launched blocks onto whatever sms are free, so the same kernel runs unchanged on gpus with 80, 108, or 132 sms

more sms just means more blocks execute simultaneously, increasing throughput with zero code changes

a gpu kernel is a function definition, but execution spawns a massive number of parallel instances

thread: smallest execution unit; runs kernel code independently; owns its registers and instruction pointer; roughly maps to an alu

block: group of threads that can share data via shared memory and synchronize (wait for each other to complete); limited in size (how many threads a block can have) because an sm has limited shared memory and registers; scheduler assigns one block at a time to an sm. (this means at a time only one block can run on a sm, but more than one block can be assigned)

grid: all blocks of a kernel launch; blocks are fully independent and never communicate; scheduler spreads them over all sms to utilize the whole gpu

look first we will look at some facts

when you write a cuda kernel, you define how many blocks your grid will have.

and you define how many threads each block will have. (these two are totally in your hands)

but how do you properly choose these two hyper params? we will go into it later in this blog post

fact #1 —> there are limits on threads per block (commonly 1024) and threads per sm (commonly 2048–4096), because each sm has finite registers, shared memory, and scheduling slots; exceeding these resources prevents additional threads from being issued

now lets see how many blocks can be alloted to a particular sm and how do we calculate that

• sm has hard caps: 2048 threads, 192 kb shared memory, 65536 registers

• each block consumes some portion of each resource; the number of blocks that fit is limited by whichever resource runs out first

• scenario a (thread-limited): 256-thread block, no shared memory → 2048/256 = 8 blocks can fit; high occupancy. so when one block is waiting for memory the sm can run another block

• scenario b (shared-memory-limited, 192kb only available): 256-thread block, 64 kb shared memory → 192/64 = 3 blocks fit; thread capacity unused because shared memory is the bottleneck

• scenario c (register-limited, 65280 registers): 256 threads × 255 registers = 65280 registers → only one block fits; low occupancy and poor latency hiding

Cordinate System

You launch a grid of 100 blocks, each with 256 threads. Total threads: 100×256=25,600. All threads run the same kernel code, so each thread must compute a unique global index to know which element (pixel, array entry, etc.) it should process.

CUDA provides built-in variables inside a kernel (__global__ void MyKernel(...)):

• threadIdx.x: Thread index within the block, 0…blockDim.x−10…blockDim.x−1.

• blockIdx.x: Block index within the grid, 0…gridDim.x−10…gridDim.x−1.

• blockDim.x: Number of threads per block (constant for that launch).

Global index

Standard 1D mapping:

int idx = blockIdx.x * blockDim.x + threadIdx.x;


Examples with blockDim.x = 256:

• Block 0, Thread 0: idx = 0 * 256 + 0 = 0

• Block 0, Thread 255:idx = 0 * 256 + 255 = 255

• Block 1, Thread 0: idx = 1 * 256 + 0 = 256

• Block 1, Thread 5: idx = 1 * 256 + 5 = 261

Each thread computes this once and then operates on data[idx].

Handling non-multiple sizes

For N=1000 elements and blockDim.x = 256, you need 4 blocks:

• 4×256=1024 threads.

• Valid indices: 0 to 999.

• Extra threads: 1000 to 1023 (must do nothing).

Standard pattern:

__global__ void MyKernel(float* data, int N) {
int idx = blockIdx.x * blockDim.x + threadIdx.x;
if (idx < N) {
data[idx] *= 2.0f;
}
}


Threads with idx >= N exit immediately, so the kernel is safe for any NN.

suppose we have defined blocks = 100, and we have a gpu with only 10 sm and each sm can only take 2 blocks. so how will our code run? our code will first run 20 blocks on those sm’s and add the remaining 80 to the queue. then again run 20 from those 80 and so on.

The Memory Bottleneck

In normal computer science, we talk a lot about Big O notation: O(N), O(N2), and so on.
Big O tells us how the number of operations grows as the input size grows.

On GPUs, this is not enough.

For GPU speed, the main limit is often not how many math operations you do.
The main limit is how many bytes you move to and from memory.

So instead of only counting operations, you must also count bytes moved.

3.1 Arithmetic Intensity

The key metric for GPU performance is called Arithmetic Intensity.

This number tells you:

For each byte read or written from global memory, how many floating point operations do you do?

Higher intensity means you do more math per byte of data moved.
Lower intensity means you move a lot of data but do very little math with it.

Example 1: Element-wise Addition

Take a simple vector addition:

C[i]=A[i]+B[i]C[i]=A[i]+B[i]

For each element ii:

• Math:

  • 1 addition ⇒⇒ 1 FLOP.

• Memory:

  • Read A[i: 4 bytes (assuming 32-bit float).

  • Read B[i]: 4 bytes.

  • Write C[i]: 4 bytes.

• Total data per element:

  • 4+4+4=12 bytes.

So the arithmetic intensity is:

Intensity=1 FLOP/12 Bytes≈0.083 FLOPs per byte

This is a very low number.
You do very little math for each byte you move.

Example 2: Matrix Multiplication (N×N)(N×N)

Now look at matrix multiplication of two N×NN×N matrices.

• Math:

  • Roughly 2N^3 FLOPs.

• Memory:

  • Read matrix A: N^2 elements.

  • Read matrix B: N^2 elements.

  • Write matrix C: N^2 elements.

  • Total elements: 3N^2.

  • If each element is 4 bytes: 3N^2×4 bytes.

Arithmetic intensity:

So intensity grows linearly with N.

This is a very high intensity.
You do a lot of math for each byte you move.

Why this matters on a real GPU (A100)

Take an NVIDIA A100 GPU:

• Compute speed: about 312,000 GFLOPs (tensor operations).

• Memory bandwidth: about 1,555 GB/s.

The hardware ratio is:

312,000/1,555 ≈ 200

This means:

To fully use the compute units, your kernel should do about 200 FLOPs for every 1 byte of global memory traffic.

This is the golden rule for this GPU.

Now compare our two examples:

• Vector Add (intensity ≈0.083):

  • Much lower than 200.

  • The kernel will wait on memory most of the time.

  • The compute units will sit idle.

  • This kernel is memory bound.

• Matrix Multiply with N=2048 (intensity ≈341):

  • Higher than 200.

  • Memory bandwidth is enough to feed the compute units.

  • The compute units can run at or near full speed.

  • This kernel is compute bound.

So, when you design GPU kernels for deep learning, you must ask:

• How many FLOPs does this kernel do?

• How many bytes from global memory does it read and write?

• What is the arithmetic intensity, and how does it compare to the GPU’s hardware ratio?

If your intensity is too low, the GPU will be limited by memory bandwidth, not compute.
If your intensity is high enough, you can use the full compute power of the GPU.

note B (bytes/second) is fixed

there is a very interesting graph which we can analyze here

on x-axis here is the matrice size, now remember that Intensity for matrice multiplication are proportional to size of matrice right

so first you see a linear line like curve and later it flatens out because the peak performance of gpu hits.

there is a lot more to this curve which we will be covering in further concepts so lets move on !

Machine learning is full of matrix multiplication.
This is not because neural networks “prefer” matrices in some deep way.

The real reason is simple:

Matrix multiplication is one of the few operations that can beat the memory bottleneck on GPUs.

Element-wise operation (ReLU, addition, etc.)

Task: Take a vector of length N.
For each element: read it, change it, write it back.

• Math operations:

  • You touch each element once.

  • Total math ≈N.

• Memory accesses:

  • Read each element: N.

  • Write each element: N.

  • Total ≈2N

• Intensity:

  Intensity=0.5

• Scaling:

  • If you make the vector 1000 times bigger, the intensity remains constant

So the intensity stays constant as N grows.
You never reach a point where you do “a lot” of math per memory access.

Global Memory Optimization

In C++ code, memory access looks simple:

float x = data[i];


It feels like the GPU goes to memory, reads that one float (4 bytes), and comes back.

This is not what really happens.

Global memory (DRAM) is not designed to read single bytes or single floats efficiently.

It is designed to read data in fixed-size chunks.

The physical wires between the GPU and the memory chips are called the memory bus.

Data moves over this bus in fixed-size units called transactions (also known as cache lines).

• On modern NVIDIA GPUs:

  • Minimum transaction size is usually 32 bytes.

  • For high-throughput loads, the effective unit is often 128 bytes.

So if your code needs 4 bytes, the hardware does this:

• The memory controller reads a 32-byte (or 128-byte) block from DRAM.

• It sends that block over the memory bus.

• You only use 4 bytes from that block.

• The rest of the bytes are unused for that instruction.

You still pay:

• The latency cost for the full transaction size.

• The bandwidth cost for the full transaction size.

If you only needed 4 bytes from a 32-byte transaction, you used:

4/32=12.5% of the data

This is low efficiency.

GPUs are designed around warps: groups of 32 threads that run together.

The memory controller expects that threads in the same warp will access nearby addresses.

If thread 0 reads address X, the hardware expects thread 1 to read X+4, thread 2 to read +8, and so on.

If this happens, the hardware can bundle these requests into a small number of large transactions.

Warp memory requests

When a warp runs a load instruction (for example, a global load), the hardware does the following:

• It looks at the memory addresses requested by all 32 active threads in the warp.

• It finds how many 128-byte transactions are needed to cover all these addresses.

• It issues that many transactions to DRAM.

The number of transactions depends on how the threads access memory.

Scenario A: Sequential (coalesced) access – ideal

• Thread 0 requests address 0

• Thread 1 requests address 4

• Thread 2 requests address 8

• ...

• Thread 31 requests address 124

Here:

• The addresses go from byte 0 to byte 127.

• This is exactly 128 bytes of contiguous memory.

The memory controller sees that all 32 requests fall inside one 128-byte block.

• It issues a single 128-byte transaction.

• All 32 loads are served by this one transaction.

Result:

• The bus moves 128 useful bytes.

• All 128 bytes are used by the program.

Efficiency: 100%.

This is the best case.

Scenario B: Strided or scattered access – worst case

• Thread 0 requests address 0

• Thread 1 requests address 1000

• Thread 2 requests address 2000

• ...

Now the addresses are far apart.

• They do not fit into one 128-byte block.

• They are in different regions of memory.

The memory controller cannot group these into one transaction.

So it must:

• Issue 1 transaction for thread 0’s address.

• Issue 1 transaction for thread 1’s address.

• ...

• Issue 1 transaction for each of the 32 threads.

Total:

• 32 separate transactions.

• Each transaction transfers at least 32 bytes.

So:

• Total data moved on the bus: 32×32=1024 bytes.

• Useful data actually used by your code: 32×4=128 bytes.

Efficiency:

Efficiency=128/1024=12.5%

That means you get only a small fraction of the possible bandwidth.

The rest is wasted on unused bytes.

Example :- reading elements from a matrice

• matrix is 32×32 stored in row-major, memory is linear row by row

• row 0 has 32 floats, then row 1 has 32 floats, and so on

• case 1: reading a row with 32 threads

• each thread reads one element from the same row

• addresses increase by 4 bytes each time

• all 32 addresses fall inside one continuous 128-byte range

• hardware combines them into one memory transaction

• result: highest possible memory efficiency

• case 2: reading a column with 32 threads

• each thread reads one element from the same column

• element in next row is 128 bytes ahead because each row is 32 floats

• addresses are 0, 128, 256, … for each thread

• addresses are far apart and not in one continuous segment

• hardware cannot merge them

• result: 32 separate memory transactions and much lower efficiency

but what happens if i read half elements from one row and other half from another, but the elements are continuous

• gpu checks only the final memory addresses requested by the warp, not which row they belong to

• if the 32 threads read addresses that form one continuous 128-byte range, it counts as coalesced

• it does not matter whether those addresses are from one row or split across two rows

• if the boundary between rows happens to align such that addresses are still continuous in memory, gpu still issues one transaction

Memory Alignment

We established that the GPU reads memory in 128-byte transactions. However, these transactions are not arbitrary windows. They are aligned to specific boundaries.

The Grid:
Memory is like a grid of 128-byte slots.

• Slot 0: Addresses 0 to 127.

• Slot 1: Addresses 128 to 255.

• Slot 2: Addresses 256 to 383.

The GPU can fetch “Slot 0” or “Slot 1”. It generally cannot fetch “The 128 bytes starting at Address 4.”

The Problem: Misaligned Data

Imagine you have a perfectly coalesced access pattern (threads reading sequential addresses 0-127).

• Case A (Aligned): The array starts at Address 0.

  • The request covers 0 to 127.

  • This falls exactly into Slot 0.

  • Cost: 1 Transaction.

• Case B (Misaligned): The array starts at Address 4 (maybe you offset the pointer).

  • Thread 0 reads Address 4.

  • Thread 31 reads Address 131.

  • The Request: Bytes 4 to 131.

  • The Mismatch: This range crosses the boundary. It touches the end of Slot 0 (4-127) and the beginning of Slot 1 (128-131).

  • Hardware Action: The Memory Controller must fetch Slot 0 AND Slot 1.

  • Cost: 2 Transactions.

On-Chip Memory and Tiling

We established in Chapter 3 that Matrix Multiplication is Compute Bound (Intensity ~ N/6 ).

However, that math assumes you only read each number from Global Memory once.

If you write a naive kernel where every thread reads its own data from Global Memory over and over again, you are Memory Bound.

let us prove this mathematically

We are multiplying two N×N matrices.

We want to calculate one single element of the output matrix: C[row][col].

The formula:

C[row][col] = sum from k = 0 to N-1 of (A[row][k] * B[k][col])

So we must calculate the dot product of row i of A and column j of B.

Scenario 1: The Naive Kernel (No Shared Memory)

In this kernel, every thread is responsible for calculating one element of C.

The kernel logic (pseudo):

• Thread calculates C[y][x]

• Initialize sum = 0.0

• For k from 0 to N-1:

  • Read A[y][k] from global memory

  • Read B[k][x] from global memory

  • Compute sum += A[y][k] * B[k][x]

• Write C[y][x] = sum

Math analysis (for the whole grid):

Total threads:
N×N threads (one per output element).

Operations per thread:

• The loop runs N times.

• Each iteration does 1 multiply + 1 add = 2 FLOPs.

Total FLOPs:

• Per thread: 2N

• All threads: (N×N threads) × (2N FLOPs per thread) = 2N^3 FLOPs.

• This matches the theoretical complexity of matrix multiplication.

Memory accesses per thread:

• The loop runs N times.

• In each iteration, we read A (4 bytes) and B (4 bytes) from global memory.

Reads per thread:
2N reads.

Total global reads (all threads):
(N×N threads) × (2N reads) = 2N^3 reads.

Bytes transferred:

• Each read is 4 bytes.

• Total bytes = 2N^3 × 4 bytes = 8N^3 bytes.

Arithmetic intensity:
Intensity = (total FLOPs) / (total bytes)
= (2N^3 FLOPs) / (8N^3 bytes)
= 0.25 FLOPs per byte.

Verdict:

• Arithmetic intensity is 0.25 ops/byte.

• This is very low.

• An NVIDIA A100 GPU needs about 200 ops/byte to reach peak compute performance.

• So this naive kernel will run at around 0.1% of the GPU’s peak performance.

so this does not scale with size and we are in a big problem

Shared memory comes to save us.
It is Programmable. Unlike a CPU’s L1/L2 cache, which handles itself automatically, Shared Memory does nothing unless you write code to load data into it.

Variable Scope
When you declare a variable in CUDA, the keyword determines where it lives:

1. float x; (Local Variable)

   • Memory: Register.

   • Scope: Private to one thread. Thread A cannot see Thread B’s x.

2. __shared__ float tile[256]; (Shared Variable)

   • Memory: Shared Memory (SRAM).

   • Scope: Visible to all threads in the same Block.

   • Lifetime: Exists as long as the Block is running. Once the Block finishes, this memory is wiped.

   • Isolation: Block 0 cannot see Block 1’s shared memory.

The Danger: Race Conditions
Since all 256 threads in a block see the same tile array, they can crash into each other.

• Thread 0 tries to write to tile[0].

• Thread 1 tries to read from tile[0] at the same time.

• Result: Undefined behavior. Garbage data.

The Solution: __syncthreads()
This is the most important function in Shared Memory programming.

• Action: It creates a barrier.

• Rule: “No thread in the block is allowed to proceed past this line of code until ALL threads in the block have reached this line.”

The General Pattern

Tiling

The goal

We want to calculate C = A × B.
The output matrix C has size N×N.
We divide C into small square blocks of size TILE_SIZE × TILE_SIZE.
We choose TILE_SIZE = 32.

We focus on one block of C, call it Block_C.
This block covers rows 0…31 and columns 0…31 of the output.
One CUDA thread block (with 32×32 = 1024 threads) is assigned to this Block_C.
The goal of this thread block is to compute all 32×32 elements of Block_C.

Step 2: The data dependency

To calculate Block_C (rows 0–31, cols 0–31), we need the following data:

From matrix A:

• We need rows 0 to 31.

• For each of these rows, we need all columns from 0 to N−1.

• So we need a strip of size 32×N from A.

From matrix B:

• We need columns 0 to 31.

• For each of these columns, we need all rows from 0 to N−1.

• So we need a strip of size N×32 from B.

Visualization example

Assume A and B are 2048×2048.
To compute our small 32×32 output block, we need:

• From A: a horizontal strip of size 32×2048.

• From B: a vertical strip of size 2048×32.

Shared memory limitation

We cannot fit these large strips into shared memory.
For A’s strip: 32×2048 floats = 65536 floats.
Each float is 4 bytes, so size = 65536 × 4 bytes = 262144 bytes ≈ 256 KB.
Typical shared memory per SM is about 48 KB on many GPUs (up to about 100 KB on A100).
Therefore, a full 32×2048 strip does not fit into shared memory.

insight

lets break the matrix multiply even more

this above can be written like this below.

taking the assumption that N is a multiple of 32, we will relax it soon

Our goal: a thread block must compute a 32×32 tile of C.
Inside this tile, the local row index i goes from 0 to 31 and the local column index j goes from 0 to 31.

We now trace a single thread: the one responsible for C within this tile.
For this thread, i = 0 and j = 0.

Its job is to calculate:

The outer loop (the phases ph)

The for loop in the tiled kernel iterates over the outer sum

Let us step through it.

Phase 0 (ph = 0)

In this phase, the thread needs to compute the first partial sum:

PartialSum0=

Simplified:

Data needed in phase 0

For this computation, thread (0,0) needs row 0 of A from columns 0 to 31 and column 0 of B from rows 0 to 31.
Thread (0,1) needs row 0 of A and column 1 of B.
Thread (1,0) needs row 1 of A and column 0 of B.
In fact, all threads in the thread block need access to the same 32×32 block of A (rows 0–31, cols 0–31) and the same 32×32 block of B (rows 0–31, cols 0–31).

Collaborative load for phase 0

Instead of each thread loading all the data it individually needs, the threads cooperate to load these tiles into shared memory:

• Each thread loads one element from A and one element from B into shared memory.

• Thread (0,0) loads A[0][0] into As[0][0] and B[0][0] into Bs[0][0].
  Thread (5,10) loads A[5][10] into As[5][10] and B[5][10] into Bs[5][10].

• This continues similarly for all 1024 threads in the 32×32 block.

A barrier (such as __syncthreads()) is then used so that all threads wait until the entire tile has been loaded into shared memory.
Now the entire 32×32 tiles of A and B needed for this phase are cached in shared memory.

Computation of the inner sum in phase 0

Now thread (0,0) runs its inner summation ∑klocal=0 to 31 using As and Bs in shared memory, which is fast.
Using these tiles, it computes PartialSum_0 and adds it to its running result for C.

Phase 1 (ph = 1)

In the next phase, the thread must compute the second partial sum:

PartialSum1=

Simplified:

Data needed in phase 1

Now thread (0,0) needs A’s data from row 0, columns 32 to 63, and B’s data from column 0, rows 32 to 63.
Again, every thread in the thread block needs this new 32×32 chunk of A and B, but with the appropriate offset for its own row and column.

Collaborative load for phase 1

The threads repeat the collaborative loading process, but now with an offset in the global indices:

• Thread (0,0) loads A[0][32] into As[0][0] and B[32][0] into Bs[0][0].
  Thread (5,10) loads A[5][42] into As[5][10] and B[42][10] into Bs[5][10].

• All threads in the block similarly load one element of the current tile of A and one element of the current tile of B.

Another synchronization (__syncthreads()) ensures all data for this new phase is present in shared memory before computation proceeds.

Computation of the inner sum in phase 1

Thread (0,0) again runs its inner loop over klocal=0…31, this time using the newly loaded tiles in As and Bs.
It computes PartialSum_1 and adds it to its running total for C.
After all phases ph from 0 to (N/32)−1(N/32)−1 are processed in this way, the thread has accumulated the full value of C.

here is the code implementation

lets see the efficiency

how did we derived these values for the last tiled kernel part? it’s your homework

also you might be thinking this does not scale with N so it is not efficient. there is another optimization which makes that possible but we are not going over that

Wraps

So far, we have been using a convenient simplification: we imagined that 1024 threads in a block all run independently and in parallel.

Now, we will look at how the SM actually executes these threads. This will explain many of the strange performance artifacts you see in real-world GPU code.

The hardware scheduler inside the SM does not see 1024 individual threads. It sees groups of 32.

A Warp is a group of 32 consecutively numbered threads that execute in lockstep.

• In a block of 256 threads:

  • Threads with threadIdx.x from 0 to 31 form Warp 0.

  • Threads with threadIdx.x from 32 to 63 form Warp 1.

  • ...and so on.

  • A block of 256 threads is composed of 8 Warps (256 / 32).

The number 32 is a hardware design choice by NVIDIA, and it is fundamental to the architecture.

1. The SM’s instruction scheduler fetches one instruction from memory (e.g., ADD R1, R2, R3).

2. It issues this single instruction to a Warp.

3. All 32 threads in that Warp execute that exact same instruction at the exact same time.

They operate on different data because each thread has its own private registers (R1 for Thread 0 is different from R1 for Thread 1), but the operation itself is identical for all of them.

now you must be thinking the code we wrote in the tiling part, had both + and multiplication in a single thread. actually some operations like AB + C, has been made a single operation by gpu cores.

Control Divergence

We’ve established the core rule of SIMT: All 32 threads in a Warp must execute the same instruction at the same time.

The Problem: What happens when the code has an if/else statement?

The Setup:
Let’s analyze a simple kernel with the parameters you suggested.

• Block Size: 256 threads.

• This means: 8 Warps (Warp 0 to Warp 7).

consider this cod

Let’s trace what the hardware does, Warp by Warp.

Trace 1: Warp 1 through Warp 7 (Threads 32 to 255)

• Consider any thread in these Warps, for example, Thread 40.

• Its tid is 40.

• The condition 40 < 16 is false.

• Every single thread in Warp 1, 2, 3, 4, 5, 6, and 7 evaluates the condition to false.

• Result: They all agree. The hardware simply skips Path A and executes the instructions for Path B.

• Performance: Perfect. No time is wasted. This is Uniform Control Flow.

Trace 2: Warp 0 (Threads 0 to 31)
This is where the problem occurs.

• Threads 0 to 15:

  • Their tid is less than 16. The condition is true.

  • They want to take Path A.

• Threads 16 to 31:

  • Their tid is 16 or greater. The condition is false.

  • They want to take Path B.

The Warp has a disagreement. The hardware cannot execute Path A and Path B at the same time. It must serialize them.

The Hardware’s Solution (Predicate Registers):

1. Evaluate: All 32 threads evaluate the condition tid < 16.

2. Masking: A “mask” is created for the Warp.

   • Threads 0-15 are marked as “active.”

   • Threads 16-31 are marked as “inactive.”

3. Execute Path A: The hardware executes the instruction data[tid] = 100.0f;.

   • Only the “active” threads (0-15) are allowed to actually write to memory.

   • The “inactive” threads (16-31) do nothing. They are stalled. They waste this clock cycle.

4. Invert Mask: The hardware flips the mask.

   • Threads 0-15 are now “inactive.”

   • Threads 16-31 are now “active.”

5. Execute Path B: The hardware executes the instruction data[tid] = 200.0f;.

   • Only the “active” threads (16-31) write to memory.

   • Threads 0-15 are now stalled.

The Cost:

• A uniform Warp would have taken 1 cycle (executing either Path A or Path B).

• Our diverged Warp took 2 cycles (executing Path A, then Path B).

• Performance on Warp 0 was cut in half.

This phenomenon is called Control Divergence.

Divergence only happens when threads within the same Warp disagree on which path to take.

Divergence between different Warps is fine (Warp 0 can take Path A while Warp 1 takes Path B).

The performance penalty is proportional to the number of divergent paths. An if/else costs 2x. A switch statement with 4 paths could cost 4x.

Rule: To write fast code, try to structure your data and logic so that threads in a Warp agree on control flow. For example, if you are processing different types of particles, sort them by type first so that Warp 0 only processes “Type A” particles and Warp 1 only processes “Type B”.

Wave Quantization

The Setup

Let’s use a concrete example.

• GPU: An NVIDIA A100 with 108 SMs.

• Kernel: A Tiled Matrix Multiplication.

• Workload: We are calculating a large output matrix. This work is divided into many independent Thread Blocks.

• Key Fact: Each Thread Block is a single “unit of work” that must be scheduled to an SM.

The GPU can process work in waves. In a perfect world, it processes 108 Blocks in the first wave, then the next 108 Blocks, and so on.

The “Good” Case: Perfect Divisibility

Imagine our matrix dimensions are such that we need to launch exactly 216 Thread Blocks.

• Wave 1: The GPU scheduler assigns Blocks 0-107 to the 108 SMs. All SMs are 100% busy.

• Wave 2: As soon as the first wave finishes, the scheduler assigns Blocks 108-215 to the 108 SMs. All SMs are 100% busy again.

• Result: The GPU is fully occupied for the entire duration. Performance is at its peak for that algorithm.

The “Bad” Case: The Tail Effect

Now, let’s say we change the matrix dimension by just one pixel.
This tiny change causes the number of required Thread Blocks to become 217.

• Wave 1: The scheduler assigns Blocks 0-107. All 108 SMs are busy.

• Wave 2: The scheduler assigns Blocks 108-215. All 108 SMs are busy.

• Wave 3 (The “Tail Wave”):

  • The scheduler has only one Block left to run (Block #216).

  • It assigns this block to SM 0.

  • What are the other 107 SMs doing? Nothing. They are completely idle.

  • The entire GPU, a multi-thousand-dollar accelerator, is sitting and waiting for one SM to finish this single, final block of work.

The Performance Impact:
The total time taken is the time for Wave 1 + Wave 2 + Wave 3.
Even though Wave 3 has almost no work, it still takes the same amount of time as a full wave.

• Time for 216 Blocks: ~2 units.

• Time for 217 Blocks: ~3 units.

We added 1/216 (~0.5%) more work, but the runtime increased by 1/2 (50%).
This causes a massive, sudden drop in achieved TFLOP/s.

Reductions

A reduction turns many input elements into one output value using an associative binary operator like +, max, or min. For summation, this means combining all elements of an array into a single scalar.

Sequential vs parallel

A naive sum on the CPU uses a single loop:

Initialize sum = 0.

For i from 0 to N−1, do sum = sum + data[i].

Each iteration depends on the previous one because the new sum uses the old sum. This makes the loop inherently sequential, so a single GPU thread doing this loop would not exploit parallelism.

Tree reduction idea

Addition is associative, so the grouping of terms does not change the result:

• Sequential: (((a0 + a1) + a2) + a3) + ...

• Tree: (a0 + a1) and (a2 + a3) can be computed in parallel, then their partial sums added, and so on.

With 8 elements and 4 threads:

• Step 1 (stride = 4): 4 threads each add pairs 4 apart, producing 4 partial sums in the first half of the array.

• Step 2 (stride = 2): 2 threads add those partial sums, producing 2 values.

• Step 3 (stride = 1): 1 thread adds the last two values, leaving the final result in element 0.

The number of active threads halves every step, giving a tree of depth log⁡2(N)log2(N) instead of doing all N additions in a chain.​

CUDA shared-memory implementation

Within a block, the algorithm is:

Load: All threads cooperatively copy one element each from global memory into a shared-memory array sdata.

Sync: Call __syncthreads() so every thread sees all loaded values.

Iterative Reduction: We loop, cutting the number of active threads in half each time.

Final Write: After the loop, sdata[0] holds the sum for the entire block. Thread 0 is responsible for writing this partial sum back to a global results array.

what if the number of elements we want to add are greater than the number of threads available to us?
suppose we want to sum 200 million numbers

Instead of each thread processing one element, each thread processes many elements.

1. Launch a Fixed Grid: We launch enough blocks to saturate the GPU (e.g., 1000 blocks of 256 threads = 256,000 threads total).

2. Looping: These 256,000 threads act like a swarm of “pac-men.” They eat the first 256,000 numbers. Then they all jump forward by the grid size and eat the next 256,000 numbers. They repeat this until the 200 million numbers are consumed.

3. Local Accumulation: While looping, each thread maintains a private running total in a register.

lets calculate intensity for this operation

Reads: We read N floats to populate sdata.
Bytes Read = 4 * N.
Writes: We write one single float (the final sum) back.
Bytes Written = 4.
Total Global Bytes: 4N + 4. For large N, this is approximately 4N

The reduction performs N-1 additions to sum N numbers.
For example, to sum 8 numbers, you do 4 adds, then 2 adds, then 1 add. Total = 7 adds.
Total FLOPs: N - 1. For large N, this is approximately N.

I = N/4N = 1/4

Atomic Operations

If you try to do parallel reduction by having all threads add to a single shared counter (like global_sum), you can run into a race condition. Here’s what happens:

• You have a memory location (global_sum), initialized to 0.

• Thread A reads its value (for example, data is 5).

• Thread B reads its value (for example, data is 10).

• Both threads try to add their own value to global_sum at the same time.

The problem:

• Thread A reads global_sum (0).

• Thread B reads global_sum (0).

• Thread A calculates 0 + 5 = 5.

• Thread B calculates 0 + 10 = 10.

• Thread A writes 5 to global_sum.

• Thread B writes 10 to global_sum.

• Final result: global_sum is 10 (not 15). Thread A’s addition was overwritten.

Why did this happen?

• Both threads read the same starting value before either finished updating global_sum.

• They “race” each other—whichever thread writes last “wins,” and the other’s update is lost.

The Solution: Atomic Operations
Hardware provides “atomic” instructions, which guarantee the entire sequence (read, modify, write) is performed without interruption.

In CUDA, the function is atomicAdd(&address, value).

What happens when you use atomicAdd:

• The hardware locks the memory location so no other thread can read or write until it’s done.

• It reads the current value.

• It adds the new value.

• It writes the new value back.

• It unlocks the location.

If another thread tries to do atomicAdd at the same address while it’s locked, it must wait.

A Simple Kernel using Atomics:
Each thread just does:

if (i < N) {
atomicAdd(global_sum, data[i]);
}

Is it correct?
Yes—the result will be correct.

What’s the problem?
It’s very slow if many threads do atomicAdd on the same memory address at the same time:

• Thousands of threads try to add to global_sum at once.

• Only one can succeed at a time; the others wait.

• This serializes the computation—the GPU is idle most of the time, waiting for each atomic operation to finish.

• As a result, parallelism is destroyed and performance drops.

When to use atomics:

• They’re great if collisions are rare—for example, histogram updates where most threads update different bins.

• They’re useful for debugging or for writing a very simple, baseline-correct algorithm.

• For high performance with frequent updates to the same memory address, you should use methods like tree reduction with shared memory.

Softmax

In practice, the values e^xi can become enormous, leading to floating point overflow (infinity). To prevent this, We first find the maximum value in the vector, let’s call it m=max⁡(x^j)

The formula becomes:

Kernel 1 (Find Max):
Read: The entire row from Global Memory (N elements).
Work: Perform a tree reduction to find the max.
Write: The single max value back to Global Memory.
Memory Traffic: ~4N bytes

Kernel 2 (Exponentiate):
Read: The entire row again, and the max value (N+1 elements).
Work: Compute e(x^i−m) for each element.
Write: The entire exponentiated row to a new temporary buffer in Global Memory.
Memory Traffic: ~8N bytes

Kernel 3 (Sum):
Read: The entire temporary row from Kernel 2 (N elements).
Work: Perform another tree reduction to find the sum.
Write: The single sum value back to Global Memory.
Memory Traffic: ~4N bytes

Kernel 4 (Divide):
Read: The temporary row from Kernel 2 again, and the sum value (N+1 elements).
Work: Perform the final division.
Write: The final result row to the output matrix in Global Memory.
Memory Traffic: ~8N bytes

but this is quite inefficient, think about how can we make this efficient what if we fuse this into a single kernel?

in the first kernel you store the m in shared memory of each sm, then in the same kernel you do not need to fetch xi again, you can just modify them to e^ (xi-m)
then in the same kernel you calculate sum of e^(xi-m) and store in shared memory of each sm. and then modify e^(xi-m) to the softmax

this works pretty fine until N gets too big

A100 Shared Memory: ~164 KB per SM

128k Context Window Size of row: 128,000×4 bytes=512 KB128,000×4 bytes=512 KB

now you might be thinking “but we do not need to do the whole reduction on a single sm”

actually you do need to, else you will have to write to global memory making everything incredibly slow

cool so now how do we do such a large softmax on a single SM?

the answer is Tiling

you break down the whole array of xi into small chunks, and calculate m-local, and

in a single sm. and using these we can reform the original softmax. How? that’s a homework. use some brains

so this completes basics of writing cuda kernels

what is the issue with just Normal (0, 1) initialization? the problem is it causes exploding and vanishing gradients over multiple layers and even while backprop.

we are going to prove how does dividing by sq root of embedding dimension helps Statistically

Let’s consider a single neuron in one layer.

Its output y is a linear combination of its inputs x and weights w.

y = w1*x1 + w2*x2 + ... + wn*xn

Here, n is the number of input dimensions (let’s call it n_in).

We will make three simple assumptions about our inputs and weights.

1. The inputs x are normalized. mean(x) = 0 and variance(x) = 1.

2. The weights w are initialized with mean(w) = 0 and variance(w) = 1. This is the Normal(0, 1) case we want to analyze.

3. Inputs xi and weights wi are all independent of each other.

Our goal is to find the variance of the output y. A stable network should have outputs with a variance close to 1. If the variance keeps increasing layer by layer, the outputs will explode to huge values.

Let’s calculate the variance of y.

Var(y) = Var(w1*x1 + w2*x2 + ... + wn*xn)

A property of variance is that for independent variables, the variance of a sum is the sum of their variances.

Var(y) = Var(w1*x1) + Var(w2*x2) + ... + Var(wn*xn)

Another property of variance for two independent variables A and B with zero mean is: Var(A*B) = Var(A) * Var(B).

Applying this:

Var(y) = (Var(w1)*Var(x1)) + (Var(w2)*Var(x2)) + ... + (Var(wn)*Var(xn))

We assumed Var(wi) = 1 and Var(xi) = 1 for all i.

Var(y) = (1 * 1) + (1 * 1) + ... + (1 * 1)
Var(y) = 1 + 1 + ... + 1 (n_in times)
Var(y) = n_in

This is the problem.

The variance of the output of the layer is equal to the number of input dimensions.

If the input dimension n_in is 512, the variance of the output is 512. The standard deviation is sqrt(512), which is about 22.

To understand what this means, we need to know what standard deviation represents. Standard deviation is a measure of how spread out numbers are from their mean. Our mean is zero. For a normal distribution, about 68% of all values lie within one standard deviation of the mean. About 95% lie within two standard deviations.

Our inputs had a mean of 0 and a standard deviation of 1. So most input values were between -2 and 2.
Our outputs now have a mean of 0 and a standard deviation of 22. So most output values will be between -44 and 44.

The typical magnitude of an output value is now 22 times larger than the typical magnitude of an input value.

Pass this through a few layers. Let’s prove how the variance explodes.

• Layer 1:

  • Input to Layer 1: x_l1. Var(x_l1) = 1.

  • Output of Layer 1: y_l1.

  • As we proved, Var(y_l1) = n_in * Var(w_l1) * Var(x_l1) = 512 * 1 * 1 = 512.

• Layer 2:

  • The input to Layer 2 is the output of Layer 1. So, x_l2 = y_l1.

  • The variance of the input to Layer 2 is Var(x_l2) = Var(y_l1) = 512.

  • The output of Layer 2 is y_l2. We use the same formula for variance.

  • Var(y_l2) = n_in * Var(w_l2) * Var(x_l2)

  • The weights of Layer 2 are also initialized from Normal(0, 1), so Var(w_l2) = 1.

  • Var(y_l2) = 512 * 1 * 512 = 262,144.

The variance of the output from Layer 2 is (n_in)^2. The standard deviation is n_in = 512.

After just two layers, the typical magnitude of the output values is 512 times larger than the original input values. After a third layer, it would be (n_in)^3.

The numbers explode. These large outputs are then passed to an activation function (like tanh or sigmoid). These functions saturate for large inputs (their output is close to 1 or -1).

When the activation function is saturated, its gradient is almost zero. During backpropagation, these zero gradients are multiplied back through the network. The weights do not get updated. The network does not learn.

Now, let’s fix it.

Our goal is to make the output variance Var(y) equal to 1.

We saw that: Var(y) = n_in * Var(w) * Var(x)

Assuming Var(x) = 1, we have: Var(y) = n_in * Var(w)

We want Var(y) = 1.

So, 1 = n_in * Var(w)

This means we must choose our weights w to have a variance of:

Var(w) = 1 / n_in

How do we get a random variable with Var(w) = 1 / n_in?

We start with a variable W_standard from a standard normal distribution Normal(0, 1). It has Var(W_standard) = 1.
If we scale this variable by a constant c, the new variance is Var(c * W_standard) = c^2 * Var(W_standard) = c^2.

We want the new variance to be 1 / n_in.
So, c^2 = 1 / n_in
c = 1 / sqrt(n_in)

This is the solution. We initialize weights not from Normal(0, 1), but by taking a sample from Normal(0, 1) and then dividing it by sqrt(n_in).

This makes the variance of the weights equal to 1 / n_in. Let’s re-calculate the output variance with this new weight initialization.

Var(y) = n_in * Var(w)
Var(y) = n_in * (1 / n_in)
Var(y) = 1

Now, the output of the layer has the same variance as the input. The variance does not explode or vanish from layer to layer. The inputs to activation functions stay in a range where their gradients are non-zero, and the network can learn effectively. This same logic applies to the gradients during the backward pass, keeping them stable as well.

and these bugs are not detectable by the compiler. so you get them on Runtime.

Rust Solves this, Rust gives you these Errors on Compile Time.

The Performance of Rust and C++ are Same.

in today’s article we will be going over various ways in which you can write buggy c++ code and how to prevent that

There are majorly Three Categories of memory bugs

1. use-after-free

2. memory-leak

3. double-free

Use-After-Free

Basically, when you reference a pointer after calling free() on it, it no longer points to a valid memory location, so de-referencing it causes errors.

Memory Leaks

you typically allocate memory using a pointer inside a local scope (usually inside functions), or indirectly by creating an object that contains a pointer as a member. When the scope ends, the pointer itself (being a stack variable) is destroyed, but the memory it was pointing to is not automatically freed. If you don’t call free() or delete before the scope ends, the allocated memory remains reserved, but you have no way to access it anymore — this is a memory leak.

Double free

it’s basically a special case of use-after-free

A double-free bug occurs when you call free() (or delete) on the same pointer more than once. This typically happens when multiple functions assume they “own” the pointer and each one tries to free it.

there are some other interesting ways through which you can run into these memory errors

Rule of Three

if a class needs a user-defined destructor then it also needs a user-defined copy constructor and vice versa

Iterator Invalidation

An iterator is an object that acts like a pointer to an element within a container. For a std::vector, an iterator might be implemented as a simple raw pointer to an element in the vector’s internal, contiguous array.

Certain modifications to a container can force it to reallocate its internal storage. The most common example is adding an element to a std::vector that has reached its capacity.

1. The vector allocates a new, larger block of memory on the heap.

2. It copies (or moves) all existing elements from the old memory block to the new one.

3. It deallocates the old memory block.

At this point, any iterators that were pointing to elements in the old memory block are now invalidated. They have become dangling pointers. Dereferencing them is a use-after-free, resulting in Undefined Behavior.

In a single-threaded context, this is a common but often manageable bug. In a multi-threaded context, it becomes a nightmare. If one thread is iterating over a shared vector while another thread concurrently adds an element to it, the first thread’s iterator can be invalidated mid-loop, leading to crashes or silent data corruption that are extremely difficult to reproduce and debug.

c++ has tried to come up with solutions to these problems with smart pointers

two types of smart pointers

1. unique pointers

2. shared pointers

Unique Pointers

when you define a unique pointer, the scope in which you define it, when the pointer goes out of scope, then the delete is automatically called on its memory so the issue of memory leakage is partially solved ( i will explain why partially ahead )

now if you want to pass that pointer to a function, there are two options

Transfer the ownership

the origional pointer becomes null, and the new scope (the function) is responsible for deleting the memory to the new pointer and it is automatically deleted once the scope ends

Borrowing

Delete is not called

it still doesn’t solve the problem completely. suppose while borrowing the pointer you call delete on it accidentally then you will get into the use-after-free error

Shared Pointers

unique pointers are great but what if you want to pass ownership between multiple functions? can’t do it via them

so we introduce shared_pointers

shared_pointer point to a memory block which is a struct with two members 1. pointer to actual data 2. a counter

when you initialize this shared pointer the counter is 1

this shared_pointer is only destroyed when the counter goes to zero

when you pass this shared_ptr to any function the counter gets incrimented and when the pointer goes out of the scope the counter decrements

but shared_ptr has a major drawback of Cyclic Dependencies

this can be solved using another type of smart pointers called free_ptr but it’s not very elegent

• A weak_ptr is created from a shared_ptr.

• It “observes” the object but does not contribute to the reference count. It does not own the object.

• Because it doesn’t own the object, the object can be deleted out from under it. Therefore, you cannot dereference a weak_ptr directly.

• To access the object, you must call the .lock() method on the weak_ptr. This method checks if the object still exists. If it does, .lock() returns a new shared_ptr to it (atomically incrementing the ref count), which you can then safely use. If the object has already been deleted, .lock() returns a null shared_ptr

that’s it for this blog, now I will be going to learn rust.

hope another blog on rust coming out soon

lets see how audio data is stored in your wav files

it mostly a list of integers representing amplitude of the sound.

sampling rate :- how many times per second the amplitude is calculated and stored in the list of integers. its generally 44,100 Hz, means 44k times sampled in a second

and the other is bit rate, it’s about how accurate is the amplitude value stored, commonly it’s 16 bit, meaning it can range from −32,768 to +32,767

this format while being okay is not that great for machine learning models.

we try to make this 1d data into 2d using the fourier transform

a fourier transform tells us how strong different frequency bands (for example 0–1 kHz, 1–2 kHz, 10–20 kHz, etc.) are in the signal

applying it once on the whole audio only gives the overall frequency distribution (no information about when a frequency occurred) → still 1d

to capture time-specific frequency information, we split the audio into short chunks (commonly 20–30 ms, e.g. 25 ms)

then we run the fourier transform on each chunk → this is called short-time fourier transform (STFT)

stacking these results gives us a 2d representation:

• one axis = time (chunks)

• one axis = frequency

• values in the grid = amplitude/energy for each frequency at each time

this 2d representation is often visualized as a spectrogram

example:

• pure 440 Hz tone → spectrogram shows a straight line at 440 Hz across time

• speech → spectrogram shows changing frequency bands (formants) over time

mel-spectrogram

mel-spectrogram is derived from the spectrogram by mapping frequencies onto the mel scale

mel scale is a perceptual scale where equal steps sound equally spaced to human ears

humans perceive pitch roughly logarithmically (we notice differences at low frequencies more than at high)

so instead of linear frequency bins, mel-spectrogram compresses high-frequency ranges and expands low ones

result = representation closer to how humans actually hear sound

widely used in speech recognition and music analysis

mel-frequency cepstral coefficients (mfccs)

• derived from mel-spectrogram by applying a discrete cosine transform (dct) to decorrelate frequency bands

• captures the overall spectral envelope (timbre/shape of sound) rather than fine detail

• commonly used in speech recognition because it represents how humans perceive phonemes

• typical dimension: 12–13 coefficients per frame (sometimes + energy + delta + delta-delta → ~39 features per frame)

chroma features

• group the spectrum into 12 bins, one for each pitch class (c, c#, d … b), regardless of octave

• captures harmonic and melodic characteristics, useful for music analysis (e.g., chord detection, key recognition)

• ignores exact frequency scale, focuses on pitch class energy distribution

• typical dimension: 12 features per frame

all these mel-spectrogram and mel-frequency cepstral coefficients and chroma features they are like images so CNN’s work very well on them.

they are different from your OS-Threads in the way that they are very light weight, each with a stack size of 2kb that can grow and shrink compared of 1-8 mb of a local os-thread

syntax

package main

import (
	"fmt"
	"time"
)

func sayHello() {
	fmt.Println("Hello from the sayHello goroutine!")
}

func main() {
	
	go sayHello()

	fmt.Println("Hello from the main goroutine.")
	
	time.Sleep(100 * time.Millisecond)
}

what happens if you remove time.sleep at the last? the sayHello function will not print.

think of it like this, the main function is run by a main go-routine and it is not in the habit of waiting for smaller go-routines.

Using time.Sleep is a fragile hack. We need a deterministic synchronization mechanism. This is the perfect segue into sync.WaitGroup, which we will cover next.

It's essentially a concurrent counter that allows a goroutine (usually the main goroutine) to block until a collection of other goroutines have finished their tasks.

A WaitGroup has three core methods:

1. Add(delta int): This increments the WaitGroup's internal counter by delta.

   • You call this before you launch the goroutine(s). If you have N goroutines to wait for, you would call wg.Add(N).

2. Done(): This decrements the WaitGroup's counter by one.

   • This is called by the worker goroutine itself, typically as the last action before it returns, often using defer. It's a signal from the goroutine saying, "I have finished my work."

3. Wait(): This blocks the goroutine that calls it until the WaitGroup's internal counter becomes zero.

   • This is called by the goroutine that needs to wait (e.g., the main goroutine).

let’s see this in code

Execution Flow and Output:

Main: Starting workers...
Main: Waiting for workers to finish...
Worker 3 starting
Worker 5 starting
Worker 1 starting
Worker 4 starting
Worker 2 starting
Worker 3 finished
Worker 5 finished
Worker 1 finished
Worker 4 finished
Worker 2 finished
Main: All workers finished. Exiting.

now you might be wondering why the output is not sequential, that is because there is a difference between starting a go routine and scheduling a go routine.
the scheduling part was sequential. and places this new goroutine into a runnable queue. but when the go routines are run they are not sequential, the queue does not take care of the order

Channels

channels are a way for go-threads to communicate information with each other, obviously they could have communicated by read global variables but that is discouraged in golang’s philosophy and channel communication is encouraged.

You declare a channel using the chan keyword followed by the type of data it will carry.

var myIntChannel chan int     // A channel that carries integers
var myStringChannel chan string // A channel that carries strings
var myStructChannel chan MyStruct // A channel that carries values of type MyStruct

Like maps and slices, a channel is a reference type. Its zero value is nil. Before you can use a channel, you must initialize it with the built-in make() function.

myIntChannel = make(chan int) 

Unbuffered Channels and Rendezvous

An unbuffered channel is created with a capacity of zero. This is the default.

ch := make(chan int) // or make(chan int, 0)

An unbuffered channel has a unique and powerful synchronization property: it forces a rendezvous.

• A send operation on an unbuffered channel will block until another goroutine is ready to receive from that same channel.

• Likewise, a receive operation will block until another goroutine performs a send.

let’s look this through a concrete code example

The Transactional View

The Go runtime acts as a mediator for the unbuffered channel.

1. Sender Arrives First:

   • Goroutine A executes ch <- "hello".

   • The runtime sees there is no receiver waiting.

   • The runtime suspends Goroutine A. It is now blocked, waiting for a partner.

2. Receiver Arrives:

   • Goroutine B executes msg := <-ch.

   • The runtime sees there is a sender (Goroutine A) waiting.

   • The Rendezvous Transaction Begins:

     • The runtime takes the value ("hello") directly from the sending goroutine (A).

     • It passes this value to the receiving goroutine (B).

     • The value is assigned to the variable msg.

     • Now that the transaction is complete, the runtime marks both Goroutine A and Goroutine B as runnable again.

   • The Transaction Ends.

3. Execution Continues:

   • At this point, the statement msg := <-ch is complete in Goroutine B. The line of code is finished. Goroutine B can now move on to the next statement, which is fmt.Printf.

   • Simultaneously, the statement ch <- "hello" is complete in Goroutine A. Goroutine A can now move on to its next statement.

it’s also possible that instead of the send, our execution first gets to the receive point and blocks the main goroutine until the send is done by another goroutine

Buffered Channels and Decoupling

A buffered channel is created with a capacity greater than zero.

ch := make(chan int, 3) // A channel that can hold up to 3 integers

A buffered channel decouples the sender and receiver.

• A send operation on a buffered channel will only block if the channel's buffer is full. If there is space, the send completes immediately, and the value is stored in the channel's buffer.

• A receive operation will only block if the channel's buffer is empty.

This allows the sender and receiver to work at different paces, as long as the buffer doesn't fill up or empty out.

Deadlocks with Channels

A deadlock occurs when all goroutines in a program are blocked, waiting for something that can never happen.

close(channel)

• Signals that no more values will ever be sent on this channel.

• It is a final "goodbye" from the sender(s).

Properties of a Closed Channel:

1. Sending to a closed channel will cause a panic. This is a strict rule: once you say you're done sending, you must be done.

2. Receiving from a closed channel never blocks. It immediately returns a value.

   • If there are values still in the buffer, it returns them one by one.

   • Once the buffer is empty, any subsequent receives will immediately return the zero value for the channel's type (e.g., 0 for int, "" for string, nil for pointers).

How does a receiver know the difference between a legitimate zero value and a zero value from a closed, empty channel? The receive operator has a special two-variable form:

value, ok := <-channel

• ok is a boolean.

• If ok is true, the value was a legitimate value sent on the channel.

• If ok is false, it means the channel is closed and empty. The value will be the zero value for its type

This brings us to the most elegant way to receive all values from a channel until it is closed: a for...range loop.

for item := range channel {
    // This loop will automatically receive values from the channel
    // and assign them to 'item'.
    
    // The loop will automatically break when the channel is closed
    // and all values have been drained from its buffer.
}

let’s see this all in action through a code sampel

if you see carefully, there is a race condition between producer function and for…range loop.

what if the loop win? then the jobs channel would be empty and remember what we said above? empty channels are blocking. so this for loop will get blocked until producer sends some value in this.

Directional Channels (chan<- and <-chan) for API Safety

• <-chan T: A receive-only channel of type T. You can only receive from it (val := <-ch). You cannot send to it or close it.

• chan<- T: A send-only channel of type T. You can only send to it (ch <- val). You cannot receive from it or close it.

This is extremely useful for writing clear APIs.

• A function that produces data should accept a send-only channel as an argument.

• A function that consumes data should accept a receive-only channel as an argument.

Common Channel Use Cases

This section is about patterns—recipes for structuring concurrent code. We'll look at three of the most fundamental: Worker Pools, Fan-in/Fan-out, and Pipelines

Worker Pools

The idea is to control the level of concurrency for a set of tasks. Instead of launching a new goroutine for every single task (which could be thousands or millions), you launch a fixed number of persistent "worker" goroutines. You then feed tasks to these workers via a channel.

Real-World Example: Concurrent Thumbnail Generator

Fan-in / Fan-out

this one honestly I also do not understand much, so I am just pasting the code snippet here

Pipelines

A pipeline is a chain of processing stages connected by channels. Each stage is a goroutine that:

1. Receives values from an upstream channel.

2. Performs some function on that value.

3. Sends the result to a downstream channel.

This creates a concurrent assembly line. It's a very powerful and elegant way to structure data processing tasks.

The select Statement

A select statement blocks until one of its cases can run, then it executes that case. If multiple cases are ready at the same time, it chooses one at random to execute. This randomness is important because it ensures fairness and prevents a "busy" channel from always starving out another channel.

Waiting on Multiple Channels

This is the primary use case for select. Imagine a goroutine that needs to process work coming from two different producers.

A select statement normally blocks. However, you can make it non-blocking by adding a default case.

The most powerful and common pattern is combining for and select. This creates a goroutine that acts like a server, continuously processing events from multiple channels until it receives a signal to stop.

note : instead of the last sleep we could also have used wait groups or there is a nice cool trick which simulates wait groups using channels itself

Timeouts and Tickers

Often, a goroutine can't afford to wait forever on a channel operation. You might be making a network request or waiting for a job from a queue, and you need to give up after a certain amount of time.

The time.After function is the perfect tool for this.

When you combine this channel with a select statement, you can create a timeout.

Output:

Scenario 1: Waiting for result with a 3-second timeout...
   -> Received result: result 1

Scenario 2: Waiting for result with a 1-second timeout...
   -> Timed out!

time.Ticker for Periodic Tasks

Graceful Shutdown and Cancellation with context

The Problem It Solves:

Imagine you have a web server. A user sends an HTTP request.

1. Your server handler starts a goroutine to handle the request.

2. This goroutine makes a call to a database.

3. It also makes a call to a microservice.

4. The microservice itself might make other calls.

Now, what happens if the user closes their browser? The initial HTTP request is cancelled. All the downstream work being done by the database and microservice calls is now pointless. We need a way to tell all the goroutines involved in this request, “Stop your work, the result is no longer needed.”

The context package provides this cancellation signal.

Error Handling in Concurrent Code

In a standard sequential program, error handling is straightforward: a function returns an error, and the caller immediately checks if err != nil.

In a concurrent program, this breaks down. When you launch a goroutine with go myFunction(), you can’t get a return value. So how does the goroutine report back if it fails? What if you launch 100 goroutines and need to know if any of them failed?

Simply logging the error from within the goroutine is not enough. The main goroutine, which launched the work, often needs to know about the failure to decide on a course of action (e.g., cancel other workers, retry the operation, or exit).

We need a pattern to propagate errors out of goroutines and aggregate them for the calling function.

5.2.1: Using a Dedicated Error Channel

The most idiomatic Go way to solve this is to use the same tool we use for data: a channel. We can create a dedicated channel just for passing error values.

The Pattern:

1. Create a buffered channel for errors. The buffer size is typically the number of workers, so that no worker will block when trying to send an error.

2. Pass this error channel to each worker goroutine.

3. Inside the worker, if an error occurs, send it into the error channel instead of just logging it. If no error occurs, the worker does not send anything.

4. The main goroutine, after launching the workers, must have a way to collect and process these errors.

5.2.2: Combining WaitGroup and Error Aggregation

The main goroutine cannot simply range over the error channel, because it doesn’t know how many errors to expect. If it tries to read N errors (for N workers), it will deadlock if only one worker fails.

The correct pattern is to use a sync.WaitGroup to know when all workers have finished their attempts. Then, and only then, can we safely close the error channel and read whatever errors were sent.

The Full Pattern:

1. Create a tasks channel and an errs channel.

2. Create a sync.WaitGroup.

3. Start a fixed number of workers. In each worker:

   • defer wg.Done().

   • Process tasks.

   • If a task fails, send the error to the errs channel.

4. Start a separate “closer” goroutine. This goroutine’s only job is to wg.Wait() and then close(errs). This is the key to breaking the deadlock.

5. The main goroutine is now free to range over the errs channel. This loop will block until the closer goroutine closes the channel, at which point it will process any received errors and then terminate.

this is it for this blog, to be honest I am also not an expert on go-concurrency as of now but will try to learn as much as we can.

some facts on threads first

• A thread is the smallest unit of execution that a CPU can schedule and run

• A process can have one or more threads; they share the same memory space but run independently

• Each thread has its own stack (function calls, local variables) but shares heap, globals, and resources with other threads of the same process

• Threads let multiple tasks run seemingly at the same time within a single program

• Concurrency comes from starting multiple threads so that while one is waiting (for I/O, lock, etc.), another can use the CPU

• On multi-core CPUs, different threads can actually run in parallel on different cores

• The downside: since threads share memory, you must use synchronization (locks, semaphores, etc.) to prevent data corruption when multiple threads access the same data

the last point of what this whole article is going to be about. so lets lock-in 💪

Race Condition

Consider a simple, common operation: incrementing a shared counter. This appears atomic at a high level, but at the machine instruction level, it's typically a sequence of three distinct operations:

1. Read: Load the current value of the counter into a CPU register.

2. Modify: Increment the value in the register.

3. Write: Store the new value from the register back into memory.

Now, imagine two threads, Thread A and Thread B, both attempting to increment the same shared integer counter which is initially 0.

Scenario 1: Desired Outcome (Sequential Execution)

• Thread A:

  • Reads counter (value is 0)

  • Increments register (register value is 1)

  • Writes register to counter (counter is now 1)

• Thread B:

  • Reads counter (value is 1)

  • Increments register (register value is 2)

  • Writes register to counter (counter is now 2)

Final counter value: 2. This is correct.

Scenario 2: Race Condition Leading to Incorrect Outcome

• Thread A: Reads counter (value is 0). Context switch.

• Thread B:

  • Reads counter (value is 0)

  • Increments register (register value is 1)

  • Writes register to counter (counter is now 1). Context switch.

• Thread A:

  • Increments register (register value, which was 0 from its initial read, becomes 1)

  • Writes register to counter (counter is now 1).

Final counter value: 1. This is incorrect. Both threads executed their increment operation, but the shared counter only reflects one increment because Thread A overwrote Thread B's update with an outdated value. This is a classic example of a "lost update."

#include <stdio.h>
#include <pthread.h>

int shared_counter = 0;

void* increment_function(void* arg) {
    int num_iterations = *((int*)arg);
    for (int i = 0; i < num_iterations; i++) {
        shared_counter = shared_counter + 1;
    }
    return NULL;
}

int main() {
    pthread_t thread1, thread2;
    int iterations_per_thread = 100000;

    pthread_create(&thread1, NULL, increment_function, &iterations_per_thread);
    pthread_create(&thread2, NULL, increment_function, &iterations_per_thread);

    pthread_join(thread1, NULL);
    pthread_join(thread2, NULL);

    printf("Expected: %d\n", 2 * iterations_per_thread);
    printf("Actual: %d\n", shared_counter);

    return 0;
}

Atomicity

From the perspective of any other thread, an atomic operation is either not yet started or fully complete—there is no observable intermediate state.

As we saw, the C statement shared_counter = shared_counter + 1; is not atomic. It is composed of multiple machine instructions (read, modify, write). A race condition occurs precisely because a thread can be preempted in the middle of this non-atomic sequence.

If we could make that increment operation atomic, the race condition would disappear.

How is Atomicity Achieved?

1. Hardware Support: Modern CPUs provide special atomic instructions. These instructions are guaranteed by the hardware to execute without being interrupted. Common examples include:

   • Test-and-Set: Atomically writes a value to a memory location and returns its old value.

   • Fetch-and-Add: Atomically increments a value in memory and returns the old value.

   • Compare-and-Swap (CAS): Atomically compares the contents of a memory location to a given value and, only if they are the same, modifies the contents of that memory location to a new given value. This is a cornerstone of many advanced lock-free algorithms.

   In C, using the <stdatomic.h> header (since C11), you can perform these operations. For example, atomic_fetch_add(&shared_atomic_counter, 1); would perform a hardware-guaranteed atomic increment.

2. Software Implementation (via Locks): The more common way to achieve atomicity for a sequence of operations is by using synchronization primitives. By placing a lock before a sequence of instructions (the critical section) and releasing it after, we create a block of code that is effectively atomic from the perspective of other threads.

CPU Level Reordering

// Shared global variables
int data = 0;
int is_ready = 0;

// Thread 1 (Producer)
void producer() {
    data = 42;      // Write 1
    is_ready = 1;   // Write 2
}

// Thread 2 (Consumer)
void consumer() {
    if (is_ready == 1) { // Read 1
        // The programmer expects that if is_ready is 1,
        // then the write to 'data' must have happened first.
        printf("Data is %d\n", data); // Read 2
    }
}

On a weakly-ordered system, it is possible for the CPU or compiler to reorder the writes in producer. The is_ready = 1 write might become globally visible to the consumer thread before the data = 42 write does. The consumer would then read is_ready == 1, proceed, and print "Data is 0". This is a subtle but critical form of race condition caused by memory reordering.

Axiom 1: Private Caches

• Each CPU core has its own private L1/L2 cache.

• A write operation by a thread on Core A updates Core A's private cache first.

• This write is not instantly visible to a thread on Core B. It becomes visible only after the cache coherence protocol propagates the change. This propagation is not instantaneous.

Axiom 2: Store Buffers

• To avoid stalling on slow memory writes, each core has a "store buffer." When a thread performs a write, the data is often placed in this buffer first. The CPU can then continue executing other instructions while the memory system works on committing the buffered write to the cache in the background.

• A write sitting in Core A's store buffer is completely invisible to Core B.

Axiom 3: Instruction Reordering

• To maximize performance, the compiler and the CPU are permitted to reorder instructions.

• Rule 3a (Load/Load and Store/Store Reordering): Weak architectures (like ARM) can reorder two read operations or two write operations. Stronger architectures (like x86) generally do not reorder writes with other writes.

• Rule 3b (Store/Load Reordering): Nearly all architectures (including x86) allow a write to be reordered with a subsequent, independent read. This is the most common and problematic source of reordering. The CPU can speculatively execute a read before a preceding write has been fully committed from its store buffer.

Axiom 4: The Programmer's View vs. Reality

• The order of instructions in your C code is merely a suggestion to the machine.

• The final order of execution is a complex combination of compiler reordering and CPU runtime reordering, governed by the axioms above.

With these axioms, let's analyze the execution flow.

Execution Flow: WITHOUT Locks

Shared Data:
int data = 0;
int is_ready = 0;

Producer Code (Thread P on Core A):

1. data = 42;

2. is_ready = 1;

Consumer Code (Thread C on Core B):

1. if (is_ready == 1)

2. print(data);

Let's trace a possible, and perfectly valid, sequence of events according to our axioms.

This scenario demonstrates a producer-side reordering problem, made possible by the Store Buffer (Axiom 2). A similar scenario could be constructed for a consumer-side reordering (Axiom 3b) where the consumer reads data before it reads is_ready. The result is the same: incorrect behavior.

Execution Flow: WITH Locks

Now, let's introduce a new set of rules that apply when we use a mutex.

The Axioms of a Mutex

Axiom 5: Mutual Exclusion

• Only one thread can hold a given mutex at any one time. A thread calling lock() will block (wait) until the mutex is released by its current owner.

Axiom 6: The Memory Barrier Handshake

• A mutex_unlock operation creates a release barrier. This acts as a command: "Commit all my buffered writes now. Ensure all my previous memory operations are visible to the rest of the system before the lock is actually released."

• A mutex_lock operation creates an acquire barrier. This acts as a command: "Before I proceed, I must synchronize my local view of memory with the latest changes from whoever last released this lock. Invalidate my stale cache entries."

• These two barriers work together. The release publishes changes; the acquire receives them. This creates a "happens-before" relationship. The unlock on Thread A happens-before the subsequent lock on Thread B.

Producer Code (Thread P on Core A):

1. lock(m);

2. data = 42;

3. is_ready = 1;

4. unlock(m);

Consumer Code (Thread C on Core B):

1. lock(m);

2. if (is_ready == 1)

3. print(data);

4. unlock(m);

Let's trace the execution flow with these new, powerful rules.

Deadlock

A deadlock is a state in which two or more threads are blocked forever, each waiting for a resource held by the other. The most common cause of deadlock with mutexes is acquiring multiple locks in an inconsistent order.

The Deadly Embrace

Imagine two threads, Thread A and Thread B, and two mutexes, M1 and M2.

1. Thread A locks M1.

2. Thread B locks M2.

3. Thread A tries to lock M2, but it's held by Thread B. Thread A blocks.

4. Thread B tries to lock M1, but it's held by Thread A. Thread B blocks.

pthread_mutex_t M1, M2;

// Transferring a resource from an object protected by M1
// to an object protected by M2.

void transfer_resource() {
    // Enforce a global lock order: Lock the mutex with the lower memory address first.
    if (&M1 < &M2) {
        pthread_mutex_lock(&M1);
        pthread_mutex_lock(&M2);
    } else {
        pthread_mutex_lock(&M2);
        pthread_mutex_lock(&M1);
    }
}

"Any thread that needs to lock both M1 and M2 must always lock M1 first, then lock M2."

Semaphores

A semaphore is a synchronization primitive that is a generalization of a mutex. It was invented by the legendary Dutch computer scientist Edsger W. Dijkstra in the mid-1960s.

A semaphore is essentially a non-negative integer counter that is manipulated by two atomic operations:

1. P() (from the Dutch proberen, "to test" or "to probe"). Also known as wait() or down().

2. V() (from verhogen, "to increment"). Also known as post(), signal(), or up().

The semaphore's internal integer counter tracks the number of available "permits" or "resources".

The logic of the two operations is as follows:

• wait(semaphore):

  1. Atomically decrement the semaphore's internal counter.

  2. If the counter's value becomes negative, the calling thread is blocked and placed into a waiting queue.

  3. If the counter's value is zero or positive, the thread continues execution.

• post(semaphore):

  1. Atomically increment the semaphore's internal counter.

  2. If there are any threads blocked waiting on this semaphore (i.e., the counter was negative before the increment), wake one of them up.

Key Difference from a Mutex:
A mutex has the concept of "ownership". The thread that locks it must be the one to unlock it. A semaphore is a signaling mechanism; it has no concept of ownership. Any thread can post to a semaphore, even if it never called wait. This makes it a powerful tool for more complex synchronization scenarios beyond simple mutual exclusion.

There are two main types of semaphores:

• Binary Semaphore: The counter is initialized to 1. It can only be 1 (resource available) or 0 (resource unavailable). It behaves almost identically to a mutex. If you wait when it's 1, it becomes 0 and you proceed. If you wait when it's 0, you block. post changes it from 0 to 1 and wakes a waiting thread.

• Counting Semaphore: The counter can be initialized to any non-negative integer N. This allows up to N threads to pass the wait operation without blocking. It is used to guard a pool of N identical resources.

example use case :- suppose there is a function which can only run on a GPU but your GPU only has 4 cores, so you want this function to be at a time picked up by only 4 threads, this is where semaphores can be useful

Spinlocks

A spinlock is a type of lock that avoids blocking in the traditional sense. Instead of yielding the CPU when the lock is unavailable, the thread enters a tight loop, repeatedly checking the lock's status until it becomes free. This action of looping is called "spinning." The thread remains active and scheduled, consuming 100% of its CPU core while it waits.

When and Why to Use Spinlocks

The decision to use a spinlock versus a mutex is a pure performance trade-off, hinging on the expected wait time for the lock.

• Cost of a Mutex: The overhead of a context switch. This is a relatively high, fixed cost. It involves saving the thread's state, updating scheduler data structures, and then later restoring the thread's state. This can take thousands of CPU cycles.

• Cost of a Spinlock: The CPU cycles wasted while spinning. This cost is variable and directly proportional to how long the lock is held by another thread.

Use a spinlock when the critical section is very short

Spinlocks are only effective on multi-core systems. On a single-core system, they are a disaster. If a thread starts spinning on a single-core machine, it is consuming the only available CPU

Condition Variables

A condition variable is a synchronization primitive that allows threads to block (wait) until a particular condition, related to shared data, becomes true

lets look at what problem they solve through an example

// BROKEN AND INEFFICIENT "SOLUTION"
pthread_mutex_lock(&queue_mutex);
while (is_queue_empty(&my_queue)) {
    // What do we do here? If we just loop, we are holding the lock
    // and spinning, preventing the producer from ever getting the lock
    // to add an item. This is a deadlock.
}
// consume item
pthread_mutex_unlock(&queue_mutex);

how condition variables solve this

1. A thread acquires a mutex to protect the shared data.

2. It checks if a condition is true.

3. If the condition is false, the thread atomically unlocks the mutex and goes to sleep on the condition variable. This is the magic step.

4. Another thread (the producer/modifier) acquires the mutex, changes the state of the shared data, and potentially makes the condition true.

5. The producer thread then signals the condition variable, which wakes up one or more of the waiting threads.

6. A woken-up thread then re-acquires the mutex and checks the condition again before proceeding.

Read-Write Locks

• A shared data structure that is read often but written rarely doesn’t need every access to be exclusive.

• A standard mutex forces both reads and writes to wait for one another, so even purely read-only threads are serialized, creating needless slowdown.

• A read-write lock (rwlock) fixes this by offering
  – read (shared) lock: many threads can read simultaneously because they don’t change the data.
  – write (exclusive) lock: a writer gains sole access, ensuring updates stay consistent.

The rules of a read-write lock are:

1. Multiple Readers Allowed: Any number of threads can hold a read lock on the resource simultaneously.

2. Exclusive Writer: Only one thread can hold a write lock.

3. Writers Exclude Everyone: If a thread holds a write lock, no other thread (neither reader nor writer) can acquire any lock.

4. Readers Exclude Writers: If any thread holds a read lock, any thread attempting to acquire a write lock must wait until all readers have released their locks.

what if a stream of new readers keeps arriving while a writer is waiting?

This leads to two common implementation strategies, which address the "reader-writer problem."

Reader-Preference:

• Policy: If the lock is held by readers, an incoming reader is granted access immediately. A waiting writer will only get the lock when all readers (including the newly arrived ones) have finished.

Writer-Preference:

• Policy: If a writer is waiting for the lock, any newly arriving readers will be blocked. They will be queued up behind the waiting writer. The writer gets the lock as soon as the current readers finish.

Barriers

A barrier is a synchronization primitive that forces a group of participating threads to all wait at a specific point (the "barrier point") until every thread in the group has reached that point. Once the last thread arrives, all threads are released simultaneously and can proceed with their next phase of computation.

Barriers are ideal for algorithms that are broken down into distinct phases or stages, where the results of one phase are needed by all threads before the next phase can begin

Atomics

Starting with the C11 standard, the C language gained native support for atomic operations. This is a massive improvement over relying on platform-specific inline assembly or compiler intrinsics. The features are primarily defined in the <stdatomic.h> header.

An atomic operation is one that is guaranteed by the hardware to execute indivisibly. When a thread performs an atomic operation on a variable, no other thread can see the variable in a half-modified state. The entire operation (read, modify, and write) completes as a single, uninterruptible step.

C11 _Atomic Types and Operations

You can make any integer type, pointer, or struct/union (of a certain size) atomic by using the _Atomic keyword (or the more convenient atomic_int, atomic_bool, etc. type aliases from <stdatomic.h>).

Once a variable is declared as atomic, you cannot use standard operators like ++, +=, or = on it. You must use the special atomic functions

that’s it for this blog guys, next will be diving deeper into Golang concurrency, until then bbye.

over-fitting —> when the model instead of learning starts memorizing.

Exploding Gradients —> when a lot of parameters in your llm become >1, because the gradients are generally multiplication of lot of parameters , this becomes NaN number and the trianing crashes

Vanishing Gradients —> similar to Exploading Gradients, just the parameters become < 1

Surprisingly, the solutions are simple; they’re familiar you just didn’t notice their use cases.

over-fitting

Dropout Layer

1. For a given layer's output (or input to the next layer), you apply a Dropout layer.

2. During each forward pass of a training step, each neuron (or activation) in that layer's output has a probability p (e.g., p=0.1) of being randomly set to zero.

3. The remaining non-zero neurons are scaled up by a factor of 1 / (1 - p). This is called "inverted dropout" and it ensures that the expected sum of the outputs remains the same, which keeps the learning dynamics stable.

It prevents a neuron from becoming overly dependent on the presence of a few specific other neurons

Weight Decay (L2 Regularization)

L_total = L_main + (λ/2) * Σ(w^2)

• Σ(w^2) is the sum of the squares of all the individual weight parameters w in the entire model. This is the L2 norm of the weight vector.

• λ (lambda) is a hyperparameter called the weight decay rate

By adding this term to the loss, we are telling the optimizer: "Minimize the prediction error, BUT ALSO try to keep the weights as small as possible."

A model with large weights is often a sign of overfitting. Large weights mean the model is making very sharp, specific decisions based on small changes in the input

Exploading/Vanishing Gradients

Layer Normalization

They are inserted between other layers (typically after a linear layer and before an activation function) and their sole job is to rescale their input tensor to have a standard, predictable distribution, usually with a mean of 0 and a variance of 1. This keeps the numbers flowing through the network in a "well-behaved" range

All normalization layers also introduce two learnable parameters, gamma (scale) and beta (shift), after normalizing. This allows the network to learn if, for some reason, a different mean and variance is actually optimal for the next layer. y_out = gamma * y_normalized + beta

RMSNorm —> just a better version of layerNorm

• LayerNorm: y = (x - mean(x)) / sqrt(variance(x) + eps)

• RMSNorm: y = x / sqrt(mean(x^2) + eps)

its because of this image, this representation is flawed.
1. it forces you to think in scalars and becomes very confusing when you take this mental model to vectors

2. there is no description where the learnable parameters lie, so never think in this image.

here is a better way to think

so how did I reach here? back in August 2024, I was friends with Sik and Arnav. we all wanted to work on a startup but we were not getting a good idea. one fine day we got an idea that “why are these dating apps so costly? all they have to do is run some servers, They have hired all those unnecessary people. we think that’s the reason of their cost”

so we started building a cheap dating app

we soon enough realized that this was not a good idea.

in dating apps only two things matter for short term success

For Men

1. why will people download your app

2. why will they stick

in dating apps men stick when they are getting matches and men download your app when “they think they will get matches”

For Women

1. Quality of Profiles

but for long term success only one thing matters

number of matches formed on your app, which depends on number of women in the app and quality of male profiles.

interestingly number of women on your app also depends on quality of male profiles and how good your marketing is.

This cheap dating app thing in short would make people feel like they will get matches because they can swipe more after purchasing the premium, but what this would result in women getting a lot of likes and swipes from a lot of low-quality profiles, which would drive women away from app

we soon understood that the idea was not good enough, so we started doing more research. talking to females, approaching through linkedin (which a lot of them told us was weird, but females do not reply on whatsapp to an unknown demand like this)

after a lot of failed and stupid insights we finally got onto our two main insights

1.

Affluent class of India (India 1 - 120 million people / 30 million households) keep switching their dating apps, because after some years India 2 comes on the app and The experience is ruined.

so we would segregate profiles in India1 and India 2 at the time of onboarding itself and only allow matching within the groups

2.

we would show you detail insights on your profile, so that even if you are not getting matches. you would get some response from the app.

twitter and substack do this very nicely, to prevent posters from getting demotivated

3.

that’s a secret, I am going to use it on my other apps too lol

The Idea sounds good right? even we had the funds. so why did we close the idea?

to be honest I still do not have any concrete answer to this, it’s mostly vibes but here are some facts

1. at this time no-one is willing to invest in social apps in India. so there is funding winter in social space

2. we live an IIT Roorkee, no-body here uses dating apps. so to test a dating app. you have to launch in delhi or banglore and to market there you need live there which is costly

3. if we were in Delhi and would have friends who used dating apps, it was very easy to know what real problems are. It is totally possible that those two insights are all in the air and nobody needs it

4. subscriptions in india don’t make money. Netflix has a revenue of mere 37 crores in India

5. We are not marketers and film makers. We are tech people. The CAC without crazy marketing is too high to operate on

6. I personally do not have time. I want to get rich fast. I hate working (even though I work almost all day long from past 4 years) and social apps take around 10 years to really become something, so the time horizons are very long

having said all of this, This is the most exciting space to work on.

Nobody gets up being excited to do a B2B SaaS but this, I can’t even express you the addreline rush.

I did not wanted this post to be too long, so there a lot of things I did not include.

If you have any questions do hit me up at https://x.com/goyalayus

There will be three components in the B2C world

1. The Interface to Interact with LLM’s

2. The LLM’s

3. Tools LLM’s will use

we will be deep diving one by one in each layer

The Interface to Interact with LLM’s

There will be some standard interface to interact with AI.
for example :- your Laptop AI Assistant, Meta Glasses, Your mobile phone AI Assistant, The Robot in your house.

The Thing to note is a lot of big players will be playing here and it will get commotized. as mobile phones and laptops have got commoditised. you will have Xiomi, Samsung and Apple of Home Robots

winning in this market is super hard. Getting Acquired is possible.

The LLM’s

These are LLM’s which the Interface providers will use.

why do I say LLM’s and not LLM? yes because just like many interfaces there would be many LLM’s but the reason for this is quite different

LLM’s can hold 3.6 bits per parameter, which is a very less of a memory. This is the reason LLM’s are forced to generalise and not remember. This is the Reason why LLM’s can not generate Interesting Oppinions because They are made to generalize.

so Generalized LLM’s can not work great in specialized domains, there is allways chance to fine tune an LLM and make a superior domain specialized model.

which is why I say LLM’s and It is the reason I believe that Fine tunning is hear to stay.

Tools LLM’s will use

many people do not know that Perpexility uses a search engine beneath it to do search. No you can not search the web with an agent or llm, It has to use a search engine beneath. there are tools. LLM’s need to use.

A big way this tool usage is being made feasable is by MCP. Harkirat Singh has a Great video on MCP

another example : if AI would have to book an AirBnB for you, it would have to use AirBnB MCP. so now AirBnB becomes a tool.

I personally think that Money is to be made in The Tools.

B2B

I think they will become tools and expose an mcp endpoint.

twitter is weird. The first 2-3 times you use it, it sucks. then one day you open the app and it becomes addictive.

so when I used twitter initially, Paras would appear a lot on my feed. I thought he was a influencer who gives gyan and has no substance.

but then I heard the news that he sold wingify for 300 million and just out of curiosity I searched for his writings on internet ( Idk why I did it tbh) and I found his blog.

I was just exploring and stumbled upon the mental models for Entreprenure section, and it was just too good, Almost Paul graham level writing. completed all the articles in one reading.

then some days back I was lost, had just closed my second startup. did’nt knew what to do, was just writing Tech Articles on substack, and thought to give those a re-read.

but this time I made notes and below are my notes of all his mental model essays

• whats not going to change in next 5 years

• don’t go for what people say but what do they do. spirited airlines is the most hated one but also the most profitable. this proves that saving money > comfort for people

• desires remain same but solutions keep changing, startups with innovation solutions replace the once with old solutions. a toothpaste company is not only competing with the other toothpaste company but also with the mouthfreshner company

• Researching why customers are doing what they’re doing can provide deep insights into their desires

• look for instances where customers are innovating by themselves by modifying or re-imagining existing products

• salesforce did not create the desire for crms, it just provided them better fast cheaper

• rather than starting with an idea and then doing research, it’s much better to start with a blank slate and start observing customer behavior and trends. Sooner or later, you’ll find yourself full of bright ideas that are derived from actual customer behavior

• Both having no competitors or having too many competitors is a clear sign of wrong timing. there is a window where you can see that the first movers product is not good, learn from what early adopters liked and disliked and launch a much better product in market

• cultural startups ( you have to spot it, new desires derivitives of old desired) electric cars, people started caring about enviroment. cultural shift vs technical startups (gogle) existing desires but better faster cheaper. there is allways a spectrum, uber spotter both

• a narrow market also allows for laser-sharp distribution and marketing

• aiming for a fully built out product on day 1 is a recipe for disaster

• apple lisa is the greatest example of this

• start with a simple product that excels only on one (or a few) aspects that customers care about

• an improvement in one aspect will only be appreciated by that segment and get ignored by everyone else in the market. For example, if the customers in a particular segment are price-insensitive, your discounts won’t work on them. In your mind, a discount should clearly work but for a certain segment of customers, it may actually decrease the appeal of your product for them

• An improvement over existing solutions is not an improvement unless a large enough customer segment cares about it

• People have busy lives and they usually don’t think much about the products and services they use in their lives. It’s a myth that people are on a constant lookout to (marginally) improve their lives. it has to be atleast 2x

• rather than dive headlong into development, it’s wise to spend at least a few weeks researching the competition, customer needs, pricing, distribution channels, and development challenges

• But don’t be too patient. Although it doesn’t happen often, it’s possible to do too much research. Entrepreneurship requires a dash of ignorance about the difficulty of solving some problems in the market. Do too much research and you may get dissuaded to even take the first step

• In my experience, for most software / consumer products, research on the order of weeks is enough. After all that research, if you’re still excited about the opportunity, go for it

• Moreover, getting several things right in one go is always significantly more difficult than getting one thing right. Therefore, an entrepreneur should strive to have clarity on what few aspects of the total solution delivered to customers need originality and everything else should be borrowed from current best practices

• smartphones have become one of the biggest industries because a self-supporting ecosystem has emerged around it. Without smartphones, many useful apps like Uber or Google Maps wouldn’t have been possible, but without such apps, smartphones would have limited use and likely wouldn’t be as big as it is today

• similarly microsoft also took advantage and care of this thign

• For truly maximizing value, the entrepreneur needs to push for the growth of the entire ecosystem (and not just her own company)

• switching cost - Historical data storage, Habits and familiarity, Sunk cost, Customized solutions

• Your competitors are just like you: smart and hard-working

• Introspect and list down all your unfair advantages and then use all of them. Each unfair advantage matters and all of them add up to providing a significant deterrent to competitors keen on copying what you’ve built

• When you are starting out, you may not have unfair advantages. If you know how to code, so do many other people. Lacking an unfair advantage, you can succeed only via good market timing and luck. As someone with no unfair advantage over others, be ready to fail multiple times to give luck and timing a chance to be in your favor

• A word of caution. For an advantage to be unfair, it has to be both: an advantage and unfair. Due to our confirmation bias, it’s easy to fall into the trap of confusing non-advantages with advantages and common knowledge as something exclusively known to you. You must be totally honest about what’s an unfair advantage and what’s not. It’ll help you to take feedback on this from someone else

• “If everything you do needs to work on a three-year time horizon, then you’re competing against a lot of people,”

• Other competitive advantages could be doing what’s unsexy or boring (like waste management, tax, and accounting, or mainframes)

• Assume most people are lazy but market to those who aren’t

• All new products compete with Instagram for attention

• When it comes to costs, it is important to understand that people are loss averse. They’d much rather not lose what they have than gain something new

• If you make a big ask from them before they’ve gotten enough value from you, they’ll drop off. If you deliver value first and then ask for something later, they’ll oblige

• What people pay for something is determined by its perceived alternatives

• Consumers want to conform, companies want to differentiate

• Consumers hate getting sold to, companies love it

• Consumers want stuff for free, companies want to pay

• Your 30 second pitch shouldn’t be about you but about the user

• Get press by giving journalists something surprising

• Just like different customer segments have different needs, different talent segments have different drives

• Exceptional people never look for jobs; jobs look for them

• If you feel people will work for you because of the salary you’re offering, you’ll end up attracting only those kinds of people who want a safe, stress-free job that pays a regular salary

Follow me for more Tech and Startup blogs

For many computer vision tasks, images are resized to a standard square dimension before being fed into the model. A very common size, inherited from famous datasets like ImageNet, is 224x224 pixels.

A Transformer model doesn't look at one pixel at a time. That would be computationally enormous. Instead, it breaks the image down into a grid of smaller, non-overlapping square "patches." A standard patch size used in many ViT (Vision Transformer) models is 16x16 pixels.

Now, how many of these 16x16 patches fit into the 224x224 image.

• Along the width: How many 16-pixel patches fit into 224 pixels?
  224 / 16 = 14 patches

• Along the height: How many 16-pixel patches fit into 224 pixels?
  224 / 16 = 14 patches

This creates a grid that is 14 patches wide by 14 patches tall.

To get the total number of patches, you multiply the number of patches along the width by the number along the height:

14 patches × 14 patches = 196 patches

now suppose there are 50k tokens in gpt-2 vocab, we create 8k more reserved tokens (from 1 - 8k) for images and text tokens now start from 8k-58k

so when you pass an text+image to gpt, it is passed text tokens (8k - 56k tokens) and these extra 196 tokens ( each of these token can be anything between 0-8k).

and the llm is then fine-tuned on a missive text + image dataset

now we will see how it is decided which patch will get which token.

we train a saperate model( it is generally an CNN) which is trained for doing one thing that is reconstructing the patch from the initial patch and the loss is calculated against how good it is able to reconstruct the image, in this process in between we make it output a token between 0-8k ( I know it feels a little vague, actually this is done through codebooks, you can read about them online or ask gpt about them )

That’s it for this essay, almost all the multimodal llms works in similar ways. suscribe for more of these “How things really work Lectures”

you can read this here on my personal website and I promise a great experience.

you can read this here on my personal website and I promise a great experience.

you can read this here on my personal website and I promise a great experience.