From my experience working at GreyOrange. Refactored my article a bit with help of GPT.

I was working on a component of a golang-service that intercepted InfluxDB writes (proxy), to enforce few rules like dropping unoptimised queries, emitting query metrics, mirroring writes to Kafka, etc. As a part of exploring the service, I learned about how different databases actually receive writes under the hood. The protocol a DB uses determines everything — I have discussed the same in this article at a high level.

There are two fundamentally different classes of DB write protocols.

Class 1: HTTP-Based Databases

InfluxDB, Elasticsearch, and CouchDB expose plain HTTP REST endpoints. A write is an HTTP request — nothing more.

• InfluxDB — Line Protocol over HTTP

  You call:

  client.WritePoint("cpu,host=web01 usage=42.3 1609459200000000000")
  

  The library sends:

  POST /write?db=mydb&precision=ns HTTP/1.1
  Host: localhost:8086
  Content-Type: application/octet-stream
  
  cpu,host=web01 usage=42.3 1609459200000000000
  

  The body is InfluxDB’s Line Protocol (a plain-text format InfluxDB invented for writing time-series data — each line encodes one data point as measurement name + tags + fields + timestamp): measurement,tag_key=tag_val field_key=field_val timestamp. Multiple points are newline-separated in one request body. No handshake, no session negotiation, no binary encoding — just HTTP POST with a text payload.

  For reads:

  GET /query?db=mydb&q=SELECT+usage+FROM+cpu+WHERE+time+>+now()-1h HTTP/1.1
  

  or POST /query with the InfluxQL in the body.

• Elasticsearch — JSON over HTTP

  Single document write:

  POST /my-index/_doc HTTP/1.1
  Content-Type: application/json
  
  {"@timestamp": "2024-01-01T00:00:00Z", "level": "error", "msg": "disk full"}
  

  Bulk write uses the _bulk API with NDJSON (Newline-Delimited JSON — each JSON object on its own line, no outer array; chosen because it lets the server stream-parse millions of records without loading the whole body into memory) — alternating action lines and document lines:

  POST /_bulk HTTP/1.1
  Content-Type: application/x-ndjson
  
  {"index": {"_index": "logs"}}
  {"level": "error", "msg": "disk full"}
  {"index": {"_index": "logs"}}
  {"level": "warn", "msg": "cpu high"}
  

  The library serializes your objects to JSON, adds action metadata lines, and POSTs the NDJSON body. Under the hood: just HTTP.

• CouchDB — REST + JSON

  PUT /mydb/doc-id HTTP/1.1
  Content-Type: application/json
  
  {"key": "value", "_rev": "1-abc"}
  

  CouchDB is perhaps the purest HTTP DB — every document operation maps directly to an HTTP verb on a URL. No special client library needed; curl works fine.

Class 2: TCP Wire Protocol Databases

Postgres, MySQL, Redis, and MongoDB do not use HTTP. They define their own binary (or text) framing over a raw TCP socket. When your application writes, the driver opens a TCP connection and speaks the DB’s private protocol directly.

• PostgreSQL — Frontend/Backend Protocol (v3)

  Postgres defines its own binary protocol (called “Frontend/Backend Protocol” — frontend = client, backend = DB server) for communication over TCP. Each message is [1-byte type][4-byte length][payload]. After a startup + auth handshake, a write looks like:

  'Q'                         -- message type: Simple Query
  00 00 00 21                 -- message length (33 bytes including length itself)
  49 4E 53 45 52 54 20 49 4E 54 4F 20 74 20 56 41 4C 55 45 53 20 28 31 29 00
  -- "INSERT INTO t VALUES (1)\0"  (null-terminated SQL string)
  

  Server responds with CommandComplete (C, e.g., INSERT 0 1) and the connection stays open for the next query.

  For prepared statements, the driver splits this into three messages — Parse (send the SQL template), Bind (send the parameter values), Execute (run it). This is how $1, $2 placeholders work: the SQL and the values travel separately.

• MySQL — Client/Server Protocol

  MySQL uses a 4-byte header per packet: 3 bytes payload length + 1 byte sequence number.

  After a handshake, writes go via COM_QUERY (COM_ prefix is MySQL’s naming convention for client commands; QUERY means “execute this SQL string” — command byte 0x03 identifies it in the binary packet):

  [payload_length 3B][sequence 1B][0x03][SQL bytes in UTF-8]
  

  For example, INSERT INTO t VALUES (1):

  19 00 00    -- payload length: 25 bytes (1 command byte + 24 SQL bytes)
  00          -- sequence: 0
  03          -- COM_QUERY
  49 4E 53 45 52 54 20 49 4E 54 4F 20 74 20 56 41 4C 55 45 53 20 28 31 29
  

  For prepared statements: COM_STMT_PREPARE + COM_STMT_EXECUTE, where parameters are sent as binary-encoded typed values separately from the SQL template.

• Redis — RESP (Redis Serialization Protocol)

  Redis is a TCP wire-protocol DB, but unlike Postgres/MySQL its protocol is text-based, not binary. RESP (the format Redis invented for client-server communication — simple, human-readable, easy to parse line by line) looks like this:

  *3\r\n          -- array of 3 elements
  $3\r\n          -- bulk string, 3 bytes
  SET\r\n
  $5\r\n          -- bulk string, 5 bytes
  mykey\r\n
  $7\r\n          -- bulk string, 7 bytes
  myvalue\r\n
  

  Every Redis command is sent as a RESP array where the first element is the command name. The client library serializes client.Set("mykey", "myvalue") into exactly this text and writes it to the TCP socket. RESP3 (Redis 6+) adds typed responses (maps, sets, doubles) but the request framing is the same.

• MongoDB — Wire Protocol (OP_MSG + BSON)

  MongoDB’s wire protocol wraps BSON (Binary JSON) in a message envelope: [total_length 4B][opcode 4B][payload] (plus some bookkeeping fields). Since MongoDB 3.6, the primary opcode is OP_MSG (opcode 2013 — OP_MSG is the unified message type that replaced the older separate OP_INSERT, OP_UPDATE, OP_DELETE opcodes; all operations now go through this one envelope). An insert looks like:

  OP_MSG flags (4 bytes)
  Section kind=0 (body document in BSON):
    {
      "insert": "mycollection",
      "ordered": true,
      "documents": [{"_id": ObjectId("..."), "field": "value"}],
      "$db": "mydb"
    }
  

  The driver serializes your document to BSON (Binary JSON — MongoDB’s binary encoding of JSON-like documents; each field has a type tag byte followed by the value in binary. An integer is stored as 4 raw bytes, not ASCII digits like "42". Compact, fast to parse, but not human-readable), wraps it in an OP_MSG body section, and writes the envelope to the TCP socket.

Why the Protocol Matters

This also determines how hard it is to debug:

• HTTP-based DBs: tcpdump or Wireshark gives you readable traffic immediately.
• TCP wire protocol DBs: you see binary bytes — you need a protocol dissector (Wireshark has Postgres and MySQL dissectors built in) or run the DB in verbose logging mode.

If You Were to Build a Proxy in Front of These DBs

• HTTP-Based DBs (InfluxDB, ES, CouchDB)

  Straightforward — your proxy is just an HTTP server with a forwarding client. You intercept the request, read the body, inspect/filter, then forward or reject.

  InfluxDB write interception example:

  http.HandleFunc("/write", func(w http.ResponseWriter, r *http.Request) {
      body, _ := io.ReadAll(r.Body)
        
      for _, line := range bytes.Split(body, []byte("\n")) {
          measurement := extractMeasurement(line) // everything before first comma or space
          if isBlocked(measurement) {
              http.Error(w, "measurement blocked", http.StatusForbidden)
              return
          }
      }
        
      resp, _ := http.Post(influxAddr+"/write?"+r.URL.RawQuery,
          r.Header.Get("Content-Type"), bytes.NewReader(body))
      io.Copy(w, resp.Body)
  })
  

  For query filtering, parse with the official InfluxQL library — gives you an AST to inspect WHERE clauses, time ranges, measurements, etc.:

  import "github.com/influxdata/influxql"
  
  stmt, _ := influxql.ParseStatement(queryString)
  if sel, ok := stmt.(*influxql.SelectStatement); ok {
      if sel.Condition == nil {
          return fmt.Errorf("query must have a WHERE clause")
      }
      min, max := sel.TimeRange()
      if max.Sub(min) > 35*24*time.Hour {
          return fmt.Errorf("time range too large")
      }
  }
  

  The standard HTTP middleware ecosystem applies — auth, rate limiting, logging, body size limits, all without touching any DB-specific code.

• TCP Wire Protocol DBs (Postgres, MySQL, Redis, MongoDB)

  Harder. Your proxy opens two TCP connections (client-side and server-side) and pipes bytes between them — but must parse the stream to actually read queries.

  Redis is easiest since RESP is text:

  func parseRESP(r *bufio.Reader) ([]string, error) {
      line, _ := r.ReadString('\n')
      count, _ := strconv.Atoi(strings.TrimSpace(line[1:])) // line is "*N\r\n"
        
      args := make([]string, count)
      for i := 0; i < count; i++ {
          r.ReadString('\n')                 // "$N\r\n"
          val, _ := r.ReadString('\n')
          r.ReadString('\n')                 // trailing "\r\n"
          args[i] = strings.TrimSpace(val)
      }
      return args, nil
  }
  // args[0] = "SET", args[1] = "mykey", args[2] = "myvalue"
  // Block FLUSHDB, FLUSHALL, DEBUG etc. by checking args[0]
  

  Postgres requires parsing binary message framing:

  func readPgMessage(conn net.Conn) (msgType byte, payload []byte, err error) {
      header := make([]byte, 5)
      io.ReadFull(conn, header)
      msgType = header[0]
      length := int(binary.BigEndian.Uint32(header[1:])) - 4
      payload = make([]byte, length)
      io.ReadFull(conn, payload)
      return
  }
  
  msgType, payload, _ := readPgMessage(clientConn)
  if msgType == 'Q' {
      sql := string(payload[:len(payload)-1]) // strip null terminator
      if violatesPolicy(sql) {
          sendPgError(clientConn, "query not allowed")
          continue
      }
  }
  writeToDBConn(dbConn, msgType, payload)
  relayResponse(dbConn, clientConn)
  

  For production, don’t write the protocol parser from scratch — use existing libraries:

  • Postgres: jackc/pgproto3 (Go) — full Frontend/Backend protocol implementation
  • MySQL: go-mysql-org/go-mysql (Go)
  • MongoDB: mongodb/mongo-go-driver internals, or mongonet
  • Redis: RESP is simple enough to write inline; tidwall/redcon if you need a full Redis server framework

• What you can enforce at proxy level once you can read queries

  • Reject queries missing a time filter / WHERE clause
  • Rate limit per client IP or per measurement/collection
  • Block specific commands or statements (FLUSHDB, DROP, DELETE without WHERE)
  • Restrict access to specific measurements / indexes / collections
  • Cost estimation before execution — run EXPLAIN first, reject if estimated rows exceed threshold (adds one extra round-trip but catches expensive queries that look innocent in text)

The Core Insight

HTTP-based DBs are just web services with domain-specific body formats. Any standard HTTP middleware — reverse proxies, API gateways, service meshes — can sit in front of them and inspect traffic without knowing anything about the DB.

TCP wire protocol DBs speak their own language. A proxy must implement or embed a parser for that binary protocol before it can read a single query. The upside: once you’ve parsed the framing, the query is just a string and the same filtering logic applies as for HTTP DBs.

The filtering logic itself — time bounds, rate limits, blocklists, cost estimation — is identical for both classes. The only difference is how much work you do to get the query into a readable form in the first place.

From my experience working at GreyOrange. Refactored my article a bit with help of GPT.

I was working on a problem statement to sync data from multi-tenant kafkas to central kafka. Upon research, I came across MirrorMaker2.

┌──────────────────────────────────────────────────────────────────────┐
│                        Central Kafka Cluster                         │
│   Topics: orders, payments, inventory  (aggregated from all tenants) │
└──────────────────────────────────────────────────────────────────────┘
         ▲              ▲              ▲
         │              │              │
    MM2 Instance   MM2 Instance   MM2 Instance
    (Tenant A)     (Tenant B)     (Tenant C)
         │              │              │
         ▼              ▼              ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Tenant A     │ │ Tenant B     │ │ Tenant C     │
│ Kafka        │ │ Kafka        │ │ Kafka        │
│ orders       │ │ orders       │ │ orders       │
│ payments     │ │ payments     │ │ inventory    │
└──────────────┘ └──────────────┘ └──────────────┘

MirrorMaker2 (introduced in Kafka 2.4) is a multi-cluster replication tool built on top of Kafka Connect. It replicates topics from a source cluster to a target cluster, tracking offsets, syncing consumer group checkpoints, and managing heartbeat topics.

It runs as a set of Kafka Connect connectors:

• MirrorSourceConnector — replicates topic data
• MirrorCheckpointConnector — syncs consumer group offsets
• MirrorHeartbeatConnector — emits liveness signals

By default, MM2 uses DefaultReplicationPolicy, which renames topics at the destination:

source.my-topic  →  clusterA.my-topic

With IdentityReplicationPolicy, the topic name is preserved (which we used in our setup):

source.my-topic  →  my-topic

This matters when:

• Downstream consumers should not need to know which cluster a message originated from
• You want a single consumer configuration pointing at central Kafka to consume the same topic name regardless of source
• You’re aggregating the same logical topic from multiple tenant clusters into one place

Config:

replication.policy.class=org.apache.kafka.connect.mirror.IdentityReplicationPolicy

Now, I have been working on testing MM2 (GCP managed not opensource). Multiple mm2 instances/connectors can be added in the GCP managed kafka connect cluster. Below are some of the scenarios and their answers, hopefully when I scale things and encounter more scenarios I will update the article.

• MM2 introduces inherent replication lag because it is a consumer + producer pipeline. Although for smaller traffic, I did not observe this.
• MM2 going down does not affect the tenant Kafka cluster or the central Kafka cluster. Both continue operating independently.
• On MM2 restart: MM2 reads its last committed offset from the source cluster (stored in the MM2 consumer group). It resumes from where it left off. Data is not lost if it was within rentention window of source kafka.
• When a new tenant kafka is added, we can add a new MM2 instance in the cluster (cluster doesn’t restart). Topic patterns are matched based on config, and they sync.
• Partition count scenarios:
  • Case 1: Tenant has FEWER partitions than Central: MM2 replicates messages from partitions 0–3 of the source. Partitions 4–7 on central will not receive any data from this tenant. They sit empty. If another tenant (Tenant B) also has orders with 4 partitions, its MM2 instance will also only write to partitions 0–3 — mixing data from both tenants in those 4 partitions while 4–7 remain empty. Messages are not lost, just unevenly distributed.

    Tenant A: orders (4 partitions)
    Central:  orders (8 partitions)
    

  • Case 2: Tenant has MORE partitions than Central: M2 maintains a strict 1:1 partition mapping. Considering below tenant(8)/central(4) topic partitions example. MM2 will attempt to automatically increase the Central topic’s partition count to 8 (if sync.topic.configs.enabled=true). If it lacks permissions to alter the topic, the connector task will crash when it attempts to write a record to a non-existent partition (e.g., partition 5, but for remaining partitions the syncing it will work as usual). Not that this can cause data skew in already existing partitions (i.e partition 1-4) due to other tenants incoming data, hence it has to be kept in mind.

    Tenant A: orders (8 partitions)
    Central:   orders (4 partitions)
    

    • From what I’ve read so far, assume this case: orders topic in tenant A kafka is bulky so we partition it to say 10. orders topic in tenant B kafka is light and we partition it to 2. In central kafka, the order topic, since it will have mix of messages from both tenants if I say initially created topic with 5 partitions. Now if I want my kafka-connect mirrormaker2 to basically sync messages across tenants from the topic into these 5 partitions probably using hash partitioning or something, then it is NOT possible. It will try to increase the central kafka topic partitions to 10 and sync, else won’t sync. We have to understand that MM2 is a replication tool, not repartitioning tool.

From my experience working at GreyOrange. Refactored my article a bit with help of GPT.

I was working on a service with a senior engineer that did some light processing and produced events to Kafka in the background. This component was a part of broader pipeline that had to be exactly-once semantics. Hence as part of work, I came across interesting producer configs, which I will discuss.

Although before that, a quick brush up of Kafka. I am assuming below Kafka configs, based on which I will discuss few things, pretty similar to a setup I was working on:

Three brokers, each pinned to a different GCP zone. Every partition has one leader and one follower — almost certainly in different zones.

• Kafka is a durable, ordered, high-throughput log on a network. You write messages into it (produce), and other systems read from it (consume) at their own pace.
• Topic: A named stream of messages. Think of it like a table name or a named channel. Example: warehouse.rack.size.
• Partition: A topic is split into N ordered sub-logs called partitions. Each partition is an append-only file on disk. Messages within one partition are strictly ordered; across partitions, there is no ordering guarantee.

Topic: warehouse.rack.size  (4 partitions)

Partition 0:  [msg1] [msg5] [msg9]  [msg13] ──► (append-only)
Partition 1:  [msg2] [msg6] [msg10] ──►
Partition 2:  [msg3] [msg7] [msg11] ──►
Partition 3:  [msg4] [msg8] [msg12] ──►

• Offset: Every message in a partition has a monotonically increasing integer ID called an offset — like an array index. A consumer remembers “I’ve read up to offset 42 in partition 2” and picks up from there on restart.
• Broker: A single Kafka server process. It stores partitions on disk, accepts produce requests, and serves fetch requests to consumers. In a cluster, each broker owns a subset of partitions.
• Producer: Any process that writes messages to Kafka.
• Consumer Group: A set of processes that together read a topic. Kafka assigns each partition to exactly one consumer in the group at a time so the group reads every message exactly once as a whole under normal operation, with individual consumers each handling a subset of partitions.

Consumer Group "analytics" reading topic with 4 partitions

  Consumer A ◄── Partition 0, Partition 1
  Consumer B ◄── Partition 2, Partition 3

  Add a third consumer → Kafka rebalances:
  Consumer A ◄── Partition 0
  Consumer B ◄── Partition 1
  Consumer C ◄── Partition 2, Partition 3

• Multi-Broker Cluster Architecture: A single broker is a single point of failure and a throughput ceiling. In production you run a cluster. Three reasons: fault tolerance, throughput, and storage capacity.

• Every partition has one leader and zero-or-more followers (replicas). All produce and consume traffic goes to the leader. Followers pull from the leader to stay caught up called replication.

Partition 0  (replication factor = 3)

  Broker 1 (LEADER) ──► Broker 2 (follower) ──► Broker 3 (follower)
       │                      │                       │
   writes go here       copies from leader      copies from leader
   reads go here

  If Broker 1 dies:
  Broker 2 or 3 is elected new leader → traffic shifts automatically

• Kafka spreads partition leaders evenly across brokers. With 12 partitions and 3 brokers, each broker leads roughly 4 partitions — distributing both CPU and network load.

3-broker cluster, 12 partitions:

  Broker 1: leads P0, P3, P6, P9    (also follows P1,P2,P4,P5,P7,P8,P10,P11)
  Broker 2: leads P1, P4, P7, P10
  Broker 3: leads P2, P5, P8, P11

In our 3-broker, RF=2 setup, each partition has 1 leader and 1 follower:
  Topic: orders  (4 partitions, replication factor: 2)
    Partition 0:  Leader → Broker 1 (zone-a)  |  Replica → Broker 2 (zone-b)
    Partition 1:  Leader → Broker 2 (zone-b)  |  Replica → Broker 3 (zone-c)
    Partition 2:  Leader → Broker 3 (zone-c)  |  Replica → Broker 1 (zone-a)
    Partition 3:  Leader → Broker 1 (zone-a)  |  Replica → Broker 3 (zone-c)

• Historically Kafka used ZooKeeper — a separate cluster — to store metadata: which broker is controller, which partitions have which leaders, ISR lists, etc. This meant running and maintaining a separate ZooKeeper ensemble alongside every Kafka cluster. Modern Kafka (3.x+) replaces this with KRaft — Kafka’s own Raft-based consensus built directly in. Brokers elect a controller amongst themselves. No external dependency, simpler operations, and faster metadata operations.

• A Produce request’s journey through a cluster

1. Producer asks any broker: "who leads partition 2 of topic warehouse.events?"
2. Broker responds: "Broker 3 leads that partition"
3. Producer connects directly to Broker 3, sends the compressed batch
4. Broker 3 writes the batch to its local log
5. Broker 1, Broker 2 (followers) pull the batch and write to their local logs
6. Once ISR quorum acks, Broker 3 replies to producer: "OK, offset 10042"

Producer ──► Broker 3 (leader)
                  │
                  ├──► Broker 1 (follower, acks)
                  └──► Broker 2 (follower, acks)
                  │
                  └──► "OK, offset 10042" ──► Producer

acks - Durability vs Throughput

• When a producer sends a record, it can ask Kafka for different levels of confirmation before considering the write “done.” This is the acks setting.
  • acks=0: Fire and Forget. The producer sends the message and does not wait for any acknowledgement from the broker. As soon as the message hits the network socket buffer, the producer considers it sent. Kafka may or may not have written it to disk. If the broker crashes between receiving and persisting, the message is gone. The producer has no idea whether it was received. Returned offset is always -1 (meaningless). Retries do nothing — the producer can’t know what failed.
    • When to use: Metrics, logs, or telemetry where occasional loss is acceptable and maximum throughput is the goal. Never for financial, transactional, or auditable data.
    • Throughput: Maximum possible — no network roundtrip for acks.
  • acks=1: Leader Acknowledgement Only. The leader writes the record to its local log and immediately acknowledges the producer. It does not wait for followers to replicate. The risk — “leader fails after ack, before follower replicates”:. On Spot VMs, evictions are sudden, the VM can vanish with little warning.
    • When to use: Medium-criticality streams where some data loss is tolerable but throughput matters.
    • Throughput: High — one network roundtrip, no follower coordination.

  • acks=all (or acks=-1): Full ISR (in-sync replica) Acknowledgement. The leader waits until all in-sync replicas have written the record before acknowledging the producer. This is the strongest durability guarantee Kafka offers. The critical companion — min.insync.replicas: This sets the minimum number of in-sync replicas that must be available for Kafka to accept a write when using acks=all. For example:

    acks=all
    min.insync.replicas=2
    With replication.factor=3, a partition normally has three replicas:
    
    Broker A   Broker B   Broker C
       │          │          │
     leader     replica    replica
    
    If Broker C fails, the ISR can shrink to two replicas: ISR = {Broker A, Broker B}. Because ISR size (2) >= min.insync.replicas (2), Kafka can continue accepting writes with acks=all.
    
    If another replica falls out of the ISR: ISR = {Broker A}. Then ISR size (1) < min.insync.replicas (2), so Kafka rejects the write rather than acknowledging a record that has fewer than the required number of in-sync replicas.
    
    This is why a common production configuration is as described earlier: replication.factor=3, min.insync.replicas=2, acks=all. It allows the cluster to continue accepting writes after losing one replica while still requiring two in-sync replicas for acknowledged writes.
        
    Important for an RF=2 cluster: If your topic has replication.factor=2 and min.insync.replicas=2, losing either replica leaves only one replica in the ISR, so acks=all writes will fail until the ISR is restored. RF=2 therefore provides less write availability during a broker failure than RF=3 with min.insync.replicas=2.
    

    • When to use: Any data where loss is unacceptable. Required for idempotent and exactly-once producers.
    • Throughput: Lower than acks=1 — the roundtrip includes follower replication latency. In a same-region multi-zone cluster, this is typically 5–20ms added latency.

batch.size and linger.ms — Throughput Tuning

• The producer accumulates records into batches before sending. Two configs control when a batch is flushed:
  • batch.size — maximum bytes in a single produce batch. Once a batch hits this size, it is sent immediately.
  • linger.ms — how long the producer waits to fill a batch before sending even if it isn’t full yet. Default is 5 in Kafka 4.x. In older versions it used to be 0ms (send immediately).
  • Setting linger.ms to 5–20ms dramatically improves throughput on high-volume topics by allowing more records to accumulate per batch, at the cost of tiny and usually imperceptible latency increases.

linger.ms=0:
  Record arrives → batch sent immediately (low latency, low throughput)

linger.ms=10:
  Record arrives → wait up to 10ms → more records accumulate → larger batch sent
  (slightly higher latency, dramatically better throughput)

max.in.flight.requests.per.connection — Parallelism vs Ordering

• How many produce requests can be in-flight simultaneously to one broker. More in-flight requests = more parallelism = higher throughput. But there is a catch:
• Without idempotence: if you send batches A and B, A fails and retries after B succeeds, the partition log ends up with B before A — ordering is violated.
• With the idempotent producer, this is capped at 5 — Kafka’s enforced maximum that still maintains ordering via sequence numbers. You get parallelism without reordering risk.

# With idempotent producer (required):
max.in.flight.requests.per.connection=5   # max allowed; kafka enforces ordering via seq numbers

# Without idempotence (higher throughput, ordering risk):
max.in.flight.requests.per.connection=10+  # reordering possible on retry

ProduceRequestTimeout — Single RPC Timeout

• How long the producer waits for a response to a single produce RPC. If the broker does not respond in this window, the request fails and is retried. Think of it as: “how patient am I with one individual network call?”
  • Too low → spurious timeouts under transient load spikes trigger unnecessary retries.
  • Too high → a stuck broker ties up the producer for a long time before retrying.

request.timeout.ms=5000   # 5 seconds per RPC attempt

RecordDeliveryTimeout — Total Retry Budget per Record

• Kafka clients retry failed batches automatically. delivery.timeout.ms sets the upper bound on how long the producer will attempt to deliver a record, including retries. It is not a fixed number of retry attempts. The relationship between the two:
  • request.timeout.ms = timeout for one attempt
  • delivery.timeout.ms = total budget across all attempts
  • delivery.timeout.ms must be ≥ (request.timeout.ms + linger.ms)

t=0:00  Record enqueued, Kafka unreachable
t=0:05  Retry 1 → still unreachable (RPC timeout: 5s)
t=0:15  Retry 2 → still unreachable
t=0:30  Retry 3 → ...
...
t=2:00  delivery.timeout.ms expires → ErrRecordTimeout
        → record is dropped, callback fires with error

delivery.timeout.ms=120000   # 2 minutes total retry window

MaxBufferedRecords — In-Memory Cap

• The producer keeps an internal in-memory buffer of records waiting to be batched and sent. Depending on your client implementation you can buffer based on size or number of records.
• When the buffer is full (e.g., Kafka is unreachable and records pile up), new produce attempts are rejected immediately — no blocking, no OOM.
• Without a cap, a slow or dead broker causes the producer to accumulate records in memory until the process OOMs. The cap trades some data loss for process stability — an acceptable trade-off for most systems. Eg:

HTTP request → produce() → [internal buffer, max N records]
                                    │
                                    └──► batcher → broker → ack
If buffer fills (Kafka unreachable):
  produce() → ErrMaxBuffered → caller handles the drop

ProducerBatchCompression — Compress Before Sending

• The producer compresses entire batches (not individual messages) before sending to the broker. zstd achieves better ratios than gzip/snappy at similar CPU cost. Compressing at batch level amortises the CPU cost across many records per batch.
• Note: also set compression.type=producer on the kafka broker topic. This tells the broker: “store batches exactly as I sent them, do not recompress.” Without this, the broker may decompress your zstd batch and re-compress with the cluster default codec — wasting CPU on both ends and potentially changing the wire format for downstream consumers.

compression.type=zstd   # on the producer

Idempotent Producer: Eliminating Duplicates on Retry

• Every messaging system makes one of three promises about delivery.

• Even with acks=all, there is a subtle problem: producer retries create duplicates.

Producer                    Broker
   │                            │
   ├──── Batch (seq=0) ────────►│  broker writes to log, prepares ack
   │                            │
   │         [network blip]     │
   │◄──── (ack never arrives)   │
   │                            │
   │  "I didn't get an ack,     │
   │   I'll retry"              │
   ├──── Batch (seq=0) ────────►│  ← broker writes it AGAIN (duplicate!)
   │◄──── "OK" ─────────────────│

• The producer cannot distinguish “ack was lost” from “write failed.” So it retries. The broker, without idempotence, has no memory of the first write. You get a duplicate record.
• Idempotent producer tries to solve this, enabled via: enable.idempotence=true (modern clients enable this by default when acks=all). It works by:
  • The broker assigns the producer a Producer ID (PID) — a unique integer for this producer instance’s lifetime.
  • Each partition gets its own monotonically increasing sequence number.
  • Every batch is stamped with (PID, partition, sequence_number).
  • If the broker receives a batch with a sequence number it has already written, it accepts the request (returns success) but silently discards the duplicate.

Producer (PID = 42)              Broker
     │                               │
     ├── Batch (seq=0) ─────────────►│  written; seq=0 recorded for PID 42
     │          [ack lost]            │
     ├── Batch (seq=0) ─────────────►│  "already have seq=0 for PID 42 — discard"
     │◄────── "OK" ──────────────────│  producer thinks it succeeded (it did, the first time)
     │                               │
     ├── Batch (seq=1) ─────────────►│  written; seq=1 recorded
     │◄────── "OK" ──────────────────│

• Caveat: PID resets on producer restart: The PID is assigned fresh on each producer startup. If the process crashes and a new instance starts, it gets a new PID. The broker will not recognise retry attempts from the new instance as duplicates of the old one. A record that was in-flight at crash time could be written twice — once by the old instance (before crash) and once by the new instance (on startup retry). For most telemetry and event-streaming workloads, a single duplicate point on a process restart is acceptable — it is a known, bounded limitation of the idempotent producer without full transactions.
• Idempotent producer gives you exactly-once from producer to broker within a single session. It does not give you:
  • Exactly-once across multiple topics simultaneously
  • Exactly-once from broker to consumer (the consumer commits its own offsets separately)
  • End-to-end exactly-once across the full pipeline (producer → Kafka → consumer → database)
  • Exactly-once across producer restarts (PID resets)

For those, you need Kafka Transactions. But I read online, that if related configs are not tuned properly then Kafka transactions hurt throughput.

Eg: Transactions: Atomic Multi-Partition Writes
Producer:
  beginTransaction()
    write(topic=A, partition=0, message=M1)
    write(topic=B, partition=2, message=M2)
  commitTransaction()   ← both M1 and M2 become visible atomically
  # or
  abortTransaction()    ← neither M1 nor M2 becomes visible

• A transactional producer is assigned a stable transactional.id that persists across restarts. When the producer restarts, it uses the same transactional.id to recover transactional state and fence the previous instance, providing transactional guarantees across restarts.

  • Producer config for full exactly-once semantics:

    enable.idempotence=true
    transactional.id=my-unique-producer-id    # stable across restarts
    acks=all
    

  • Consumer config — only read committed messages:: Without read_committed, consumers see messages from uncommitted (in-flight) transactions — which may later be aborted, causing phantom reads: the consumer processes a message that was never actually committed.

    isolation.level=read_committed
    

Other info:

• Partition Key — Routing and Ordering: The record key controls which partition a record lands in. Records with the same key always go to the same partition (for a fixed partition count). Records with a null key are distributed by the client.
  • Keyed records: gives ordering guarantees — all records for the same key arrive at the same partition in order. The risk: if one key generates disproportionately high traffic, one partition gets disproportionate load — the hot partition problem. A more granular key (e.g., user_id + metric_type) distributes load across partitions while still giving ordering within a category.
  • Null-keyed records: since Kafka 2.4, the default partitioner for null-keyed records is the sticky partitioner, not round-robin. It writes to one partition until a batch fills or linger.ms expires, then rotates. This meaningfully improves batch fill rates on high-volume keyless topics over round-robin. If you need key identity for consumers (for routing or filtering) but don’t want it to affect partitioning, put it in a record header instead of the key.
• A producer does not fail immediately when a broker becomes unreachable. Retries and buffering absorb short outages transparently.
• Two decisions made at topic creation that you mostly can’t undo:
  • Partition count — more partitions = more consumer parallelism, but also more replication overhead at the cluster level (100 partitions × 3 replicas = 300 replica slots to manage). Partition count should be sized from expected producer throughput, consumer parallelism, recovery requirements, and broker capacity. Kafka can add partitions to an existing topic. It cannot remove them. If you auto-create topics programmatically, guard against a config service returning a nonsense value (like 50,000 partitions) with a hard cap, and make sure your reconciliation logic only ever increases partition count, never decreases.
  • Replication factor — typically 3 in production. With min.insync.replicas=2 and acks=all, you can lose one broker and keep writing without pause.
• Running Kafka on Spot VMs is economical but introduces real eviction risk.
• When your process shuts down, records may still be sitting in the buffer or a linger window, unsent. The correct pattern:
  • Stop accepting new records.
  • Call flush() with a reasonable timeout — give in-flight records a chance to drain.
  • Call close(). Ensure the client library doesn’t skip pending callbacks/flush(), as the object is being torn down.

From my experience working at GreyOrange. Refactored my article a bit with help of GPT.

I had the fortune of working in data teams of different companies, and one thing that I noticed is the evolution of setting up OSS (open-source software) on cloud.

Tldr; From VMs to Docker to Helm-Kubernetes

For instance, let’s talk about Airflow. Apache Airflow is an open-source workflow orchestration platform. You define data pipelines as DAGs (Directed Acyclic Graphs) in Python — each DAG describes what tasks to run, in what order, and on what schedule. Airflow handles scheduling, execution, retries, logging, and a web UI for visibility.

It has a few core components like scheduler, workers, webserver/apiserver, etc.

When setting up for the first time on cloud - the deployment model for all these components have changed over years. I would divide it as:

1) VM era:

Teams would spin up one or more virtual machines and install Airflow directly on the OS.

# Typical setup steps on Ubuntu/Debian
sudo apt-get install python3-pip
pip install apache-airflow
export AIRFLOW_HOME=~/airflow
airflow db init
airflow users create --username admin --role Admin ...
airflow webserver --port 8080 &
airflow scheduler &

Workers would be started similarly, either on the same machine or additional VMs. The metadata database (Postgres) was either installed on the same VM or pointed to an external managed instance.

Managing looked like:

• DAGs were deployed by SSHing into the VM and copying .py files into $AIRFLOW_HOME/dags/
• Dependencies were installed with pip install directly on the VM, shared across all DAGs
• Config changes required editing airflow.cfg and restarting processes via systemctl or screen sessions
• Scaling workers meant provisioning a new VM, installing everything again manually, and hoping nothing drifted
• Upgrades were painful: pip install --upgrade apache-airflow could break existing DAGs or dependencies

Pros:

• Simple mental model — it’s just a process on a server
• No container or orchestration knowledge required
• Easy to debug: SSH in, look at logs, check processes

Cons:

• No isolation: all DAGs share the same Python environment; one DAG’s pip install can break another’s
• Manual scaling: adding a worker means provisioning and configuring a new VM by hand
• Config drift: servers diverge over time; “works on my VM” is a real problem
• No HA out of the box: if the scheduler process crashes, nothing restarts it automatically
• Dependency hell: conflicting package versions across DAGs are a constant headache
• Upgrades are risky: a bad pip upgrade can take down the entire Airflow installation

2) Container era:

The era of Docker came. Instead of installing Airflow on a host OS, you’d pull (or build) a Docker image containing Airflow and all its dependencies, then run it as a container.

# Custom Airflow image with extra packages
FROM apache/airflow:3.x.x
RUN pip install pandas boto3 google-cloud-bigquery
COPY dags/ /opt/airflow/dags/

# Running Airflow webserver in Docker
docker run -d \
  -p 8080:8080 \
  -e AIRFLOW__CORE__EXECUTOR=LocalExecutor \
  -e AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@postgres/airflow \
  -v $(pwd)/dags:/opt/airflow/dags \
  apache/airflow:3.x.x webserver

Airflow project provided an official docker-compose.yaml as well to run entire stack.

Managing looked like:

• DAGs were mounted as a volume into the container, or baked into a custom Docker image
• Custom dependencies went into a custom Dockerfile extending the base Airflow image
• Config was passed as environment variables (AIRFLOW__SECTION__KEY=value)
• Scaling meant increasing container replicas on a host; scaling across machines required additional orchestration
• Upgrades meant pulling a new image tag and running docker-compose up -d

Pros:

• Reproducibility: same image runs the same way on any machine
• Isolation: Airflow’s Python environment is sealed inside the container
• Easier upgrades: swap image tags, rebuild, redeploy
• Local dev: any engineer can run the full Airflow stack on their laptop with docker-compose up
• Simpler dependency management: pip installs go into the Dockerfile, not the host

Cons:

• Still manual scaling: docker-compose is single-host; scaling across machines requires extra tooling (Docker Swarm, or moving to Kubernetes)
• Not production-grade HA: docker-compose has no self-healing, no rolling restarts, no health-based rescheduling
• DAG deployment is still a concern: you either remount volumes (operational complexity) or rebuild and redeploy images (slow CI loop)
• No cloud-native integration: no automatic secrets injection, no auto-scaling, no native logging to cloud log sinks
• CeleryExecutor needs MessageBroker: Like RabbitMQ/Redis extra service to manage and keep healthy

3) K8s era:

We can now setup Airflow using helm charts on K8s. K8s provides:

• Self-healing: crashed containers are automatically restarted
• Horizontal scaling: add more pods with a single command or HPA rule
• Rolling deployments: update with reduced deployment downtime
• Resource management: CPU and memory limits per component
• Native secrets/config management: Kubernetes Secrets and ConfigMaps
• Cloud integration: persistent volumes, load balancers, IAM, logging — all first-class

But deploying a complex multi-component application like Airflow on Kubernetes by hand (writing Deployments, Services, PVCs, ConfigMaps, Secrets for each component) is repetitive and error-prone. That’s where Helm comes in. Helm is the package manager for Kubernetes. A Helm chart is a collection of templated Kubernetes manifests packaged together with configurable values.

We can think of it like apt or pip but for Kubernetes applications.

# Install Airflow on Kubernetes with Helm
helm repo add apache-airflow https://airflow.apache.org
helm install airflow apache-airflow/airflow \
  --namespace airflow \
  --create-namespace \
  -f values.yaml

One command which you could run from a bastion machine that has access to the K8s namespace and Airflow’s entire stack — scheduler, webserver, workers, triggerer, metadata DB connection, ingress, RBAC — deployed on that namespace.

Notice in the command, we are passing a values.yaml file, it basically has our complete Airflow deployment config, it may look like below for example:

# Airflow image — use official or custom
images:
  airflow:
    repository: your-registry/custom-airflow
    tag: "3.x.x"
    pullPolicy: IfNotPresent

# Executor choice
executor: KubernetesExecutor   # or CeleryExecutor, LocalExecutor

# Webserver
webserver:
  replicas: 2
  resources:
    requests:
      memory: "1Gi"
      cpu: "500m"
    limits:
      memory: "2Gi"
      cpu: "1000m"

# Scheduler
scheduler:
  replicas: 1
  resources:
    requests:
      memory: "2Gi"
      cpu: "1000m"

# Workers (CeleryExecutor)
workers:
  replicas: 3
  resources:
    requests:
      memory: "4Gi"
      cpu: "2000m"

...Similarly database, env variables, and other details... 

There are 3 common patterns to deploy/update DAGs via helm:

• Git-sync sidecar: A sidecar container runs alongside the scheduler and workers, continuously polling a Git repo and syncing DAGs to a shared volume. No manual file copying, no image rebuilds for DAG changes.
• GCS bucket mounted via GKE FUSE (Cloud-native alternative): On GKE (Google K8s), you can mount a Google Cloud Storage bucket directly into Airflow pods as a filesystem using GCSFuse (via the GKE Cloud Storage FUSE CSI driver). DAGs live in GCS, are mounted at /opt/airflow/dags inside the pod, and updates to the bucket are visible to pods without any restart or re-sync delay generally. You could sync DAGs to GCS buckets via any simple CI pipeline.
• Bake DAGs into custom image: Build a custom Docker image that includes your DAGs. Every DAG change requires a new image build and Helm upgrade. Slower iteration but more reproducible.

Similarly, upgrading airflow via helm is simple as well.

Pros:

• Self-healing: Kubernetes restarts crashed pods automatically
• Rolling updates: zero-downtime upgrades via Helm upgrade
• GitOps-friendly: values.yaml in Git; CD pipelines trigger helm upgrade
• Per-task isolation: KubernetesExecutor gives each task its own pod and Python environment
• Auto-scaling: HPA can scale workers based on external/custom metrics
• Cloud-native: native secrets, PVCs, service accounts, IAM integration
• Resource governance: CPU/memory limits per component prevent one runaway DAG from killing the scheduler
• Multi-environment parity: same Helm chart, different values.yaml per env (dev/staging/prod)

Cons:

• Kubernetes knowledge required: your team must understand pods, PVCs, RBAC, ingress
• Complexity of values.yaml: the official Airflow Helm chart has hundreds of configurable values
• Cold start latency (KubernetesExecutor): each task (if using K8sExecutor) incurs pod scheduling latency (~5–30s) vs. a warm Celery worker
• Spot/preemptible nodes need care: if the scheduler pod is on a Spot node, eviction causes a brief scheduling pause — use node affinity or priority classes to keep the scheduler on stable nodes
• Log management: logs are ephemeral in pods; must configure remote logging (GCS, S3, Elasticsearch) from day one

Hence, for teams setting up OSS Airflow rather than using any managed service, helm is preffered way. Note that a lot of other OSS like say: Superset, Trino, Metabase, their helm charts exist as well, and can be setup similarly.

From my experience working at Punch. Refactored my article a bit with help of GPT.

I was building a system from scratch using a Go service with a NATS + JetStream stack to simulate and replay financial markets in a staging environment.

At a high level, live market ticks were persisted into JetStream streams, and the replay service consumed this historical data and re-published it onto NATS while preserving the original inter-tick timing. Replays could be triggered either through cron jobs or internal APIs, allowing downstream systems like charts and trading engines to behave similarly to live market conditions.

Financial markets such as the National Stock Exchange of India (NSE) and BSE Limited operate between 9:15 AM and 3:30 PM, during which financial instruments like stocks, futures, and options are actively traded.

The requirement was to enable replaying the market activity of a specific trading day outside market hours or during holidays. For example, if a team wanted to test a new trading or charting feature against real historical market conditions, the system could recreate the entire market session on demand in a controlled staging environment.

When replaying historical market data for testing or simulation, simply publishing messages as fast as possible is not enough.

The inter-tick timing (the delay between consecutive market updates for stocks/F&O instruments) had to closely match the original market behavior to ensure charts, trading logic, and downstream systems behaved similarly to production.

I will discuss a few learnings related to building high-precision replay engine.

The Problem: time.Sleep(5 * time.Millisecond) Is Not Precise. It guarantees a minimum sleep duration, not an exact wake-up time. The actual wake-up depends on: OS scheduler, timer granularity/resolution, CPU load, runtime scheduling, context switching, power-saving behavior, etc.

For high-frequency systems like market feeds, this can create timing errors. Eg:

Real market tick gap: 200µs
Replay gap using Sleep: 1ms

The replay timing becomes significantly slower than the original market behavior. A replay system must preserve these intervals. Eg:

Inter-tick gaps must be preserved:

T2 - T1 = 200µs
T3 - T2 = 250µs

The Solution: Busy Spinning. Instead of sleeping, the CPU can continuously check the clock until the desired time arrives. This technique is called busy spinning. Eg:

target := time.Now().Add(delta)

for time.Now().Before(target) {
}

The loop repeatedly checks the current time until the target time is reached. Conceptually it works like this:

check time
check time
check time
check time
exit when target reached

Why Busy Spinning Is Precise: Modern systems implement time.Now() very efficiently. On Linux, Go uses: clock_gettime(CLOCK_MONOTONIC) via vDSO (Virtual Dynamic Shared Object). This means: No system call, No kernel context switch, Extremely fast execution. Typical cost: ~20–40 nanoseconds.

So if we spin for 500 microseconds (theoretically):

500µs = 500,000ns
500,000 / 20 ≈ 25,000 iterations

The CPU checks the clock roughly 25,000 times, making the wait extremely accurate. Even with busy spinning, perfect determinism is not guaranteed because OS scheduling, goroutine preemption, and GC pauses can still introduce latency. However, it is significantly more precise than relying solely on time.Sleep() for sub-millisecond timing.

Downside of Busy Spinning: It consumes CPU.

The Hybrid Strategy: Sleep + Spin: A better approach in my use case was to combine both techniques. Strategy: Sleep most of the time; Busy-spin near the target time.

This reduced CPU usage theoretically, although in practice I did not observe a significant difference in infrastructure metrics, likely because most inter-tick delays were already in the microsecond range.

Below is a simplified version of the approach used in the replay engine I worked on:

const defaultBusySpinThreshold = 2 * time.Millisecond
const sleepMargin = 1 * time.Millisecond

func WaitPrecise(delta time.Duration) {

    if delta <= 0 {
        return
    }

    target := time.Now().Add(delta)

    if delta > defaultBusySpinThreshold {

        sleepDuration := delta - sleepMargin

        if sleepDuration > 0 {
            time.Sleep(sleepDuration)
        }
    }

    for time.Now().Before(target) {
    }
}

How it works:

Example: Waiting for 10ms

sleepDuration = 10ms - 1ms = 9ms
So: 
|-------- sleep --------|-- spin --|
0                       9ms       10ms

CPU usage occurs only during the final 1ms. This avoids scheduler overshoot while preserving timing accuracy.

Also (as observed in sample code), instead of repeatedly sleeping for fixed intervals, we compute a target timestamp directly. This avoids cumulative drift, where small scheduling inaccuracies compound over thousands of ticks and eventually drift the replay by seconds.

Busy spinning is appropriate when: Precision below 1ms is required; Timing accuracy matters; Wait durations are short; CPU availability is acceptable. Else time.Sleep() is sufficient.

From my experience working at Punch. Read docs, youtube, GPT explanations.

Learning Golang, gRPC, Protobuf. I may take occasional detours as a part of understanding things ‘properly’.

Fundamentals

Nuances in Go:

• Strict compile time checks, i.e: If you have declared a variable or imported a package in code, you MUST use it. Unused entities in code will lead to compile time errors thrown.
  • Compiled vs Interpreted language:
    • Your CPU runs instructions in 0s and 1s. So every instruction defined via a language must eventually become machine code. The difference is WHEN and HOW that translation happens.
    • Compiled languages Flow: Source code → Compiler → Machine code executable → Run the executable. Compiler: Reads the entire program; Does static analysis; Optimizes globally. Interpreted languages Flow: Source code → Interpreter → Execute line by line. Interpreter: Reads one statement, Executes it immediately, Moves to the next. Compiled languages came first. Note that today almost all modern “interpreted” languages compile internally.

    • What did interpreted languages solve?: Early computers were painful; Compilation took minutes to hours hence debugging meant: Write code, Compile, Run, Crash, Repeat. Interpreters solved this: Immediate feedback, Interactive programming (REPL), Dynamic behavior. For example:

      Eg1: — Calculator 
      In Python (Interpreted, can do interactive exploration): 
      >>> 10 * 3
      30
      >>> 10 * 3 + 5
      35
      >>> (10 * 3 + 5) / 7
      5.0
      In C (Compiled): 
      - Write the code first 
      #include <stdio.h>
      int main() {
          printf("%d\n", (10 * 3 + 5) / 7);
      }
      - Then run gcc calc.c -> ./a.out. 
      > Interpreter lets you think with the computer, compiler forces you to prepare a program first. Interpreters were not invented to replace compilation rather solve human feedback speed, not program execution speed.
      > Interpreted languages shine when you don’t yet know what the program should be, if you know, then Compiled language/Interpreted language would anyways work in similar way. 
            
      Eg2: Parsing unknown / messy data
      In Python: 
      import json
      with open("data.json") as f:
          data = json.load(f)
      type(data)
      len(data)
      data[0].keys()
      You inspect:
      >>> data[0]["user_id"]
      >>> data[0].get("timestamp")
      >>> [x for x in data if "error" in x]
      You discover the data while writing code.
      In C you must: 
      Decide struct layout upfront, Handle parsing manually, Recompile every structural change, Print + inspect. C forces decisions early. Python lets decisions happen late.
      

• Strong + static typing (but with inference).
  • It is statically typed language, i.e: The type of every variable/expression is known before the program runs. Dynamic typic means the type of variable is know during runtime of program.
  • Go gives capability to infer the type of variable even if you don’t mention the type in code, during compile time, using it’s type-inference. Eg: Usual code eg: var x int = 10. But if you don't mention 'int', it will still be able to infer the type during compile time, i.e: var x = 10.

• If you don’t assign a value to variable, Go assumes default values, it never leaves variables uninitialized. Eg:

  | Type    | Zero value |
  | ------- | ---------- |
  | int     | 0          |
  | string  | ""         |
  | bool    | false      |
  | pointer | nil        |
  | slice   | nil        |
  | map     | nil        |
  

• No semicolons (mostly) to end code in line. Go takes care of it. Eg: var x int = 10 is fine, no need for var x int = 10;.
• Explicit conversions must be done if required in values. Eg: var y float64 = float64(x).
• if / for / switch need no parentheses.
• Only ONE loop keyword: for. No while, no do-while.

• Functions can return multiple values.

  func divide(a, b int) (int, error) {
    if b == 0 {
        return 0, errors.New("divide by zero")
    }
    return a / b, nil
  }
  

• Error handling is explicit. Eg:

  result, err := divide(10, 2)
  if err != nil {
      return err
  }
  

• No classes, no inheritance. Go uses: structs, interfaces, composition.

• Interfaces are implicit. Eg:

  type Reader interface {
      Read() string
  }
  If a struct has Read(), it automatically implements Reader. No need to use a keyword like implements which we do like in Java. 
  

  • Pointers but no pointer arithmetic. Like: p := &x, is fine, but can't do things like: p++ // illegal. Some more info on pointers:

    var x int = 10
    var y *int
    y = &x
    

    Expression	Meaning	Type	Value
    x	normal variable	int	10
    &x	address of x	*int	memory address
    y	pointer to x	*int	address of x
    *y	value at address y	int	10
    &y	address of pointer y	**int	memory address

    *int      → pointer to int
    *string   → pointer to string
    *float64  → pointer to float64
    *struct{} → pointer to struct
    *int guarantees dereferencing gives an int ~ Type safety (prevents invalid memory access)
    

    • Interesting read on pointers.

• Arrays vs slices: Arrays are fixed, Slices in Go are dynamic sized (mostly used).

  var a [3]int // fixed size of 3 - arr
  var s []int // dynamic - slice
  
  Note:
  1. nil slice ≠ empty slice. Eg: 
     var s []int     // nil
     s := []int{}    // empty
  2. To add elements in slice, eg: s = append(s, "hello")
  3. Conversions:
       Array to Slice:
         a := [5]int{1, 2, 3, 4, 5}
         s := a[:] // Could also have partial slice [1:4]
       Slice to Array
         s := []int{1, 2, 3, 4, 5}
         var a [5]int
         copy(a[:], s)
  

• Maps must be initialized

  var m map[string]int
  Doing: m["a"] = 1 // Wrong
  Rather: m := make(map[string]int)
  

  Note:

  new	make
  allocates memory	allocates + initializes
  returns pointer	returns value
  rarely used	commonly used

• func init() in Go is a special function that runs automatically during a package’s initialization, before any other functions in the package are called, including main(). It’s used for setup and configuration tasks.

• No function overloading or hardcore OOPs kind of concept in Go.

  Ex: 
  Java style OOP:
    class User {
        private String name;
        User(String name) {
            this.name = name;
        }
        public String greet() {
            return "Hi " + name;
        }
    }
  Go style OOP:
    type User struct {
        name string
    }
    func NewUser(name string) *User {
        return &User{name: name}
    }
    func (u *User) Greet() string {
        return "Hi " + u.name
    }
    
    > Creating a User value (without pointers) can be done like: u := User{name: "Suraj"}.
    > Now, coming back to our pointer OOP example:
       User{name: name} → creates a User value.
       & → takes its address.
       Result type → *User (pointer to User).
       Memory picture: 0x1000 ─▶ User{name: "Suraj"}.
    > In the (u *User) the receiver function, u is a pointer, u.name automatically dereferences the pointer (Go does this for you, else you would've to write like: (*u).name)
    > There is no separate “address type”. The only way to represent an address is with a pointer type (*User). Go does NOT allow raw memory addresses like C: return 0x7ffeefbff5a8; Only return &User{name: "Suraj"}. Can imagine as pointer types being safe abstraction over addresses.
  

Phase 1

• Keywords used:
  • Declarations: package, import, var, const, type, func
  • Control flow: if, for, switch, select
  • Concurrency: go, chan
  • Memory/lifecycle: new, make, defer
  • Error/exit: panic, recover, return

• Variable declarations:

  For Single variable:
  1. var x string = "" -> Used and declared inside/outside (global variable) functions; Declares + Assigns value
  
  2. var x = "" -> Used inside/outside functions; Go infers 'x' is a string during compile time (Go is statically typed language); Declares + Assigns value
  
  3. x := "" -> Used ONLY inside functions; Declares + Assigns value; It's var only, not const (we learn about this later)
  
  4. x = "hello" -> Used inside/outside functions; Assigns ONLY, hence 'x' must exist already
  
  For Multiple variables:
  1. 
  var name1, name2 string
  var age int
  var isAdmin bool
  name1 = "foo"
  name2 = "bar"
  age = 25
  isAdmin = false
  
  2. 
  var (
      name string = "suraj"
      age int = 25
      active bool = true
  )
  
  3. (type-interface taking care)
  var name, age, active = "suraj", 25, true
  
  4. 
  func main() {
      name, age, active := "suraj", 25, true
      _, _, _ = name, age, active
  }
  _ -> It is a blank identifier. Since Go requires every declared variable MUST be used, blank identifier flags to compiler that the variable exists, not deliberately using it, but may use in future. 
  

• Data Structures in Go:
  • Keywords:
    • var: mutable, package-level declaration
    • const: immutable
    • type: define new types, used for: Structs, Interfaces, etc.
    • func: functions
    • import: dependency management and package: namespace
      • A namespace is a named logical container that groups identifiers (functions, variables, types) so they don’t clash with others. In Go, packages are namespaces.

      • If a function/variable/struct/interface/const, etc., variable is capitalized - Means access is public, if smallcase then access is private (accessible only inside same package)

        func CreateUser() User {   // public function
            return User{Name: "Suraj", Age: 10}
        }
        func createUser() User {   // private function
            return User{Name: "Suraj", Age: 10}
        }
        

  • Data Structures:

    The type is always on the RIGHT side, if not inferring. Eg: var x int, var s []string, var m map[string]int
    
    Normal variables ~
    > x := 10
    > var x string = ""
    > var name string
      name = "Suraj"
    
    Arrays (Fixed size) ~ 
    > var a [3]string
      a[0] = "a" 
    > x := [3]int{1, 2, 3}
    
    Slices (Dynamic size) ~
    > s := []int{1, 2, 3} // or 
    > s2 := []string{}
      s2 = append(s, "hello") // append is only in slice, not arrays
    > arr := [5]int{1, 2, 3, 4, 5}
      s := arr[1:4] // [2 3 4] // slice from an array 
    
    Maps (key → value) ~
    > m := map[string]int{
            "apple":  10,
            "banana": 20,
          }
      price := m["apple"]
      // Note: map[string]interface{} - A map whose key is a string and whose value can be ANY type (int, float64, slice, etc.). Often used. 
    
    Pointers ~ 
    > x := 10
      p := &x   // pointer to x
    
    Structs ~ Holds data
    > type User struct {
          Name string
          Age  int
      }
      u := User{
          Name: "suraj",
          Age:  25,
      }
      fmt.Println(u.Name) // Capitalized, hence public access from all packages, if lowercase letters named in struct then private access.
      > Struct tags: They are defined as key-value pairs enclosed in backticks `` immediately following the field declaration. They are small pieces of metadata. They are ignored by normal Go code execution but are highly useful for tasks like data serialization, database mapping, and validation - which third-party libraries leverage, like Viper. Ex: 
        type User struct {
            Name     string `json:"user_name" db:"name,unique"`
            Age      int    `json:"age,omitempty"`
            Password string `json:"-"`
        }
    
    Interface (Projects behaviour, unlike struct) ~
    > type Speaker interface {
        Speak() string
      }
      // Any type that has Speak() string automatically implements Speaker interface, no need to use stuff like override, etc. Use eg: 
      func (p Person) Speak() string {
          return "Hello, I am " + p.Name
      }
    

    • Functions & Methods
      • Function: standalone, not tied to any type. Method: function attached to a type (receiver), called on a value.
      • Methods give behavior to structs; functions are just helpers. Eg:

      Normal/Free function:
      func add(a int, b int) int {
        return a + b
      }
      
      Variadic functions: Accepts any number of arguments of the same type. Can be used for multiple arguments or slices.
      // Variadic Mode: ...int → becomes []int inside function. Note that only last parameter can be variadic.
      func sum(nums ...int) int {
          total := 0
          for _, n := range nums {
              total += n
          }
          return total
      }
      sum(1, 2)
      sum(1, 2, 3)       
      sum()           // Valid: nums is a nil slice
      s := []int{1, 2, 3, 4} // using slice 
      sum(s...)       // Must "unpack" with ...
      // You could also pass slices normally, eg: 
      func sumForSlice(s []int) int { // Type must be []int
          total := 0
          for _, n := range s {      
              total += n
          }
          return total
      }
      s2 := []int{1, 2, 3, 4}
      sumForSlice(s2) // Pass directly; no unpacking needed   
      	
      Multiple returns function: 
      func divide(a, b int) (int, error) {
        if b == 0 {
            return 0, errors.New("divide by zero")
        }
        return a / b, nil
      }
      		
      Methods: Assume a struct, 
      type User struct {
          name string
      }
      Plain function (NOT attached to anything): 
      	func sayHello(a, b int) {
          	fmt.Println("hello", a, b)
      	}
      Method (attached to a struct): 
      	func (u *User) sayHello(a, b int) {
      		fmt.Println("hello", u.name, a, b)
      	}
      (u *User) means:
      	This function is attached to User and operates on a User object.
      How it becomes accessible: 
      	u := &User{name: "Suraj"}
      	u.sayHello(1, 2) // Even though method receiver is *User, u.sayHello() works, as Go automatically takes care of addresses: (&u).sayHello()
      	  
      A struct can have multiple behaviors by implementing multiple interfaces.
      type Flyer interface {
         Fly() string
      }
      type Swimmer interface {
         Swim() string
      }
      type Duck struct {
         Name string
      }
      func (d Duck) Fly() string {
         return d.Name + " is flying"
      }
      func (d *Duck) Swim() string {
         return d.Name + " is swimming"
      }
      func main() {
         d := Duck{Name: "Donald"}
         var f Flyer = d // Way 1 to do it
         var s Swimmer = &d // Way 2 to do it, since go handles pass by reference. Also note we are assigning struct d inside interface Flyer/Swimmer - will learn about it
         fmt.Println(f.Fly())
         fmt.Println(s.Swim())
      }
      How can an interface "equal" a struct?: var f Flyer = d.
        An interface value is two things: (interface type, concrete value). Hence above one is internally stored as:
        Flyer interface
        └── concrete type: Duck
        └── concrete value: Duck{Name: "Donald"}
        The interface does NOT become the struct. The struct is stored INSIDE the interface. This is different from java / python.
        
      > Interfaces can only be satisfied by methods, not free functions. Consider same above example: 
        func Fly(d Duck) string {
            return d.Name + " is flying"
        }
        Fly(d)
        func (d Duck) Fly() string {
            return d.Name + " is flying"
        }
        d.Fly()
        Both are conceptually same, second form enables interface and method sets. 
      
      > Value vs Pointer receiver nuance; Say the receiver is: 
          > d Duck -> It cannot modify struct.
          > d *Duck -> It can modify struct.  
        Another ex:
          type Rectangle struct {
            width, height int
        }
        func (r Rectangle) Area() int { // Define a method for the Rectangle struct using a value receiver
            return r.width * r.height
        }
        func (r *Rectangle) Scale(factor int) { // Define a method with a pointer receiver to modify the original struct
            r.width *= factor
            r.height *= factor
        }
        func main() {
            myRect := Rectangle{width: 10, height: 5} // Create an instance of the struct	    
            fmt.Println("Area:", myRect.Area()) // Output: Area: 50
            myRect.Scale(2)
            fmt.Println("New Width:", myRect.width) // Output: New Width: 20
        }
      

  • DataTypes:
    • Note: Use string when you need immutable text representation, such as configuration keys or constant text. Use []byte for mutable data sequences, file/network I/O, binary encoding/decoding, or when performance is critical and avoiding allocations is necessary.

    10      // int, we do have int8, int64, uint/uint8... - unsigned int which has positive values, etc... 
    3.14    // float64
    true    // bool
    "hi"    // string
    'A'     // byte
    '世'    // rune
    

• Loops/If-Else/Switches

  Normal loop:
  for i := 0; i < 5; i++ {
    fmt.Println(i)
  }
  
  Infinite loop:
  for {
    fmt.Println("running")
  }
  
  Loop over slice/map:
  for i, v := range nums {
    fmt.Println(i, v)
  }
  // i=index, v=value
  
  Run like while loop:
  for sum < 1000 {
  	sum += 1 // doubles the value of sum
  }
  
  If-Else:
  if x > 10 {
    fmt.Println("big")
  } else {
      fmt.Println("small")
  }
  
  Switch:
  switch day {
    case "Mon":
        fmt.Println("Start")
    case "Sun": 
        fmt.Println("Rest")
    default:
        fmt.Println("Other")
  }
  // Could also give multiple conditions like: case "Sun", "Tues":...
  // It's same as writing: case day == "Sun" || day == "Sun2":...
  

Phase 2

File structure in Go: Assume:

hello-go/
├── go.mod
├── main.go
└── mathutils/
    └── add.go

Create go.mod using: go mod init hello-go. Note for production related projects, consider naming convention like: github.com/XYZCompanyOrName/project

go.mod can be imagined as: requirements.txt + project identity + version lock.

Keeping module name same as folder name is best practice, else when you import other packages, you’ll have to do an explicit handling.

• Go mod or module name you define in command becomes the prefix for all imports inside this project. Hence use proper module name like: go mod init github.com/X/myproject therefore import "github.com/X/myproject/internal/utils".
• Go does not support relative imports (except very special cases you should avoid) like: import "./utils". Note that you could rename the folder, it wouldn’t matter, Go builds based on module identity, not folder naming.
• Hence file in /Users/X/work/nats-consumer/... doesn’t matter, Go sees it as it’s inside github.com/X/myproject/nats-consumer/...

Files:

mathutils/add.go:
package mathutils
// Add is a public function
func Add(a int, b int) int {
    return a + b
}
// Note: Package name mathutils would be same across all files in that folder. Note that main.go, is a special package, since it's entrypoint file.  

main.go:
package main
import (
    "fmt"               // standard library
    "hello-go/mathutils" // local package
)
func main() {
    sum := mathutils.Add(3, 4)
    fmt.Println("Sum:", sum)
}
// Note: Import path = (module name initialized) + (folder or relative path from go.mod to the package directory).

Run program using: go run .

To install third party packages: go get github.com/google/uuid

When you do so, a go.sum file is created (you don’t edit this). It stores checksums, ensures integrity; Can be imagined as pip-lock/poetry.lock file in Python. Version selection happens via go.mod (what versions). Integrity is enforced via go.sum (prove this code hasn’t changed).

Key Go Directory Naming Conventions:

- cmd/: Contains entry points for executable binaries. Each subdirectory (cmd/app1, cmd/app2) acts as a main package, allowing a single repository to generate multiple binaries.
- internal/: Contains code intended only for this project, enforced by the Go compiler. Code in internal/ cannot be imported by other projects, making it ideal for encapsulated application logic.
- pkg/: Contains library code designed to be consumed by external applications or other projects, serving as a shared library.
- api/: Houses API definitions such as Swagger/OpenAPI specs, JSON schemas, or Protocol Buffers.
- configs/: Stores configuration files or default configuration templates.
- web/: Holds front-end components, such as static assets, HTML templates, or CSS/JS files.
- scripts/: Contains build, installation, analysis, or administrative scripts.
- testdata/: Stores data files required for tests; Go tools automatically ignore this directory during building.
- vendor/: Contains application dependencies. Although becoming less common with Go modules, it's still a standard directory name for vendored code.
- test/ (or tests/): Used for system or integration tests, rather than unit tests which usually reside alongside the code. 

Essential Go commands:

go run .                 # run main package
go build                 # build binary
go build ./cmd/consumer  # build specific binary
go get github.com/google/uuid   # add dependency
go mod tidy                    # clean unused deps (VERY important)
go list -m all                 # list modules
go fmt ./...       # auto-format code
go vet ./...       # static analysis
go test ./...      # run all tests

Phase 3

Go - Memory model

• Memory model
  • A Go binary compiled for macOS will not run on Windows because binaries are OS and architecture-specific (like ARM64, x86_64), but Go allows cross-compiling by targeting the desired OS and CPU (like GOOS=windows GOARCH=amd64 go build). Interesting read.
  • Every program uses two main memory regions (both reside in RAM):
    • Stack: Fast, Automatically managed, Function-scoped, Freed when function returns
    • Heap: Slower than stack, Manually (C) or GC-managed (Go/Java/Python), Used when data must live longer
  • Go decides stack vs heap at compile time using escape analysis - you don’t do it, whereas Java relies on runtime JIT optimizations and Python allocates everything on the heap by design.
  • What Go GC does:
    • Find live objects, Free unreachable objects, Run concurrently with your program. This is called Concurrent mark-and-sweep (tricolor) GC.
      • White → not yet seen (assumed garbage)
      • Gray → seen, but children not scanned
      • Black → seen and fully scanned. A black object must never point to a white object for invariant GC.
    • Properties: Mostly concurrent, Small pause times, Optimized for server workloads
    • Go’s GC runs at the same time as your program.
    • Write barrier: When your program changes a pointer while GC is running, Go must inform the GC, called WB, handled by Go compiler itself. Problem without write barrier: GC thinks an object is unreachable -> Your code suddenly points to it -> GC frees it anyway → Crash. Write barrier prevents this. GC must be told about new pointers created while it is running, because the GC is making decisions based on a partial, moving snapshot of the heap.
    • sync.Pool: It is a temporary object recycling bin. Instead of: Allocate → use → GC frees. You do: Allocate once → reuse many times. Helps with heap allocations, fewer objects for GC to scan, shorter GC cycles, etc. Note: sync.Pool should NOT be used everywhere, its specific tool meant for temporary objects that reduce GC pressure, not a general cache or reuse mechanism.
    • unsafe keyword: It lets you break Go’s rules like: Type safety, Pointer safety, GC visibility guarantees. You gain: Speed, Control, Zero-copy tricks, etc. Risk: Crashes, Memory corruption, GC bugs.
  • In Go, how fast you allocate matters more than how much memory you use.
  • new(T) vs make(T)
    • In summary:
      • new: Allocates memory, Returns *T, Does NOT initialize runtime structures
      • make: Allocates + initializes, Returns T, Used for: slices, maps, channels
    • In detail: (Consider slices data structure as example)
      • Arrays in Go are static data structures with a fixed type and size. Slices are dynamic and built on top of arrays, defined by three components:
        • Data: pointer to the underlying array
        • Len: length of the slice
        • Cap: capacity of the slice (max length or array size)
      • Difference:

      Feature	new	make
      Purpose	Allocates memory but does not initialize runtime structures (memory is zeroed)	Allocates and initializes slices, maps, and channels
      Return value	Pointer to zeroed memory of type T (*T)	Initialized (non-zero) value of type T
      Slice underlying array	Not allocated; pointer is nil	Allocates underlying array with specified length and capacity
      Resulting slice	nil slice (no backing array)	Non-nil slice with backing array
      Usage	Returns a pointer → needs dereferencing	Returns the value directly
      Applicable types	Any type (struct, int, array, etc.)	Only slice, map, channel
      Ready to use?	Often not usable directly	Immediately usable

      • Zeroing means setting allocated memory to the zero value of the type (discussed in Fundamentals sections as well): Numeric types- 0, String- empty string “”, Boolean- false, Pointer/slice/map/channel- nil, Struct- all fields zeroed by their respective zero values.
      • When using new for a slice, the slice’s data pointer is nil, meaning no underlying array is allocated.
      • When using make, the underlying array is allocated and initialized to its zero values, making the slice ready for use.

      • There is no difference in observable behavior. But performance-wise, there is, Nil slices (created with new) will trigger automatic memory allocations and array resizing when elements are appended, potentially causing overhead due to repeated allocations and copying. Slices created with make and a predefined capacity avoid repeated allocations since the underlying array is pre-allocated.

        Note:
        sMake := make([]string, 3, 5)
        This means:
        | Property                  | Value      |
        | ------------------------- | ---------- |
        | Type                      | `[]string` |
        | Length (`len`)            | `3`        |
        | Capacity (`cap`)          | `5`        |
        | Underlying array size.    | `5`        |
        Visually:
        Underlying array (size = 5)
        +---------+---------+---------+---------+---------+
        |   ""    |   ""    |   ""    |    ?    |    ?    |
        +---------+---------+---------+---------+---------+
          ↑         ↑         ↑
          |--------- len=3 ---|
          |--------------- cap=5 ----------------|
        First 3 elements exist and are initialized ("")
        Last 2 slots exist but are not part of the slice yet
        
        Length (len): Number of elements you can access, Valid indices: 0 → len-1, Anything beyond len cannot be indexed. 
        Capacity (cap): Total space available before reallocation, How much you can grow using append without allocating new memory. 
        
        Now, if we do: sMake = append(sMake, "a"), sMake = append(sMake, "b")
        +---------+---------+---------+---------+---------+
        |   ""    |   ""    |   ""    |  "a"    |  "b"    |
        +---------+---------+---------+---------+---------+
        If you append beyond capacity. What Go does internally (also called Reallocation): 
        > Allocate a new, bigger array
        > Copy old elements
        > Append new element
        > Point slice header to new array
        > Old array is garbage-collected
        Capacity growth strategy is not guaranteed — don’t depend on exact numbers. 
        Very common pattern to initialize in Go: s := make([]T, 0, N) -> Means I have no elements yet, but I know how many I’ll need.
        

Phase 4

Go - Error handling

• Error handling

  • defer: schedules a function call to run when the surrounding function returns, in other words defer runs after return is executed, but before the function actually exits. It executes in LIFO order. Although, avoid using it in extreme tight loops. Using defer to clean up resources is very common in Go.

    Eg 1: 
    func main() {
      defer fmt.Println("world")
      fmt.Println("hello")
    }
    Output: 
    hello
    world
    
    Eg 2: 
    func main() {
      defer fmt.Println(1)
      defer fmt.Println(2)
      defer fmt.Println(3)
    }
    Output: 
    3
    2
    1
    

    • Note: Immediately Invoked Function Expression (IIFE) allows you to define a function without a name and execute it at the same moment. Use func() { }() only if you need at least one: go, defer, isolated scope, closure over variables, inline one-time logic. Else direct code.

      Eg: 
      func(msg string) {
        fmt.Println(msg)
      }("Hello Go")
      > Breakdown:
          func(msg string) → define function
          { ... } → logic
          ("Hello Go") → pass arguments & execute
      > With go keyword → run function in new goroutine
      > defer needs a function
          Invalid: 
            defer file.Close()
          Valid: 
            defer func() {
              file.Close()
              db.Disconnect()
            }()
      

  • panic: immediately stops normal execution of the current goroutine and begins stack unwinding. There are certain operations in Go that automatically return panics and stop the program like indexing an array beyond its capacity, performing type assertions, etc. We can also generate panics of our own using the panic built-in function. It’s a last-resort mechanism for programmer errors or truly unrecoverable states. Only the panicking goroutine unwinds its stack (will learn about this in ex).
    • Deferred functions or other goroutines still run.
    • Program crashes unless the panic is ‘recovered’. recover() stops stack unwinding and only works inside a deferred function.

    • Note that if the caller can handle it → return an error. If the program is broken → panic.

      Ex 1: 
      func main() {
        panic("something went wrong")
      }
      Output: panic: something went wrong
      
      Ex 2: 
      func main() {
        defer fmt.Println("cleanup")
        panic("boom")
      }
      Output: 
      cleanup
      panic: boom
      
      Ex 3: Unwinding of stack: 
      Normal function calls: f1 → f2 → f3 → return → return → return
      Assume in panic: f1 → f2 → f3 → panic 
      At this point: 
        Go stops normal execution
        Go says: “I am not returning normally”
        Go enters panic mode
        The call stack looks like this at panic time:
        [f3 stack frame]  ← panic here
        [f2 stack frame]
        [f1 stack frame]
        Stack unwinding means: Go starts destroying stack frames one by one, from top to bottom.
        But before destroying each frame, Go runs its defers.
        Hence sequence: 
        panic →
        run defers of f3 →
        remove f3 frame →
        run defers of f2 →
        remove f2 frame →
        run defers of f1 →
        remove f1 frame →
        (no more frames)
        → program crash
      
      Ex 4: 
      func f3() {
        defer func() {
            fmt.Println("f3 defer")
        }()
        panic("boom")
      }
      func f2() {
        defer fmt.Println("f2 defer")
        f3()
      }
      func f1() {
        defer fmt.Println("f1 defer")
        f2()
      }
      func main() {
        f1()
      }
      Execution timeline: 
      panic in f3
      ↓
      run f3 defer
      ↓
      destroy f3 frame
      ↓
      run f2 defer
      ↓
      destroy f2 frame
      ↓
      run f1 defer
      ↓
      destroy f1 frame
      ↓
      no recover → crash
      Our crash could cascade all the way down. 
      Panic should be used in cases of: programmer errors, impossible states, initialization failures. Else errors. 
      Now add recover:
      func f3() {
          defer func() {
              if r := recover(); r != nil {
                  fmt.Println("recovered:", r)
              }
          }()
          panic("boom")
      }
      Execution now: 
      panic in f3
      ↓
      run f3 defer
      ↓
      recover() stops panic
      ↓
      f3 returns normally
      ↓
      f2 continues
      ↓
      f1 continues
      ↓
      program continues
      Stack unwinding stops immediately at the recover point. 
      Note: Recover only stops panics in the same goroutine
      

  • In Go, all failures fall into two buckets:
    • Bucket A — Expected, possible, recoverable. Handled with error. These are things that can legitimately happen even if your code is perfect. Examples:
      • File not found
      • Invalid user input
      • Network timeout
      • Permission denied
      • API returned 500
      • Database connection lost

      • JSON malformed from external source

        data, err := os.ReadFile("config.json")
        if err != nil {
            return err
        }
        

    • Bucket B — Impossible, programmer mistake, corrupted state. Handled with panic. These are situations where continuing makes no sense. Examples:
      • Index out of bounds
      • Nil pointer dereference
      • Map accessed concurrently without lock
      • Invariant violated

      • Impossible switch case

        if user == nil {
          panic("user must never be nil here")
        }
        

    • Should everything else use panic/defer/recover?: No. It happens rarely, but good to know. Use error for 99% of cases. panic for impossible cases. Most panic calls are added after a bug is discovered in production or during testing. The evolution of code from a “crashing bug” to a “robust feature” usually follows this three-stage lifecycle:
      • The Implicit Crash (The “Unknown” Phase)
        • The Bug: You assume data is perfect (e.g., a pointer is never nil).
        • The Result: The Go Runtime panics for you with a generic error (e.g., “nil pointer dereference”).
        • The Outcome: Hard to debug; the program stops without explaining why the state was invalid.
      • The Explicit Panic (The “Defensive” Phase)
        • The Action: You add a manual if check that calls panic(“descriptive message”).
        • The Goal: To turn a “mysterious crash” into a clear assertion.
        • The Outcome: You’ve defined a “Programmer Error.” You are signaling to other developers that they are using your function incorrectly.
      • The Graceful Error (The “Maturity” Phase)
        • The Action: You realize the “impossible state” might actually happen in production (e.g., a database record was deleted). You replace panic with return err.
        • The Goal: To move from crashing to communicating.
        • The Outcome: The program remains running. The caller now has the power to log the issue, retry, or show a friendly message to the user.

          Another ex: 
        func main() {
            divideByZero()
            fmt.Println("we survived dividing by zero!")
        		
        }
        func divideByZero() {
            defer func() {
                if err := recover(); err != nil {
                    log.Println("panic occurred:", err)
                }
            }()
            fmt.Println(divide(1, 0))
        }
        func divide(a, b int) int {
            if b == 0 {
                panic(nil)
            }
            return a / b
        }
        

Phase 5

Go - Concurrency

• Concurrency
  • Go runs goroutines using its own scheduler on top of the OS scheduler. The OS schedules threads on CPU cores. The Go runtime schedules goroutines onto those threads using the G-M-P model, minimizing OS context switches and making concurrency cheap.

  • Python/Java use OS level threads which is heavy. Goroutine is NOT an OS thread. Under the hood (we would learn more):

    Millions of Goroutines
          ↓
    Go Scheduler
          ↓
    Few OS Threads
          ↓
    CPU Cores
        
    There is M:N scheduling. M goroutines & N OS threads. 
    

  • GMP Model:
    • G – Goroutine: Lightweight execution unit; Starts with ~2KB stack; Millions possible; Scheduled by Go runtime
    • M – Machine (OS Thread): Real OS thread (pthread, etc.); Generally ~1MB; Scheduled by OS scheduler; Executes Go code only when it owns a P
    • P – Processor (Logical Processor): Go runtime abstraction; Holds: Run queue of goroutines, Scheduler context; Count = GOMAXPROCS

    • A goroutine runs only when an M holds a P.

      CPU Core
        ↓
      OS Scheduler → M (thread)
        ↓
      Go Scheduler → P → G (goroutine)
      

    • Assume configuration: Physical CPU cores = 4, Ps(GOMAXPROCS) = 10, OS threads (Ms) = 5. Gs could be say 100s.
      • Maximum true parallelism = number of CPU cores. Hence, Max parallel execution = 4 goroutines
      • Ps do not map 1:1 to cores, they are logical.
      • For Ps: At most 4Ms can be running simultaneously (one per core). Each running M must own 1P. So at most 4Ps can be active at a time. Remaining 6Ps are idle.
      • For Ms: OS scheduler runs 4Ms max. 1M will be waiting / sleeping.
      • Having more Ms/Ps than physical capacity is waste of resources.
  • In case of any blocking scenario: Goroutine blocks → Go detaches it; M runs another G; OS thread stays busy.
  • Context switching:
    • OS thread switch: Save registers, Kernel mode, Expensive. 100k threads -> impossible
    • Goroutine switch: User-space, Save small state, Very cheap. 100k goroutines -> fine

  • main() function is initial/default goroutine.

    Sample Go routine: 
    func main() {
        sayHello("Alice") // Normal function call, assume it prints - blocks until complete
        go sayHello("Bob") // Goroutine - runs concurrently - simply add go keyword 
        time.Sleep(time.Second) // Without this sleep, main would exit before the goroutine runs hence you will not see "Bob" being printed
    }
    

    • When main exits, the entire program exits, killing all goroutines regardless of whether they’ve finished their work.
    • time.Sleep “works” but is wrong, for obvious reasons like although it gives time for goroutine to complete, you can’t be guessing the timing, its flaky. We should wait for events, not time, which leads to concept of WaitGroups.
  • WaitGroups: A sync.WaitGroup lets one goroutine wait until a set of goroutines finish.
    • Key rules:
      • Add(n) → number of goroutines to wait for
      • Done() → call once per goroutine
      • Wait() → blocks until counter reaches zero
    • Important: WaitGroup does NOT protect data. It only synchronizes completion

    Ex: 
    package main
    import (
      "fmt"
      "sync"
    )
    var wg sync.WaitGroup
    func sayHello(name string) {
      defer wg.Done()   // must be called once per goroutine
      fmt.Println("Hello", name)
    }
    func main() {
      names := []string{"Alice", "Bob", "Charlie", "Diana"}
      wg.Add(len(names)) // tell WaitGroup how many goroutines to wait for. Always call Add() before starting the goroutine
      for _, name := range names {
          go sayHello(name)
      }
      wg.Wait() // blocks until all Done() calls are made
      fmt.Println("All greetings printed")
    }
    
     **Goroutines interleave unpredictably. Hence the print order in above example may not be same as in list string**
    

    • For a WaitGroup, you must correctly account for every goroutine you want to wait for. WaitGroup is just a counter. What if goroutines ≠ wg.Add() count? Cases for count of:
      • Waitgroups > Code Goroutines: wg.Wait blocks forever (logical deadlock), as counter doesn’t reach 0.
      • Waitgroups < Code Goroutines: You get panic: sync: negative WaitGroup counter.
    • What if number of waitgroups to add is unknown/unbounded?: Then WaitGroup may be the wrong tool. Better alternatives: Channel + close(), Worker pool with fixed workers, Context cancellation (discussed later).

  • Mutex: A Mutual Exclusion lock ensures only one goroutine accesses critical data at a time. Helps in race conditions (concurrent access of data by multiple goroutines). In Java ecosystem, we use volatile/synchronized keywords.

    Race condition: 
      func increment() {
          counter++
      }
      go increment()
      go increment()
      This may NOT produce 2. Read → modify → write is not atomic. 
      Although if we add waitGroups in this, you will get visible output, but that doesn't erase the fact that your code might be in race condition. Hence we use mutex locks. 
      Note: To know if your code is in race condition use: go run --race .
      Other example for race condition can be: Multiple goroutines calling APIs and appending result to some list. 
    

    • Basic Mutex: Lock before accessing shared data. Unlock immediately after. Using defer to unlock is a good practice.

      var (
          counter int
          mu      sync.Mutex
      )
      func increment() {
          mu.Lock()
          counter++
          mu.Unlock()
      }
      

    • Types of Mutexes:

      • sync.Mutex: Exclusive lock; Only one goroutine can access the critical section at a time; Simple, fast, commonly used. Limitation: Readers and writers are treated the same; Even read-only operations block each other

      • sync.RWMutex: Read-Write lock; Multiple readers allowed concurrently; Only one writer allowed; Writers block readers and other writers. If your program has many reads, less writes, use this.

        Eg: 
        package main
        import (
          "fmt"
          "net/http"
          "sync"
        )
        var (
          wg      sync.WaitGroup
          mu      sync.Mutex
          signals []string
        )
        func getStatusCode(endpoint string) {
          defer wg.Done()
          res, err := http.Get(endpoint)
          if err != nil {
              fmt.Println("OOPS in endpoint")
              return
          }
          defer res.Body.Close()
          mu.Lock()
          signals = append(signals, endpoint)
          mu.Unlock()
          fmt.Printf("%d status code for %s\n", res.StatusCode, endpoint)
        }
        func main() {
          endpoints := []string{
              "https://google.com",
              "https://github.com",
              "https://golang.org",
          }
          wg.Add(len(endpoints)) // MUST be before starting goroutines
          for _, ep := range endpoints {
              go getStatusCode(ep)
          }
          wg.Wait() // blocks until all wg.Done() calls complete
          fmt.Println("Signals:", signals)
        }
        What happens: 
        1. wg.Add(len(endpoints)) - Tells WaitGroup how many goroutines to wait for.
        2. go getStatusCode(ep) - Launches each HTTP call concurrently.
        3. defer wg.Done() inside getStatusCode - Signals completion of one goroutine.
        4. mu.Lock() / mu.Unlock() - Protects shared slice signals from data races.
        5. wg.Wait() - Blocks main() until all HTTP calls finish.
        

  • Channels:
    • Go provides three major concurrency tools: sync.Mutex, sync.WaitGroup, channels - each solving a different class of problem.
      • What Mutex and WaitGroup Actually Solve:
        • sync.Mutex: Protects shared memory, Ensures exclusive access, Prevents data races.
        • sync.WaitGroup: Waits for goroutines to finish execution, Does NOT pass data, Does NOT control access
    • Channel is a typed conduit through which goroutines communicate. Eg: ch := make(chan int).
      • Think of a channel as: A thread-safe queue (like a message passing queue); With built-in blocking (for backpressure handling from producer-consumer); That transfers data + control.
      • Philosophy: Do not communicate by sharing memory, share memory by communicating.
      • They guarantee synchronization at the point of communication, not global ordering.
        • Send: ch <- value
        • Receive: value := <-ch. Note only <- symbol exists.
        • Close: close(ch). Only the sender (producer) should close the channel. (No more values will be sent, Receivers can still drain existing values)

          • If you read from a closed channel you get:

            v, ok := <-ch 
            ok == false → channel is closed
            v → zero value
            

        • Channel Ownership:

          // Compiler enforces ownership rules. You can restrict direction when passing a channel, but you cannot widen it again. Depending your use case, you may use bi-directional channel or restrict it 
          func produceJobs(jobs chan<- int, n int) {
            jobs <- 1      // ✅ allowed
            <-jobs         // ❌ compile-time error
          }
          

    • Blocking Rules:

      • Unbuffered Channel ch := make(chan int): Send and Receive must happen at the same time like a handshake. Synchronization first, data second. If you send, but no receiver then its blocked (pending state), vice versa. Ex:

        go func() {
          ch <- 10
        }()
        fmt.Println(<-ch) // main goroutine listening from the subroutine which pushed data to some channel
        

      • Buffered Channel ch := make(chan int, 2): Buffered channels decouple timing, but still synchronize.

        As seen above we initialized a channel of size 2. 
        ch <- 1 // ok
        ch <- 2 // ok
        ch <- 3 // blocks (buffer full)
        
        Other ex: Multiple Producers, Single Consumer
        ch := make(chan int)
        go func() { ch <- 1 }()
        go func() { ch <- 2 }()
        go func() { ch <- 3 }()
        for i := 0; i < 3; i++ {
            fmt.Println(<-ch)
        }
        All sends are received. Order is non-deterministic. If you want order to be deterministic, then of course a single sender must send data in guaranteed order. 
        

    • Blocked vs Deadlocked:
      • A goroutine is blocked when it is waiting for something. A program is deadlocked when: All goroutines are blocked, and no goroutine can ever make progress
      • In Go, the runtime’s deadlock detector is only triggered when every single goroutine is blocked. As long as there is at least one “active” or “runnable” goroutine in the program’s ecosystem, it will not panic, even if other goroutines are permanently blocked.
      • Scenario 1: One sender, no receiver in main. If your main function (which is its own goroutine) tries to send to an unbuffered channel without a concurrent receiver, the program will panic immediately. Why? The runtime sees that the only goroutine in existence (the main one) is stuck. There is no other goroutine that could ever perform a receive to unblock it. Error: fatal error: all goroutines are asleep - deadlock!.
      • Scenario 2: Two goroutines, one blocked, one “healthy” If you have one goroutine permanently blocked on a channel but another goroutine is still running (e.g., performing a long calculation or sleeping), the program will not panic. The “Healthy” Ecosystem: The runtime sees that progress is still being made elsewhere. It assumes the blocked goroutine might eventually be unblocked by the active ones.
        • Goroutine Leak: This is considered a goroutine leak. The blocked goroutine will stay in memory forever, consuming resources until the entire program terminates naturally.

    • Solving a problem using Mutex + Waitgroups vs Channels

      Eg 1: 
      -- Mutex+Waitgroup:
      var (
      mu      sync.Mutex
      wg      sync.WaitGroup
      results []int
      )
      func worker(n int) {
        defer wg.Done()
        mu.Lock()
        results = append(results, n*n)
        mu.Unlock()
      }
      func main() {
        for i := 1; i <= 5; i++ {
            wg.Add(1)
            go worker(i)
        }
        wg.Wait()
      }
      -- Channels: 
      jobs := make(chan int)
      results := make(chan int)
      go func() {
        for i := 1; i <= 5; i++ {
            jobs <- i
        }
        close(jobs)
      }()
      go func() {
        for job := range jobs {
            results <- job * job
        }
        close(results)
      }()
      for res := range results {
        fmt.Println(res)
      }
      
      > Also imagine, if we had a case for backpressure, creating that only using Mutex would be difficult. 
      
      Differences: 
      | Feature             | Mutex | WaitGroup | Channel |
      | ------------------- | ----- | --------- | ------- |
      | Protect memory      | ✅     | ❌        | ❌      |
      | Wait for completion | ❌     | ✅        | ✅      |
      | Transfer data       | ❌     | ❌        | ✅      |
      | Enforce order       | ❌     | ❌        | ✅      |
      | Backpressure        | ❌     | ❌        | ✅      |
      | Lifecycle signaling | ❌     | ❌        | ✅      |
      
      Eg 2:
      -- Mutex + Waitgroup 
      Consider earlier example related to APIs, pseudocode: 
      func main() {
        endpoints := []string{
            "https://google.com",
            "https://github.com",
            "https://golang.org",
        }
        wg.Add(len(endpoints))...}...
        // If we reimagine it with channels (below)
      -- Channel  
      import (
      "fmt"
      "net/http"
      )
      func getStatusCode(endpoint string, ch chan<- string) {
        res, err := http.Get(endpoint)
        if err != nil {
            fmt.Println("OOPS in endpoint")
            return
        }
        defer res.Body.Close()
        fmt.Printf("%d status code for %s\n", res.StatusCode, endpoint)
        ch <- endpoint // send result
      }
      func main() {
        endpoints := []string{
            "https://google.com",
            "https://github.com",
            "https://golang.org",
        }
        ch := make(chan string)
        for _, ep := range endpoints {
            go getStatusCode(ep, ch)
        }
        var signals []string
        for i := 0; i < len(endpoints); i++ {
            signals = append(signals, <-ch)
        }
        fmt.Println("Signals:", signals)
      }
      What did it replace?: // Receiving N values is equivalent to waiting for N goroutines.
      | Old          | New                                   |
      | ------------ | ------------------------------------- |
      | `WaitGroup`  | Receive loop (`len(endpoints)` times) |
      | `Mutex`      | Single owner of data                  |
      | Shared slice | Message passing                       |
      | `wg.Done()`  | `ch <- value`                         |
      | `wg.Wait()`  | `<-ch` loop                           |
      

    • When to Use Channels:
      • Goroutines need to communicate
      • Execution order matters
      • You want backpressure
      • You want pipeline or worker pool
      • You want clean shutdown signaling
  • Select: It allows a goroutine to wait on multiple channel operations simultaneously, executing whichever becomes ready first, with optional non-blocking behavior via default.
    • If no case is ready and no default exists, select blocks.
    • Even if multiple cases are ready, Go executes only one.
    • If multiple cases are ready: Go picks one at random, this prevents starvation.
    • default case prevents blocking

    Ex1: 
    select {
    case msg := <-ch1: // Receive case is ready when ch1 has a value already available/ ch1 is closed
        fmt.Println(msg)
    case ch2 <- 10: // Send case is ready when ch2 has buffer space /there is a receiver already waiting
        fmt.Println("sent")
    default: // default is ready when no other case is ready
        fmt.Println("nothing ready")
    }
    
    Ex2: Fan-in (Multiple Inputs → One Output)
    select {
    case v := <-worker1:
        fmt.Println("worker1:", v)
    case v := <-worker2:
        fmt.Println("worker2:", v)
    }
    

  • context.Context: It is a signal carrier carrying cancellation/deadline/request-scoped signals. Imagine: A request comes in and you start 5 goroutines to process it, but the user disconnects or request times out; Now how to stop all those goroutines? We can’t kill goroutines or force stop functions, hence Go gives you cooperative cancellation. Syntax: ctx := context.Background(). Note:
    • Any function that blocks or loops must listen to ctx.Done().
    • Cancellation propagates downward, never upward. parent → child → grandchild. If grandchild cancels(), parent is unaffected. This prevents goroutine leaks/zombie process, etc.

    Ex 1:
    func worker(ctx context.Context) {
      for {
        select {
        case <-ctx.Done():
          fmt.Println("worker stopped:", ctx.Err())
          return
        default:
          fmt.Println("working...")
          time.Sleep(500 * time.Millisecond)
        }
      }
    }
    func main() {
      ctx, cancel := context.WithCancel(context.Background())
      go worker(ctx)
      time.Sleep(2 * time.Second)
      cancel() // broadcast stop signal
      time.Sleep(1 * time.Second)
      fmt.Println("main exits")
    }
    What it does: 
      context.WithCancel creates: ctx, a hidden done channel
      Worker runs and selects on ctx.Done()
      cancel() is called
      ctx.Done() closes. Note: This blocks forever until someone cancels, then it unblocks immediately for ALL goroutines sharing the context, hence context scales.
      <-ctx.Done() unblocks instantly
      Worker exits cleanly
      No leak. Clean shutdown.
    
    Ex 2: 
    func fetchData(ctx context.Context) error {
    select {
      case <-time.After(3 * time.Second):
        fmt.Println("data fetched")
        return nil
      case <-ctx.Done():
        return ctx.Err()
      }
    }
    func main() {
      ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
      defer cancel()
      err := fetchData(ctx)
      fmt.Println("result:", err)
    }
    What it does: 
      main() — context creation: context.Background(), its root context (never cancels on its own). context.WithTimeout(...) creates: a child context, a timer, a Done channel. After 1 second, Go automatically calls cancel() internally.
      main and fetchData share the same context.
      Inside fetchData: time.After(3s) returns a channel that receives a value after 3 seconds. Until then → blocked. This represents slow work (API call, DB query, etc.)
      ctx.Done() is also a channel. It closes when the context is cancelled. Closing a channel unblocks all receivers immediately. 
      Its like: Try to finish work in 3s, but if the caller gives up in 1s — stop immediately.
    
    Ex 3: 
    Fan-Out (One → Many): Distribute work across multiple goroutines
      for i := 0; i < 4; i++ {
          go worker(jobs)
      }
    Fan-In (Many → One): Merge multiple result channels
      select {
      case r := <-c1:
      case r := <-c2:
      }
    

Phase 6

Miscellaneous stuff in Go

• Testing: Other doc

  func TestAdd(t *testing.T) {
    tests := []struct {
        name string
        a, b int
        want int
    }{
        {"both positive", 2, 3, 5},
        {"with zero", 0, 5, 5},
        {"negative", -1, 1, 0},
    }
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            if got := Add(tt.a, tt.b); got != tt.want {
                t.Fatalf("got %d, want %d", got, tt.want)
            }
        })
    }
  }
  

• Benchmarking:

  func BenchmarkAdd(b *testing.B) {
      for i := 0; i < b.N; i++ {
          Add(2, 3)
      }
  }
  Command: go test -bench=.
  

• Fuzzing: It finds edge cases you didn’t think of.

  func FuzzParseInt(f *testing.F) {
    f.Add("123")
    f.Add("-1")
    f.Fuzz(func(t *testing.T, input string) {
        _, _ = strconv.Atoi(input)
    })
  }
  Command: go test -fuzz=.
  

• Race Detector

  var counter int
  go func() { counter++ }()
  go func() { counter++ }()
  Commands: 
  go test -race
  go run -race main.go
  

• Performance Profiling using pprof

  import _ "net/http/pprof"
  go http.ListenAndServe(":6060", nil)
  go tool pprof http://localhost:6060/debug/pprof/profile
  
  Runtime metrics: Use import "runtime/metrics"
  

• Tracing: It shows execution flow over time.

  trace.Start(os.Stdout)
  defer trace.Stop()
  go test -trace trace.out
  go tool trace trace.out
  

• Logging:

  import "log" or "slog"
  

• HTTP API Calls: Interesting read

  Use net/http package. It: 
  
  Creates a TCP listener
  Accepts connections
  For each connection: 
    Spawns a goroutine One goroutine per connection, not per request
    Parses HTTP requests
    Reuses the same connection (keep-alive)
    Dispatches to handlers
  Places you MUST set timeouts
    Server side
    http.Server{
        ReadTimeout:       5 * time.Second,
        ReadHeaderTimeout: 2 * time.Second,
        WriteTimeout:      10 * time.Second,
        IdleTimeout:       60 * time.Second,
    }
    Client side
    client := &http.Client{
        Timeout: 5 * time.Second,
    }
  

• Marshalling, sometimes also known as serialization, is the process of transforming program data in memory into a format that can be transmitted or saved elsewhere. The json.Marshal function, then, is used to convert Go data into JSON data.
• Reflection in Go: Go is a strictly typed language. Usually, the Go compiler acts like a strict bouncer at a club: if it doesn’t know exactly who you are (your type) and what you are carrying (your value) before the program even runs, it won’t let you in. Reflection is an X-ray machine. It allows your program to accept a “mystery box” (an any / empty interface) while the program is already running, X-ray it, and ask: “What kind of data are you?” (Type), “What is inside you?” (Value).
  • Imagine you are writing a function that converts any Go struct into JSON format (just like json.Marshal does). The Problem Without Reflection: You would have to write a custom JSON converter for every single struct in your app, because Go demands to know the exact type.

    func UserToJson(u User) string { ... }
    func OrderToJson(o Order) string { ... }
    func ProductToJson(p Product) string { ... }
    

  • The Solution With Reflection: You write one generic function that accepts an empty interface (any). When a struct is passed in, reflection “X-rays” it, dynamically loops through whatever fields it finds, and builds the JSON string on the fly.

    func AnythingToJson(mysteryBox any) string { 
    // 1. Use reflection to see what's inside the mystery box
    // 2. Dynamically loop through the fields (Name, Age, Price, etc.)
    // 3. Turn it into a JSON string
    }
    

  • Takeaway: We use reflection to build universal tools (JSON serializers, Database ORMs, Logging tools) that can handle any data type we throw at them, without needing to know what that data type is when we write the code. When you pass your “mystery box” into the reflect package, it splits the X-ray into two specific tools:

    Tool	What it tells you	Example Question it Answers
    reflect.TypeOf()	The Blueprint	“Are you an integer or a struct? If you are a struct, what are your field names?”
    reflect.ValueOf()	The Actual Data	“I know you are an integer, but is your number 10 or 42?”

  • Eg: This function doesn’t know what struct it is receiving, but it prints the fields anyway.

    // We have two totally different structs
    type User struct{ Name string; Age int }
    type Product struct{ Title string; Price float64 }
    // A universal function that accepts ANYTHING
    func PrintAnyStruct(mysteryBox any) {
        // X-ray the box to get its blueprint and data
        blueprint := reflect.TypeOf(mysteryBox)
        data := reflect.ValueOf(mysteryBox)
        fmt.Printf("--- Inspecting: %s ---\n", blueprint.Name())
        // Dynamically loop through however many fields it has!
        for i := 0; i < blueprint.NumField(); i++ {
            fieldName := blueprint.Field(i).Name
            fieldData := data.Field(i).Interface()
            fmt.Printf("%s: %v\n", fieldName, fieldData)
        }
    }
    func main() {
        PrintAnyStruct(User{Name: "Alice", Age: 30})
        PrintAnyStruct(Product{Title: "Laptop", Price: 999.99})
    }
    

  • Nuances:
    • It’s Slow: X-raying boxes at runtime takes extra processing power. Regular Go code is compiled and lightning-fast; reflection code is evaluated on the fly and is much slower.
    • It’s Unsafe: The compiler can’t protect you. If you use reflection to try and extract a string out of a box that actually contains an int, your entire app will panic and crash.
• Similarly other things like Interface implementation patterns, Error design patterns, Time related functions, etc.

Phase 7

Extras - gRPC, Protobuf

1) The Mental Model Shift — REST vs gRPC

• When you use REST APIs, you think in terms of resources and URLs. You call POST /users or GET /orders/123. The URL is the thing you’re targeting, and you manually define every route, handle the HTTP method, parse the JSON body, and write JSON back in the response. You own all of that plumbing.
• gRPC flips this entirely. You think in terms of functions (procedures). You’re not hitting a URL — you’re calling a method on a remote object, just like calling a function in your own code. The networking is abstracted away from you.
• This concept is called RPC — Remote Procedure Call. The “remote” part means the function lives on another machine. The “procedure call” part means it feels like a local function to the caller. gRPC is Google’s implementation of this idea, built on top of HTTP/2 and Protobuf.
• So if you’re coming from REST and wondering “what URL do I POST to?” — that question becomes irrelevant in gRPC. You just call a method. The framework handles the transport layer entirely.

2) What is Protobuf? (It’s Actually Three Things)

• Protobuf (short for Protocol Buffers) is where most confusion starts, because it is actually three things bundled into one, and people often only describe one of them.
• Thing 1 — A Schema Language (the .proto file): You write a .proto file that describes what your data looks like and what methods your service exposes. This is the “schema definition” or “schema validation” role. Think of it like TypeScript interfaces or a database schema — it defines the shape of your data and the contract between systems.
• Thing 2 — A Binary Encoding Format: When your data actually travels over the network, Protobuf encodes it into a compact binary format. This is the “encode/decode” role — similar to what JSON does, but binary instead of human-readable text. The binary format uses field numbers (not field names) to identify fields, which is why it’s significantly smaller and faster to parse than JSON.
• Thing 3 — A Code Generator: You run the protoc compiler on your .proto file, and it automatically generates real working code (classes, methods, serializers, deserializers) in your language of choice — Python, Go, Java, Rust, etc. This is the “function generation” role. You never write the serialization logic by hand; protoc writes it for you.
• All three of these are part of “Protobuf”. That’s the source of the confusion — when someone says “we use Protobuf”, they mean all three things at once.

3) What is gRPC and What Does the Name Mean?

• gRPC stands for gRPC Remote Procedure Call — yes, it’s recursive, like GNU (GNU’s Not Unix). The “g” technically changes with every version of the project and has meant things like “google”, “good”, “green”, and “glorious” at different times. The important part is RPC — Remote Procedure Call.
• gRPC is a framework built by Google that lets a program on one computer call a function on another computer as if it were a local function. It uses:
  • Protobuf as the data format and schema system (though JSON is technically possible)
  • HTTP/2 as the transport layer (not HTTP/1.1)
  • Generated code on both the client and server to handle all communication automatically
• gRPC is particularly dominant in microservices architectures where many services need to talk to each other efficiently, because it offers strong type safety, very high performance, and an ergonomic developer experience once the initial setup is done.

4) Why gRPC Uses Protobuf Instead of JSON

• gRPC can technically use JSON (there’s a spec called gRPC-JSON transcoding), but almost nobody does, because Protobuf is the entire reason gRPC is worth using in the first place. Here’s why Protobuf wins:
  • Speed. Binary formats are much faster for machines to parse than text-based formats. A machine reading a Protobuf message doesn’t need to scan for quote characters, parse key names as strings, or handle escape sequences. It reads field numbers and jumps directly to the data.
    • Sidenote: A major performance advantage comes from HTTP/2 itself. Unlike typical REST setups using HTTP/1.1, gRPC uses persistent connections, multiplexed streams, header compression (HPACK), and efficient binary framing. This reduces repeated TCP handshakes, avoids many head-of-line blocking problems, and improves bandwidth utilization under heavy load.
  • Size. Protobuf is significantly smaller than JSON. Consider sending { "name": "Alice" } — as JSON that’s roughly 16 bytes of payload (plus 400–800 bytes of HTTP/1.1 headers). As Protobuf binary, the same data is about 7 bytes, with HTTP/2 header compression further reducing the overhead. At scale, across millions of requests per day, this compounds dramatically.
  • Strictness and Type Safety. JSON is flexible to a fault — you can send a string where an integer was expected, include unexpected fields, or omit required ones, and the receiver might silently mishandle it. Protobuf’s schema is enforced at compile time. The generated code enforces field types and schema structure, catching many integration bugs at development time instead of production.
  • Schema as Documentation. With JSON REST APIs, you typically need separate documentation (OpenAPI/Swagger specs, Postman collections, Confluence pages) to describe what fields are expected. With Protobuf, the .proto file is the documentation, the contract, and the SDK generator all at once.
  • Protobuf Evolution & Compatibility: Once deployed, field numbers are part of your contract and must never change. Safe changes: adding new fields, adding new RPC methods. Breaking changes: changing a field number, reusing a deleted field number, changing a field type. If a client adds a field the server doesn’t know about, the server silently ignores it. If a client removes a field, the server sees the default value. If either side changes a field type, you get decode failures or silent data corruption.

5) The .proto File — The Contract

• Everything in gRPC starts with a .proto file. Think of it as a contract between the server and the client — like a restaurant menu that both the waiter and the kitchen agree on before any order is placed. Here is a full example with explanations inline:

    // greeter.proto
  	
    syntax = "proto3";  // Tells the compiler which version of protobuf syntax to use.
                        // proto3 is the current standard.
  	
    // This defines the "service" — essentially a remote class with callable methods.
    // Each "rpc" line is one callable endpoint/function.
    service Greeter {
      rpc SayHello (HelloRequest) returns (HelloReply) {}
      //  ^ method name  ^ input message type  ^ output message type
      rpc SayGoodbye (GoodbyeRequest) returns (GoodbyeReply) {}
    }
  	
    // A "message" is like a struct or class — it defines the shape of a request or response.
    message HelloRequest {
      string name = 1;
      // The "= 1" is a FIELD NUMBER, not a default value.
      // Field numbers are the real identifiers used on the wire.
      // JSON sends field names repeatedly:
      // { "name": "Alice" }
      // Protobuf instead sends:
      // [field_number + wire_type] + [value bytes]
      // For this field:
      // string name = 1;
      // if:
      // name = "Alice"
      // the binary payload becomes roughly:
      // 0A 05 41 6C 69 63 65
      // Where:
      // 0A -> field number 1 + wire type for length-delimited data
      // 05 -> string length (5 bytes)
      // 41 6C 69 63 65 -> ASCII bytes for "Alice"
      // Notice:
      // the string "name" never appears on the wire at all.
      // This is one reason protobuf messages are much smaller
      // and faster to parse than JSON.
    }
   
    message HelloReply {
      string message = 1;
    }
   
    message GoodbyeRequest {
      string name = 1;
    }
   
    message GoodbyeReply {
      string message = 1;
    }
  

• This single file simultaneously serves as your API documentation, your data schema, your type definitions, and the input to your code generator. Every other piece of the system is derived from it.

6) Code Generation — The Magic Step

• Once you have your .proto file, you run the protoc compiler on it. This is what “code generation” means — protoc reads your schema and writes real, working code in your chosen language.

    # For Python
    protoc --python_out=. --grpc_python_out=. greeter.proto
  	
    # For Go
    protoc --go_out=. --go-grpc_out=. greeter.proto
  

• This generates two files (in Python’s case):
  • greeter_pb2.py contains the Python classes for your messages — HelloRequest, HelloReply, GoodbyeRequest, GoodbyeReply. These classes have the serialization and deserialization logic baked in. You never write these by hand.
  • greeter_pb2_grpc.py contains two things. First, a Stub class for the client — this is the “remote control” object that has SayHello() and SayGoodbye() as methods you can call. Second, a Servicer base class for the server — this is the class you inherit from and implement with your business logic.
• This is the “function generation” aspect of Protobuf. You defined SayHello in the .proto file, and now you have a real Python method stub.SayHello(...) to call without writing any of that infrastructure yourself.
• What gRPC generates for you: client stubs, server interfaces, serialization/deserialization logic, transport plumbing.
• What it does NOT generate: business logic, database access, validation rules, authorization, caching, observability. You still write the actual application behavior yourself.

7) The Server — No net/http, No Manual Routes

• This is one of the most important practical differences from REST. In a traditional Go REST server, you manually wire up every route:

    // REST way — you own all of this plumbing
    mux := http.NewServeMux()
    mux.HandleFunc("/users", handleUsers)       // manual route
    mux.HandleFunc("/orders", handleOrders)     // manual route
    http.ListenAndServe(":8080", mux)           // manual server start
  

• With gRPC, you throw all of that away. The gRPC server is your HTTP server. It manages the port, the HTTP/2 connections, the routing, serialization, and deserialization. You only implement the business logic.

• Here is a full Python gRPC server:

    # server.py
    import grpc
    import greeter_pb2        # generated message classes (HelloRequest, HelloReply, etc.)
    import greeter_pb2_grpc   # generated service classes (Servicer base class)
    from concurrent import futures
    # You inherit from the generated Servicer base class and implement each method.
    # This is YOUR business logic — the generated code handles all the networking.
    class GreeterServicer(greeter_pb2_grpc.GreeterServicer):
        def SayHello(self, request, context):
            # "request" is already a HelloRequest Python object.
            # The binary Protobuf bytes were automatically deserialized for you.
            # You just work with normal Python objects.
            name = request.name  # e.g., "Alice"
            # You return a HelloReply object.
            # gRPC automatically serializes this back to binary before sending.
            return greeter_pb2.HelloReply(
                message=f"Hello, {name}! Welcome to gRPC."
            )
        def SayGoodbye(self, request, context):
            return greeter_pb2.GoodbyeReply(
                message=f"Goodbye, {request.name}. See you soon."
            )
    # Standard boilerplate to start the server
    server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
    # "Registering" your service is the gRPC equivalent of adding routes.
    # But notice — you specify NO URLs. gRPC derives the routing from the proto schema.
    greeter_pb2_grpc.add_GreeterServicer_to_server(GreeterServicer(), server)
    server.add_insecure_port('[::]:50051')  # listen on port 50051
    server.start()
    server.wait_for_termination()
   	-- The add_GreeterServicer_to_server call does what mux.HandleFunc(...) was doing in REST, but instead of you specifying URL paths, gRPC automatically creates routes from the service and method names in your proto file. In normal application code, you rarely think about URLs, HTTP methods, or JSON parsing directly, the gRPC framework handles most transport concerns automatically.
  

8) The Client — Calling Remote Functions Like Local Ones

• The client is where the RPC abstraction is most apparent. There is no URL construction, no requests.post(), no JSON serialization, no response parsing. You just call a method:

    # client.py
    import grpc
    import greeter_pb2
    import greeter_pb2_grpc
    # Step 1: Open a connection to the server (equivalent of creating an HTTP session)
    channel = grpc.insecure_channel('localhost:50051')
    # Step 2: Create a Stub — this is your "remote control" object.
    # It has SayHello() and SayGoodbye() as real callable methods.
    stub = greeter_pb2_grpc.GreeterStub(channel)
    # Step 3: Call the remote method exactly like a local function.
    # Under the hood: creates a HelloRequest, serializes to binary,
    # sends over HTTP/2, receives binary response, deserializes to HelloReply.
    # You see none of that — it's all handled by the generated code.
    response = stub.SayHello(greeter_pb2.HelloRequest(name='Alice'))
    print(response.message)  # Output: "Hello, Alice! Welcome to gRPC."
    # Calling a second method is just calling another method — no new config needed
    farewell = stub.SayGoodbye(greeter_pb2.GoodbyeRequest(name='Alice'))
    print(farewell.message)  # Output: "Goodbye, Alice. See you soon."
  

• From the developer’s perspective, SayHello feels like a local function. The fact that it’s making a network call to another machine — serializing your object to binary, sending it over HTTP/2, receiving a binary response, and deserializing it back — is entirely invisible to you.

9) Where Does the HTTP Request Actually Go?

• This is the question that trips up developers coming from REST: “If there’s no URL, where does the request go?”: The answer is that gRPC does use HTTP/2 under the hood, and there is a URL — but it’s automatically derived from your proto schema, and you never write it yourself. The pattern is always:

    POST https://<host>:<port>/<PackageName>.<ServiceName>/<MethodName>
  

• So for the SayHello call in our example, the actual HTTP/2 request that goes over the wire is:

    POST http://localhost:50051/Greeter/SayHello
    Content-Type: application/grpc
    [binary protobuf body — NOT JSON]
  

• The body is the binary-encoded HelloRequest. It is not JSON. It’s a compact sequence of bytes that only makes sense if you have the proto schema to decode it.
• You never write this URL. You never serialize the body. gRPC generates this mapping from your .proto file and handles it automatically. This is the fundamental difference from REST — in REST, you design and manage the URLs. In gRPC, the framework owns the transport layer entirely.
• gRPC Communication Patterns: So far we’ve only looked at unary RPC — one request, one response, exactly like a REST call. But gRPC supports three other patterns that REST simply cannot do without bolting on WebSockets or long-polling:
  • Server streaming — client sends one request, server streams back many responses over the same connection. Useful for live logs, real-time metrics, or replacing paginated polling.
  • Client streaming — client streams many requests, server replies once. Useful for batch uploads or telemetry ingestion where you want one connection rather than thousands of small HTTP calls.
  • Bidirectional streaming — client and server stream independently and simultaneously over one persistent connection. This is the natural fit for chat systems, real-time collaboration, or online gaming — use cases where REST would push you toward WebSockets.

  • Eg: These patterns are defined directly in the .proto file using the stream keyword:

    service Greeter {
      rpc SayHello (HelloRequest) returns (HelloReply) {}                            // unary
      rpc StreamUpdates (UpdateRequest) returns (stream UpdateResponse) {}           // server streaming
      rpc UploadChunks (stream ChunkRequest) returns (UploadReply) {}               // client streaming
      rpc Chat (stream ChatMessage) returns (stream ChatMessage) {}                 // bidirectional
    }
    

10) Both Sides Must Speak gRPC

• This is a critical architectural point. Unlike REST — where any client (a browser, curl, Postman, a Python script using requests) can talk to any server because everyone agrees on HTTP/1.1 + JSON — gRPC requires both sides to understand the same protobuf contract.
• The client needs the generated stub code so it knows how to serialize a HelloRequest into binary and send it over HTTP/2. The server needs the generated servicer code so it knows how to deserialize that binary back into a real object and route it to the right method. If either side doesn’t have the generated code from the .proto file, they literally cannot communicate — the binary format is meaningless without the schema to interpret it.
• This is why in companies running microservices with gRPC, teams publish their .proto files to a shared repository (often called a “proto registry” or “buf registry”). Every team that wants to call your service pulls your .proto file, runs protoc in their language, and gets a fully working, type-safe client. The .proto file is your API documentation, your contract, and your SDK generator all in one.

11) Adding New Endpoints

• Adding a new “endpoint” in gRPC means adding a new rpc line in your .proto file. The workflow is clean and consistent every time.

Step 1 — Update the proto file.

service Greeter {
  rpc SayHello   (HelloRequest)   returns (HelloReply)   {}
  rpc SayGoodbye (GoodbyeRequest) returns (GoodbyeReply) {}  // New endpoint — just one line
}
// Add the new message types
message GoodbyeRequest {
  string name = 1;
}
message GoodbyeReply {
  string message = 1;
}

Step 2 — Regenerate the code by re-running protoc. The generated stub and servicer now automatically include SayGoodbye on both the client side and the server side.

Step 3 — Implement the method on the server. In strongly-typed languages like Go, the compiler will actually refuse to compile until you implement every method declared in the proto service. This prevents you from accidentally shipping a server with unimplemented endpoints — a guarantee REST has no equivalent for.

class GreeterServicer(greeter_pb2_grpc.GreeterServicer):
    def SayHello(self, request, context):
        return greeter_pb2.HelloReply(message=f"Hello, {request.name}!")
    # Must implement this now — gRPC framework will raise errors if you don't
    def SayGoodbye(self, request, context):
        return greeter_pb2.GoodbyeReply(message=f"Goodbye, {request.name}!")

Step 4 — The client gets the updated .proto file, regenerates, and can immediately call stub.SayGoodbye(...). There is no API documentation to update, no Postman collection to edit, no URL to communicate to other teams. The proto file is all of that.

This is a significant developer experience win over REST, where adding a new endpoint means updating route handlers, updating documentation, updating any shared Postman collections, and manually communicating the change to all consumers.

12) Error Handling in gRPC

• gRPC defines its own application-level status codes on top of HTTP/2. Common status codes include: OK, InvalidArgument, NotFound, Unauthenticated, PermissionDenied, Internal. These status codes are language-neutral and consistent across all gRPC clients and servers.

13) Testing gRPC — The curl Equivalent

• This is one of the real friction points when first moving to gRPC, because curl simply does not work with gRPC. The reason is straightforward: curl speaks HTTP/1.1 and sends plain text, but gRPC expects HTTP/2 and a binary protobuf body. If you tried to curl a gRPC endpoint, the server would reject the connection entirely.
• The ecosystem has built dedicated tools that act as the curl equivalent for gRPC.
  • Option 1: grpcurl — The True curl Equivalent: grpcurl is a command-line tool that works almost identically to curl, but speaks gRPC natively. Install it inside your container and use it from the terminal. Option A — Point grpcurl at your proto file: This works but requires the proto file to be accessible wherever you’re running the command, which can be inconvenient inside containers. Option B — Enable Server Reflection (recommended): Server reflection is a built-in gRPC feature where your server advertises its own schema at runtime. You enable it once with two lines of code, and then grpcurl (or any tool) can query the server to discover its own API without needing the .proto file present.
  • Option 2: GUI Tools (Postman, BloomRPC). Postman now supports gRPC natively. You point it at your server (with reflection enabled, or by importing your proto file) and get a visual interface to fill in fields and call methods — exactly like using Postman for REST. BloomRPC is another dedicated gRPC GUI tool. These are great for exploratory testing but not useful inside a container via terminal.
  • Option 3: Write a Small Test Client Script.

14) The Full Picture — End to End Flow

• Here is what happens, step by step, from the moment you write a proto file to the moment a client gets a response:

1. You write greeter.proto
         │
         ▼
2. Run protoc compiler
         │
         ├──► greeter_pb2.py          (message classes: HelloRequest, HelloReply, etc.)
         └──► greeter_pb2_grpc.py     (GreeterStub for client, GreeterServicer for server)
                      │
         ┌────────────┴────────────┐
         │                         │
      CLIENT                    SERVER
      imports Stub               imports Servicer
      calls stub.SayHello()      implements SayHello() with business logic
         │                         │
         │   HTTP/2 POST           │
         │   /Greeter/SayHello     │
         │   [binary protobuf] ───►│
         │                         │ deserializes binary → HelloRequest object
         │                         │ runs your SayHello() method
         │                         │ serializes HelloReply → binary
         │◄─── [binary response] ──┘
         │
      deserializes binary → HelloReply object
      response.message is available as a normal Python string

The key insight is that the binary serialization, HTTP/2 transport, routing, and deserialization are all handled invisibly by the generated code and the gRPC framework. You write the schema, you write the business logic, and the framework handles everything in between.

15) Quick Reference Comparison — REST vs gRPC

16) When to use gRPC, and when not to

• gRPC is often an excellent default for internal microservices, high-throughput systems, anything with low-latency requirements, and any use case involving streaming. It pays off most when strong contracts across teams matter — the proto file eliminates entire categories of miscommunication.
• Avoid it when you’re building a public API, when your clients are browsers (gRPC requires a proxy like gRPC-Web in browser environments), or when you have a simple CRUD service with no performance pressure and no streaming needs. REST is simpler to debug, easier to test ad-hoc, and universally supported — don’t reach for gRPC just because it’s faster.

From my experience working at Punch.

Recently, I was discussing the issue with devops which made me brush up concepts related to DNS. In short, one of the pages of main website of Punch had a broken CSS/styling. This happened because the DNS record for that page was probably pointing to the wrong Application Load Balancer (staging service or other). Devops fixed the Route 53 record to point to the correct ALB, which restored CSS loading and fixed the UI. Now, my brushup behind this:

• Understanding DNS: The Internet’s Address Book
  • DNS translates human-readable domain names into IP addresses that computers use to connect. When you type xyz.punch.trade into your browser, a lookup process begins that walks through a hierarchy of DNS servers.
  • Your browser queries a recursive DNS resolver, typically your ISP’s DNS or a public service like Google’s 8.8.8.8.
  • This resolver contacts one of thirteen root DNS servers asking “who manages .trade domains?” The root server responds with addresses of .trade TLD servers.
  • The resolver then asks these TLD servers “who is authoritative for punch.trade?” and receives Route 53’s nameservers as the answer. (Other nameservers ex: Route 53, Cloudflare, Google DNS, Akamai DNS, etc.)
  • Finally, the resolver queries Route 53 directly for xyz.punch.trade and receives an IP address or load balancer DNS name.
  • The resolver caches this answer based on TTL settings and returns it to your browser. (Good videos around this: video 1, video2).
• How Domains Enter the DNS System
  • Before DNS resolution can work, the domain must exist in the global system.
  • Punch purchased punch.trade through a registrar like GoDaddy, which acts as a middleman to the official .trade registry.
  • During registration, Punch specified Route 53 nameservers as the authoritative DNS provider.
  • The registrar communicated this to the .trade registry, which updated its global database.
  • This information propagated to TLD servers worldwide, making Route 53 the authoritative source for all punch.trade DNS queries.
• What DNS Records Actually Control
  • DNS operates exclusively at the hostname level. When you create a Route 53 record for xyz.punch.trade, you map that complete hostname to a destination server or load balancer.
  • Route 53 knows nothing about files, application code, or URL paths. There’s no DNS record for xyz.punch.trade/index.html or xyz.punch.trade/assets/main.css.
  • DNS simply answers “what server hosts this domain name?” The server itself handles all file routing and application logic.
• How Websites Load After DNS Resolution
  • Once your browser obtains the IP address from DNS, it establishes a TCP connection to the Application Load Balancer and performs an SSL/TLS handshake for secure connections.
  • The browser sends an HTTP request, which the ALB forwards to an ECS container running your web application. That container generates and returns the HTML response.
  • As the browser parses the HTML, it discovers additional resources like CSS files, JavaScript, and images. For each resource, the browser makes separate HTTP requests. When it encounters <link rel="stylesheet" href="https://xyz.punch.trade/assets/career/main-1234.css">, it performs DNS resolution again for xyz.punch.trade (typically served from cache) and requests that specific file path.
  • The ALB forwards this request to an ECS container where your application or web server like Nginx uses its internal routing logic to serve the CSS file from the correct directory.
• About the bug
  • The Route 53 record for xyz.punch.trade pointed to the wrong ALB. When browsers loaded the page, they received HTML successfully, possibly from cache or because the initial connection worked. However, when browsers attempted to fetch the CSS file referenced in the HTML, DNS resolution returned the wrong destination. The CSS request either failed completely or reached a server where the file didn’t exist.
  • Without CSS, HTML renders as unstyled text. All content appears on the page but without layouts, colors, fonts, or visual formatting. The site looked broken even though the actual content and structure were intact.
  • Once Devops updated the Route 53 record to point to the correct ALB and DNS caches expired according to TTL settings, CSS requests began reaching the correct server. The stylesheet loaded successfully and the UI displayed properly.
• Insight
  • DNS sits at the network layer, completely separate from your application. Route 53 points browsers to your ALB, which forwards requests to ECS containers running your application code.
  • That application code came from GitHub through a CI/CD pipeline that builds Docker images and deploys them to ECS (update image in ECR & ECS pulls and runs container from the image).
  • DNS has no involvement in this deployment process. Its single job is mapping hostnames to server addresses.
  • When that mapping breaks, resources referenced by absolute URLs fail to load even though the initial page may appear functional.

Tried to get sense of AI hype going around.

With AI advancing around, I tried to get a sense of the fundamentals on which it is built and touched some of the trending terms watching their youtube videos, reading articles, chatting with GPT, like - RAG, MCP, LLMs, VectorDBs, ML/DL/Neural Networks, etc.

Things make a little more sense now, I found neural networks quite interesting. Below is a condensed snippet of an interesting conversation I was having with GPT making me understand neural networks.

Explain me neural network with a mathematical example. 

Let's work through a concrete, simplified example with actual numbers. I'll make this small enough that you can verify every calculation by hand.
Setup: Recognizing the digit "1" from a blurry 4×4 image
The Input Image
Let's say our blurry image is 4×4 pixels (grayscale, 0=black, 1=white):
Image of "1":
0.1  0.7  0.2  0.1
0.2  0.8  0.2  0.1
0.1  0.6  0.3  0.2
0.1  0.7  0.1  0.1
We flatten this into a vector of 16 input values:
Input = [0.1, 0.7, 0.2, 0.1, 0.2, 0.8, 0.2, 0.1, 0.1, 0.6, 0.3, 0.2, 0.1, 0.7, 0.1, 0.1]

Network Architecture
Input layer: 16 neurons (one per pixel)
Hidden layer 1: 4 neurons
Hidden layer 2: 3 neurons
Output layer: 2 neurons (outputs: "is it 1?" and "is it 0?")

Layer 1: Input → Hidden Layer 1
Each neuron computes: output = activation(weighted_sum + bias)
Neuron 1 in Hidden Layer 1
Weights (I'll give you 16 weights, one for each input):
w = [0.5, 0.8, 0.1, 0.0, 
     0.3, 0.9, 0.2, 0.0,
     0.4, 0.7, 0.1, 0.1,
     0.2, 0.8, 0.0, 0.0]
Bias: b = -0.5
Calculation:
z₁ = (0.1×0.5) + (0.7×0.8) + (0.2×0.1) + (0.1×0.0) +
     (0.2×0.3) + (0.8×0.9) + (0.2×0.2) + (0.1×0.0) +
     (0.1×0.4) + (0.6×0.7) + (0.3×0.1) + (0.2×0.1) +
     (0.1×0.2) + (0.7×0.8) + (0.1×0.0) + (0.1×0.0) - 0.5
z₁ = 0.05 + 0.56 + 0.02 + 0.0 + 0.06 + 0.72 + 0.04 + 0.0 +
     0.04 + 0.42 + 0.03 + 0.02 + 0.02 + 0.56 + 0.0 + 0.0 - 0.5
z₁ = 2.54 - 0.5 = 2.04

Activation function (ReLU: max(0, x)):
a₁ = ReLU(2.04) = 2.04
Let me give you simpler weights for the remaining 3 neurons:

Neuron 2:
Weights: [0.1, 0.2, 0.3, 0.1, 0.2, 0.1, 0.2, 0.1, 0.3, 0.2, 0.1, 0.1, 0.2, 0.3, 0.1, 0.1]
Bias: -0.2
Calculate: z₂ = sum of (input × weight) - 0.2 = 0.31 - 0.2 = 0.11
a₂ = ReLU(0.11) = 0.11

Neuron 3:
Weights: [0.2, 0.1, 0.4, 0.2, 0.1, 0.2, 0.3, 0.1, 0.2, 0.1, 0.3, 0.2, 0.1, 0.2, 0.2, 0.1]
Bias: -0.3
Calculate: z₃ = 0.49 - 0.3 = 0.19
a₃ = ReLU(0.19) = 0.19

Neuron 4:
Weights: [0.1, 0.3, 0.2, 0.1, 0.1, 0.2, 0.1, 0.1, 0.1, 0.2, 0.2, 0.1, 0.1, 0.3, 0.1, 0.1]
Bias: -0.1
Calculate: z₄ = 0.36 - 0.1 = 0.26
a₄ = ReLU(0.26) = 0.26

Hidden Layer 1 Output: [2.04, 0.11, 0.19, 0.26]

Layer 2: Hidden Layer 1 → Hidden Layer 2
Now these 4 values become inputs to the next layer.
Neuron 1 in Hidden Layer 2:
Weights: [0.6, 0.3, 0.2, 0.1]
Bias: -0.2
z = (2.04×0.6) + (0.11×0.3) + (0.19×0.2) + (0.26×0.1) - 0.2
z = 1.224 + 0.033 + 0.038 + 0.026 - 0.2 = 1.121
a = ReLU(1.121) = 1.121

Neuron 2:
Weights: [0.2, 0.5, 0.4, 0.3]
Bias: -0.1
z = (2.04×0.2) + (0.11×0.5) + (0.19×0.4) + (0.26×0.3) - 0.1
z = 0.408 + 0.055 + 0.076 + 0.078 - 0.1 = 0.517
a = ReLU(0.517) = 0.517

Neuron 3:
Weights: [0.1, 0.4, 0.3, 0.5]
Bias: -0.15
z = (2.04×0.1) + (0.11×0.4) + (0.19×0.3) + (0.26×0.5) - 0.15
z = 0.204 + 0.044 + 0.057 + 0.13 - 0.15 = 0.285
a = ReLU(0.285) = 0.285

Hidden Layer 2 Output: [1.121, 0.517, 0.285]

Layer 3: Hidden Layer 2 → Output Layer
Final layer! 2 outputs: "probability it's a 1" and "probability it's a 0"
Output Neuron 1 (for digit "1"):
Weights: [0.8, 0.4, 0.2]
Bias: -0.3
z = (1.121×0.8) + (0.517×0.4) + (0.285×0.2) - 0.3
z = 0.897 + 0.207 + 0.057 - 0.3 = 0.861

Output Neuron 2 (for digit "0"):
Weights: [0.1, 0.3, 0.5]
Bias: -0.2
z = (1.121×0.1) + (0.517×0.3) + (0.285×0.5) - 0.2
z = 0.112 + 0.155 + 0.143 - 0.2 = 0.210
Raw outputs: [0.861, 0.210]
Apply Softmax (converts to probabilities):
exp(0.861) = 2.366
exp(0.210) = 1.234
Sum = 2.366 + 1.234 = 3.600
P(digit is "1") = 2.366 / 3.600 = 0.657 = 65.7%
P(digit is "0") = 1.234 / 3.600 = 0.343 = 34.3%

Result: The network predicts "1" with 65.7% confidence!

Takeaways:

- Each neuron multiplies inputs by weights, adds them up, adds a bias, then applies activation. Weights are learned during training to recognize patterns (like vertical edges for "1"). Layers stack: outputs of one layer become inputs to the next. The network transforms raw pixels → abstract features → decision

- How Are Weights Defined? (Training Process)
You DON'T manually set weights! That would be impossible for large networks. Instead, the network learns them automatically through a process called training.
The Training Process:
     Step 1: Start with Random Weights
     Initially, all weights are random small numbers like:
     w = [0.03, -0.12, 0.08, ...]

     Step 2: Forward Pass (What We Just Did)
     Feed an image through the network
     Get a prediction (e.g., "65.7% it's a 1")

     Step 3: Calculate Error (Loss)
     True label: "1" → [1, 0] (100% sure it's 1, 0% it's 0)
     Network prediction: [0.657, 0.343]
     Error = How wrong are we?
     Loss = (1 - 0.657)² + (0 - 0.343)² = 0.235

     Step 4: Backpropagation (The Magic) This is where calculus comes in! The algorithm:
     Calculates: "If I change weight w₁ by a tiny bit, how much does the error change?"
     Adjusts weights in the direction that reduces error
     Mathematical update rule:
     new_weight = old_weight - (learning_rate × gradient)
     Example:
     w₁ = 0.5 - (0.01 × 0.3) = 0.497

     Step 5: Repeat Thousands of Times
     Show the network 1000s of images of digits
     Each time, adjust weights slightly
     Over time, weights learn to recognize patterns:
     Some weights become sensitive to vertical edges (good for detecting "1")
     Some learn curves (good for detecting "0", "6", "8")

- Do You Retrain for Dogs vs Cats?: 
Yes, but: 
Option A: Train from Scratch
Start with random weights. Feed 10,000 images of dogs and cats. Let the network learn weights that detect: Fur textures, Ear shapes (pointy cat ears vs floppy dog ears), Nose patterns, Eye positions
Option B: Transfer Learning (Much Faster)
Why retrain everything? Low-level features are similar across tasks. 
Layer 1 learns: edges, corners, colors
Layer 2 learns: textures, simple shapes
Layer 3 learns: fur patterns, facial features
Layer 4 learns: "this is a dog face" or "this is a cat face"
Process:
Take a network already trained on 1 million images
Freeze the first 3 layers (keep their weights)
Only retrain the last layer for dogs vs cats
Done in 100s of images instead of 10,000s. 

- Why Activation Functions? (ReLU, Softmax)
Without Activation Functions:
Layer 1: z₁ = w₁×input + b₁
Layer 2: z₂ = w₂×z₁ + b₂
Layer 3: z₃ = w₃×z₂ + b₃
Substitute:
z₃ = w₃×(w₂×(w₁×input + b₁) + b₂) + b₃
   = (w₃×w₂×w₁)×input + (combined biases)
   = W×input + B  ← Just a single linear equation!
Problem: Multiple layers collapse into ONE layer. Deep networks become useless.

ReLU (Rectified Linear Unit)
Definition: ReLU(x) = max(0, x)
Graph:
  |
  |     /
  |    /
  |   /
__|__/______
  |
Why use it?
Introduces Non-linearity
If neuron gets negative input → outputs 0 (neuron "turns off")
If positive → passes through unchanged
This lets networks learn complex patterns like:
- "Activate only if there's a vertical edge AND in the center"
- "Detect curves but ignore straight lines"
Biological Inspiration Real neurons fire only when stimulated enough (threshold behavior)
Computationally Simple
ReLU: just compare to 0 vs Sigmoid: e^x / (1 + e^x)  ← expensive
Hand Calculation Example:

z = -1.5  → ReLU(-1.5) = 0
z = 0.3   → ReLU(0.3) = 0.3
z = 2.7   → ReLU(2.7) = 2.7

Softmax (Output Layer)
Purpose: Convert raw numbers into probabilities that sum to 1.
Formula:
For outputs [z₁, z₂, ..., zₙ]:
softmax(z₁) = e^z₁ / (e^z₁ + e^z² + ... + e^zₙ)
Example (Dog vs Cat):
Raw outputs: [2.3, 0.8]
              dog  cat
Step 1: Exponentiate
e^2.3 = 9.97
e^0.8 = 2.23
Step 2: Normalize
P(dog) = 9.97 / (9.97 + 2.23) = 9.97 / 12.2 = 0.817 = 81.7%
P(cat) = 2.23 / 12.2 = 0.183 = 18.3%
Check: 81.7% + 18.3% = 100% 
Why Exponential?
Amplifies differences: high scores get MUCH higher probability
Always positive: probabilities can't be negative
Differentiable: needed for backpropagation

In further detail: 
The neural network is fundamentally a mathematical function. You feed it numbers (the pixel values of your image), and it performs a series of calculations, giving you back numbers that represent probabilities. The entire process is just multiplication, addition, and a few simple functions applied over and over.
The beautiful part is that we do not tell the network what makes a dog a dog or what makes a cat a cat. Instead, we show it thousands of examples, and it figures out the patterns by itself through a process called training.

Think of a neuron as a tiny decision maker. It looks at several inputs, weighs their importance, and produces one output.
Here is the mathematical formula for one neuron:
output = activation_function(w₁×x₁ + w₂×x₂ + w₃×x₃ + ... + wₙ×xₙ + b)
Let me break down each component:
The inputs (x₁, x₂, x₃, etc.) are the data coming into this neuron. For the first layer, these might be pixel brightness values. For deeper layers, these are outputs from previous neurons.
The weights (w₁, w₂, w₃, etc.) represent how important each input is. A large positive weight means "this input strongly influences me to activate." A large negative weight means "this input strongly influences me to stay quiet." A weight near zero means "I don't really care about this input."
The bias (b) is like a threshold that shifts the neuron's sensitivity. If the bias is negative, the neuron needs more total input to activate. If positive, it activates more easily.
The activation function introduces non-linearity, which I will explain in detail shortly.

Hand Calculation Example - Single Neuron
Let me give you a tiny example you can verify right now:
Inputs: x₁ = 0.8, x₂ = 0.3, x₃ = 0.5
Weights: w₁ = 0.5, w₂ = 0.7, w₃ = 0.2
Bias: b = -0.3
Calculate the weighted sum (called z):
z = (0.5 × 0.8) + (0.7 × 0.3) + (0.2 × 0.5) + (-0.3)
z = 0.4 + 0.21 + 0.1 - 0.3
z = 0.41
Now apply the ReLU activation function (which I will explain soon):
ReLU(z) = max(0, z) = max(0, 0.41) = 0.41
So this neuron outputs 0.41. 

Understanding Activation Functions
Why Do We Need Them?
This is crucial to understand. Let me show you what happens without activation functions.
Imagine a three-layer network where each layer just does weighted sums:
Layer 1: y₁ = W₁×input + b₁
Layer 2: y₂ = W₂×y₁ + b₂ = W₂×(W₁×input + b₁) + b₂
Layer 3: y₃ = W₃×y₂ + b₃ = W₃×(W₂×(W₁×input + b₁) + b₂) + b₃
If you multiply this all out (which you can try on paper), you get:
y₃ = (W₃×W₂×W₁)×input + (some combined biases)
This simplifies to just one big weighted sum! All three layers collapse into a single operation. This means your deep network gains no benefit from having multiple layers. It is like stacking multiple straight ramps, you still just get one straight ramp in the end.
Activation functions curve these ramps, allowing the network to learn complex, curved decision boundaries. This is what lets neural networks approximate virtually any function.

ReLU (Rectified Linear Unit)
The ReLU function is beautifully simple:
ReLU(x) = x    if x > 0
ReLU(x) = 0    if x ≤ 0
Or more compactly: ReLU(x) = max(0, x)
When a neuron's weighted sum is negative, ReLU makes it output zero. The neuron is essentially "off" or "silent." When positive, the neuron passes the value through unchanged. The neuron is "on."
This creates a threshold effect similar to real biological neurons. A neuron fires when it receives enough stimulation, and stays silent otherwise.
Sample calculations:
ReLU(-2.3) = 0
ReLU(0.0) = 0
ReLU(1.7) = 1.7
ReLU(5.2) = 5.2

Sigmoid Function
Another common activation function is the sigmoid, which squashes any input into a range between zero and one: σ(x) = 1 / (1 + e^(-x))
The sigmoid has a characteristic S-shaped curve. For large negative numbers, it outputs values close to zero. For large positive numbers, it outputs values close to one. For numbers near zero, it produces values near 0.5.
Sample calculations:
σ(0) = 1 / (1 + e^0) = 1 / (1 + 1) = 0.5
σ(2) = 1 / (1 + e^(-2)) = 1 / (1 + 0.135) = 1 / 1.135 = 0.88
σ(-2) = 1 / (1 + e^2) = 1 / (1 + 7.389) = 1 / 8.389 = 0.12
The sigmoid was popular historically, but ReLU has largely replaced it in hidden layers because ReLU is simpler to compute and helps networks train faster.

Softmax (For Output Layer)
When we want to classify into multiple categories (dog, cat, bird, etc.), we need outputs that represent probabilities. The softmax function converts a vector of numbers into a probability distribution:
For inputs [z₁, z₂, ..., zₙ]:
softmax(zᵢ) = e^zᵢ / (e^z₁ + e^z² + ... + e^zₙ)
The key properties are that all outputs are positive, and they sum to exactly one.
Hand calculation example:
Raw scores: [2.0, 1.0, 0.1]
             dog  cat  bird
Step 1: Exponentiate each score
e^2.0 = 7.389
e^1.0 = 2.718
e^0.1 = 1.105
Step 2: Sum them
Sum = 7.389 + 2.718 + 1.105 = 11.212
Step 3: Divide each by the sum
P(dog) = 7.389 / 11.212 = 0.659 = 65.9%
P(cat) = 2.718 / 11.212 = 0.242 = 24.2%
P(bird) = 1.105 / 11.212 = 0.099 = 9.9%
Check: 65.9% + 24.2% + 9.9% = 100.0% 
The exponential function amplifies differences, so the highest score gets an even higher probability.

---

Let me design an extremely small network again (another example) that you can fully calculate by hand. This will be unrealistically small (real networks are much larger), but it captures all the essential concepts.

Network Architecture:
Input Layer: 4 pixels (our "image" is just 2×2 pixels, grayscale)
Hidden Layer: 3 neurons
Output Layer: 2 neurons (dog probability and cat probability)
Our image is represented as four numbers between zero and one, where zero is black and one is white.

Complete Forward Pass (Prediction)
Let me walk you through predicting whether an image is a dog or cat with actual numbers you can verify.
Step 1: The Input Image
Our 2×2 grayscale image:
[0.9  0.2]
[0.8  0.3]
Flattened input vector: x = [0.9, 0.2, 0.8, 0.3]
Imagine this represents a simple dark blob on the right (maybe part of a dog's nose) and brightness on the left (maybe fur).

Weights and Biases (First Hidden Layer)
I am going to give you the weights for all three neurons in the hidden layer. In a real network, these start random and are learned. For now, pretend we already trained the network and these are the learned values.
Neuron 1 in Hidden Layer:
Weights: w₁ = [0.6, 0.4, 0.5, 0.3]
Bias: b₁ = -0.4
Calculate the weighted sum:
z₁ = (0.6 × 0.9) + (0.4 × 0.2) + (0.5 × 0.8) + (0.3 × 0.3) - 0.4
z₁ = 0.54 + 0.08 + 0.40 + 0.09 - 0.4
z₁ = 1.11 - 0.4
z₁ = 0.71
Apply ReLU activation:
a₁ = ReLU(0.71) = 0.71
Neuron 2 in Hidden Layer:
Weights: w₂ = [0.2, 0.7, 0.3, 0.6]
Bias: b₂ = -0.3
Calculate:
z₂ = (0.2 × 0.9) + (0.7 × 0.2) + (0.3 × 0.8) + (0.6 × 0.3) - 0.3
z₂ = 0.18 + 0.14 + 0.24 + 0.18 - 0.3
z₂ = 0.74 - 0.3
z₂ = 0.44
a₂ = ReLU(0.44) = 0.44
Neuron 3 in Hidden Layer:
Weights: w₃ = [0.3, 0.5, 0.2, 0.8]
Bias: b₃ = -0.5
Calculate:
z₃ = (0.3 × 0.9) + (0.5 × 0.2) + (0.2 × 0.8) + (0.8 × 0.3) - 0.5
z₃ = 0.27 + 0.10 + 0.16 + 0.24 - 0.5
z₃ = 0.77 - 0.5
z₃ = 0.27
a₃ = ReLU(0.27) = 0.27
Output of Hidden Layer: [0.71, 0.44, 0.27]
Output Layer
Now these three hidden neuron outputs become inputs to our final two output neurons.
Output Neuron 1 (Dog):
Weights: w_dog = [0.8, 0.6, 0.3]
Bias: b_dog = -0.2
Calculate:
z_dog = (0.8 × 0.71) + (0.6 × 0.44) + (0.3 × 0.27) - 0.2
z_dog = 0.568 + 0.264 + 0.081 - 0.2
z_dog = 0.913 - 0.2
z_dog = 0.713
Output Neuron 2 (Cat):
Weights: w_cat = [0.3, 0.7, 0.9]
Bias: b_cat = -0.3
Calculate:
z_cat = (0.3 × 0.71) + (0.7 × 0.44) + (0.9 × 0.27) - 0.3
z_cat = 0.213 + 0.308 + 0.243 - 0.3
z_cat = 0.764 - 0.3
z_cat = 0.464
Raw outputs: [0.713, 0.464]
Step 4: Apply Softmax
Convert these raw scores to probabilities:
e^0.713 = 2.040
e^0.464 = 1.590
Sum = 2.040 + 1.590 = 3.630
P(dog) = 2.040 / 3.630 = 0.562 = 56.2%
P(cat) = 1.590 / 3.630 = 0.438 = 43.8%
Final prediction: DOG (56.2% confidence)
This is the entire forward pass through the network.

Training - How Weights Are Learned
Now comes the most important question: where did those weights come from? This is where training happens.
The Training Dataset: Before training, we need labeled data. Imagine we have collected one thousand images:
Five hundred images of dogs, each labeled "dog"
Five hundred images of cats, each labeled "cat"
Each image is our input, and the label is what we want the network to output.
Training Process Overview: Training happens in iterations called epochs. In each epoch, we show the network every image in our training set, and we adjust the weights to make predictions better.

Here is the cycle:
> First: Initialize all weights randomly (small numbers like 0.01, negative 0.03, 0.05, etc.)
> Second: For each training image, do a forward pass (like we just did) and get a prediction.
> Third: Calculate how wrong the prediction was. This is called the loss or error.
> Fourth: Use calculus to figure out how to adjust each weight to reduce the error. This is called backpropagation.
> Fifth: Update all weights slightly in the direction that reduces error.
> Sixth: Repeat for all images, then repeat the entire process for many epochs until the network gets good at predictions.

Calculating Loss (Mean Squared Error)
Let me show you how we measure how wrong our prediction was.
Suppose our network predicted [0.562, 0.438] for dog and cat, but the true label was "dog", which we represent as [1.0, 0.0] (one hundred percent dog, zero percent cat).
The loss function measures the difference:
Loss = (1/2) × [(predicted_dog - true_dog)² + (predicted_cat - true_cat)²]
Loss = (1/2) × [(0.562 - 1.0)² + (0.438 - 0.0)²]
Loss = (1/2) × [(-0.438)² + (0.438)²]
Loss = (1/2) × [0.192 + 0.192]
Loss = (1/2) × 0.384
Loss = 0.192
A perfect prediction (outputting exactly [1.0, 0.0]) would give us a loss of zero. The larger the loss, the worse our prediction.

Gradient Descent and Backpropagation
This is where calculus enters the picture, but I will explain it conceptually first, then mathematically.
The Concept: Imagine you are standing on a hilly landscape in thick fog. You cannot see where the lowest point is, but you can feel the slope under your feet. If you always walk downhill, eventually you will reach a valley (a low point). That is gradient descent.
The "landscape" is actually a mathematical surface where the height represents the loss (error). Each weight in your network corresponds to one dimension in this landscape. Our goal is to find the combination of weights that gives us the minimum loss.
The Mathematics: For each weight, we calculate the derivative of the loss with respect to that weight. This derivative tells us: "if I increase this weight by a tiny amount, how much does the loss change?"

Let me show you a simplified example with one weight:
Suppose we have one weight w = 0.5
After forward pass, loss L = 0.192
We compute: dL/dw (the derivative of loss with respect to w)
Suppose dL/dw = 0.35
This means: "if I increase w slightly, the loss will increase by about 0.35 times that amount"
The update rule is:
new_w = old_w - (learning_rate × dL/dw)
The learning rate (often 0.01 or 0.001) controls how big our steps are. Let's say learning rate equals 0.01:
new_w = 0.5 - (0.01 × 0.35)
new_w = 0.5 - 0.0035
new_w = 0.4965
We moved the weight slightly in the direction that decreases loss!

Backpropagation: The Chain Rule
Computing these derivatives for all weights is complex because the network has many layers. Backpropagation uses the chain rule from calculus to efficiently compute all derivatives.
The chain rule states: If y depends on z, and z depends on x, then: dy/dx = (dy/dz) × (dz/dx)
For our network:
Loss depends on output layer activations
Output activations depend on output layer weights
Output activations also depend on hidden layer activations
Hidden activations depend on hidden layer weights
Hidden activations depend on input
By applying the chain rule backwards through the network (hence "back" propagation), we can compute the derivative of the loss with respect to every single weight.

Let me show you a concrete calculation for one weight in the output layer. This requires some calculus, but I will go step by step.
Computing Gradient for Output Weight:
Remember our output neuron for "dog":
z_dog = w₁×h₁ + w₂×h₂ + w₃×h₃ + b
      = (0.8 × 0.71) + (0.6 × 0.44) + (0.3 × 0.27) - 0.2
      = 0.713
After softmax: p_dog = 0.562
True label: y_dog = 1.0
The derivative of the loss with respect to the output (before softmax) is:
dL/dz_dog = p_dog - y_dog = 0.562 - 1.0 = -0.438
Now, to find how much the loss changes with respect to the first weight (connecting hidden neuron one to the dog output):
dL/dw₁ = dL/dz_dog × dz_dog/dw₁
       = dL/dz_dog × h₁
       = -0.438 × 0.71
       = -0.311
Update the weight:
w₁_new = w₁_old - (learning_rate × dL/dw₁)
       = 0.8 - (0.01 × (-0.311))
       = 0.8 + 0.00311
       = 0.80311
Notice the weight increased slightly! That is because the derivative was negative, meaning increasing this weight will decrease the loss. The network wants to predict "dog" more strongly (which makes sense since the true label was dog).
You would repeat this calculation for every single weight in the network.

Complete Training Loop
Pseudocode that shows the entire training process:
Initialize all weights randomly
For each epoch (let's say 100 epochs):
    For each image in training set:
        # Forward Pass
        1. Feed image through network
        2. Get prediction (probabilities)
        # Calculate Loss
        3. Compare prediction to true label
        4. Calculate error (loss)
        # Backward Pass (Backpropagation)
        5. For each weight in output layer:
           - Calculate derivative dL/dw
           - Update: w_new = w_old - (learning_rate × dL/dw)
        6. For each weight in hidden layers (going backwards):
           - Use chain rule to calculate derivative dL/dw
           - Update: w_new = w_old - (learning_rate × dL/dw)
    # After seeing all images once:
    Calculate average loss across all training images
    Print progress: "Epoch 10: Average Loss = 0.143"
    If loss is low enough, stop training
After many epochs, the loss decreases, and the network gets better at distinguishing dogs from cats.

What Is the Network Actually Learning?
The weights encode patterns:
> First layer weights might learn to detect simple features like edges, corners, or color blobs. One neuron might activate strongly when it sees a vertical edge. Another might respond to horizontal edges.
> Second layer weights combine these simple features into more complex patterns. They might detect textures (fur), shapes (triangular ears), or patterns (stripes, spots).
> Third layer weights combine complex features into complete concepts. They learn to recognize "this combination of features means dog" versus "this combination means cat."
The network builds a hierarchy of understanding, from simple to complex, all automatically from the data!

The Complete ML Lifecycle
Now let me walk you through the entire process from start to finish, as you would do in a real project.

Phase 1: Problem Definition and Data Collection
First, clearly define what you want to predict. In our case: given an image, classify it as dog or cat. Next, collect your dataset. You might scrape images from the internet, use an existing dataset like ImageNet, or take your own photos. You need thousands of images, ideally balanced (equal numbers of dogs and cats). Each image must be labeled. This is often done manually or using crowdsourcing platforms. The quality of your labels directly affects your model's quality.

Phase 2: Data Preprocessing
Real-world data is messy. You need to clean and standardize it:
Resize images to a consistent size (say 224×224 pixels). Neural networks expect fixed-size inputs. Normalize pixel values from the range [0, 255] to [0, 1] by dividing by 255. This helps training converge faster. Split your data into three sets:
Training set (70% of data): used to train the network
Validation set (15% of data): used to tune hyperparameters
Test set (15% of data): used only at the end to evaluate final performance
Augment your data: Create variations of training images by randomly flipping, rotating, cropping, or adjusting brightness. This helps the network generalize better.

Phase 3: Model Architecture Design
Decide on your network structure. For image classification, convolutional neural networks (CNNs) work best, but the principles are the same as what we discussed.
Choose:
Number of layers
Number of neurons per layer
Activation functions
Output layer structure (softmax with two outputs for dog vs cat)

Phase 4: Training
Initialize weights randomly. Set hyperparameters like learning rate (0.001 is a common starting point), batch size (how many images to process before updating weights, often 32 or 64), and number of epochs (how many times to go through the entire dataset, often 50-200).
Run the training loop I described earlier. Monitor two metrics:
Training loss: Error on the training set. This should decrease steadily.
Validation loss: Error on the validation set (data the network has never seen during training). This should also decrease, but might level off or increase if the network starts overfitting (memorizing training data instead of learning general patterns).
If validation loss stops decreasing while training loss keeps dropping, you are overfitting. Solutions include:
Getting more training data
Simplifying the model (fewer layers or neurons)
Using regularization techniques like dropout
Stopping training earlier

Phase 5: Hyperparameter Tuning
Use the validation set to experiment with different settings. Try different learning rates (0.01, 0.001, 0.0001). Try different architectures (more or fewer layers). Try different batch sizes.
This phase is iterative. You train many models with different settings and pick the one that performs best on the validation set.

Phase 6: Final Evaluation
Once you are satisfied with your model's performance on the validation set, evaluate it one final time on the test set. This gives you an unbiased estimate of how well the model will perform on completely new data in the real world.
Calculate metrics like:
Accuracy: What percentage of images did you classify correctly?
Precision: Of the images you labeled as "dog," what percentage were actually dogs?
Recall: Of all the actual dog images, what percentage did you correctly identify?

Phase 7: Deployment
If the test performance is good enough for your use case, deploy the model. This might mean:
Creating an API that accepts images and returns predictions
Embedding the model in a mobile app
Running the model on a web server

Phase 8: Monitoring and Maintenance
After deployment, monitor the model's performance on real-world data. Performance might degrade over time if the data distribution changes (for example, if people start uploading different types of dog breeds).
Periodically retrain the model with new data to keep it accurate.

Part 9: Key Insights and Common Pitfalls
Neural networks are universal function approximators. Given enough neurons and layers, they can theoretically approximate any continuous function. This is why they work for such diverse tasks.
More data usually beats better algorithms. A simple neural network trained on one million images will often outperform a sophisticated network trained on ten thousand images.
Training is expensive, inference is cheap. Training might take hours or days on powerful computers. But once trained, making predictions is fast, often milliseconds per image.
Overfitting is the main challenge. Networks are so powerful they can memorize training data perfectly, but then fail on new data. Regularization, dropout, data augmentation, and early stopping help combat this.
Deeper networks learn hierarchical features. Early layers learn simple patterns, later layers learn complex combinations. This is why deep learning works so well for images, where natural hierarchies exist (pixels → edges → shapes → objects).

From my experience working at Simpl.

• Some brief: We had a monolith Ruby-Rails service which was fairly complex having frontend/backend/kafka consumers related components. It interacted with various postgres RDS tables, external APIs, Kafka, RabbitMQ cluster(s). It was deployed on ECS, running over 30 tasks in parallel.
• Issue: Our alerts (Cloudwatch+Zenduty) notified us of an unusual drop in database connection counts (these were constant-workload tables by the way). The service appeared healthy on the surface as well as the frontend UI component of the service was working fine, but there were issues underneath.
• Debugging issue:
  • We were looking at various metrics, one of them being the ECS task status, where we discovered 80%+ containers were caught in a restart loop, continuously failing and being restarted by ECS. Only some containers remained stable and running. Since it was an internal service, traffic volume was relatively low, so few surviving containers were able handle the incoming load without any obvious performance degradation. Had the traffic been high, the service would have definitely impacted.
  • Tracing backward from the database connection drop, we checked the service logs and it showed connection errors, but not to our primary databases, instead failed connections to a RabbitMQ cluster that we had deprecated few days back.
  • As we traced back, the codebase still contained initialization code that attempted to establish connections to the old RabbitMQ cluster during container startup. It probably wasn’t cleaned up properly since the logic is buried in legacy modules.
  • So this was the presumed issue flow: ECS launches a new container task -> Ruby application begins initialization -> Legacy module attempts to connect to deprecated RabbitMQ cluster -> Connection fails (cluster no longer accessible) -> Failure triggers application crash or health check failure -> ECS detects unhealthy container and restarts it -> DB connections drop triggering alerts -> Cycle repeats indefinitely.
  • Few running containers were pretty much a ticking time-bomb, that would’ve also failed anyways.
  • Graceful degradation is important, but sometimes, your code should fail-fast.
  • We removed the loose connections from the codebase & cleaned up the old code. Also improved handling during connection initializations and later deployed all changes.

From my experience working at Simpl.

I worked on an interesting task along with the devops team of cleaning up a bulky legacy Postgres table in the db, which was causing cluster-wide performance degradation (Eg: Auto-vacuum processes were triggering frequently, locking table and impacting writes across the entire database cluster). Example: The table was loaded with real-time data from a Kafka consumer running round the clock. It had over ~ 1.6B rows having roughly a month of data; ~ 600GB total size; Of which indexes size ~ 125GB, toast size ~ 125GB. A daily batch job was running to delete data older than 30 days. Command which it used: DELETE FROM <tableNameX> WHERE created_at < '<current_time_minus_thirty_days>' on the writer postgres instance - and this is not an effective command (discussed later)

Query to get metadata of all tables in db:

SELECT
    t.table_schema || '.' || t.table_name AS table_full_name,
    COALESCE(c.reltuples, 0) AS estimated_rows,
    pg_size_pretty(pg_total_relation_size(quote_ident(t.table_schema) || '.' || quote_ident(t.table_name))) AS total_size,
    pg_size_pretty(pg_relation_size(quote_ident(t.table_schema) || '.' || quote_ident(t.table_name))) AS table_size,
    pg_size_pretty(pg_indexes_size(quote_ident(t.table_schema) || '.' || quote_ident(t.table_name))) AS indexes_size,
    pg_size_pretty(pg_total_relation_size(quote_ident(t.table_schema) || '.' || quote_ident(t.table_name)) - pg_relation_size(quote_ident(t.table_schema) || '.' || quote_ident(t.table_name))) AS toast_size
FROM
    information_schema.tables t
JOIN
    pg_class c ON c.relname = t.table_name
JOIN
    pg_namespace n ON n.nspname = t.table_schema AND n.oid = c.relnamespace
WHERE
    t.table_schema NOT IN ('information_schema', 'pg_catalog')
    AND t.table_type = 'BASE TABLE'
ORDER BY
    pg_total_relation_size(quote_ident(t.table_schema) || '.' || quote_ident(t.table_name)) DESC;

Crisp points learned & strategy:

• Post discussing with teams - it was found concrete requirement of data was only for 1 week, hence we could proceed with deleting data older than 7 days. Expected downtime was communicated in advance.
• Key PostgreSQL Concepts

  • By default, PostgreSQL is a single-node OLTP (Online Transaction Processing) database, with: one primary (writable) instance & no built-in read replicas. In real-world deployments, PostgreSQL is often extended like below. This setup is implemented using streaming replication or tools like: Amazon RDS/Aurora for PostgreSQL (managed reader endpoints), etc.

    Role	Description
    Writer (Primary)	Handles all write operations: INSERT, UPDATE, DELETE
    Reader (Replica)	Handles read-only queries. Replicated from primary.

  • Table Bloat:
    • When rows are updated or deleted in PostgreSQL, the old versions are not physically removed immediately. This leads to table bloat.
    • What Happens Internally:
      • PostgreSQL uses MVCC (Multi-Version Concurrency Control).
      • An UPDATE creates a new row version (tuple) and marks the old one as dead.
      • A DELETE marks the row as dead, but doesn’t remove it.
      • These dead tuples still occupy disk space.
      • Over time, with many updates/deletes, the table grows in size (bloats), even if the row count doesn’t.
    • Impact:
      • Slower sequential scans and index usage
      • Increased I/O due to reading bloated pages
      • Sluggish performance for frequently updated tables
  • Auto-vacuum:
    • Auto-Vacuum is PostgreSQL’s background process that automatically cleans up dead tuples to control bloat and maintain visibility maps.
    • What Happens Internally:
      • PostgreSQL tracks how many tuples are updated or deleted.
      • When thresholds are crossed (autovacuum_vacuum_threshold + fraction of table), the autovacuum daemon kicks in. It: Scans the table and visibility map, Removes dead tuples (if not visible to any active transaction), Updates the free space map (FSM) and visibility map
      • If it’s doing an aggressive freeze (e.g., nearing vacuum_freeze_max_age), it may take heavier locks or consume more I/O.
      • Notes:
        • Typically holds an ACCESS SHARE lock, which doesn’t block reads/writes.
        • On very large tables, it can compete with application queries for CPU and I/O.
        • If not tuned properly, autovacuum may lag behind, leading to excessive bloat or even transaction wraparound issues.
  • TOAST (The Oversized-Attribute Storage Technique):
    • TOAST handles storage of large data types like text, bytea, or jsonb that exceed a threshold (typically ~2KB).
    • What Happens Internally:
      • When a row contains a large column (e.g., a big text field), PostgreSQL:
        • Compresses the value (if possible)
        • If still too large, stores the value in a separate TOAST table
        • The main table stores a pointer to the TOAST data
      • The TOAST table is created automatically, one per main table.
      • TOAST data is stored in chunks (usually 2KB) in the TOAST table.
    • Cleanup Complexity:
      • VACUUM and autovacuum must also manage the TOAST table.
      • Large updates or deletes may leave dead TOAST tuples as well.
      • If TOAST cleanup is missed or delayed, it can cause hidden bloat.
• Coming back, existing deletion strategy using DELETE was not effective.
  • DELETE operations don’t reclaim disk space immediately
  • Creates massive amounts of dead tuples (bloat)
  • Triggers aggressive auto-vacuum cycles
  • Auto-vacuum locks table during cleanup
  • Impacts performance of concurrent writes
  • TOAST data cleanup is particularly expensive
• Solution Options Evaluated
  • Option 1: New Table with Different Name
    • Minimal downtime
    • Risk: Consumer services might miss updating table name
    • Rejected due to operational risk/backward compatibility in workflows.
  • Option 2: Recreate with Daily Partitions
    • Same table name maintained
    • 1-3 hours downtime as copying data to new table, dropping old table, renaming new table with old table name.
    • Data builds back over 7 days
  • Option 3: Drop and Recreate (Selected)
    • Was discussed and ensured losing old data is fine, and reloading fresh data.
    • 10-20 minute downtime
    • No backfill required
    • Minimal impact
      • Step 0: Readers (eg: Analytics workflows) and writers (eg: Kafka consumer) to the table were stopped temporarily, turned on later once the activity was complete
      • Step 1: Rename existing table

        ALTER TABLE tableNameX RENAME TO tableNameX_Old;
        

      • Step 2: Create partitioned table

        CREATE TABLE tableNameX (
        event_id                        uuid NOT NULL,
        event_timestamp                 timestamptz,
        event_type                      varchar(50),
        ....
        PRIMARY KEY (event_id, event_timestamp)
        ) PARTITION BY RANGE (event_timestamp);
        

      • Step 3: Configure pg_partman

        -- Create parent partitioning structure
        SELECT partman.create_parent(
            p_parent_table => 'public.tableNameX',
            p_control => 'event_timestamp',
            p_type => 'native',
            p_interval=> 'daily',
            p_premake => 365
        );
        -- Set retention policy
        UPDATE partman.part_config 
        SET infinite_time_partitions = true,
            retention = '6 days', 
            retention_keep_table = false 
        WHERE parent_table = 'public.tableNameX';
        

      • Step 4: Create indexes

        -- No need to create explicit index on event_timestamp and event_id as PostgreSQL automatically creates a unique B-tree index for the primary keys
        CREATE INDEX tableNameX_user_id_idx ON public.tableNameX (user_id);
        

• Key Benefits of Partitioning Approach:
  • Automatic partition management: pg_partman creates future partitions via cron
  • Efficient data removal: DROP PARTITION vs DELETE (instant vs hours)
  • Query transparency: Applications query main table; PostgreSQL routes to correct partition
  • No bloat accumulation: Old partitions dropped entirely
  • Predictable performance: Each partition remains manageable size
  • No manual intervention: Retention automatically enforced
• Others:
  • DELETE is not suitable for time-series data at scale
    • Creates bloat instead of freeing space
    • Triggers expensive auto-vacuum cycles
  • Partitioning is essential for large time-series tables
    • DROP PARTITION is instantaneous
    • No bloat, no vacuum needed
  • pg_partman simplifies partition management
    • Automatic partition creation and built-in retention policies
    • No custom scripts needed
  • Partition constraints are important
    • Once partitioned, you can add partitions but not remove partitioning
    • Plan partition strategy carefully
• Metrics monitored:
  • Memory/CPU usage
  • Running queries, Auto-vacuum frequency, etc.
  • Query performance
  • Disk space reclamation
• Alternative Approaches (Not Used)
  • pg_repack:
    • Would reclaim space but lock table
    • Not suitable for high-write tables
    • Temporary solution only
  • TRUNCATE:
    • Would lose all data
    • Not viable for production
  • Manual partition management:
    • Error-prone
    • Requires custom scripts
    • pg_partman is superior

Impact: Decrease in load (blue)