Mikhail Mikhasenko

Original Lecture-1A, slide 1. Visual source: Keynote PDF export.

Mikhail Mikhasenko Automatic differentiationLecture 1A

Why automatic differentiation?

From early neural networks to the need for reliable derivatives

Mark I Perceptron

Original Lecture-1A, slide 2. Visual source: Keynote PDF export.

Mark I Perceptron to reporters by Frank Rosenblatt. July 7, 1958 1958, The New York Times

Expected at ~1960

Original Lecture-1A, slide 3. Visual source: Keynote PDF export.

Expected at ~1960 •Walk, talk, see, and write happened in 2025 chatGPT voice mode Robot  Atlas by Boston Dynamics

What held the progress?

Original Lecture-1A, slide 4. Visual source: Keynote PDF export.

What held the progress? Data Compute Algorithms

Ways to compute derivatives

Problem formulation, symbolic differentiation, and finite differences

Diff: Problem formulation

Original Lecture-1A, slide 5. Visual source: Keynote PDF export.

Diff: Problem formulation f1(x): ⃗x∈Rnf1 R(scalar) Rnf Rm f1(x),…,fm(x)(multiple) f1,…,fm x1 ⋅ ⋅ ⋅ xn •Function •Derivative •Gradient •Jacobian ∂(f1,…,fm) ∂(x1,…,xn) ∂f1 ∂x1 (∂f1 ∂x1 ,∂f1 ∂x2 ,…,∂f1 ∂xn)

Several ways to get derivative

Original Lecture-1A, slide 6. Visual source: Keynote PDF export.

Several ways to get derivative Symbolic, Numeric, Automatic f(x)=f(x)= df dx=? Symbolic Numeric “Automatic”

Analytic Diff

Original Lecture-1A, slide 7. Visual source: Keynote PDF export.

Analytic Diff Warm up •Follow standard rules compute a derivative

Finite Diff

Original Lecture-1A, slide 8. Visual source: Keynote PDF export.

Finite Diff Choose right answer A B C D df dx≈fx+Δx−f(x−Δx) 2Δx

Forward mode

Propagate values and derivative components together

Forward Diff

Original Lecture-1A, slide 9. Visual source: Keynote PDF export.

Forward Diff Dual numbers •Augment each number with a derivative component. •We use dual numbers: , where 
 •Define operations on dual numbers •For higher dims, one column per pass ̂x=x+·xε ε2=0 f(x+·xε)=f(x)+f′ (x)·xε J=[f(x1+ε,x2,…,xn)f(x1,x2+ε,…,xn)…]

Example to work out

Original Lecture-1A, slide 10. Visual source: Keynote PDF export.

Example to work out c→(c,0) x→(x,1)

Reverse mode and backpropagation

Evaluate forwards, then propagate sensitivities backwards

Reverse Diff

Original Lecture-1A, slide 11. Visual source: Keynote PDF export.

Reverse Diff •Operate with a computation graph, pulling back effects on the final computation node to the leaves. •Two operations to understand:
 * for the chain rule
 + node combination •Not expression tree but computation graph

Backpropagation, step by step

We want the derivative of one loss with respect to every parameter.

$$L(\boldsymbol\theta),\qquad \nabla_{\boldsymbol\theta}L =\left(\frac{\partial L}{\partial\theta_1},\ldots, \frac{\partial L}{\partial\theta_n}\right)$$

Evaluate the program forwards and retain the values needed for derivatives.

Then propagate sensitivities backwards, using one local derivative at a time.

We will compute every step by hand before running the demonstration.

One graph, two paths

$$a=wx,\qquad b=a^2,\qquad c=b+a,\qquad L=\tfrac12(c-t)^2$$

The value $a$ affects the loss through two paths.

We train $w$. The input $x=2$ and target $t=5$ stay fixed.

Trace both paths from a to the loss. This shared intermediate is why the reverse pass must accumulate derivatives. The graph is an illustrative scalar program; the neural-network connection follows later in the block.

1. Forward pass

Start with $w=1$, $x=2$, and $t=5$.

$$a=1\cdot2=2,\quad b=2^2=4,\quad c=4+2=6$$ $$L=\tfrac12(6-5)^2=0.5$$

Keep the intermediate values. The backward pass will use them.

2. What travels backwards?

The adjoint of an intermediate $v$ is its sensitivity:

$$\bar v\equiv\frac{\partial L}{\partial v}$$

For a small perturbation, $\Delta L\approx\bar v\,\Delta v$.

Forward value

“What number did this operation produce?”

Backward sensitivity

“How does this number affect the loss?”

$$\bar L=\frac{\partial L}{\partial L}=1$$

The seed is one because the loss changes by one unit per unit change of itself. The number travelling backwards is a derivative, not simply the numerical loss.

Local backward rules

For an incoming sensitivity $\bar z$:

Addition

$$z=u+v$$ $$\bar u\mathrel{+}=\bar z$$ $$\bar v\mathrel{+}=\bar z$$ Send the sensitivity to both inputs.

Multiplication

$$z=uv$$ $$\bar u\mathrel{+}=v\bar z$$ $$\bar v\mathrel{+}=u\bar z$$ Use the other input’s forward value.

Squaring

$$z=u^2$$ $$\bar u\mathrel{+}=2u\bar z$$ Use the local slope at the stored value.

The notation $\mathrel{+}=$ means add this contribution to the adjoint.

3. Backwards through the loss

$$L=\tfrac12(c-t)^2,\qquad c=6,\quad t=5$$

$$\bar c=\bar L\,\frac{\partial L}{\partial c} =1\cdot(c-t)=1$$

At this point, increasing $c$ slightly would increase the loss.

4. Backwards through the sum

$$c=b+a,\qquad\bar c=1$$

$$\bar b\mathrel{+}=\bar c\cdot1=1,\qquad \bar a\mathrel{+}=\bar c\cdot1=1$$

This is only the direct-path contribution to $\bar a$.

5. Backwards through the square

$$b=a^2,\qquad a=2,\quad\bar b=1$$

$$\bar a\mathrel{+}=\bar b\,\frac{\partial b}{\partial a} =1\cdot2a=4$$

Is $\bar a$ now 1, 4, or 5?

Pause here. Ask students to justify whether to overwrite the stored derivative or add to it. Do not reveal the next slide until the two paths are identified.

6. Accumulation at a shared node

$$\bar a= \underbrace{\bar c\frac{\partial c}{\partial a}}_{\text{direct path}} +\underbrace{\bar c\frac{\partial c}{\partial b} \frac{\partial b}{\partial a}}_{\text{path through }b} =1+4=5$$

Multiply derivatives along each path. Add contributions from different paths.

Only propagate further once all contributions to the node have arrived.

7. Backwards to the parameter

$$a=wx,\qquad w=1,\quad x=2,\quad\bar a=5$$

$$\bar w=\bar a\,x=5\cdot2=10,\qquad \bar x=\bar a\,w=5\cdot1=5$$

The derivative with respect to the trainable parameter is $\partial L/\partial w=10$. We can also compute $\bar x$, even though $x$ is fixed during this update.

8. A parameter update

Backpropagation has finished. Now apply gradient descent with $\eta=0.01$.

$$w_{\mathrm{new}}=w-\eta\bar w=1-0.01\cdot10=0.9$$

	Before	After
Parameter $w$	1	0.9
Prediction $c$	6	5.04
Loss $L$	0.5	0.0008

The new loss requires a new forward pass. A larger step may increase it.

With w=0.9: a=1.8, b=3.24, c=5.04, L=0.0008. Keep the roles separate: reverse mode calculates the gradient; the optimizer chooses and applies the parameter update. The fixed target remains five.

A neuron uses the same rules

$$z=w_1x_1+w_2x_2+b,\qquad h=\sigma(z),\qquad L=\tfrac12(h-t)^2$$

Work backwards, starting at the scalar loss:

$$\bar h=h-t,\qquad\bar z=\bar h\,\sigma'(z)$$ $$\bar w_1=\bar z\,x_1,\qquad \bar w_2=\bar z\,x_2,\qquad\bar b=\bar z$$

For a sigmoid, $\sigma'(z)=h(1-h)$: reuse the activation from the forward pass.

Backpropagation through layers

$$\mathbf z^{(\ell)}=W^{(\ell)}\mathbf h^{(\ell-1)}+\mathbf b^{(\ell)}, \qquad \mathbf h^{(\ell)}=\sigma(\mathbf z^{(\ell)})$$

Given the incoming sensitivity $\bar{\mathbf h}^{(\ell)}$:

$$\bar{\mathbf z}^{(\ell)}= \bar{\mathbf h}^{(\ell)}\odot\sigma'(\mathbf z^{(\ell)})$$ $$\bar W^{(\ell)}=\bar{\mathbf z}^{(\ell)} \big(\mathbf h^{(\ell-1)}\big)^T,\qquad \bar{\mathbf b}^{(\ell)}=\bar{\mathbf z}^{(\ell)}$$ $$\bar{\mathbf h}^{(\ell-1)}= \big(W^{(\ell)}\big)^T\bar{\mathbf z}^{(\ell)}$$

Repeat for the previous layer. Here $\odot$ denotes elementwise multiplication.

These equations are for one sample and a componentwise activation. For a batch, sum or average according to the loss definition. Shared parameters accumulate contributions from every use. No full network Jacobian needs to be constructed.

The reverse sweep in code

# Forward: record operations and their needed input values.
value = evaluate_in_dependency_order(graph)

# Backward: start with zero contributions everywhere.
adjoint = {node: 0.0 for node in graph}
adjoint[loss] = 1.0

for node in reversed(dependency_order):
    for input_edge in node.input_edges:
        parent = input_edge.source
        adjoint[parent] += (
            adjoint[node] * local_derivative(input_edge, value)
        )

Process edges, including repeated inputs. For $z=u\cdot u$, both edges contribute.

This is scalar pseudocode, not a framework API. The saved order must respect dependencies; reversing it ensures all downstream contributions are available. For vector operations the local derivative action is a vector-Jacobian product.

Checking a gradient

For our example, compute an independent central finite difference:

$$g_{\mathrm{FD}}=\frac{L(w+\varepsilon)-L(w-\varepsilon)}{2\varepsilon}$$

At $w=1$, $x=2$, $t=5$, and $\varepsilon=10^{-5}$:

$$g_{\mathrm{FD}}\approx10.000000004,\qquad g_{\mathrm{backprop}}=10$$

Use this as a diagnostic on a small, smooth example.

Very small finite-difference steps amplify roundoff. At a nondifferentiable point, the comparison needs special care.

One reverse pass, many parameters

For one scalar loss, the reverse sweep computes all parameter derivatives.

Each operation sends sensitivities to its inputs. The work scales with the computation graph, with a constant-factor overhead for standard operations.

The tradeoff is memory: backward rules need values from the forward pass.

Checkpointing saves selected values and recomputes others.

This prepares the original Jacobian-cost and memory slides that immediately follow the demonstration. Distinguish a scalar-output gradient from a full multi-output Jacobian. The precise factor depends on operations and implementation.

Backpropagation demonstration

First step through the preset example. Ask for the next value before revealing it. Then change w, repeat, and try a larger learning rate. The final step compares the old and new loss but does not silently update w. Reset restores all defaults. The PDF shows the fully worked default example.

Computation cost

Original Lecture-1A, slide 12. Visual source: Keynote PDF export.

Computation cost of Jacobian matrix f1,…,fm x1 ⋅ ⋅ ⋅ xn f1,…,fm x1 ⋅ ⋅ ⋅ xn x1 ⋅ ⋅ ⋅ xn f1,…,fm x1 ⋅ ⋅ ⋅ xn Rnf Rm f1(x),…,fm(x)(multiple) Forward AD Reverse AD∂(f1,…,fm) ∂(x1,…,xn) 1. dual 2. dual 3.… x1 x2 1.effect on
2.effect on 3.… f1 f2

Higher-Order Derivatives

Original Lecture-1A, slide 13. Visual source: Keynote PDF export.

Higher-Order Derivatives Hessian •Forward mode can be applied twice, or 
 use Hyper-dual numbers •Reverse mode can be applied to each gradient component to get row-wise Hessians f1(x): ⃗x∈Rnf1 R(scalar) ∂2f1 ∂xi∂xj f1,…,fm x1 ⋅ ⋅ ⋅ xn x1 ⋅xn Rnf Rm f1(x),…,fm(x)(multiple)

Practical Considerations

Original Lecture-1A, slide 14. Visual source: Keynote PDF export.

Practical Considerations Memory & Complexity •fAD has a small memory footprint — carry derivative alongside the computation •rAD must remember all intermediate results
 (GPT-3: 175 billion parameters Float16 ~350 GB),
 some mitigated with re-computation (Gradient- checkpointing) https://en.wikipedia.org/wiki/GPT-3 https://arxiv.org/pdf/1904.10631

Automatic differentiation in practice

Frameworks, transformations, compilers, and implementation trade-offs

Existing AD techniques

Original Lecture-1A, slide 15. Visual source: Keynote PDF export.

Existing AD techniques •Differentiable DSLs (TensorFlow, PyTorch) provide a new language where everything is differentiable. Must rewrite code in DSL. •Operator Overloading (Adept, JAX) tools provide differentiable versions of language constructs (e.g. np.sum => jax.sum). Require rewriting non-standard utlities for support. •Source Rewriting tools statically analyze code to produce a new gradient function in the source language. Requires all code available ahead of time, difficult with external libraries.

AD in Major ML Frameworks

Original Lecture-1A, slide 16. Visual source: Keynote PDF export.

AD in Major ML Frameworks Dynamic Tape reverse Trace to HLO Transform reverse Eager → Opt graph Tape or grad ops Default graph AD style PyTorch JAX TF 2 Baydin et al. “AD in ML” (JMLR 2018)

Python ecosystem

JAX · PyTorch · TensorFlow

Tool	Role	Differentiation interface
NumPy / SciPy	Numerical arrays and scientific algorithms	Derivatives supplied separately or approximated
SymPy	Symbolic expressions	diff(expression, variable)
JAX	Array programs and function transformations	jvp, vjp, grad
PyTorch	Tensors, neural networks, training	torch.func, .backward()
TensorFlow	Tensors, models, training	ForwardAccumulator, GradientTape
marimo	Reactive Python notebooks	Runs the chosen library’s code

NumPy and SciPy themselves are not general-purpose reverse-mode AD systems. Some SciPy routines provide finite-difference derivatives, accept derivatives, or support particular array backends. SymPy computes symbolic derivatives. marimo is the notebook interface, not an AD implementation. SymPy source: https://docs.sympy.org/latest/tutorials/intro-tutorial/calculus.html. marimo source: https://docs.marimo.io/.

Forward evaluation and forward-mode AD

For $\mathbf y=F(\mathbf x)$, ordinary evaluation computes $\mathbf y$.

Forward-mode AD also propagates an input direction $\mathbf v$:

$$\dot{\mathbf y}=J_F(\mathbf x)\,\mathbf v$$

Value

$$z=uv$$

Tangent

$$\dot z=\dot u\,v+u\,\dot v$$

A Jacobian-vector product (JVP) answers: how does the output change along this input direction?

A model’s ordinary forward pass is not automatically forward-mode differentiation. Forward-mode AD evaluates values and tangents together. The tangent seed is an infinitesimal direction, not a finite-difference step size. JAX API: https://docs.jax.dev/en/latest/_autosummary/jax.jvp.html.

Reverse mode: an output sensitivity

Evaluate $\mathbf y=F(\mathbf x)$, then choose an output sensitivity $\mathbf u$.

$$\bar{\mathbf x}=J_F(\mathbf x)^T\,\mathbf u$$

y, pullback = jax.vjp(F, x)
(sensitivity_x,) = pullback(u)

The pullback maps an output sensitivity back to the inputs.

For a scalar loss, seed the output with $1$ to obtain its gradient.

JVP: a direction through the inputs. VJP: a weighted combination of output sensitivities.

The library calls this a vector-Jacobian product (VJP). With sensitivities represented as column vectors, its returned input sensitivity is J transpose u. Source: https://docs.jax.dev/en/latest/_autosummary/jax.vjp.html.

JAX: the lecture’s scalar loss

import jax
jax.config.update("jax_enable_x64", True)

def loss(w):
    a = 2.0 * w
    return 0.5 * (a**2 + a - 5.0)**2

value, tangent = jax.jvp(loss, (1.0,), (1.0,))
value, pullback = jax.vjp(loss, 1.0)
(gradient,) = pullback(1.0)

print(value, tangent, gradient)   # 0.5, 10.0, 10.0
print(jax.grad(loss)(1.0))        # 10.0

The input direction is $1$. The scalar-loss output seed is also $1$.

The two seeds have different meanings despite having the same numerical value. The trailing comma constructs a one-element tuple, matching the function’s argument structure. grad is the convenient reverse-mode interface for a scalar output. Sources: JAX jvp, vjp, and grad API documentation.

marimo: compare the two modes

Live notebook · Snapshot · Python source

Change $w$ from $1$ to $0.9$: predict the loss and gradient before rerunning.

At w=0.9 the loss is 0.0008 and the gradient is 0.368. This is a screenshot of the executed notebook, not a live control. The source notebook is reactive when opened in marimo. The HTML link is a static snapshot. Launch: ../../.venv-lectures/bin/marimo edit notebooks/01_jax_scalar.py.

A vector function

$$F(x_1,x_2)=\begin{pmatrix}x_1x_2\\x_1^2+x_2\end{pmatrix}, \qquad \mathbf x=\begin{pmatrix}2\\3\end{pmatrix}, \qquad J=\begin{pmatrix}3&2\\4&1\end{pmatrix}$$

Forward: select a column

$$\mathbf v=\begin{pmatrix}1\\0\end{pmatrix},\qquad J\mathbf v=\begin{pmatrix}3\\4\end{pmatrix}$$

Reverse: select a row

$$\mathbf u=\begin{pmatrix}1\\0\end{pmatrix},\qquad J^T\mathbf u=\begin{pmatrix}3\\2\end{pmatrix}$$

For general seeds, compute linear combinations of columns or rows.

JAX: products and full Jacobians

import jax.numpy as jnp

def F(x):
    return jnp.array([x[0] * x[1], x[0]**2 + x[1]])

x = jnp.array([2.0, 3.0])
seed = jnp.array([1.0, 0.0])
y, Jv = jax.jvp(F, (x,), (seed,))  # [3., 4.]
y, pullback = jax.vjp(F, x)
(JTu,) = pullback(seed)            # [3., 2.]

J_forward = jax.jacfwd(F)(x)       # [[3., 2.],
J_reverse = jax.jacrev(F)(x)       #  [4., 1.]]

Both Jacobian functions return the same matrix. JVP and VJP need not build it.

This code continues from the previous JAX import. The notebook is standalone. Source: https://docs.jax.dev/en/latest/jacobian-vector-products.html.

marimo: input direction and output seed

Live notebook · Snapshot · Python source

Switch each seed to $(0,1)$. Which column or row should appear?

JVP with (0,1) gives (2,1). VJP with (0,1) gives (4,1). Open the live source to use the dropdowns; the slide and linked HTML are static.

PyTorch: forward and reverse transforms

PyTorch function transforms

import torch
from torch.func import jvp, vjp

w = torch.tensor(1.0, dtype=torch.float64)
one = torch.ones_like(w)

value, tangent = jvp(loss, (w,), (one,))
value, pullback = vjp(loss, w)
(gradient,) = pullback(one)

print(value.item())      # 0.5
print(tangent.item())    # 10.0
print(gradient.item())   # 10.0

Use the same scalar loss function. These transforms return derivatives directly.

PyTorch: a training step

w = torch.tensor(1.0, dtype=torch.float64, requires_grad=True)
optimizer = torch.optim.SGD([w], lr=0.01)

optimizer.zero_grad(set_to_none=True)
value = loss(w)                 # forward evaluation: 0.5
value.backward()               # reverse mode
print(w.grad.item())           # 10.0

optimizer.step()               # optimizer changes w
print(w.item())                # 0.9
print(loss(w).item())          # approximately 0.0008

.backward() accumulates into .grad. Clear gradients before the next step.

The optimizer update is separate from autodiff. zero_grad is normally called once per training step, before the backward call. Intentional accumulation across microbatches is an exception. A new forward graph is built on the next evaluation. Source: https://docs.pytorch.org/docs/stable/generated/torch.Tensor.backward.html.

marimo: one optimizer step

Live notebook · Snapshot · Python source

Increase the learning rate to $0.5$. The gradient is still correct. What happens to the loss?

The new weight is -4 and the new loss is 1300.5. Each reactive execution starts from the selected initial weight and takes exactly one step; there is no hidden training history. Open the Python source in marimo for the live sliders.

Choosing and checking an AD mode

For $F:\mathbb R^n\to\mathbb R^m$, think about what derivative you need.

Task	Usual starting point
One input direction	JVP / forward mode
Scalar loss, many parameters	Gradient / reverse mode
Full Jacobian, few inputs	jacfwd
Full Jacobian, few outputs	jacrev

Keep operations inside the library’s differentiable computation. Convert to NumPy or Python numbers after computing the derivative.

Check small examples analytically or with finite differences, and benchmark the real workload.

Mode choice is a rule of thumb, not a guarantee about wall-clock performance. JAX examples use pure functions and floating-point inputs. PyTorch detach, conversion to NumPy, and rebuilding a tensor can disconnect a gradient path. A model forward pass alone computes values; forward-mode AD also propagates tangents. Source: https://docs.jax.dev/en/latest/jacobian-vector-products.html.

Example: mutations and branchings

Original Lecture-1A, slide 17. Visual source: Keynote PDF export.

Example: mutations and branchings

Example: differentiating LLVM

Original Lecture-1A, slide 18. Visual source: Keynote PDF export.

Example: differentiating LLVM

From Source to Machine Code

Original Lecture-1A, slide 19. Visual source: Keynote PDF export.

From Source to Machine Code XLA & LLVM •Pipeline
 Source (Julia/Python) → Front-end IR → Optimizer → Machine Code •XLA: linear-algebra-centric compiler used by JAX / TF; IR = HLO •LLVM: language-agnostic compiler back-end (Julia, C/C++, Rust)

Historical and conceptual epilogue

An old algorithm, modern scale, and the cheap-gradient principle

Backpropagation is older than deep learning

1970 — Seppo Linnainmaa
Systematic reverse accumulation for differentiating a program.

1974 — Paul Werbos
Connected backward differentiation with learning systems.

1986 — Rumelhart, Hinton & Williams
Demonstrated and popularized backpropagation for multilayer neural networks.

Backpropagation is reverse-mode automatic differentiation applied to a learning problem.

The milestones are deliberately selective. Linnainmaa described reverse accumulation without framing it as neural-network training. Werbos connected backward differentiation to learning systems. Rumelhart, Hinton and Williams demonstrated that the procedure could learn useful internal representations and made it widely known in neural networks. Do not say that Hinton invented backpropagation.

Sources: Baydin et al., “Automatic Differentiation in Machine Learning: a Survey,” JMLR 18 (2018), section 3.3, https://www.jmlr.org/papers/volume18/17-468/17-468.pdf; Werbos, Beyond Regression (1974), https://gwern.net/doc/ai/nn/1974-werbos.pdf; Rumelhart, Hinton & Williams, Nature 323 (1986), https://doi.org/10.1038/323533a0. Illustration supplied by the lecturer and cropped to the image column.

The same algorithm at larger scale

1989 — handwritten ZIP codes
LeCun and colleagues trained a convolutional network with backpropagation.

1990s — document recognition
Neural systems moved into deployed cheque and document processing.

2012 — AlexNet
GPU training, about 60 million parameters, and 1.2 million ImageNet training images.

Data, compute, architecture and scale changed dramatically. The chain rule did not.

The 1989 reference is LeCun et al., “Backpropagation Applied to Handwritten Zip Code Recognition,” Neural Computation 1(4), 541–551. The later overview describes industrial document-recognition systems: LeCun, Bottou, Bengio & Haffner, “Gradient-Based Learning Applied to Document Recognition,” Proc. IEEE 86(11), 1998, https://doi.org/10.1109/5.726791. AlexNet reports 60 million parameters and training on 1.2 million high-resolution ImageNet images: Krizhevsky, Sutskever & Hinton, NeurIPS 2012, https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf. Illustration supplied by the lecturer and cropped to the image column.

The surprising fact: gradients are cheap

For a scalar objective

$$f:\mathbb R^n\to\mathbb R,$$

one reverse sweep produces the complete gradient:

$$\operatorname{cost}(\nabla f)=O\!\left(\operatorname{cost}(f)\right).$$

In practice, the backward pass costs a small constant multiple of the forward computation. It also needs stored or recomputed intermediate values.

$10^6$ or $10^9$ parameters do not require $10^6$ or $10^9$ separate derivative calculations.

Pause on the asymptotic statement. The important scaling is with the number of inputs to a scalar output. Reverse mode has a memory cost, and the constant factor depends on the computation and implementation. Baur and Strassen proved an arithmetic-circuit version of the cheap-gradient principle: Walter Baur & Volker Strassen, “The Complexity of Partial Derivatives,” Theoretical Computer Science 22 (1983), 317–330, https://doi.org/10.1016/0304-3975(83)90110-X. Illustration supplied by the lecturer and cropped to the image column.

EndBackup follows

Original Lecture-1A, slide 20. Visual source: Keynote PDF export.

EndBackup follows

Julia AD: Enzyme

Original Lecture-1A, slide 21. Visual source: Keynote PDF export.

Julia AD: Enzyme •LLVM plug-in; differentiates after compiler optimizations •Handles mutation & C/Fortran kernels; cross-language •Benchmarks: gradient ≈ 4.5× faster vs unoptimized source-level AD

Julia AD: Zygote

Original Lecture-1A, slide 22. Visual source: Keynote PDF export.

Julia AD: Zygote •Source-to-source transformation → gradient code compiled by Julia •No runtime tape; functional style, uses ChainRules definitions •Call: grads = Zygote.gradient(f, params)