Deep Learning · MT2 · index

Formula reference

  1. Mean, variance and standard deviation
  2. Standardization
  3. Power, constant and sum derivatives
  4. Chain rule
  5. Dot product and a neuron
  6. Matrix multiplication
  7. Layer forward pass
  8. Sigmoid and its derivative
  9. Tanh and its derivative
  10. ReLU and its derivative
  11. Leaky ReLU
  12. Stable softmax
  13. Squared and absolute losses
  14. Binary cross entropy
  15. Categorical cross entropy
  16. Output delta for half squared loss
  17. Sigmoid plus BCE; softmax plus CCE
  18. Hidden delta, weight gradient and bias gradient
  19. Gradient descent
  20. Batch and mini-batch gradient
  21. Momentum: signed velocity
  22. Nesterov: look-ahead gradient
  23. AdaGrad
  24. RMSProp
  25. Adam: both memories
  26. Adam: correction and update
  27. L1 penalty and update
  28. L2 penalty and update with a half
  29. Inverted dropout
  30. Convolution output dimensions
  31. Convolution value
  32. Max, min and average pooling
  33. Convolution parameter count
  34. Flatten and dense parameter count
  35. Convolution multiply-accumulate count
  36. Concatenation and residual addition
  37. Batch normalization and learned scale/shift
  38. Global average pooling
  39. Top-k error percentage