Deep Learning · TEE · index

Formula reference

  1. Mean, variance and standard deviation
  2. Standardization
  3. Power, constant and sum derivatives
  4. Chain rule
  5. Dot product and a neuron
  6. Matrix multiplication
  7. Layer forward pass
  8. Sigmoid and its derivative
  9. Tanh and its derivative
  10. ReLU and its derivative
  11. Leaky ReLU
  12. Stable softmax
  13. Squared and absolute losses
  14. Binary cross entropy
  15. Categorical cross entropy
  16. Output delta for half squared loss
  17. Sigmoid plus BCE; softmax plus CCE
  18. Hidden delta, weight gradient and bias gradient
  19. Gradient descent
  20. Batch and mini-batch gradient
  21. Momentum: signed velocity
  22. Nesterov: look-ahead gradient
  23. AdaGrad
  24. RMSProp
  25. Adam: both memories
  26. Adam: correction and update
  27. L1 penalty and update
  28. L2 penalty and update with a half
  29. Inverted dropout
  30. Convolution output dimensions
  31. Convolution value
  32. Max, min and average pooling
  33. Convolution parameter count
  34. Flatten and dense parameter count
  35. Convolution multiply-accumulate count
  36. Concatenation and residual addition
  37. Vanilla RNN
  38. Perceptron update
  39. BPTT hidden delta
  40. BPTT recurrent-weight gradient
  41. LSTM gates and candidate
  42. LSTM candidate and output gate
  43. LSTM cell and hidden state
  44. GRU gates, candidate and update
  45. GRU state under the retain-old convention
  46. Batch normalization and learned scale/shift
  47. Global average pooling
  48. Top-k error percentage