Build up a Neural Network with Python
Use NumPy to realize forward propagation, backward propagation
The purpose of this blog is to use package NumPy in python to build up a neural network. Although well-established packages like Keras and Tensorflow make it easy to build up a model, yet it is worthy to code forward propagation, backward propagation and gradient descent by yourself, which helps you better understand this algorithm.
Overview
Figure above shows how information flows, when a neural network model is trained. After input Xn is entered, a linear combination of weights W1 and bias B1 is applied to Xn. Next, an activation function is applied to have a non-linear transformation to get A1. Then A1 is entered as input for next hidden layer. Same logic is applied to generate A2 and A3. The procedure to generate A1, A2 and A3 is called forward propagation. A3, also regarded as output of the neural network, is compared with independent variable y to calculate cost. Then derivative of cost function is calculated to get dA3. Take a partial derivative of dA3 for W3 and B3 to get dW3 and dB3. Same logic is applied to get dA2, dW2, dB2, dA1, dW1 and dB1. The procedure to generate a list of derivatives is called backward propagation. Finally gradient descent is applied and parameters are updated. Then a new round iteration starts with updated parameters. The algorithm will not stop until it converges.
Create Testing Data
Create a small set of testing data to verify functions created.
How to build a simple Neural Network from scratch with Python
A quick guide to set up a Neural Network without using a framework.
Neural Networks are becoming more and more popular every day and as a core field of Machine Learning and Artificial Intelligence, they are going to play a major role in technology, science and industry over the next years. This high popularity has given rise to many frameworks that allow you to implement Neural Networks very easily without knowing the complete theory behind them. On the other side, the strict theoretical explanation of neural network mechanisms demands some knowledge on high-level mathematics.
In this post, we will do something between. Specifically, in order to get a more solid understanding of neural networks, we will go through the actual implementation of a NN from scratch without using any framework, but we will omit the proofs for the sake of simplicity. This may be a bit more difficult than using a framework but you will gain a much better understanding of the mechanism behind the algorithm. Of course, in large scale projects a framework implementation is preferred as it is easier and faster to set up.
The tools used in this tutorial are just Python with numpy library (Scientific library for Linear Algebra operations). Supposing that you have python and pip installed, you can install numpy by running this command:
A Neural Network is actually a function of many variables: It takes an input, makes computations and produces an output. We like to visualize it as neurons in different layers, with each neuron in a layer connected with all neurons in the previous and the next layer. All the computations take place inside those neurons and depend on the weights that connect the neurons with each other. So all we have to do is learn the right weights in order to get the desired output.
Their structure is generally very complex including a lot of layers and even more than a million (Dec 2020 Update: GPT-3 now uses 175B parameters! ) neurons in order to be able to handle the large datasets of our era. However, in order to understand how large deep neural networks work one should start with the simplest ones.