DeepLearning/NeuralNetworks/neural_networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Wait,-Wait,-Wait...-Why-a-Neural-Network?\" data-toc-modified-id=\"Wait,-Wait,-Wait...-Why-a-Neural-Network?-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Wait, Wait, Wait... Why a Neural Network?</a></span></li><li><span><a href=\"#Starting-with-a-Perceptron\" data-toc-modified-id=\"Starting-with-a-Perceptron-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>Starting with a Perceptron</a></span><ul class=\"toc-item\"><li><span><a href=\"#A-Diagram\" data-toc-modified-id=\"A-Diagram-2.1\"><span class=\"toc-item-num\">2.1&nbsp;&nbsp;</span>A Diagram</a></span></li><li><span><a href=\"#A-Scenario---Logic\" data-toc-modified-id=\"A-Scenario---Logic-2.2\"><span class=\"toc-item-num\">2.2&nbsp;&nbsp;</span>A Scenario - Logic</a></span><ul class=\"toc-item\"><li><span><a href=\"#Logical-AND\" data-toc-modified-id=\"Logical-AND-2.2.1\"><span class=\"toc-item-num\">2.2.1&nbsp;&nbsp;</span>Logical <code>AND</code></a></span><ul class=\"toc-item\"><li><span><a href=\"#Solution\" data-toc-modified-id=\"Solution-2.2.1.1\"><span class=\"toc-item-num\">2.2.1.1&nbsp;&nbsp;</span>Solution</a></span></li></ul></li><li><span><a href=\"#Logical-OR\" data-toc-modified-id=\"Logical-OR-2.2.2\"><span class=\"toc-item-num\">2.2.2&nbsp;&nbsp;</span>Logical <code>OR</code></a></span><ul class=\"toc-item\"><li><span><a href=\"#Solution\" data-toc-modified-id=\"Solution-2.2.2.1\"><span class=\"toc-item-num\">2.2.2.1&nbsp;&nbsp;</span>Solution</a></span></li></ul></li></ul></li></ul></li><li><span><a href=\"#Neural-Networks-Overview\" data-toc-modified-id=\"Neural-Networks-Overview-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>Neural Networks Overview</a></span><ul class=\"toc-item\"><li><span><a href=\"#Couple-ways-to-think-of-neural-networks\" data-toc-modified-id=\"Couple-ways-to-think-of-neural-networks-3.1\"><span class=\"toc-item-num\">3.1&nbsp;&nbsp;</span>Couple ways to think of neural networks</a></span></li><li><span><a href=\"#Parts-of-a-Neural-Network\" data-toc-modified-id=\"Parts-of-a-Neural-Network-3.2\"><span class=\"toc-item-num\">3.2&nbsp;&nbsp;</span>Parts of a Neural Network</a></span><ul class=\"toc-item\"><li><span><a href=\"#Layers\" data-toc-modified-id=\"Layers-3.2.1\"><span class=\"toc-item-num\">3.2.1&nbsp;&nbsp;</span>Layers</a></span></li><li><span><a href=\"#Weights\" data-toc-modified-id=\"Weights-3.2.2\"><span class=\"toc-item-num\">3.2.2&nbsp;&nbsp;</span>Weights</a></span></li><li><span><a href=\"#Activation-Functions\" data-toc-modified-id=\"Activation-Functions-3.2.3\"><span class=\"toc-item-num\">3.2.3&nbsp;&nbsp;</span>Activation Functions</a></span></li><li><span><a href=\"#Other-Hyperparameters\" data-toc-modified-id=\"Other-Hyperparameters-3.2.4\"><span class=\"toc-item-num\">3.2.4&nbsp;&nbsp;</span>Other Hyperparameters</a></span></li></ul></li><li><span><a href=\"#Training-a-Neural-Network\" data-toc-modified-id=\"Training-a-Neural-Network-3.3\"><span class=\"toc-item-num\">3.3&nbsp;&nbsp;</span>Training a Neural Network</a></span><ul class=\"toc-item\"><li><span><a href=\"#Backpropagation\" data-toc-modified-id=\"Backpropagation-3.3.1\"><span class=\"toc-item-num\">3.3.1&nbsp;&nbsp;</span>Backpropagation</a></span></li></ul></li></ul></li><li><span><a href=\"#Bring-in-more-complexity!\" data-toc-modified-id=\"Bring-in-more-complexity!-4\"><span class=\"toc-item-num\">4&nbsp;&nbsp;</span>Bring in more complexity!</a></span><ul class=\"toc-item\"><li><span><a href=\"#🧠-Knowledge-Check:-Why-not-more-complex-all-the-time?\" data-toc-modified-id=\"🧠-Knowledge-Check:-Why-not-more-complex-all-the-time?-4.1\"><span class=\"toc-item-num\">4.1&nbsp;&nbsp;</span>🧠 Knowledge Check: Why not more complex all the time?</a></span></li></ul></li><li><span><a href=\"#Let's-see-it-in-action!\" data-toc-modified-id=\"Let's-see-it-in-action!-5\"><span class=\"toc-item-num\">5&nbsp;&nbsp;</span>Let's see it in action!</a></span></li></ul></div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-05T17:49:01.027667Z",
     "start_time": "2019-12-05T17:49:00.607445Z"
    }
   },
   "outputs": [],
   "source": [
    "# Some initial setup\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "np.random.seed(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Wait, Wait, Wait... Why a Neural Network?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "You really should take a second to realize what tools we already have and ask yourself, \"Do we really need to use this 'neural network' if we already have so many machine learning algorithms?\"\n",
    "\n",
    "And in short, we don't need to default to a neural network but they have advantages in solving very complex problems. It might help to know that idea of neural networks was developed back in the 1950s (perceptron network). It wasn't until we had a lot of data and computational power where they became reasonably useful."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Starting with a Perceptron"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## A Diagram"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "<img src='https://cdn-images-1.medium.com/max/1600/0*No3vRruq7Dd4sxdn.png' width=40%/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Notice the similarity to a linear regression:\n",
    "\n",
    "\n",
    "$$ x_1 w_1 + x_2 w_2  + x_3 w_3 = \\text{output}$$\n",
    "$$ XW = \\text{output}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## A Scenario - Logic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "A perceptron basically allows us separate data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-05T17:50:33.979166Z",
     "start_time": "2019-12-05T17:50:33.974624Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def test_logic(inputs, correct_outputs, weights, bias):\n",
    "    our_outputs = []\n",
    "    w1,w2 = weights\n",
    "    # Check each output\n",
    "    for test_input, correct_output in zip(inputs, correct_outputs):\n",
    "        linear_combination = w1 * test_input[0] + w1 * test_input[1] + bias\n",
    "        output = int(linear_combination >= 0)\n",
    "        is_correct_string = 'Yes' if output == correct_output else 'No'\n",
    "        our_outputs.append([test_input[0], test_input[1], linear_combination, output, is_correct_string])\n",
    "\n",
    "    # Print results\n",
    "    num_wrong = len([output[4] for output in our_outputs if output[4] == 'No'])\n",
    "    output_frame = pd.DataFrame(\n",
    "                        our_outputs, \n",
    "                        columns=[\n",
    "                            'Input 1', \n",
    "                            '  Input 2', \n",
    "                            '  Linear Combination', \n",
    "                            '  Activation Output', \n",
    "                            '  Is Correct']\n",
    "    )\n",
    "\n",
    "    print(output_frame.to_string(index=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Logical `AND`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "$A \\land B = C$\n",
    "\n",
    "Truth table:\n",
    "\n",
    "| A | B | C |\n",
    "|---|---|---|\n",
    "| 1 | 1 | 1 |\n",
    "| 1 | 0 | 0 |\n",
    "| 0 | 1 | 0 |\n",
    "| 0 | 0 | 0 |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "<!--TODO: Real image and plot-->\n",
    "<pre>\n",
    "(x1) ⇒ (w1) ⇒\n",
    "        \n",
    "(x2) ⇒ (w2) ⇒    ⇒  (linear combination)  ⇒ (>= 0 ?) ⇒ (Output)\n",
    "\n",
    "(b)  ⇒ (1)  ⇒  \n",
    "</pre>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-05T17:50:35.027839Z",
     "start_time": "2019-12-05T17:50:35.025409Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Inputs and outputs\n",
    "test_inputs = [(0, 0), (0, 1), (1, 0), (1, 1)]\n",
    "correct_outputs = [False, False, False, True]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-05T17:50:35.216370Z",
     "start_time": "2019-12-05T17:50:35.214204Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Set your values\n",
    "w1 = 0.0\n",
    "w2 = 0.0\n",
    "bias = 0.0\n",
    "\n",
    "weights = (w1,w2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-05T17:50:41.121966Z",
     "start_time": "2019-12-05T17:50:41.114450Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Test it out!\n",
    "test_logic(test_inputs, correct_outputs, weights, bias)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "#### Solution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-05T17:51:03.926522Z",
     "start_time": "2019-12-05T17:51:03.924275Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "w1 = 0.5\n",
    "w2 = 0.5\n",
    "bias = -1.0\n",
    "\n",
    "weights = (w1,w2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-05T17:51:04.305666Z",
     "start_time": "2019-12-05T17:51:04.300196Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "test_logic(test_inputs, correct_outputs, weights, bias)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Logical `OR`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "$A \\lor B = C$\n",
    "\n",
    "Truth table:\n",
    "\n",
    "| A | B | C |\n",
    "|---|---|---|\n",
    "| 1 | 1 | 1 |\n",
    "| 1 | 0 | 1 |\n",
    "| 0 | 1 | 1 |\n",
    "| 0 | 0 | 0 |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "<!--TODO: Real image and plot-->\n",
    "<pre>\n",
    "(x1) ⇒ (w1) ⇒\n",
    "        \n",
    "(x2) ⇒ (w2) ⇒    ⇒  (linear combination)  ⇒ (>= 0 ?) ⇒ (Output)\n",
    "\n",
    "(b)  ⇒ (1)  ⇒  \n",
    "</pre>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-05T17:52:16.919596Z",
     "start_time": "2019-12-05T17:52:16.917233Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Inputs and outputs\n",
    "test_inputs = [(0, 0), (0, 1), (1, 0), (1, 1)]\n",
    "correct_outputs = [False, True, True, True]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-05T17:52:17.816940Z",
     "start_time": "2019-12-05T17:52:17.814447Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Set your values\n",
    "w1 = 0.0\n",
    "w2 = 0.0\n",
    "bias = 0.0\n",
    "\n",
    "weights = (w1,w2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-05T17:52:18.611934Z",
     "start_time": "2019-12-05T17:52:18.606151Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Test it out!\n",
    "test_logic(test_inputs, correct_outputs, weights, bias)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "#### Solution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-05T17:53:16.664802Z",
     "start_time": "2019-12-05T17:53:16.662384Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "w1 = 1\n",
    "w2 = 1\n",
    "bias = -1.0\n",
    "\n",
    "weights = (w1,w2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-05T17:53:17.444972Z",
     "start_time": "2019-12-05T17:53:17.439382Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "test_logic(test_inputs, correct_outputs, weights, bias)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Neural Networks Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## Couple ways to think of neural networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "> We can think of neural networks as finding ways to take inputs and creating something like latent features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "![](images/neural_network_with_human_readable_labels.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "> But we can also think of them as creating linear separators and then combining them together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-05T17:49:01.133636Z",
     "start_time": "2019-12-05T17:49:00.624Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "x = np.random.rand(25)\n",
    "y = np.random.rand(25)\n",
    "z = (x + y) <= 0.8\n",
    "plt.scatter(x,y,c=z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Thinking in the more mathematical way, allows us to use our linear algebra knowledge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "![](images/neural_network_mathematics.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## Parts of a Neural Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "- **Input Layer**: the initial parameters (these will be the parts we feed to our network)\n",
    "- **Output Layer**: the classification (or regression predictions)\n",
    "- **Hidden Layer(s)**: the other neurons potentially in a neural network to find more complex patterns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "> The weights from our inputs are describing how much they should contribute to the next neuron\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "But we can also think of the weights of hidden layer neurons telling us how much of these linear separations should be combined."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Activation Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Let's discuss what kind activation functions we have and what we can do with them:\n",
    "\n",
    "> [Activation Functions Notebook](activation_functions.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Other Hyperparameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "We'll talk more about this in [optimizing our neural networks](optimizations.ipynb) but some hyperparameters include:\n",
    "\n",
    "- **Learning Rate ($\\alpha$)**: how big of a step we take in gradient descent\n",
    "- **Number of epochs**: how many times we repeat this process\n",
    "- **batch-size**: how many data points we use in a single training session (1 epoch)\n",
    "\n",
    "Remember, any parameter adjusted to enhance the neural network's learning _is_ a hyperparameter (this includes the actual structure of the neural net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## Training a Neural Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Imagine that our neural network doesn't do great after creating. What would you do to improve it?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Backpropagation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The **backpropagation** algorithm takes the idea of optimally adjusting the parameters (weights) to get a better result. \n",
    "\n",
    "We do this tuning by propogating the (average) error back through the network, with the cost function $J$ guiding us and adjusting via gradient descent.\n",
    "\n",
    "> Turn down previous neurons that give a bad result\n",
    ">\n",
    "> Turn up previous neurons that give a good result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "> Great video explanation of backpropogation by 3Blue1Brown (part of a full playlist): [Backpropagation calculus | Deep learning, chapter 4](https://www.youtube.com/watch?v=tIeHLnjs5U8&list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi&index=4)\n",
    "\n",
    "![](images/neural_network_graph_3blue1brown.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Bring in more complexity!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "But can what if the data is more complicated? Can we separate these?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-05T17:49:01.134636Z",
     "start_time": "2019-12-05T17:49:00.628Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "x = np.random.rand(40)\n",
    "y = np.random.rand(40)\n",
    "z = (x + y) < np.random.rand(40)*2\n",
    "\n",
    "plt.scatter(x,y,c=z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "By adding in more parts (layers) leads us to **deep learning**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "<img src='images/layered-neural-net.jpg' width='90%'/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "In fact, neural networks can (in theory) approximate any continuous function! (https://en.wikipedia.org/wiki/Universal_approximation_theorem)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## 🧠 Knowledge Check: Why not more complex all the time?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "> More complexity can increase our chances of overfitting\n",
    ">\n",
    "> More parameters mean more computation (takes longer to train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "We'll talk about ways to tune our neural network and still attempt to avoid overfititng:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "- [Avoiding Overfitting](avoiding_overfitting.ipynb)\n",
    "- [Optimizations to Neural Networks](optimizations.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Let's see it in action!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Now we know the different parts, let's try it out for ourselves!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "- [playground.tensorflow.org](https://playground.tensorflow.org): A visual playground for us to train a neural network\n",
    "- [Keras Implementation Notebook](keras_implementation.ipynb): Some code we can try out for ourselves"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}