From Linear Regression to Deep Learning: Your First Neural Network

ML
Deep Learning
What a weight really is, why a single neuron is just linear regression, and how hidden layers and depth let a network learn curves. Interactive, step by step.
Author

MLStation

Published

October 1, 2026

Deep learning sounds mysterious, but its building block is something you already know: multiply inputs by weights, add them up. That is linear regression. A neural network simply stacks many of these small calculations in layers, with a bend (an activation) in between. Stack enough layers and you have deep learning.

TipWhat you will learn
  • What a neuron and a weight are, with a live diagram
  • Why one neuron with no bend is linear regression, weight by weight
  • How a hidden layer lets the model learn a curve linear regression cannot
  • How an ANN and a deeper DNN solve a five-feature problem
  • How to save the trained model and load it again

The building block: one neuron

A neuron takes inputs x_1, x_2, \dots, multiplies each by its own weight, adds a bias, then passes the total through an activation function f:

z = w_1 x_1 + w_2 x_2 + w_3 x_3 + b, \qquad a = f(z) \tag{1}

ImportantWhat is a weight, exactly?

A weight is a single number attached to one connection. It answers one question: if this input goes up by 1, how much does it push the neuron’s total z, and in which direction?

  • Large weight: the input matters a lot. Near zero: the input is almost ignored.
  • Positive weight: the input pushes the total up. Negative: it pushes it down.
  • The bias b is the starting value before any input is added.

Think of each weight as a volume knob on one microphone: it sets how loud, and with what sign, that input is heard. Training means turning all the knobs until the predictions match the data.

Play with the knobs. Line thickness shows the size of each weight, colour shows its sign.

ReLU (rectified linear unit) is the most common activation: it outputs \max(0, z), so it is zero until the total crosses zero, then rises in a straight line. That one bend is what gives neural networks their power.

Weights vs linear regression

Linear regression with one feature is \hat{y} = \beta_0 + \beta_1 x. Now look at a neuron with one input and no activation: \hat{y} = b + w\,x. Same formula, different names.

Table 1: Weights in linear regression vs a neural network
Linear regression Neural network
Multiplier on an input Coefficient \beta_1 (slope) Weight w
Constant term Intercept \beta_0 Bias b
How it is found One exact formula (least squares) Gradual adjustment (gradient descent)
Meaning of the multiplier Fixed effect of x everywhere Effect of x on one neuron; the total effect depends on all neurons
Can it bend? No, always a straight line Yes, once a hidden layer with activation is added
Number of parameters Features + 1 Can be thousands or billions

Experiment 1: one feature, one target

A building’s electricity demand depends on the outside temperature. Below about 22 °C demand is flat. Above it, the air conditioning turns on and demand climbs. We simulate 120 days.

Simulate 120 days of temperature and demand 
import numpy as np, pandas as pd, warnings
warnings.filterwarnings("ignore")
from sklearn.linear_model import LinearRegression
from sklearn.neural_network import MLPRegressor
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.compose import TransformedTargetRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score, mean_absolute_error
import plotly.graph_objects as go
from ms_charts import COLORS, PLOTLY_CONFIG

rng = np.random.default_rng(7)
n = 120
temp = rng.uniform(5, 40, n)                                   # °C
cool = np.maximum(temp - 22, 0)                                # air conditioning kicks in at 22 °C
demand = 50 + 2.6 * cool + 0.04 * cool**2 + rng.normal(0, 3, n)  # kW
x = temp.reshape(-1, 1)
grid = np.linspace(5, 40, 200)

Model A: linear regression vs a single neuron

We fit ordinary linear regression, then train one neuron with no activation using gradient descent: start with w = 0, b = 0, measure the error, nudge both numbers downhill, and repeat.

Train both models 
lr = LinearRegression().fit(x, demand)

# a single neuron trained by gradient descent (on scaled x for stability, then converted back to °C)
mu, sd = temp.mean(), temp.std()
z = (temp - mu) / sd
w, b, eta, path = 0.0, 0.0, 0.1, []
for epoch in range(80):
    pred = w * z + b
    err = pred - demand
    path.append(dict(epoch=epoch, w=w / sd, b=b - w * mu / sd, mse=float((err**2).mean())))
    w -= eta * 2 * (err * z).mean()          # gradient of mean squared error w.r.t. w
    b -= eta * 2 * err.mean()                # gradient w.r.t. b
w_neuron, b_neuron = w / sd, b - w * mu / sd

Drag the slider to watch the neuron learn. Its line starts flat at zero and settles exactly on the linear regression line.

Parameter Linear regression Single neuron Same?
Slope / weight 1.644 1.644 Yes
Intercept / bias 26.79 26.79 Yes
R² 0.758 0.758 Yes

Similarity: with no hidden layer and no activation, the neuron’s weight is the regression slope and its bias is the intercept. Both say “each extra degree adds about 1.64 kW, everywhere”. Difference: regression solves it in one exact step, the neuron gets there by trial and correction. Both are wrong in the same way: the data bends, a straight line cannot.

Model B: add a hidden layer with two ReLU neurons

Now we put two hidden neurons with ReLU between the input and the output:

\hat{y} = v_1\,\text{ReLU}(w_1 x + b_1) + v_2\,\text{ReLU}(w_2 x + b_2) + c \tag{2}

Train the 1-2-1 network 
ann1 = MLPRegressor(hidden_layer_sizes=(2,), activation="relu", solver="lbfgs",
                    max_iter=5000, random_state=5).fit(z.reshape(-1, 1), demand)
W1 = ann1.coefs_[0][0] / sd                                   # input-to-hidden weights in °C units
B1 = ann1.intercepts_[0] - ann1.coefs_[0][0] * mu / sd        # hidden biases
V, C = ann1.coefs_[1].ravel(), float(ann1.intercepts_[1][0])  # hidden-to-output weights and bias
hinge = -B1 / W1                                              # temperature where each neuron switches on/off
parts = [V[j] * np.maximum(W1[j] * grid + B1[j], 0) for j in range(2)]
r2_ann1 = r2_score(demand, sum(V[j] * np.maximum(W1[j] * temp + B1[j], 0) for j in range(2)) + C)

Each hidden neuron draws one hinge: flat on one side, a straight slope on the other. The output adds the hinges together. Toggle them to see how the curve is built.

Now compare the parameters weight by weight:

Parameter What it does Value Linear regression equivalent
w_1, b_1 Hidden neuron 1: switches at 22.3 °C -0.77, 17.2 None
w_2, b_2 Hidden neuron 2: switches at 4.7 °C 0.42, -2.0 None
v_1, v_2 How strongly each hinge counts in the output 4.45, 7.99 None
c Output bias -10.3 Like the intercept
Total 7 parameters 2 parameters
R² 0.978 0.758
NoteThe key difference

In linear regression, the slope is one number that applies at every temperature. In the network, no single weight is “the slope”. The slope below 22 °C is v_1 w_1 + v_2 w_2 \approx -0.10 kW per °C (flat), and above it is 3.35 kW per °C. The network discovered the air-conditioning threshold near 22 °C without being told. That is the trade: more flexible, less directly readable.

Experiment 2: five features, one target

Real buildings depend on more than temperature. We simulate 3,000 hourly readings with five features:

Feature Meaning
temp Outside temperature (°C)
humidity Relative humidity (%)
hour Hour of day (0 to 23)
weekend 1 on Saturday and Sunday
occupancy People in the building
Simulate 3,000 hourly readings 
N = 3000
F = pd.DataFrame({
    "temp": rng.uniform(-5, 40, N),
    "humidity": rng.uniform(20, 90, N),
    "hour": rng.integers(0, 24, N),
    "weekend": (rng.random(N) < 2 / 7).astype(int),
})
F["occupancy"] = rng.uniform(0, 200, N) * (1 - 0.7 * F.weekend)

def true_demand(t, h, hr, wk, occ):
    open_ = ((hr >= 8) & (hr < 19)).astype(float) * (1 - wk)        # office open on weekdays
    return (60 + 0.10 * (t - 18)**2                                 # heating and cooling (U shape)
            + 0.04 * h * np.maximum(t - 24, 0)                      # humid heat is extra costly
            + 35 * open_ + 0.25 * occ * open_)                      # lights, computers, people

y = true_demand(F.temp, F.humidity, F.hour, F.weekend, F.occupancy) + rng.normal(0, 6, N)
Xtr, Xte, ytr, yte = train_test_split(F, y, test_size=0.25, random_state=0)

Linear regression first, then a network with one hidden layer of 8 neurons. We scale the inputs and the target, which helps the network train reliably.

def network(layers, seed=0):
    mlp = MLPRegressor(hidden_layer_sizes=layers, activation="relu",
                       solver="lbfgs", alpha=1e-3, max_iter=3000,
                       random_state=seed)
    net = make_pipeline(StandardScaler(), mlp)            # scale inputs
    return TransformedTargetRegressor(regressor=net,      # scale target
                                      transformer=StandardScaler())

linreg = make_pipeline(StandardScaler(), LinearRegression()).fit(Xtr, ytr)
ann = network((8,)).fit(Xtr, ytr)

This is the map of the network we just trained: 5 inputs, 8 hidden neurons, 1 output. Hover any line to read its weight.

Blue lines are positive weights, red are negative, thicker means stronger. With 5 × 8 + 8 × 1 = 48 weights plus 9 biases, the ANN has 57 parameters. Linear regression has 6.

Going deep: the DNN

A deep neural network (DNN) is simply a network with two or more hidden layers. The first layer learns simple patterns (for example “it is hot” or “the office is open”), the next layer combines them (“hot and open”). That stacking of simple ideas into richer ones is what the “deep” in deep learning means.

# two hidden layers: 8 neurons, then 6 neurons
dnn = network((8, 6)).fit(Xtr, ytr)

How the three models compare

Model Parameters Test R² Mean error
Linear regression 6 0.181 27.2 kW
ANN (1 hidden layer: 8) 57 0.871 9.6 kW
DNN (2 hidden layers: 8, 6) 109 0.969 5.2 kW

Linear regression cannot capture the U-shaped temperature effect or the office switching on and off, so it explains very little. The ANN captures most of it, and the extra layer in the DNN lifts accuracy further. On a small problem like this, a wider single layer can also get close. Depth pays off most on complex data such as images, audio and text.

Try it yourself

Change the conditions and watch all three models respond. The “true” value comes from the simulation formula, without noise.

Set the hour to 2 (night) or switch on Weekend: the networks drop sharply, the linear model barely moves.

Save the model and load it later

Training takes time, so you save the trained model once and reuse it. With scikit-learn, joblib stores the whole pipeline: the scalers, every weight and every bias.

import joblib, os
path = "models/building_demand_dnn.joblib"
os.makedirs("models", exist_ok=True)

joblib.dump(dnn, path)                 # save
loaded = joblib.load(path)             # load, any time later

new_hour = pd.DataFrame([{"temp": 33, "humidity": 70, "hour": 14,
                          "weekend": 0, "occupancy": 120}])
print(f"Saved file size: {os.path.getsize(path) / 1024:.0f} KB")
print(f"Predicted demand: {loaded.predict(new_hour)[0]:.1f} kW")
same = np.allclose(loaded.predict(Xte), dnn.predict(Xte))
print("Same predictions as the original:", same)
Saved file size: 10 KB
Predicted demand: 174.2 kW
Same predictions as the original: True

The same idea in the two most popular deep learning libraries:

keras_save.py
model.save("building_demand_dnn.keras")
model = keras.models.load_model("building_demand_dnn.keras")
torch_save.py
torch.save(model.state_dict(), "building_demand_dnn.pt")    # weights only
model = DemandNet()                                          # same architecture
model.load_state_dict(torch.load("building_demand_dnn.pt"))
model.eval()
WarningOnly load files you trust

joblib, pickle and older PyTorch files can run code when loaded. Never load a model file from an unknown source.

The journey in one picture

flowchart LR
  A["Linear regression<br/>y = b + w·x"] --> B["One neuron<br/>same thing"] --> C["Hidden layer + ReLU<br/>ANN: learns bends"] --> D["2+ hidden layers<br/>DNN: combines patterns"] --> E["Save with joblib<br/>reuse anywhere"]

Summary

  • A weight is the number that says how much, and in which direction, one input pushes one neuron.
  • A neuron with no activation and no hidden layer is linear regression: weight = slope, bias = intercept.
  • A hidden layer with ReLU adds bends. No single weight is the slope any more; the effect of an input depends on which neurons are switched on.
  • A DNN stacks two or more hidden layers to combine simple patterns into richer ones.
  • Save the trained model so you can reuse it without retraining.

For the full theory, see Goodfellow et al. (2016).

Back to top

References

Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. https://www.deeplearningbook.org.