Linear Regression from Scratch(Including Python Code)
LINEAR REGRESSION
In this article we are going to discuss about the linear regression algorithm in detail along with its practical implementation in laymen terms.
• Linear Regression is a supervised machine learning algorithm where the predicted output is continuous and has a constant slope.
• It’s used to predict values within a continuous range, (e.g. sales, price) rather than trying to classify them into categories (e.g. cat, dog).
• There are 2 types of linear regression model:
• Simple Linear Regression
• Multiple Linear Regression
SIMPLE LINEAR REGRESSION
• Simple linear regression uses traditional slope-intercept form, where a and b are the variables our algorithm will try to “learn” to produce the most accurate predictions. There we have only one input column.
• Formula: 𝑦=a𝑥+b
‘𝑥’ represents our input data , ‘𝑦’ represents our prediction , ‘a’ represents the coefficient( weightage of input variable on the output variable) , ‘b’ represents constant/bias.
Value of ‘a’ and ‘b’ can be find using two approaches :
Closed form solution (Direct Formula)
Non – closed solution (Gradient Descent)
Closed form solution :
Formula for a and b are
a = Σ ((Xi - X ̅)(Yi - Y ̅)) / Σ (Xi - X ̅)^2
b = Y ̅ - mX ̅
Where Y ̅,X ̅ represents the mean values of output and input variables.
Where Xi , Yi represents the data points.
Sklearn library uses this formula only.
We will discuss the gradient descent approach in later part of this article.
BEST FIT LINE
- Dark black line in this figure represents best fit line ( line which is making the minimum error)
- In order to find the best fit line, we have to find the minimum value of cost function/loss function.
- The line for which the the error between the predicted values and the observed values is minimum is called the best fit line or the regression line.
- These errors are also called as residuals.
- The residuals can be visualized by the vertical lines from the observed data value to the regression line.
MULTIPLE LINEAR REGRESSION
• A more complex, multi-variable linear equation might look like this, where 𝑤w represents the coefficients, or weights, our model will try to learn.
• F(𝑥,𝑦,𝑧) = (𝑤1 *𝑥)+(𝑤2 * 𝑦)+(𝑤3 * 𝑧)
• The variables 𝑥, 𝑦, 𝑧 represent the attributes, or distinct pieces of information, we have about each observation.
• For sales predictions, these attributes might include a company’s advertising spend on radio, TV, and newspapers.
• 𝑆a𝑙𝑒𝑠 = (𝑤1 *𝑅a𝑑𝑖𝑜)+(𝑤2 *𝑇𝑉)+(𝑤3 *𝑁𝑒𝑤𝑠)
COST FUNCTION: gives you the estimation of the error that your model is making.
We always try to keep the value of the cost function as small as possible.
NOTE: Cost function and Loss Function are synonymous and used interchangeably but they are different. A loss function is for a single training example/input. A cost function is average loss over the entire training dataset.
In the formula,
Theta --> Input parameter
m --> Total number of data points
h (xi) --> predicted value
yi --> actual value
WE TAKE THE SQUARE OF THE RESIDUALS AND NOT THE ABSOLUTE VALUE OF THE RESIDUALS BECAUSE :
• We want to penalize the points which are farther from the regression line much more than the points which lie close to the line.( Penalizing means giving more weights to those points which are farther from the best fit line . )
• Our objective is to find the model parameters so that the cost function is minimum.
GRADIENT DESCENT
- Gradient descent is a generic optimization algorithm used in many machine learning algorithms. (In a nutshell, it is method of finding the minima.)
- It iteratively tweaks the parameters of the model in order to minimize the cost function.
STEPS IN GRADIENT DESCENT
- We first initialize the model parameters with some random values. This is also called as random initialization.
- Now we need to measure how the cost function changes with change in its parameters. Therefore we compute the partial derivatives of the cost function w.r.t to the parameters θ₀, θ₁… θ.
PARTIAL DERIVATIVE OF COST FUNCTION W.R.T. ANY PARAMETER
PARTIAL DERIVATIVES OF ALL PARAMETERS
UPDATING THE PARAMETERS
UPDATING ALL PARAMETERS
LEARNING RATE (α)
α is the learning rate.
• If the value of α is too small, the cost function takes larger time to converge. When the learning rate is very slow, the gradient descent takes larger time to find the best fit line.
• If α is too large, gradient descent may overshoot the minimum and may finally fail to converge.
NOTE: learning rate=0.01 is usually taken .
In order to reach the global minima , we have to perform this step n number of times.
WHEN TO STOP THIS PROCESS:
1. difference between (theta)^old and (theta)^new =< 0.0001
2. By predefining iteration (epochs) before i.e. 1000 or 100
BEST PARAMETERS TO EVALUATE PERFORMANCE OF LINEAR REGRESSION MODEL
• We will be using Root mean squared error (RMSE) and Coefficient of Determination (R² score) to evaluate our model.
• RMSE is the square root of the average of the sum of the squares of residuals.
• R² score or the coefficient of determination explains how much the total variance of the dependent variable can be reduced by using the least square regression.
• SSₜ is the total sum of errors if we take the mean of the observed values as the predicted value.
• SSᵣ is the sum of the square of residuals.
RMSE FORMULA
R² SCORE FORMULA
Where SUM OF SQUARES Of Total
y ̅ = mean of target variable
SUM OF RESIDUALS
ASSUMPTIONS
- There should be linear relationship between input and output variable.
- No multi collinearity between input variables.
- Normality: For any fixed value of X, Y is normally distributed.
- Homoscedasticity: The variance of residual is the same for any value of X.
ADVANTAGES
- Simple Implementation
- Highly interpretable
- Scientifically acceptable
- Widespread availability
- Performance on linearly separable datasets
- Over fitting can be reduced by regularization
DISADVANTAGES
- Prone to underfitting
- Sensitive to outliers
- Linear regression also looks at a relationship between the mean of the dependent variables and the independent variables. Just as the mean is not a complete description of a single variable, linear regression is not a complete description of relationships among variables.
WHERE CAN LINEAR REGRESSION BE USED ?
• It is a very powerful technique and can be used to understand the factors that influence Profitability.
• It can be used to forecast sales in the coming months by analysing the sales data for previous months.
• It can also be used to gain various insights about customer behavior.
Practical implementation of Simple Linear Regression
- Ankit Yadav
- Dec, 27 2022