Introduction

P-value is one of the most important topics in Statistics and Machine Learning. If you are new in Data Science like me, then you also probably face problems in understanding p-value. And you cannot avoid p-value in your Data Science journey. It is everywhere! From Hypothesis testing to Machine Learning models, you will find the concept of p-value. Initially, I did not understand the interpretation of p-value. Later, after seeing some youtube videos, and reading articles and blogs, I got some understanding on this topic.

In this article, I will share my thoughts and understanding about p-value. I have gained these information and interpretations from various sources. I hope after reading this article, you will get a good idea about p-value, its relationship with significance value, and its importance in Machine Learning.

I divide this blog basically into four parts. In the first 2-3 paragraphs, I give some bookish definitions of the p-value. After that, I try to interpret the p-value using an example. Later you will find, the relationship between p-value and significance value and how p-value is important in ML models.

What is p-value?

A p-value is a number that tells you how likely it is that your results are due to chance. In other words, it measures the strength of evidence against the null hypothesis, which assumes that there is no real difference between two groups or no real effect of a treatment.

The p-value ranges from 0 to 1, with a smaller p-value indicating stronger evidence against the null hypothesis. Typically, a p-value of 0.05 or less is considered statistically significant, meaning there is less than a 5% chance that the results are due to chance.

It's important to note that the p-value does not tell you the size or practical significance of the effect. It only tells you whether the effect is likely to be real or not. Additionally, the p-value should be interpreted in conjunction with other statistical measures, such as effect size and confidence intervals, to fully understand the implications of the results.

Interpreting p-value in a simple way

Let's try to understand the p-value with an example!

Imagine you're playing a game with a friend, and you're trying to see who's better at throwing a ball into a basket. You each get three tries, and whoever gets more balls in the basket wins.

Now, let's say you win with a score of 2-1. But, your friend thinks maybe you just got lucky and it wasn't because you were actually better at throwing the ball. So, you decide to do a test to see if your win was really just luck or if you really are better at throwing the ball.

Here comes the p-value. It's like a score that tells you how likely it is that your win was just luck or if you really are better at throwing the ball. If the p-value is really low, that means it's very unlikely that your win was just luck, and you really are better at throwing the ball. But if the p-value is high, that means it's more likely that your win was just luck, and you and your friend are probably equally good at throwing the ball.

So, the p-value helps us figure out if something we did really made a difference, or if it was just luck. In statistics, it's used to help us make sure our experiments and tests are fair and accurate.

Relationship between p-value and significance value

P-value and significance value are two ways to measure the strength of evidence against the null hypothesis in statistical hypothesis testing. They are related, but they are not the same thing.

The p-value is a probability value that represents the likelihood of obtaining the observed test results, or more extreme results if the null hypothesis is true. In other words, it measures how much evidence there is against the null hypothesis. A small p-value (typically less than 0.05) indicates strong evidence against the null hypothesis and suggests that we should reject the null hypothesis and accept the alternative hypothesis.

On the other hand, the significance level (or alpha level) is a predetermined threshold value that is used to determine whether the p-value is small enough to reject the null hypothesis. It represents the maximum probability of making a Type I error (i.e., rejecting the null hypothesis when it is actually true) that we are willing to accept. The most commonly used significance level is 0.05, which means that we are willing to accept a 5% chance of making a Type I error.

So, in summary, the p-value is a measure of the strength of evidence against the null hypothesis, while the significance level is a predetermined threshold value that is used to determine whether the evidence is strong enough to reject the null hypothesis. The two are related in that the p-value must be less than or equal to the significance level to reject the null hypothesis.

How p-value is related to Machine Learning?

In Machine Learning, the p-value is often used to determine the significance of a model's performance or the importance of a feature.

For example, in hypothesis testing for classification models, the p-value can be used to determine whether the difference in accuracy between two models or two sets of parameters is statistically significant. If the p-value is small (e.g., less than 0.05), we can conclude that the difference in performance is unlikely to be due to chance and that one model or set of parameters is significantly better than the other.

In feature selection, the p-value can be used to assess the importance of each feature in predicting the outcome variable. We can calculate the p-value for each feature's coefficient in a regression model, and if the p-value is small, we can conclude that the feature is likely to be an important predictor. Conversely, if the p-value is large, we can conclude that the feature is likely not important and can be removed from the model to improve its performance.

Conclusion

This is all about p-value. Thank you for reading this article. If you find out any mistake, please let me know in the comments section.

My LinkedIn Profile: https://www.linkedin.com/in/rounak-show-211131174/