{"id":1888,"date":"2022-04-11T10:56:26","date_gmt":"2022-04-11T03:56:26","guid":{"rendered":"https:\/\/mintea.blog\/?p=1888"},"modified":"2022-04-11T10:56:49","modified_gmt":"2022-04-11T03:56:49","slug":"1888","status":"publish","type":"post","link":"https:\/\/mintea.blog\/?p=1888","title":{"rendered":"Overfitting – Understanding overfitting"},"content":{"rendered":"

Understanding overfitting: an inaccurate meme in Machine Learning<\/strong><\/p>\n

Preamble<\/strong>
\nThere is a lot of confusion among practitioners regarding the concept of\u00a0overfitting<\/a>. It seems like, a kind of\u00a0an<\/em>\u00a0urban legend<\/em>\u00a0or a\u00a0meme, a folklore\u00a0<\/em>is circulating\u00a0in data science or allied fields with the following statement:<\/p>\n

Applying\u00a0<\/em>cross-validation<\/em><\/a>\u00a0prevents overfitting and a good out-of-sample performance, low generalisation error in unseen data, indicates not an overfit.<\/em><\/p>\n

This statement is of course not true: cross-validation does not prevent your model to overfit and good out-of-sample performance does not guarantee not-overfitted model. What actually people refer to in one aspect of this statement is called\u00a0overtraining<\/em>. Unfortunately, this meme is not only propagated in industry but in some academic papers as well. This might be at best a confusion on\u00a0jargon<\/em>. But, it will be a good practice if we set the\u00a0jargon<\/em>\u00a0right and clear on what do we refer to when we say\u00a0overfitting,<\/em>\u00a0in communicating our results.<\/p>\n

Aim<\/strong>
\nIn this post, we will give an intuition on why\u00a0model validation<\/a>\u00a0as approximating\u00a0generalization error of a\u00a0model fit<\/a>\u00a0and detection of overfitting can not be resolved simultaneously on a single model. We will work on \u00a0a concrete example workflow in understanding\u00a0overfitting<\/em>,\u00a0overtraining<\/em>\u00a0and a typical final model building stage \u00a0after some conceptual introduction. We will avoid giving a reference to the Bayesian interpretations and regularisation and restrict the post to regression and cross-validation. While regularisation has different ramification due to its mathematical properties and prior distributions have different implications in Bayesian statistics. We assume an introductory background in machine learning, so this is not a beginners tutorial.<\/p>\n

A recent question from\u00a0Andrew Gelman<\/a>, a Bayesian guru, regarding\u00a0What is overfitting?<\/em>\u00a0was one of the reasons why this post is developed along with my frustration to see practitioners being muddy on the meaning of\u00a0overfitting<\/em>\u00a0and continuing recently published data science related technical articles circulating around and even in some academic papers claiming the above statement.<\/p>\n

What do we need to satisfy in supervised learning?\u00a0<\/strong>
\nOne of the most basic tasks in mathematics is to find a solution to a function: If we restrict ourselves to real numbers in\u00a0n<\/em>-dimensions and our domain of interest would be\u00a0R<\/strong>n<\/sup><\/em>. Now imagine set of\u00a0p<\/em>\u00a0points living in this domain\u00a0x<\/em>i<\/sub>\u00a0form a dataset, this is actually\u00a0a partial<\/em>\u00a0\u00a0solution to a function. The main purpose of modelling is to find an explanation of the\u00a0dataset, meaning that we need to determine\u00a0m<\/em>-parameters,\u00a0a\u2208R<\/strong>m<\/sup>\u00a0which are\u00a0unknown. (Note that a non-parametric model does not mean no parameters.) Mathematically speaking this manifests as a function as we said before, \u00a0f(x,a). This modelling is usually called\u00a0regression<\/em>,\u00a0interpolation<\/em>\u00a0or\u00a0supervised learning\u00a0<\/em>depending on the literature you are reading. This is a form of\u00a0an inverse problem<\/a>, while we don\u2019t know the parameters but we have a partial information regarding variables. The main issue is that solutions are not\u00a0well-posed<\/a>. Omitting axiomatic technical details, practical problem is that we can find many functions\u00a0f(x,a)\u00a0or models, explaining the\u00a0dataset.<\/p>\n

So, we seek the following two concepts to be satisfied by our model solution,\u00a0\u00a0f(x,a)=0.<\/p>\n

1. Generalized: A model should not depend on the dataset. This step is called\u00a0model validation.<\/em>
\n2. Minimally complex: A model should obey\u00a0Occam\u2019s razor or principle of parsimony<\/a>. This step is called\u00a0model selection<\/em>.<\/p>\n

\"Diagram\n\nDescription<\/p>\n

Figure 1: A workflow for model validation and selection in supervised learning.<\/p>\n

Generalization of a model can be measured by\u00a0goodness-of-fit<\/a>. It essentially tells us how good our model (chosen function) explains the dataset. To find a\u00a0minimally\u00a0complex model\u00a0<\/strong>requires comparison against another model.<\/p>\n

Up to now, we have not named a technique how to check if a\u00a0model is generalized and selected as the best model. Unfortunately, there is no unique way of doing both and that\u2019s the task of data scientist or quantitative practitioner that requires human judgement.<\/p>\n

Model validation: An example\u00a0<\/strong>
\nOne way to check if a model is generalized enough is to come up with a metric on how good it explains the dataset. Our task in model validation is to estimate the model error. For example,\u00a0root mean square deviation<\/a>\u00a0(RMDS) is one metric we can use. \u00a0If \u00a0RMSD is low, we could say that our model fit is good, ideally it should be close to zero. \u00a0But it is not generalized enough if we use the same dataset to measure the goodness-of-fit. \u00a0We could use different dataset, specially out-of-sample dataset, to validate this as much as we can, i.e. so called hold out method. \u00a0Out-of-sample is just\u00a0a fancy way of saying we did not use the same dataset to find the value of parameters\u00a0a. An improved way of doing this is cross-validation. We split our dataset into\u00a0k partitions, and we obtain\u00a0k RMDS values to averaged over. This is summarised in Figure 1. \u00a0Note that, different parameterisation of the same model does not constitute a different model.<\/p>\n

Model Selection: Detection of overfitting\u00a0<\/strong>
\nOverfitting comes into play when we try to satisfy \u2018minimally complex model\u2019. This is a comparison problem and we need more than one model to judge if a given model is an overfit.\u00a0Douglas Hawkins in his classic paper\u00a0The Problem of Overfitting<\/em><\/a>, states that<\/p>\n

Overfitting of models is widely recognized as a concern. It is less recognized however that overfitting is not an absolute but involves a comparison. A model overfits if it is more complex than another model that fits equally well.<\/em><\/p>\n

The important point here what do we mean by complex model, or how can we quantify model complexity? Unfortunately, again there is no unique way of doing this. One of the most used approaches is that a model having more parameters is getting more complex. But this is again a bit of a\u00a0meme<\/em>\u00a0and not generally true. One could actually resort to different measures of complexity. For example, by this definition\u00a0f1(a,x)=ax\u00a0and\u00a0f2(a,x)=ax2<\/sup>\u00a0have the same complexity by having the same number of free parameters, but intuitively\u00a0f2\u00a0is more complex, while it is nonlinear. There are a lot of information theory based measures of complexity but discussion of those are beyond the scope of our post. For demonstration purposes, we will consider more parameters and degree of\u00a0nonlinearity as more complex a model.<\/p>\n

\"Chart,<\/p>\n

Figure 2: Simulated data and the non-stochastic part of the data.<\/p>\n

Hand on example<\/strong>
\nWe have intuitively covered the reasons behind how we can\u2019t resolve model validation and judge overfitting simultaneously. Now try to demonstrate this with a simple dataset and models, yet essentially capturing the above premise.<\/p>\n

A usual procedure is to generate a synthetic dataset, or simulated dataset, from a model, as a gold standard and use this dataset to build other models. Let\u2019s use the following functional form, from\u00a0classic text of Bishop<\/a>, but with an added Gaussian noise<\/p>\n

f(x)=sin(2\u03c0x)+N(0,0.1).<\/p>\n

We generate large enough set, 100 points to avoid sample size issue discussed in Bishop\u2019s book, see Figure 2. Let\u2019s decide on two models we would like to apply to this dataset in supervised learning task. Note that, we won\u2019t be discussing Bayesian interpretation here, so equivalency of these model under a strong prior assumption is not an issue as we are using this example for ease of demonstrating the concept. A polynomial model of degree 3 and degree 6, we call them g(x) and h(x) respectively, are used to learn from the simulated data.<\/p>\n

g(x)=a0<\/sub>\u00a0+ a1<\/sub>x + a2<\/sub>x2<\/sup>\u00a0+ a3<\/sub>x3<\/sup><\/p>\n

and<\/p>\n

h(x)=b0<\/sub>\u00a0+ b1<\/sub>x + b2<\/sub>x2<\/sup>\u00a0+ b3<\/sub>x3<\/sup>\u00a0+ b4<\/sub>x4<\/sup>\u00a0+ b5<\/sub>x5<\/sup>\u00a0+ b6<\/sub>x6<\/sup><\/p>\n

\"Diagram\n\nDescription<\/p>\n

Figure 3: Overtraining occurs at around after 40 percent of the data usage for g(x).<\/p>\n

Overtraining is not overfitting<\/strong>
\nOvertraining<\/em>\u00a0means a model performance degrades in learning model parameters against an objective variable that effects how model is build, for example, an objective variable can be a training data size or iteration cycle in neural network. This is more prevalent in neural networks (see\u00a0Dayhoff 2011<\/a>). In our practical example, this will manifest in hold out method to measure RMSD in modelling with g(x). In other words finding an optimal number of data points to use to train the model to give a better performance on unseen data, See Figure 3 and 4.<\/p>\n

Overfitting with low validation error<\/strong>
\nWe can also estimate 10-fold cross-validation error, CV-RMSD. For this sampling, g and h have 0.13 and 0.12 CV-RMSD respectively. So as we can see, we have a situation that more complex model reaches similar predictive power with cross validation and we can not distinguish this overfitting by just looking at CV-RMSD value or detecting \u2018overtraining\u2019 curve from Figures 4. We need two models to compare, hence both Figure 3 and 4, with both CV-RMSD values. We might argue that in small data sets we might be able tell the difference by looking at test and training error differences, this is exactly how Bishop explains\u00a0overfitting;<\/em>\u00a0where he points out\u00a0overtraining<\/em>\u00a0in small datasets.<\/p>\n

Which trained model to deploy?<\/strong>
\nNow the question is, we found out best performing model with minimal complexity empirically. All well, but which trained model should we use in production?
\nActually we have already build the model in model selection. In above case, since we got similar
\npredictive power from g and h, we obviously will use g, trained on the splitting sweet spot from Figure 3.<\/p>\n

\"Diagram\n\nDescription<\/p>\n

Figure 4: Overtraining occurs at around after 30 percent of the data usage for h(x)<\/p>\n

Conclusion<\/strong>
\nThe essential message here is good validation performance would not guarantee the detection of an\u00a0overfitted<\/em>\u00a0model. As we have seen from examples using synthetic data in one dimension.\u00a0Overtraining<\/em>\u00a0is actually what most practitioners mean when they use the term\u00a0overfitting<\/em>.<\/p>\n

Outlook<\/strong>
\nAs more and more people are using techniques from machine learning or inverse problems, both in academia and industry, some key technical concepts are deviated a bit and take different definitions and meaning for different people, due to the fact that people learn some concepts not from reading the literature carefully but from their line managers or senior colleagues verbally. This creates\u00a0memes<\/em>\u00a0which are actually wrong or at least creating lots of confusion in jargon. It is very important for all of us as practitioners that we must\u00a0question<\/em>\u00a0all technical concepts and try to seek origins from the published scientific literature and not rely entirely on verbal explanations from our experienced colleagues. Also, we should strongly avoid ridiculing question from colleagues even they sound too simple, at the end of the day we don\u2019t stop learning and naive looking questions might have very important consequences in fundamentals of the field.<\/p>\n

P.S.\u00a0<\/strong>As I mentioned above, the inspiration of writing this post was, a recent post from Gelman (post<\/a>). He defined \u2018overfitting\u2019 as follows:<\/p>\n

Overfitting is when you have a complicated model that gives worse predictions, on average, than a simpler model.<\/em><\/p>\n

Priors and\u00a0equivalence\u00a0of two models aside, Gelman\u2019s definition is weaker than Hawkins definition, that he accepts a complex model having a similar predictive power. So, if we use Gelman\u2019s definitions it is ok to deploy either of\u00a0g<\/em>\u00a0or\u00a0h<\/em>in our toy example above. But strictly speaking from Hawkins perspective we need to deploy\u00a0g<\/em>.<\/p>\n

Original<\/a>. Reposted with permission.<\/p>\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

Understanding overfitting: an inaccurate meme in Machine Learning Preamble There is a lot of confusion among practitioners regarding the concept of\u00a0overfitting. It seems like, a kind of\u00a0an\u00a0urban legend\u00a0or a\u00a0meme, a folklore\u00a0is circulating\u00a0in data science or allied fields with the following statement: Applying\u00a0cross-validation\u00a0prevents overfitting and a good out-of-sample performance, low generalisation error in unseen data, indicates … Continue reading Overfitting – Understanding overfitting<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":1890,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[25],"tags":[32,26,66,58,52,51,59],"class_list":["post-1888","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-bookmarked-articles","tag-analytic","tag-data","tag-data-fallacies","tag-data-modelling","tag-machine-learning","tag-modelling","tag-statistic"],"_links":{"self":[{"href":"https:\/\/mintea.blog\/index.php?rest_route=\/wp\/v2\/posts\/1888","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mintea.blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mintea.blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mintea.blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mintea.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1888"}],"version-history":[{"count":2,"href":"https:\/\/mintea.blog\/index.php?rest_route=\/wp\/v2\/posts\/1888\/revisions"}],"predecessor-version":[{"id":1894,"href":"https:\/\/mintea.blog\/index.php?rest_route=\/wp\/v2\/posts\/1888\/revisions\/1894"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/mintea.blog\/index.php?rest_route=\/wp\/v2\/media\/1890"}],"wp:attachment":[{"href":"https:\/\/mintea.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1888"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mintea.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1888"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mintea.blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1888"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}