Jekyll2022-10-01T20:07:54-07:00https://xinchen236.github.io//feed.xmlXin Chen / HomePostdoctoral Fellow at Harvard UniveristyChen, Xin (陈欣)xinch512@mit.eduNotes on Gaussian Process2020-07-29T00:00:00-07:002020-07-29T00:00:00-07:00https://xinchen236.github.io//posts/2020/07/gaussianprocess<p>My study notes on Gaussian Process and some useful resources.</p> <script id="MathJax-script" async="" src="&lt;url-to-your-site&gt;/mathjax/tex-chtml.js"></script> <h2 id="useful-resources">Useful Resources</h2> <ul> <li> <p>GP Toolbox in Matlab: <a href="http://www.gaussianprocess.org/gpml/code/matlab/doc/">GPML</a>. A GP theme website is <a href="http://www.gaussianprocess.org/">here</a>.</p> </li> <li> <p>A good GP textbook: <a href="http://www.gaussianprocess.org/gpml/chapters/RW.pdf">Gaussian Processes for Machine Learning</a>.</p> </li> </ul> <h2 id="notes-on-gaussian-process"><a href="https://xinchen236.github.io/files/ACC2020slides.pdf">Notes on Gaussian Process</a></h2> <h1 id="i-introduction">I. Introduction</h1> <p>Gaussian process (GP) is a non-parametric supervised machine learning method, which has been widely used to model nonlinear system dynamics as well. GP works to infer an unknown function $y = f(x)$ based on the training set $\mathcal{D}:= \{(x_i, y_i): i=1,\cdots,n\}$ with $n$ noisy observations. Comparing with other machine learning techniques, GP has the following main merits:</p> <ul> <li> <p>GP provides an estimate of uncertainty or confidence in the predictions through the predictive variance, in addition to using the predictive mean as the prediction.</p> </li> <li> <p>GP can work well with small datasets.</p> </li> <li> <p>In the nature of Bayesian learning, GP incorporates prior domain knowledge of the unknwon system by defining kernel covariance function or setting hyperparameters.</p> </li> </ul> <p>Formally, a GP is defined as a collection of random variables, any Gaussian process finite number of which have a joint Gaussian distribution. A GP is fully specified by a mean function $m(x)$ and a (kernel) covariance function $k(x,x')$, which is denoted as \begin{align} f(x)\sim\mathcal{GP}(m(x),k(x,x’)) \end{align}</p> <p>It aims to infer the function value $f(x_*)$ on a new point $x_{*}$ based on the observations $\mathcal{D}$. According to the formal definition, the collection $(\boldsymbol f_{\mathcal{D}}, f(x_*))$ follows a joint Gaussian distribution with</p> $[\boldsymbol f_{\mathcal{D}}; f(x_*)] \sim \mathcal{N} \Big( [ \boldsymbol m_{\mathcal{D}}; m(x_*) ], [ K_{\mathcal{D},\mathcal{D}}, \boldsymbol k_{ *,\mathcal{D}}; \boldsymbol k_{ *,\mathcal{D}}^\top, k(x_*,x_*) ] \Big)$ <p>where vector $\boldsymbol k_{*, \mathcal{D}}:= [ k(x_*,x_1); \cdots; k(x_*, x_n)]$, and matrix $K_{\mathcal{D},\mathcal{D}}$ is the covariance matrix, whose $ij$-component is $k(x_i,x_j)$. Then conditioning on the given observations $\mathcal{D}$, it is known that the posterior distribution $f(x_*)|(\boldsymbol f_{\mathcal{D}} =\boldsymbol y_{\mathcal{D}})$ is also a Gaussian distribution $\mathcal{N}(\mu_{*|\mathcal{D}}, \sigma^2_{*|\mathcal{D}} )$ with the closed form</p> <p>\begin{align<em>} \mu_{</em>|\mathcal{D}} &amp; = m(x_<em>) + <br /> \sigma^2_{</em>|\mathcal{D}} &amp; = \end{align*}</p> $\mu_{*|\mathcal{D}} &amp; = m(x_*) + \\\\ \sigma^2_{*|\mathcal{D}} &amp; =$Chen, Xin (陈欣)xinch512@mit.eduMy study notes on Gaussian Process and some useful resources.Notes on Gaussian Process2014-08-14T00:00:00-07:002014-08-14T00:00:00-07:00https://xinchen236.github.io//blog-post-3My study notes on Gaussian Process and some useful resources. Useful Resources ------ - GP Toolbox in Matlab: [GPML](http://www.gaussianprocess.org/gpml/code/matlab/doc/). A GP theme website is [here](http://www.gaussianprocess.org/). - A good GP textbook: [Gaussian Processes for Machine Learning](http://www.gaussianprocess.org/gpml/chapters/RW.pdf). I. Introduction ====== Gaussian process (GP) is a non-parametric supervised machine learning method, which has been widely used to model nonlinear system dynamics as well. GP works to infer an unknown function $$y = f(x)$$ based on the training set $$\mathcal{D}:= \{(x_i, y_i): i=1,\cdots,n\}$$ with $$n$$ noisy observations. Comparing with other machine learning techniques, GP has the following main merits: - GP provides an estimate of uncertainty or confidence in the predictions through the predictive variance, in addition to using the predictive mean as the prediction. - GP can work well with small datasets. - In the nature of Bayesian learning, GP incorporates prior domain knowledge of the unknwon system by defining kernel covariance function or setting hyperparameters. Formally, a GP is defined as a collection of random variables, any Gaussian process finite number of which have a joint Gaussian distribution. A GP is fully specified by a mean function $$m(x)$$ and a (kernel) covariance function $$k(x,x')$$, which is denoted as \begin{align} f(x) & \sim\mathcal{GP}(m(x),k(x,x')) \\ g(y) & = \end{align} <script type="text/x-mathjax-config"> MathJax.Hub.Config({ tex2jax: { inlineMath: [ ['$','$'], ["\$","\$"] ], processEscapes: true } }); </script> <script type="text/javascript" charset="utf-8" src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML" > </script> <script type="text/javascript" charset="utf-8" src="https://vincenttam.github.io/javascripts/MathJaxLocal.js" > </script>Chen, Xin (陈欣)xinch512@mit.eduMy study notes on Gaussian Process and some useful resources.Blog Post number 22013-08-14T00:00:00-07:002013-08-14T00:00:00-07:00https://xinchen236.github.io//posts/2013/08/blog-post-2<p>This is a sample blog post. Lorem ipsum I can’t remember the rest of lorem ipsum and don’t have an internet connection right now. Testing testing testing this blog post. Blog posts are cool.</p> <h1 id="headings-are-cool">Headings are cool</h1> <h1 id="you-can-have-many-headings">You can have many headings</h1> <h2 id="arent-headings-cool">Aren’t headings cool?</h2>Chen, Xin (陈欣)xinch512@mit.eduThis is a sample blog post. Lorem ipsum I can’t remember the rest of lorem ipsum and don’t have an internet connection right now. Testing testing testing this blog post. Blog posts are cool.