Hendra Bunyamin

[EN] Completing the Square for Normal Model with Multiple Observations

2021-04-12T00:00:00+00:00

The subchapter 2.5 of Bayesian Data Analysis Third Edition explains how to estimate a normal mean with known variance; particularly, the subchapter extends the development of a normal model with a single observation into the more realistic situation where a sample of independent and identically distributed observations \(y = (y_1, \ldots, y_n)\) are available.

\(\pmb{\text{Figure 1}}\): Example of a normal distribution consisting a horde of rabbits. Image taken from Casey Dunn, some rights reserved.

The posterior density of the normal model consists of a likelihood distribution, \(\Pr(y \mid \theta)\), and a prior distribution, \(\Pr(\theta)\). Specifically,

\[\begin{align} y_i \mid \theta &\sim \text{N}(\theta, \sigma^2) && \text{A normal distribution with mean = }\theta \text{ and variance = }\sigma^2\text{, for }i=1, \ldots, n \\ \theta &\sim \text{N}(\mu_0, \tau_0^2) && \text{A normal distribution with mean = }\mu_0 \text{ and variance = }\tau_0^2. \end{align}\]

Proceeding formally, the posterior density is

\(\begin{align} \Pr(\theta \mid y) &\propto \Pr(\theta) \Pr(y \mid \theta) && \text{posterior definition} \tag{1}\label{eq:definition}\\ &= \Pr(\theta) \prod_{i=1}^{n} \Pr(y_i \mid \theta) && \text{i.i.d observations} \tag{2}\label{eq:iid} \\ &\propto \exp \left( -\frac{1}{2 \tau_0^2} (\theta - \mu_0)^2 \right) \prod_{i=1}^n \exp \left( - \frac{1}{2 \sigma^2} (y_i - \theta)^2 \right) && \text{normal distributions} \tag{3}\label{eq:exposition-normal} \\ &= \exp \left( -\frac{1}{2} \left( \frac{1}{\tau_0^2} (\theta - \mu_0)^2 + \frac{1}{\sigma^2} \sum_{i=1}^{n} (y_i - \theta)^2 \right) \right) && \text{sum all terms} \tag{4}\label{eq:sum-all-terms} \\ &= \exp \left( -\frac{1}{2} \left( \frac{1}{\tau_0^2} \theta^2 - \frac{2 \theta \mu_0}{\tau_0^2} + \frac{\mu_0^2}{\tau_0^2} + \frac{1}{\sigma^2} \sum_{i=1}^n (y_i^2 - 2 \theta y_i + \theta^2) \right) \right) && \text{expand all squares} \tag{5}\label{eq:expand-all} \\ &= \exp \left( -\frac{1}{2} \left( \frac{1}{\tau_0^2} \theta^2 - \frac{2 \theta \mu_0}{\tau_0^2} + \frac{\mu_0^2}{\tau_0^2} + \frac{\sum_{i=1}^n y_i^2}{\sigma^2} - \frac{2 \theta \sum_{i=1}^n y_i}{\sigma^2} + \frac{n \theta^2}{\sigma^2} \right) \right) && \text{expand the last term} \tag{6}\label{eq:expand-again} \\ &= \exp \left( -\frac{1}{2} \left( \frac{\theta^2}{\tau_0^2} + \frac{n \theta^2}{\sigma^2} - 2 \theta \left( \frac{\mu_0}{\tau_0^2} + \frac{\sum_{i=1}^n y_i}{\sigma^2} \right) + \frac{\mu_0^2}{\tau_0^2} + \frac{\sum_{i=1}^n y_i^2}{\sigma^2} \right) \right) && \text{group all }\theta s \text{ & } \theta^2 s \tag{7}\label{eq:collect-all} \\ &= \exp \left( -\frac{1}{2} \left( \theta^2 \left( \frac{1}{\tau_0^2} + \frac{n}{\sigma^2} \right) - 2 \theta \left( \frac{\mu_0}{\tau_0^2} + \frac{\sum_{i=1}^n y_i}{\sigma^2} \right) + \frac{\mu_0^2}{\tau_0^2} + \frac{\sum_{i=1}^n y_i^2}{\sigma^2} \right) \times \frac{\frac{1}{\frac{1}{\tau_0^2} + \frac{n}{\sigma^2}}}{\frac{1}{\frac{1}{\tau_0^2} + \frac{n}{\sigma^2}}} \right) && \text{use a trick} \tag{8}\label{eq:multiply-by} \\ &= \exp \left( - \frac{1}{2} \frac{ \left( \theta^2 - 2 \theta \frac{ \frac{\mu_0}{\tau_0^2} + \frac{\sum y_i}{\sigma^2}}{ \frac{1}{\tau_0^2} + \frac{n}{\sigma^2}} + \frac{\frac{\mu_0^2}{\tau_0^2}}{\frac{1}{\tau_0^2} + \frac{n}{\sigma^2}} + \frac{\frac{\sum y_i^2}{\sigma^2}}{\frac{1}{\tau_0^2} + \frac{n}{\sigma^2}} \right) }{\frac{1}{\frac{1}{\tau_0^2} + \frac{n}{\sigma^2}}} \right) \tag{9}\label{eq:atas-bawah} \\ &= \exp \left( - \frac{1}{2} \frac{\left( \theta - \frac{\frac{\mu_0}{\tau_0^2} + \frac{\sum y_i}{\sigma^2} }{ \frac{1}{\tau_0^2} + \frac{n}{\sigma^2} } \right)^2 + C}{\frac{1}{\frac{1}{\tau_0^2} + \frac{n}{\sigma^2}}} \right) && \text{with }C \text{ is a constant} \tag{10}\label{eq:a-constant} \\ &\propto \exp \left( -\frac{1}{2} \frac{(\theta - \mu_n)^2}{\tau_n^2} \right) \tag{10}\label{eq:almost} \\ &\propto \text{N}(\mu_n, \tau_n^2) && \text{a normal distribution} \tag{11}\label{eq:finally} \end{align}\) with

\(\begin{align} \mu_n &= \frac{\frac{\mu_0}{\tau_0^2} + \frac{\sum_{i=1}^n y_i}{\sigma^2}}{\frac{1}{\tau_0^2} + \frac{n}{\sigma^2} } \tag{12}\label{eq:mu-n} \\ &= \frac{\frac{\mu_0}{\tau_0^2} + \frac{n \bar{y}}{\sigma^2}}{\frac{1}{\tau_0^2} + \frac{n}{\sigma^2} } && \text{because }\bar{y} = \frac{\sum_{i=1}^n y_i}{n} \tag{13}\label{eq:mu-n-2} \end{align}\) and

\[\begin{equation} \frac{1}{\tau_n^2} = \frac{1}{\tau_0^2} + \frac{n}{\sigma^2}. \tag{14}\label{eq:sigma-n} \end{equation}\]

At last, we have shown that the posterior distribution of the normal model is also a normal distribution as it is explained by Equation (2.11) and (2.12) on page 42 of the book.

[EN] A Quantum of the Fisher Information Derivation

2021-02-23T00:00:00+00:00

This post elaborates a derivation of Equation (2.20) on page 53 of Bayesian Data Analysis Third Edition.

\(\pmb{\text{Figure 1}}\): Sir Ronald Aylmer Fisher (17 February 1890 - 29 July 1962). One of his many great contributions to Statistics is Fisher Information. Image taken from Wikipedia, some rights reserved.

Concretely, we want to show the derivation \(J(\theta)\), the Fisher Information, from

\[\begin{equation} J(\theta) = \text{E}\left( \left( \frac{d \log \Pr(y \mid \theta )}{d\theta} \right)^2 \, \middle| \, \theta \right) \tag{1}\label{eq:start-point} \end{equation}\]

\[\begin{equation} J(\theta) = - \text{E}\left( \frac{d^2 \log \Pr(y \mid \theta)}{d\theta^2} \, \middle| \, \theta \right). \tag{2}\label{eq:end-point} \end{equation}\]

The idea of this derivation comes from a lecture note by John Duchi from Stanford Statistics class. The difference between this post and the lecture note is that the lecture note deals with multi-variables which employs second derivatives for multi-values (Hessian matrix); on the other hand, this post deals with a single variable and employs a second derivative for just one value.

Let’s start with computing \(\begin{equation} \text{E} \left( \frac{d \log \Pr(y \mid \theta)}{d\theta} \, \middle| \, \theta \right) \end{equation}\).

\[\require{cancel} \begin{align} \text{E}\left( \frac{d \log \Pr(y \mid \theta)}{d\theta} \, \middle| \, \theta \right) &= \int \frac{d \log \Pr(y \mid \theta)}{d\theta} \Pr(y \mid \theta) d\theta && \text{definition of expectation} \tag{3}\label{eq:dlog-1} \\ &= \int \frac{d \Pr(y \mid \theta)}{d\theta} \frac{1}{\Pr(y \mid \theta)} \; \Pr(y \mid \theta) d\theta && \text{derivation of }\frac{d \log \Pr(y \mid \theta)}{d\theta} \tag{4}\label{eq:dlog-2} \\ &= \int \frac{d \Pr(y \mid \theta)}{d\theta} \frac{1}{\cancel{\Pr(y \mid \theta)}} \; \cancel{\Pr(y \mid \theta)} d\theta \tag{5}\label{eq:dlog-3} \\ &= \int \frac{d \Pr(y \mid \theta)}{d\theta} d\theta \tag{6}\label{eq:dlog-4} \\ &= \frac{d}{d\theta} \int \Pr(y \mid \theta) d\theta && \text{exchange }\frac{d}{d\theta} \text{ and } \int \tag{7}\label{eq:dlog-5} \\ &= \frac{d}{d\theta} \underbrace{\int \Pr(y \mid \theta) d\theta}_{1} && \text{property of a pdf}\tag{8}\label{eq:dlog-6} \\ &= \frac{d}{d\theta} (1) \tag{9}\label{eq:dlog-7} \\ &= 0. \tag{10}\label{eq:dlog-8} \end{align}\]

Consider Equation \eqref{eq:dlog-5}, we shall utilize this exchangeability between integral and differentiation again later.

Equation \eqref{eq:dlog-2} states that

\[\begin{equation} \frac{d \log \Pr(y \mid \theta)}{d\theta} = \underbrace{\frac{1}{\Pr(y \mid \theta)}}_{u} \underbrace{\frac{d \Pr(y \mid \theta)}{d\theta}}_{v}. \tag{11}\label{eq:first-order-derivation} \end{equation}\]

Therefore,

\[\begin{align} \frac{d^2 \log \Pr(y \mid \theta)}{d\theta^2} &= \underbrace{- \frac{1}{\Pr( y \mid \theta )^2} \frac{d \Pr(y \mid \theta)}{d\theta}}_{u^{\prime}} \underbrace{\frac{d \Pr(y \mid \theta)}{d\theta}}_{v} + \underbrace{\frac{1}{\Pr(y \mid \theta)}}_{u} \underbrace{\frac{d^2 \Pr(y \mid \theta)}{d\theta^2}}_{v^{\prime}} && \text{based on } u^{\prime} v + u v^{\prime} \tag{12}\label{eq:second-order-1} \\ &= \frac{d^2 \Pr(y \mid \theta)}{d\theta^2} \frac{1}{\Pr(y \mid \theta)} - \left( \frac{d\Pr(y \mid \theta)}{d\theta} \frac{1}{\Pr(y \mid \theta)} \right) \left( \frac{d\Pr(y \mid \theta)}{d\theta} \frac{1}{\Pr(y \mid \theta)} \right) && \text{just rearranging} \tag{13}\label{eq:second-order-2} \\ &= \frac{d^2 \Pr(y \mid \theta)}{d\theta^2} \frac{1}{\Pr(y \mid \theta)} - \left( \frac{d \log \Pr( y \mid \theta)}{d\theta} \right) \left( \frac{d \log \Pr( y \mid \theta)}{d\theta} \right) && \text{based on Equation }\eqref{eq:first-order-derivation} \tag{14}\label{eq:second-order-3} \\ &= \frac{d^2 \Pr(y \mid \theta)}{d\theta^2} \frac{1}{\Pr(y \mid \theta)} - \left( \frac{d \log \Pr( y \mid \theta)}{d\theta} \right)^{2} \tag{15}\label{eq:second-order-4} \end{align}\]

From Equation \eqref{eq:second-order-4} we obtain

\[\begin{align} \frac{d^2 \log \Pr(y \mid \theta)}{d\theta^2} = \frac{d^2 \Pr(y \mid \theta)}{d\theta^2} \frac{1}{\Pr(y \mid \theta)} - \left( \frac{d \log \Pr( y \mid \theta)}{d\theta} \right)^{2} &\Longleftrightarrow \left( \frac{d \log \Pr( y \mid \theta)}{d\theta} \right)^{2} = - \frac{d^2 \log \Pr(y \mid \theta)}{d\theta^2} + \frac{d^2 \Pr(y \mid \theta)}{d\theta^2} \frac{1}{\Pr(y \mid \theta)} \tag{16}\label{eq:second-order-last} \end{align}\]

Now we are ready to calculate \(\begin{equation} \text{E}\left( \left( \frac{d \log \Pr(y \mid \theta )}{d\theta} \right)^2 \, \middle| \, \theta \right). \end{equation}\)

\[\require{cancel} \begin{align} \text{E}\left( \left( \frac{d \log \Pr(y \mid \theta )}{d\theta} \right)^2 \, \middle| \, \theta \right) &= \int \left( \frac{d \log \Pr(y \mid \theta )}{d\theta} \right)^2 \Pr(y \mid \theta) d\theta && \text{by definition} \tag{17}\label{eq:final-showdown-1}\\ &= \int \left( - \frac{d^2 \log \Pr(y \mid \theta)}{d\theta^2} + \frac{d^2 \Pr(y \mid \theta)}{d\theta^2} \frac{1}{\Pr(y \mid \theta)} \right) \Pr(y \mid \theta) d\theta && \text{by Equation }\eqref{eq:second-order-last} \tag{18}\label{eq:final-showdown-2}\\ &= \int \left( - \frac{d^2 \log \Pr(y \mid \theta)}{d\theta^2} \right) \Pr(y \mid \theta) d\theta + \int \frac{d^2 \Pr(y \mid \theta)}{d\theta^2} \frac{1}{\Pr(y \mid \theta)} \Pr(y \mid \theta) d\theta && \text{by distributive} \tag{19}\label{eq:final-showdown-3}\\ &= \int \left( - \frac{d^2 \log \Pr(y \mid \theta)}{d\theta^2} \right) \Pr(y \mid \theta) d\theta + \int \frac{d^2 \Pr(y \mid \theta)}{d\theta^2} \frac{1}{\cancel{\Pr(y \mid \theta)}} \cancel{\Pr(y \mid \theta)} d\theta \tag{20}\label{eq:final-showdown-4}\\ &= \int \left( - \frac{d^2 \log \Pr(y \mid \theta)}{d\theta^2} \right) \Pr(y \mid \theta) d\theta + \int \frac{d^2 \Pr(y \mid \theta)}{d\theta^2} d\theta \tag{21}\label{eq:final-showdown-5}\\ &= \int \left( - \frac{d^2 \log \Pr(y \mid \theta)}{d\theta^2} \right) \Pr(y \mid \theta) d\theta + \frac{d^2}{d\theta^2} \left( \int \Pr(y \mid \theta) d\theta \right) && \text{by exchangeability again} \tag{22}\label{eq:final-showdown-6}\\ &= \int \left( - \frac{d^2 \log \Pr(y \mid \theta)}{d\theta^2} \right) \Pr(y \mid \theta) d\theta + \underbrace{\frac{d^2}{d\theta^2} \left( \int \Pr(y \mid \theta) d\theta \right)}_{0} && \text{similar to Equation }\eqref{eq:dlog-8} \tag{23}\label{eq:final-showdown-7}\\ &= \int \left( - \frac{d^2 \log \Pr(y \mid \theta)}{d\theta^2} \right) \Pr(y \mid \theta) d\theta \tag{24}\label{eq:final-showdown-8}\\ &= - \int \left( \frac{d^2 \log \Pr(y \mid \theta)}{d\theta^2} \right) \Pr(y \mid \theta) d\theta \tag{25}\label{eq:final-showdown-9}\\ &= - \text{E}\left( \frac{d^2 \log \Pr(y \mid \theta)}{d\theta^2} \, \middle| \, \theta \right). && \text{by definition} \tag{26}\label{eq:final-showdown-10} \end{align}\]

At last, we have finally shown that

\[\begin{equation} J(\theta) = \text{E}\left( \left( \frac{d \log \Pr(y \mid \theta )}{d\theta} \right)^2 \, \middle| \, \theta \right) = - \text{E}\left( \frac{d^2 \log \Pr(y \mid \theta)}{d\theta^2} \, \middle| \, \theta \right) \end{equation}\]

as it is explained by Equation (2.20) on page 53 of the book.

[EN] Variance and Covariance of Categorical Distribution

2020-09-18T00:00:00+00:00

This post is inspired by the lecture given by David Blei on Thursday, 17 September 2020. One of the topics he explained was related to a categorical variable and a categorical distribution. This post will elaborate those two concepts. Let’s get started.

\( \pmb{\text{Figure 1}} \) : A categorical variable ( \( \text{Color} \)) and its values ( \( \text{Red} \), \( \text{Yellow} \), and \( \text{Green} \) ). Image taken from Kaggle, some rights reserved.

\(\text{Figure 1}\) shows an example of categorical values stored in a categorical variable, \(\text{Color}\). Basically, a categorical variable takes one of \(K\) values and each categorical value is represented by a \(K\)-vector with a single \(1\) and otherwise \(0\)s.

Let’s denote a categorical variable as \(x^{(k)}\) which means that the \(k\)th component of it has a a single \(1\) and otherwise \(0\)s.

For example, the categorical variable in \(\text{Figure 1}\) has \(3\) values ( \(\text{Red}\), \(\text{Yellow}\), and \(\text{Green}\) ) and each value is represented by \(3\)-vector with a single \(1\) and otherwise \(0\)s as follows:

\[\begin{align} \text{Red} &= x^{(1)} = (1, 0, 0 ) \\ \text{Yellow} &= x^{(2)} = (0, 1, 0 ) \tag{1}\label{eq:yellow} \\ \text{Green} &= x^{(3)} = (0, 0, 1 ). \end{align}\]

The \(K\)-vector with single \(1\) and otherwise \(0\)s is commonly named one-hot vector.

A categorical distribution is parameterized by \(\theta\). Moreover, \(\pmb{\theta}\) specifies the probability of each categorical value. Suppose we have \(K\) categorical values; therefore,

\[\begin{equation} \theta = (\theta_1, \theta_2, \ldots, \theta_K) \tag{2}\label{eq:theta} \end{equation}\]

with

\[\begin{equation} \sum_{k=1}^{K}{\theta_k} = 1 \text{ and } 0 \leq \theta_k \leq 1 \text{ for }k=1, \ldots, K. \tag{3}\label{eq:theta-constraints} \end{equation}\]

Consider that \(X^{(k)}\) is a random categorical variable which takes one of \(K\) values. Moreover, since \(X^{(k)}\) is random variable, it has categorical distribution that is described by a discrete probability distribution,

\(\begin{equation} \text{p}(x^{(k)}) = \prod_{l=1}^{K}{\theta_{l}^{x^{(l)}}} \tag{4}\label{eq:pdf-categorical} \end{equation}\)
with \(x^{(l)}\) is the \(l\)th component of \(x^{(k)}\). Additionally, we elaborate Equation \eqref{eq:pdf-categorical} into

\[\begin{align} \text{p}(x^{(k)}) &= \prod_{l=1}^{K}{\theta_{l}^{x^{(l)}}} \\ &= \theta_{1}^{x^{(1)}} \times \theta_{2}^{x^{(2)}} \times \cdots \times \theta_{k}^{x^{(k)}} \times \cdots \times \theta_{K}^{x^{(K)}} \\ &= \theta_{1}^{0} \times \theta_{2}^{0} \times \cdots \times \theta_{k}^{1} \times \cdots \times \theta_{K}^{0} \\ &= \theta_{k}. \tag{5}\label{eq:pdf-categorical-simplified} \end{align}\]

Let’s put Equation \eqref{eq:pdf-categorical-simplified} into practice and demonstrate it in one example. Suppose we want to compute \(\text{p}(\text{Yellow})\) in Equation \eqref{eq:yellow},

\[\begin{align} \text{p}(\text{Yellow}) &= \text{p}(x^{(2)}) \\ &= \text{p}((0,1,0)) & \Rightarrow \text{1st} = 0, \text{2nd} = 1, \text{3rd} = 0 \\ &= \prod_{l=1}^{3}{\theta_{l}^{x^{(l)}}} \\ &= \theta_{1}^{x^{(1)}} \times \theta_{2}^{x^{(2)}} \times \theta_{3}^{x^{(3)}} \\ &= \theta_{1}^{0} \times \theta_{2}^{1} \times \theta_{3}^{0} \\ &= \theta_{2}. \end{align}\]

With Equation \eqref{eq:pdf-categorical-simplified} in hand, we are now ready to compute the expectation of \(X^{(k)}\) as

\[\begin{align} \text{E}(X^{(k)}) &= \sum_{l=1}^{K}{x^{(l)} \text{p}(x^{(l)})} \\ &= \underbrace{0 \times \text{p}(x^{(1)})}_{1\text{st}} + \underbrace{0 \times \text{p}(x^{(2)})}_{2\text{nd}} + \cdots + \underbrace{1 \times \text{p}(x^{(k)})}_{k\text{th}} + \cdots + \underbrace{0 \times \text{p}(x^{(K)}}_{K\text{th}}) \\ &= \text{p}(x^{(k)}) \\ &= \theta_k. \tag{6}\label{eq:expectation} \end{align}\]

Next, we compute the Variance, \(\text{Var}\), as follows:

\[\begin{align} \text{Var}(X^{(k)}) &= \underbrace{\text{E}((X^{(k)})^2)}_{\text{Part I}} - \underbrace{(\text{E}(X^{(k)}))^2}_{\text{Part II}}. & \text{the definition of variance} \tag{7}\label{eq:variance-definition} \\ \end{align}\]

Next, we compute \(\text{Part I}\), \(\text{E}((X^{(k)})^2)\), as follows:

\[\begin{align} \text{E}((X^{(k)})^2) &= \sum_{l=1}^{K}{(x^{(l))^2} \text{p}(x^{(l)})} \\ &= \underbrace{0^2 \times \text{p}(x^{(1)})}_{1\text{st}} + \underbrace{0^2 \times \text{p}(x^{(2)})}_{2\text{nd}} + \cdots + \underbrace{1^2 \times \text{p}(x^{(k)})}_{k\text{th}} + \cdots + \underbrace{0^2 \times \text{p}(x^{(K)}}_{K\text{th}}) \\ &= \text{p}(x^{(k)}) \\ &= \theta_k. \tag{8}\label{eq:expectation-x-square} \end{align}\]

Now, we can finalize computing the Variance in Equation \eqref{eq:variance-definition},

\[\begin{align} \text{Var}(X^{(k)}) &= \text{E}((X^{(k)})^2) - (\text{E}(X^{(k)}))^2 && \text{by definition of variance} \\ &= \theta_k - (\theta_k)^2 && \text{using Equation }\eqref{eq:expectation} \text{ and }\eqref{eq:expectation-x-square} \\ &= \theta_k (1 - \theta_k). && \text{using distributive property} \tag{9}\label{eq:variance} \end{align}\]

Last but not least, we shall compute the Covariance, \(\text{Cov}(X^{(j)}, X^{(k)})\). We start by the definition of Covariance,

\[\begin{align} \text{Cov}(X^{(j)}, X^{(k)}) &= \underbrace{\text{E}(X^{(j)} X^{(k)})}_{\text{Part I}} - \underbrace{(\text{E}(X^{(j)}) E(X^{(k)}))}_{\text{Part II}}. && \text{by definition} \tag{10}\label{eq:covariance} \end{align}\]

Let’s compute the \(\text{Part I}\) as follows:

\[\begin{align} \text{E}(X^{(j)} X^{(k)}) &= (0)(0) \theta_1 + \cdots + \underbrace{(1)(0) \theta_j}_{j\text{th}} + \cdots + \underbrace{(0)(1) \theta_k}_{k\text{th}} + \cdots + (0)(0) \theta_K \\ &= 0. \tag{11}\label{eq:covariance-zero} \end{align}\]

Eventually, we can finalize Equation \eqref{eq:covariance} as

\[\begin{align} \text{Cov}(X^{(j)}, X^{(k)}) &= \text{E}(X^{(j)} X^{(k)}) - (\text{E}(X^{(j)}) E(X^{(k)})) \\ &= 0 - \theta_j \theta_k && \text{using Equation } \eqref{eq:expectation} \text{ and }\eqref{eq:covariance-zero} \\ &= - \theta_j \theta_k. \end{align}\]

To conclude, we have shown how to derive the expectation, variance, and covariance of a categorical distribution. We hope this post helps anyone who wants to understand a categorical distribution.

[ID] Contoh Pembuktian Ruang Vektor dengan sangat Detil

2019-03-22T00:00:00+00:00

Tunjukkan bahwa himpunan dari semua matriks berukuran \(2 \times 3\) beserta operasi matrix addition dan scalar multiplication merupakan sebuah ruang vektor.

Bukti:
Misalkan himpunan dari semua matriks berukuran \(2 \times 3\) beserta operasi matrix addition dan scalar multiplication adalah \(V\).

Diketahui juga \(A\), \(B\), dan \(C\) adalah matriks berukuran \(2 \times 3\) dan \(k\), \(l\) adalah skalar dengan spesifikasi sebagai berikut:

\(A = \begin{bmatrix}a_{11} & a_{12} & a_{13}\\a_{21} & a_{22} & a_{23}\end{bmatrix}\), \(B = \begin{bmatrix} b_{11} & b_{12} & b_{13} \\ b_{21} & b_{22} & b_{23}\end{bmatrix}\), dan \(C = \begin{bmatrix} c_{11} & c_{12} & c_{13} \\ c_{21} & c_{22} & c_{23} \end{bmatrix}\).

Larson (2016) subbab 4.2 hlm. 161 menyatakan bahwa pembuktian suatu himpunan merupakan ruang vektor harus memenuhi 10 aksioma. Berikut akan dibuktikan untuk 10 aksioma tersebut.

Apakah \(A+B\) juga ada di dalam \(V\)?
Ya karena
\(\begin{align} A + B &= \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \end{bmatrix} + \begin{bmatrix} b_{11} & b_{12} & b_{13} \\ b_{21} & b_{22} & b_{23} \end{bmatrix} \\ &= \begin{bmatrix} a_{11}+b_{11} & a_{12}+b_{12} & a_{13}+b_{13} \\ a_{21}+b_{21} & a_{22}+b_{22} & a_{23}+b_{23} \end{bmatrix}. \end{align}\)

\(\begin{bmatrix} a_{11}+b_{11} & a_{12}+b_{12} & a_{13}+b_{13} \\ a_{21}+b_{21} & a_{22}+b_{22} & a_{23}+b_{23} \end{bmatrix}\) merupakan matriks berukuran \(2 \times 3\), berarti \(A+B\) juga ada di dalam \(V\).
Apakah \(A+B = B+A\)?
Ya karena
\(\begin{align} A+B &= \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \end{bmatrix} + \begin{bmatrix} b_{11} & b_{12} & b_{13} \\ b_{21} & b_{22} & b_{23} \end{bmatrix} \\ &= \begin{bmatrix} a_{11}+b_{11} & a_{12}+b_{12} & a_{13}+b_{13} \\ a_{21}+b_{21} & a_{22}+b_{22} & a_{23}+b_{23} \end{bmatrix} \\ &= \begin{bmatrix} b_{11}+a_{11} & b_{12}+a_{12} & b_{13}+a_{13} \\ b_{21}+a_{21} & b_{22}+a_{22} & b_{23}+a_{23} \end{bmatrix} \\ &= \begin{bmatrix} b_{11} & b_{12} & b_{13} \\ b_{21} & b_{22} & b_{23} \end{bmatrix} + \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \end{bmatrix} \\ &= B + A. \end{align}\)
Apakah \(A + (B + C) = (A+B) + C\)?
Ya karena
\(\begin{align} A+(B+C) &= \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \end{bmatrix} + \left( \begin{bmatrix} b_{11} & b_{12} & b_{13} \\ b_{21} & b_{22} & b_{23} \end{bmatrix} + \begin{bmatrix} c_{11} & c_{12} & c_{13} \\ c_{21} & c_{22} & c_{23} \end{bmatrix} \right) \\ &= \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \end{bmatrix} + \left( \begin{bmatrix} b_{11} + c_{11} & b_{12} + c_{12} & b_{13} + c_{13} \\ b_{21} + c_{21} & b_{22} + c_{22} & b_{23} + c_{23} \end{bmatrix} \right) \\ &= \begin{bmatrix} a_{11} + b_{11} + c_{11} & a_{12} + b_{12} + c_{12} & a_{13} + b_{13} + c_{13} \\ a_{21} + b_{21} + c_{21} & a_{22} + b_{22} + c_{22} & a_{23} + b_{23} + c_{23} \end{bmatrix} \\ &= \begin{bmatrix} a_{11} + b_{11} & a_{12} + b_{12} & a_{13} + b_{13} \\ a_{21} + b_{21} & a_{22} + b_{22} & a_{23} + b_{23} \end{bmatrix} + \begin{bmatrix} c_{11} & c_{12} & c_{13} \\ c_{21} & c_{22} & c_{23} \end{bmatrix} \\ &= \left( \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \end{bmatrix} + \begin{bmatrix} b_{11} & b_{12} & b_{13} \\ b_{21} & b_{22} & b_{23} \end{bmatrix} \right) + \begin{bmatrix} c_{11} & c_{12} & c_{13} \\ c_{21} & c_{22} & c_{23} \end{bmatrix} \\ &= (A+B)+C. \end{align}\)
Apakah \(V\) mempunyai matriks nol \(\textbf{0}\) sedemikian sehingga untuk setiap matriks \(A\) di \(V\), \(A + \textbf{0} = A\)?
Ada, matriks \(\mathbf{0}\) tersebut adalah
\(\begin{bmatrix} 0 & 0 & 0 \\ 0 & 0 & 0 \end{bmatrix}\) karena \(A + \mathbf{0} = \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \end{bmatrix} + \begin{bmatrix} 0 & 0 & 0 \\ 0 & 0 & 0 \end{bmatrix} = \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \end{bmatrix} = A.\)
Untuk setiap matriks \(A\) di dalam \(V\), apakah \(V\) mempunyai matriks yang dilambangkan dengan \(-A\) sedemikian sehingga \(A + (-A) = \mathbf{0}\)?
Ada, matriks \(-A\) tersebut adalah \(\begin{bmatrix} -a_{11} & -a_{12} & -a_{13} \\ -a_{21} & -a_{22} & -a_{23} \end{bmatrix}\) karena \(\begin{align} A + (-A) &= \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \end{bmatrix} + \begin{bmatrix} -a_{11} & -a_{12} & -a_{13} \\ -a_{21} & -a_{22} & -a_{23} \end{bmatrix} \\ &= \begin{bmatrix} a_{11}-a_{11} & a_{12}-a_{12} & a_{13}-a_{13} \\ a_{21}-a_{21} & a_{22}-a_{22} & a_{23}-a_{23} \end{bmatrix} \\ &= \begin{bmatrix} 0 & 0 & 0 \\ 0 & 0 & 0 \end{bmatrix} \\ &= \mathbf{0}. \end{align}\)
Apakah \(kA\) juga ada di dalam \(V\)?
Ya, karena \(kA = k \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \end{bmatrix} = \begin{bmatrix} ka_{11} & ka_{12} & ka_{13} \\ ka_{21} & ka_{22} & ka_{23} \end{bmatrix}.\)

\(\begin{bmatrix} ka_{11} & ka_{12} & ka_{13} \\ ka_{21} & ka_{22} & ka_{23} \end{bmatrix}\) adalah matriks berukuran \(2 \times 3\) sehingga \(cA\) juga berada di dalam \(V\).
Apakah \(k(A+B) = kA + kB\)?
Ya karena
\(\begin{align} k(A+B) &= k \left( \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \end{bmatrix} + \begin{bmatrix} b_{11} & b_{12} & b_{13} \\ b_{21} & b_{22} & b_{23} \end{bmatrix} \right) \\ &= k \left( \begin{bmatrix} a_{11} + b_{11} & a_{12} + b_{12} & a_{13} + b_{13} \\ a_{21} + b_{21} & a_{22} + b_{22} & a_{23} + b_{23} \end{bmatrix} \right) \\ &= \begin{bmatrix} k(a_{11} + b_{11}) & k(a_{12} + b_{12}) & k(a_{13} + b_{13}) \\ k(a_{21} + b_{21}) & k(a_{22} + b_{22}) & k(a_{23} + b_{23}) \end{bmatrix} \\ &= \begin{bmatrix} ka_{11} + kb_{11} & ka_{12} + kb_{12} & ka_{13} + kb_{13} \\ ka_{21} + kb_{21} & ka_{22} + kb_{22} & ka_{23} + kb_{23} \end{bmatrix} \\ &= \begin{bmatrix} ka_{11} & ka_{12} & ka_{13} \\ ka_{21} & ka_{22} & ka_{23} \end{bmatrix} + \begin{bmatrix} kb_{11} & kb_{12} & kb_{13} \\ kb_{21} & kb_{22} & kb_{23} \end{bmatrix} \\ &= k \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \end{bmatrix} + k \begin{bmatrix} b_{11} & b_{12} & b_{13} \\ b_{21} & b_{22} & b_{23} \end{bmatrix} \\ &= kA + kB. \end{align}\)
Apakah \((k+l)A = kA + lA\)?
Ya karena
\(\begin{align} (k+l) A &= (k+l) \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \\ \end{bmatrix} \\ &= \begin{bmatrix} (k+l) a_{11} & (k+l) a_{12} & (k+l) a_{13} \\ (k+l) a_{21} & (k+l) a_{22} & (k+l) a_{23} \end{bmatrix} \\ &= \begin{bmatrix} k a_{11} + l a_{11} & k a_{12} + l a_{12} & k a_{13} + l a_{13} \\ k a_{21} + l a_{21} & k a_{22} + l a_{22} & k a_{23} + l a_{23} \end{bmatrix} \\ &= \begin{bmatrix} k a_{11} & k a_{12} & k a_{13} \\ k a_{21} & k a_{22} & k a_{23} \end{bmatrix} + \begin{bmatrix} l a_{11} & l a_{12} & l a_{13} \\ l a_{21} & l a_{22} & l a_{23} \end{bmatrix} \\ &= k \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \end{bmatrix} + l \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \end{bmatrix} \\ &= kA + lA. \end{align}\)
Apakah \(k(lA) = (kl)A\)?
Ya karena
\(\begin{align} k(l A) &= k \left( l \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \end{bmatrix} \right) \\ &= k \left( \begin{bmatrix} l a_{11} & l a_{12} & l a_{13} \\ l a_{21} & l a_{22} & l a_{23} \end{bmatrix} \right) \\ &= k \begin{bmatrix} l a_{11} & l a_{12} & l a_{13} \\ l a_{21} & l a_{22} & l a_{23} \end{bmatrix} \\ &= \begin{bmatrix} kl a_{11} & kl a_{12} & kl a_{13} \\ kl a_{21} & kl a_{22} & kl a_{23} \end{bmatrix} \\ &= \begin{bmatrix} (kl)a_{11} & (kl)a_{12} & (kl)a_{13} \\ (kl)a_{21} & (kl)a_{22} & (kl)a_{23} \end{bmatrix} \\ &= (kl) \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \end{bmatrix} \\ &= (kl) A. \end{align}\)
Apakah \(1(A) = A\)?
Ya karena
\(\begin{align} 1(A) &= 1 \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \end{bmatrix} \\ &= \begin{bmatrix} 1a_{11} & 1a_{12} & 1a_{13} \\ 1a_{21} & 1a_{22} & 1a_{23} \end{bmatrix} \\ &= \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \end{bmatrix} \\ &= A. \end{align}\)

Karena \(V\) memenuhi 10 aksioma dari definisi suatu ruang vektor, \(V\) atau himpunan dari semua matriks berukuran \(2 \times 3\) beserta operasi matrix addition dan scalar multiplication merupakan sebuah ruang vektor. \(\square\)