# Probability review

Basic probability review

• Sample space ${\Omega}$ can be viewed as the set of results obtained from the probabilistic experiment under consideration. e.g. If the experiment consists of flipping a coin twice, then the sample space would consist of ${\{HH, HT, TT, TH \}}$. Elements of ${\Omega}$ are termed elementary events. Subsets of ${\Omega}$ are termed events. e.g. ${\{HH, HT\}}$ is the event that the first coin flip results in a Head.
• Definition A ${\sigma-}$ field ${(\Omega, \mathcal{F})}$ consists of a sample space ${\Omega}$ and a set ${\mathcal{F}}$ which is a collection of subsets of ${\Omega}$, and satisfies the following properties:
• ${\phi \in \mathcal{F}}$
• closure under complement ${\mathcal{\epsilon} \in \mathcal{F} \Rightarrow \bar{\mathcal{\epsilon}} \in \mathcal{F}}$
• closure under union ${ \forall i \in I, }$ ${ \mathcal{\epsilon}_i \in \mathcal{F} \Rightarrow \cup_{i \in I} \mathcal{\epsilon}_i \in \mathcal{F}}$.

Note that closure under union and closure and complement imply closure under intersection (since ${A \cap B = \overline{ \bar{A} \cup \bar{B} } }$). Note also that ${\Omega \in \mathcal{F}}$. Elements of ${\mathcal{F}}$ can be termed as observable events, and ${\mathcal{F}}$ can be termed as an information structure.

• Definition Given a ${\sigma-}$ field ${(\sigma, \mathcal{F}}$, a probability measure ${Pr : \mathcal {F} \rightarrow \mathbb{R}^{+}}$ is a function which satisfies:
• For all ${\mathcal{\epsilon} \in \mathcal{F}}$, ${0 \leq Pr [ \mathcal{\epsilon}] \leq 1}$.
• ${Pr [ \Omega] = 1}$.
• For mutually disjoint events, ${\mathcal{\epsilon}_1, \mathcal{\epsilon}_2, \ldots }$, ${Pr [ \cup_i \mathcal{\epsilon}_i ] = \sum_{i} Pr [ \mathcal{\epsilon}_i ]}$.

Some other properties that follow from the above:

• ${Pr [ \overline{\mathop{\mathbb E}\mathcal{\epsilon}}] = 1 - Pr [ \mathcal{\epsilon}] }$
• ${Pr [ A \cup B ] = Pr [ A ] + Pr [ B] - Pr [ A \cap B]}$.
• Definition A probability space ${(\sigma, \mathcal{F}, Pr)}$ consists of a ${\sigma-}$ field ${(\sigma, \mathcal{F})}$ together with a probability measure ${Pr}$ defined on it.
• Principle of inclusion-exclusion$\displaystyle Pr [ \cup_{i=1}^{n} \mathcal{\epsilon}_i ] = \sum_i Pr [\mathcal{\epsilon}_i] - \sum_{i < j} Pr [ \mathcal{\epsilon}_i \cap \mathcal{\epsilon}_j] + \sum_{i < j < k} Pr [ \mathcal{\epsilon}_i \cap \mathcal{\epsilon}_j \cap \mathcal{\epsilon}_k] -/+ \ldots$
• Definition Condition probability of event ${A}$ given event ${B}$ is defined as :$\displaystyle Pr [ A | B ] = \frac { Pr [ A \cap B ] } { Pr [B] }$.

From this, we have that : ${ Pr [ X \cap Y ] = Pr [ Y ] Pr [ X | Y ] = Pr [X ] Pr [Y | X ]}$.

• If ${\mathcal{\epsilon}_1, \ldots, \mathcal{\epsilon}_k}$ denote a partition of the sample space ${\Omega}$, then, for any event ${\mathcal{\epsilon}}$, :$\displaystyle Pr [ \mathcal{\epsilon} ] = \sum_{i=1}^{k} Pr [\mathcal{\epsilon}_i ] Pr [ \mathcal{\epsilon} | \mathcal{\epsilon}_i]$
• Bayes’ rule If ${\mathcal{\epsilon}_1, \ldots, \mathcal{\epsilon}_k}$ denote a partition of the sample space ${\Omega}$, then, for any event ${\mathcal{\epsilon}}$, :$\displaystyle Pr [ \mathcal{\epsilon}_i | \mathcal{\epsilon}] = \frac { Pr [ \mathcal{\epsilon}_i] Pr [ \mathcal{\epsilon} | \mathcal{\epsilon}_i ] } { \sum_{j=1}^{k} Pr [\mathcal{\epsilon}_j] Pr [\mathcal{\epsilon} | \mathcal{\epsilon}_j]}$
• Definition A collection of events ${\{ \mathcal{\epsilon}_i | i \ in I\} }$ is \emph {independent} if for all subsets ${ S \subseteq I }$, :$\displaystyle Pr [\displaystyle \cap_{i \in S} \mathcal{\epsilon}_i ] = \displaystyle \prod_{ i \in S } Pr [ \mathcal{\epsilon}_i]$
• Equivalent defintion A collection of events ${\{ \mathcal{\epsilon}_i | i \ in I\} }$ is \emph {independent} if for all ${j \in I}$ and for all subsets ${ S \subseteq I \setminus \{j\} }$,$\displaystyle Pr [ \mathcal{\epsilon}_j | \displaystyle \cap_{i \in S} \mathcal{\epsilon}_i ] = Pr [ \mathcal{\epsilon}_j]$
• A collection of events ${\{ \mathcal{\epsilon}_i | i \ in I\} }$ is \emph {k-wise independent} if every set of ${k}$ events in the collection is independent. 2-wise independence is called pairwise independence.
• Definition A random variable ${X}$ is a real-valued function over the sample space ${X : \Omega \rightarrow \mathbb{R}}$ such that for all ${x \in \mathbb{R}}$:$\displaystyle \{ \omega \in \Omega | X(\omega) \leq x \} \in \mathcal{F}$
• The distribution function ${ F : \mathbb{R} \rightarrow [0,1]}$ for a random variable ${X}$ is defined as ${ F_X (x) = Pr [ X \leq x]}$.
• The density function ${p : \mathbb{R} \rightarrow [0,1]}$ is defined as ${p_X (x) = Pr [X = x ]}$.
• The joint distribution function and joint density function are defined likewise :$\displaystyle F_{X,Y} (x,y) = Pr [ \{X \leq x\} \cap \{ Y \leq y \} ]$$\displaystyle p_{X,Y} (x,y) = Pr [ \{X =x\} \cap \{ Y = y \} ]$

.