Information Theory & Digital Communications

• Bandwidth: The range of frequencies allocated to transmission.
• Reliability: Characterized by the Output Signal-to-Noise Ratio (SNR).

• Bandwidth Utilization (Spectral Efficiency): Defined as the transmission rate per unit bandwidth $\eta = R_s / B$ or $R_b / B$ (symbols/s/Hz or bits/s/Hz).
• Reliability: Characterized by the Bit Error Rate (BER) $P_b$ or Symbol Error Rate (SER) $P_s$, as a function of the energy per bit or symbol to noise ratio ($E_b/N_0$ or $E_s/N_0$). $$ P_b = \frac{\text{Incorrect Bits}}{\text{Total Transmitted Bits}}, \quad P_s = \frac{\text{Incorrect Symbols}}{\text{Total Transmitted Symbols}} $$
  • For Binary systems ($M = 2$): $P_b = P_s$.
  • For $M$-ary systems ($M > 2$): $P_b < P_s$.
  • Under Natural Binary Coding: $$ P_b = \frac{M}{2(M-1)} P_s $$
  • Under Gray Coding (where adjacent symbols differ by only 1 bit): $$ P_b \approx \frac{1}{\log_2 M} P_s $$

• Efficiency: Transmission bandwidth is $ B = 2f_m $. Power efficiency for single-tone: $ \eta = \frac{\beta^2}{2 + \beta^2} $.
• Input Signal and Noise Powers: $$ S_i = \frac{1}{2} A_c^2 \left( 1 + E[f^2(t)] \right), \quad N_i = n_0 B = 2 n_0 f_m $$
• Output Signal and Noise Powers (Coherent/Envelope Demodulation): $$ S_o = A_c^2 E[f^2(t)], \quad N_o = n_0 B = 2 n_0 f_m $$
• Modulation Gain: $$ G_1 = \frac{S_o / N_o}{S_i / N_i} = \frac{2 E[f^2(t)]}{1 + E[f^2(t)]} \le \frac{2}{3} \quad (\text{since } |f(t)| \le 1) $$

• Efficiency: Bandwidth $ B = 2f_m $; carrier is suppressed, giving $\eta = 100\%$.
• Input Signal and Noise Powers: $$ S_i = \frac{1}{2} A_c^2 E[f^2(t)], \quad N_i = n_0 B = 2 n_0 f_m $$
• Output Signal and Noise Powers (Coherent Demodulation): $$ S_o = \frac{1}{4} A_c^2 E[f^2(t)], \quad N_o = \frac{1}{4} n_0 B = \frac{1}{2} n_0 f_m $$
• Modulation Gain: $$ G_1 = \frac{S_o / N_o}{S_i / N_i} = 2 $$

• Efficiency: Bandwidth is halved, $ B = f_m $; power efficiency $\eta = 100\%$.
• Input Signal and Noise Powers: $$ S_i = \frac{1}{4} A_c^2 E[f^2(t)], \quad N_i = n_0 B = n_0 f_m $$
• Output Signal and Noise Powers (Coherent Demodulation): $$ S_o = \frac{1}{16} A_c^2 E[f^2(t)], \quad N_o = \frac{1}{4} n_0 B = \frac{1}{4} n_0 f_m $$
• Modulation Gain: $$ G_1 = \frac{S_o / N_o}{S_i / N_i} = 1 $$

Requires the sideband shaping filter to satisfy the vestigial symmetry condition around the carrier frequency:

• Frequency Modulation (FM): $$ s_{FM}(t) = A_c \cos\left( \omega_c t + K_{FM} \int_{-\infty}^t f(\tau) d\tau \right) $$
• Phase Modulation (PM): $$ s_{PM}(t) = A_c \cos\left( \omega_c t + K_{PM} f(t) \right) $$

In wideband FM, the transmission bandwidth is dominated by the frequency deviation and is independent of the message frequency $f_m$. This allows FM to achieve significant noise suppression and high output SNR compared to PM. Thus, FM is generally preferred.

• Transmission Bandwidth (Carson's Rule): $$ B \approx 2(f_m + \Delta f) = 2(\beta_{FM} + 1) f_m $$
• Reliability: Output SNR exhibits a threshold effect. When the input carrier-to-noise ratio is above the threshold (typically $\approx 10$ dB), output SNR increases quadratically with bandwidth, trading bandwidth for improved noise immunity.

A typical FM receiver consists of the following cascade block diagram:

• Condition: $ |K_{FM} f(t)|_{\text{max}} = \Delta f_{\text{max}} \gg f_m $
• Bandwidth: $ B \approx 2 \Delta f_{\text{max}} $
• Modulation Gain: $$ G_{FM} \approx 6 \left( \frac{\Delta f_{\text{max}}}{f_m} \right)^3 \frac{E[f^2(t)]}{f^2_{\text{max}}(t)} $$

• Spectrum representation: $$ s(t) = A_c \sum_{n=-\infty}^{\infty} J_n(\beta_{FM}) \cos[(\omega_c + n \omega_m)t] $$
• Bandwidth: $ B \approx 2 \Delta f_{\text{max}} $
• Modulation Gain: $$ G_{FM} \approx 3 \beta_{FM}^3 $$

As the bandwidth is expanded ($B \uparrow$), the output SNR initially increases because the modulation gain $G$ increases. However, the total noise power entering the discriminator also increases ($N_i \uparrow$). When the input SNR falls below the threshold, click noise occurs, causing a rapid degradation in output SNR. This establishes the limit on bandwidth expansion for SNR improvement.

In FM demodulation, the output noise PSD is parabolic, rising quadratically with frequency ($S_n(f) \propto f^2$). To counteract this high-frequency noise, we use pre-emphasis at the transmitter to boost high-frequency signal components, and de-emphasis at the receiver to restore the signal spectrum while attenuating high-frequency noise.

• System layout: $$ f(t) \rightarrow \boxed{\text{Pre-emphasis Filter } H_{pe}(f)} \rightarrow \boxed{\text{FM Modulator}} \rightarrow \text{Channel} \rightarrow \boxed{\text{FM Demodulator}} \rightarrow \boxed{\text{De-emphasis Filter } H_{de}(f)} \rightarrow y(t) $$
• Typical filter transfer functions: $$ H_{pe}(f) \approx 1 + j\frac{f}{f_1}, \quad H_{de}(f) = \frac{1}{1 + j f/f_1} $$

• Self-Information: Measures the uncertainty of an event $x_i$: $$ I(x_i) = \log_2\left(\frac{1}{P(x_i)}\right) = -\log_2 P(x_i) \quad (\text{bits}) $$
• Joint Information: For two events $x_i$ and $y_j$: $$ I(x_i, y_j) = -\log_2 P(x_i, y_j) $$
• Mutual Information: Information shared between $x_i$ and $y_j$: $$ I(x_i; y_j) = \log_2\left(\frac{P(x_i \mid y_j)}{P(x_i)}\right) = \log_2\left(\frac{P(x_i, y_j)}{P(x_i)P(y_j)}\right) $$

• Source Entropy $H(X)$: The average self-information of source $X$: $$ H(X) = \sum_{i=1}^M P(x_i) \log_2\left(\frac{1}{P(x_i)}\right) \le \log_2 M $$
• Conditional Entropy $H(Y \mid X)$: The average uncertainty of $Y$ given $X$: $$ H(Y \mid X) = \sum_{i=1}^M \sum_{j=1}^N P(x_i, y_j) \log_2\left(\frac{1}{P(y_j \mid x_i)}\right) \le H(Y) $$
• Joint Entropy $H(X, Y)$: The total uncertainty of the joint system: $$ H(X, Y) = \sum_{i=1}^M \sum_{j=1}^N P(x_i, y_j) \log_2\left(\frac{1}{P(x_i, y_j)}\right) = H(X) + H(Y \mid X) $$
• Average Mutual Information $I(X; Y)$: The average information transmitted through a channel: $$ I(X; Y) = H(X) - H(X \mid Y) = H(Y) - H(Y \mid X) \ge 0 $$

• Capacity $C$ is the maximum mutual information over all input distributions $P(X)$: $$ C = \max_{P(X)} I(X; Y) = \max_{P(X)} \{ H(Y) - H(Y \mid X) \} \quad (\text{bits/symbol}) $$
• For symmetric channels, $H(Y \mid X)$ is independent of $P(X)$. Uniform input distribution maximizes $H(Y)$, giving: $$ C = \log_2 M - H(\text{row of transition matrix}) $$

If the transmission rate $R < C$, there exist error-correcting codes of block length $n$ such that the average block error probability $P_e$ satisfies:

where $E_r(R)$ is the error exponent. As $n \to \infty$, $P_e \to 0$. If $R > C$, error-free transmission is impossible.

• Differential (Relative) Entropy: $$ H(X) = -\int_{-\infty}^{\infty} p(x) \log_2 p(x) dx \quad (\text{bits/symbol}) $$
• Maximum Entropy for Bounded Range $[0, A]$: Achieved by uniform distribution: $$ H(X) \le \log_2 A $$
• Maximum Entropy for Given Variance $\sigma^2$: Achieved by Gaussian distribution: $$ H(X) \le \frac{1}{2} \log_2(2\pi e \sigma^2) $$
• Continuous Joint, Conditional, and Mutual Information: $$ H(X, Y) = -\int_{-\infty}^{\infty} \int_{-\infty}^{\infty} p(x, y) \log_2 p(x, y) dx dy $$ $$ H(Y \mid X) = -\int_{-\infty}^{\infty} \int_{-\infty}^{\infty} p(x, y) \log_2 p(y \mid x) dx dy \le H(Y) $$ $$ I(X; Y) = \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} p(x, y) \log_2 \left( \frac{p(x, y)}{p(x)p(y)} \right) dx dy = H(Y) - H(Y \mid X) \ge 0 $$

• Let the received signal be $ Y = X + N $, where noise $ N \sim \mathcal{N}(0, \sigma_N^2) $ is independent of signal $X$. $$ H(Y \mid X) = H(X+N \mid X) = H(N) = \frac{1}{2}\log_2(2\pi e \sigma_N^2) $$
• Mutual Information: $$ I(X; Y) = H(Y) - H(N) $$
• For average power constraints $E[X^2] \le S$, the output power is $E[Y^2] = S + N$. Since $H(Y)$ is maximized when $Y$ is Gaussian: $$ C = \max_{P(X)} I(X; Y) = \frac{1}{2} \log_2\left( 1 + \frac{S}{N} \right) \quad (\text{bits/symbol}) $$

For a bandlimited channel of bandwidth $B$ (Hz) sampled at the Nyquist rate $2B$ samples/second, the capacity in bits/second is:

where $N_0$ is the single-sided noise power spectral density ($N = N_0 B$).

• Infinite Bandwidth Limit ($B \to \infty$): $$ C_{\infty} = \lim_{B\to\infty} B \log_2\left(1 + \frac{S}{N_0 B}\right) = \frac{S}{N_0} \log_2 e \approx 1.44 \frac{S}{N_0} $$
• Shannon Limit in terms of $E_b/N_0$:

  Since the transmission rate $R_b \le C_t$ and signal power is $S = R_b E_b$:

  $$ \frac{E_b}{N_0} \ge \ln 2 \approx -1.6 \text{ dB} $$

  This represents the absolute limit below which error-free communication is mathematically impossible.

• Impulse/Natural Sampling: $$ x_s(t) = x(t) \cdot \sum_{n=-\infty}^{\infty} \delta(t - n T_s) $$
• Flat-Top Sampling (Zero-Order Hold): $$ x_s(t) = \left[ x(t) \cdot \sum_{n=-\infty}^{\infty} \delta(t - n T_s) \right] * \text{rect}\left(\frac{t}{T_0}\right) $$

  Introduces aperture distortion, requiring an equalization filter.

To quantize a continuous source $x$ with probability density $p_x(x)$ into $L$ levels $\{y_k\}$ with boundaries $\{x_k\}$, we minimize the mean squared error (distortion $\sigma_q^2$):

• Optimal Boundaries $x_k$ (for given $\{y_k\}$): $$ \frac{\partial \sigma_q^2}{\partial x_k} = 0 \Rightarrow x_k = \frac{1}{2}(y_k + y_{k+1}) $$
• Optimal Reconstruction Levels $y_k$ (for given $\{x_k\}$): $$ \frac{\partial \sigma_q^2}{\partial y_k} = 0 \Rightarrow y_k = \frac{\int_{x_{k-1}}^{x_k} x p_x(x) dx}{\int_{x_{k-1}}^{x_k} p_x(x) dx} \quad (\text{centroid of the partition interval}) $$

• Spacing: $\Delta = \frac{2V}{L}$.
• Distortion (Quantization noise variance): $$ \sigma_q^2 = \frac{\Delta^2}{12} = \frac{V^2}{3L^2} $$
• Signal-to-Quantization Noise Ratio (SQNR): $$ \text{SQNR} = \frac{\sigma_x^2}{\sigma_q^2} = L^2 = 2^{2B} \approx 6.02 B \text{ dB} $$ where $B = \log_2 L$ is the number of bits.

• Density: $ p_x(x) = \frac{1}{\sqrt{2}\sigma_x} e^{-\frac{\sqrt{2}|x|}{\sigma_x}} $
• Distortion for large $L$: $$ \sigma_q^2 \approx \frac{2\sigma_x^2}{3L^2} $$

• Using uniform quantization with dynamic range parameter $D = V/\sigma_x$: $$ \text{SQNR (dB)} \approx 6.02 B + 4.77 - 20\log_{10} D $$

Used to maintain a nearly constant SQNR over a wide dynamic range (e.g., $\mu$-law or $A$-law):

• Unipolar NRZ (Non-Return-to-Zero):
  • "1" $\to$ pulse of amplitude $A$; "0" $\to$ 0. Has DC offset, poor clock recovery.
• Polar NRZ:
  • "1" $\to$ $+A$; "0" $\to$ $-A$. Better noise margin, no DC offset if symmetric.
• Bipolar NRZ (Alternate Mark Inversion - AMI):
  • "1" alternates between $+A$ and $-A$; "0" $\to$ 0. Zero DC component, allows error detection.
• Return-to-Zero (RZ) codes:
  • Unipolar RZ: "1" $\to$ pulse of amplitude $A$ for half the bit interval, then returns to 0. Excellent clock recovery.
  • Polar RZ: "1" $\to$ $+A$ (half-interval); "0" $\to$ $-A$ (half-interval).
• Manchester (Split-Phase) Coding:
  • "1" $\to$ high-to-low transition; "0" $\to$ low-to-high transition. Guaranteed transition at center of every bit interval, zero DC component.

• Codeword: $ \bm{c} = \bm{m} \bm{G} $, where $ \bm{G} = [\bm{I}_k \mid \bm{P}] $ is the generator matrix.
• Parity check matrix: $ \bm{H} = [-\bm{P}^T \mid \bm{I}_{n-k}] $, satisfying $ \bm{G} \bm{H}^T = \bm{0} $.
• Syndrome decoding: $ \bm{s} = \bm{r} \bm{H}^T = \bm{e} \bm{H}^T $. If $ \bm{s} = \bm{0} $, no detectable errors; otherwise, $ \bm{s} $ maps to the error vector $ \bm{e} $.
• Hamming codes: $(2^m-1, 2^m-1-m)$ cyclic/linear codes with $d_{\text{min}} = 3$, correcting single errors.

• Codewords are represented as polynomials: $ c(D) = m(D) g(D) $, where $g(D)$ is the generator polynomial of degree $n-k$.
• Systematic form: $$ c(D) = D^{n-k} m(D) + [D^{n-k} m(D) \bmod g(D)] $$
• Syndrome polynomial: $ s(D) = r(D) \bmod g(D) $.

A generalized class of cyclic codes for multiple error corrections. Decoded using algebraic methods like the Berlekamp-Massey algorithm.

Scrambles symbol sequences across a matrix of depth $M$ to convert burst errors caused by fading into isolated random errors, which are easily corrected by block codes.

• Parameters: $(n, k, K)$, where $K$ is the constraint length. Codewords depend on current and past bits.
• Maximum Likelihood decoding is implemented using the Viterbi Algorithm on the code trellis.

To transmit at the Nyquist rate $2B$ symbols/second over a physical channel of bandwidth $B$ without ISI, duobinary coding introduces controlled correlation between adjacent symbols. To prevent error propagation, we pre-code the input data $\{d_n\}$ (where $d_n \in \{0, 1, \dots, L-1\}$):

The transmitted sequence $\{c_n\}$ is:

At the receiver, the data is decoded without error propagation using:

• Effectiveness: Compresses the required bandwidth, allowing signaling at the theoretical Nyquist rate over a practical channel.
• Limitations: Requires a higher SNR because the received signal has $2L-1$ levels instead of $L$. It is also sensitive to phase offset and timing jitter.

• Baseband Bipolar PSD: $$ \Phi_s(f) = A^2 T_s \left( \frac{\sin(\pi f T_s)}{\pi f T_s} \right)^2 $$
• M-ary Frequency Shift Keying (MFSK):

  Orthogonality condition requires carrier spacing $\Delta f = \frac{1}{2T_s}$.

  $$ \Phi_{FSK}(f) \propto \sum_{m=1}^M \left[ \Phi_s(f - f_m) + \Phi_s(f + f_m) \right], \quad B \approx 2B_0 + (M-1)\Delta f $$
• M-ary Amplitude Shift Keying (MASK) / MPSK: $$ \Phi_{PSK}(f) \propto \Phi_s(f - f_c) + \Phi_s(f + f_c) $$
• Minimum Shift Keying (MSK):

  A continuous-phase FSK (CPFSK) scheme with modulation index $h = 0.5$ ($\Delta f = \frac{1}{2 T_s}$). Constant envelope, zero phase transitions, and rapid sidelobe decay.

• BPSK / QPSK: $$ P_b = Q\left( \sqrt{\frac{2E_b}{N_0}} \right) $$
• MASK: $$ P_s = \frac{2(M-1)}{M} Q\left( \sqrt{\frac{6\log_2 M}{M^2 - 1} \frac{E_b}{N_0}} \right) $$
• MFSK (Coherent): $$ P_s \le (M-1) Q\left( \sqrt{\frac{E_s}{N_0}} \right) $$
• MQAM: $$ P_s \approx 4\left( 1 - \frac{1}{\sqrt{M}} \right) Q\left( \sqrt{\frac{3}{M-1} \frac{E_s}{N_0}} \right) $$
• MPSK: $$ P_s \approx 2 Q\left( \sqrt{\frac{2E_s}{N_0}} \sin\left(\frac{\pi}{M}\right) \right) $$

• Non-Coherent FSK (BFSK): $$ P_b = \frac{1}{2} e^{-\frac{E_b}{2N_0}} $$
• Non-Coherent DPSK: $$ P_b = \frac{1}{2} e^{-\frac{E_b}{N_0}} $$

Let received waveform be $ x(t) = s_i(t) + n(t) $ for $t \in [0, T_s]$ in AWGN. The Likelihood function is:

• Decision Rule for Equiprobable Symbols (Minimum Distance): $$ \hat{m} = \arg\min_i \int_0^{T_s} [x(t) - s_i(t)]^2 dt $$
• Decision Rule for Equal Energy and Equiprobable Symbols (Correlation Receiver): $$ \hat{m} = \arg\max_i \int_0^{T_s} x(t) s_i(t) dt $$

The linear filter that maximizes output SNR at sampling instant $t = T_s$ is matched to the signal $s_i(t)$:

• Maximum Output SNR: $$ \text{SNR}_{\text{max}} = \frac{2 E_s}{N_0} $$