This thesis considers a full-duplex MIMO relay which amplifies and forwards the received signals, between a source and a destination that do not a have line of sight. Full-duplex mode raises the problem of self-interference. Though all the links in the system undergo frequency flat fading, the end-to-end effective channel is frequency selective. This is due to the imperfect cancellation of the self-interference at the relay and this residual self-interference acts as intersymbol interference at the destination which is treated by equalization. This also leads to complications in form of recursive equations to determine the input-output relationship of the system. This also leads to complications in the form of recursive equations to determine the input-output relationship of the system.

To overcome this, a signal flow graph approach using Mason's gain formula is proposed, where the effective channel is analyzed with keen notice to every loop and path the signal traverses. This gives a clear understanding and awareness about the orders of the polynomials involved in the transfer function, from which desired conclusions can be drawn. But the complexity of Mason's gain formula increases with the number of antennas at relay which can be overcome by the proposed linear algebraic method. Input-output relationship derived using simple concepts of linear algebra can be generalized to any number of antennas and the computation complexity is comparatively very low.

For a full-duplex amplify-and-forward MIMO relay system, assuming equalization at the destination, new mechanisms have been implemented at the relay that can compensate the effect of residual self-interference namely equal-gain transmission and antenna selection. Though equal-gain transmission does not perform better than the maximal ratio transmission, a trade-off can be made between performance and implementation complexity. Using the proposed antenna selection strategy, one pair of transmit-receive antennas at the relay is selected based on four selection criteria discussed. Outage probability analysis is performed for all the strategies presented and detailed comparison has been established. Considering minimum mean-squared error decision feedback equalizer at the destination, a bound on the outage probability has been obtained for the antenna selection case and is used for comparisons. A cross-over point is observed while comparing the outage probabilities of equal-gain transmission and antenna selection techniques, as the signal-to-noise ratio increases and from that point antenna selection outperforms equal-gain transmission and this is explained by the fact of reduced residual self-interference in antenna selection method.]]>