Free View Point Imaging over Wireless Mesh Networks

Having a signal representation comprising multiple views of the same scene is increasingly becoming popular due to emerging applications such as free view point TV and immersive 3D gaming, to name a few. At the same time working with such signal sets raises a new set of challenges in terms of signal processing, compression, and communication that need to be met. An additional degree of difficulty is introduced by the frequent requirement for operating multi-view setups in a distributed or at least a semi-independent fashion. The present project investigates the problem of efficient delivery of multiple correlated media streams over overlay or wireless mesh networks. The streams are transmitted to heterogeneous clients, which subscribe to several streams or generate new sequences from multiple correlated streams. The project aims to develop the theoretical foundations for rate-distortion efficient delivery of correlated media streams (see Figure 1). It explores resource allocation across several stages of the multiple view system. Specifically, the rate allocation to each of the views under a constraint on the overall bit budget is important as it will determine the overall performance of the system. Moreover, the delivery of the multi-view signal to its audience necessitates a careful design of the distribution of transmission resources across the multiple views, as it can also affect the performance of the system as a whole. Finally, the statistics of the audience in terms of view preference as well as their capabilities in terms of computational power and network connectivity are also taken into consideration for a truly optimized end-to-end multiple view experience.



Figure 1. Multiple view imaging system: R-D characteristic of synthetic source VS depends on the R-D characteristics of sources V1 and V2 and on the relative positions of V1, V2, and VS as expressed through θ and φ.

As part of the project, we investigate sample scenarios, such as free viewpoint rendering of video information or iterative walkthrough (see Figure 2). In particular, we study models of multiview video representation that can be used toward efficient streaming solutions for free viewpoint TV. Furthermore, we design algorithms for adaptive and seamless rendering of successive viewpoints. Finally, a special attention is placed on distributed coding and streaming strategies in order to enable future streaming applications in large scale ad hoc scenarios with user-generated content.



Figure 2. Virtual tour of a museum. Trajectory of visitor in dashed black. Camera selection order sequentially numbered and in dotted light gray.