Using multiobjective optimization to map the entropy region

Mapping the structure of the entropy region of at least four jointly distributed random variables is an important open problem. Even partial knowledge about this region has far reaching consequences in other areas in mathematics, like information theory, cryptography, probability theory and combinatorics. Presently, the only known method of exploring the entropy region is, or equivalent to, the one of Zhang and Yeung from 1998. Using some non-trivial properties of the entropy function, their method is transformed to solving high dimensional linear multiobjective optimization problems. Benson’s outer approximation algorithm is a fundamental tool for solving such optimization problems. An improved version of Benson’s algorithm is presented, which requires solving one scalar linear program in each iteration rather than two or three as in previous versions. During the algorithm design, special care is taken for numerical stability. The implemented algorithm is used to verify previous statements about the entropy region, as well as to explore it further. Experimental results demonstrate the viability of the improved Benson’s algorithm for determining the extremal set of medium-sized numerically ill-posed optimization problems. With larger problem sizes, two limitations of Benson’s algorithm is observed: the inefficiency of the scalar LP solver, and the unexpectedly large number of intermediate vertices.

[1]  Walter B. McRae,et al.  An Algorithm for the Extreme Rays of a Pointed Convex Polyhedral Cone , 1973, SIAM J. Comput..

[2]  Benjamin A. Burton,et al.  Projective geometry and the outer approximation algorithm for multiobjective linear programming , 2010, 1006.3085.

[3]  Randall Dougherty,et al.  Non-Shannon Information Inequalities in Four Random Variables , 2011, ArXiv.

[4]  T. H. Chan,et al.  Balanced information inequalities , 2003, IEEE Trans. Inf. Theory.

[5]  Frantisek Matús,et al.  Infinitely Many Information Inequalities , 2007, 2007 IEEE International Symposium on Information Theory.

[6]  Nikolai K. Vereshchagin,et al.  A new class of non-Shannon-type inequalities for entropies , 2002, Commun. Inf. Syst..

[7]  Yeow Meng Chee,et al.  Coding and Cryptology, Second International Workshop, IWCC 2009, Zhangjiajie, China, June 1-5, 2009. Proceedings , 2009, IWCC.

[8]  Milan Studeny.  Probabilistic Conditional Independence Structures: With 42 Illustrations (Information Science and Statistics) , 2004 .

[9]  Prasad Tetali,et al.  Information-theoretic inequalities in additive combinatorics , 2010, 2010 IEEE Information Theory Workshop on Information Theory (ITW 2010, Cairo).

[10]  Tarik Kaced.  Equivalence of two proof techniques for non-shannon-type inequalities , 2013, 2013 IEEE International Symposium on Information Theory.

[11]  Andreas H. Hamel,et al.  Benson type algorithms for linear vector optimization and applications , 2013, J. Glob. Optim..

[12]  Demetres Christofides,et al.  Multiple unicasts, graph guessing games, and non-Shannon inequalities , 2013, 2013 International Symposium on Network Coding (NetCod).

[13]  Raymond W. Yeung,et al.  A First Course in Information Theory (Information Technology: Transmission, Processing and Storage) , 2006 .

[14]  Raimund Seidel,et al.  How Good Are Convex Hull Algorithms? , 1997, Comput. Geom..

[15]  C.J.H. Mann,et al.  Probabilistic Conditional Independence Structures , 2005 .

[16]  Andreas Löhne,et al.  Geometric Duality in Multiple Objective Linear Programming , 2008, SIAM J. Optim..

[17]  Raymond W. Yeung,et al.  A First Course in Information Theory , 2002 .

[18]  Harold P. Benson,et al.  An Outer Approximation Algorithm for Generating All Efficient Extreme Points in the Outcome Set of a Multiple Objective Linear Programming Problem , 1998, J. Glob. Optim..

[19]  Zhen Zhang,et al.  On Characterization of Entropy Function via Information Inequalities , 1998, IEEE Trans. Inf. Theory.

[20]  Rahim Tafazolli,et al.  Network Coding Theory: A Survey , 2013, IEEE Communications Surveys & Tutorials.

[21]  Terence Chan.  Recent Progresses in Characterising Information Inequalities , 2011, Entropy.

[22]  Lizhen Shao,et al.  A dual variant of Benson’s “outer approximation algorithm” for multiple objective linear programming , 2012, J. Glob. Optim..

[23]  F. Matús,et al.  Two Constructions on Limits of Entropy Functions , 2007, IEEE Transactions on Information Theory.

[24]  Milan Studený,et al.  Probabilistic conditional independence structures , 2006, Information science and statistics.

[25]  Amos Beimel,et al.  Secret-Sharing Schemes: A Survey , 2011, IWCC.

[26]  David Avis,et al.  On the complexity of vertex and facet enumeration for convex polytopes , 1998 .

[27]  František Matúš,et al.  Conditional Independences among Four Random Variables III: Final Conclusion , 1999, Combinatorics, probability & computing.

[28]  László Csirmaz.  Book Inequalities , 2014, IEEE Transactions on Information Theory.

[29]  Komei Fukuda,et al.  Double Description Method Revisited , 1995, Combinatorics and Computer Science.

[30]  S. Weber,et al.  Relationships among bounds for the region of entropic vectors in four variables , 2010, 2010 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton).