A reduced-memory multi-layer perceptron with systematic network weights generated and trained through distribution hyper-parameters

Abstract

A multi-layer perceptron (MLP) consists of a number of forward-connected weights (W_ijk_) from each feeding layer node (n_ij_) to the many initially equivalent nodes (n_i+1,k_) in the next layer. Exact a priori order and search space of these weights (W_ijk_) is random and prone to redundancy, irreproducibility and non-optimality. We demonstrate that a weight subspace (W_ijk_ for each i and j), generated systematically using a statistical distribution with predefined breakpoints and Genetic algorithm-trained hyper-parameters substantially reduces the computational complexity of an MLP and produces comparable or better performance than similarly trained equivalent models with fully defined weights. This distribution based neural network (DBNN) provides a novel framework to create very large neural network models with currently prohibitive memory requirements.