We introduce a generative model based on training deep architectures. The model consists of K networks that are trained together to learn the underlying distribution of a given data set. The process starts with dividing the input data into K clusters and feeding each into a separate network. We use an EM-like algorithm to train the networks together and update the clusters of the data. We call this model Mixture of Networks. The provided model is a platform that can be used for any deep structure and be trained by any conventional objective function for distribution modeling. As the components of the model are neural networks, it has a high capability in characterizing complicated data distributions as well as clustering data. We apply the algorithm on MNIST handwritten digits and Yale faces datasets. The model can learn the distribution of these data sets. One can sample new data points from these distributions that look like a real handwritten digit or a real face. We also demonstrate the clustering ability of the model using some real-world and toy examples.