Atypical observations or mild outliers are commonly encountered in real data. Modelling these together with typical observations can substantially impact the model fit. However, identifying such atypical observations can be challenging in a clustering framework. Various approaches for identifying atypical observations are available in a clustering framework for continuous data but there is a dearth of approaches for multivariate discrete data. Here, we propose a contaminated mixture model that identifies atypical observations in a clustering framework and mitigates their influence in the model fit. For this, we utilize a mixture of multivariate Poisson-lognormal (MPLN) models that accounts for both correlation among the variables and over-dispersion through a hierarchical structure. Atypical observations are identified in the latent space using a contaminated mixture framework. Parameter estimation is done using a variational variant of the expectation maximization algorithm.