I will introduce the recent findings of my research group about the integration of neural networks in the analysis of functional data. First, we propose a new layer architecture, a functional output layer, allowing us to use neural networks to output functional responses. This can be used to solve the function-on-scalar regression problem in a non-linear way. Second, we propose a concept for functional weights which can project functional data to a scalar representation. By combining these weights with the functional output layer previously established, we create a functional autoencoder. This model is both able to return a finite representation of functional data and a smooth representation of that data. Finally, inspired by the successes of convolutional neural networks, we propose to address the connections in the functional output layer in a meaningful way to both reduce the number of parameters drastically and to capture the temporal correlation of functional data.