In this work, we introduce Gumbel Graph Network (GGN), a model-free, data-driven deep learning framework to accomplish network reconstruction and dynamics simulation. Our method can reconstruct many kind of dynamics such as continuous, discrete, and even binary.