Skeletal bone age assessment is a common clinical practice to investigate endocrinology, genetic and growth disorders of children. However, clinical interpretation and bone age analyses are time-consuming, labor intensive and often subject to inter-observer variability. This advocates the need of fully automated method for bone age assessment. We propose a regression convolutional neural network (CNN) to automatically assess the pediatric bone age from hand radiograph. Our network is specifically trained to place more attention to those bone age related regions in the X-ray images. Specifically, we first adopt the attention module to process all images and generate the coarse/fine attention maps as inputs for the regression network. Then, the regression CNN follows the supervision of the dynamic attention loss during training, thus it can estimate the bone age of the hard (or “outlier”) images more accurately. Specially, we evaluate our method on 12,390 subjects in total, ranking our study on the largest scale world-wide bone age dataset in children from our cooperative hospital. The experimental results show that our method achieves an average discrepancy of 5.2-5.3 months between clinical and automatic bone age evaluations on large datasets. Meanwhile, the entire pipeline can process a subject within ~1.5s, which is highly efficient for clinical usage. In conclusion, we propose a fully automated deep learning solution to process X-ray images of the hand for bone age assessment, with the accuracy comparable to human experts but with much better efficiency.