Computer-aided estimation of brain connectivity aims to reveal information propagation in brain automatically, which has great potential in clinical applications, e.g. epilepsy foci diagnosis. Granger causality is an effective tool for directional connection analysis in multivariate time series. However, most existing methods based on Granger causality assume fixed time lags in information transmission, while the propagation delay between brain signals is usually changing constantly. We propose a Granger causality estimator based on the recurrent neural network, called RNN-GC, to deal with the multivariate brain connectivity detection problem. Our model takes input of time series signals with arbitrary length of transmission time lags and learns the information flow from the data using the gated RNN model, i.e. long short-term memory (LSTM). The LSTM model can sequentially update the gates in memory cells to determine how many preceding points should be considered for prediction. Therefore, the LSTM-based RNN-GC estimator works well on varying length time lags and shows effectiveness even on very long transmission delays. Results: Experiments are carried out in comparison with other methods using both simulation data and epileptic electroencephalography (iEEG) signals. The RNN-GC estimator achieves superior performance in brain connectivity estimation and shows robustness in modeling multivariate connections with varying length time lags. Conclusion: The RNN-GC method is capable of modeling nonlinear and varying-length lagged information transmission, and effective in directional brain connectivity estimation. The proposed method is promising to serve as a robust brain connection analysis tool in clinical applications.