As a senile chronic, progressive and currently incurable disease, dementia has an enormous impact on society and life quality of the elderly. The development of teleoperation technology has changed the traditional way of care delivery and brought a variety of novel applications for dementia care. In this paper, a telerobotic system is presented which gives the caregivers the capability of assisting dementia elderly remotely. The proposed system is composed of a dual-arm collaborative robot (YuMi) and a wearable motion capture device. The communication architecture is achieved by the robot operation system (ROS). The position-orientation data of the operator’s hand are obtained and used to control the YuMi robot. Besides, a path-constrained mapping method is designed for motion trajectory tracking between the robot and the operator in the progress of teleoperation. Meanwhile, corresponding experiments are conducted to verify the performance of the trajectory tracking using the path-constrained mapping method. Results show that the position tracking deviation between the trajectory of the operator and the robot measured by dynamic time warping distance is 1.05 mm at the sampling frequency of 7.5 Hz. Moreover, the practicability of the proposed system was verified by teleoperating the YuMi robot to pick up a medicine bottle and further demonstrated by assisting an elderly woman in picking up a cup remotely. The proposed telerobotic system has potential utility for improving the life quality of dementia elderly and the care effect of their caregivers.