Objective: Colorectal cancer (CRC) patients respond differently to treatments and are sub-classified by different approaches. We evaluated a deep learning model, which adopted endoscopic knowledge learnt from AI-doscopist, to characterise CRC patients by histopathological features. Results: Data of 461 patients were collected from TCGA-COAD database. The proposed framework was able to 1) differentiate tumour from normal tissues with an Area Under Receiver Operating Characteristic curve (AUROC) of 0.97; 2) identify certain gene mutations (MYH9, TP53) with an AUROC > 0.75; 3) classify CMS2 and CMS4 better than the other subtypes; and 4) demonstrate the generalizability of predicting KRAS mutants in an external cohort. Conclusions: Artificial intelligent can be used for on-site patient classification. Although KRAS mutants were commonly associated with therapeutic resistance and poor prognosis, subjects with predicted KRAS mutants in this study have a higher survival rate in 30 months after diagnoses.