Objective: Diabetic macular edema (DME) and retinal vein occlusion (RVO) are the leading causes of visual impairments across the world. Vascular endothelial growth factor (VEGF) stimulates breakdown of blood-retinal barrier that causes accumulation of fluid within macula. Anti-VEGF therapy is the first-line treatment for both the diseases; however, the degree of response varies for individual patients. The main objective of this work was to identify the (i) texture-based radiomics features within individual fluid and retinal tissue compartments of baseline spectral-domain optical coherence tomography (SD-OCT) images and (ii) the specific spatial compartments that contribute most pertinent features for predicting therapeutic response. Methods: A total of 962 texture-based radiomics features were extracted from each of the fluid and retinal tissue compartments of OCT images, obtained from the PERMEATE study. Top-performing features selected from the consensus of different feature selection methods were evaluated in conjunction with four different machine learning classifiers: Linear Discriminant Analysis (LDA), Quadratic Discriminant Analysis (QDA), Random Forest (RF), and Support Vector Machine (SVM) in a cross-validated approach to distinguish eyes tolerating extended interval dosing (non-rebounders) and those requiring more frequent dosing (rebounders). Results: Combination of fluid and retinal tissue features yielded a cross-validated area under receiver operating characteristic curve (AUC) of 0.78±0.08 in distinguishing rebounders from non-rebounders. Conclusions: This study revealed that the texture-based radiomics features pertaining to IRF subcompartment were most discriminating between rebounders and non-rebounders to anti-VEGF therapy. Clinical Impact: With further validation, OCT-based imaging biomarkers could be used for treatment management of DME patients.