The concrete materials in structures are most likely to experience elevated temperatures during their service period due to fire. Therefore, the relative properties of concrete after exposure to fire are of particular interest in the serviceability of buildings and the safety of inhabitants. In this study, a comprehensive experimental program was performed regarding the use of recycled aggregate produced by crushing refractory bricks. To do so, ten mix designs were employed to prepare 210 specimens with the replacement ratios of 0, 25, 50, 75, and 100% of refractory brick fine aggregate instead of natural sand. The specimens were prepared in the two classes, namely the specimens containing ordinary Portland cement and the ones containing calcium aluminate cement. The physico-mechanical properties of the concrete specimens including compressive strength, modulus of elasticity, and loss of weight of the concrete after being exposed to the temperatures of 110, 200, 400, 600, 800, and 1000 C, together with the porosity, water absorption, and density of the concrete prior to experiencing fire were investigated. The results showed that the refractory brick aggregate and aluminate cement improve the concrete residual strength to twice as much beyond the temperature of 800 C. Furthermore, the effect of using refractory brick fine aggregate together with aluminate cement on the improvement of concrete modulus of elasticity under fire was not significant