In this paper, the compressive stress-strain behavior of recycled aggregate concrete with different replacement levels of conventional fine aggregate by recycled particles obtained from used refractory bricks (0, 25, 50, 75, and 100% replacement level by volume) is investigated in two groups, one containing ordinary portland cement and the other containing calcium aluminate cement, after exposure to elevated temperatures (110, 200, 400, 600, 800, and 1,000°C). For this purpose, the parameters that affect the compressive behavior of recycled refractory brick aggregate concrete (RRBC), including compressive strength, strain at peak stress, modulus of elasticity, and toughness, as well as the shape of the stress-strain curve following exposure to elevated temperatures were evaluated and compared to those predicted by international codes. Finally, a stress-strain model to predict the compressive behavior of RRBC at elevated temperatures was proposed. The results indicate that a significant degradation occurs for most of the mechanical properties of the ordinary cement containing concrete at 400°C and for the aluminate cement-containing concrete at 110°C. Furthermore, higher contents of refractory brick fine aggregate improve the concrete compressive behavior at higher temperatures.