Earthquake-induced structural wastes and other factors causing destruction, on one hand, and issues regarding the damaging effects of acidic environments, on the other, comprise some of the most challenging concerns of humanity, thus eliminating these challenges is one of the major goals in today’s construction industry. To face this challenge, this research set out to investigate the compressive behavior of concrete containing fine recycled refractory brick aggregate together with calcium aluminate cement (CAC) and polyvinyl alcohol (PVA) fibers under an acidic environment. In order to achieve this objective, 96 fine aggregate concrete specimens were exposed to 5% sulfuric acid attack for periods of 0, 7, 21, and 63 days, then they were subjected to various experiments. First, to determine the extent of the specimens’ corrosion, the weight variations test was conducted, then, the ultrasonic pulse velocity (UPV) test, a non-destructive concrete test to obtain the level of porosity and density, was carried out, and at the end of the experimental phase, the specimens were subjected to compressive testing after being visually inspected to examine the corrosion level. Finally, using response surface method (RSM), an optimum solution for the design parameters was presented by maximizing the compressive strength of fibrous concrete as well as making the concrete mix cost-effective. Generally, the results indicate that the specimens containing calcium aluminate cement together with PVA fibers demonstrate proper mechanical properties in terms of corrosion control against acid attack, while the specimens containing fine refractory brick aggregate showed a rather unsatisfactory performance in this regard.