Reefs are biogenic structures that result in three-dimensional accumulations of calcium carbonate. Over geological timescales, a positive balance between the production and accumulation of calcium carbonate versus erosional and off-reef transport processes maintains positive net accretion on reefs. Yet, how ecological processes occurring over decadal timescales translate to the accumulation of geological structures is poorly understood, in part due to a lack of studies with detailed time-constrained chronologies of reef accretion over decades to centuries. Here, we combined ecological surveys of living reefs with palaeoecological reconstructions and high-precision radiometric (U-Th) age-dating of fossil reefs represented in both reef sediment cores and surficial dead in situ corals, to reconstruct the history of community composition and carbonate accumulation across the central and southern Saudi Arabian Red Sea throughout the late Holocene. We found that reefs were primarily comprised of thermally tolerant massive Porites colonies, creating a consolidated coral framework, with unconsolidated branching coral rubble accumulating among massive corals on shallow (5–8 m depth) exposed (windward), and gently sloping reef slopes. These unconsolidated reef rubble fields were formed primarily from ex situ Acropora and Pocillopora coral fragments, infilled post deposition within a sedimentary matrix. Bayesian age-depth models revealed a process of punctuated deposition of post-mortem coral fragments transported from adjacent reef environments. That a large portion of Saudi Arabian Red Sea reef slopes is driven by allochthonous deposition (transportation) has important implications for modeling carbonate budgets and reef growth. In addition, a multi-decadal lag exists between the time of death for branching in situ coral and incorporation into the unconsolidated reef rubble. This indicates that recent climate related degradation in the 21st century has not had an immediately negative effect on reef building processes affecting a large portion of the reef area in the Saudi Arabian Red Sea.