Enhancing the resilience of corals to rising temperatures is now a matter of urgency, leading to growing efforts to explore the use of heat tolerant symbiont species to improve their thermal resilience. The notion that adaptive traits can be retained by transferring the symbionts alone, however, challenges the holobiont concept, a fundamental paradigm in coral research. Holobiont traits are products of a specific community (holobiont) and all its co-evolutionary and local adaptations, which might limit the retention or transference of holobiont traits by exchanging only one partner. Here we evaluate how interchanging partners affect the short- and long-term performance of holobionts under heat stress using clonal lineages of the cnidarian model system Aiptasia (host and Symbiodiniaceae strains) originating from distinct thermal environments. Our results show that holobionts from more thermally variable environments have higher plasticity to heat stress, but this resilience could not be transferred to other host genotypes through the exchange of symbionts. Importantly, our findings highlight the role of the host in determining holobiont productivity in response to thermal stress and indicate that local adaptations of holobionts will likely limit the efficacy of interchanging unfamiliar compartments to enhance thermal tolerance.