I have cavity wall insulation. Yes it stops heat from the inside escaping. But it also stops heat from the outside getting in. On many days I have gone outside and thought that it is warmer outside than inside, because the insulation has not allowed the warmth outside to get through to the inside which I am paying to heat
However, you might welcome the insulation on a very hot summer day. Ideally, a house should have the windows (with blinds or shutters) on the south side to catch the sun when available and wanted. My previous house had the main windows west-facing and got too hot on a summer evening and consequently my current house has south-facing windows.
Unfortunately, planners and architects fail to properly consider welcome and unwelcome solar gain and put standard house designs on a map with whatever orientation and layout looks nice on the map. Trees in the wrong places area also a problem.