I can't speak for the harm delaying a CVT fluid change might or might not do but for the past 20 years I've serviced my own cars and change all fluids on schedule. For instance, the Mercedes 190E that I recently sold only travelled around a thousand miles last year, but still had new engine oil and filter as it's recommended annually regardless of mileage.
Personally, I'd keep to the manufacturers servicing schedules. Better to over-service than under-service in my view.