From an exercise physiology point of view, the length of the rests is designed for your maximum training success. For intense exercises with few repetitions, you can expect a longer rest than for less intense exercises with many repetitions. After a power-intensive exercise, your central nervous system and muscles need more time to recover for the next set.
Therefore, try to stick to the length of the rest. If you need to take a longer break, you can of course take it. However, try to reduce the break times in the following training sessions to the suggested maximum time.
If the breaks are too long and you feel ready for the next set, you can skip the break and start with the next exercise.