Cut the cake into small cubes. If desired, toss blueberries with sugar and let sit for 10 minutes to enhance sweetness.
In a large trifle dish or individual serving cups, layer the ingredients starting with a layer of cake cubes.
Spoon lemon curd over the cake layer.
Sprinkle fresh blueberries on top of the lemon curd.
Add a layer of whipped cream to complete the first round.
Continue layering cake, lemon curd, blueberries, and whipped cream until the dish is full.
Top with a final dollop of whipped cream, fresh blueberries, and lemon zest if desired.
Refrigerate for at least 1 hour before serving to allow the flavors to meld together.