For the frangipane, cream the butter and sugar together in a large, deep bowl using the beater attachment, then beat in the eggs. Finely grate in the orange and lemon zest, then fold through with the almond extract, ground almonds and flour.
Roll out the pastry onto a floured surface to fit a 25cm x 35cm lined baking tray, going up the sides slightly. Bake in the middle of the oven for 15 minutes, or until lightly golden. Remove the tray and lower the heat to 160ºC/325ºF/gas 3.
Spread the frangipane over the pastry and stud with the plums and flaked almonds. Bake for 30 minutes or until sticky and golden, then remove and leave to cool slightly.