Preheat the oven to 400 degrees F. Line a baking sheet with parchment paper. Toss the peaches with the olive oil and brown sugar. Roast for 15 minutes, then toss and roast for 10 minutes more until caramely. Remove the peaches and let them cool.
In a large bowl, whisk together the flour, sugar, lavender, baking powder, soda and salt. Add the butter and using your fingers, a fork or a pastry blender mix it together and combine until the butter forms coarse crumbs. I usually take a full 2-3 minutes and do this with my fingers. Add almost all of the peach slices, reserving about 1/3 cup of slices and a little juice. Toss the slices with the flour and butter crumbs.
Make a well in the center and add in buttermilk, vanilla extract and lavender extract (if using it). Mix with a large spoon until a dough forms and comes together – it will be sticky! Just make sure your hands are floured and so is your work surface. Divide the dough in half and pat it into two 6-inch circles on the floured surface or one larger circle. Cut the circles into 6 or 8 wedges. Brush each with some heavy cream, then sprinkle on a bit of raw sugar. Place the scones on a baking sheet and bake for 10 to 12 minutes – or about 15ish minutes for larger scones. Top with the glaze before serving.
To make the glaze, add the remaining peaches and syrupy juice to a food processor. Blend until pureed then pour the mixture into a bowl. Stir in the powdered sugar. You want a syrupy glaze, so you might need to add more sugar 1/4 cup at a time to thicken it. If it becomes too thick, whisk in 1 teaspoon of milk or water.