In general people have the "right to roam" in the countryside, but not in your garden nor to pick your flowers.
I have seen people parking on mass and all jumping over a gate and picking "wild" flowers in a field and also I've seen several little old ladies run into fields to pick a solitary mushroom. I think this is accepted practice at least if not allowed in law. I expect this boils down to the definition or "garden" and "wild".
My suggestion would be to put up notices on the fence saying something like "jardin priveé" or "espace gardeé" and make the field look more like a garden. In practice however, if people have taken the flowers for a number of years they will expect to be able to do it again and so short of digging them up or putting dogs in the field, I expect you must be pragmatic about it.