Farmers have a lot of rights over land that they use to farm and it can be very difficult to get land returned, however taking the hay from a field is not considered working the land only cleaning it. Using the land to graze livestock is a different matter.
You should meet with the farmer and make it clear that
- a) you own the land (be sure of this and have documentation if possible - be sure you understand where your parcels are on the ground (hedge rows may not be on the parcel edge. Also check that you do actually own the parcel in question).
- b) you did not rent it to him.
- c) you don't want him to use the land and he should immediately remove his fences and cattle.
- d) you are happy that he continues to clean the land and take the hay (or not as the case maybe).
Once he starts to work the land for a period of time that is considered to be enough for its use to be essential to his business, then getting him off will be difficult.
If you get any guff then go to a solicitor that specialises in land issues and get a letter written. Do not delay and don't accept anything like "I did it in the past", "It's just this once" or "I'll take them off next week" etc. He must stop immediately and be under no illusion that he can choose when or how to use the land. If possible get someone else to take the hay for a year or so (hay is in very short supply so there will be no shortage of folks who might do this).
I've had similar issues with a old farmer who was very difficult. He knew it was always in his interest to do nothing and wait things out and to continually try to do more than he was entitled because it's so long-winded and expensive to sort it out legally.