I guess I'd like to start by pointing out that with someone like you describe, laws aren't likely to help you. Getting all legal on him is likely to be a big mistake since it will just make him angry. And the police are NOT going to send him off to jail for long - if at all - for trespassing on your property.
He also has a rain gutter that comes off his house and the spout comes out into my yard (but still on his property, if that makes sense. Just an extra bit of info for you.
Is that his house in the photo, beyond the section of fence? How far from the property line is that wall, and how far from the property line is the outer edge of his roof overhang? Do you have survey pins from an official recorded survey so you can tell exactly where the property line is?
You should be able to look up your city's zoning code, or your county's zoning code if you do not live in a city, online to see how close to the property line you could build a wall or a fence, or plant a hedge, if you wanted to do that. It's a little more difficult to figure out whether HIS wall, fence, or house violate applicable zoning codes, since it's the code that was in effect when each of those was built (not the current code) that would apply. The year the house was built is probably in the tax assessor's database, but if you didn't observe when the wall and fence were built I'm not sure how you'd find that out.
Did you get a title report when you bought your house? Are there any easements for your neighbor's benefit recorded in the county land records for your property? If you don't have a title report to look through, you could try searching the property records online (not the assessor's website, this would have to be the county recorder or equivalent official), but usually only relatively recent documents are available online, so if you don't find anything online there could still BE something.
Normally according to the law you don't have to let him walk on your property to get to his unless he has an easement giving him that right.
But I still think that researching who has what legal rights here is not likely to help you much with someone who has anger management problems - and could even backfire severely, if you try to insist on your legal rights.