Minecraft 1.16 Spigot] How can test if monsters are near x & z = 8 and y = 5?

Do you do that somehow with vectors or .getNearbyEntitys?

First you create an armor stand with the tag "Armorstand" at your coordinates (here 8 5 8)

/ summon armor_stand 8 5 8 {Invisible: 1b, Tags: ["Armorstand"]}

then you check if a monster (here a zombie) is near (in this case 5 blocks)

/ execute as @e [tag = Armorstand] at @s if entity @e [type = minecraft: zombie, distance = .5]

The distance and the monster can be changed at will.