Tough question because it depends on many variables.
Genetics, handling/quick movements, weather, stressors -mite/shb, dearth, ... Maybe they can just sense your fear and disgust in the attitudes
Ultimately its a judgement call; What you feel is aggressive could be fine to another beekeeper.
I typically say wait for a several weeks, if its NOT conditional just replace the queen - Life is too short to deal with aggressive/defensive bees.
