Bring you down

If people are looking to bring you down

it only means that you are above them.