Maybe it’s the stuff you’re posting. Is it your best work? Is it only your photography or are you slipping in some vacation pics or unrelated content? I recently watched a Scott Kelby session on this very thing. People that follow you for photography content will be turned off by off-topic posts and will likely unfollow you. Have a look back at your posts and see if you can find off-topic posts and then remove them. Then do some research on the very best hashtags to use.