In the last couple of days I found that my sitemap.xml was not automatically updating when I published a new post. As far as I know, sitemap.xml will automatically build using wp-cron.php. And after digging a little bit, I found that something was blocking wp-cron.php from running. This is what the error log showed:
[error] [client xxx.xxx.xxx.xxx] client denied by server configuration: /home/username/public_html/wp-cron.php
[error] [client xxx.xxx.xxx.xxx] client denied by server configuration: /home/username/public_html/index.php
#xxx.xxx.xxx.xxx: my own ip addressSo the error log says that “client denied by server configuration“. This error may be caused by many possibilities, but in my case I accidentally blocked the WordPress agent in my htaccess — see my previous post Block Unwanted Visitors To Your Website With .htaccess.
There was a bot scraping my website without honoring the robots.txt — the bot used a “WordPress 1.01” user agent. So I blocked it with ‘*’ (wordpress*) which means blocking all user agents which start with the word ‘wordpress’. And this caused me to accidentally block the WordPress cron itself. So after I deleted that line, everything came back to normal.
So if you are having this issue, be sure to check your .htaccess file, file and directory permissions, and your Apache configuration.