Multipage sitemap.xml extension

Magento already has a built-in Google sitemap generator. Nevertheless, there is one problem – it is generating a single file which includes all the category, product and CMS page links. But, due to the Google specifications it should not contain more than 50 000 items. So, if your shop is large enough, then your XML sitemap file will be quite big and will cause the error from the crawlers site. That is why, we decided to write an article that describes the extension which generates a sitemap with the index file and numerous sub-files for each type of the pages (products, categories, CMS).

Let’s start. First of all, we need to inform Magento about our new extension (note, that the current and further file locations are specified at the top of the file):

<?xml version="1.0"?>
<!--

Location:
magento_root/app/etc/modules/Atwix_Sitemap.xml

-->
<config>
    <modules>
        <Atwix_Sitemap>
            <active>true</active>
            <codePool>local</codePool>
        </Atwix_Sitemap>
    </modules>
</config>

Next what we need is a config file. It is pretty simple, due to our needs – so, let’s check the full code. The proper sections will be described with the corresponding extension parts:

<?xml version="1.0"?>
<!--

Location:
magento_root/app/code/local/Atwix/Sitemap/etc/config.xml

-->
<config>
    <modules>
        <Atwix_Sitemap>
            <version>1.0.0</version>
        </Atwix_Sitemap>
    </modules>
    <global>
        <models>
            <atwix_sitemap>
                <class>Atwix_Sitemap_Model</class>
            </atwix_sitemap>
            <sitemap>
                <rewrite>
                    <sitemap>Atwix_Sitemap_Model_Sitemap</sitemap>
                </rewrite>
            </sitemap>
        </models>
        <helpers>
            <atwix_sitemap>
                <class>Atwix_Sitemap_Helper</class>
            </atwix_sitemap>
        </helpers>
    </global>
    <adminhtml>
        <acl>
            <resources>
                <all>
                    <title>Allow Everything</title>
                </all>
                <admin>
                    <children>
                        <system>
                            <children>
                                <config>
                                    <children>
                                        <atwix_sitemap>
                                            <title>Atwix Sitemap</title>
                                        </atwix_sitemap>
                                    </children>
                                </config>
                            </children>
                        </system>
                    </children>
                </admin>
            </resources>
        </acl>
    </adminhtml>
</config>

So, to begin with, we need to provide a possibility of enabling and disabling our extension and set the limit of the items per sub-file. This goal can be reached through the Magento admin panel settings. Here we’ll just add a section to the System Configuration. Furthermore, the corresponding code in the config.xml specifies the proper ACL resource:

        <acl>
            <resources>
                <all>
                    <title>Allow Everything</title>
                </all>
                <admin>
                    <children>
                        <system>
                            <children>
                                <config>
                                    <children>
                                        <atwix_sitemap>
                                            <title>Atwix Sitemap</title>
                                        </atwix_sitemap>
                                    </children>
                                </config>
                            </children>
                        </system>
                    </children>
                </admin>
            </resources>
        </acl>

The system.xml will have only two settings so far:

<?xml version="1.0"?>
<!--

Location:
magento_root/app/code/local/Atwix/Sitemap/etc/system.xml

-->
<config>
    <tabs>
        <atwix translate="label" module="atwix_sitemap">
            <label>Atwix Extensions</label>
            <sort_order>100</sort_order>
        </atwix>
    </tabs>
    <sections>
            <atwix_sitemap translate="label" module="atwix_sitemap">
                <class>separator-top</class>
                <label>Atwix Sitemap</label>
                <tab>atwix</tab>
                <sort_order>10</sort_order>
                <show_in_default>1</show_in_default>
                <show_in_website>1</show_in_website>
                <show_in_store>1</show_in_store>
                <groups>
                    <general translate="label">
                        <label>General Options</label>
                        <frontend_type>text</frontend_type>
                        <sort_order>10</sort_order>
                        <show_in_default>1</show_in_default>
                        <show_in_website>0</show_in_website>
                        <show_in_store>0</show_in_store>
                        <fields>
                            <enabled translate="label">
                                <label>Enable extension</label>
                                <frontend_type>select</frontend_type>
                                <source_model>adminhtml/system_config_source_yesno</source_model>
                                <sort_order>100</sort_order>
                                <show_in_default>1</show_in_default>
                                <show_in_website>1</show_in_website>
                                <show_in_store>1</show_in_store>
                            </enabled>
                            <limit translate="label">
                                <label>Item limit per page</label>
                                <frontend_type>text</frontend_type>
                                <sort_order>200</sort_order>
                                <show_in_default>1</show_in_default>
                                <show_in_website>1</show_in_website>
                                <show_in_store>1</show_in_store>
                                <comment>
                                    <![CDATA[<span class="notice">50000 if empty</span>]]>
                                </comment>
                            </limit>
                        </fields>
                    </general>
                </groups>
            </atwix_sitemap>
    </sections>
</config>

Then, the final thing that we need to make the backend settings work is an empty helper. It is declared in the config.xml like here:

        <helpers>
            <atwix_sitemap>
                <class>Atwix_Sitemap_Helper</class>
            </atwix_sitemap>
        </helpers>

So, here is our “dummy helper” code:

<?php
/**
 * Location:
 * magento_root/app/code/local/Atwix/Sitemap/Helper/Data.php
 */

class Atwix_Sitemap_Helper_Data extends Mage_Core_Helper_Data
{

}

Okay, now we are ready to rewrite the core model responsible for the sitemap generation. Declaring our extension model and rewriting the Mage_Sitemap_Model_Sitemap are made with this config.xml code:

        <models>
            <atwix_sitemap>
                <class>Atwix_Sitemap_Model</class>
            </atwix_sitemap>
            <sitemap>
                <rewrite>
                    <sitemap>Atwix_Sitemap_Model_Sitemap</sitemap>
                </rewrite>
            </sitemap>
        </models>

The model code as follows:

<?php
/**
 * Location:
 * magento_root/app/code/local/Atwix/Sitemap/Model/Sitemap.php
 */

class Atwix_Sitemap_Model_Sitemap extends Mage_Sitemap_Model_Sitemap
{
    const     ITEM_LIMIT = 50000;
    protected $_io;
    protected $_subfiles = array();

    public function generateXml()
    {
        $enabled = (bool) Mage::getStoreConfig('atwix_sitemap/general/enabled');
        if(!$enabled) {
            return parent::generateXml();
        }
        $helper = Mage::helper('atwix_sitemap');
        
        $limit = (int) Mage::getStoreConfig('atwix_sitemap/general/limit');
        if ($limit == 0) {
            $limit = self::ITEM_LIMIT;
        }
        $this->fileCreate();

        $storeId = $this->getStoreId();
        $date = Mage::getSingleton('core/date')->gmtDate('Y-m-d');
        $baseUrl = Mage::app()->getStore($storeId)->getBaseUrl(Mage_Core_Model_Store::URL_TYPE_LINK);

        /**
         * Generate categories sitemap
         */
        $changefreq = (string) Mage::getStoreConfig('sitemap/category/changefreq');
        $priority = (string) Mage::getStoreConfig('sitemap/category/priority');
        $collection = Mage::getResourceModel('sitemap/catalog_category')->getCollection($storeId);

        /**
         * Delete old category files
         */
        try {
            foreach(glob($this->getPath() . substr($this->getSitemapFilename(), 0, strpos($this->getSitemapFilename(), '.xml')) . '_cat_*.xml') as $f) {
                unlink($f);
            }
        } catch(Exception $e) {
            Mage::getSingleton('adminhtml/session')->addError(
                $helper->__('Unable to delete old categories sitemaps') . $e->getMessage()
            );
        }

        /**
         * Brake to pages
         */
        $pages = ceil( count($collection) / $limit );
        $i = 0;
        while( $i < $pages ) {
            $name = '_cat_' . $i . '.xml';
            $this->subFileCreate($name);
            $subCollection = array_slice($collection, $i * $limit, $limit);
            foreach ($subCollection as $item) {
                $xml = sprintf(
                    '<url><loc>%s</loc><lastmod>%s</lastmod><changefreq>%s</changefreq><priority>%.1f</priority></url>',
                    htmlspecialchars($baseUrl . $item->getUrl()),
                    $date,
                    $changefreq,
                    $priority
                );
                $this->sitemapSubFileAddLine($xml, $name);
            }
            $this->subFileClose($name);
            /**
             * Add link of the subfile to the main file
             */
            $xml = sprintf('<sitemap><loc>%s</loc><lastmod>%s</lastmod></sitemap>', htmlspecialchars( $this->getSubFileUrl($name)), $date);
            $this->sitemapFileAddLine($xml);
            $i++;
        }

        unset($collection);

        /**
         * Generate products sitemap
         */
        $changefreq = (string) Mage::getStoreConfig('sitemap/product/changefreq');
        $priority = (string) Mage::getStoreConfig('sitemap/product/priority');
        $collection = Mage::getResourceModel('sitemap/catalog_product')->getCollection($storeId);

        /**
         * Delete old products files
         */
        try {
            foreach(glob($this->getPath() . substr($this->getSitemapFilename(), 0, strpos($this->getSitemapFilename(), '.xml')) . '_prod_*.xml') as $f) {
                unlink($f);
            }
        } catch(Exception $e) {
            Mage::getSingleton('adminhtml/session')->addError(
                $helper->__('Unable to delete old products sitemaps') . $e->getMessage()
            );
        }

        /**
         * Brake to pages
         */
        $pages = ceil( count($collection) / $limit );
        $i = 0;
        while( $i < $pages ) {
            $name = '_prod_' . $i . '.xml';
            $this->subFileCreate($name);
            $subCollection = array_slice($collection, $i * $limit, $limit);
            foreach ($subCollection as $item) {
                $xml = sprintf(
                    '<url><loc>%s</loc><lastmod>%s</lastmod><changefreq>%s</changefreq><priority>%.1f</priority></url>',
                    htmlspecialchars($baseUrl . $item->getUrl()),
                    $date,
                    $changefreq,
                    $priority
                );
                $this->sitemapSubFileAddLine($xml, $name);
            }
            $this->subFileClose($name);
            /**
             * Add link of the subfile to the main file
             */
            $xml = sprintf('<sitemap><loc>%s</loc><lastmod>%s</lastmod></sitemap>', htmlspecialchars($this->getSubFileUrl($name)), $date);
            $this->sitemapFileAddLine($xml);
            $i++;
        }

        unset($collection);

        /**
         * Generate cms pages sitemap
         */
        $changefreq = (string) Mage::getStoreConfig('sitemap/page/changefreq');
        $priority = (string) Mage::getStoreConfig('sitemap/page/priority');
        $collection = Mage::getResourceModel('sitemap/cms_page')->getCollection($storeId);

        /**
         * Delete old cms pages files
         */
        try {
            foreach(glob($this->getPath() . substr($this->getSitemapFilename(), 0, strpos($this->getSitemapFilename(), '.xml')) . '_pages_*.xml') as $f) {
                unlink($f);
            }
        } catch(Exception $e) {
            Mage::getSingleton('adminhtml/session')->addError(
                $helper->__('Unable to delete old products sitemaps') . $e->getMessage()
            );
        }

        /**
         * Brake to pages
         */
        $pages = ceil( count($collection) / $limit );
        $i = 0;
        while( $i < $pages ) {
            $name = '_pages_' . $i . '.xml';
            $this->subFileCreate($name);
            $subCollection = array_slice($collection, $i * $limit, $limit);
            foreach ($subCollection as $item) {
                $xml = sprintf(
                    '<url><loc>%s</loc><lastmod>%s</lastmod><changefreq>%s</changefreq><priority>%.1f</priority></url>',
                    htmlspecialchars($baseUrl . $item->getUrl()),
                    $date,
                    $item->getUrl() == 'home' ? 'always' : $changefreq,
                    $item->getUrl() == 'home' ? '1' : $priority
                );
                $this->sitemapSubFileAddLine($xml, $name);
            }
            $this->subFileClose($name);
            /**
             * Adding link of the subfile to the main file
             */
            $xml = sprintf('<sitemap><loc>%s</loc><lastmod>%s</lastmod></sitemap>', htmlspecialchars($this->getSubFileUrl($name)), $date);
            $this->sitemapFileAddLine($xml);
            $i++;
        }
        unset($collection);

        $this->fileClose();

        $this->setSitemapTime(Mage::getSingleton('core/date')->gmtDate('Y-m-d H:i:s'));
        $this->save();

        return $this;
    }

    /**
     * Create sitemap subfile by name in sitemap directory
     *
     * @param $name
     */
    protected function subFileCreate($name)
    {
        $io = new Varien_Io_File();
        $io->setAllowCreateFolders(true);
        $io->open(array('path' => $this->getPath()));
        $io->streamOpen( substr($this->getSitemapFilename(), 0, strpos($this->getSitemapFilename(), '.xml')) . $name);

        $io->streamWrite('<?xml version="1.0" encoding="UTF-8"?>' . "\n");
        $io->streamWrite('<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">');
        $this->_subfiles[$name] = $io;
    }

    /**
     * Add line to sitemap subfile
     *
     * @param $xml
     * @param $name
     */
    public function sitemapSubFileAddLine($xml, $name) {
        $this->_subfiles[$name]->streamWrite($xml);
    }

    /**
     * Create main sitemap file
     */
    protected function fileCreate() {
        $io = new Varien_Io_File();
        $io->setAllowCreateFolders(true);
        $io->open(array('path' => $this->getPath()));
        $io->streamOpen($this->getSitemapFilename());

        $io->streamWrite('<?xml version="1.0" encoding="UTF-8"?>' . "\n");
        $io->streamWrite('<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">');
        $this->_io = $io;
    }

    /**
     * Add closing tag and close sitemap file
     */
    protected function fileClose() {
        $this->_io->streamWrite('</sitemapindex>');
        $this->_io->streamClose();
    }

    /**
     * Add closing tag and close sitemap subfile by the name
     *
     * @param $name
     */
    protected function subFileClose($name) {
        $this->_subfiles[$name]->streamWrite('</urlset>');
        $this->_subfiles[$name]->streamClose();
    }

    /**
     * Get URL of sitemap subfile by the name
     *
     * @param $name
     * @return string
     */
    public function getSubFileUrl($name)
    {
        $fileName = substr($this->getSitemapFilename(), 0, strpos($this->getSitemapFilename(), '.xml')) . $name;
        $filePath = Mage::app()->getStore($this->getStoreId())->getBaseUrl(Mage_Core_Model_Store::URL_TYPE_LINK) . $this->getSitemapPath();
        $filePath = str_replace('//','/',$filePath);
        $filePath = str_replace(':/','://',$filePath);
        return $filePath . $fileName;
    }

    /**
     * Add line to the main file
     *
     * @param $xml
     */
    public function sitemapFileAddLine($xml)
    {
        $this->_io->streamWrite($xml);
    }
}

The main method here is generateXml() . Basically, it is creating the main file, running generation of items of each type (categories, products and cms pages) and closing the file (with model saving) at the end. It is true for both our and original method class. The only thing is really important here is that we’re splitting items collection to pages and write those pages to the separate files. Moreover, we’re also taking care of deleting the old sitemap files after each sitemap generation.

If you have some additional components in your store such as a blog, which has its own pages, you should add the corresponding logic to the generateXml() method.

The extension repo is available through this link. You can check our Magento ERP extention guide too. Thanks for reading us and do not hesitate to leave your feedback about our post in the comments.
Read more:

Adding new category attribute in Magento

Cookie	Duration	Description
__jid	1 day	Used to add comments to the website and remember the user's Disqus login credentials across websites that use said service.
cf_ob_info	past	The cf_ob_info cookie is set by Cloudflare to provide information on HTTP Status Code returned by the origin web server, the Ray ID of the original failed request and the data center serving the traffic.
cf_use_ob	past	Cloudflare sets this cookie to improve page load times and to disallow any security restrictions based on the visitor's IP address.
cookielawinfo-checkbox-analytics	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional	11 months	The cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
li_gc	2 years	Linkeding cookie that stores the user's cookie consent state for the current domain
viewed_cookie_policy	11 months	The cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.

Cookie	Duration	Description
_ga	2 years	Google Analytics cookie. Used to identify unique users and expires after two years
_gat	1 day	Used by Google Analytics to throttle request rate
_gid	24 hours	Google Analytics cookie. Used to identify user and expires in 24 hours.
_hjAbsoluteSessionInProgress	30 minutes	Hotjar cookie. Used to detect the first pageview session of a user. 30 minutes duration. Boolean true/false data type.
_hjFirstSeen	User session	Hotjar cookie. Identifies a new user’s first session. Used by Recording filters to identify new user sessions. Session duration. Boolean true/false data type.
_hjIncludedInPageviewSample	30 minutes	Hotjar cookie. Set to determine if a user is included in the data sampling defined by your site's pageview limit. 30 minutes duration. Boolean true/false data type.
_hjIncludedInSessionSample	30 minutes	Hotjar cookie. Set to determine if a user is included in the data sampling defined by your site's daily session limit. 30 minutes duration. Boolean true/false data type.
_hjRecordingLastActivity	User session	Hotjar cookie. Sets a unique ID for the session. This allows the website to obtain data on visitor behaviour for statistical purposes
_hjSession_2881021	30 minutes	Hotjar cookie. Holds current session data. Ensures subsequent requests in the session window are attributed to the same session. 30 minutes duration. JSON data type.
_hjSessionRejected	User session	Hotjar cookie. If present, set to '1' for the duration of a user's session, when Hotjar has rejected the session from connecting to our WebSocket due to server overload. Applied in extremely rare situations to prevent severe performance issues. Session duration. Boolean true/false data type.
_hjSessionUser_2881021	1 year	Hotjar cookie. Set when a user first lands on a page. Persists the Hotjar User ID which is unique to that site. Ensures data from subsequent visits to the same site are attributed to the same user ID. 365 days duration. JSON data type.
_hjTLDTest	User session	Hotjar cookie. Registers statistical data on users' behaviour on the website. Used for internal analytics by the website operator.
AnalyticsSyncHistory	29 days	Linkedin cookie. Used in connection with data-synchronization with third-party analysis service.
disqus_unique	1 year	Disqus cookie. Collects statistics related to the user's visits to the website, such as number of visits, average time spent on the website and loaded pages.
https://#.#/	User session	leadfeeder.com cookie. Registers statistical data on users' behaviour on the website. Used for internal analytics by the website operator.
juggler/event.gif	User session	Disqus cookie. Identifies the visitor across devices and visits, in order to optimize the chat-box function on the website.

Cookie	Duration	Description
aet-dismiss	User session	Disqus cookie. Necessary for the functionality of the website's comments system
drafts.queue		Disqus cookie. Necessary for the functionality of the website's comments system
lang		Linkedin cookie. Remembers the user's selected language version of a website
submitted_posts_cache	User session	Disqus cookie. Necessary for the functionality of the website's comments system

Cookie	Duration	Description
_hjRecordingEnabled	User session	Hotjar cookie. This cookie is used to identify the visitor and optimize ad-relevance by collecting visitor data from multiple websites – this exchange of visitor data is normally provided by a third-party data-center or ad-exchange.
_lfa	2 years	Leadfeeder cookie. Used to store and track audience reach and expires in 2 years.
_lfa_expiry	persistent	Leadfeeder cookie. : Contains the expiry-date for the cookie with corresponding name
_lfa_test_cookie_stored	1 day	Leadfeeder cookie. Used in context with Account-Based-Marketing (ABM). The cookie registers data such as IP-addresses, time spent on the website and page requests for the visit. This is used for retargeting of multiple users rooting from the same IP-addresses. ABM usually facilitates B2B marketing purposes
ads/ga-audiences	User session	Used by Google AdWords to re-engage visitors that are likely to convert to customers based on the visitor's online behaviour across websites.
badges-message	Persistent	Disqus cookie. Used for comments functionality
bcookie	2 years	Used by the social networking service, LinkedIn, for tracking the use of embedded services.
bscookie	2 years	Used by the social networking service, LinkedIn, for tracking the use of embedded services.
lidc	1 day	Used by the social networking service, LinkedIn, for tracking the use of embedded services.
UserMatchHistory	29 days	Linkedin cookie. Ensures visitor browsing-security by preventing cross-site request forgery. This cookie is essential for the security of the website and visitor.

Multipage sitemap.xml Magento module

Subscribe to Our Newsletter