Бесплатный парсер товаров онлайн магазина Cabela’s

С помощью этого бесплатного парсера товаров онлайн магазина Cabela’s вы сможете собрать данные о десятках тысяч товаров для активного отдыха, охоты, рыбалки и туризма. Компания Cabela’s является одним из ведущих американских ретейлеров в области товаров для различных занятий и спорта на открытом воздухе. Компания была основана в 1961 году Ричардом Кабела в городе Сидней, штат Небраска и до сих пор управляется членами семьи Кабела.

Примерное количество товаров: 70000
Примерное количество запросов: 70000
Рекомендуемый план подписки: Small

ВНИМАНИЕ! Количество запросов может превышать количество товаров, потому что данные о вариациях, изображениях и др. могут парсится используя запросы к дополнительным ресурсам. Также часть данных о товаре может доставляться с помощью XHR запросов, что также увеличивает общее количество необходимых запросов.

Как использовать этот парсер товаров

Для использования парсера товаров онлайн магазина Cabela’s вы должны иметь учетную запись в нашем сервисе Diggernaut.

1. Пройдите по этой ссылке для регистрации в сервисе Diggernaut
2. После регистрации и подтверждения email адреса войдите в свою учетную запись
3. Создайте проект с любый именем и описанием, если вы не знаете как, обратитесь к нашей документации
4. Войдите во вновь созданный проект и создайте в нем диггер с любым именем, если вы не знаете как, обратитесь к нашей документации
5. Скопируйте в буфер обмена приведенный ниже сценарий диггера и вставьте его в созданный вами диггер, если вы не знаете как, обратитесь к нашей документации
6. Переключите режим работы диггера с Debug на Active, если вы не знаете как, обратитесь к нашей документации
7. Запустите ваш диггер и дождитесь окончания его работы, если вы не знаете как, обратитесь к нашей документации
8. Скачайте собранный набор данных в нужном вам формате, если вы не знаете как, обратитесь к нашей документации

В дальнейшем вы можете установить расписание для запуска вашего парсера и забирать информацию регулярно.

Сценарий парсера

---
config:
    debug: 2
    agent: Chrome
do:
- walk:
    to: http://www.cabelas.com
    do:
    - find: 
        path: div.shopDropdown>a
        do: 
        - parse:
            attr: href
            filter: ^(.+\/\_\/N\-\d+)
        - space_dedupe
        - trim
        - if:
            match: \w+
            do:
            - normalize:
                routine: url
            - link_add:
                pool: catalog
- walk:
    to: links
    pool: catalog
    do:
    - sleep: 2
    - variable_clear: good
    - find:
        path: div.leftnav_content
        do:
        - variable_set:
            field: good
            value: "yes"
    - find:
        path: div.leftnav_content a
        do:
        - parse:
            attr: href
            filter: ^(.+\/\_\/N\-\d+)
        - space_dedupe
        - trim
        - if:
            match: \w+
            do:
            - normalize:
                routine: url
            - link_add:
                pool: catalog
    - find:
        path: a.entry:contains('Next')
        do:
        - parse:
            attr: href
        - space_dedupe
        - trim
        - if:
            match: \w+
            do:
            - normalize:
                routine: url
            - link_add:
                pool: catalog
    - find:
        path: div.productContentBlock>a
        do:
        - variable_set:
            field: good
            value: "yes"
        - parse:
            attr: href
        - space_dedupe
        - trim
        - if:
            match: \w+
            do:
            - normalize:
                routine: url
            - link_add:
                pool: pages
    - find:
        path: html
        do:
        - variable_get: good
        - if:
            match: "yes"
            else:
            - nocontent_category
- walk:
    to: links
    pool: pages
    do:
    - variable_clear: good
    - sleep: 2
    - find:
        path: 'div#productDetailsTemplate'
        do:
        - variable_set:
            field: good
            value: "yes"
        - variable_clear: list
        - variable_clear: desc
        - variable_clear: pid
        - variable_clear: cid
        - object_new: product
        - eval:
            routine: js
            body: '(function (){var d = new Date(); return d.toISOString()})();'
        - object_field_set:
            object: product
            field: date
        - static_get: url
        - object_field_set:
            object: product
            field: url
        - find:
            path: span.itemNumber
            do:
            - node_remove: b
            - parse
            - space_dedupe
            - trim
            - variable_set: pid
            - object_field_set:
                object: product
                field: sku
        - find:
            path: h1.label
            do:
            - parse
            - space_dedupe
            - trim
            - object_field_set:
                object: product
                field: name
        - register_set: Cabela's
        - variable_set: brand
        - find:
            in: doc
            path: script[type="application/ld+json"]
            do:
            - parse
            - normalize:
                routine: replace_substring
                args:
                - Luck\s+"E"\s+Strike: Luck E Strike
                - \"@type\"\:\s*\"Product\"\,\s+\"name\"\:.+: ''
                - \"description\"\:.+: ''
                - \n+: ''
                - \}\,\]: '}]'
            - normalize:
                routine: json2xml
            - to_block
            - find:
                path: brand>name
                do:
                - parse
                - space_dedupe
                - trim
                - variable_set: brand
            - find:
                path: offers:has(standardPrice:not(:contains("null")))
                slice: 0
                do:
                - find:
                    path: standardPrice
                    do:
                    - parse
                    - object_field_set:
                        object: product
                        type: float
                        field: price
                - find:
                    path: price
                    do:
                    - parse
                    - object_field_set:
                        object: product
                        type: float
                        field: price
                - register_set: USD
                - object_field_set:
                    object: product
                    field: currency
        - variable_get: brand
        - object_field_set:
            object: product
            field: brand
        - find:
            in: doc
            path: meta[name="description"]
            do:
            - parse:
                attr: content
            - space_dedupe
            - trim
            - variable_set: desc
        - find:
            path: div.pdp-desc-long
            do:
            - parse
            - space_dedupe
            - trim
            - variable_set: desc
        - find:
            path: 'div#description'
            do:
            - node_replace:
                path: br
                with: "\n"
            - split:
                context: text
                delimiter: \n+
            - find:
                path: div.splitted
                slice: 0
                do:
                - parse
                - space_dedupe
                - trim
                - variable_set: desc
        - variable_get: desc
        - object_field_set:
            object: product
            field: description
        - find:
            path: select.js-dropdown:has(option:contains("Select COLOR"))
            do:
            - find:
                path: option
                slice: 1:-1
                do:
                - parse
                - space_dedupe
                - trim
                - if:
                    match: \w+
                    do:
                    - object_field_set:
                        object: product
                        joinby: "|"
                        field: variations
        - find:
            path: script:contains('params_viewlarger.push("asset",')
            do:
            - parse:
                filter: \s+params_viewlarger\.push\(\"asset\"\,\s+"([^"]+)"\)\;
            - to_block
            - split:
                context: text
                delimiter: \s*[\;\,]
            - find:
                path: div.splitted
                do:
                - parse:
                    filter: ^([^\?]+)
                - space_dedupe
                - trim
                - if:
                    match: \w+
                    do:
                    - if:
                        match: _sw_
                        else:
                        - register_set: http://images.cabelas.com/is/image/<%register%>?wid=1000
                        - object_field_set:
                            object: product
                            joinby: "|"
                            field: images
        - find:
            path: script:contains('params_viewlarger.push("asset",')
            do:
            - parse:
                filter: \s+altviewparams_viewlarger\.push\(\"asset\"\,\s+"([^"]+)"\)\;
            - to_block
            - split:
                context: text
                delimiter: \s*[\;\,]
            - find:
                path: div.splitted
                do:
                - parse:
                    filter: ^([^\?]+)
                - space_dedupe
                - trim
                - if:
                    match: \w+
                    do:
                    - if:
                        match: _sw_
                        else:
                        - register_set: http://images.cabelas.com/is/image/<%register%>?wid=1000
                        - object_field_set:
                            object: product
                            joinby: "|"
                            field: images
        - find:
            path: ul.breadcrumb>li>a
            do:
            - parse
            - space_dedupe
            - trim
            - if:
                match: \w+
                do:
                - object_field_set:
                    object: product
                    joinby: "|"
                    field: categories
        - object_save:
            name: product
    - find:
        path: html
        do:
        - variable_get: good
        - if:
            match: "yes"
            else:
            - nocontent_product

Пример данных, собранных парсером

Ниже приведен пример датасета с несколькими товарами в формате JSON (для наглядности). Датасет может быть скачан и как CSV, XLSX, XML, и любой другой текстовый формат используя темплейтный подход.

[{
    "product": {
        "brand": "Cabela's",
        "categories": "Hunting|Hunting Bags & Packs",
        "currency": "USD",
        "date": "2017-12-27T13:24:43.531Z",
        "description": "For total concealment and game-stalking quiet construction, look no further than our Traditional Hydration Pack. Incorporating low-nap poly tricot with PVC backing, this 70-oz.-capacity pack features comfort shoulder straps and a low-profile design. Taste-free hydration-tube system. External zippered pocket and shock-cord cargo panel. Imported.",
        "images": "http://images.cabelas.com/is/image/Cabelas/s7_518535_510_01?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_518535_510_01?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_518535_510_alt01_01?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_518535_510_alt02_01?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_518535_510_alt03_01?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_518535_510_alt04_01?wid=1000",
        "name": "Cabela's Traditional Hydration Pack",
        "price": 23.88,
        "sku": "IK-518535",
        "url": "http://www.cabelas.com/product/hunting/hunting-bags-packs/pc/104791680/c/104392080/cabelas-traditional-hydration-pack/728032.uts"
    }
}
,{
    "product": {
        "brand": "Badlands",
        "categories": "Hunting|Hunting Bags & Packs",
        "currency": "USD",
        "date": "2017-12-27T13:24:46.525Z",
        "description": "Long, challenging hunting trips require you to be prepared for any obstacle you might encounter. With Cabela's-exclusive Badlands' Release Day Pack, you'll have the pocket space, support and carry capabilities you need to be just that – prepared. The legendary Air-Track™ Suspension system combines with the technical, ultralight construction to deliver easy mobility for long treks over the most rugged terrain. Made of super-quiet KXO-32 fabric with rugged Hypalon- and Kevlar-reinforced stress points for durability that stands up to years of use in the field, it also offers a multitude of pockets - five to be exact - that team up to keep all of your must-have hunting gear and accessories secure while you're on the move. It even boasts bow-carrying capabilities for added hands-free convenience. External bedroll and compression straps. Accepts up to a 110-oz. reservoir (sold separately). Comes with Badlands' Unconditional Lifetime Warranty. Imported.",
        "images": "http://images.cabelas.com/is/image/Cabelas/s7_463111_999_04?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_463111_999_04?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_463111_999_alt01_04?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_463111_999_alt02_04?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_463111_999_alt03_04?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_463111_999_alt04_04?wid=1000",
        "name": "Badlands Release Day Pack – Cabela's Exclusive",
        "price": 99.99,
        "sku": "IK-463111",
        "url": "http://www.cabelas.com/product/hunting/hunting-bags-packs/pc/104791680/c/104392080/badlands-release-day-pack/2000208.uts"
    }
}
,{
    "product": {
        "brand": "Herter's",
        "categories": "Hunting|Hunting Bags & Packs",
        "currency": "USD",
        "date": "2017-12-27T13:24:48.912Z",
        "description": "Transport your calls, ammo and other duck-hunting accessories safely and securely in Herter's Waterfowl Field Bag. A large zippered opening delivers access to the main compartment, which has an adjustable divider to separate your gear. Exterior pockets on the front, back and sides keep essentials at the ready. Adjustable shoulder strap and neoprene-wrapped handle for easy carry. Rugged 600-denier polyester fabric resists snags and tears. Imported.",
        "images": "http://images.cabelas.com/is/image/Cabelas/s7_421520_560_02?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_421520_560_02?wid=1000",
        "name": "Herter's® Waterfowl Field Bag",
        "price": 24.99,
        "sku": "IK-421520",
        "url": "http://www.cabelas.com/product/hunting/hunting-bags-packs/pc/104791680/c/104392080/herters-waterfowl-field-bag/1643686.uts"
    }
}
,{
    "product": {
        "brand": "Cabela's",
        "categories": "Hunting|Hunting Bags & Packs",
        "currency": "USD",
        "date": "2017-12-27T13:24:51.318Z",
        "description": "You won’t find a better gear bag for a lower price. Its durable, weather-resistant 600-denier polyester construction makes it ideal for toting everything from packable rain gear to extra odds and ends. Sturdy 1-1⁄2\" nylon web carry straps can be joined by a hand-friendly wrap handle. Six exterior pockets, including zippered mesh pockets on top and side, provide multiple storage and organization options. Embroidered Cabela’s logo on front pocket. Imported.",
        "images": "http://images.cabelas.com/is/image/Cabelas/s7_580032_999_03?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_580032_999_03?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_580032_014_01?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_580032_014_01?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_580032_022_01?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_580032_022_01?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_580032_027_01?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_580032_027_01?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_580032_148_01?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_580032_148_01?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_580032_928_01?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_580032_928_01?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_580032_896_05?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_580032_896_05?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_580032_014_alt01_01?wid=1000|http://images.cabelas.com/is/image/Cabelas/s7_580032_148_alt01_01?wid=1000",
        "name": "Cabela's Catch-All Gear Bags",
        "price": 14.99,
        "sku": "IK-580032",
        "url": "http://www.cabelas.com/product/hunting/hunting-bags-packs/pc/104791680/c/104392080/cabelas-catch-all-gear-bags/753422.uts",
        "variations": "BLUE|TAN|PINK|GRAY|CAMO|O2 OCTANE"
    }
}]