Парсим товары и цены в интернет-магазине Bloomingdales

Posted on: Январь 5, 2018 /

Categories: Бесплатные парсеры, Парсер магазина

Bloomingdale’s — мульти-брендовая сеть магазинов, основанная в апреле 1872. В настоящий момент сетью владеет компания Macy’s, Inc. С помощью парсера, представленного ниже, вы сможете отпарсить интернет-магазин bloomingdales.com и собрать данные о товарах, ценах и изображениях.

Примерное количество товаров: 350000
Примерное количество запросов: 350000
Рекомендуемый план подписки: Medium

ВНИМАНИЕ! Количество запросов может превышать количество товаров, потому что данные о вариациях, изображениях и др. могут парсится используя запросы к дополнительным ресурсам. Также часть данных о товаре может доставляться с помощью XHR запросов, что также увеличивает общее количество необходимых запросов.

Для его использования вы должны иметь учетную запись в нашем сервисе Diggernaut.

Пройдите по этой ссылке для регистрации в сервисе Diggernaut
После регистрации и подтверждения email адреса войдите в свою учетную запись
Создайте проект с любый именем и описанием, если вы не знаете как, обратитесь к нашей документации
Войдите во вновь созданный проект и создайте в нем диггер с любым именем, если вы не знаете как, обратитесь к нашей документации
Скопируйте в буфер обмена приведенный ниже сценарий диггера и вставьте его в созданный вами диггер, если вы не знаете как, обратитесь к нашей документации
ВНИМАНИЕ! Базовые прокси сервера могут не работать с данным сайтом и вам могут потребоваться ваши собственные прокси. Вы должны будете прописать ваш прокси сервер в сценарии, как указано. Если этот пункт вам непонятен и вы затрудняетесь как это сделать, свяжитесь с нами через систему поддержки или используя наш онлайн чат, мы обязательно вам поможем.
Переключите режим работы диггера с Debug на Active, если вы не знаете как, обратитесь к нашей документации
Запустите ваш диггер и дождитесь окончания его работы, если вы не знаете как, обратитесь к нашей документации
Скачайте собранный набор данных в нужном вам формате, если вы не знаете как, обратитесь к нашей документации

В дальнейшем вы можете установить расписание для запуска вашего парсера и забирать информацию регулярно.

Сценарий парсера:

---
config:
    debug: 2
    agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36
    proxy: #используйте ваш список прокси серверов тут
do:
- walk:
    to: https://www.bloomingdales.com/index
    headers:
        accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
        accept-language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7
        cache-control: no-cache
    do:
    - find: 
        path: '#globalFlyouts a'
        do: 
        - pool_clear: main
        - parse:
            attr: href
            filter:
                - \?id=(\d+)
        - variable_set: pur
        - variable_set: 
            field: first
            value: 1
        - if:
            match: (\d)
            do:
            - register_set: https://www.bloomingdales.com/api/navigation/categories/facet?categoryId=<%pur%>&facet=false&pageIndex=1&bcomNavPPP=undefine
            - link_add:
                pool: main
            - walk:
                to: links
                pool: main
                headers:
                    accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
                    accept-language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7
                    cache-control: no-cache
                do:
                - find: 
                    path: productids
                    do: 
                    - parse
                    - if:
                        match: (\d)
                        do:
                        - register_set: https://www.bloomingdales.com/shop/product/?ID=<%register%>&CategoryID=<%pur%>
                        - link_add:
                            pool: sub
                - find: 
                    path: productcount
                    do: 
                    - parse
                    - if:
                        match: (\d)
                        do:
                        - variable_set: count
                        - variable_get: first
                        - if:
                            match: (\d+)
                            do:
                            - variable_clear: first
                            - eval:
                                routine: js
                                body: (function () {var pages = []; for (var i=2; i*90 <= <%count%>; i++) {pages.push(i)}; return pages.join(",");})();
                            - to_block
                            - split:
                                context: text
                                delimiter: ","
                            - find: 
                                path: .splitted 
                                do: 
                                - parse
                                - register_set: https://www.bloomingdales.com/api/navigation/categories/facet?categoryId=<%pur%>&facet=false&pageIndex=<%register%>&bcomNavPPP=undefine
                                - link_add:
                                    pool: main
- walk:
    to: links
    headers:
        accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
        accept-language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7
        cache-control: no-cache
    pool: sub
    do:
    - proxy_switch
    - cookie_reset
    - variable_clear: allli
    - variable_clear: descr
    - variable_clear: n
    - object_new: product
    - find: 
        in: doc
        path: head 
        do: 
        - eval:
            routine: js
            body: '(function (){var d = new Date(); return d.toISOString()})();'
        - object_field_set:
            object: product
            field: date
        - static_get: url
        - filter:
            args:
             - (.+\?[idID]+=\d+)\&
        - object_field_set:
            object: product
            field: url
        - register_set: Bloomingdale
        - object_field_set:
            object: product
            field: brand
    - find: 
        path: '#productId' 
        do: 
        - parse:
            attr: value
        - if:
            match: (\d)
            do:
            - object_field_set:
                object: product
                field: sku
    - find: 
        path: '#brandNameLink' 
        do: 
        - parse
        - space_dedupe
        - trim
        - object_field_set:
            object: product
            field: brand
    - find: 
        path: '#productName, #productTitle' 
        do: 
        - variable_get: n
        - if:
            match: (\d)
            else:
            - parse
            - space_dedupe
            - trim
            - object_field_set:
                object: product
                field: name
            - variable_set: 
                field: n
                value: 1
    - find: 
        path: .selectedFOB
        do: 
        - parse
        - space_dedupe
        - trim
        - normalize:
            routine: lower
        - object_field_set:
            object: product
            field: category
            joinby: "|"
    - find: 
        path: 'script#pdp_data' 
        do: 
        - parse
        - normalize:
            routine: json2xml
        - to_block
        - find: 
            path: colorwayadditionalimages > *, colorwayprimaryimages > *, additionalimages, imagesource
            do: 
            - parse
            - split:
                context: text
                delimiter: ','
            - find: 
                path: .splitted 
                do: 
                - parse
                - if:
                    match: (\S)
                    do:
                    - register_set: https://images.bloomingdales.com/is/image/BLM/products/<%register%>
                    - object_field_set:
                        object: product
                        field: images
                        joinby: "|"
        - find: 
            path: colorfamily > * 
            do: 
            - parse
            - if:
                match: (\S)
                do:
                - object_field_set:
                    object: product
                    field: variations
                    joinby: "|"
        - find: 
            path: product > seokeywords
            slice: 0:-2
            do: 
            - parse
            - space_dedupe
            - trim
            - normalize:
                routine: lower
            - if:
                match: (\S)
                do:
                - object_field_set:
                    object: product
                    field: category
                    joinby: "|"
        - find: 
            path: longdescription 
            do: 
            - parse
            - space_dedupe
            - trim
            - if:
                match: (\S)
                do:
                - object_field_set:
                    object: product
                    field: description
        - find: 
            path: product > price 
            do: 
            - parse
            - space_dedupe
            - trim
            - if:
                match: (\d)
                do:
                - object_field_set:
                    object: product
                    field: price
                    type: float
                - register_set: USD
                - object_field_set:
                    object: product
                    field: currency
    - object_save:
        name: product

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

---

config:

debug: 2

agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36

proxy: #используйте ваш список прокси серверов тут

do:

- walk:

to: https://www.bloomingdales.com/index

headers:

accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8

accept-language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7

cache-control: no-cache

do:

- find:

path: '#globalFlyouts a'

do:

- pool_clear: main

- parse:

attr: href

filter:

- \?id=(\d+)

- variable_set: pur

- variable_set:

field: first

value: 1

- if:

match: (\d)

do:

- register_set: https://www.bloomingdales.com/api/navigation/categories/facet?categoryId=<%pur%>&facet=false&pageIndex=1&bcomNavPPP=undefine

- link_add:

pool: main

- walk:

to: links

pool: main

headers:

accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8

accept-language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7

cache-control: no-cache

do:

- find:

path: productids

do:

- parse

- if:

match: (\d)

do:

- register_set: https://www.bloomingdales.com/shop/product/?ID=<%register%>&CategoryID=<%pur%>

- link_add:

pool: sub

- find:

path: productcount

do:

- parse

- if:

match: (\d)

do:

- variable_set: count

- variable_get: first

- if:

match: (\d+)

do:

- variable_clear: first

- eval:

routine: js

body: (function () {var pages = []; for (var i=2; i*90 <= <%count%>; i++) {pages.push(i)}; return pages.join(",");})();

- to_block

- split:

context: text

delimiter: ","

- find:

path: .splitted

do:

- parse

- register_set: https://www.bloomingdales.com/api/navigation/categories/facet?categoryId=<%pur%>&facet=false&pageIndex=<%register%>&bcomNavPPP=undefine

- link_add:

pool: main

- walk:

to: links

headers:

accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8

accept-language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7

cache-control: no-cache

pool: sub

do:

- proxy_switch

- cookie_reset

- variable_clear: allli

- variable_clear: descr

- variable_clear: n

- object_new: product

- find:

in: doc

path: head

do:

- eval:

routine: js

body: '(function (){var d = new Date(); return d.toISOString()})();'

- object_field_set:

object: product

field: date

- static_get: url

- filter:

args:

- (.+\?[idID]+=\d+)\&

- object_field_set:

object: product

field: url

- register_set: Bloomingdale

- object_field_set:

object: product

field: brand

- find:

path: '#productId'

do:

- parse:

attr: value

- if:

match: (\d)

do:

- object_field_set:

object: product

field: sku

- find:

path: '#brandNameLink'

do:

- parse

- space_dedupe

- trim

- object_field_set:

object: product

field: brand

- find:

path: '#productName, #productTitle'

do:

- variable_get: n

- if:

match: (\d)

else:

- parse

- space_dedupe

- trim

- object_field_set:

object: product

field: name

- variable_set:

field: n

value: 1

- find:

path: .selectedFOB

do:

- parse

- space_dedupe

- trim

- normalize:

routine: lower

- object_field_set:

object: product

field: category

joinby: "|"

- find:

path: 'script#pdp_data'

do:

- parse

- normalize:

routine: json2xml

- to_block

- find:

path: colorwayadditionalimages > *, colorwayprimaryimages > *, additionalimages, imagesource

do:

- parse

- split:

context: text

delimiter: ','

- find:

path: .splitted

do:

- parse

- if:

match: (\S)

do:

- register_set: https://images.bloomingdales.com/is/image/BLM/products/<%register%>

- object_field_set:

object: product

field: images

joinby: "|"

- find:

path: colorfamily > *

do:

- parse

- if:

match: (\S)

do:

- object_field_set:

object: product

field: variations

joinby: "|"

- find:

path: product > seokeywords

slice: 0:-2

do:

- parse

- space_dedupe

- trim

- normalize:

routine: lower

- if:

match: (\S)

do:

- object_field_set:

object: product

field: category

joinby: "|"

- find:

path: longdescription

do:

- parse

- space_dedupe

- trim

- if:

match: (\S)

do:

- object_field_set:

object: product

field: description

- find:

path: product > price

do:

- parse

- space_dedupe

- trim

- if:

match: (\d)

do:

- object_field_set:

object: product

field: price

type: float

- register_set: USD

- object_field_set:

object: product

field: currency

- object_save:

name: product

Ниже приведен пример датасета с несколькими товарами в формате JSON (для наглядности). Датасет может быть скачан и как CSV, XLSX, XML, и любой другой текстовый формат используя темплейтный подход.

[{
    "product": {
        "brand": "Michael Aram",
        "category": "home|#homegoals",
        "currency": "USD",
        "date": "2017-12-07T21:43:20.868Z",
        "description": "In the designer's own words, the Molten collection is distinguished by \"streamlined, timeless shapes... objects which reverberate with the skill of their maker and yet do not fit into a traditional interpretation of craft. The pieces possess a soulfulness and organic energy only possible through the handmade process.\"",
        "images": "https://images.bloomingdales.com/is/image/BLM/products/5/optimized/8722225_fpx.tif|https://images.bloomingdales.com/is/image/BLM/products/5/optimized/8722225_fpx.tif",
        "name": "Michael Aram Molten 5-Piece Place Setting",
        "price": 80,
        "sku": "1197985",
        "url": "https://www.bloomingdales.com/shop/product/michael-aram-molten-5-piece-place-setting?ID=1197985",
        "variations": "Silver"
    }
}
,{
    "product": {
        "brand": "Lagostina",
        "category": "home|#homegoals",
        "currency": "USD",
        "date": "2017-12-07T21:43:24.007Z",
        "description": "Showcasing Lagostina's core values of impeccable Italian craftsmanship, technical innovation and elegant design, this ultrastrong grill pan triple-wall construction and a sturdy grooved surface for perfect grilling and searing.",
        "images": "https://images.bloomingdales.com/is/image/BLM/products/6/optimized/8694046_fpx.tif|https://images.bloomingdales.com/is/image/BLM/products/6/optimized/8694046_fpx.tif",
        "name": "Lagostina Accademia Bistecchiera 11\" Grill Pan",
        "price": 180,
        "sku": "1205514",
        "url": "https://www.bloomingdales.com/shop/product/lagostina-accademia-bistecchiera-11-grill-pan?ID=1205514"
    }
}
,{
    "product": {
        "brand": "Iittala",
        "category": "home|#homegoals",
        "currency": "USD",
        "date": "2017-12-07T21:43:26.117Z",
        "description": "Designed by Kaj Franck for Iittala, the Kartio carafe is a perfect balance of pure material and simple geometric form. Stripped of the superfluous, it is clean and timeless.",
        "images": "https://images.bloomingdales.com/is/image/BLM/products/5/optimized/1232615_fpx.tif|https://images.bloomingdales.com/is/image/BLM/products/5/optimized/1232615_fpx.tif",
        "name": "Iittala Kartio Carafe/Pitcher, 1 quart",
        "price": 100,
        "sku": "1239359",
        "url": "https://www.bloomingdales.com/shop/product/iittala-kartio-carafe-pitcher-1-quart?ID=1239359",
        "variations": "White"
    }
}]

[{

"product": {

"brand": "Michael Aram",

"category": "home|#homegoals",

"currency": "USD",

"date": "2017-12-07T21:43:20.868Z",

"description": "In the designer's own words, the Molten collection is distinguished by \"streamlined, timeless shapes... objects which reverberate with the skill of their maker and yet do not fit into a traditional interpretation of craft. The pieces possess a soulfulness and organic energy only possible through the handmade process.\"",

"images": "https://images.bloomingdales.com/is/image/BLM/products/5/optimized/8722225_fpx.tif|https://images.bloomingdales.com/is/image/BLM/products/5/optimized/8722225_fpx.tif",

"name": "Michael Aram Molten 5-Piece Place Setting",

"price": 80,

"sku": "1197985",

"url": "https://www.bloomingdales.com/shop/product/michael-aram-molten-5-piece-place-setting?ID=1197985",

"variations": "Silver"

}

"product": {

"brand": "Lagostina",

"category": "home|#homegoals",

"currency": "USD",

"date": "2017-12-07T21:43:24.007Z",

"description": "Showcasing Lagostina's core values of impeccable Italian craftsmanship, technical innovation and elegant design, this ultrastrong grill pan triple-wall construction and a sturdy grooved surface for perfect grilling and searing.",

"images": "https://images.bloomingdales.com/is/image/BLM/products/6/optimized/8694046_fpx.tif|https://images.bloomingdales.com/is/image/BLM/products/6/optimized/8694046_fpx.tif",

"name": "Lagostina Accademia Bistecchiera 11\" Grill Pan",

"price": 180,

"sku": "1205514",

"url": "https://www.bloomingdales.com/shop/product/lagostina-accademia-bistecchiera-11-grill-pan?ID=1205514"

}

"product": {

"brand": "Iittala",

"category": "home|#homegoals",

"currency": "USD",

"date": "2017-12-07T21:43:26.117Z",

"description": "Designed by Kaj Franck for Iittala, the Kartio carafe is a perfect balance of pure material and simple geometric form. Stripped of the superfluous, it is clean and timeless.",

"images": "https://images.bloomingdales.com/is/image/BLM/products/5/optimized/1232615_fpx.tif|https://images.bloomingdales.com/is/image/BLM/products/5/optimized/1232615_fpx.tif",

"name": "Iittala Kartio Carafe/Pitcher, 1 quart",

"price": 100,

"sku": "1239359",

"url": "https://www.bloomingdales.com/shop/product/iittala-kartio-carafe-pitcher-1-quart?ID=1239359",

"variations": "White"

}

}]

Михаил Сисин

Парсим товары и цены в интернет-магазине Bloomingdales

Добавить комментарий Отменить ответ

Свежие записи

Архивы

Рубрики