{"flag":true,"single":true,"pageTitle":"Simple Grabing tutorial | Create Spider, Extract data, Save data in json, with pagination scraping | scrapy cheatsheet Python","post":{"id":38,"user_id":"1","slug":"simple-grabing-tutorial-create-spider-extract-data-save-data-in-json-with-pagination-scraping-scrapy-cheatsheet-python-skcm","title":"Simple Grabing tutorial | Create Spider, Extract data, Save data in json, with pagination scraping | scrapy cheatsheet Python","body":"

We will grabe https:\/\/quotes.toscrape.com\/<\/a><\/p>\r\n

1.Run command below to start a scrapy project <\/p>\r\n

scrapy startproject tutorial<\/code><\/pre>\r\n
It will create a new folder tutorial and scrapy.cfg file<\/p>\r\n
scrapy.cfg  <\/strong>          # deploy configuration file<\/p>\r\n
    tutorial\/ <\/strong>            # project's Python module, you'll import your code from here<\/p>\r\n
        items.py <\/strong>         # project items definition file<\/p>\r\n
        middlewares.py <\/strong>   # project middlewares file<\/p>\r\n
        pipelines.py <\/strong>     # project pipelines file<\/p>\r\n
        settings.py <\/strong>      # project settings file<\/p>\r\n
        spiders\/ <\/strong>         # a directory where you'll later put your spiders<\/p>\r\n
Spiders are classes that you define and that Scrapy uses to<\/p>\r\n
    \r\n
  • scrape information from a website .<\/li>\r\n
  • how to follow links in the pages,<\/li>\r\n
  • how to parse the downloaded page content to extract data.<\/li>\r\n<\/ul>\r\n
    2. change directory to tutorial folder<\/p>\r\n
    cd tutorial<\/code><\/pre>\r\n
    3. Create a new file quotes_spider.py<\/strong> inside spiders directory.<\/p>\r\n
    from pathlib import Path\r\n\r\nimport scrapy\r\n\r\n\r\nclass QuotesSpider(scrapy.Spider):\r\n    name = \"quotes\"\r\n\r\n    def start_requests(self):\r\n        urls = [\r\n            'https:\/\/quotes.toscrape.com\/page\/1\/',\r\n            'https:\/\/quotes.toscrape.com\/page\/2\/',\r\n        ]\r\n        for url in urls:\r\n            yield scrapy.Request(url=url, callback=self.parse)\r\n\r\n    def parse(self, response):\r\n        page = response.url.split(\"\/\")[-2]\r\n        filename = f'quotes-{page}.html'\r\n        Path(filename).write_bytes(response.body)\r\n        self.log(f'Saved file {filename}')<\/code><\/pre>\r\n
    name:<\/strong> identifies the Spider. It must be unique within a project, that is, you can’t set the same name for different Spiders.<\/p>\r\n
    start_requests():<\/strong> must return an iterable of Requests which the Spider will begin to crawl from.<\/p>\r\n
    parse():<\/strong> handle the response downloaded for each of the requests made. The response parameter<\/strong> is an instance of TextResponse that holds the page content<\/p>\r\n
    4.Run the spider<\/strong><\/p>\r\n
    scrapy crawl quotes<\/code><\/pre>\r\n
    It will save two new files have been created: quotes-1.html and quotes-2.html <\/strong>in root directory.<\/p>\r\n
    background Working:<\/strong><\/p>\r\n
      \r\n
    • Scrapy schedules the scrapy.Request objects <\/strong>returned by the start_requests method of the Spider.<\/li>\r\n
    • instantiates the Response objects and calls the callback method associated with the request (in this case, the parse method)<\/li>\r\n<\/ul>\r\n
      parse() is Scrapy’s default callback method so we can use short code<\/strong> also<\/p>\r\n
      from pathlib import Path\r\nimport scrapy\r\nclass QuotesSpider(scrapy.Spider):\r\n    name = \"quotes\"\r\n    start_urls = [\r\n        'https:\/\/quotes.toscrape.com\/page\/1\/',\r\n        'https:\/\/quotes.toscrape.com\/page\/2\/',\r\n    ]\r\n    def parse(self, response):\r\n        page = response.url.split(\"\/\")[-2]\r\n        filename = f'quotes-{page}.html'\r\n        Path(filename).write_bytes(response.body)<\/code><\/pre>\r\n
      Extract Data<\/strong><\/em><\/p>\r\n
      Run the command <\/p>\r\n
      scrapy shell \"https:\/\/quotes.toscrape.com\/page\/1\/\"<\/code><\/pre>\r\n
      Now play with below commands<\/p>\r\n
      Using CSS Selector<\/strong><\/p>\r\n
      response.css('title') #return selectorList object\r\nresponse.css('title').getall() # return list of elements with content\r\nresponse.css('title::text').getall() #return list of all text from element\r\nresponse.css('title::text').get() # first result only\r\nresponse.css('title::text')[0].get() # first result only\r\nresponse.css('title::text').re(r'Quotes.*') # return all elements that start with Qoutes\r\nresponse.css('title::text').re(r'(\\w+) to (\\w+)')\r\nresponse.css(\"div.quote\")\r\nSUB Queries using css selector\r\nquote = response.css(\"div.quote\")[0]\r\ntext = quote.css(\"span.text::text\").get() #return first found element\r\ntags = quote.css(\"a.tag::text\").getall() #return list<\/code><\/pre>\r\n
      Using Xpath<\/strong><\/p>\r\n
      Xpaths : more powerfull then css selectors\r\nresponse.xpath('\/\/title') #return selectorList object\r\nresponse.xpath('\/\/title\/text()').get() # return text <\/code><\/pre>\r\n
      Details: <\/strong>https:\/\/docs.scrapy.org\/en\/latest\/topics\/selectors.html#topics-selectors<\/p>\r\n
      ________________________<\/p>\r\n
      Extract Data from our project <\/p>\r\n
      A Scrapy spider typically generates many dictionaries<\/strong> containing the data extracted from the page.<\/p>\r\n
      \"yield\" is a keyword used in generator functions to return a sequence of values one at a time.<\/p>\r\n
      from pathlib import Path\r\nimport scrapy\r\nclass QuotesSpider(scrapy.Spider):\r\n    name = \"quotes\"\r\n    start_urls = [\r\n        'https:\/\/quotes.toscrape.com\/page\/1\/',\r\n        'https:\/\/quotes.toscrape.com\/page\/2\/',\r\n    ]\r\n    def parse(self, response):\r\n        for quote in response.css('div.quote'):\r\n            yield {\r\n                'text': quote.css('span.text::text').get(),\r\n                'author': quote.css('small.author::text').get(),\r\n                'tags': quote.css('div.tags a.tag::text').getall(),\r\n            }<\/code><\/pre>\r\n
      Below command will generate a quotes.json file containing all scraped items, serialized in JSON.<\/p>\r\n
      scrapy crawl quotes -O quotes.json<\/code><\/pre>\r\n
      -O: overwrite the data with same file<\/p>\r\n
      save data using json lines, its append the data in previous file<\/strong><\/p>\r\n
      scrapy crawl quotes -o quotes.jsonl<\/code><\/pre>\r\n
      With Pagination:<\/strong><\/em><\/p>\r\n
      Get Attribute of elements<\/p>\r\n
      response.css('li.next a::attr(href)').get()\r\nOR\r\nresponse.css('li.next a').attrib['href']<\/code><\/pre>\r\n
      Scrape all https:\/\/quotes.toscrape.com\/<\/strong><\/p>\r\n
      import scrapy\r\nclass QuotesSpider(scrapy.Spider):\r\n    name = \"quotes\"\r\n    start_urls = [\r\n        'https:\/\/quotes.toscrape.com\/page\/1\/',\r\n    ]\r\n    def parse(self, response):\r\n        for quote in response.css('div.quote'):\r\n            yield {\r\n                'text': quote.css('span.text::text').get(),\r\n                'author': quote.css('small.author::text').get(),\r\n                'tags': quote.css('div.tags a.tag::text').getall(),\r\n            }\r\n        next_page = response.css('li.next a::attr(href)').get()\r\n        if next_page is not None:\r\n            next_page = response.urljoin(next_page)\r\n            yield scrapy.Request(next_page, callback=self.parse)<\/code><\/pre>","category_id":"11","is_private":"0","created_at":"2023-03-21T11:43:06.000000Z","updated_at":"2023-03-21T12:58:26.000000Z","category":{"id":11,"user_id":"1","name":"Scrapy Python","slug":"python-vk7t","parent_id":"5","created_at":"2023-03-21T09:29:28.000000Z","updated_at":"2023-03-21T09:29:43.000000Z"},"user":{"id":1,"name":"R GONDAL","email":"rizikmw@gmail.com","email_verified_at":null,"two_factor_confirmed_at":null,"current_team_id":"1","profile_photo_path":null,"created_at":"2023-03-12T10:49:33.000000Z","updated_at":"2025-01-10T12:59:00.000000Z","profile_photo_url":"https:\/\/ui-avatars.com\/api\/?name=R+G&color=7F9CF5&background=EBF4FF"}},"pageDesc":"We will grabe https:\/\/quotes.toscrape.com\/ 1.Run command below to start a scrapy project  scrapy startproject tutorial It will create a - Simple Grabing tutorial | Create Spider, Extract data, Save data in json, with pagination scraping  | scrapy cheatsheet Python (Updated: March 21, 2023) - Read more about Simple Grabing tutorial | Create Spider, Extract data, Save data in json, with pagination scraping  | scrapy cheatsheet Python at my programming site [SITE]","categories":[]}