{"flag":true,"single":true,"pageTitle":"Scrapy Basic tutorial to scrape a site using peewee, grabe and store data to database using pipelines","post":{"id":49,"user_id":"1","slug":"scrapy-basic-tutorial-to-scrape-a-site-using-peewee-grabe-and-store-data-to-database-using-pipelines-ollf","title":"Scrapy Basic tutorial to scrape a site using peewee, grabe and store data to database using pipelines","body":"<p><strong>Spider:<\/strong><\/p>\r\n<pre class=\"language-markup\"><code>import scrapy\r\nfrom ..items import TutorialItem\r\nfrom ..db_model import *\r\n\r\n\r\nclass QuotesSpider(scrapy.Spider):\r\n    name = \"quotes\"\r\n    url_lists = []\r\n    with database.atomic():\r\n        query = Links.select().where((Links.link.contains(\"pet-supplies\")) &amp; (Links.status == \"\"))\r\n        for data in query:\r\n            url_lists.append(data.link)\r\n            query = Links.update(status=\"fetched\").where(Links.id == data.id)\r\n            query.execute()  # Returns the number of rows that were updated.\r\n    start_urls = url_lists\r\n\r\n    def parse(self, response):\r\n        items = TutorialItem()\r\n        all_cards = response.css(\".col-xl-9 .card \")\r\n        page = response.url.split(\"\/\")\r\n        for quote in all_cards:\r\n            items['category'] = page[-2]\r\n            items['sub_category'] = page[-1]\r\n            items['link'] = quote.css(\".card-body::attr(href)\").extract_first()\r\n            items['title'] = quote.css(\".card-title::text\").get()\r\n            items['image'] = quote.css(\".avatar::attr(src)\").extract_first()\r\n            items['products'] = quote.css(\".card-text::text\").extract_first()\r\n            yield items\r\n<\/code><\/pre>\r\n<p><strong>2.Items<\/strong><\/p>\r\n<pre class=\"language-markup\"><code>import scrapy\r\nclass TutorialItem(scrapy.Item):\r\n    # define the fields for your item here like:\r\n    title = scrapy.Field()\r\n    image = scrapy.Field()\r\n    products = scrapy.Field()\r\n    category = scrapy.Field()\r\n    sub_category = scrapy.Field()\r\n    link = scrapy.Field()<\/code><\/pre>\r\n<p><strong>3.pipeline<\/strong><\/p>\r\n<pre class=\"language-markup\"><code>from itemadapter import ItemAdapter\r\nfrom db_model import *\r\nclass TutorialPipeline:\r\n    def process_item(self, item, spider):\r\n        print(\"pipeline\", item['title'], item['image'], item['products'])\r\n\r\n        insert_it = Products(link=item['link'], title=item['title'], category=item['category'],\r\n                             sub_category=item['sub_category'], image=item['image'], products_count=item['products'],status=\"1\")\r\n        insert_it.save()\r\n        return item<\/code><\/pre>\r\n<p><strong>4. activate pipeline in settings.py, <\/strong>when you add this code all data in items.py will be sent to pipeline<\/p>\r\n<pre class=\"language-markup\"><code>ITEM_PIPELINES = {\r\n   \"tutorial.pipelines.TutorialPipeline\": 300,\r\n}<\/code><\/pre>\r\n<p>&nbsp;<\/p>\r\n<p>Also please install peewee module to interact with database.<\/p>","category_id":"11","is_private":"0","created_at":"2023-03-25T17:17:20.000000Z","updated_at":"2023-03-25T17:17:20.000000Z","category":{"id":11,"user_id":"1","name":"Scrapy Python","slug":"python-vk7t","parent_id":"5","created_at":"2023-03-21T09:29:28.000000Z","updated_at":"2023-03-21T09:29:43.000000Z"},"user":{"id":1,"name":"R GONDAL","email":"rizikmw@gmail.com","email_verified_at":null,"two_factor_confirmed_at":null,"current_team_id":"1","profile_photo_path":null,"created_at":"2023-03-12T10:49:33.000000Z","updated_at":"2025-01-10T12:59:00.000000Z","profile_photo_url":"https:\/\/ui-avatars.com\/api\/?name=R+G&color=7F9CF5&background=EBF4FF"}},"pageDesc":"Spider: import scrapy from ..items import TutorialItem from ..db_model import *   class QuotesSpider(scrapy.Spider):     name = \"quotes\"     - Scrapy Basic tutorial to scrape a site using peewee, grabe and store data to database using pipelines (Updated: March 25, 2023) - Read more about Scrapy Basic tutorial to scrape a site using peewee, grabe and store data to database using pipelines at my programming site [SITE]","categories":[]}