5-3 Crawling and scraping techniques

5-3 Crawling and scraping techniques#

Streamlit and and Playwright
Crawling and scraping techniques
Pagination, multiple pages, multiple items

Playwright + Streamlit#

Due to the multi-threaded nature of Streamlit, and playwright, there are some compatability on windows platforms.

https://discuss.streamlit.io/t/using-playwright-with-streamlit/28380

Therefore we recommend using writing the playwright code in a separate file, and then calling it from the streamlit app. using the code_helper.py module.

See:

5-3-streamlit.py #

and

5-2-2a.py #

for examples.

Playwright + Streamlit: Exchanging Data#

To send data into run_python_script we provide text that would be input(), and then we print() what should be returned.

get-page-text.py #

from playwright.sync_api import Playwright, sync_playwright, expect

def run(playwright: Playwright, url) -> None:
    browser = playwright.chromium.launch(headless=False)
    context = browser.new_context()
    page = context.new_page()
    page.goto(url)
    # get body content as text
    content = page.query_selector("body").text_content() 

    # ---------------------
    context.close()
    browser.close()

    return content


with sync_playwright() as playwright:
    # input from other program
    url = input()
    
    text = run(playwright, url)

    # output to other program
    print(text)

Challenge 5-3-1: Streamlit + playwright#

Use the get_page_text.py plus the run_python_script() in code_helper.py to create a streamlit app that takes a URL and returns the text of the page.

Build a dataset through scraping#

A common approach for web scraping is to build a dataset by scraping multiple pages.

In this example we will build a JSON dataset of course data from the SU course catalog.

Inputs:

Course numbers as a list IST 256, IST 101, etc…
current date as a string in YYYY-MM-DD format

Output:

list of dict as JSON
Example:

[
    { 'course': 'IST 256', 
      'title': 'Introduction to Python for the Information Profession', 
      'credits': 3,
      'description' : '.....'.
      'date': '2021-01-01' },
    { .... }
]

5-3-course-scraper.py #

Challenge 5-3-2: Build a dataset through scraping#

Build a JSON dataset of stock prices from the Yahoo Finance website.

https://finance.yahoo.com/quote/NET/

Inputs:

List of stock symbols in your portfolio: AAPL, AMZN, GM, HD, META, NET
current date eg. 2024-03-21

Output:

list of dict as JSON of stocks, date, and price at date
Example:

[
    { 'symbol': 'AAPL', 'price': 123.45, 'date': '2021-01-01' },
    { 'symbol': 'GOOG', 'price': 234.56, 'date': '2021-01-01' },
]

Save the file in the cache folder as {date}-stock-prices.json

5-3 Crawling and scraping techniques

Contents