Deploy with Flask

For the times you want to share your work!

Jun 10, 2021

Hey so um yeah, I’m back again. This week is special because its the kickoff so I want to cover a bit more so we can start deploying projects together (aiming to finish the Whole Foods scraper by the end of this weekend so you all can see what this newsletter is about learning to do) - as a lil nugget of info at the top here the newsletter will be structure in the following manner:

Monday - Scraping Deep (a guide to a method or process of scraping that will be used in the ByteScrape of the week),

Thursday - Deployment Concepts (how to share, save, run, and generally everything around your scraper),

Saturday - ByteScrape (a weekly project series that will have posted solutions the following week to solidify concepts - there will also be a little contest incentive where you can submit and get your name/pseudonym featured in the next ByteScrape),

Free subscriptions will typically get this content biweekly but for the beginning, I will be making everything accessible to all.

TLDR

The Upshot: Learn to build a flask app and have the foundational file structure to then be ready to deploy the app on a web hosting service (will be covered in the next newsletter). Bonus material at the bottom shows how to turn a Dataframe to HTML and have that render in a browser (all from your flask app!)

Use Case: Rather than share a csv or a GitHub link, have a webpage you can host that can scrape or run complex machine learning tasks on the backend after scraping your data in real-time. This is the way to share what you learn after you scrape/build it.

Blech enough of that ok now let’s get to the fun stuff - Flask. I’m sure you’re asking yourself why I am talking about some library that doesn’t have to do with scraping on a scraping newsletter - well, I want you to have the tools to show people, share your work, and be proud of it - so that’s what we can do with Flask.

Flask is a python package that is basically a web development tool, it allows you to deploy websites and create website logic in python - seems scary, to be honest, it’s a bit slow at the beginning but once you get the hang of it you’ll be totally fine.

So, of course, we need the all-powerful -

pip install flask

If there are dependency issues here, feel free to www.google.com or hit up the discord because I’m just going to assume this worked. Yay! To the code!

File structure, is key (and will help later) so here come some screenshots so you can see exactly how I’ve set up the folders.

Let’s talk about each one:

assets - this will be important in later projects when we want to make the displayed content ~ beautiful ~

models - these are the python scripts that you will run, kinda like the brain, and we keep them here so they’re all in a nice and organized place

templates - these are the templates for the output, think of the blueprints for a building where the assets (from above) are the furniture, painting, etc. - as I said, makes things ~beautiful~

app.py - this is the flask logic driver (in my own words) but it basically serves as the navigation panel between all files.

Now, you do not HAVE to follow this structure (but please please do, you will thank yourself infinitely at 2 am while debugging that you set it up this way so… I mean kinda do it)

Oof okok almost done with setup - for the last bit of non-python, you will need to navigate from your terminal to inside the flask_deployment folder. I have a Mac so I will show you how I did this in the terminal, if you have trouble, www.google.com is your friend again (too many possible setups that I can’t possibly give the instructions to each :( i’m sorry)

I open my terminal and type this command:

cd ~/Desktop/ScrapeWell/flask_deployment 
'''the elements between the backslashes are the folder path you can also find the "file path" of your flask_deployment folder by searching for how to do so in your operating system.'''

once we are in this folder, we are good to start coding and testing!

open the app.py file and proceed by writing the following code:

import flask from flask import Flask #  

app = Flask(__name__)  

@app.route("/") 
def index():
    return "Hello World!"

this basically tells our computer to print the text “Hello World” on an initial (front) page, but don't take my word for it let’s run it. Go back to the terminal/console and write:

flask run

!!!remember you must be in the “flask_deployment” folder!!! The output should be the following:

Ugh I swear every time I execute this command the red scares me and I think I’ve done something wrong but no. Ok, so nows the time to test out code, copy the “http://127.0.0.1:5000/” and paste it into your browser (I’ll be honest and say I have no idea if that number is the same for everyone but either way you see it in the output so copy whatever http://number yours shows). Et voila, or maybe you’ve done this before and you’re not surprised but you should see:

Exciting, I know right! Now here is the magic - let’s use some python syntax from the heavens to print the whole food list we scraped from the BeautifulSoup article. How do we do this? On to the next section we go.

Take a look at my file structure (note the location of each file) because this is how you “use” the scraper in your flask app.

now, we need to make a quick change to the ScrapeWell_Les_1.py file from the BeautifulSoup task but I’ve written the code here:

# this is the ScrapeWell_Les_1.py file!!!!!!!!!!! BEWARE
import requests
import bs4
from bs4 import BeautifulSoup

def go():
	page_url = "https://www.wholefoodsmarket.com/sales-flyer?store-id=10005"
	page_sourced = requests.get(page_url).content 
	html_content = BeautifulSoup(page_sourced, "html.parser")
	sale_items = html_content.findAll('h4', class_="w-sales-tile__product")
 	return (sale_item_titles)

#try not to leave anything outside this function

Why the change? Well, first because I didn’t realize we would be using this for a flask write up (eek i’m sorry) but actually we are just defining a function inside the script that can be “called” from the flask app (python jargon, but basically, just think about it like we put something inside the python that we can use to “trigger” it when we need it).

Now back to the flask app! We can change 1 line and now have our BeautifulSoup scraped sales right on the browser page!! (You need to quit the terminal task (basically flask just runs until you tell it to stop) and you can do this by pressing ctrl + c (again hard to know what hardware you all are using but you can find some info if that doesn't work on well, www.google.com). The terminal should look like this:

and it should stop. Open the app.py file now and insert the following code:

import flask
from flask import Flask

import models
from models import ScrapeWell_Les_1

app = Flask(__name__)

@app.route("/")
def index():
    sale_listings = ScrapeWell_Les_1.go()
    return str(sale_listings)

the “import models” is a magic feature of python that is very useful, hard to describe everything it does and if you want me to explain, just shoot me a discord message, email, or comment below. Notice we also “import ScrapeWell_Les_1”. This allows us to use that “trigger” that we put in place when we changed the file just a little bit ago.

OK phew, we are ready to run this now - as before flask run, and let’s go copy that http://number to the browser it’s exciting!!!

Sweet ok this is great, we can check everything there and, what’s cool is if you change the scraper (everything inside go() and return some other content, you’ll see it on the webpage too!)

At this point, you have a flask app that outputs your scraper right into the index of a webpage, this is great, you’re almost at the point where you can put this on the web (trust me one more of these and you’ll be able to deploy a web app that returns whatever you can scrape accessible for others on the web)

Speaking from the heart I just want to say:

The ability to deploy something on a webpage as a tool accessible for people to see is HUGE, people love to interact with these projects and you can share them on Twitter/Reddit/Facebook, anywhere you want! This is massive, whether it be for a portfolio or just to share, it makes a huge difference to people and takes the project to a whole new level (it’s also fun to see your work in a beautiful format)

Bonus: The df.to_html()

This is the quickest way of getting a table from your python code to render nicely on the screen (you can always make it prettier later but at this point, you will add no-frills yet)

If you were following along in the article above, we are going to keep everything the same but add two new lines in ScrapeWell_Les_1.py, the file now looks like this:

import requests
import bs4
from bs4 import BeautifulSoup
import pandas as pd

def go():
	page_url = "https://www.wholefoodsmarket.com/sales-flyer?store-id=10005"
	page_sourced = requests.get(page_url).content 
	html_content = BeautifulSoup(page_sourced, "html.parser")
	sale_items = html_content.findAll('h4', class_="w-sales-tile__product")
	sale_item_titles = [i.text for i in sale_items]

	return pd.DataFrame({'Sale Items':sale_item_titles}).to_html()

What did I do? Ok well, we added the pandas library – if you’re unfamiliar, I will be using it in future newsletters so it would be nice to get to know it (for now you can just copy and paste but this is an awesome library for data analysis). I then used the library to change a dataframe (basically a fancy python name for a table) into “HTML” text (basically a fancy string). Now, rerun the flask app with “flask run” and see what happens…

Prettier right? In a future letter, I will be addressing how to edit this table to make it look a bit nicer on the eyes, but even now it looks pretty good.

A challenge to attempt: If you’re advancing through these with ease, try to add another column to the dataframe to get the prices of the Whole Foods sale items next to the sale name - you will need to add lines to your scraper file, not the flask - if you have questions, you know where to find me

If you made it here, man I really appreciate it, I have fun writing these, and I hope you have fun reading them - if you have any comments/suggestions please let me know as I take any feedback to heart. As always have a good one and if you haven’t, please share these with your friends/anyone you think would enjoy them, in my world, the more the merrier!

Peace,

bs354