There comes a point when learning Python where watching another tutorial isn’t enough.
You understand variables.
You’ve seen loops.
You know what a function is supposed to do.
You’ve followed someone else’s code and successfully made "Hello World" appear on your screen.
Great.
But I wanted to build something I could actually use.
So for my first proper Python SEO automation, I decided to keep things simple: take a list of URLs, check them automatically, and return useful information about each page.
Nothing revolutionary.
But that wasn’t the point.
The point was turning Python from something I was learning into something I could use.
I deliberately chose a boring SEO task
I didn’t want my first project to be an “AI-powered SEO agent.”
I wanted something boring.
SEO involves a lot of URL checking.
You might have a spreadsheet containing hundreds of URLs and need to know which return a 200 status code, which redirect and which are broken.
You might also want basic information from the page.
The title tag.
Meta description.
H1.
Canonical.
Things I’ve checked manually thousands of times during my career.
Doing that for five URLs isn’t a problem.
Doing it for 500 becomes annoying.
That’s exactly the type of problem I want Python to help me solve.
The idea was simple
Give Python a list of URLs.
For each URL:
Request the page.
Record the HTTP status code.
Extract some basic SEO elements.
Store the results.
Export everything into a file I can analyse.
In my head, this sounded incredibly straightforward.
Then I actually tried building it.
That’s when the learning started.
Making a request changed how I thought about crawling
One of the first things I needed Python to do was request a webpage.
That’s something I’ve talked about for years in technical SEO.
A crawler requests a URL.
The server responds.
You get a status code.
- 200
- 201
- 202
- 203
I’ve explained this process countless times.
But writing code that actually sends the request makes the concept feel different.
Suddenly you’re responsible for what happens next.
What if the website takes too long to respond?
What if the connection fails?
What if the URL redirects?
What if the server blocks the request?
What if the page doesn’t contain the element you’re expecting?
The neat diagram of request → response becomes much messier when you’re the one writing the instructions.
My script worked beautifully
For about five minutes.
Then I gave it URLs that weren’t as cooperative.
That’s when I discovered one of the most important lessons from my first automation:
Real-world data doesn’t behave itself.
A URL might fail.
A title tag might be missing.
A page might contain multiple H1s.
A redirect might lead somewhere unexpected.
A request might time out.
The HTML might not look anything like what your code expected.
My first version assumed everything would work.
That’s a dangerous assumption when you’re automating anything.
So I had to start thinking about errors.
Error handling suddenly made sense
I’d seen error handling while learning Python.
At first, it felt like another programming concept to remember.
Then my script failed halfway through a list of URLs.
Now I understood why it existed.
If I’m checking 1,000 pages, I don’t want URL number 237 to fail and stop the entire process.
I want the script to say:
Something went wrong here.
Record the error.
Move to URL 238.
That feels obvious now.
It wasn’t obvious when I was simply following tutorials.
That’s the difference between learning syntax and solving a problem.
Extracting SEO elements was surprisingly satisfying
Once I could retrieve the HTML, I started extracting information from it.
Page titles.
Meta descriptions.
H1s.
Canonicals.
These are things I normally see inside Screaming Frog or another SEO tool.
Seeing my own script return them was strangely satisfying.
Not because I’d built anything better than Screaming Frog.
I absolutely hadn’t.
Screaming Frog can do far more than my little script.
What mattered was understanding how some of that process works.
Request the document.
Parse the HTML.
Find an element.
Extract its value.
Store it.
Repeat.
A tool I’d used professionally for years suddenly felt slightly less magical.
Exporting the results was the moment it felt useful
Eventually, I had Python collecting information.
But information sitting inside a terminal isn’t particularly useful for most SEO work.
I wanted a spreadsheet.
So the next step was structuring the results and exporting them.
URL.
Status code.
Title.
Meta description.
H1.
Canonical.
Once I opened the output and saw everything organised into columns, something clicked.
I’d taken a repetitive SEO task and turned it into a process.
Input goes in.
Python does the work.
Structured data comes out.
That’s exactly what I wanted to learn.
Building it taught me more than watching tutorials
This was probably the biggest lesson.
You can watch someone explain a loop and think you understand loops.
Then you have to write one yourself.
You can read about functions.
Then you have to decide which part of your own script should become a function.
You can learn about handling errors.
Then your program crashes and suddenly error handling becomes extremely interesting.
Building something gives programming concepts context.
I stopped asking:
“What does this Python feature do?”
And started asking:
“Can this help solve the problem I’m having?”
That’s a much more useful way for me to learn.
AI helped — but I didn’t want it to build everything
I’m learning Python at an interesting time because AI can generate most of the code I need.
I could describe the automation, ask an AI tool to write it and probably have something functional fairly quickly.
That’s useful.
But it would defeat part of the reason I’m doing this.
I want AI to help me understand code, troubleshoot errors and suggest better approaches.
I don’t want to become someone who can only build a Python script while an AI tool is holding my hand.
So when AI gives me code, I’m trying to ask another question:
Can I explain what this is doing?
If I can’t, I haven’t finished learning from it yet.
I also learned not to automate something just because I can
There’s another important lesson here.
My script isn’t replacing professional SEO crawlers.
If I need to crawl a large website tomorrow, I’m probably opening Screaming Frog.
It already solves the problem extremely well.
Python becomes interesting when the workflow gets more specific.
Maybe I want to take crawl data and combine it with Search Console.
Maybe I want to run a very particular check across thousands of pages.
Maybe I want to pull information from several APIs.
Maybe I need to transform data in a way my existing tools don’t easily support.
That’s where I think programming becomes powerful for SEO.
Not replacing every tool.
Connecting them and filling the gaps between them.
My first automation wasn’t impressive
And I’m completely fine with that.
It didn’t use machine learning.
It didn’t have a beautiful interface.
Nobody is going to fund my URL checker startup.
It took a list of URLs, checked them, extracted some information and gave me a file.
But I built it.
More importantly, I understood much more about Python after building it than I did before.
And now my brain has started doing something slightly dangerous.
I’m looking at repetitive SEO tasks and thinking:
Could I automate that?
Now I want to make it better
There are plenty of directions I could take the project.
I could compare titles against recommended lengths.
Flag missing metadata.
Identify broken internal links.
Check indexability.
Compare canonicals.
Connect Search Console data.
Add Core Web Vitals data.
Group problems automatically.
Maybe eventually I’ll build something completely different.
That’s part of the fun.
I’m no longer learning Python because somebody told me it was a useful skill for SEO.
I’m starting to see the possibilities myself.
The biggest thing I built wasn’t the script
My first Python SEO automation was simple.
But it changed how I look at repetitive work.
Previously, my options were often:
Do it manually.
Use an SEO tool.
Build a spreadsheet formula.
Now there’s another possibility.
Build something.
I won’t always know how.
I’ll write bad code.
I’ll break things.
I’ll probably spend an hour automating a task that would have taken 20 minutes manually.
But that’s okay.
Right now, the goal isn’t saving time.
The goal is learning how to build things that eventually will.
