How do I build my own feed generator?

This guide covers the high-level concepts you need to consider when building a system that checks websites for new content on a schedule and turns it into RSS feeds for your own private use. It doesn’t include any actual code or step-by-step build instructions — just the ideas around finding data sources, extracting content, respecting servers, and hosting the output so LightWatch can subscribe to it.

This is for people who are comfortable with code or coding agents. I don’t offer support for any of the concepts here — it’s a starting point for your own DIY project. You can provide the URL for this page to a coding agent to get a project started with these considerations baked in.

Before you start, read Is it ethical to make my own feed? The rest of this guide assumes that stance: your feeds are for you alone, and your scraper behaves like you would in a browser.

What you’re building

OK, now that we’re all confident we’re not being a-holes, let’s get into it. You want a system that runs on a schedule, checks websites for new content, and turns that content into RSS feeds that LightWatch can subscribe to. The high-level pieces are:

1. A pipeline that fetches content and turns it into structured post data.
2. A feed writer that outputs valid RSS XML from that data.
3. A scheduler that runs the whole thing periodically.
4. A host that serves the generated feed files privately, somewhere LightWatch can reach.

If you’re only generating feeds from one or two sites, this can be a single script, but you might want to consider a more robust pipeline if you intend to do more than that.

The pipeline

Most content sources follow the same two-level pattern:

• An index — a page or endpoint that lists posts (a blog homepage, a gallery page, an API response).
• Articles — the individual posts linked from the index, which often have richer content than the index itself.

Sometimes the index contains everything you need. Other times the index just has titles and thumbnails, and the good stuff is on the article pages.

Design your pipeline as two agnostic steps:

1. Fetch the index and return a list of post objects (title, URL, date, thumbnail, etc.). How you fetch and parse the index is up to the implementation — it could be an API call, structured data extraction, or HTML parsing. What matters is that it returns the same structured result regardless.
2. Enrich each post by optionally following its URL and extracting additional content — higher resolution images, more media, better metadata. This step is additive: it layers data on top of what the index already provided rather than replacing it.

This separation keeps your system flexible. You can swap out the index fetcher for a different source without changing the enrichment step. You can skip enrichment entirely for sources where the index has everything. And you can add new sources without rearchitecting the whole thing.

Caching

Keep a local cache of posts you’ve already processed (a JSON file, a SQLite database, whatever works). On each run, check new posts against your cache and skip anything that’s already been processed. This makes each run faster and avoids putting unnecessary load on the server by re-fetching pages you’ve already seen.

A side effect of this is that you can also provide feeds with more content than the source if you keep the cached data around.

Filtering

You might want a filtering system that lets you exclude posts by keyword, user, tag, or other criteria. This can be handy for situations like “I want this feed but not these annoying users.” Building this in from the start is easier than adding it later. Consider filtering at the global and also the source level. They each have their benefits, but it’s more likely you’ll care per-source than globally.

Finding your data source

Scraping is great and all, but — actually no, scraping sucks. It’s slow, hard to do, and fragile. Avoid it if you can.

First check if there’s a better data source:

1. Public API — Some sites have a documented API you can use directly. Check for a developer portal or API docs. This is the most reliable option.
2. Internal API — Open the site in a browser and inspect the network traffic. Many sites load content from internal APIs that return structured JSON. If you can find one, use it. The data will be cleaner and less likely to break when the site updates its design.
3. Structured data in the page — Check the <head> for JSON-LD scripts, Open Graph tags, or microformats. Many sites embed structured metadata for SEO that you can extract without parsing the visual layout.
4. Sitemaps — Most sites publish a sitemap. They often group content pages by category and include publish dates you can use to order posts.
5. HTML parsing — If none of the above work, parse the HTML directly.

Whatever data source you use, the goal is the same: get structured post data into your pipeline.

HTML-specific concerns

If you’re resorting to HTML parsing, there are some specific things to watch for.

Extracting media

When parsing HTML for images, check for srcset attributes and extract the largest available size. Many sites serve responsive images, and the srcset will have much better versions than the src attribute alone.

Many sites also use lazy loading, which means the real image URLs aren’t in src or srcset at all — they’re usually in data attributes like data-src, data-srcset, data-lazy-src, etc. Build your image extraction with a hierarchy: prefer srcset and src first, but fall back to data attributes that look like they contain URLs.

For videos, when available, extract the poster image in addition to the video URL. LightWatch uses poster images if they are provided, rather than generating them, which will provide a quicker loading experience than a video with no preview. See the feed optimization guide for exactly how to include these in your RSS output.

Getting larger images

Many sites serve thumbnailed or downsized versions of images, but larger sizes are available at a predictable URL. Often it’s a simple string substitution — changing thumb to full, _small to _large, or w=400 to w=1600 in the URL. It’s worth inspecting a few image URLs from a site to see if there’s a pattern.

Some website platforms have standardized URL schemes for image sizes. If you can identify the platform (Shopify, WordPress, Squarespace, etc.), you can often reverse-engineer the URL to request the largest available version. This is worth building into your pipeline as a per-platform transform.

Server etiquette

Let’s go back to that not being a-holes thing. We should be respectful to the source servers. Remember, the goal is to emulate your own personal browsing. Smart servers let us know what they’re cool with us doing. And for the rest, we can make some default choices that keep us polite citizens.

Conditional requests

Every request should send If-None-Match (ETag) and If-Modified-Since headers if the server provided them on the previous response. If the server returns 304 Not Modified, skip parsing entirely. This reduces load for both you and the server.

Rate limiting

Start with a reasonable frequency for your routine checks. Once a day is enough for most sources.

If a response comes back with a 429 Too Many Requests status or a Retry-After header, stop and wait the specified time before trying again. Don’t retry immediately.

If the server sends Cache-Control: max-age=3600, it’s telling you the content won’t change for an hour. Don’t fetch more often than that.

Error handling

If the server returns a 5xx error, back off exponentially, or just stop until the next scheduled run. Don’t hammer a server that’s struggling.

Scope

If you only need the front page, don’t crawl the entire site. Fetch one page, extract what you need, and stop.

Platform coordination

If you’re generating multiple feeds from the same platform (e.g., several Instagram users or Tumblr blogs), build a shared client for that platform that coordinates rate limiting across all feeds. A full check across 20 feeds shouldn’t fire 20 requests simultaneously — stagger them and share rate limit state so one blocked request pauses the whole batch.

User-Agent and robots.txt

For personal use, using a browser User-Agent string to get around scraping protection is reasonable. You’re making the same requests a browser would, just on a schedule. A server that serves content to your browser shouldn’t get to decide that the same request from a script is forbidden.

If a site’s robots.txt blocks scrapers, that’s a gut call for you to make. My take is that it’s there to block bad actors, not one person who wants to see the content in a different app. As long as you’re emulating your own personal browsing, it’s fine.

Hotlinking images

For personal use, hotlinking is fine. You’re one person making the same requests a browser would. Keep in mind that if the source deletes those images, they disappear from your feed too. If you want to keep them, download the images and serve them yourself alongside the feed. As long as you’re not redistributing them, that’s fine too.

Output format

Now the good part. You’ve got your data. It’s time to build the feeds. Actually, this part is pretty easy. LightWatch is pretty robust at figuring out what content is available. If you put the content in the feed using standard HTML conventions, LightWatch will figure it out.

However, to be more specific, LightWatch supports RSS 2.0, Atom, and JSON Feed. I personally recommend RSS. See the feed optimization guide for the exact properties and media formats that LightWatch supports.

Running it

Scheduling

Your system needs to run on a schedule, on a machine that can also serve the feeds privately (see Hosting below). Some options:

• cron or launchd on a Mac, Raspberry Pi, or NAS that’s always on.
• A Shortcuts automation on a Mac that’s always on.
• cron on a rented cloud server (a VPS) that you’ve added to a private network like the one described below.

How often to run depends on how often the source updates. For most sources, every 24 hours is perfectly reasonable. You could tighten that to every 4–6 hours for content that updates more often. If you’re tempted to go closer to once an hour, consider whether it’s necessary. Sources that update that often are better sampled anyway.

Hosting

Once your script generates the feed files, they need to be somewhere LightWatch can fetch them. This part is a little trickier in the ethics department. The goal is to avoid redistribution, so the feeds need to stay private. That rules out the usual free static hosts — GitHub Pages, an S3 bucket, Netlify, and friends all publish your files to the whole internet. Instead, serve them from a device you own:

• A local server on your home network — Run a simple HTTP server on an always-on Mac or Raspberry Pi and point LightWatch at it using its .local address (e.g. http://my-mac.local:8080/beans.xml). LightWatch accepts plain http:// for .local addresses, so there’s no certificate to set up. The trade-off is that LightWatch can only check the feed while you’re on the same network. If you can live with that, this can be a great, easy solution.
• A private network that follows you around — A mesh VPN links your devices into a private network that works from anywhere. Your phone can reach the server at home from a coffee shop, but nobody else on the internet can. Tailscale is a great example, and its free plan covers personal use. However, the addresses your feeds will use aren’t .local ones, so LightWatch requires https:// and a valid certificate. Tailscale and similar services provide ways to serve your feeds over HTTPS. Just remember that your phone also needs to stay connected to the VPN for LightWatch to refresh in the background.

Go forth and build

Let’s be honest, this is the perfect project for a coding agent. It can yield great personal returns, but it’s complicated and low priority. Give your agent this URL and get it going! Or build it yourself if that’s your thing. Either way, don’t @ me about it. This is a guide to get you started, but I don’t offer support for things you build yourself.

If there’s one final thing I can leave you with it’s this: You can make anything a feed. LightWatch is a visual feed reader watcher. You can follow a lot of different kinds of content this way, and building your own feed generator is the step to unlocking that. I have a feed called Cool Beans that is just new releases from artisanal dry bean brands. You could make a feed from a SaaS company’s about page to see when they add employees, or follow a sneaker brand to see when the new hot shoes drop. The weirder and more niche the ideas you come up with, the more personally satisfying the results will be. Have fun.