Happy Holidays! 🎅🏿🎄

What?

It's time for me to admit it.

I'm a data hoarder.

Movies, music, twitch vods, tiktoks, you name it. I hoard all of it. However, I've kept running into the same friction point with this stuff. How do I both share and retain these files, at scale?

Well, that's where the idea of tom.christmas comes in.

My goal? Build a simple, fast, and secure way to download media from various platforms and either keep it, or share it with others.

The currently available stuff for this was in my case, bloated, riddled with ads or paywalls, or was simply dead. Yeah, YT-DLP exists, cobalt.tools exists, even my own site—museisfun.com exists, but I want something near-instant, something that didn't have a ton of dependencies, and most importantly, something simple enough that it'd run on my own containers.

tom.christmas is honestly not just a tool, but also a study in reverse engineering modern web APIs. I'm lowkey just bored too and this felt more interesting than watching another season of whatever is trending rn.

Where it all starts

Long story short, we all know that advertisers like YouTube, Twitch, and Instagram are pretty paranoid over who accesses their media. When say your or my browser loads a video, it picks up on so many things. User agent, session cookies, signatures embedded in the HTML, and a bunch of validation logic that says "yes, this is a real fucking user on our site."

This makes it incredibly difficult to simply "call their APIs" to get the media.

I tried just proxying the URLs at first, thinking that would be enough. It wasn't. The platform servers check for a multitude of things, including the Referer header, the user agent, sometimes even timing windows. It became clear pretty quickly that for scraping purposes, I need to be a bit more clever.

Signed URL Tunneling

By never exposing the raw media URL to the client, I can create an encrypted tunnel through my server.

The server can then:

1. Store the URL, cookies, and all the authentication stuff in an encrypted blob
2. Generate a signed URL with an expiry, allowing me to control the lifespan of the stream
3. When someone inevitably hits that URL, the server decrypts everything and proxies the stream with all the aformentioned headers

This ended up being pretty sick since it was stateless (no database of active streams), reasonably secure (you can't tamper with the encrypted data without breaking the signature), and cacheable if I ever wanted to add it later.

Not gonna lie, I didn't plan all of this upfront. It definitely evolved as I realized what was actually needed. The first version just stored everything in memory and broke whenever the server restarted, which was...not great, but that's how ya learn.

The Stream Manager

This object is responsible for creating, storing, and verifying the signed URLs.

1class StreamManager {
2    private streams: Map<string, StreamData> = new Map();
3}

To encrypt the stream data, we need a secret key and an initialization vector. So I declare all of that here.

1const streamId = randomBytes(16).toString("hex");
2const secret = randomBytes(32);
3const iv = randomBytes(16);
4
5const exp = Date.now() + STREAM_LIFESPAN * 1000;
6const streamDataWithExp = { ...mediaData, exp };

Then, it's of course essential to encrypt the entire metadata payload using those keys.

1const cipher = createCipheriv("aes-256-cbc", secret, iv);
2const encrypted = Buffer.concat([
3    cipher.update(JSON.stringify(streamDataWithExp), "utf8"),
4    cipher.final(),
5]);

Finally, creating an HMAC signature to prevent tampering:

1const hmac = createHmac("sha256", STREAM_SALT);
2hmac.update(
3    `${streamId},${exp},${iv.toString("base64url")},${secret.toString("base64url")}`
4);
5const sig = hmac.digest("base64url");
6
7streamCache.set(streamId, {
8    encrypted: encrypted.toString("base64"),
9    expiresAt: exp,
10});

The design here is inspired by how AWS does presigned URLs, except simpler and probably with a few more rough edges both because I wrote it and not some tenured security expert.

Anyway, I then cache the encrypted data instead of the plaintext URL, compute the signature over the parameters that the client will send back, and bind the expiration to the signature so someone can't just extend it arbitrarily. I'm sure there are things I haven't thought of, but its held up so far.

Verifying it

When a user (or the Discord bot) accesses the tunnel URL, the server verifies it.

1st, it recomputes the HMAC signature:

1const hmac = createHmac("sha256", STREAM_SALT);
2hmac.update(`${streamId},${exp},${iv},${sec}`);
3const expectedSig = hmac.digest("base64url");

Then it decrypts using the secret and IV provided by the client, and returns the stream data:

1const entry = streamCache.get(streamId);
2const decipher = createDecipheriv(
3    "aes-256-cbc",
4    Buffer.from(sec, "base64url"),
5    Buffer.from(iv, "base64url"),
6);
7
8const decrypted = Buffer.concat([
9    decipher.update(Buffer.from(entry.encrypted, "base64")),
10    decipher.final(),
11]);
12
13return JSON.parse(decrypted.toString("utf8")) as StreamData;

If you're interested, I'd consider making this project public, but for now, just message me on Discord if you want to talk more about it more in depth.

Handling platform specific stuff

This is where it got interesting. And by interesting I mean "staring at network tabs in devtools for hours trying to figure out why a request works in the browser but fails in code."

TikTok

TikTok's website is heavily obfuscated. The DOM is a nightmare of minified variable names and dynamically loaded content. I originally tried parsing the HTML directly and quickly gave up after finding variables like __webpack_require__.n(t)()(e.default) scattered everywhere.

But wait. I noticed something. When you view the source of any given TikTok video page, there's this massive JSON blob in a script tag called __UNIVERSAL_DATA_FOR_REHYDRATION__¹.

This is the data TikTok's React app uses to hydrate on the client side, and it contains literally everything.

The video URL, author info, stats, all of it. Jackpot.

1<script id="__UNIVERSAL_DATA_FOR_REHYDRATION__" type="application/json">
2  {
3    "__DEFAULT_SCOPE__": {
4      "webapp.video-detail": {
5        "itemInfo": {
6          "itemStruct": {
7            "video": { "playAddr": "https://..." },
8            "author": { "uniqueId": "username" }
9          }
10        }
11      }
12    }
13  }
14</script>

By extracting and parsing this JSON, we get direct access to playAddr (the raw MP4 URL) without having to execute JavaScript or deal with their API rate limits. And that's exactly what we're gonna do.

How it works

First, collect cookies and set the user agent:

1let cookie = "";
2
3function appendCookies(headers: Headers): void {
4    const set = headers.getSetCookie();
5    if (!set) return;
6    for (const c of set) {
7        cookie += `${c.name}=${c.value}; `;
8    }
9}
10
11const UA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36";

Resolve short links (like vt.tiktok.com/wutdafuc) to get the post ID:

1if (!postId && obj.shortLink) {
2    const res = await fetch(`https://vt.tiktok.com/${obj.shortLink}`, {
3        redirect: "manual",
4        headers: { "user-agent": UA }
5    });
6    const html = await res.text();
7    const match = html.match(/<a href="(https:\/\/[^"]+)"/);
8    if (match) {
9        const idMatch = match[1].match(/video\/(\d+)/);
10        if (idMatch) postId = idMatch[1];
11    }
12}

Extract the hydration data to get the video details (the "magic" when put together):

1const res = await fetch(`https://www.tiktok.com/@i/video/${postId}`, {
2    headers: { "user-agent": UA, cookie }
3});
4appendCookies(res.headers);
5const page = await res.text();
6
7const scriptStart = '<script id="__UNIVERSAL_DATA_FOR_REHYDRATION__" type="application/json">';
8const parts = page.split(scriptStart);
9const json = parts[1].split("</script>")[0];
10const data = JSON.parse(json);
11
12const videoDetail = data.__DEFAULT_SCOPE__?.["webapp.video-detail"]?.itemInfo?.itemStruct;

I also handle different content types (video vs. image post):

1const isPhoto = videoDetail.imagePost?.images;
2const playAddr = videoDetail.video?.playAddr;
3
4if (playAddr && !isPhoto) {
5    return {
6        type: "video",
7        url: playAddr,
8        filename: `tiktok_${videoDetail.author.uniqueId}_${postId}.mp4`,
9        cookie
10    };
11}
12
13if (isPhoto) {
14    const images = videoDetail.imagePost.images
15        .map((img: ImageData, i: number) => ({
16            url: img.imageURL.urlList[0],
17            filename: `tiktok_${videoDetail.author.uniqueId}_img_${i + 1}.jpg`
18        }));
19    return { type: "image_post", images, audio: videoDetail.music?.playUrl, cookie };
20}

Because then, by including cookies in the tunnel, we basically pretend to be a legitimate browser.

Twitch

We see that behind authentication and permissions, Twitch's REST API actually does support downloading clips.

However, for unauthenticated users, their web player has to fetch clip data somehow, and that's all happening through GraphQL. I opened up devtools, loaded a clip, and watched the network tab to see what the player was actually requesting.

Turns out you need two things: the clip metadata (title, broadcaster, available qualities) and a playbackAccessToken that proves you're allowed to access that specific quality.

Interestingly, Twitch uses persisted queries for this. That works by instead of sending say the full GraphQL query string, they just send a SHA256 hash that identifies the query on the server².

It's clever on their part (but not really) because it saves bandwidth and makes it slightly harder to reverse engineer, but the hash itself is just sitting there in the JavaScript bundle, so it's not really that hard to find if you know where to look.

Extraction

As always, set up the headers for the request (in this case, the GraphQL endpoint, and the client ID are unique to Twitch):

1const twitchGraphQLURL = "https://gql.twitch.tv/gql";
2const twitchClientId = "kimne78kx3ncx6brgo4mv6wki5h1ko";
3
4const headers = {
5    "Client-ID": twitchClientId,
6    "Content-Type": "application/json",
7    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
8};

Then, get the metadata with a GraphQL query:

1const queryBody = {
2    query: `{
3        clip(slug: "${input.clipId}") {
4            broadcaster { login }
5            durationSeconds
6            id
7            title
8            videoQualities { quality, sourceURL }
9        }
10    }`,
11};
12
13const metadataRes = await fetch(twitchGraphQLURL, {
14    method: "POST",
15    headers,
16    body: JSON.stringify(queryBody),
17});
18
19const clip = (await metadataRes.json())?.data?.clip;

Fetch the access token using a persisted query²:

1const tokenRes = await fetch(twitchGraphQLURL, {
2    method: "POST",
3    headers,
4    body: JSON.stringify([{
5        operationName: "VideoAccessToken_Clip",
6        variables: { slug: input.clipId },
7        extensions: {
8            persistedQuery: {
9                version: 1,
10                sha256Hash: "36b89d2507fce29e5ca551df756d27c1cfe079e2609642b4390aa4c35796eb11",
11            },
12        },
13    }]),
14});
15
16const accessToken = (await tokenRes.json())[0]?.data?.clip?.playbackAccessToken;

Construct the final URL with the access token and its signature:

1const mediaUrl = `${selectedQuality.sourceURL}?${new URLSearchParams({
2    sig: accessToken.signature,
3    token: accessToken.value,
4})}`;

SoundCloud

SoundCloud's API needs a client_id parameter for basically every request. The annoying part? This ID isn't just sitting in a meta tag somewhere. It's buried in one of like thirty different script files, and they rotate it pretty frequently. Sometimes weekly, sometimes less.³

My first attempt was to hardcode the client ID and just update it manually when it broke. That lasted about three days before I got tired of it. So instead, I crawl the main page, find the current script version (which SoundCloud helpfully exposes in window.__sc_version) LMFAO, and then pattern match through their bundles looking for where they construct API URLs with the client ID. Then cache it until the version changes.

Client ID Discovery

1const cachedID = { version: '', id: '' };
2
3async function findClientID(): Promise<string | null> {
4    const sc = await fetch('https://soundcloud.com/').then(r => r.text());
5    const versionMatch = sc.match(/<script>window\.__sc_version="([0-9]{10})"<\/script>/);
6    const scVersion = versionMatch[1];
7
8    if (cachedID.version === scVersion && cachedID.id) {
9        return cachedID.id;
10    }
11
12    const scripts = sc.matchAll(/<script.+src="(.+?)">/g);
13    let clientid;
14
15    for (const script of scripts) {
16        const url = script[1];
17        if (!url.startsWith('https://a-v2.sndcdn.com/')) continue;
18
19        const scrf = await fetch(url).then(r => r.text());
20        const idMatch = scrf.match(/\("client_id=([A-Za-z0-9]{32})"\)/);
21        if (idMatch[1]) {
22            clientid = idMatch[1];
23            break;
24        }
25    }
26
27    if (clientid) {
28        cachedID.version = scVersion;
29        cachedID.id = clientid;
30    }
31
32    return clientid;
33}

Track Resolution

Query SoundCloud's API to get the track details:

1const resolveURL = new URL("https://api-v2.soundcloud.com/resolve");
2resolveURL.searchParams.set("url", link);
3resolveURL.searchParams.set("client_id", clientId);
4
5const json = await fetch(resolveURL).then(r => r.json());

I do this to handle various content restrictions on SoundCloud:

1if (json.duration > durationLimit * 1000) {
2    return { error: "content.too_long" };
3}
4
5if (json.policy === "BLOCK") {
6    return { error: "content.region" };
7}
8
9if (json.policy === "SNIP") {
10    return { error: "content.paid" };
11}

Then, fetch the actual stream URL:

1const fileUrl = new URL(selectedStream.url);
2fileUrl.searchParams.set("client_id", clientId);
3if (json.track_authorization) {
4    fileUrl.searchParams.set("track_authorization", json.track_authorization);
5}
6
7const file = await fetch(fileUrl)
8    .then(async r => {
9        const data = await r.json();
10        return data.url ? new URL(data.url) : null;
11    });

The Tunnel

When someone accesses the tunnel URL, the server must:

1. Decrypt and verify the payload
2. Fetch from the original platform with the correct headers
3. Stream the response back without loading everything into memory first

This is why it's called a "tunnel".

How

Validate and decrypt the payload to verify its authenticity:

1const { id, sig, iv, sec, range } = query;
2
3if (!id || !sig || !iv || !sec) {
4    set.status = 400;
5    return { error: "Missing required parameters" };
6}
7
8const streamData = await streamManager.verifyAndDecrypt(
9    id as string,
10    sig as string,
11    exp,
12    iv as string,
13    sec as string,
14);

Handle range requests for video seeking (so clients can seek to a specific time in the video):

1const clientRange = typeof range === "string" && range.length > 0
2    ? range
3    : incomingRequest.headers.get("range") || "bytes=0-";

Proxy the request to the original platform:

1const { body, headers, statusCode } = await request(streamData.urls, {
2    headers: {
3        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
4        Cookie: streamData.headers.cookie,
5        Referer: originalUrl,
6        Range: clientRange,
7    },
8});

Determine content type and build the response:

1let contentType: string;
2if (streamData.type === "audio_only") {
3    if (streamData.service === "soundcloud") {
4        contentType = streamData.isHLS ? "application/vnd.apple.mpegurl"
5            : streamData.bestAudio === "mp3" ? "audio/mpeg" : "audio/opus";
6    } else {
7        contentType = "audio/m4a";
8    }
9} else {
10    contentType = "video/mp4";
11}
12
13const responseHeaders: Record<string, string> = {
14    "Content-Type": contentType,
15    "Content-Disposition": `inline; filename="${streamData.filename}"`,
16    "Accept-Ranges": "bytes",
17};
18
19return new Response(body as ReadableStream<Uint8Array>, {
20    status: statusCode,
21    headers: responseHeaders,
22});

The key thing here is streaming. We don't load the entire video into memory. That would be a disaster for a 500MB file. Instead, tom.christmas opens a stream from the selected source and pipes it directly back to the client. This was one of those things that seemed obvious in hindsight but took me a while to get right. The first version buffered everything and the server just died silently whenever someone tried to download a long clip.

---

Exposing this through a Discord bot

At the end of building the MVP with the original TikTok source, I decided to add a Discord bot to the mix. It's a lot more accessible this way. The bot simply sits on top of this entire system by exposing the tom.christmas APIs through Discord's application layer.

Here's a flowchart overview of how the Discord bot processes messages:

Extracting URLs

I use simple regex to extract URLs from the message content. It's honestly nothing fancy, just a few patterns to match the different URL types.

1const extractCandidates = (content: string) => {
2    const tiktok = new Set<string>();
3    const soundcloud = new Set<string>();
4    const twitch = new Set<string>();
5
6    const directUrls = content.match(/https?:\/\/[^\s<>()\[\]]+/gi) ?? [];
7    const extraTikTok = content.match(/(?:www\.)?(?:vt|vm)?\.?tiktok\.com\/\S+/gi) ?? [];
8    const extraSoundCloud = content.match(/(?:www\.)?soundcloud\.com\/\S+|on\.soundcloud\.com\/\S+/gi) ?? [];
9    const extraTwitch = content.match(/(?:www\.)?(?:clips\.)?twitch\.tv\/\S+/gi) ?? [];
10
11    const candidates = [...directUrls, ...extraTikTok, ...extraSoundCloud, ...extraTwitch];
12
13    for (const raw of candidates) {
14        const normalized = normalizeCandidate(raw);
15        try {
16            const url = new URL(normalized);
17            if (TIKTOK_HOSTS.has(url.hostname)) tiktok.add(normalized);
18            else if (SOUNDCLOUD_HOSTS.has(url.hostname)) soundcloud.add(normalized);
19            else if (TWITCH_HOSTS.has(url.hostname)) twitch.add(normalized);
20        } catch { continue; }
21    }
22
23    return { tiktok: [...tiktok], soundcloud: [...soundcloud], twitch: [...twitch] };
24};

Re-encode we must

Discord has strict file size limits depending on the server boost level. A 1080p Twitch clip at 60fps definitely exceeds the limit. The bot handles such cases by re-encoding using FFmpeg, on the server, before sending it to the client:

1const reencodeVideo = async (buffer: Buffer, filename: string) => {
2    const tempDir = await fs.mkdtemp(join(tmpdir(), "get-tt-"));
3    const inputName = join(tempDir, `input_${filename.replace(/[\\/]/g, "_")}`);
4    const outputName = join(tempDir, "output.mp4");
5
6    await fs.writeFile(inputName, buffer);
7
8    await new Promise<void>((resolve, reject) => {
9        ffmpeg()
10            .input(inputName)
11            .outputOptions([
12                "-c:v libx264",
13                "-preset veryfast",
14                "-crf 30",
15                "-maxrate 900k",
16                "-bufsize 1800k",
17                "-vf scale=min(854\\,iw):-2",
18                "-c:a aac",
19                "-b:a 96k",
20                "-movflags +faststart",
21            ])
22            .on("end", resolve)
23            .on("error", reject)
24            .save(outputName);
25    });
26
27    const outputBuffer = await fs.readFile(outputName);
28    await fs.rm(tempDir, { recursive: true, force: true });
29    return outputBuffer.length > 0 ? outputBuffer : null;
30};

The FFmpeg options here took some time to get just right. veryfast preset because nobody wants to wait five minutes for a video to load, crf 30 which is honestly on the low-quality side but keeps file sizes reasonable, and cap the bitrate at 900k to make absolutely sure I stay under Discord's limits. The scale filter downscales to around 480p width while preserving aspect ratio, which looks fine on mobile where most people are on Discord anyway.

Sending to Discord

Finally, I send the attachment to the channel.

1const sendAttachment = async (tunnel: string, data: ExtractResponse, message: Message) => {
2    const response = await fetch(tunnel);
3    if (!response.ok) return;
4
5    let buffer = Buffer.from(await response.arrayBuffer());
6    let filename = getFilename(data, response.headers.get("content-type"));
7
8    if (response.headers.get("content-type")?.includes("video/")
9        && buffer.length > DISCORD_MAX_UPLOAD_BYTES) {
10        const reencoded = await reencodeVideo(buffer, filename);
11        if (reencoded) {
12            buffer = reencoded;
13            filename = getCompressedFilename(filename);
14        }
15    }
16
17    const attachment = new AttachmentBuilder(buffer, { name: filename });
18    try {
19        await message.reply({ files: [attachment] });
20    } catch {
21        await message.channel.send({ files: [attachment] });
22    }
23};

---

Deploying

The system runs on Bun with ElysiaJS.

The architecture is a pretty simple one, with the API and the web frontend being served from the same Elysia instance.

I use Bun's blazing fast zig-native streaming API to avoid memory bloat and Elsyia to handle the routing and serving of the static files.

1app.listen(
2    { port: env.PORT, hostname: env.HOSTNAME },
3    (server) => {
4        console.log(`Elysia started at ${server.url}`);
5    },
6);

The application is deployed to Railway using their Docker BuildKit. After setting up the application, I wrote a Dockerfile to essentially bundle everything together and run in a container. The bot is an executable built with Bun, and the server is a regular Bun application, serving the static client files and API.

1FROM oven/bun:1.2.15 AS builder
2WORKDIR /app
3COPY bun.lock package.json turbo.json biome.json bunfig.toml ./
4COPY client/package.json client/biome.json client/tsconfig*.json client/components.json ./client/
5COPY server/package.json server/biome.json server/tsconfig.json ./server/
6COPY bot/package.json bot/tsconfig.json ./bot/
7RUN bun install --frozen-lockfile
8COPY client/index.html client/vite.config.ts ./client/
9COPY client/src ./client/src
10COPY client/public ./client/public
11COPY server/src ./server/src
12COPY bot/index.ts ./bot/
13RUN bun run --cwd client build
14RUN bun build --compile --target=bun-linux-x64 --minify ./bot/index.ts --outfile ./bot/bot
15FROM oven/bun:1.2.15-slim AS runner
16WORKDIR /app
17ENV NODE_ENV=production \
18    PORT=3000 \
19    HOSTNAME=0.0.0.0
20COPY --from=mwader/static-ffmpeg:latest /ffmpeg /usr/bin/ffmpeg
21COPY --from=mwader/static-ffmpeg:latest /ffprobe /usr/bin/ffprobe
22RUN chmod +x /usr/bin/ffmpeg /usr/bin/ffprobe
23COPY --from=builder /app/bun.lock /app/package.json /app/turbo.json ./
24COPY --from=builder /app/node_modules ./node_modules
25COPY --from=builder /app/server ./server
26COPY --from=builder /app/client/dist ./client/dist
27COPY --from=builder /app/bot/bot ./bot/bot
28RUN chmod +x ./bot/bot
29EXPOSE 3000
30CMD ["sh", "-c", "./bot/bot & bun server/src/index.ts"]

What's Next?

Pretty much just scaling the sources to more platforms! Instagram Reels would be nice but their obfuscation and security is tighter. I want to add better HLS / MP4 handling for platforms that use adaptive streaming. I'd also like to use a non-native audio player.

Maybe metadata enrichment too so I actually pull full artist and producer info in SoundCloud's case. But honestly, this already does what I need it to do, so these are all "if I get bored again" additions.

---

Tbh, this christmas project ended up being A LOT more interesting than I expected. I was telling my friend Devine this the other day, when people ask me "Why do you love frontend so much?" I tell them that this exact ubiquity of frontend is what makes it so interesting.

And for people out there reading this and struggling to ship their own project(s):

Remember that not every project needs to solve a massive problem. Literally, sometimes the best ones are just the ones where you learn a ton and end up with something useful at the end. And I believe this project is a clear example of that.

The stack

• Bun
• ElysiaJS
• Discord.js

Footnotes

1. The __UNIVERSAL_DATA_FOR_REHYDRATION__ script is a pattern used by many server-side rendered apps (Next.js, Nuxt, etc.) to avoid duplicate API calls. It's meant for the browser's hydration process but is trivially accessible via parsing. ↩

2. Twitch's persisted query hash system is documented in their GraphQL explorer. The specific hash for VideoAccessToken_Clip has been stable for years, suggesting it's a core operation unlikely to change soon. ↩ ↩²

3. SoundCloud's client ID rotation is the platform's way of limiting third party API usage. However, the client ID is embedded in publicly served JavaScript, making it discoverable and cacheable. ↩