Welcome back to our journey in video scraping! In the previous lesson, you learned how to download videos from public Google Drive links using Go. In this lesson, we will expand your skills by downloading videos from LinkedIn using Go's standard library and the powerful open-source tool yt-dlp. You'll learn how to recognize LinkedIn video URLs, validate them, and download publicly accessible video files directly to your local machine.

In this lesson, you will:

• Identify and validate a range of LinkedIn URLs.
• Understand how to use yt-dlp to download video files from LinkedIn posts.
• Download video files from LinkedIn using Go's standard library and yt-dlp.
• Handle temporary files and consider important legal aspects when downloading videos.

Our goal is to download videos from LinkedIn posts using Go. The first step is to recognize valid LinkedIn URLs that point to video content. LinkedIn video URLs can appear in several formats, such as:

• Full activity: https://www.linkedin.com/feed/update/urn:li:activity:VIDEO_ID
• Post-based: https://www.linkedin.com/posts/USERNAME_activity-VIDEO_ID

Recognizing these URL structures is essential for starting the download process. Once a valid URL is detected, we will use yt-dlp to fetch and download the video file.

Before you can use the code in this lesson, you need to have yt-dlp installed on your system. yt-dlp is a command-line program that can download videos from a wide variety of sites, including LinkedIn.

Why use yt-dlp?
LinkedIn does not provide direct video file links in the page HTML, and the URLs can be protected or obfuscated. yt-dlp is a robust, open-source tool that can extract and download videos from many platforms, handling authentication, cookies, and video formats for you.

To install yt-dlp, run one of the following commands in your terminal:

On macOS (using Homebrew):

brew install yt-dlp

On Linux (using pip):

pip install -U yt-dlp

On Windows:

• Download the latest Windows executable from the yt-dlp releases page.
• Or, if you have Python, run:
  pip install -U yt-dlp

Make sure yt-dlp is available in your system's PATH so that Go can invoke it.
If you are using our CodeSignal IDE, yt-dlp is already installed and available for you—no setup required.

The first step in our workflow is to check if a given URL is a LinkedIn video post. This is important to avoid running the downloader on unsupported or invalid URLs.

Here's how you can do this in Go:

package utils

import (
    "net/url"
    "strings"
)

// isValidDomain checks if the host exactly matches or is a subdomain of the expected domain
func isValidDomain(host, expectedDomain string) bool {
    host = strings.ToLower(host)
    // Remove port if present
    if idx := strings.Index(host, ":"); idx != -1 {
        host = host[:idx]
    }
    // Exact match or valid subdomain
    return host == expectedDomain || strings.HasSuffix(host, "."+expectedDomain)
}

// IsLinkedInURL checks if the URL is a valid LinkedIn post URL
func IsLinkedInURL(urlStr string) bool {
    parsed, err := url.Parse(urlStr)
    if err != nil {
        return false
    }

    validPaths := []string{
        "/feed/update/urn:li:activity:",
        "/posts/",
    }

    isLinkedInDomain := isValidDomain(parsed.Host, "linkedin.com")
    hasValidPath := false

    for _, path := range validPaths {
        if strings.Contains(parsed.Path, path) {
            hasValidPath = true
            break
        }
    }

    return isLinkedInDomain && hasValidPath
}

Explanation:

• The isValidDomain helper function performs secure domain validation by checking for exact matches or valid subdomains, preventing host spoofing attacks (e.g., linkedin.com.evil.com).
• The function parses the input URL.
• It checks if the host is exactly linkedin.com or a valid subdomain like www.linkedin.com.
• It checks if the path matches known LinkedIn video post patterns.
• Returns true if both conditions are met.

This function helps ensure that only valid LinkedIn video URLs are processed in the next step.

After validating the LinkedIn URL, the next step is to download the video. Instead of manually parsing HTML (which is brittle and often fails due to LinkedIn's dynamic content), we'll use the robust open-source tool yt-dlp from Go. This approach works for a wide range of LinkedIn video posts and handles the extraction for you.

package utils

import (
    "errors"
    "fmt"
    "os"
    "os/exec"
    "path/filepath"
)

// DownloadLinkedInVideo uses yt-dlp to download the video and returns the path
func DownloadLinkedInVideo(link string) (string, error) {
    if !IsLinkedInURL(link) {
        return "", errors.New("invalid LinkedIn video URL")
    }

    // Make temporary download directory
    tempDir, err := os.MkdirTemp("", "linkedin_video")
    if err != nil {
        return "", fmt.Errorf("failed to create temp dir: %w", err)
    }

    // Set output template
    outputTemplate := filepath.Join(tempDir, "%(title)s.%(ext)s")

    // Build base command arguments
    args := []string{
        "-f", "bestvideo+bestaudio/best",
        "--merge-output-format", "mp4",
        "-o", outputTemplate,
        "--quiet",
        "--no-warnings",
    }

    // Optional: attach cookies if needed
    if _, err := os.Stat("cookies.txt"); err == nil {
        args = append(args, "--cookies", "cookies.txt")
    }

    // Add separator and URL
    args = append(args, "--", link)

    // Use bestvideo+bestaudio to ensure we get both video and audio streams
    // This is critical for transcription. Fall back to 'best' if separate streams aren't available
    cmd := exec.Command("yt-dlp", args...)

    // Run the command
    out, err := cmd.CombinedOutput()
    if err != nil {
        return "", fmt.Errorf("yt-dlp failed: %v\nOutput: %s", err, string(out))
    }

    // Find downloaded file
    entries, err := os.ReadDir(tempDir)
    if err != nil || len(entries) == 0 {
        return "", errors.New("downloaded file not found")
    }

    // Return full path to the first file
    return filepath.Join(tempDir, entries[0].Name()), nil
}

Explanation:

• The function first checks if the URL is a valid LinkedIn video post.
• It creates a temporary directory for the download, ensuring your workspace stays clean.
• It constructs a yt-dlp command to download the video, using a robust format selector.
• If a cookies.txt file is present, it is used for authentication (useful for downloading private or restricted videos).
• The -- separator is added before the URL to prevent URLs starting with - from being interpreted as flags.
• The command is run, and the function checks for errors.
• It returns the path to the downloaded video file.

Why this format string?
The -f bestvideo+bestaudio/best format selector ensures we download both video and audio streams (essential for transcription). If LinkedIn doesn't provide separate streams, it falls back to the best single-file format. The --merge-output-format mp4 ensures the final output is always an mp4 file, regardless of the source formats.

Why use a temporary directory?
This keeps your downloads organized and avoids cluttering your project directory. You can later move or process the file as needed.

What about authentication?
If you need to download private videos (e.g., from your own account), you can export your browser cookies to a cookies.txt file and place it in your working directory. yt-dlp will use these cookies to authenticate your session.

Being able to download videos from LinkedIn using Go and yt-dlp allows you to collect educational content, access videos offline, and back up your own posts. This approach is robust, maintainable, and leverages the strengths of both Go and the open-source community.

Legal Note:
Always ensure you comply with LinkedIn's terms of service and copyright laws when downloading content. Only download videos you have rights to access or for which you have permission.

Now that you know how to detect and download LinkedIn videos with Go and yt-dlp, try the practice section to reinforce your understanding with hands-on exercises.