Raspberry Pi Usenet: Turn Your Pi into an Always-On Machine

Expanding Your Raspberry Pi Download Station with Usenet

Previously, we demonstrated the process of configuring a Raspberry Pi as a continuous BitTorrent server. This setup is designed to minimize energy consumption while maintaining optimal tracker ratios.

Now, we will expand upon this foundation by integrating Usenet access. This addition will transform your Raspberry Pi into a versatile and complete downloading solution.

Why Combine BitTorrent and Usenet?

Utilizing both BitTorrent and Usenet provides a more robust downloading experience. Each method has its strengths, and combining them offers greater access to content and improved download speeds.

Usenet is particularly valuable for accessing files that may be difficult to find or slow to download via BitTorrent.

Setting Up Usenet Access on Your Raspberry Pi

Integrating Usenet requires a Usenet service provider. Numerous providers offer varying levels of speed, retention, and features.

Once you've selected a provider, you'll need to configure a Usenet client on your Raspberry Pi. Popular options include NZBGet and SABnzbd.

Configuring Your Usenet Client

The configuration process will vary depending on the client you choose. Generally, you'll need to enter your Usenet provider's server address, username, and password.

Ensure your firewall allows outbound connections to the Usenet server. Proper configuration is crucial for optimal performance.

Downloading with NZB Files

Usenet downloads are typically initiated using NZB files. These files contain information about the files you want to download, including their names, sizes, and locations on the Usenet servers.

You can find NZB files using dedicated search engines or Usenet indexers.

Integrating with Your BitTorrent Setup

Your Raspberry Pi can now handle both BitTorrent and Usenet downloads concurrently. This creates a powerful and efficient downloading hub.

Consider automating the download process using scripts or scheduling tools to maximize efficiency.

Benefits of a Combined Approach

Increased content availability.
Faster download speeds for certain files.
A more versatile downloading solution.
Optimized resource utilization on your Raspberry Pi.

By adding Usenet access to your Raspberry Pi BitTorrent server, you've created a comprehensive and cost-effective downloading solution.

The Advantages of Utilizing a Raspberry Pi for Downloads

As previously discussed in our article, “How to Turn a Raspberry Pi into an Always-On BitTorrent Box,” a key motivation for transferring your downloading tasks to a Raspberry Pi is the substantial reduction in energy consumption. Substituting a Raspberry Pi for a conventional computer functioning as a home server or download station will result in significant cost savings.

Referring back to “The How-To Geek Guide to Measuring Your Energy Use,” it was noted that a typical office server can accumulate approximately $200 in electricity costs annually. In contrast, a Raspberry Pi’s annual energy expenditure is roughly $3.

Even when factoring in the addition of external hard drives, exceeding $10 per year in energy usage remains a challenge.

Although Usenet downloads are a one-way process – lacking the seeding, trackers, and ratio requirements of peer-to-peer networks like BitTorrent – maintaining 24/7 operation still proves advantageous.

For instance, if you rely on Usenet to acquire your preferred television programs, a consistently active Raspberry Pi download system will immediately obtain them upon release.

Furthermore, employing an always-on download box alleviates the strain on your primary computer. You won’t need to keep your desktop running overnight to complete a file, nor will you have to postpone gaming due to ongoing download unpacking and verification.

Continue reading to discover how to transform your Raspberry Pi into a highly efficient, quiet, and dedicated Usenet downloading device.

Essential Requirements for the Project

This guide presumes you possess a Raspberry Pi with the Raspbian operating system installed. You should also be capable of accessing the device, either through a direct connection with a monitor and keyboard or remotely using SSH and VNC. A USB drive, or multiple drives, connected to the Pi is also necessary.

Prerequisite Guides

Should you require foundational knowledge in these areas, we recommend reviewing the following guides in the sequence presented:

The HTG Guide to Getting Started with Raspberry Pi
How to Configure Your Raspberry Pi for Remote Shell, Desktop, and File Transfer
How to Turn a Raspberry Pi into a Low-Power Network Storage Device

Completion of the first tutorial is vital. The second is optional, though remote access significantly simplifies management of this project, making it ideal for a headless setup. The most crucial aspect of the third tutorial involves setting up and configuring the hard drive for automatic mounting upon system startup.

Understanding Usenet

For those unfamiliar with Usenet, we advise reviewing this tutorial before proceeding:

The How-To Geek Guide to Getting Started with Usenet

In brief, Newshosting is a highly recommended provider within the Usenet community.

If you already have a Usenet account with a reputable provider, you are well-prepared. However, if you lack an account, it is essential to consult our guide to gain a comprehensive understanding. Unlike torrenting, where public trackers can be utilized, dependable and free public Usenet servers are unavailable.

A subscription with a reliable provider is a necessity—our guide offers general information about Usenet and guidance on selecting a suitable provider.

With all preparatory materials reviewed and the Raspberry Pi properly configured, you can now begin transforming it into a quiet and energy-efficient downloading machine.

Updating Apt-Get and Installing SABnzbd

Initially, the apt-get installer needs to be updated and upgraded. If you recently completed the guide, "How to Turn a Raspberry Pi into an Always-On BitTorrent Box," this step can be bypassed, as the update and upgrade were already performed.

Execute the following commands in the terminal:

sudo apt-get update

sudo apt-get upgrade

Allow sufficient time for the process to complete, especially if it has been a while since the last update.

With the system updated, the installation of SABnzbd can begin. The choice of SABnzbd warrants explanation. For a minimal setup with low resource usage, command-line tools such as NZBGet could be considered.

However, this comes at the cost of the extensive third-party applications, plugins, and integrations available with a robust tool like SABnzbd.

Testing was conducted on the platform, and while NZBGet demonstrably uses fewer system resources, the benefits of SABnzbd’s refined interface and features were deemed more valuable.

To initiate the SABnzbd installation, use the following command in the terminal:

sudo apt-get install sabnzbdplus

This command installs the necessary dependencies for SABnzbd, including Python tools like the RSS Feed Parser and Cheetah template engine, alongside core SABnzbd themes such as Classic and Plush.

During the installation, a warning message may appear:

[....] SABnzbd+ binary newsgrabber: not configured, aborting. See /etc/default/s[warndplus ... (warning).

This message is normal and indicates that SABnzbd has not yet been configured, which is expected during the installation phase.

Once the installation is complete, proceed to the subsequent section for configuration.

Initiating the SABnzbd Configuration Wizard

Following the completion of the sabnzbdplus installation process, you will be returned to the command line interface. To initiate SABnzbd for the first time, execute the following command:

sabnzbdplus --server 0.0.0.0

This command launches the SABnzbd daemon and activates the WebUI. A substantial amount of textual output will be displayed, potentially creating the impression that the application has stalled. However, this is not the case; the application is simply managing the terminal and displaying new functionalities as they are implemented.

To avoid inadvertently terminating the daemon, either open a new terminal window or establish a new SSH connection. Using CTRL+C will interrupt the process and shut down the daemon.

Accessing the Configuration Wizard

The configuration wizard can now be accessed from a web browser, either directly on the Raspberry Pi or remotely from your desktop computer. Utilizing a remote browser is highly recommended for improved usability and performance.

Navigate to the following address in your preferred web browser:

http://[Your Pi's IP]:8080/wizard/

Select your desired language and then click "Start Wizard" to begin the configuration process. The initial stage involves entering the details provided by your Usenet provider. While Newshosting is suggested, any provider of your choosing can be utilized.

how-to-turn-a-raspberry-pi-into-an-always-on-usenet-machine-3.jpg

Provider Information and Connection Settings

Input the host, port, username, password, and the desired number of connections. Although a higher number of connections (20+) may be suitable for desktop or server installations, it is advisable to begin with 5 connections on your Raspberry Pi. This number can be incrementally increased if greater concurrency is required.

Click "Test Server" to verify the accuracy of your login credentials with your provider.

Access Control Configuration

The second step of the wizard focuses on access control settings:

how-to-turn-a-raspberry-pi-into-an-always-on-usenet-machine-4.jpg

It is crucial to select the option "I want my SABnzbd to be viewable by any pc on my network". Furthermore, ensure that the "Launch my internet browser with the SABnzbd page when the program starts" checkbox is unchecked. As this will be a headless system, automatically launching the browser on the Pi is unnecessary and will consume valuable system resources.

Setting a username and password is optional at this stage.

Completing the Wizard

Step three of the quick-start wizard can be bypassed, as the supplemental services it suggests are no longer functional. Click "next" to proceed to step four. This final step is automated; the SABnzbd daemon will restart, and the wizard will display the web addresses through which you can access the WebUI, such as:

http://192.168.1.102:8080/sabnzbd/

http://raspberrypi:8080/sabnzbd/

http://127.0.1.1:8080/sabnzbd/

Click "Go to SABnzbd" to exit the wizard and access the main SABnzbd user interface.

Installing UNRAR for Automated Unpacking

The UNRAR utility is essential for automatically extracting files from RAR archives. This is particularly useful in automated systems where manual intervention isn't feasible.

Why UNRAR is Needed

RAR archives are a popular method for compressing and distributing files. Without UNRAR installed, systems cannot natively process these archives, hindering automated workflows.

Installation on Debian/Ubuntu

On Debian-based distributions like Ubuntu, installation is straightforward using the package manager. Simply execute the following command in your terminal:

sudo apt update

Followed by:

sudo apt install unrar

Installation on CentOS/RHEL

For CentOS and Red Hat Enterprise Linux systems, the installation process differs slightly. You'll typically need to enable the EPEL repository first.

Execute these commands:

sudo yum install epel-release

Then, install UNRAR:

sudo yum install unrar

Installation on macOS

macOS users can easily install UNRAR using Homebrew, a popular package manager for Apple platforms.

If you don't have Homebrew, install it first from brew.sh. Then, run:

brew install unrar

Verifying the Installation

After installation, it's crucial to verify that UNRAR is correctly installed and accessible from the command line.

Open your terminal and type:

unrar

If UNRAR is installed correctly, you should see a list of available command-line options.

Using UNRAR in Scripts

UNRAR can be seamlessly integrated into shell scripts for automated unpacking. The basic command for extracting a RAR archive is:

unrar x archive.rar

Where "archive.rar" is the name of your RAR file. The 'x' option instructs UNRAR to extract files with full paths.

Troubleshooting Common Issues

If you encounter issues, ensure the RAR archive isn't corrupted. Also, verify that you have the necessary permissions to write to the destination directory.

Incorrect file paths are another common source of errors. Double-check that the archive name and path are accurate.

Further Options and Documentation

UNRAR offers a wide range of options for customizing the extraction process. For detailed documentation, refer to the official UNRAR documentation available online.

You can access the manual page by typing man unrar in your terminal.

how-to-turn-a-raspberry-pi-into-an-always-on-usenet-machine-5.jpg

Upon initial inspection of the new interface, a warning may be observed: "No UNRAR program found, unpacking RAR files is not possible".

The standard SABnzbd installation doesn't include a RAR program, which presents a challenge. While files can be downloaded from Usenet, automatic extraction won’t occur. Manually unpacking all downloads would be a tedious process.

To enable automated file unpacking, the unrar-nonfree application, though freely available, requires compilation. Fortunately, a user on RaspberryPi.StackExchange has provided detailed instructions for installation on Raspian.

To begin, enter the following command in the terminal to modify your sources.list and incorporate the repository containing unrar-nonfree:

sudo nano /etc/apt/sources.list

Within the nano editor, append this line to the .list file:

deb-src http://archive.raspbian.org/raspbian wheezy main contrib non-free rpi

Press CTRL+X to exit nano, and confirm the save/overwrite with Y. Returning to the command prompt, update your sources list to apply the changes:

sudo apt-get update

Following the update’s completion – which should be relatively quick if you’ve recently updated – create and navigate into a working directory:

mkdir ~/unrar-nonfree && cd ~/unrar-nonfree

Next, download the dependencies required by unrar-nonfree:

sudo apt-get build-dep unrar-nonfree

Once this process is finished and you are back at the prompt, execute the following command to obtain the source code and construct the installation package:

sudo apt-get source -b unrar-nonfree

The package can now be installed. If this tutorial is followed after a new version of unrar-nonfree is released, the filename may need adjustment. The version number can be verified by listing the downloaded files with the command "ls":

sudo dpkg -i unrar_4.1.4-1_armhf.deb

Upon successful installation, verify the functionality by typing "unrar" at the command prompt. A list of available switches and their descriptions should be displayed. If the installation completed without errors, clean up the working directory with:

cd && rm -r ~/unrar-nonfree

Now, address the error within SABnzbd. Restart SABnzbd through the WebUI (Options -> Restart). The error message should disappear from the upper left corner of the WebUI upon restart. Confirm an empty error log by clicking the Status link in the upper left corner:

how-to-turn-a-raspberry-pi-into-an-always-on-usenet-machine-6.jpg

The system is now clear. Unrar-nonfree is installed, enabling automatic unpacking of downloads. However, the default download directories currently point to the Raspberry Pi’s small SD card. These will be redirected to the external HDD.

Adjusting SABnzbd Directory Settings

Initially, all downloaded files are directed to the /home/pi/downloads directory. Rapid accumulation of files, even with a moderate internet speed, can quickly exhaust the storage capacity of the SD card. This will cause SABnzbd to automatically pause downloads, effectively halting the entire download process.

To prevent this, it’s crucial to relocate the essential directories from the SD card to an external hard drive. If you haven't already connected a USB hard drive to your Raspberry Pi and configured it for automatic mounting during startup, we recommend consulting our guide on transforming a Raspberry Pi into a low-power network storage device for detailed instructions.

We will maintain the same HDD naming conventions and directory structure as outlined in that previous tutorial. Therefore, please adapt the commands provided in this section to align with the specific location of your HDD.

Let's begin by establishing the necessary directories for SABnzbd:

sudo mkdir /media/USBHDD1/shares/SABnzbd/downloadingsudo mkdir /media/USBHDD1/shares/SABnzbd/completedsudo mkdir /media/USBHDD1/shares/SABnzbd/watchsudo mkdir /media/USBHDD1/shares/SABnzbd/watch/nzb-backupsudo mkdir /media/USBHDD1/shares/SABnzbd/scripts

With the directories created, navigate back to the SABnzbd WebUI to modify the default directory locations. Access Config -> Folders within the WebUI. This section is divided into User Folders and System folders. Adjust the following entries within these sections, utilizing the directories we’ve just established. Employing absolute paths is essential to ensure SABnzbd utilizes folders outside the default /home/pi/ location.

Temporary Download Folder:
/media/USBHDD1/shares/SABnzbd/downloading
Completed Download Folder:
/media/USBHDD1/shares/SABnzbd/completed
Watched Folder:
/media/USBHDD1/shares/SABnzbd/watch
Scripts Folder:
/media/USBHDD1/shares/SABnzbd/scripts
.nzb Backup Folder:
/media/USBHDD1/shares/SABnzbd/watch/nzb-backup

Furthermore, you can define the "Minimum Free Space for Temporary Download Folder" using units like 900M for 900 megabytes or 20G for 20 gigabytes. Maintaining 10-20GB of free space on the disk is generally advisable as a buffer.

After implementing these changes, save your settings by clicking the "Save" button at the bottom of the menu. A restart is required for the changes to take effect. Return to the main WebUI by clicking "Downloads," then select Options -> Restart in the upper right corner.

Evaluating Your SABnzbd Setup

Once SABnzbd has been restarted, it’s necessary to perform a test run. For the purpose of this evaluation, Binsearch.info was visited to locate a Linux Mint distribution for download.

To initiate the download process, the corresponding .NZB file was placed within the SABnzbd /watch/ directory. SABnzbd automatically identifies and processes files placed in this folder.

The download will initially be visible in the Queue. Subsequently, it transitions to the History section of the WebUI as it progresses through the stages of downloading, verification, and unpacking, as illustrated in the provided screenshot.

Performance Comparison

An assessment was conducted to compare the performance of SABnzbd running on a Raspberry Pi versus a standard desktop computer. Identical files were downloaded on both systems to facilitate a direct comparison.

The Raspberry Pi demonstrated surprisingly strong performance, achieving speeds approximately 15% slower than the desktop machine.

Specifically, a 1GB download completed in roughly 14 minutes on the desktop, while the Raspberry Pi required approximately 16 minutes for the same task. This represents a commendable result considering the Pi’s hardware limitations.

SABnzbd’s functionality was successfully validated through these tests.

Configuring SABnzbd for Automatic Startup

Unlike some of our previous projects, configuring SABnzbd+ to launch automatically upon boot is a relatively straightforward process. The installation process itself generates a script located in the /init.d/ directory. The primary step involves modifying the SABnzbd configuration file located in /etc/default/ to specify the user account under which the daemon should operate.

Editing the SABnzbd Configuration File

To make this adjustment, utilize the following command within the terminal to open the SABnzbd configuration file using the nano editor:

sudo nano /etc/default/sabnzbdplus

Within this file, locate the variable designated as USER=; input the username associated with the account you intend to use for running the daemon.

For our setup, we employed the default user account, "pi," which was also the account used during the SABnzbd installation. After entering the desired username, press CTRL+X to save the changes and exit the editor.

Verifying the Configuration

You can confirm the effectiveness of these changes through one of two methods. Either reboot your Raspberry Pi device, or directly test the service startup using the following command:

sudo service sabnzbdplus start

Successful application of the changes will result in the following output sequence:

[....] Starting SABnzbd+ binary newsgrabber:

[ ok ] Starting SABnzbd+ binary newsgrabber:.

It’s important to note that if you attempt to start the service while SABnzbd is already running, the output will indicate a "fail" status instead of "ok." This signifies that the daemon attempted to launch but was prevented because an instance was already active.

SABnzbd+ offers a convenient way to automate downloads, and ensuring it starts on boot maximizes its utility.

This configuration ensures SABnzbd+ is always available to process your Usenet downloads.

Optimizing SABnzbd Performance

Our previous testing demonstrated that a Raspberry Pi can achieve download speeds comparable to a desktop computer when running SABnzbd. However, several configuration adjustments can be implemented to further enhance performance and maximize throughput. These optimizations will be detailed below, presented in order of expected impact.

Configuring Server Settings

Access the main WebUI and select "Config" to begin modifying settings. We will examine each pertinent section and its associated tweaks individually.

Within the Config -> Servers section, consider the following:

Disable SSL Encryption. Unless stringent privacy concerns dictate otherwise, switching from a provider’s SSL-enabled server to a standard server is recommended. Testing revealed that SSL encryption introduces significant overhead, reducing download speeds by approximately 50% and hindering unpacking speeds due to the processing demands placed on the Raspberry Pi.

Adjusting SABnzbd Switches

Navigate to the Config -> Switches section to refine download behavior.

Enable Duplicate Download Detection. Avoid unnecessary consumption of time, bandwidth, and processing resources by preventing the re-downloading of previously acquired files. The available options are "Discard" and "Pause." Utilizing "Pause" allows for periodic queue review and informed decisions regarding potential re-downloads.

Activate Pause During Post-Processing. While the Raspberry Pi efficiently handles single downloads and unpacking, processing a large backlog concurrently can strain its resources. Enabling this feature instructs SABnzbd to temporarily halt downloads while completing post-processing tasks.

General Configuration Tweaks

Further performance gains can be achieved within the Config -> General section.

Define Article Cache Limit. Although currently left unmanaged, numerous users on the SABnzbd forum have reported positive outcomes on Raspberry Pi systems by setting this value to 32M or 64M.

These adjustments, when implemented, should contribute to a noticeably improved SABnzbd experience on your Raspberry Pi.

Setting Up Mobile Access for SABnzbd

To manage your SABnzbd installation conveniently from a mobile device, you have two primary options. These involve either installing a mobile-optimized WebUI template or utilizing dedicated control applications available on the App Store or Google Play.

During the initial package installation, both the Classic and Plush themes were included. Now, let’s proceed with installing a theme specifically designed for mobile viewing.

Installing Mobile Themes

To install the standard mobile theme, as depicted in the screenshot, execute the following command in your terminal:

sudo apt-get install sabnzbdplus-theme-mobile

Alternatively, if you prefer a theme resembling the iOS iPhone interface, use this command instead:

sudo apt-get install sabnzbdplus-theme-iphone

After installing your chosen mobile theme, navigate to the WebUI. Access the configuration settings by going to Config -> General -> SABnzbd Web Server.

Within the Secondary Web Interface subsection, you will find the option to select the mobile skin you have installed.

how-to-turn-a-raspberry-pi-into-an-always-on-usenet-machine-11.jpg

Remember to scroll down and save your changes. Then, click the "Restart SABnzbd" button, located next to the save button. Upon completion of the restart, you can access the secondary, mobile-optimized interface via this URL:

http://[Your Pi's IP]:8080/m/

Mobile Apps for Enhanced Control

For an even more comprehensive Usenet experience, consider utilizing dedicated mobile apps. These apps extend the functionality beyond the basic mobile interface.

iPhone and iPad users can explore these options:

SABMobile
myNZB
SABCommand

Android users have a similarly robust selection, including:

NZB Unity
SABcontrol
SABMobile
NZBAir

Many of these mobile applications provide advanced features, such as RSS-to-NZB transfers and sophisticated queue management capabilities.

Having now completed the installation of SABnzbd, addressed its dependencies, implemented optimizations, and configured a mobile interface or control app, you are well-equipped to fully utilize the power of Usenet. Stay tuned to this page for further guides on maximizing your Raspberry Pi experience.

Topics

More