My Newspaper

Author: Iñigo Serna, inigoserna AT gmail DOT com
Version: 4.0, April 13th. 2014
Home page:http://inigo.katxi.org/devel/mynewspaper/
License:
Copyright © 2005-14, Iñigo Serna
This software has been released under the GNU Affero GPL License version 3 or later, see the COPYING file that comes with this package.
There is NO WARRANTY.
Last update:Sun Apr 13 13:18:26 2014

Table of Contents

Introduction

This is the documentation for MyNewspaper, a web-based personal RSS/ATOM news aggregator and feeds reader written in Python 3.

It incorporates its own web server and store all the information in a database; but don't be afraid, it's very easy to install and use, and powerful enough for most users.

Can be completely controlled with the keyboard. It even has an adapted web interface for mobile devices.

Code and development is publically available in the BitBucket repository.

Want to know how it looks like?

Web interface on Firefox 28 on Fedora Linux 20
ss1.jpeg ss2.jpeg
Home Folders
ss3.jpeg ss4.jpeg
Reading articles 1 Reading articles 2
Mobile web interface on an Android smartphone with Firefox 28
ss5.jpeg ss6.jpeg ss7.jpeg
Summary List of articles Reading an article

A bit of history

MyNewspaper can be considered an old project, as first version dates from 2005, and since then it has adopted and adapted different web paradigms: CGI running on a web server in v1.0, pure web app in v2.0, javascript and AJAX for v3.0.

A couple of years after v3.0 release I abandoned it to join Google Reader wave, but since the announce of its shutdown I turned to MyNewspaper as I didn't like any of the alternatives. Thus I rewrote from scratch all the code. It was fast, I had a working version in a month, many weeks before Google Reader closed, but then it took about 10 months more to write the documentation…

Anyway, this is now version 4.0, a very fast and nice web-based feeds reader for personal use.

Technically…

MyNewspaper is a web application and, as such, it has backend and frontend code.

Backend is written in Python and based upon BottlePy web microframework, and frontend use HTML and JavaScript with the help of jQuery library.

Permanent storage is saved in a SQLite DB, without any abstraction modules or ORMs to speed up DB interaction.

It uses Mark Pilgrim's Universal FeedParser module to download and handle feeds in RSS, Atom and other popular formats.

All the needed modules are already included within the package. More information can be found in the Dependencies section.

Dependencies

All the required dependencies are already packed within MyNewspaper so you don't need to download and install any module.

I specify the versions I currently use, note different versions could not work.

JavaScript code minimized with Closure Compiler.

License

MyNewspaper has been released under the GNU Affero GPL License version 3 or later, read the COPYING file that comes with this package. There is NO WARRANTY.

Contact

In case you have any criticism, comment, suggestion, etc you can use the BitBucket Issue Tracker or send me an email to the address stated above.

Download and Installation

First of all, download MyNewspaper from here. It's all you need to do, no additional modules or libraries to install, only Python 3.2+.

You can run MyNewspaper in different ways. I'll show you different methods as examples, simple and more elaborated, but note that you can also mix from both approaches.

Easy way: uncompress & run

This is an easy and fast way to get MyNewspaper running, but it's not the proper method if you want to keep it running permanently.

Uncompress source package in a directory of your choice and go there. You don't need to configure anything. Just be sure you have Python version 3.2 or upper installed on your machine.

Now run the application:

$ python3 ./mynewspaper.py
Bottle v0.12.5 server starting up (using WSGIRefServer())...
Listening on http://127.0.0.1:8887/
Hit Ctrl-C to quit.

It will automatically start its own web server on IP 127.0.0.1 and port 8887, then you can open a web browser to that URL in the same computer. You can change both IP and port editing the function run_standalone() at the end of src/web.py file.

This method is convenient for testing or taking a look at the program but consider a more robust approach for production.

Production: guru's way

If you want to install MyNewspaper as a system service this is the right choice.

We will configure a running web server as a frontend for MyNewspaper, install a cron recipe to update feeds periodically in an automatic and transparent way and finally install an init.d script to make MyNewspaper start with your system.

  1. As before, uncompress package in a directory of your choice. In this case it is a good idea to use a subdirectory under your web documents hierarchy
  2. Then change the ownership of the whole directory to that user of your web server

For example:

# cd /var/www
# tar xvfx path_to/mynewspaper-X.Y.tar.gz
# chown -R <www_user> mynewspaperX

Note that depending on your distribution, web documents root path could be other than /var/www.

Next, let's study how to do some advanced configurations.

Run behind a web server

Note MyNewspaper (in fact the micro web framework it's based on, BottlePy) does not directly provide user authentication, ssl or traffic compression on his own. Thus it's a worth idea to run behind a web server, specially if you want to deploy it on a internet accessible computer. You will get many other advantages as well:

  • use the public URL you like, and let the web server map it to the program internal URL
  • run under SSL transparently
  • use transparent authentication for the program
  • compress traffic between MyNewspaper and the web browser
  • integrate visits information with your site logs

With nginx

nginx is the web server I use nowdays, fast and powerful.

Example 1: use a virtual host: https://mynewspaper.domain.com

[...]
# enable and configure SSL and gzip compression
[...]
server {
    listen       443;
    server_name  mynewspaper.domain.com;
    ssl          on;
    access_log   /var/log/nginx/mynewspaper-access.log  main;

    location / {
        auth_basic            "My Newspaper";
        auth_basic_user_file  "/etc/nginx/domain.com.users";
        proxy_pass            http://127.0.0.1:8887;
    }
    location /images {
        disable_symlinks  off;
        alias /var/www/mynewspaper4/images;
    }
    location /static {
        disable_symlinks  off;
        alias /var/www/mynewspaper4/static;
    }
    location /docs {
        disable_symlinks  off;
        alias /var/www/mynewspaper4/docs;
    }
}
[...]

With lighttpd

lighttpd is a fast, powerful web server with lot of features.

Example 1: use a virtual host: http://mynewspaper.domain.com

[...]
server.modules = ( ..., "mod_proxy", "mod_simple_vhost", ... )
[...]
$HTTP["host"] == "mynewspaper.domain.com" {
  server.document-root = "/var/www/mynespaper/"
  accesslog.filename = "/var/log/lighttpd/mynewspaper-access_log"
  proxy.server = ( "" => (("host" => "127.0.0.1", "port" => 8887)) )
}
[...]

Example 2: rewrite url: http://www.domain.com/mynewspaper [1]

[...]
server.modules = ( ..., "mod_rewrite", "mod_proxy", ... )
[...]
$HTTP["host"] == "www.domain.com" {
  [...]
  $HTTP["url"] =~ "^/mynewspaper" {
    server.document-root = "/var/www/mynewspaper2"
    url.rewrite-once = ( "^/mynewspaper(.*)$" => "/$1" )
    proxy.server = ( "" => (("host" => "127.0.0.1", "port" => 8887)) )
  }
  [...]
}
[...]
[1]This doesn't work in some lighttpd v1.4.x versions due to a bug in lighttpd code, see http://trac.lighttpd.net/trac/ticket/164

With apache

apache, the most popular web server.

The concepts are quite similar to any other web server.

Other

There are other alternatives: FCGI, WSGI, running as CGI, etc. More ideas in BottlePy_ documentation.

Automatic updates with cron

  1. Copy tools/mynewspaper.cron file to /etc/cron.d with the name mynewspaper
  2. Uncomment the proper line depending on how you want to access the application: internal web or web front. By default, it will update feeds each 30 minutes
  3. Finally, restart cron daemon
# cp tools/mynewspaper.cron /etc/cron.d/mynewspaper
# [edit] /etc/cron.d/mynewspaper
# systemctl restart crond # or /etc/init.d/crond restart

If you don't understand anything, consult your cron documentation.

Of course, you can fetch new articles from the web UI too.

Running as a system service

Nowdays I use supervisor to start, run and control my python web applications.

This is the configuration file I use, from /etc/supervisor.d/web_mynewspaper.ini:

[program:web_mynewspaper]
command=/home/www/mynewspaper4/mynewspaper.py
directory=/home/www/mynewspaper4
priority=100
autostart=true
autorestart=true
user=www
log_stdout=true
log_stderr=true
logfile=/var/log/supervisord/web_mynewspaper.log
logfile_maxbytes=1MB
logfile_backups=5

Anyway you could easily create your own systemctl or init.d scripts.

Using MyNewspaper

First steps

Initial wizard

Once you have the application installed and running, open your web browser and navigate to the URL shown on the terminal, http://127.0.0.1:8887 by default. If you run MyNewspaper behind a web server and use url rewriting you could use that public URL too.

Note

You can also manually upgrade a MyNewspaper v3 database from command line using the migrate_db_from_v3_to_v4.py shell script in tools directory. Remember to backup the database file before.

ss-tut-wizard.jpeg

First time you will be redirected to the initial wizard where you will be able to import feeds from an OPML File, import feeds from Google Reader Takeout file, migrate data from a MyNewspaper v3 database or start without feeds.

Main screen

After you have completed the wizard, the application will download the new articles for the feeds you have defined (if any) and then you will be redirected to the summary screen of the application. This could take some time if there are a lot of feeds or you have a slow internet connection.

ss-tut-main.jpeg

Once the database is properly configured and application is running you can always go to http://127.0.0.1:8887 (or configured URL) to see the summary page.

Now your are ready to enjoy MyNewspaper.

User Interface

As you can see in the image above, the screen is divided into 3 parts:

  • at the top, there is the header
  • at the left, the sidebar
  • at the right, the body of the application

The Header

Logo with a couple of buttons:

  • hide/show sidebar toggle button, at the left
  • refresh view button, at the right of logo

The Sidebar

The sidebar is placed at the left of the page. It can be hidden and showed clicking the button on the header or with ESC key shortcut.

It contains some links: Home view, Folders view, Archive, Search.

Below them there is the list with feeds folders, and the count if there are unread articles. The groups can be expanded or collapsed to show or hide feeds in the folder. Clicking on folders or feeds will lead you to the Articles view.

The Body

It can contain different pages:

Home view

A summary with links to some management actions, reports or help.
For example, you can add new subscriptions, update feeds, import feeds from an OPML file, export current feeds as OPML, export database, etc.
You can also manage the database (clean the data): Compact (vacuum), or Remove old entries from cache (> 3 months).

Folders view

A table with the folders (groups) defined showing the name, number of feeds, and number of unread, starred, postponed, archived and total articles.
Some actions at top: new folder, stats, and update feeds.
Name and numbers are hyperlinks to other views.

Group view

A table with the feeds defined in the group showing the name, and number of unread, starred, postponed, archived and total articles.
Some actions at top: add subscription, rename folder, delete folder, stats, and update feeds.
Name and numbers are hyperlinks to other views.

Feed view

Feed name, feed URL, home page, state (enabled, disabled, broken), date of last update, and number of unread, starred, postponed, archived and total articles.
Some actions at top: edit, delete, stats, update.
Hyperlinks are used too.

Articles view

Note

An article is or can be marked in any of the following states:

  • st1 Delete
  • st2 Unread
  • st3 Starred (favourite)
  • st4 Later (postponed for later)
  • st5 Archived

A list contaning a batch of the selected articles. By default there is a maximum of 100 entries per page, but you have buttons to load 100 more or all.

Button bar

ss-tut-arts_btns.jpeg

There is a button bar at the top providing different actions:

  • Process button (more on this later)
  • Pop-over to change which articles to show (by state)
  • Pop-over to filter articles by state
  • Pop-over to change sort mode
  • Toggle button to expand articles contents or show header only
  • Toggle button to hide/show articles images
  • Pop-over to change the state of all shown (non filtered out) articles

Article areas

ss-tut-arts.jpeg

Each line represents an article. They are divided in several areas, from left to right:

  • article state icon
  • feed name, in color green
  • feed favicon
  • article title, in bold
  • brief summary
  • time stamp
  • article contents, hidden by default

The two orange vertical bars around an article represent the cursor, and it marks the currently selected entry. You can operate on current article using Keyboard navigation.

Mouse actions:

  • Placing the mouse over the status icon will open a small window allowing to change it
  • Clicking on the article title will open the original source of the article in another tab/window
  • Clicking on the summary will show or hide article contents

Process

Clicking on the Process button will perform for each article the action marked in its state: delete, let it as unread, starred, mark for later use, or archive.

Remember Process only operate on loaded and visible entries.

Keyboard navigation

If you prefer keyboard navigation you can avoid the mouse to control MyNewspaper.

  • Global
    • ESC -> Toggle sidebar
    • h -> Go to Home
    • f -> Folders
    • / -> Search
    • u -> Unread items
    • r -> Refresh view
    • ? -> Show Keyboard Shortcuts
  • Articles view
    • j -> Select next article
    • k -> Select previous article
    • SPC -> Move one page down and select article
    • BackSpace -> Move one page up and select article
    • n -> Select and open next article
    • p -> Select and open previous article
    • Home -> Move to first article and select it
    • End -> Move to last article and select it
    • 1 -> View with articles expanded
    • 2 -> View with only articles titles shown
    • i -> Show or hide the images in articles
    • v, Enter -> Expand or collapse selected article
    • s -> Cycle actions for current article
    • o -> Open article in new window or tab
    • Shift + a -> Process visible articles
    • Shift + m -> Load more articles
    • Shift + t -> Load all articles

MyNewspaper Mobile

MyNewspaper contains a minimalist web interface for mobile devices like smartphones and tablets, where screen size and lack of proper keyboard and mouse make difficult to manage the application in a traditional way.

The URL is located at: http://127.0.0.1:8887/m or http://mynewspaper.domain.com/m.

There are 3 main views:

Summary

List of all the folders with unread articles.
You can change selected state (unread by default) by tapping on the state icon at the header.
To expand a folder click on its name.
To go Articles list view click on the number at right.

Articles list

A list of articles with their title, feed, timestamp and a few lines of the contents as a brief summary.
Clicking on the back image will return to Summary view.
Touching on the icon at the left of an article will open a popup window to change its state.
Tap article summary to open Article contents view.

Article contents

This page shows the contents of the selected article.
Clicking on the back image will return to Articles list view.
There are up and down arrows to navigate to previous or next article.

FAQ

ChangeLog & TODO

Releases

v4.0 "I hate writing documentation"

[2014/04/13] Completely rewritten. Download from here

- General:
    . rewritten everything from scratch
    . all dependencies are included within the application
    . new documentation, including a brief tutorial
- Core:
    . use BottlePy web framework instead of CherryPy
    . don't use any ORM, access DB directly
    . use dateutil module to parse timestamps
- UI:
    . UI completely rewritten
    . new look & feel: darker theme, new icons
    . use newer versions of jQuery and jQueryUI
- Mobile version:
    . completely rewritten
    . don't use jQuery Mobile
v3.0 "a new one is here"

[2010/12/29] Javascript & AJAX, mobile version

- Core:
    . articles new "postponed" state
    . AJAX calls between python core and UI javascript
    . option to disable a feed temporary
    . create http://mynewspaper/update entry point
    . speed, speed, speed
    . clean code
    . remove cheetah templating engine
- UI:
    . UI completely rewritten, using jQuery and jQueryUI
    . added visual effects
    . drag&drop to sort groups or feeds
    . new icons
    . navigation using keys
    . new navigation bar
    . show/hide images
- Mobile version of the web interface
    . http://mynewspaper/m
- Admin:
    . configuration
    . init wizard
    . feeds report
    . new error 404 - not found page
v2.0: "living with no name"

[2006/07/XX] Unreleased

- Core:
    . use CherryPy insted of CGI
    . articles new "favs" and "archived" states
    . improved fetching feeds, using threads
- UI:
    . toggle sidebar
    . new "virtual" folder in sidebar
    - Admin:
    . opml import/export (http://www.opml.org/)
    . stats: no. of fav/unread/total articles per group/feed,
             older/newer, .db type and size
v1.4: "even faster and sexier than before"

[2005/08/01] Speed, usability, UI improvements

- New features:
    . Unread virtual folder
    . Favourites
- Faster:
    . with v1.2 summary page was very slow if >3k articles, it
      has been profiled and optimised. Basically I've substituted
      len(a_group.feeds) and len(a_feed.articles) by
      select(XXX).count() I've also built OR-expressions
      dinamically instead of for-loops when dealing with SQLObject
- UI:
    . show/hide heading/contents for all articles or per article
    . "toggle" button to invert selections
    . user can see number of selected/total articles
    . show articles in batches of 50 articles
    . un/selecting articles should update count button
- bugs fixed:
    . crash in maintenance when no feeds defined
    . crash in archive/unmark_as_read if no articles selected
v1.2: "faster (25-35%), cleaner (100%), smaller (69%) and even more robust (100%)"

[2005/07/24] A major rewrite and clean up of the code

- UI:
    . use Cheetah as html template
    . generate valid html
    . clean css
- Use SQLObject as database abstraction
- bugs fixed:
    . fix 'XXXadminXXX' hack
    . now no names are passed in urls or forms, id instead =>
      => no more crashes due to encoding problems
    . crash: 'Stiri Ziare.ro': 'http://www3.ziare.ro/backend.php'
    . feeds_edit doesn't expand group in sidebar
    . timeout for feed downloading => feedparser does it
    . a pile of them
v1.0: "initial release"
[2005/07/17] Initial release.

TODO

TODO file with some ideas and eventual future enhancements is at TODO.