Importer 3000!

importer3000

Importer 3000 is here! What is it? It could be the most important moment in the history of Booktype! It could be, but in reality it is just a basic upload form I did for testing new ebook library.

importer_basic

And yes, it could easily look like this but would anyone believe me I have spend more then 5 minutes on this task? Can that simple form explain why am I so excited about new ebook library? Probably not and that is why we have Google fonts and shiny colors! If you are trying to present couple of months of research and work you have spend on backend code, try to use them. It can not hurt.

github

 

E-Book Web Reader ~ Rheya ~

info

I had a small coding sprint over the weekend. It would probably be better if I played Minecraft entire weekend but somehow i couldn’t stay still until I put my thoughts into code….

The code is not really representative, I just wanted to check the feeling of using this new Web interface for the books. I have two not fully functional interfaces: desktop and mobile one. I am aware it does not work on Android (the footer), the inner frame do not resize as they should and some buttons do not work. Like I said… just a proof of concept for now. But a proof of concept which has support for keyboard shortcuts and you can change color/text theme. What more do you need?!

mobile_font

But when it is finished… ohhhhh when it is finished! It will work in offline mode, it will support bookmarks, sharing, annotations and many many more things. It can read epub2/epub3 books for now but I will integrate it with Booktype online books when I get some free time.

You can check the code here:
* https://github.com/aerkalov/ebook-web-reader/wiki

Demo books:
* http://www.binarni.net/projects/rheya/metropolis/ (Desktop version)
* http://www.binarni.net/projects/rheya/the-binary-chronicles/ (Desktop version)
* http://www.binarni.net/projects/rheya/metropolis/mobile.html (iPad version)
* http://www.binarni.net/projects/rheya/the-binary-chronicles/mobile.html (iPad version)

Soooo…. what do you think? Would this be a good interface for online/offline books?

Booktype-scrolls: Ubuntu installation

First in a series of screencasts – How to use Booktype-scrolls script to install Booktype on Ubuntu.

Script is smart enough to figure out what needs to be installed on Ubuntu, Debian, CentOS or Mac OS X. You will be informed about every step and nothing will be installed on the system without your confirmation or knowledge. At the moment script works with Sqlite and builtin Python server. That will be changed in next version but for now this is more then enough if you had problems installing it for yourself. You can read more about the script in my previous post – http://www.binarni.net/2012/07/booktype-easy-install/.

For more information check out official installation docs – https://wiki.sourcefabric.org/display/Booktype/Installing.

Styling your book on the Web

As you know, you can use Booktype to publish your books into different formats but you can also view them on the web. In this blog post I will try to show how that web view can be customized with simple template modification. By default Booktype comes with predefined templates but they can all be easily customized to your own needs (templates can also be multilingual).

Let us imagine we are publishing shop “Awesome Sun” who is interested in publishing books about solar power. Booktype project has been installed in /var/www/awesomesun/ directory on our web hosting server. We are not so happy with the default look and feel so we decided to change it. Where should our web designers look at? In the case of “Awesome Sun” template files should be placed in /var/www/awesomesun/templates/ directory while all static files (CSS files, JavaScript files, images, …) should be placed in /var/www/awesomesun/static/ directory. Fairly simple for now. Before doing any work it would be wise to check documentation how to work with Django templates. As you could guess, examples in this blog post are oversimplified.

By default, to present books on the web Booktype comes with Django application called “Reader” and we will customize templates for that application. Original templates for this application are placed in $BOOKTYPESOURCE/lib/booki/reader/templates/ directory and our customized templates will be placed in /var/www/awesomesun/templates/reader/ directory. All you have to do is either copy original template files into awesomesun project and modify them or write new templates from scratch.

What kind of modifications are we making? As you can see on the screenshots bellow we want to have very simple book view, we want our “Table of contents” to be always visible while we scroll the page and we want our header to be fixed on top with name of the book and chapter we are currently reading.

To make it work I created new base template called new_base.html and placed it in /var/www/awesomesun/templates/ directory. This is a base template which other templates for “Reader” application should include. This template just includes jQuery library and creates place holders for future content.


{% load i18n messaging_tags booki_tags %}



{% booki_site_favicon %}
{% booki_site_metadata %}

{% block header %}
{% endblock %}

{% block content %}
{% endblock %}



Then I had to create /var/www/awesomesun/templates/reader/book_chapter.html template file. This is where I include additional JavaScript library, CSS file, display “Table of contents” and content of chapter. Besides HTML and CSS there are just two important things to look at. Chapter content is inserted with {% booki_format content %} and “Table of contents” is inside of variable chapters. It all depends how you want to show your “Table of Contents” but here I am using “for” loop to construct unordered list with chapter titles. Everything else is just CSS styling.


{% extends "new_base.html" %}
{% load i18n booki_tags %}

{% block header %}
{% blocktrans with book.title as booktitle %}/book: {{ booktitle }}{% endblocktrans %}

{% endblock %}

{% block content %}

{% booki_format content %}

{% endblock %}

Essentialy this is how you customize Booktype templates. I have compiled all needed files (with CSS and additional JavaScript) from this blog post here. Please download it if you are interested in more details.

Booktype Easy Install

A lot of people seems to have problems installing Booktype. Until we have native packages for each distribution installing vanilla Booktype from git repository will be the way to go.

I love all of my users the same and that is why I decided to work on a small install script which would do most of the steps for them. This script should help you even when your distribution does not provide valid native packages (Old Django packages and etc…). Anyhoo, I have done some basic testing on – Ubuntu 10.04, Ubuntu 12.04, CentOS 6.3, Debian 6 and I have tried to avoid combination of outdated system packages and latest packages installed with PIP (because that is what Google told you to do). I am aware there are many places where this script could go wrong but i have tried to fix at least basic problems. For instance, if you are on CentOS it will warn you and inform how to install EPAL repository.

Like all “double click” methods this script will also install unwanted packages. To avoid conflicts with different versions of python modules it will create virtualenv environment and install only those Python packages (and their versions) you really need. It will install packages using python package tools and not system packages. For some of the packages it means compiling. Compiling means having development tools on the system. Be aware of that.

Created Booktype instance will use Sqlite to store data. Sqlite is great database but it does not support easy upgrades of schema. If you are serious about Booktype you might want to look at PostgreSQL. After all, this is version 0.0.1 and we only support built in python web server. To make it work with Apache you will need to manually activate virtualenv environment in wsgi script. In the future install script will do this for you.

Where to get it

For now, Booktype Easy Install script is part of “Booktype Scrolls” project and you can get it here:
https://github.com/aerkalov/booktype-scrolls/tree/master/scripts/install.

How to start it

Download the script and execute it. It will download and install required packages. When it is done it will tell you how to create user account and start Booktype. You don’t need to be root to start this install script but you do need sudo permissions in case it must install some system packages. You will need to confirm installation of new packages and you will be informed which commands are being executed in the background. Without your permission new packages will not be installed on the system. Feel free to analyze the script before you start it.

wget https://raw.github.com/aerkalov/booktype-scrolls/master/scripts/install/booktype_install.py 
python booktype_install.py

Use –help to get all available arguments:

python booktype_install.py --help
python booktype_install.py -p myproject
python booktype_install.py -o ubuntu -p book

What now

It needs testing and support for more platforms. Please report to me if you have problems with it. Will try to fix it together!

Booktype scrolls – Importing WordPress site!

So…you left your job and went traveling throughout Asia for 4 months.  In small cyber cafe, somewhere in south Pakistan, you created WordPress site because you were lazy to send emails to friends and family. Internets loved you and your pictures were on the front page of Reddit many many times. Who could forget that great article about food poisoning in China or that unfortunate misunderstanding with gentleman in Bangkok…

When you got back everyone was like – “You should write a book!” and you said to yourself  – “Why not… Half of the book is already written.”. This brings us to the subject of this post – importing WordPress site as a Booktype book.

Importing

There are couple of ways you can do it (this can be applied to other systems also):

  • Use some kind of API to remotely access data. For this you only need permission to access data.
  • Write your own plugin to export data from WordPress. Besides knowing how to write plugin you need permissions to install it on the remote server. You can use WordPress API to render the articles properly, something you can’t really do with other 2 methods.
  • Finally the method we are using in this post. Parsing XML export file created with WordPress export plugin.

Importing data can be complex process and each method has its good and bad points. You probably don’t want to have chapter for each post in WordPress because that could be hundred and hundred of chapters at the end. Maybe you want to choose which chapters you want and sometimes to combine couple of posts as one chapter. These are just some of the issues.

How it works

As it is visible on the picture above you should go to Tools/Export and choose what you want to export. Download the “Export File” to you computer. Export file is just extended RSS file and you can (and should) use it to export content of your site. Notice that attachments and images are not part of this export file, we will need to download them separately! Besides content is in “raw” format (before everything was rendered to HTML).

Like I said, export file is extended RSS file and for parsing we use this great library feedparser. Book name can be specified with arguments but for the default name we use title of the WordPress site. Booktype cares about two titles. One is the full title of the book and the other one is unique url name of the book (usually slugified version of full title).

from booki.utils.book import createBook

book = createBook(conf['user'],
                  conf['bookTitle'],
                  status = 'new',
                  bookURL=conf['bookTitleURL'])

As you can see we use function createBook. It takes user object as first argument (someone has to be owner of the book), full book title, default book status (books can have different statuses) and slugified title name. Export file provides us with the information about WordPress administrator and Post author. We could use that information to find or create Booktype user with specific info and set that person as Book owner.

Parsing posts

After this we just go through the list of all the posts in export file and ignore those who don’t have ‘wp_status’ set ‘publish’. Like i said, content of that chapter is in “raw” format. For instance empty lines in text are representing start of new chapter and etc. If we were using specific tags to format our code (for instance, to prettify the source code) we would not get nice and colorful HTML.

In this example we just do two modifications. One is to create paragraphs in text and the 2nd one is to put Chapter title inside of H2 tag. That is one of the Booktype requirements at the moment.

content = "\n".join(["< p >%s< /p >" % p for p in content.split('\n\n') if p.strip() != ''])
content = u'< h2 >%s< /h2 >%s' % (chapterTitle, content)

And yes… I have some extra spaces in tags P and H2 because i am too lazy to figure out how to escape it in code prettifier plus the code to make paragraphs is not perfect (but this is just an example).

Images

Now we parse Post content and search for the images. When we find one, we try to download it to our computer. Normally we wouldn’t need to do it, but Booktype really wants to have entire content of a book locally. When the image is successfully downloaded we save it as an attachment and we modify image location. All images must be placed inside of ‘static/’ directory. There are good reasons why it has to be relative path, but we will not talk about it now.

att = models.Attachment(book = book,
         version = book.version,
         status = stat)

f2 = File(StringIO(data)) f2.size = len(data) att.attachment.save(fileName, f2, save=True)
fileName = os.path.basename(att.attachment.path)
e.set('src', 'static/%s' % fileName)

Chapter

At the end we just create new Chapter. This is very basic way of creating new chapter because we are not leaving any traces in our logs and etc… We can leave that for some other example. Also notice we are not doing anything with links to other posts/chapters on the same site/book. We just ignore that for now.

chapter = models.Chapter(book = book,
                         version = book.version,
                         url_title = bookiSlugify(chapterTitle),
                         title = chapterTitle,
                         status = stat,
                         content = content,
                         created = now,
                         modified = now)

chapter.save()

Source

Look at the full source for WordPress importer in my Booktype Scrolls repository. Purpose of this code is to be more educational and less production ready.

As you can see technical part of importer is very simple. Thumblr has API for remote access and it would be fairly simple task to make Thumblr importer (or for any other CMS). The biggest problem is still how to make their HTML work nicely inside of Booktype. If author is using custom CSS on WordPress side then Booktype would be completely unaware of it. For instance – <img class=”alignnone size-full wp-image-591″ … It would be the same for different WordPress plugins, broken HTML, headings at wrong place, JavaScript plugins and etc.

Booktype Scrolls

I decided to put different kind of scripts, snippets and small Django apps in a separate project called “Booktype Scrolls“. The idea is to show how Booktype can be extended and how some of its functionality can be used.

For now i have WordPress importer, global search-replace and couple of snippets to get some basic statistics from the database. For 0.1.3 version it is more then satisfactory but I plan to extend it every couple of days with something new. So… stay tuned!

Links:

Booktype 1.5.3

Booktype 1.5.3 is out with a lot of bug fixes and new features. Check the full list of commits here – CHANGES.txt. As you could also notice we were a bit late with this release.

Major feature is something called “Booktype Control Center”. Default Django admin interface is still here but we have developed new administrative interface for Booktype administrators. Booktype is now more customizable then ever and  you  can use this web tool to adapt it to your own needs. Besides users and books you can also manage look and feel of your Booktype instance and books your produce. We started as extremely open tool for book collaboration but with this release we have let administrators adapt Booktype instance to their own needs. Normal users can not create books? Not a problem! Only administrator can create new users? It it possible now! You only want to let users output books as PDF files? You can do it now!

Because of “Booktype Control Center” we needed to develop new way of managing configuration. Read more about it in “Managing configuration” blueprint. Only some parts of code are using this new API but i plan to convert entire Booktype in new release (that would be 1.5.4). Besides us developers, new API will be very useful to administrators during migrations or daily Booktype management.

We have also upgraded our default settings.py to work with latest version of Django. Django 1.2 is still minimal requirement but now we work out of the box with Django 1.4! No more manual editing of files! This is a feature for new users, everyone else will have to manually update their old settings.py files. If it is too much work for someone (as always, INSTALL file has information how to do it) they can always create new Booktype project and just copy+update it with their existing project.

New release will be 1.5.4 (not that hard to guess) and it should be out in 2 weeks. The plan is to have release every 2 weeks. What can you expect in new release? More API cleaning, new install scripts and couple of secret projects. But nothing is a secret with Booktype, if you want to be up to date with our development watch our github page, check my tweets and read my blog!

Getting Started with Booktype – What is Booktype?

This is a first post in a series of articles about Booktype. Before I start to be all technical and “… and that is how you install new Django app” it would be good to say couple of words about Booktype history.

Booktype is an open source platform to write and publish print and digital books. It was created to run the FLOSS Manuals platform and back then it was just called Booki. FLOSS Manuals is a community of people joined together for their love of writing good quality documentation for FLOSS software. At first the  FLOSS Manuals platform was build on the  TWiki (Wiki in Perl) platform.

Using  TWiki to prototype something was a good choice. We extended it with our own plugins and some external tools. FLOSS Manuals is a collaborative community and we needed to extend TWiki to support the collaborative processes. We had our IRC-Web chat gateway called FM Transmission [post in Croatian], live Table of contents where you could see what other users are doing (view image of it), Remixer for content, Localization plugins to manage book translation, our own Web-based tool for translating content and many other things. It was good enough platform but it felt like a big hack at times. After some thinking and observing some problems in the TWiki community [post in Croatian] we decided to write a new platform –  from scratch.

We wanted to build community around software and we figured out TWiki and Perl were not the best platform for that. Mainly because a lot of our users are people who are more interested in books and less in Perl code. We decided to use Python and Django as our new platform. They both seems to be reasonably easy to master and even newbies could extend their Booktype installations. You want to have blog on the front page? Just install existing Django blog app, change one line in settings file, put couple of tags in templates and you are good to go. Simples! Anyone could do it.

Booktype

When we talk about Booktype we really talk about Booktype and Objavi. Booktype is used for editing and managing books while Objavi is used to produce book formats (as pdf, epub, mobi, …). In this post we will talk only about Booktype (the book editor).

Booktype is all about the collaborative editing of books and as such is not always what people want. It is about building community around your book. Sure, you can use it to write your diary also (and you should!), but the real power is in collaborative work. What does this mean for the architecture?

Like i said, we decided to use Django as web framework mainly because we wanted people to be able to plug Booktype into their existing web sites more easily and extend their installations with many already existing Django apps. I think we made a good choice with it. As database backend we are using PostgreSQL. Sqlite3 is also an option, but it really should be used just for testing and development purposes. We are using Redis in-memory database to store all kind of information web clients are exchanging with Booktype server using Sputnik.

What is Sputnik? Sputnik is a Django application for two-way communication with web clients. When we started there was no other Django project that could help us with this so I decided to do what everyone would normally do. I wrote one abstract API (with channels, subscriptions and etc…) for communication and implemented dummy network layer. It means when the right moments comes this network layer can be replaced with something better without any need to change the rest of the code. There are many good frameworks these days that could be used as network layer (Socket.IO for instance). Why do we have it? Mainly because of the collaborative part of Booktype. When you have 10 people working on one book at the same time you need a way of notifying all of them about the actions others are doing. Like: locking chapters, chat, changes in Table of Contents, creation of new chapters, attachments uploads and etc. Yes, it means your Table of Contents will be changed as soon as someone adds a new chapter to it. This really improves the user experience and stops you from overwriting someone’s changes.

But enough about Booktype history. In my next article I will talk about exciting world of “Booktype projects” and how you can easily install and extend your Booktype installation!

These are the Booktype links  you should probably check:

 

 

 

Glasajte za Booki!

Gdje to treba glasati ?

Transmediale je u suradnji sa Mozillom najavio Open Web Award 2011. Pročitajte više na website-u o čemu se radi.

Flossmanuals su uključeni sa projektom Booki. Ne trebam pričati koliko ću biti sretan kad bi glasali za nas i pomogli nam da se dokopamo 5000 EUR-a nagrade i vječne slave!

Što je Booki ?

Booki je naziv naše nove platforme za jednostavno i brzo objavljivanje knjiga. To uključuje softwarea koji služi za multiuser online editiranje knjiga, publish-anje knjiga u različite formate, importanje sadržaja sa drugih platformi itd… Osim samog software (platforme) to uključuje i Book sprintove koje organiziramo, a od skoro i kombi. Kombi? Oh da… Kombi sa opremom za printanje i uvezivanje koji je u stanju doći i razveseliti vaše lokalno događanje!

Zašto bi glasao/la za Booki?

Postoji nekoliko razloga zašto bi mogli glasati za nas!

Prvo razlog je što smo vam mi baš super! Super vam je ekipa uključena u projekt, super vam je ekipa koja je odlučila koristiti naš sustav za svoje projekte,… Npr. ekipa kao što je Arctic Perspective Initiative (http://arcticperspective.org) koji rade na knjizi o preživljavanju u Artiškim uvjetima, Thunderbird / Inkscape / Processing / CimCRM / Blender / Command line / OLPC i ostali manuali, ekipe koja skuplja folk priče o uzgajanju biljki, itd. Super vam je ekipa koja odvaja ono malo slobodnog vremena od svog posla i obitelji da potroši na stvari do kojih im je stalo, a vjerujte mi, takvih je dosta uključeno u projekt i stvari koje se dešavaju oko projekta. Oh… kako bi im samo Vas glas puno značio 🙂

Drugi razlog je što bi svaki vaš glas mene malo trgnuo i natjerao da mrdnem guzicu i više radim na projektu. U zadnje vrijeme sam se fino ulijenio. Što je naravno velika šteta jer se radi o jednom od najzanimljivijih projekata u mom dosadašnjem zivotu. Pa ako mislite da će Vaš glas pomoći da mrdnem guzicu od besmislenog komercijalnog posla koji ne pruža ništa osim stalnog prihoda, nemojte se previše sramiti da glasate za nas!

Kako glasati ?

Ako ste došli do ovog paragrafa, neće vam biti onda teško ni da se registrirate na websiteu i glasate za nas! Znam … znam.. ali tako stvari stoje 🙂

Prvo se registrirajte:
http://www.drumbeat.org/user/register

I onda glasajte za nas:
http://www.drumbeat.org/project/open-web-publishing

Hvala od mene i cijelog tima!