Booktype Easy Install

A lot of people seems to have problems installing Booktype. Until we have native packages for each distribution installing vanilla Booktype from git repository will be the way to go.

I love all of my users the same and that is why I decided to work on a small install script which would do most of the steps for them. This script should help you even when your distribution does not provide valid native packages (Old Django packages and etc…). Anyhoo, I have done some basic testing on – Ubuntu 10.04, Ubuntu 12.04, CentOS 6.3, Debian 6 and I have tried to avoid combination of outdated system packages and latest packages installed with PIP (because that is what Google told you to do). I am aware there are many places where this script could go wrong but i have tried to fix at least basic problems. For instance, if you are on CentOS it will warn you and inform how to install EPAL repository.

Like all “double click” methods this script will also install unwanted packages. To avoid conflicts with different versions of python modules it will create virtualenv environment and install only those Python packages (and their versions) you really need. It will install packages using python package tools and not system packages. For some of the packages it means compiling. Compiling means having development tools on the system. Be aware of that.

Created Booktype instance will use Sqlite to store data. Sqlite is great database but it does not support easy upgrades of schema. If you are serious about Booktype you might want to look at PostgreSQL. After all, this is version 0.0.1 and we only support built in python web server. To make it work with Apache you will need to manually activate virtualenv environment in wsgi script. In the future install script will do this for you.

Where to get it

For now, Booktype Easy Install script is part of “Booktype Scrolls” project and you can get it here:
https://github.com/aerkalov/booktype-scrolls/tree/master/scripts/install.

How to start it

Download the script and execute it. It will download and install required packages. When it is done it will tell you how to create user account and start Booktype. You don’t need to be root to start this install script but you do need sudo permissions in case it must install some system packages. You will need to confirm installation of new packages and you will be informed which commands are being executed in the background. Without your permission new packages will not be installed on the system. Feel free to analyze the script before you start it.

wget https://raw.github.com/aerkalov/booktype-scrolls/master/scripts/install/booktype_install.py 
python booktype_install.py

Use –help to get all available arguments:

python booktype_install.py --help
python booktype_install.py -p myproject
python booktype_install.py -o ubuntu -p book

What now

It needs testing and support for more platforms. Please report to me if you have problems with it. Will try to fix it together!

Booktype scrolls – Importing WordPress site!

So…you left your job and went traveling throughout Asia for 4 months.  In small cyber cafe, somewhere in south Pakistan, you created WordPress site because you were lazy to send emails to friends and family. Internets loved you and your pictures were on the front page of Reddit many many times. Who could forget that great article about food poisoning in China or that unfortunate misunderstanding with gentleman in Bangkok…

When you got back everyone was like – “You should write a book!” and you said to yourself  – “Why not… Half of the book is already written.”. This brings us to the subject of this post – importing WordPress site as a Booktype book.

Importing

There are couple of ways you can do it (this can be applied to other systems also):

  • Use some kind of API to remotely access data. For this you only need permission to access data.
  • Write your own plugin to export data from WordPress. Besides knowing how to write plugin you need permissions to install it on the remote server. You can use WordPress API to render the articles properly, something you can’t really do with other 2 methods.
  • Finally the method we are using in this post. Parsing XML export file created with WordPress export plugin.

Importing data can be complex process and each method has its good and bad points. You probably don’t want to have chapter for each post in WordPress because that could be hundred and hundred of chapters at the end. Maybe you want to choose which chapters you want and sometimes to combine couple of posts as one chapter. These are just some of the issues.

How it works

As it is visible on the picture above you should go to Tools/Export and choose what you want to export. Download the “Export File” to you computer. Export file is just extended RSS file and you can (and should) use it to export content of your site. Notice that attachments and images are not part of this export file, we will need to download them separately! Besides content is in “raw” format (before everything was rendered to HTML).

Like I said, export file is extended RSS file and for parsing we use this great library feedparser. Book name can be specified with arguments but for the default name we use title of the WordPress site. Booktype cares about two titles. One is the full title of the book and the other one is unique url name of the book (usually slugified version of full title).

from booki.utils.book import createBook

book = createBook(conf['user'],
                  conf['bookTitle'],
                  status = 'new',
                  bookURL=conf['bookTitleURL'])

As you can see we use function createBook. It takes user object as first argument (someone has to be owner of the book), full book title, default book status (books can have different statuses) and slugified title name. Export file provides us with the information about WordPress administrator and Post author. We could use that information to find or create Booktype user with specific info and set that person as Book owner.

Parsing posts

After this we just go through the list of all the posts in export file and ignore those who don’t have ‘wp_status’ set ‘publish’. Like i said, content of that chapter is in “raw” format. For instance empty lines in text are representing start of new chapter and etc. If we were using specific tags to format our code (for instance, to prettify the source code) we would not get nice and colorful HTML.

In this example we just do two modifications. One is to create paragraphs in text and the 2nd one is to put Chapter title inside of H2 tag. That is one of the Booktype requirements at the moment.

content = "\n".join(["< p >%s< /p >" % p for p in content.split('\n\n') if p.strip() != ''])
content = u'< h2 >%s< /h2 >%s' % (chapterTitle, content)

And yes… I have some extra spaces in tags P and H2 because i am too lazy to figure out how to escape it in code prettifier plus the code to make paragraphs is not perfect (but this is just an example).

Images

Now we parse Post content and search for the images. When we find one, we try to download it to our computer. Normally we wouldn’t need to do it, but Booktype really wants to have entire content of a book locally. When the image is successfully downloaded we save it as an attachment and we modify image location. All images must be placed inside of ‘static/’ directory. There are good reasons why it has to be relative path, but we will not talk about it now.

att = models.Attachment(book = book,
         version = book.version,
         status = stat)

f2 = File(StringIO(data)) f2.size = len(data) att.attachment.save(fileName, f2, save=True)
fileName = os.path.basename(att.attachment.path)
e.set('src', 'static/%s' % fileName)

Chapter

At the end we just create new Chapter. This is very basic way of creating new chapter because we are not leaving any traces in our logs and etc… We can leave that for some other example. Also notice we are not doing anything with links to other posts/chapters on the same site/book. We just ignore that for now.

chapter = models.Chapter(book = book,
                         version = book.version,
                         url_title = bookiSlugify(chapterTitle),
                         title = chapterTitle,
                         status = stat,
                         content = content,
                         created = now,
                         modified = now)

chapter.save()

Source

Look at the full source for WordPress importer in my Booktype Scrolls repository. Purpose of this code is to be more educational and less production ready.

As you can see technical part of importer is very simple. Thumblr has API for remote access and it would be fairly simple task to make Thumblr importer (or for any other CMS). The biggest problem is still how to make their HTML work nicely inside of Booktype. If author is using custom CSS on WordPress side then Booktype would be completely unaware of it. For instance – <img class=”alignnone size-full wp-image-591″ … It would be the same for different WordPress plugins, broken HTML, headings at wrong place, JavaScript plugins and etc.

Booktype Scrolls

I decided to put different kind of scripts, snippets and small Django apps in a separate project called “Booktype Scrolls“. The idea is to show how Booktype can be extended and how some of its functionality can be used.

For now i have WordPress importer, global search-replace and couple of snippets to get some basic statistics from the database. For 0.1.3 version it is more then satisfactory but I plan to extend it every couple of days with something new. So… stay tuned!

Links:

Booktype 1.5.3

Booktype 1.5.3 is out with a lot of bug fixes and new features. Check the full list of commits here – CHANGES.txt. As you could also notice we were a bit late with this release.

Major feature is something called “Booktype Control Center”. Default Django admin interface is still here but we have developed new administrative interface for Booktype administrators. Booktype is now more customizable then ever and  you  can use this web tool to adapt it to your own needs. Besides users and books you can also manage look and feel of your Booktype instance and books your produce. We started as extremely open tool for book collaboration but with this release we have let administrators adapt Booktype instance to their own needs. Normal users can not create books? Not a problem! Only administrator can create new users? It it possible now! You only want to let users output books as PDF files? You can do it now!

Because of “Booktype Control Center” we needed to develop new way of managing configuration. Read more about it in “Managing configuration” blueprint. Only some parts of code are using this new API but i plan to convert entire Booktype in new release (that would be 1.5.4). Besides us developers, new API will be very useful to administrators during migrations or daily Booktype management.

We have also upgraded our default settings.py to work with latest version of Django. Django 1.2 is still minimal requirement but now we work out of the box with Django 1.4! No more manual editing of files! This is a feature for new users, everyone else will have to manually update their old settings.py files. If it is too much work for someone (as always, INSTALL file has information how to do it) they can always create new Booktype project and just copy+update it with their existing project.

New release will be 1.5.4 (not that hard to guess) and it should be out in 2 weeks. The plan is to have release every 2 weeks. What can you expect in new release? More API cleaning, new install scripts and couple of secret projects. But nothing is a secret with Booktype, if you want to be up to date with our development watch our github page, check my tweets and read my blog!

Getting Started with Booktype – What is Booktype?

This is a first post in a series of articles about Booktype. Before I start to be all technical and “… and that is how you install new Django app” it would be good to say couple of words about Booktype history.

Booktype is an open source platform to write and publish print and digital books. It was created to run the FLOSS Manuals platform and back then it was just called Booki. FLOSS Manuals is a community of people joined together for their love of writing good quality documentation for FLOSS software. At first the  FLOSS Manuals platform was build on the  TWiki (Wiki in Perl) platform.

Using  TWiki to prototype something was a good choice. We extended it with our own plugins and some external tools. FLOSS Manuals is a collaborative community and we needed to extend TWiki to support the collaborative processes. We had our IRC-Web chat gateway called FM Transmission [post in Croatian], live Table of contents where you could see what other users are doing (view image of it), Remixer for content, Localization plugins to manage book translation, our own Web-based tool for translating content and many other things. It was good enough platform but it felt like a big hack at times. After some thinking and observing some problems in the TWiki community [post in Croatian] we decided to write a new platform –  from scratch.

We wanted to build community around software and we figured out TWiki and Perl were not the best platform for that. Mainly because a lot of our users are people who are more interested in books and less in Perl code. We decided to use Python and Django as our new platform. They both seems to be reasonably easy to master and even newbies could extend their Booktype installations. You want to have blog on the front page? Just install existing Django blog app, change one line in settings file, put couple of tags in templates and you are good to go. Simples! Anyone could do it.

Booktype

When we talk about Booktype we really talk about Booktype and Objavi. Booktype is used for editing and managing books while Objavi is used to produce book formats (as pdf, epub, mobi, …). In this post we will talk only about Booktype (the book editor).

Booktype is all about the collaborative editing of books and as such is not always what people want. It is about building community around your book. Sure, you can use it to write your diary also (and you should!), but the real power is in collaborative work. What does this mean for the architecture?

Like i said, we decided to use Django as web framework mainly because we wanted people to be able to plug Booktype into their existing web sites more easily and extend their installations with many already existing Django apps. I think we made a good choice with it. As database backend we are using PostgreSQL. Sqlite3 is also an option, but it really should be used just for testing and development purposes. We are using Redis in-memory database to store all kind of information web clients are exchanging with Booktype server using Sputnik.

What is Sputnik? Sputnik is a Django application for two-way communication with web clients. When we started there was no other Django project that could help us with this so I decided to do what everyone would normally do. I wrote one abstract API (with channels, subscriptions and etc…) for communication and implemented dummy network layer. It means when the right moments comes this network layer can be replaced with something better without any need to change the rest of the code. There are many good frameworks these days that could be used as network layer (Socket.IO for instance). Why do we have it? Mainly because of the collaborative part of Booktype. When you have 10 people working on one book at the same time you need a way of notifying all of them about the actions others are doing. Like: locking chapters, chat, changes in Table of Contents, creation of new chapters, attachments uploads and etc. Yes, it means your Table of Contents will be changed as soon as someone adds a new chapter to it. This really improves the user experience and stops you from overwriting someone’s changes.

But enough about Booktype history. In my next article I will talk about exciting world of “Booktype projects” and how you can easily install and extend your Booktype installation!

These are the Booktype links  you should probably check:

 

 

 

Javni natječaj oko moje nove domene



Daklem, nabavio sam si domenu “kurv.in” i nemam baš pretjerano puno ideja što da s njom napravim. Jedna od ideja je e-mail adresa “sin@kurv.in” ali tu negdje staje sva moja maštovitost.

Ako netko ima bolju ideju neka ju slobodno pošalje na aerkalov@gmail.com. Bogate nagrade su obećane ali za sad ništa ne garantiram. Nema nikakvih ograničenja i otvoren sam za sve ideje i prijedloge.

Netko ima tajni projekt….


Daklem… Aco ima tajni projekt. Iz ovog drugog drafta logoa se ne da previše zaključiti, ali samo ću reći da se radi o nečemu što sam trebao raditi prije 10 godina dok je još i imalo smisla. Ne sad… sad više nema smisla ali se i dalje isplati probati. Neću sa 60 godina valjda da uzdišem i samom sebi da govorim “ahhh… možda sam ipak trebao.”. Eh da… u ovom kriminalu imam i partnera. On je onaj B u Baco.

Pa sretno nam bilo!