Evopedia

(Dumps)
(Undo revision 46005 by 114.79.28.10 (Talk) vandalism/spam)
 
(7 intermediate revisions not shown)
Line 1: Line 1:
Evopedia is an offline Wikipedia viewer. It allows to search for articles based on their title or the geographic location on a map. If there is a connection to the internet, even images are displayed.
Evopedia is an offline Wikipedia viewer. It allows to search for articles based on their title or the geographic location on a map. If there is a connection to the internet, even images are displayed.
-
Apart from the software you also need to download a specially prepared dump of the articles available here.
+
'''The information here is partly outdated. Please visit the
 +
[http://evopedia.info project website].'''
== Dumps ==
== Dumps ==
Line 13: Line 14:
Note that you need evopedia version 0.3.0 (NOT 0.3.0 RC 3) for the search function to work properly for languages not based on the latin alphabet.
Note that you need evopedia version 0.3.0 (NOT 0.3.0 RC 3) for the search function to work properly for languages not based on the latin alphabet.
-
 
-
'''Converted dumps:'''
 
-
 
-
These dumps are from older versions of evopedia, some features may be missing.
 
-
 
-
* [http://www.reitwiessner.de/openmoko/wikipedia_dumps/wikipedia_nl_2009-10-16.zip Dutch Wikipedia], 2009-10-16 (0.8 GB)
 
-
* [http://evopedia.info/dumps/wikipedia_en_2009-02-28.tar.bz2.torrent English Wikipedia], 2009-02-28 (4.9 GB), BitTorrent, please see above for more recent version
 
-
* [http://www.reitwiessner.de/openmoko/wikipedia_dumps/wikipedia_eo_2009-11-17.zip Esperanto Wikipedia], 2009-11-17 (0.1 GB)
 
-
* [http://www.reitwiessner.de/openmoko/wikipedia_dumps/wikipedia_fr_2009-10-11.zip French Wikipedia], 2009-10-11 (1.9 GB)
 
-
* [http://www.reitwiessner.de/openmoko/wikipedia_dumps/wikipedia_it_2009-11-09.zip Italian Wikipedia], 2009-11-09 (1.7 GB)
 
-
* [http://www.reitwiessner.de/openmoko/wikipedia_dumps/wikipedia_ja_2009-03-01.zip Japanese Wikipedia], 2009-03-01 (1.5 GB) (pages are a bit broken but content should be there)
 
-
* [http://www.reitwiessner.de/openmoko/wikipedia_dumps/wikipedia_es_2009-11-16.zip Spanish Wikipedia], 2009-11-16 (1.4 GB)
 
-
 
Dumping Wikipedia is a very time-consuming task. Please be patient or [http://dumpathome.evopedia.info/contribute help creating the dumps].
Dumping Wikipedia is a very time-consuming task. Please be patient or [http://dumpathome.evopedia.info/contribute help creating the dumps].
Line 42: Line 30:
== Tweaks ==
== Tweaks ==
-
Evopedia creates a configuration file at /home/user/.evopediarc when it is first started. Most of the options are self-explanatory.
+
Evopedia creates a configuration file at <code>/home/user/.evopediarc</code> when it is first started. Most of the options are self-explanatory.
By changing listen_address to 0.0.0.0 you (or anybody else, so be careful) can also access evopedia from a different computer using the url http://<ip adress of your device>:8080/
By changing listen_address to 0.0.0.0 you (or anybody else, so be careful) can also access evopedia from a different computer using the url http://<ip adress of your device>:8080/
Line 62: Line 50:
==== 1. install needed software (please add packages if I forgot some): ====
==== 1. install needed software (please add packages if I forgot some): ====
-
apt-get install php5-cli php5-mysql python mysql-server mysql-client wget zip tidy git-core texlive texlive-math-extra  
+
apt-get install php5-cli php5-mysql python mysql-server mysql-client wget zip tidy git-core texlive texlive-math-extra  
On some distributions, texlive must be replaced by tetex and texlive-math-extra by tetex-extra
On some distributions, texlive must be replaced by tetex and texlive-math-extra by tetex-extra
==== 2. download evopedia source code ====
==== 2. download evopedia source code ====
-
 
+
git://gitorious.org/evopedia/evopedia.git
-
git clone git://github.com/crei/evopedia.git
+
 +
Note: '''Please don't use the github address anymore, it's deprecated:'''
 +
<strike>git clone git://github.com/crei/evopedia.gitr (deprecated) </strike>
==== 3. adjust settings ====
==== 3. adjust settings ====
-
Set php's memory_limit to some higher value (128 MB) in /etc/php5/cli/php.ini (or similar)
+
Set php's memory_limit to some higher value (128 MB) in <code>/etc/php5/cli/php.ini</code> (or similar)
Create empty mysql database and grant all rights to some user -  
Create empty mysql database and grant all rights to some user -  
Line 80: Line 70:
  mysql> grant all privileges on wikidb.* to thomas@localhost identified by 'x1lx';
  mysql> grant all privileges on wikidb.* to thomas@localhost identified by 'x1lx';
-
Edit paths and database settings in dumpscripts/createdump.sh
+
Edit paths and database settings in <code>dumpscripts/createdump.sh</code>
(especially: dbuser, password, REPODIR, DUMPDIR) - make sure DUMPDIR points to area with plenty of space
(especially: dbuser, password, REPODIR, DUMPDIR) - make sure DUMPDIR points to area with plenty of space
Change the dump language ("de" in "for" statement at bottom of dumpscripts/createdump.sh)
Change the dump language ("de" in "for" statement at bottom of dumpscripts/createdump.sh)
-
Make ~/evopedia/evopedia/evopedia/*.py files executable (chmod a+x ...).
+
Make <code>~/evopedia/evopedia/evopedia/*.py</code> files executable (chmod a+x ...).
==== 4. start the dump process ====
==== 4. start the dump process ====
-
call createdump.sh
+
call <code>createdump.sh</code>
(Don't be too optimistic though. After part of the day the database is filled with several gigabytes of downloaded wikipedia content, but something is broken and the final dump gives no results.)
(Don't be too optimistic though. After part of the day the database is filled with several gigabytes of downloaded wikipedia content, but something is broken and the final dump gives no results.)
[[Category:Software]]
[[Category:Software]]

Latest revision as of 06:30, 15 December 2011

Evopedia is an offline Wikipedia viewer. It allows to search for articles based on their title or the geographic location on a map. If there is a connection to the internet, even images are displayed.

The information here is partly outdated. Please visit the project website.

Contents

[edit] Dumps

BitTorrent

Since downloads of the dumps caused 2 TB of data transfer on the server in one week, the big dumps are now only available via BitTorrent. Please install a BitTorrent client (if not already installed on your system) to use the .torrent files.

New dumps:

Note that you need evopedia version 0.3.0 (NOT 0.3.0 RC 3) for the search function to work properly for languages not based on the latin alphabet.

Dumping Wikipedia is a very time-consuming task. Please be patient or help creating the dumps.

Please report if you find errors in the dumps.

Distributed dump processing

[edit] Dump Installation

Extract one (or more) of the .zip-Files to your device and select it from within evopedia. The dump that is used can be changed using the (second) link at the top-left of the search screen.

Note that you cannot directly download dumps greater than 4 GB onto your device (FAT file size limitation). Please download it to a computer with a non-FAT filesystem and then unzip it directly onto your device (via USB for example).

[edit] Tweaks

Evopedia creates a configuration file at /home/user/.evopediarc when it is first started. Most of the options are self-explanatory.

By changing listen_address to 0.0.0.0 you (or anybody else, so be careful) can also access evopedia from a different computer using the url http://<ip adress of your device>:8080/

maptile_repositories is a string that specifies which map types are available and where the tile images are stored. If it is empty, tiles are not stored at all and only OpenStreetmap is available. If this is not done automatically, you can set it to

maptile_repositories = [OpenStreetMap I|http://tile.openstreetmap.org/%d/%d/%d.png|/home/user/MyDocs/.maps/OpenStreetMap I/|0,OpenStreetMap II|http://tah.openstreetmap.org/Tiles/tile/%d/%d/%d.png|/home/user/MyDocs/.maps/OpenStreetMap II/|0,OpenCycleMap|http://c.andy.sandbox.cloudmade.com/tiles/cycle/%d/%d/%d.png|/home/user/MyDocs/.maps/OpenCycleMap/|0,Public Transport|http://tile.xn--pnvkarte-m4a.de/tilegen/%d/%d/%d.png|/home/user/MyDocs/.maps/Public Transport/|0]

(without line breaks) to use the files also used by maep.

[edit] Creating a Wikipedia Dump

Please consider joining the distributed dump system mentioned above. Also, the information below is a bit outdated.

Creating a Wikipedia dump is unfortunately a rather lengthy process (some days). If you have any suggestions for speeding it up, please step forward. Concerning space, 20 GB should be enough for creating the dump.

For the advanced users, note that the process can be distributed over many computers by calling dumpWiki with the slice number to process on each computer.

[edit] 1. install needed software (please add packages if I forgot some):

apt-get install php5-cli php5-mysql python mysql-server mysql-client wget zip tidy git-core texlive texlive-math-extra 

On some distributions, texlive must be replaced by tetex and texlive-math-extra by tetex-extra

[edit] 2. download evopedia source code

git://gitorious.org/evopedia/evopedia.git

Note: Please don't use the github address anymore, it's deprecated:

git clone git://github.com/crei/evopedia.gitr (deprecated) 

[edit] 3. adjust settings

Set php's memory_limit to some higher value (128 MB) in /etc/php5/cli/php.ini (or similar)

Create empty mysql database and grant all rights to some user - e.g. for database named wikidb, user thomas and password x1lx:

> mysql -p
mysql> create database wikidb;
mysql> grant all privileges on wikidb.* to thomas@localhost identified by 'x1lx';

Edit paths and database settings in dumpscripts/createdump.sh (especially: dbuser, password, REPODIR, DUMPDIR) - make sure DUMPDIR points to area with plenty of space

Change the dump language ("de" in "for" statement at bottom of dumpscripts/createdump.sh)

Make ~/evopedia/evopedia/evopedia/*.py files executable (chmod a+x ...).

[edit] 4. start the dump process

call createdump.sh

(Don't be too optimistic though. After part of the day the database is filled with several gigabytes of downloaded wikipedia content, but something is broken and the final dump gives no results.)