Editing Opt Problem

Warning: You are not logged in. Your IP address will be recorded in this page's edit history.

The edit can be undone. Please check the comparison below to verify that this is what you want to do, and then save the changes below to finish undoing the edit.

Learn more about Contributing to the wiki.

= The /opt Problem =

There is not enough space on the [[rootfs]] to fit all applications, therefore an additional partition has to be used. But how?

There is a section in the Maemo 5 Developer Guide explaining the basics: 
[[Documentation/Maemo_5_Developer_Guide/Packaging,_Deploying_and_Distributing/Installing_under_opt_and_MyDocs|Installing under /opt and /MyDocs]]

= Current situation =

* 256MB OneNAND chip: bootloader, kernel, root (ubifs - with compression)
* 32GB eMMC: /home ~2GB (ext3), /home/user/MyDocs ~29GB (vfat), swap 0.7GB
* symlink: /opt/ -> /home/opt/
* OneNAND is probably "much" faster than the eMMC (hard numbers? read? write? small blobs? big blobs?)
* file system [http://talk.maemo.org/showpost.php?p=504332&postcount=76 benchmark results] for different media
* OneNAND performance figures: Sustained read performance: 108MB/s. Sustained write performance: up to 17MB/s. See [http://www.samsung.com/global/business/semiconductor/products/fusionmemory/downloads/onenand_brochure_200609.pdf Samsung OneNAND specs].
* eMMC performance figures: read performance 37MB/s. Write performance 20MB/s. Write speed of 17MB/s has been achieved in this test at [http://symbianbriefs.blogspot.com/2010/01/n900-review-part-10-transferring-data.html symbianbriefs.blogspot.com]. My own read tests however show ~11MB/s (370MB file).
* Decisions were taken in a "what shall we do about /opt?" BOF at the [[Maemo Summit 2009]]. Minutes: http://lists.maemo.org/pipermail/maemo-developers/2009-October/021289.html

= Some goals =

* Firmware should be flashable.
* Should not break applications.
* One should be able to install more than 256mb worth of software.
* Standards compliance, less need to make incompatible quirks.
* A fix should be doable after shipping (before shipping, probably not?)

= Current problems / developer and packaging issues =

This should be a list of current problems developers are experiencing related to optification:

* [http://lists.maemo.org/pipermail/maemo-developers/2009-October/021419.html Optification breaks package on upgrade from non-optified older version]

* Symlinks take up space in the rootfs. This may become an issue if there are lots of them.

* [[/Non-Optified_packages|List of non-optified packages and applications]] - Please feel free to add every piece of software which is using too much space in rootfs

= Tool for detection of non-optified packages =
[[Image:Storageusage.png|thumb|300px|<br>[http://code.google.com/p/storageusage/ Storage Usage] provides a great visual way to detect non-optified packages.
]]

= Possible solutions, even the bad ones =

==== Applications are made relocatable and installed to /opt ====
* Relocatable applications can move everywhere without breaking.
* No symlinks get scattered over the rootfs (symlinks take up space, too).
* Not all applications can easily be made relocatable.
* Requires more work by the programmer.
* May not be feasible for ports from an upstream source.
* Putting a startup script into /usr/bin setting up environment variables such as LD_LIBRARY_PATH can help.
* autotools-based packages can use --prefix=/opt/<packagename>.
* Python apps can make use of the [http://docs.python.org/reference/datamodel.html#index-821 __file__] module-scope variable to determine where they are installed (e.g. installdir = os.path.dirname(__file__)).
* Could be taken into account as an alternative to maemo-optify wherever feasible.

==== Have a 2-5gb partition on the emmc for addon packages mounted to /usr/local or /opt ====
* Most apps by default usually go there
* for /opt store according to the FHS standard /opt/<package> and symlink to /opt/maemo/{bin,lib,share,...} (actually the standard is /opt/<vendor>/whatever In this case using /opt/maemo directly would work).
* Having ld.so.conf and PATH set properly avoids all the symlink madness
* Don't do symlinks for /usr/local|/opt - make it a real partition
* Explain to people why they don't have full 32GB of storage - or use LVM on the eMMC to dynamicaly change the size as things get installed through some dpkg hooks or such - we do know how much it will take ahead of time so if there isn't enough storage do a resize(VFAT should be resizable with something like gparted maybe?) ext3/2 are both online resizable
* Requires a bit more work when packaging - make some defaults pre-set in SDK
* All apps should use a normal usr layout underneath any path. Example: /opt/maemo/{bin,sbin,lib, etc...} same if it would be in /usr/local - works like GNU stow

==== store addon packages completely in /home/opt hierarchy ====
* [http://wiki.maemo.org/User:Tanner#full_optification see this page]
* like previous solution but on the standard /home partition
* very clean solution as no files (except for /var/lib/dpkg which may be moved as well) are created or modified on the NAND.
* /etc and /var could be copied there as well and mounted to root using bind. This way OneNAND would only be changed be reflashes and if /home is not mounted (e.g., by a keypress during boot) a fallback  system could boot from unmodified OneNAND.
* if OneNAND is significantly faster, mark some often used shared libraries as candidates for a /usr/lib/cache directory in which they are copied during installation until a certain cache limit is reached.

==== During packaging move all files >500kb to /opt and symlink to root ====
* Optify is basically this.
* Install is not slowed down, work is done when packaging.
* Some packages break when replaced with symlinks. How to prevent - hack on hack?
* A buildbot could force this to debian/rules automatically.

==== Use aufs/unionfs to overlay OneNAND root with eMMC filesystem. ====
* This could be partial overlay, for example only /lib and /usr. 
* Could keep essential libraries on OneNAND so they would work faster.
* Has anyone measured the speed penalty for aufs/unionfs usage?
* Is it faster or slower that moving root to eMMC?
* Might be difficult to have new programs to benefit from faster OneNAND.

==== Hack with --datadir, --libdir etc. through debian/rules ====
* Could be forced by a buildbot.
* Should move most of the big lumps on larger media if there are some.
* Works at least with packages with configure.
* Requires at least additional $LIBDIR setup?
* Programs which have hardcoded paths (ugly code) break.
* Could additionally symlink files from opt to root.

==== Require that applications have --prefix /opt ====
* Could be forced by a buildbot.
* Should not break anything.
* Programs would not benefit from faster OneNAND.
* Requires at least additional $PATH setup?
* Programs which have hardcoded paths (ugly code) break.
* Could additionally symlink files from opt to root.
* Just reminds me of 770's /var/lib/install..
* Against standard

==== Move root from OneNAND to eMMC and combine with /home to create a new root ====
* How much faster OneNAND is? Numbers?
* Can flasher flash on eMMC?
* Leave only the bootable and rescue stuff in the rootfs
* Maybe use it uncompressed for speed and add a /fast/usr folder to OneNAND to store some system files that would benefit from the speed (example: move /usr/something to /fast/usr and symlink it or just place /fast/usr in the path)

==== Use mount -o bind (or -o rbind) ====
* Method: Do not use NAND for any additional data. Boot into standard image. Any additional changes made are only available after -o bind or -o rbind which are defined in /etc/fstab . Setup provides potential fail-over; if -o bind or -o rbind fails user gets into 'rescue shell' providing 1) recovery method 2) quick install to vanilla firmware without 3rd party utilities, networking, or computer (think: no network available, expensive roaming, no computer with fwflasher or Nokia Software Downloader available, ...)

==== Turn the problem around, pivot the root with /opt ====
* After pivot /opt would be root and old root would be /opt. Symlink the old root contents to new root would make those available.
* Can the base system dependencies handle this?
* Can Nokia created software handle this? 
* Could this work? If so, then all additional packages would install directly on eMMC.

==== After install move all files >500kb to /opt and symlink to root ====
* Could be done at background after install or with post-install hook.
* There might not be enough space on / even for install.
* Install is already slow, could make it even slower.
* Some packages break when replaced with symlinks. How to prevent - hack on hack?
* Uninstall might be messy, some post-remove magic needed.

==== Chroot install in /opt, symlink to where it belongs ====
* A Bit like stow.
* Maybe somebody can see something good in this?
* Difficult hack.. would need all the tools etc to be visible in the chroot.

==== Make /usr another partition ====
You know, historically /usr exists entirely for the purpose of separating the application partition from the underlying system, which is exactly our problem.
* All the core nokia software and libraries should be relocated to / or maybe a /maemo.
* /usr and /opt should become symlinks to /home/usr and /home/opt.
* Maemo targeted GUI software would be encouraged to use /opt/[application].
* Linux libraries and CLI software could safely use /usr, minimizing the required modifications.
* Any frequently used accessed files in /home but not /home/user/MyDocs could be cached in /cache, which then gets overlaid with /home using unionfs or something more specialized.
* Another location like /alt could be used by packages that know they need the NAND's speed, but this is discouraged.

[[Category:Development]]

Please note that all contributions to maemo.org wiki may be edited, altered, or removed by other contributors. If you do not want your writing to be edited mercilessly, then do not submit it here.
You are also promising us that you wrote this yourself, or copied it from a public domain or similar free resource (see maemo.org wiki:Copyrights for details). Do not submit copyrighted work without permission!

Summary:

Cancel | Editing help (opens in new window)

Retrieved from "http://wiki.maemo.org/Opt_Problem"

Personal tools

Navigation

Views

Editing Opt Problem

@@ Line 57: / Line 57: @@
 * Having ld.so.conf and PATH set properly avoids all the symlink madness
 * Don't do symlinks for /usr/local|/opt - make it a real partition
-* Explain to people why they don't have full 32GB of storage - or use LVM on the eMMC to dynamicaly change the size as things get installed through some dpkg hooks or such - we do know how much it will take ahead of time so if there isn't enough storage do a resize(VFAT should be resizable with something like gparted maybe?) ext3/2 are both online resizeable (growth only!)
+* Explain to people why they don't have full 32GB of storage - or use LVM on the eMMC to dynamicaly change the size as things get installed through some dpkg hooks or such - we do know how much it will take ahead of time so if there isn't enough storage do a resize(VFAT should be resizable with something like gparted maybe?) ext3/2 are both online resizable
 * Requires a bit more work when packaging - make some defaults pre-set in SDK
 * All apps should use a normal usr layout underneath any path. Example: /opt/maemo/{bin,sbin,lib, etc...} same if it would be in /usr/local - works like GNU stow