The Epik-Builder Homepage
 Project Theseus |
|
Epik-Builder Epik-Browser Embedded Terminal Server TOOLS Architecture Lazarus/FPC Links Credits SourceForge Summary About Thesesus |
About
Epik-Builder is an object oriented installer and bootable image creator for the Linux Operating system. It is a tool and packaging system that supports two major functions: the creation of generic Linux install images from distribution packages and the configuration of those images to boot from CDR, hard disk, User Mode Linux or any other user defined boot process.
An architectural overview of Epik-Builder can be found here. More detailed architectural and program design information can be found here.
Status
Posted by Tom Lisjac - June 18, 2003
Epik-Builder's architecture and an initial command line prototype have been completed. Pre-release documentation can be found here. I'm currently setting up CVS for the initial release of the CLI version and the first set of EPS templates.
A Graphical User Interface is also being developed using Lazarus and should be ready for beta release within two weeks. This will make the EPS templates a lot easier to navigate.
News
June 1, 2003 Pre-release documentation is now available for Epik-Builder
Some pre-release architectural and program documentation has been posted for Epik-Builder. This is just a start... I'm working on diagrams and a few additional web pages. (Epik-Builder is a tool for building customized CD, Hard Disk or UML bootable system images directly from a distribution's generic packages.)
Epik-Builder - Overview
Summary
Epik-Builder is an object oriented installer and bootable image creator for the Linux Operating system. It is a tool and packaging system that supports two major functions: the creation of generic Linux install images from distribution packages and the configuration of those images to boot from CDR, hard disk, User Mode Linux or any other user defined boot process.
Epik-Builder provides a framework to capture package lists, configuration data and "experience" in a file structure called an "EPS template". These templates can be "inherited", making it easy to build a wide variety of complex install images from a tree of simple and well tested "base classes". EPS templates, in combination with a distribution's packages, can generate a highly customized and fully configured Linux system for any desired target media. The overall architecture of Epik-Builder is shown in the figure below.
EPS Templates
Epik-Builder uses the Epik Packaging System (EPS) templates to control all installation and image building operations. An EPS Template is a recursive data object that contains package, configuration and deployment information. EPS Templates can stand alone, inherit from parent templates or be parents to other template subclasses. The structure of an EPS Template is illustrated below.
EPS Templates only contain the data and methods that are needed to customize a distribution's packages for a given set of requirements. As a result, they are relatively small and easily exchanged. This makes it possible for dialup users to build customized CD, HD and UML images by downloading the templates that other people have created and applying them to their distribution's package repository. New EPS Templates are also easy to create, modify or "inherit".
EPS Templates simplify maintenance because new capabilities are added in separate layers that contain and encapsulate all the installation specifications, customization and specific deployment features for a given application. Keeping an entire EPS tree updated is also simplified because any security or package upgrades that are applied to the distributions's package repository will be "inherited" by all the templates that were derived from them.
Distribution, Package and Media Independent
The EPS architecture can be adapted to any distribution or packaging system by writing a small "packager.sh" script that contains three simple functions. The first maps package names to file names. The second installs an individual package using the selected distribution's packaging tool. The third returns query information about a specified package. Epik-Builder calls these functions to create a generic installation tree that can be morphed into any of the supported deployment image types... which are currently iso, cloopiso, hd and uml. Other image types can be added for any desired media by writing a "createimage.sh" script. This script contains 3 function entry points: PatchInstall, MakeImage and DeployImage.
Epik-Builder - Architecture and System Design
Overview
The Epik-Builder system consists of two components: a set of shell scripts that do most of the work and a GUI wrapper that provides a comfortable end user experience. Together, these programs create, control and maintain a filesystem based data structure called an "EPS Template". EPS Templates provide the data and methods that are required to build customized Linux installs and bootable images.
Structure: epik-builder.sh
The epik-builder.sh bash script contains the generic program logic and template verification routines. It provides a getopts command line interface for inputting user commands in addition to routines that verify those commands and their operands. Epik-Builder's parameter verification philosophy is to be demanding and to error out with an informative message if something isn't "just right". It doesn't always do this... but that's the goal!
Building your own custom distribution consists of creating a new EPS template (preferably derived from an existing one) and customizing it by writing template build scripts to make the image into what you want the new implementation to do. Epik-Builder and can then be used to verify the template and inheritance tree for correctness and perform the build sequencing and image deployment.
Details: EPS Templates
The Epik Packaging System (EPS) is based on a "project" directory that contains one or more application specific templates. The templates are a filesystem tree that contain all the components, tools, build scripts and patches to completely install, patch and format a given system for its target image and media. Currently the creation of iso, hard disk and uml images is supported... but the architecture will build images from any distribution's packaging system for any media or boot environment.
EPS templates are usually created under a "project" root. For example:
epik.eps/proj/
redhat9.eps/proj/
core.eps/
base-with-knoppix-autodetect.eps/
core-base-and-X.eps/
kiosk.eps/
office.eps/
webserver.eps/
developer.eps/
debian3.eps/proj/
core.eps/
base-with-knoppix-autodetect.eps/
core-base-and-X.eps/
slack90.eps/proj/
core.eps/
base-with-knoppix-autodetect.eps/
core-base-and-X.eps/
blfs41.eps/proj/
core.eps/
base-with-knoppix-autodetect.eps/
core-base-and-X.eps/
This shows a core template and derived kiosk, office, webserver and developer templates under an epik.eps project root. The templatesare coded to generate specific build and system images for their end applications. The eps.info file contains global options and pointers for the templates. Each template has the option to override the global eps.info by creating a local one in it's root.
The root of an EPS template (like the kiosk example above) contains the following branches:
install - Where a clean OS tree is built from distribution packages and stored
images - Where bootable images are built for the target media using the install tree
backup - Where build scripts and addon packages are archived after each build
components - Additional programs & utilities intended for target system integration
proj - Location to create new projects and templates that use this one for a root
doc - Documentation on the template's purpose, specialized tools or images goes here
bin - Tools required by Epik-Builder for maintenance and creating installs and images
logs - Place where all Epik-builder logs for the selected operation are written.
The template and inheritance tree structure is verified by all Epik-Builder commands. Most of these commands are locked directly to the following declaration... but some aren't. If you modify the structure of the EPS Template, carefully check to make sure that the commands won't be affected or even run amuck and damage your system. Remember, there is no warranty for this software! :-o
EPS Root Structure
Here is the root EPS template declaration in creation order:
EPSROOTTEMPLATE="install \
images \
backup \
doc \
logs \
components \
components/src \
proj \
bin \
bin/src \
install/src \
install/patches \
install/specs \
install/packages \
install/packages/release
install/packages/updates
install/packages/addin
install/packages/src
install/systree"
Here's a graphical representation:
EPS Template Path Definitions
Under /install:
/specs is where the package list is placed that will drive the creation of the clean install image from the distribution package cache. A createinstall.sh script is also located in this branch that will execute before and after the build process so pre and post patching of the pristine install tree can be made.
/patches is where any patches to the clean install tree should be placed in the FHS location where they are required.
/packages is where the original release package set and any additional packages (rpm, deb, etc) that aren't in the distribution's release set can be placed. The directories are searched during install creation in the following order: addin, updates and release. The configuration file eps.conf also provides a DISTROPKGS path variable that allows you to place your packages in another location. If specified, Epik-Builder will check DISTROPKGS for the same subdirectory format as the template requires: addin, updates and release subdirectories must be present.
/systree is where the actual installation tree is built from the package cache. After building, the contents of this branch is compressed into an install.tar.gz and /systree is optionally cleaned to save disk space. (Disk space can be a problem when building large system specs)
Under /images:
The images branch contains additional branches for each boot media/environment desired. For example, the /images branch would contain an /iso branch for creating a bootable CD or a uml branch for generating UML bootable images. Image types are defined by the user using the -A add new image type command or, in the usual case, inherited from existing templates. The initial release of Epik-Builder will include a base template that supports cloopiso, hd and uml support.
The following tree is created under each supported image type. The structure of this tree is verified before any operations are performed. This structure is where the work of generating customized images is done:
EPSIMAGETEMPLATE="patches src specs "
/patches is a FHS tree that contains patch files and programs to be applied to the os image that is being built.
/tmp is for temporary storage of intermediate files used in the build and for mounting loop devices used for building initrd's and boot floppies.
/specs is where the createimage.sh script is located. It is called from this script and is responsible for building the specific image, configuring the boot environment and patching the system tree for anything that is required to make the system work in the target environment. Epik-builder doesn't know anything about images or target environments... it relies on createimage.sh to do all the "heavy lifting" in getting the image built and properly configured.
/systree starts as a direct copy of the install tree that was stored in install.tar.gz. It is poked, patched and cleaned as necessary to create the specific image needed by the target image and boot environment.
/buildtree is where the final image is assembled. In the case of a compressed CD, a compressed iso is created from /systree and placed here along with the isolinux or floppy boot. For a cd, the final iso image is created using this as the root directory. The final iso is then stored in images/iso.
Buildsite directory structure
The BUILDSITE variable points to the root of a tree that is used as a workspace for all install and image building operations. It defaults to the ebtmp/ subdirectory of the selected template but can be relocated to any unrestricted area of the filesystem by setting BUILDSITE in eps.conf. By default, buildsites are cleaned to save disk space (Disk space can be a problem when building large system specs) but they can be optionally retained for debugging purposes by setting KEEPBUILDSITE in eps.conf.
BUILDSITETEMPLATE="systree buildtree tmp mnt"
# /systree is where the actual installation tree is built from the package cache or a previously installed archive. After an initial install, the contents of this branch is compressed into an TEMPLATENAME-install.tar.gz. For image builds, this subdirectory holds a decompressed copy of the install tree that was stored in this archive. It is modified, patched and cleaned as necessary to create the specific image needed by the target image and boot environment.
/buildtree is where the final image is assembled. In the case of a compressed CD, a compressed iso is created from systree and placed here along with the isolinux or floppy boot. For a cd, the final iso image is created using this as the root directory. The final iso is then stored in images/iso.
/tmp is for temporary storage of intermediate files used in the build
/mnt is used for mounting loop devices used for building initrd's and boot floppies.
Globals and search paths
The templates in a given project tree could contain a LOT of redundant data... so the concept of a "path" is supported. If the search for a particular patch, package or tool fails in the active template, the parent tree will be searched all the way to the Epik-Builder root. The search mechanism will return the item from the first location where it is found. This allows large and commonly used items (such as X drivers, build sources, common binary packages and initrd trees) to be located anywhere in the inheritance tree where they can be efficiently shared without the need to specify absolute file locations to the requester.
The same mechanism is used to search for installation packages. The template/install/packages directory is searched first, followed by the path defined in the DISTROPKGS variable.
EPS Inheritance
In order to reduce the effort required to construct new system types, Epik-Builder supports the object oriented concept of "template inheritance". This allows large and complex systems to be derived from a chain of smaller and simpler "parent classes". The implementation supports single inheritance within the createinstall.sh and createimage.sh script chains. Subclasses may call any parent methods that are defined within these scripts.
Creating a new EPS template creates prototype scripts that contain pre-formatted "Inherited" calls... so a new template will provide exactly the same behavior as the parent it was derived from. Inherited calls can be overridden by commenting them out. Code can also be added before or after the inherited call to the parent class is made.
In a derived system, inheritance calls are of the following form:
function ConfigureServices() {
Inherited $FUNCNAME "$@" # Pass the function name as the first parm... the rest as "$@"
... your specific code to customize this subclass AFTER the parent template does it's thing.
}
To completely override global inheritance, simply drop out the Inherited call:
function ConfigureNetworking(){
#Inherited $FUNCNAME "$@"
... Your specific code to COMPLETELY REPLACE the logic in the parent template...
Note: Doing this interrupts the inheritance call chain and makes this template the "top" of
the inheritance tree for this method.
}
Currently the EPS inheritance depth is set using the $MAXLEVELS variable. This is done for error trapping purposes in case there's corrupted data in an eps parent link. The number is completely arbitrary, however, and can be increased if needed.
Template Signatures
Each template is assigned an MD5 signature based on a scan of selected files and subdirectories that determine if it has been changed in a way that will affect generated installs or images. The following declarations declare what directories will be included in the signature scan.
CheckedDirs="components \
bin \
install/src \
install/patches \
install/specs"
ImageDirs="patches specs src"
EPS Template descriptor
To support template inheritance and signatures, a eps.info record descriptor is present in the root of every template. These are generated when a new template is created and updated as needed.
EPSNAME="" # The name of this template: the name of the epik.eps would be epik
DESCRIPTION="" # Optional short description of what this template is for
AUTHOR="Tom Lisjac <vlx@users.sourceforge.net>" # The template creator
CREATED="" # When this template was initially created
LASTUPDATED="" # when the template was last modified
SIGNATURE="" # MD5 signature of the template per the discussion about Template Signatures
PARENTNAME="" # If there is a parent template, this is it's name
PARENTPATH="" # This is the absolute or relative (default) path to the parent
PARENTSIG="" # This is the MD5 signature of the parent
PROVIDES="" # What this template provides (not implemented yet)
REQUIRES="" # What this template requires (not implemented yet)
EPS Configuration Defaults
All templates and inheritance trees contain a default settings file named eps.conf. The file defines distribution and package specific settings for Epik-Builder to use during install and image creation operations. There is usually only one of these files per in any given distribution tree. When a process is started, Epik-Builder looks for eps.conf in the root of the specified EPS template and will search the inheritance tree until one is found. An error is generated if one isn't. The declarations in eps.conf are listed below.
Abbreviated name of the distribution... for display and reference.
DISTRO=""
If you're going to do any building, this is an alternate location for the package repository that contains the distribution packages. The template/install/packages subdirectory is searched first in addin, updates and release order... then this directory will be searched in the same order and with the same prefixes. If specified, the structure of this directory is verified and an error is generated if it doesn't conform.
DISTROPKGS=""
Users can add their own space delimited image types with this entry. These are not checked by internal verification routines, however. It is up to the implementor to provide verification in the PatchInstall and MakeImage functions.
USERIMAGETYPES=""
Website, phone numbers or other contact info to support this deployment
SUPPORTINFO="http://theseus.sf.net"
Default Install switches... added to any found in installspec.txt for each package. These are passed to the packager.sh script for each distributions's package tool. An example for an RPM system would be: "--force --nodeps --nosignature --nodigest"
PKGSWITCHES=""
Currently unused... backups default to the template backup subdirectory
BACKUPDIR="/tmp"
Prefered architectures to select packages from... in order of preference. Package selection can be influenced by this setting. Currently implemented in the RPM packager.sh.
PREFARCH="i386 i586"
Set "Y" to backup all the scripts and add on packages for each build. Automatically saves to a tree to a tar.gz to the selected image under /epikbuild. This was of value when I was initially debugging... but I may remove it in the next version.
MAKEIMAGEBACKUPS="N"
Set to "Y" to use isolinux, "N" to use floppy emulation for booting CDs
USEISOLINUX="N"
Comment these out to erase raw install systrees after compressing them. Can save a LOT of disk space if you aren't debugging something or need to inspect the tree!
DEBUGINSTALL="N"
Same as DEBUGINSTALL but as applied to the image creation process. If set to N, deletes the buildtree where all the patches and additional packages were installed after the final image was made from it.
DEBUGIMAGES="N"
Set to "Y" to implant an MD5 image of the iso data in the application area of the CD. This allows the CD to be verified after a copy operation using information from two different areas of the CD. Very useful for mass CD distributions.
IMPLANTISOMD5="N"
Important Global Variables and naming conventions
The TEMPLATE* variables are the global context of the template we're actively working with. The following values are only set ONCE at the beginning of each command's initialization call to InitTemplateContext. They should never be written to anywhere else in the script. They can be saved, modified and restored with calls to Push and Pop... but I don't recommend it.
TEMPLATE=$PARM # Contains the full path to the template's root: /opt/epik.eps/proj/rh9.eps
TEMPLATEPATH=$PARMPATH # The parent path that contains the template
TEMPLATEBASENAME=$PARMBASENAME # The template name without the eps extension
TEMPLATENAME=$PARMNAME # short name: if TEMPLATE=/opt/epik/core.eps this would be: core
All files that are shipped outside the eps environment carry this timestamped name
TEMPLATERELEASENAME=$PARMTIMESTAMP # $TEMPLATENAME-$(date +%F-%H%M%S)
Two important variables that are used in the createinstall.sh and createimage.sh scripts are
$INSTALLROOT and $IMAGEROOT. $INSTALLROOT always points to the $TEMPLATE/install subdirectory.
$IMAGEROOT points to the $TEMPLATE/images/<the imagetype> subdirectory. For example if you're making an iso image IMAGEROOT would point to $TEMPLATE/images/iso.