Get Gentoo!
gentoo.org sites
gentoo.org Wiki Bugs Forums Packages
Planet Archives Sources
Infra Status
aboutsummaryrefslogtreecommitdiff
path: root/slave/autotua
Commit message (Collapse)AuthorAgeFilesLines
* Explicitely use tuples for except:HEADmasterNirbheek Chauhan2009-02-242-2/+2
| | | | | This behaviour will change in Python 3.0, so we should make sure it's explicit right now.
* Minor changes to follow Pythonesque idiomsNirbheek Chauhan2009-02-242-21/+30
| | | | | * try..finally instead of osp.exists() * Explicit relative imports
* [bugfix] Don't check gpghome while init_gpghome()Nirbheek Chauhan2008-10-141-0/+6
| | | | | | | | | This is the kind of bug that is easy to miss without proper testing :) init_gpghome() would never work because it would check if self.gpghome was a valid gpghome before creating it :p Reported by Anielkis Herrera González (p0w3r3d)
* Basic slave-master stateful encrypted interactionNirbheek Chauhan2008-10-103-12/+206
| | | | | | | | | | | | | | | | | | | | | | | * Slave can "take" jobs from the master now - Asymmetrical encryption is used via GPG - models.Slave stores the gpg fingerprint in models.GPGFingerprintField - Slave imports the master's GPG key (/slave_api/autotua_master.asc) * Currently, Slave registration is manual (./manage.py shell) - Slave does fancy encrypted pickle talking (autotua.talk()) :) * "Take" jobs via autotua.Jobs().takejob(maintainer, job_name) - slave/autotua/crypt/__init__.py: * Implements the glue with `gpg` (maybe pygnupg later?) * Crypto() object. Has encrypt() and decrypt() - Also see autotua.decrypt_if_required() - GNUPGHOME for the slave is /var/tmp/autotua * => Job().fetch() requires root access (userpriv/sandbox later) * Phases store state to allow pausing/stopping and resuming of jobs - Future feature, not really used ATM - Job().everything() has prelim support for "resume" * Various small bug fixes and tweaks - Yes, I know I need to make this stuff more atomic :p
* Slave is licensed under GPL-3Nirbheek Chauhan2008-09-1810-10/+10
| | | | | - For compatibility with AGPL-3 - Include COPYING
* Notify the user if autotua_master is unsetNirbheek Chauhan2008-09-171-1/+3
|
* Bugfixes to master server.Nirbheek Chauhan2008-09-031-17/+21
| | | | | | | | | - Allow the master to run with a non-installed autotua-slave by not failing if /etc/autotua/slave.cfg doesn't exist - Use os.makedirs instead of os.mkdir in config.py for creating the autotua dirs - The user needs to manually enter the URL to the master server (temporary change)
* Use + not __add__ for adding tuples.Nirbheek Chauhan2008-08-311-1/+1
|
* Allow unprivileged users to parse jobuildsNirbheek Chauhan2008-08-301-3/+5
| | | | If we can't write to the logfile, don't :p
* Numerous small bugfixes for commit 5dc25dNirbheek Chauhan2008-08-304-18/+19
| | | | | | | | | | * Compatibility with git-1.6 (git-* commands are gone) * atexit.register() needs a function *reference* you dope :) * job_data['atoms'] is no longer an array, but is a space-separated string * jobuild.name -> jobuild.atom (a while ago) * Replace string exceptions with Exception(); for compatibility with unicode objects sent by django-1.0 * jobtagedir is different when using a chroot and not (duh)
* Make the slave system-installable and configurableNirbheek Chauhan2008-08-308-68/+137
| | | | | | | | | | | | | | | * Move configurable constants from const.py to config.py - These values are now overidden by /etc/autotua/slave.cfg * config.VERBOSE is overidden in slave.cfg - Fix all packages to import config.py instead - Change USE_PROXY to IGNORE_PROXY * bai bai test_modules.py. Testing is now done by executing modules * Install git-proxy-cmd.sh into /usr/bin Advantages: * python setup.py sdist * python setup.py install --root ~/autotua-slave * python setup.py help # ;-)
* Mount PORTAGE_DIR and DISTFILES_DIR only if they are directories. Also,Nirbheek Chauhan2008-08-301-3/+5
| | | | don't mount DISTFILES_DIR if it's PORTAGE_DIR+'/distfiles'
* Use "atexit" for cleaning up instead of the test_modules.sh kludge.Nirbheek Chauhan2008-08-301-1/+2
|
* Implement basic dependency resolution:Nirbheek Chauhan2008-08-177-34/+96
| | | | | | | | | | | | | | | | | | | | | | | | slave/autotua/jobuild/__init__.py:Resolver() * Basic dependency resolution; gives back a list of the job order * Does not deal with circular dependencies * Does not parse SIDEPEND slave/autotua/__init__.py slave/autotua/bin/jobuild.sh slave/autotua/bin/jobuild-functions.sh slave/autotua/jobuild/__init__.py * Pass jobtagedir around; don't assume it's always /tmp/autotua/jobtage inside a chroot slave/autotua/daemon/__init__.py slave/autotua/jobuild/__init__.py * Change things so we don't always have to chroot for parsing. Major speedup for dep resolution. * jobtagedir needs to be passed since we might not be chrooting for parsing slave/autotua/chroot/__init__.py * const.CHAUTOTUA_DIR could change from /tmp/autotua; use that instead of hardcoding the path slave/autotua/const.py * Use abspath for AUTOTUA_DIR
* Link the master and the slave in a simplistic (for now) way:Nirbheek Chauhan2008-08-162-67/+9
| | | | | | | | | * stage url processing is shifted to the master now * Communication is via protocol 2 (binary) pickles * slave API can be accessed via /slave_api/jobs /slave_api/jobs/<username> /slave_api/jobs/<username>/<job_name>
* Reorder slave/autotua/const.pyNirbheek Chauhan2008-08-151-29/+28
|
* Typos in jobuild-functions.shNirbheek Chauhan2008-08-091-3/+3
|
* Add http proxy support for git and git+sshNirbheek Chauhan2008-08-091-0/+9
| | | | git-proxy-cmd.sh uses the http_proxy variable to tunnel the git connections
* - Implement post() and emerge()Nirbheek Chauhan2008-08-071-0/+42
| | | | - Document a few functions in jobuild-functions.sh
* Initialise some basic variables: ATOMS, P, PN, PVNirbheek Chauhan2008-08-061-1/+21
|
* Just spent the better part of an hour tracing a regression that turned out ↵Nirbheek Chauhan2008-08-061-4/+5
| | | | | | to be a missing '/' Tell me. What did I do to deserve this?
* Move old autotua tmpdirs inside the chroot to /tmp/autotua-old instead of /tmpNirbheek Chauhan2008-08-061-1/+3
| | | | (reduce clutter)
* Fix syncing of the jobtage tree (master:master not HEAD:HEAD)Nirbheek Chauhan2008-08-051-1/+1
|
* * Fix unmounting when we die prematurelyNirbheek Chauhan2008-08-051-1/+1
|
* Whoops, jobtagedir should be relative to TMPDIRNirbheek Chauhan2008-07-111-1/+1
|
* - Message to indicate status (Preparing WorkChroot)Nirbheek Chauhan2008-07-092-1/+3
| | | | | - Another bug! (armin76 you... :P) RMTREE_INIT was incomplete and incorrect
* Fix major logic problems in autotua.sync which led to *no* syncing happening ↵Nirbheek Chauhan2008-07-081-22/+44
| | | | | | | | | | in most cases. (Found by armin76) - Use names for the return values of is_repo() (I should've implemented Ford_Prefect's suggestion earlier) - Renovate git cloning/fetching; inspired from git.eclass (Thanks for git.eclass dberkholz!) - Fix detection of repository for bzr - rm -rf properly when {git,bzr}-export-ing - Add a few comments explaining what's going on
* And it works! =)Nirbheek Chauhan2008-07-076-21/+154
| | | | | | | | | | | | | | | | | | | | | | | | | | | | slave/autotua/__init__.py: - _fetch_src -> _setup_jobfiles * Fetch jobfiles and hardlink them inside the chroot * Copy them if hardlinking fails - Job().run() slave/autotua/bin/jobuild-functions.sh: - Flick unpack() from pkgcore ;p - Skeleton src_unpack() - run_all_phases() - Fix indentation in get_param() slave/autotua/bin/jobuild.sh: - run_phase() -- implement "all" -> run_all_phases() slave/autotua/chroot/__init__.py: - mkdir -p $chrootdir/{bin,jobfiles,jobtage,src} * jobfiles will be linked into jobfiles/ * Extracted into src/ - Don't need to import sys slave/autotua/jobuild/__init__.py: - Document run_phase() - Check for returncode in _msg(msg) slave/autotua/const.py: - New 2008.0 release! :) slave/autotua/test_modules.py: - Update to run jobs - Update TODO
* A revolution in how jobuilds are parsed. All parsing is now done inside the ↵Nirbheek Chauhan2008-07-066-17/+47
| | | | | | | | | | | | | | | | | | | | chroot :P Why? Because everything has to be done in that enviroment stupid! slave/autotua/jobuild/__init__.py: - self.jobuild is now relative to the jobtagedir slave/autotua/bin/jobuild.sh: - Get AUTOTUA_DIR from autotua.daemon.Spawn() - Prepend AUTOTUA_DIR (inside the chroot) to the jobuild path slave/autotua/daemon/__init__.py: - Chroot all spawned processes - Send const.CHAUTOTUA_DIR (AUTOTUA_DIR inside the chroot) to the process slave/autotua/__init__.py: - Prepare chroot before parsing
* autotua.chroot.WorkChroot():Nirbheek Chauhan2008-07-051-3/+7
| | | | | - Bind-mount read-only by default - Everything except ${DISTDIR} is bind,ro
* autotua.chroot.WorkChroot():Nirbheek Chauhan2008-07-052-24/+37
| | | | | | | | | | | | | - Revamp _clean_mounts() * Reads /proc/mounts to find all mounts inside chrootdir - Move out bind mount code in _setup_mounts() to _bind() - Bind mount ${workdir}/jobtage and autotua/bin into ${chrootdir}/tmp/autotua - Tidy up in setup() before preparing chroot - Move ${chrootdir}/tmp/autotua to ${chrootdir}/tmp/aututua-${timestamp} - Replace useless osp.joins with direct path separators autotua.Job(): - Tidy before cleaning in clean()
* WorkChroot rsync is now 'rsync-nc' (no-clobber)Nirbheek Chauhan2008-07-042-6/+6
|
* Move jobuild.sh 'ping-pong' code to autotua.daemonNirbheek Chauhan2008-07-042-6/+10
|
* - Move running of commands to autotua.daemonNirbheek Chauhan2008-07-043-11/+36
| | | | | | * Makes the jobuild processor cleaner - Fix a bug in autotua.Job()._fetch_src() -- processing is no longer handled via the Jobuild() object - Update TODO to reflect this commit
* - Make PORTDIR and DISTDIR chroot bind mounting optionalNirbheek Chauhan2008-07-032-3/+11
| | | | default will be to fetch portage snapshot
* - Implement a jobuild processor: autotua.jobuild.Processor()Nirbheek Chauhan2008-07-032-6/+45
| | | | | - Insert placeholder for running phases inside a chroot (autotua/bin/jobuild.sh) - Remove a stray debug print
* No matter how hard I try, I can't seem to be able to make atomic commits :(Nirbheek Chauhan2008-07-024-35/+75
| | | | | | | | | | | | | Here's a load of further changes: - Move jobtage repo to git; incorporate that change here * Massive changes in autotua.sync =p * No way I'm sitting and documenting the changes * Syncing is still ugly, will document how it works once it's finalised - Print some more status info in Job() and autotua.sync - autotua.Job(),autotua.chroot:s/clean/tidy/ - Introduce new variable autotua.const.FULL_CLEAN to remove chroot data (unused atm) - Update test_modules.py - Update TODO
* Cleanup workdir+jobdir after work is doneNirbheek Chauhan2008-07-011-1/+3
|
* Implement support for {>,<}= in autotua.jobuild.Jobuild()._best_jobuild()Nirbheek Chauhan2008-07-012-6/+41
| | | | FTR, r'[0-9]+\.[0-9]+' + native python sorting is the sort order
* - Remove unneeded stuff from autotua.constNirbheek Chauhan2008-07-012-28/+1
| | | | - add --progress to rsync
* * Job metadata:Nirbheek Chauhan2008-06-297-26/+40
| | | | | | | | | | | | | | - "jobuilds" -> "atoms" * autotua: - Fix bad bug where jobuild objects were created before the jobtage tree was exported - We can't parse jobuild src_uri before the objects are created => move to prepare() - Jobuild portconf will now be stored inside the jobtage tree * autotua.fetch: - If a filename is omitted, assume basename(uri) * autotua.chroot: - rename prepare() to setup() -- prepare() sounds like it's about to be ready for running jobs :P * autotua.jobuild: - Read everything instead of just one line -- results of a single request will often be on multiple lines
* - Implement jobuild atom parsing and ↵Nirbheek Chauhan2008-06-284-35/+61
| | | | | | | | | | | | | autotua.jobuild.Jobuild()._best_jobuild() (incomplete) - Always use a tuple in autotua.sync.Command()._get_args(); fixes a nasty runtime bug - Use subprocess module in autotua.chroot - Implement an (incomplete) autotua.chroot.WorkChroot().clean() - Implement an (incomplete) autotua.Job().clean() which calls autotua.chroot.WorkChroot().clean() - Pass around jobdir = '/var/tmp/autotua/work/<job_atom>/' instead of jobtagedir, chrootdir etc - Add testing for autotua.Job() in test_modules.py - scripts/init-autotua-tmpdirs.sh for initialising the tmpdir structure - Add README
* - Implement basic jobuild parsing (jobuild.sh, jobuild-functions.sh, ↵Nirbheek Chauhan2008-06-266-44/+139
| | | | | | | | | | | | jobuild/__init__.py) - autotua.chroot doesn't take a Job() object as a param, specific params are needed now - WorkChroot().prepare() calls self.pristine.prepare() - Skeleton Jobuild().split_atom() function (to be moved out later) - Only supports fully versioned atoms atm - Integrate chroot and jobuild with autotua.Job() [messy atm] - Update TODO - test_module.py updated, but cannot test Job() right now
* - Implement a check for whether the chroot is pristineNirbheek Chauhan2008-06-253-8/+39
| | | | | | | - Use the subprocess module instead of os.system() - Verbosity control via autotua.const.VERBOSE - Don't compress rsync data (add a flag|AI later) - .gitignore
* Typos.Nirbheek Chauhan2008-06-221-1/+1
|
* - Use subprocess.check_call() instead of os.system() => no more brain-dead ↵Nirbheek Chauhan2008-06-212-33/+41
| | | | | | return value checking. - Abstract out the commands used to sync and init, support for non-standard behaviour via "params" (ugly)
* Misc small fixes:-Nirbheek Chauhan2008-06-201-3/+4
| | | | | | | | | | | autotua.chroot: - /usr/portage does not exist in the stage3, os.mkdir it - Rename force_reset to reset - Quote the umount command - Fix chrootdir typo test_modules.py: - Testing for autotua.chroot
* - Change header in skeleton jobuild.shNirbheek Chauhan2008-06-205-32/+55
| | | | | | | | - chroot.PristineChroot().prepare() implemented - Use the stage filename instead of a stage fetchable for init-ing chroot.PristineChroot() - chroot.WorkChroot().prepare() works now - Define const.{TARBALL_DIR, JOBFILE_DIR} - (Finally) make sync.Syncer() consistent -- it syncs contents into destdir
* - Whoops, forgot to save this and commit in the previous commitNirbheek Chauhan2008-06-201-1/+1
|
* - Rename self.maintainer to self.maintNirbheek Chauhan2008-06-204-14/+29
| | | | | | | | | - autotua.chroot -- self.destdir -> self.workdir - Make autotua.sync.Syncer().sync() behaviour consistent (rsync was different) - autotua.const -- CHROOT_WORKDIR -> WORKDIR - MAIN_JOBTAGE_DIR -> JOBTAGE_DIR - Each job has it's own exported jobtage tree (WORKDIR+'/jobtage') - Support bzr-export as a Sync scheme