Rather than manually pulling requirements, it would be hella sweet if there was a job config into which we could just list all the URLs to mirror and their matching project names.

Then this job would scrape the .target files for the current list of URLs used, grep out the /requirements/<reqname>/<version>, fetch a new mirror for each new version, and dump an updated copy of the .target file into the job's workspace.

Thus for webtools we might simply define:
webtools,S-3.8.0M7-20160503010110,http://download.eclipse.org/webtools/downloads/drops/R3.8.0/S-3.8.0M7-20160503010110/repository/

and we'd end up with:

http://download.jboss.org/jbosstools/updates/requirements/webtools/S-3.8.0M7-20160503010110/

• Since we already have a http://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/DevStudio/view/DevStudio_Master/job/jbosstools-requirements/ job we can just pass parameters to that to invoke the mirroring steps. Would be even better if we could run multiple calls in parallel as neon and wtp can take >2hr

1. matrix job. each config is a trio of reqname/version/URL which is passed to a process akin to that of the jbosstools-requirements job to perform the mirror; process is now scripted here: https://github.com/jbosstools/jbosstools-build-ci/blob/jbosstools-4.4.x/publish/mirror.sh

2. when all children are done, a downstream job can runs TP update & validation against new .target files

• fetch http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/jbosstoolstargetplatformrequirements-mirror-matrix/lastBuild/api/xml?xpath=//description
• parse that into a list of URLs
• each URL contains REQ_NAME/VERSION, which can then be matched up with similar lines in .target files
• run p2diff between old/new URLs in .target to generate list of changes and verify new site contains all the same IUs
• resulting edited .target files will remain in the workspace, and we can then run
• https://github.com/jbosstools/jbosstools-build-ci/blob/jbosstools-4.4.x/util/verifyTarget.sh

• when done if success:
• ideally, generate a PR and attach link in email to jbosstools-dev@
• if can't generate PR, then attach patch in email to nickboldt & mistria to apply locally in jbosstools-target-platforms to create a PR
• email should includes boilerplate text to send mail to list announcing new change for review