Uploaded image for project: 'Red Hat CodeReady Studio (devstudio)'
  1. Red Hat CodeReady Studio (devstudio)
  2. JBDS-2254

clean out redundant test runtime caches on in static_build_env

    XMLWordPrintable

Details

    Description

      Noticed today that we have a lot of duplicate copies of the same runtimes. For example:

      /home/hudson/static_build_env/jbds/download-cache/soa-p-5.3.0.ER2.zip
/home/hudson/static_build_env/jbds/requirements/download/soa-p-5.3.0.ER2/soa-p-5.3.0.ER2.zip
/home/hudson/static_build_env/jbds/requirements/download/soap/soa-p-5.3.0.ER2.zip

      Doing a full search for dupes from the download-cache folder vs. the requirements/download/ folder, I see:

      $ cd /home/hudson/static_build_env/jbds/download-cache; for d in 
$(find . -type f -name "*.zip" | sort); do du -sch 
{/home/hudson/static_build_env/jbds/download-cache/,/home/hudson/static_build_env/jbds/requirements/download/*/}${d};
done

178M    /home/hudson/static_build_env/jbds/download-cache/./GateIn-3.1.0-FINAL-jbossas.zip
178M    /home/hudson/static_build_env/jbds/requirements/download/gatein-3.1.0.Final/./GateIn-3.1.0-FINAL-jbossas.zip
355M    total

8.0K    /home/hudson/static_build_env/jbds/download-cache/./jbds-seamfp-patch.zip
du: cannot access `/home/hudson/static_build_env/jbds/requirements/download/*/./jbds-seamfp-patch.zip': No such file or directory
8.0K    total

54M     /home/hudson/static_build_env/jbds/download-cache/./jboss-3.2.8.SP1.zip
54M     /home/hudson/static_build_env/jbds/requirements/download/jbossas/./jboss-3.2.8.SP1.zip
107M    total

78M     /home/hudson/static_build_env/jbds/download-cache/./jboss-4.0.5.GA.zip
78M     /home/hudson/static_build_env/jbds/requirements/download/jbossas/./jboss-4.0.5.GA.zip
155M    total

105M    /home/hudson/static_build_env/jbds/download-cache/./jboss-5.0.1.GA.zip
105M    /home/hudson/static_build_env/jbds/requirements/download/jbossas/./jboss-5.0.1.GA.zip
209M    total

128M    /home/hudson/static_build_env/jbds/download-cache/./jboss-5.1.0.GA.zip
128M    /home/hudson/static_build_env/jbds/requirements/download/jbossas-5.1.0.GA/./jboss-5.1.0.GA.zip
128M    /home/hudson/static_build_env/jbds/requirements/download/jbossas/./jboss-5.1.0.GA.zip
383M    total

270M    /home/hudson/static_build_env/jbds/download-cache/./jboss-eap-4.3.0.GA_CP10.zip
du: cannot access `/home/hudson/static_build_env/jbds/requirements/download/*/./jboss-eap-4.3.0.GA_CP10.zip': No such file or directory
270M    total

293M    /home/hudson/static_build_env/jbds/download-cache/./jboss-eap-5.1.2.zip
du: cannot access `/home/hudson/static_build_env/jbds/requirements/download/*/./jboss-eap-5.1.2.zip': No such file or directory
293M    total

110M    /home/hudson/static_build_env/jbds/download-cache/./jboss-eap-6.0.0.zip
110M    /home/hudson/static_build_env/jbds/requirements/download/jbeap-6/./jboss-eap-6.0.0.zip
220M    total

78M     /home/hudson/static_build_env/jbds/download-cache/./jboss-eap-fp-4.3.0.CP03-FP01.zip
78M     /home/hudson/static_build_env/jbds/requirements/download/seamfp/./jboss-eap-fp-4.3.0.CP03-FP01.zip
155M    total

193M    /home/hudson/static_build_env/jbds/download-cache/./jboss-eap-noauth-4.3.0.GA_CP03.zip
193M    /home/hudson/static_build_env/jbds/requirements/download/jbeap/./jboss-eap-noauth-4.3.0.GA_CP03.zip
385M    total

293M    /home/hudson/static_build_env/jbds/download-cache/./jboss-eap-noauth-5.1.2.zip
du: cannot access `/home/hudson/static_build_env/jbds/requirements/download/*/./jboss-eap-noauth-5.1.2.zip': No such file or directory
293M    total

189M    /home/hudson/static_build_env/jbds/download-cache/./jbossesb-4.11.zip
189M    /home/hudson/static_build_env/jbds/requirements/download/jbossesb-4.11/./jbossesb-4.11.zip
377M    total

157M    /home/hudson/static_build_env/jbds/download-cache/./jboss-portal-2.7.2-bundled.zip
157M    /home/hudson/static_build_env/jbds/requirements/download/JBoss-Portal-2.7.2.GA/./jboss-portal-2.7.2-bundled.zip
314M    total

26M     /home/hudson/static_build_env/jbds/download-cache/./JBossPortletBridge-2.1.0.FINAL.zip
26M     /home/hudson/static_build_env/jbds/requirements/download/JBossPortletBridge-2.1.0.FINAL/./JBossPortletBridge-2.1.0.FINAL.zip
51M     total

93M     /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.0.1.GA.zip
93M     /home/hudson/static_build_env/jbds/requirements/download/seam-2.0.1.GA/./jboss-seam-2.0.1.GA.zip
186M    total

89M     /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.0.2.SP1.zip
89M     /home/hudson/static_build_env/jbds/requirements/download/seam-2.0.2.SP1/./jboss-seam-2.0.2.SP1.zip
177M    total

89M     /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.0.3.CR1.zip
89M     /home/hudson/static_build_env/jbds/requirements/download/seam-2.0.3.CR1/./jboss-seam-2.0.3.CR1.zip
178M    total

107M    /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.1.1.GA.zip
107M    /home/hudson/static_build_env/jbds/requirements/download/seam-2.1.1.GA/./jboss-seam-2.1.1.GA.zip
213M    total

137M    /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.1.2.zip
137M    /home/hudson/static_build_env/jbds/requirements/download/seam-2.1.2.GA/./jboss-seam-2.1.2.zip
273M    total

141M    /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.2.0.CR1.zip
141M    /home/hudson/static_build_env/jbds/requirements/download/seam-2.2.0.CR1/./jboss-seam-2.2.0.CR1.zip
281M    total

113M    /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.2.0.GA.zip
113M    /home/hudson/static_build_env/jbds/requirements/download/seam-2.2.0.GA/./jboss-seam-2.2.0.GA.zip
225M    total

108M    /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.2.1.Final.zip
108M    /home/hudson/static_build_env/jbds/requirements/download/seam-2.2.1.Final/./jboss-seam-2.2.1.Final.zip
216M    total

110M    /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.3.0.ALPHA-distribution.zip
110M    /home/hudson/static_build_env/jbds/requirements/download/seam-2.3.0/./jboss-seam-2.3.0.ALPHA-distribution.zip
219M    total

24M     /home/hudson/static_build_env/jbds/download-cache/./jbpm-3.1.4.zip
24M     /home/hudson/static_build_env/jbds/requirements/download/jbpm-3.1.4/./jbpm-3.1.4.zip
47M     total

8.2M    /home/hudson/static_build_env/jbds/download-cache/./rhds-seam-gen-patch.zip
8.2M    /home/hudson/static_build_env/jbds/requirements/download/eapseamgen/./rhds-seam-gen-patch.zip
17M     total

60M     /home/hudson/static_build_env/jbds/download-cache/./riftsaw-2.3.0.Final.zip
60M     /home/hudson/static_build_env/jbds/requirements/download/riftsaw-2.3.0.Final/./riftsaw-2.3.0.Final.zip
120M    total

569M    /home/hudson/static_build_env/jbds/download-cache/./soa-4.3.0.GA_CP05.zip
569M    /home/hudson/static_build_env/jbds/requirements/download/soa-4.3.0.GA_CP05/./soa-4.3.0.GA_CP05.zip
569M    /home/hudson/static_build_env/jbds/requirements/download/soap/./soa-4.3.0.GA_CP05.zip
1.7G    total

593M    /home/hudson/static_build_env/jbds/download-cache/./soa-5.0.1.GA-signed.zip
593M    /home/hudson/static_build_env/jbds/requirements/download/soa-5.0.1.GA-signed/./soa-5.0.1.GA-signed.zip
593M    /home/hudson/static_build_env/jbds/requirements/download/soap/./soa-5.0.1.GA-signed.zip
1.8G    total

635M    /home/hudson/static_build_env/jbds/download-cache/./soa-p-5.3.0.ER2.zip
635M    /home/hudson/static_build_env/jbds/requirements/download/soa-p-5.3.0.ER2/./soa-p-5.3.0.ER2.zip
635M    /home/hudson/static_build_env/jbds/requirements/download/soap/./soa-p-5.3.0.ER2.zip
1.9G    total

629M    /home/hudson/static_build_env/jbds/download-cache/./soa-p-5.3.0.ER5.zip
du: cannot access `/home/hudson/static_build_env/jbds/requirements/download/*/./soa-p-5.3.0.ER5.zip': No such file or directory
629M    total

      Full disk footprint between the two locations:

      $ du -sch /home/hudson/static_build_env/jbds/download-cache /home/hudson/static_build_env/jbds/requirements/download/

5.6G    /home/hudson/static_build_env/jbds/download-cache
13G     /home/hudson/static_build_env/jbds/requirements/download/
18G     total

      So... seems like there are two problems:

      a) we have a lot of old/obsolete reqs which can be purged
      b) we fetch into two places - I suspect one path (download-cache?) is the new approach mistria wrote for JBDS 6, and the old path is the old Ant-based fetch approach (requirements/download/).

      Spoke with Len and he said we were good to purge these, but I wanted to double-check w/ mistria and dgolovin that wiping out all 18G and letting the builds re-fetch what they need is an acceptable way to clean up this area.

      Also, can we fix JBDS 5 (and 4?) so it uses the new maven-based fetch instead of the old Ant one, in order to avoid the duplicates and disk usage?

      Attachments

        Issue Links

          relates to

          Enhancement - An enhancement to or refactoring of existing functionality that is not configurable by an end user (typically a change made by an internal team that affects users) JBIDE-11714 Replace requirements folder & requirements bootstrap job by "self-hosted" content in pom.xml ?

          • Minor - To be worked after urgent, high, and medium priorities are resolved. This term is chosen when the severity of the issue is low, or the complexity or effort to correct it may be higher, relatively speaking. For low priority issues, known workarounds exist or are not needed due to the trivial effort needed to address the issue.
          • Closed

          Activity

            People

              nickboldt Nick Boldt
              nickboldt Nick Boldt
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: