1 2 3 Previous Next 44 Replies Latest reply on Jan 17, 2013 8:33 AM by dhladky

    (ORG-562) Automated Project Matrix - External/Awestruct sites

    aslak

      There is work being done to automate the project matrix page on jboss.org/projects (ORG-562) based on metadata from Magnolia. There is another feature request to add descriptions for each project to be included on the page (ORG-1408).

       

      According to Mark Newton in an internal mail, there will be some 'properties' files for Awestruct based sites.

       

      Where can I find some information on this file, format/ content/ location?

       

      As more and more projects move to Awestruct, won't having a manual properties file for these sites just slowly move this to a manually maintained page again?

       

      In the Arquillian project we've been look at exposing APIs on our site as well as the traditional html based view. Meaning we will be creating 'static' json/xml/yml APIs for e.g. Guide/Documentation API (tags, links, internal, external), Container/Extension API (maven artifacts, versions, configuration options). Similar we could create a Project API which would contain information about where source code is located, build server, documentation site etc etc..

       

      Is this something we could integrate in the Project Matrix automation job instead of harcoding properties files?

        • 1. Re: Automated Project Matrix - External/Awestruct sites (ORG-562)
          aslak

          Just to cleaify:

           

          So instead of having a arquillian-project.properties file located somewhere on jboss.org, you have the url: http://arquillian.org/api/project.json

           

          You probably don't need to call that API pr user request as this is mostly static information, but you could regen the matrix page e.g. once a week with fresh information.

           

          Then the content of the Matrix Project Page is in the hands of the people that actually control the info, and it's just a part of the normal project workflow. No content duplication.

          • 2. Re: Automated Project Matrix - External/Awestruct sites (ORG-562)
            lightguard

            Maybe I'm missing something, but couldn't we use the projects in JIRA for this? Certainly you'd need to do some trimming and condensing of projects, but all of our projects use the JBoss JIRA don't they?

            • 3. Re: Automated Project Matrix - External/Awestruct sites (ORG-562)
              dan.j.allen

              I tried to use JIRA for this originally, based on advice from Aslak. It gets you part of the way there, but not all the way. To flip it around, JIRA is a great input source, but it's not the only input source.

               

              Think of the architecture this way. We are creating a project catalog pipe (ala Yahoo Pipes). It will have multiple inputs. It may produce multiple outputs. One of those outputs is jboss.org/projects. Another is a REST endpoint for the catalog.

               

              Btw, I did create a script to pull from JIRA:

               

              https://gist.github.com/2699890

               

              which originally seeded this list:

               

              https://gist.github.com/7bfe5d4660d201463952

               

              Obviously, there is still work to do. I think the step to iron out right now is the API and data structures. Then we'll be ready to begin automation.

              • 4. Re: Automated Project Matrix - External/Awestruct sites (ORG-562)
                lightguard

                We really don't have a single source of truth for this information? That honestly surprises me.

                • 5. Re: Automated Project Matrix - External/Awestruct sites (ORG-562)
                  dan.j.allen

                  Nope, we don't. It surprised me too at first, but when I thought about it, I never had a place to enter the information, so how could there be.

                   

                  JIRA is also not sufficient because it's not necessarily the case that there is one JIRA per project, or that the project even uses JIRA. I don't think it's a good idea to tie metadata to a system like JIRA or Magnolia. This information should be in a technology agnostic location (just a data file). The project is free to generate this file in whatever way they want. We are going to include ours in the Arquillian website, likely. The central catalog then only has to maintain a list of URLs pointing to the data files.

                  • 6. Re: Automated Project Matrix - External/Awestruct sites (ORG-562)
                    lightguard

                    Honestly seems like some sort of script that should be run when we create a project (not that we do this all the time). The script would allow JIRA to be setup, add this information, maybe some other stuff as well. When everything was hosted like VCS then that would have been setup too. Maybe it's not such a good idea anymore. A simple datafile works just as well.

                    • 7. Re: Automated Project Matrix - External/Awestruct sites (ORG-562)
                      dan.j.allen

                      I'll copy here what I wrote in ORG-1408.

                       

                      We've been talking for a while about automating the list of projects on the index page. As it is now, the list is deficient on several accounts. There are projects missing, projects which shouldn't be listed and entries with outdated links.

                       

                      We can solve all these problems at once by pushing the project metadata (i.e., enlistment) out the projects and then using a script to aggregate the information together to generate the index page. This allows projects to maintain their own information and updates to be propagated automatically.

                       

                      For project metadata, I'm thinking about using something like DOAP or shorthand versions of it. I've explored a few formats in which to store information. Here's are three examples of the JBoss Tools project:

                       

                      DOAP

                       

                      <Project xmlns="http://usefulinc.com/ns/doap#"
                          xmlns:foaf="http://xmlns.com/foaf/0.1/"
                          xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
                          xmlns:jboss="http://api.jboss.org/doap-extensions">
                          <name>JBoss Tools</name>
                          <shortdesc>Eclipse-based tooling for developing with JBoss middleware</shortdesc>
                          <description>An integrated collection of Eclipse plugins to support the development of applications...</description>
                          <homepage rdf:resource="http://jboss.org/tools"/>
                          <download-page rdf:resource="http://jboss.org/tools/download"/>
                          <wiki rdf:resource="http://community.jboss.org/en/tools"/>
                          <mailing-list rdf:resource="https://lists.jboss.org/mailman/listinfo/jbosstools-dev"/>
                          <bug-database rdf:resource="http://issues.jboss.org/browse/JBIDE"/>
                          <license rdf:resource="http://www.eclipse.org/legal/epl-v10.html"/>
                          <maintainer>
                              <foaf:Person>
                                  <foaf:name>Max Ryhahl Andersen</foaf:name>
                                  <foaf:homepage rdf:resource="http://community.jboss.org/people/maxandersen"/>
                                  <jboss:userid>maxandersen</jboss:userid>
                              </foaf:Person>
                          </maintainer>
                          <repository>
                              <GitRepository>
                                   <location rdf:resource="git://github.com/jbosstools/jbosstools-svn-mirror.git"/>
                                   <browse rdf:resource="http://github.com/jbosstools/jbosstools-svn-mirror"/>
                              </GitRepository>
                           </repository>
                      </Project>
                      

                       

                      JSON

                       

                      {"Project":{ 
                        "name": "JBoss Tools", 
                        "shortdesc": "JBoss Tools", 
                        "description": "An integrated collection of Eclipse plugins to support the development of applications...", 
                        "status": "A",
                        "download-page": "http://jboss.org/tools/download",     
                        "mailing-list": "https://lists.jboss.org/mailman/listinfo/jbosstools-dev",     
                        "homepage": "http://jboss.org/tools",     
                        "GitRepository":{ 
                          "location": "git://github.com/jbosstools/jbosstools-svn-mirror.git",         
                          "browse": "http://github.com/jbosstools/jbosstools-svn-mirror"}, 
                        "licenses":[ {
                          "name": "Eclipse Public License v1.0",
                          "url": "http://www.eclipse.org/legal/epl-v10.html"}],     
                        "trackers":[ {
                          "name": "Bugs",
                          "location": "http://issues.jboss.org/browse/JBIDE"}],     
                        "maintainers":[ {
                          "name": "Max Ryhahl Andersen",
                          "homepage": "http://community.jboss.org/people/maxandersen"} ], }}
                      

                       

                      YAML

                       

                      - name: JBoss Tools
                        shortdesc: Eclipse-based tooling for developing with JBoss middleware
                        description: An integrated collection of Eclipse plugins to support the development of applications...
                        status: Active
                        homepage: http://jboss.org/tools
                        download_page: http://jboss.org/tools/download
                        wiki: http://community.jboss.org/en/tools
                        mailing_lists:
                          name: Development
                          url: https://lists.jboss.org/mailman/listinfo/jbosstools-dev
                        issue_trackers:
                          name: Issues
                          location: http://issues.jboss.org/browse/JBIDE
                        license:
                          name: Eclipse Public License v1.0
                          url: http://www.eclipse.org/legal/epl-v10.html
                        maintainers:
                          name: Max Ryhahl Andersen
                          homepage: http://community.jboss.org/people/maxandersen
                          username: maxandersen
                        GitRepository:
                          location: git://github.com/jbosstools/jbosstools-svn-mirror.git
                          browse: http://github.com/jbosstools/jbosstools-svn-mirror
                      

                       

                      I'm in favor of YAML. But we could allow projects to choose the format they prefer, I suppose.

                      • 8. Re: Automated Project Matrix - External/Awestruct sites (ORG-562)
                        aslak

                        The problem with the initial project setup is that this information change over time, even tho not often. So we're back at maintaining it manually again.

                         

                        A lot of projects recently moved from SVN to Git. Some projects are moving from Magnolia to Awestruct. Projects change names.

                        • 9. Re: Automated Project Matrix - External/Awestruct sites (ORG-562)
                          dan.j.allen

                          Btw, I got these ideas from the SourceForge API Wiki.

                          • 10. Re: Automated Project Matrix - External/Awestruct sites (ORG-562)
                            dan.j.allen

                            Aslak Knutsen wrote:

                             

                            Just to cleaify:

                             

                            So instead of having a arquillian-project.properties file located somewhere on jboss.org, you have the url: http://arquillian.org/api/project.json

                             

                            You probably don't need to call that API pr user request as this is mostly static information, but you could regen the matrix page e.g. once a week with fresh information.

                             

                            Then the content of the Matrix Project Page is in the hands of the people that actually control the info, and it's just a part of the normal project workflow. No content duplication.

                             

                            +1

                            • 11. Re: Automated Project Matrix - External/Awestruct sites (ORG-562)
                              aslak

                              Apache use DOAP as well: http://projects.apache.org/

                              • 12. Re: Automated Project Matrix - External/Awestruct sites (ORG-562)
                                dan.j.allen

                                Btw, the reason I created the list of projects is:

                                 

                                • To discover all the projects we have and who we need to contact
                                • To get lots of example data to use when formulating our data API

                                 

                                Plus, in the short run, we needed a full list for a separate purpose. I don't propose to maintain a centralized list. That's what we are moving away from. I just wanted to clear that up in case there was confusion.

                                 

                                The central list should be just a list of URLs (perhaps even self-discovered, though I'd be okay with explicit). This list might be important for ordering in the display. That list might be like:

                                 

                                http://arquillian.org/api/project.json

                                http://torquebox.org/api/project.json

                                http://jboss.org/jbossas/api/project.json

                                ...

                                 

                                That could even be in a github repo somewhere.

                                • 13. Re: Automated Project Matrix - External/Awestruct sites (ORG-562)
                                  dan.j.allen

                                  Jason Porter wrote:

                                   

                                  Honestly seems like some sort of script that should be run when we create a project (not that we do this all the time). The script would allow JIRA to be setup, add this information, maybe some other stuff as well. When everything was hosted like VCS then that would have been setup too. Maybe it's not such a good idea anymore. A simple datafile works just as well.

                                   

                                  The initial project setup automation is good as an independent idea. That would save a lot of time getting a project setup. I recommend filing an issue for that. A few bits of input could then kick off a script and setup all the various infrastructure for a project, permissions, etc.

                                  • 14. Re: Automated Project Matrix - External/Awestruct sites (ORG-562)
                                    dan.j.allen

                                    I think we can arm projects with some script raw material to generate the data file from sources like a pom.xml. But, I think it's reasonable to request for projects to maintain their data file. (If a project doesn't want to participate, then we'll just stick their file somewhere we can get at it until they want to maintain it).

                                    1 2 3 Previous Next