Jenkins + Groovy with the Job DSL Plugin

Jenkins + Groovy with the Job DSL Plugin



in this session we're going to be talking about how you can create your Jenkins jobs with groovy using the groovy job DSL and we'll cover what it does how it works how to get a DS DSL project set up and going some best practices and then we'll go over some examples and some of the stuff that we go over will be specific to the Jenkins domain but some of it will also be applicable to DSL in general so excuse me what we're talking about today falls under the idea of configuration is code which is kind of a subset of infrastructure as code and what that means is that we take some static edit in place kind of configuration files and we generate we generate them from code and we automate that process and that gives us all the benefits of having using a programming language as well as using a development process so that includes source control we check all of our DSL scripts into source control and that makes it a lot more visible everything's in one place searchable and traceable you can see who did what and win and win that change and stuff like that and so what we do with the DSL is we have a repository full of DSL scripts and then that's automatically synced to Jenkins so that whenever you change something in the in the repository it just automatically syncs up so also testable since this code you can you can run tests on you can pull out reusable components and test those make sure that they behave as you would expect and reviewable you can integrate it into whatever reviewing process you're already using at my job we have full requests so somebody submits a pull request and and then somebody else reviews it and merge and it's good to get a other set of eyes on that and finally you can pull out you know reuse logic and stuff like that so what is a Jenkins job we'll just cover real quick just kind of the basic structure of what what's in a job here you may have some input parameters you may be pulling some code from source control you may have some triggers you'll probably have one or more build steps and you may have some post build actions like emailing and publishing and stuff like that so I am going to just go real quick I assume those people are familiar with this UI I have a just one example here if I click on this and I go into the I go into the configuration you can see here it follows kind of that same structure we've got got a big form here a bunch of stuff hidden under these advanced buttons you've got you know source control your build steps that you can add post build actions which is also called publishers and some places but it you can look through here it's not too hard you can fill it out if you have a few jobs it's not a big deal to come in here and fill out create you know fill this form out a few times edit it once in a while but problem happens when you have you start to have more and more jobs or you want to do a job per branch or you want to you know you have very similar jobs and you have to copy and paste stuff for one from one job to the next and that that's what the DSL helps with so does anybody know how these jobs are this configuration is stored in Jenkins this Jenkins has no database everything is stored as a file so every job every view every folder has a config out XML file and you can easily view that by just appending could pick out XML to the URL so we can look at the XML for this file for this job and you can see it's basically the same structure you get the source control you get a couple get your Gradle bill here and then a couple of publishers here so so that's how that's stored and so some years ago somebody said why don't we just instead of using the UI why don't we just write a Ruby DSL that will generate its exact same XML so that's what the DSL is doing it is just a way to generate this exact same XML and so I have a example here so this is the DSL for the same job we just looked at and you can see it's very easy to just glance at and see what it's doing it follows the same structure that we described earlier and it's just very readable pretty easy to see what's going on here and so that's definitely a probe of DSL in general is that's very readable but one of the challenges can be writing one can I sit down and easily edit this or add a new job or whatever without having to go look a bunch of places to figure out what I'm supposed to what exact you know magic words I'm supposed to put in here or whatever so his next few slides go over a few things to make this more more writable first of all you can go you can take you can go I created this app a while ago where you can go in and you can put your dsl script in on the left and then you can run it and it'll generate xml for you on the right and that can be useful and I'll show you some use cases later on for that but and also if you have multiple jobs in here then it will run those and list them out here sequentially so you can generate a lot of XML from just a little bit of groovy code and next we have all the documentation for this is in the wiki so you can go in here and it describes all the job types you can create and the view types you can create and folders and so on and each of these has a page where you can go through and see all the options and the nested nested options and stuff like that so if I you know want to do a Grails I can go in here and get all the method signatures and and then down later on there's a block describing that with some examples usually so that's how the that's how the documentation is now but I've been working on a way to auto-generate the documentation there's groovy has a annotation called delegates to that you can put on closures that will indicate what the context of that closure is so all the closures in this DSL that marked up with that so you can programmatically follow the tree of options available and auto generate documentation that way so I have a script that will run it'll go in here and walk that tree and pull out all the all the Java Doc's and whatever and build some auto-generated documentation and so that's improv in it's in process now in progress I hope you have it done before this talk but it's pretty close so this is automatically generated there's it generates a JSON file and that JSON file is read it's read by this this app so this lists out all your top-level options here and then you can you know you can go in and see what all the what you can just go into all the nested options and see what's available and we haven't moved any of the text and code examples into into this yet but but all the structures there and you can see like for example if I go down to steps and I see there's Grails step requires the Grails plugin so that's indicated over here and you can follow that and get more information about that too but does not seem to be now maybe that we Jenkins weekís down or something anyway you can also go in here and search for commands oh there it is and you can also search and it lists plugins here these are all the plugins that are currently supported in here so that's that you can check that out that's in progress should be done sometime soon and the next is ID support since we're using that delegates to annotation is also used by the IDE to give you all the autocomplete options inside the closures and whatever since it it looks at that and knows what the cut what the context is so it will it will give you it all the autocomplete and documentation and stuff like that and I can show that for example so I have a script here and you can see that it it gives me that gives me those options and then it lets me get like quick documentation books about options that are there and you can step into them to see what the implementation is so so that makes it very useful makes it a lot easier to write that when you can easily get those you know available methods without even having to leave the IDE so how the plug-in works is that you the developer will have a local copy of the scripts he'll update those once he checks those in then that will change that will trigger the C job and the C job is responsible for doing the syncing so the C job will run the DSL and then do all the updating creating a deletion so the C Drive you have to configure manually but that's the only job and I will show you how to do that here if we just go here and create a new item freestyle job and so I've installed the DSL plugin and that gives me this step right here process the DSL now there's a couple ways you can provide a script you can provide one in line here which I'm just going to do as a demo that's about all it's useful for is demoing so for now I'm just going to create hundred jobs and then down here some options to how you want to handle existing jobs whether you want to ignore changes but probably want to do that and then what you want to do with jobs that were removed and I think in most cases you would want to delete those so we do that and run this C job now you can see it created a hundred jobs and go back here all those jobs now there if I go back into the C job and modify that now and delete say I only want one job now and then save that and rerun it now there's only one job and if you look at this output it says found one existing job and remove these other jobs so that's basically how it works there's one job remaining here now and so that's just some inline script but what you really want to do is you want to point it at a repository and I have a example here I'd suggest using Gradle because it gives you a bunch of benefits like you can easily run tests and set up pull down dependencies and stuff like that and this is the structure of the this example repository this repository it's been up here for a while you can check it out it's got all the examples from that we'll go over today and it's a good place to start if you're going to set up on these projects you can fork this or whatever but we put all the jobs and the jobs the folder all the DSL scripts and then any shell scripts or stuff that's included by us jobs put those in the resources folder and then if you pull out any classes or you want to have some support classes you can put them in source main groovy there's a special file here that indicates that he'll I do you support it tells the tells IntelliJ to what the context is for the for these script files and then any test can go into the source test groovy so that's how that's laid out and then I will point I'm going to point this C job at this repository here so I'll go back in here and going to add this pull this from source and I'll go down here there's a there's some instructions on this how to set up a C job here pretty simple I just need to tell it where my jobs are and so I switch that to look on the file system and then I just need to add some additional whatever additional class path I have so in this case I've got some stuff in the source main grew 8 and then normally we put like a Gradle test step before this just to make sure that the tests are running so now if I build this again so now it's got all my examples from that repository and I put each example in a separate folder just to keep them separated and so I'm going to step through some of those examples they're all in the in that repo if you want to check them out I'm gonna got them locally here I'm gonna step through so so in this first example is just basic job in polar creation here I'm creating a folder up here called example one and then I use that I use path notation in my job name to indicate that I want it to be in that folder so this is just a couple of sample jobs nothing uh nothing difficult here but example two is a little more interesting is this one shows a little bit more about you know power of using a programming language is that you can do stuff like a built for branch so what this is doing is pulling down its exact same jobs but it's pulling down from the github API all the branches for that repository and then iterating over each branch and then creating a folder per branch and then a set of jobs in each folder and we do something like this at my job we actually use the there's a Java wrapper for the github API we use that and that way we can we can check the dates on the branch and only pull in the most active branches and we can also delete branches that have been merged already so automatically whenever you create new branches jobs automatically created for that branch and then whenever you merge that branch jobs are automatically deleted so and this is just a simplified version of that so you can see now in this repository and example two we've got you know a folder for each branch and and each of those folders has a its own set of jobs and they're all pointing at specifically at that branch so that was example to this next example shows how to use a configure block now you may probably will run across something that you have in Jenkins that isn't supported yet but you can easily add that through this configure block which allows you to manually edit the XML in this case we're adding a node like this I'll show you like so if I was at if I was back here at this example original example and I want to add something that was not supported Nik say this thing and this actually is supported but just for this demo pretend it's not if I look at the XML and I say I can go down and find the XML for that and in this case it's right here and I can use that playground to help me you know make sure I've got the syntax right stuff but I can easily yeah add that here like that so there's nothing that you can't accomplish with the configure block anything it manually in this case there's the Dib operator is overloaded for the node here and so that allows you it does a it will find or create the notes in this case it'll find or create this in project publishers so it's going to create it and then add this node inside that so that's how that works and this next example may be kind of obvious but it comes up a lot where you have some jobs that are different just a couple properties you can easily pull them out into like a list of maps and then iterate over those and it's just something that pretty common so that's what that would look like and in this fifth example this shows how you can pull out chunks of just uh chunks of code here that you find that are being reused for example if I had a Gradle build and I was always doing the same you know proxying or whatever I had a bunch of options that were always the same I could easily pull those out into like static method like we have down here I have to remember to pass always past the delegate as in all these in all these closures there's a different delegate implicit variable and that is what all these methods are being resolved against so you have to if you pull that out and use it anywhere other than in that closure you have to pass the delegate and once you know you pass the delegate so like in the static method I can use I've got this this delegate variable here I can call the object out with method to make that the delegate again and then I can just you know resume with my DSL like I normally would and so that's pretty useful you can definitely eliminate a lot of copy and paste with something like this and you can also pull out I said earlier you get there's a resources directory where you put a shell scripts and stuff you can you can define them in line like this and even intend to them and then strip out the indent at the end to kind of make it look nice and it's groovy as well but it's a lot cleaner if you can pull those out and put them in you know put them in the resource directory and then just reference them and that makes those files reusable and other jobs and you can also it's a lot easier to you know edit one of these files if it's in its own file with the same file extension so that option is available and then the last example here it shows you how to pull out like whole jobs like if you want you want to create just a template for a whole job so we have these job builder classes where you pass you have some like default options that you can override and you just pass all the properties in that you want and and then and then build a job from there in this case you have to pass this variable which is the route context so we pass that in and that looks something like this so you have put all your jobs stuff in here and then you can reference you know the stuff that you've passed in and you can put logic in here and stuff like that if necessary so in here I'm passing I'm actually you don't have to declare this type but if you want if you want a you know Auto completion and stuff you have to tell it what type that is so so that is that's the last example any questions on any of those you yeah exactly so so testing there's a couple of different tests here I can show you the so I have one test here called the job that will iterate over all the job scripts and validated it's about at Val groovy and that you didn't miss type anything and that could be pretty handy I will you can see if I run that it's iterating over all scripts here and validating that they compile without an exception so if I was for example in one of these jobs and I had you know miss type something for some reason then I could immediately find out about that before checking it in and then also there's some examples in here for you know testing some of these support classes in this case we're just testing that these uh these options can be overridden and that if I specify emails the mail letters presents stuff like that so also if there's options like this will this DSL can automate all your Jenkins jobs but it doesn't include your plugins and your you know built your global configuration and all that kind of stuff and Eric helgesen has created a chef cookbook that includes the job ESL that you can use to do that if you want and that he's gonna be talking about that later today his automation talk talking about that and a bunch of other stuff so that's at 1:30 if you want to check that out and I totally breezed by that super fast so you can if you have any questions you can always hit up the forum and that's basically it or any other questions all right thank you

10 Comments

Leave a Reply

Your email address will not be published. Required fields are marked *