Showing posts with label Continuous Integration. Show all posts
Showing posts with label Continuous Integration. Show all posts

Friday, October 8, 2010

Simple NAnt dependency manager for the TeamCity repository

I did not found any suitable tool that can be used to load dependencies when building .NET project in a TeamCity environment (using NAnt scripts).
Until now all dependencies where stored in VCS together with the source, not so sophisticated.

All our TeamCity projects contains one SDK-configuration that is a zipped file with all output assemblies and executables.

The TeamCity offers URL patterns to access build artifacts. For example:

http://teamcity.mycompany.com/guestAuth/repository/download/AGroup::AProduct/1.1.124/SDK/AProduct_SDK-1.1.124.zip

These two facts trigged me to build a custom NAnt task.
It simply downloads SDK-artifacts from the TeamCity repository, and unpacks the assemblies in the current build environment before the build is started.

I added a “pom”-file to all TeamCity configuration projects that defines the dependencies:

<project>
 <dependencyManagement>
   <repositories>
     <repository>http://teamcity.mycompany.com/guestAuth/repository/download/</repository>
   </repositories>
   <dependencies>
     <dependency>
       <groupId>AGroup</groupId>
       <artifactId>AProduct</artifactId>
       <version>1.1.124.0</version>
     </dependency>
     <dependency>
       <groupId>AGroup</groupId>
       <artifactId>BProduct</artifactId>
       <version>5.4.42.0</version>
     </dependency>
     <dependency>
       <groupId>BGroup</groupId>
       <artifactId>CProduct</artifactId>
       <version>2.5.24.0</version>
     </dependency>
   </dependencies>
 </dependencyManagement>
</project>

The NAnt task is executed in the build file:

<target name="loadDependencies">
  <loaddependencies filename="${Build.Base}\dependencies.xml" target="${Build.Base}\Dependencies"/>
</target>
The two parameters defines the name of the dependency file and where I want the assemblies to be copied.

And here is the source if someone is interested:

using System;
using System.Collections.Generic;

namespace MyNAnt.Build.Tasks
{
    using System.IO;
    using System.Net;
    using System.Xml;

    using global::NAnt.Core;
    using global::NAnt.Core.Attributes;
    using ICSharpCode.SharpZipLib.Zip;

    [TaskName("loaddependencies")]
    public class LoadDependenciesTask : Task
    {
        // NAnt parameters
        [TaskAttribute("filename", Required = true)]
        [StringValidator(AllowEmpty = false)]
        public string DependencyFile { get; set; }

        [TaskAttribute("target", Required = true)]
        [StringValidator(AllowEmpty = false)]
        public string Target { get; set; }

        // Lokal parameters
        private List<Dependency> Dependencies { get; set; }
        private List<string> Repositories { get; set; }

        /// <summary>
        /// Executes the NAnt task
        /// </summary>
        protected override void ExecuteTask()
        {
            // load all dependency information
            LoadDependencies();

            // download dependencies from TeamCity
            foreach (var dependency in Dependencies)
            {
                DownloadArtifact(Repositories, dependency.Group, dependency.Name, dependency.Version, Target);
            }
        }

        /// <summary>
        /// Reads the dependency list from configuration file
        /// </summary>
        private void LoadDependencies()
        {
            var doc = new XmlDocument();

            Log(Level.Info, "Loading dependencies from '{0}'.", DependencyFile);
            doc.Load(DependencyFile);

            // load all repositories to search for assemblies
            var repList = new List<string>();
            var repElemList = doc.GetElementsByTagName("repository");
            foreach (XmlNode repository in repElemList)
            {
                repList.Add(repository.InnerText);
            }
            Repositories = repList;

            // load all dependency assemblies
            var depList = new List<Dependency>();
            var depElemList = doc.GetElementsByTagName("dependency");
            foreach (XmlNode dependency in depElemList)
            {
                var item = new Dependency();
                if (dependency != null)
                {
                    item.Group = dependency["groupId"].InnerText;
                    item.Name = dependency["artifactId"].InnerText;
                    item.Version = dependency["version"].InnerText;
                }
                depList.Add(item);
            }
            Dependencies = depList;
        }

        /// <summary>
        /// Downloads and unzip artifact from TeamCity repository
        /// </summary>
        /// <param name="repositoryList"></param>
        /// <param name="group"></param>
        /// <param name="name"></param>
        /// <param name="version"></param>
        /// <param name="destination"></param>
        private void DownloadArtifact(List<string> repositoryList, string group, string name, string version, string destination)
        {
            Log(Level.Info, "Destination folder: '{0}'.", destination);

            // create destination folder if it not exist
            Directory.CreateDirectory(destination);

            foreach (var repository in repositoryList)
            {
                // URL-example
                // http://teamcity.mycompany.com/guestAuth/repository/download/AGroup::AProduct/1.1.124/SDK/AProduct_SDK-1.1.124.zip
                var address = repository + group + "::" + name + "/" + version + "/SDK/" + name + "_SDK-" + version + ".zip";

                using (var wc = new WebClient())
                {
                    try
                    {
                        using (var streamRemote = wc.OpenRead(new Uri(address)))
                        {
                            Log(Level.Info, "Found artifact '{0}'.", address);

                            var zis = new ZipInputStream(streamRemote);
                            ZipEntry ze;
                            while ((ze = zis.GetNextEntry()) != null)
                            {
                                if (ze.IsDirectory)
                                {
                                    Directory.CreateDirectory(ze.Name);
                                }
                                else
                                {
                                    var buffer = new byte[2048];
                                    var fileName = Path.GetFileName(ze.Name);

                                    Log(Level.Info, "Unzipping '{0}'.", fileName);

                                    using (
                                        Stream outstream = new FileStream(
                                            destination + "\\" + fileName, FileMode.Create))
                                    {
                                        while (true)
                                        {
                                            var bytes = zis.Read(buffer, 0, 2048);
                                            if (bytes > 0) outstream.Write(buffer, 0, bytes);
                                            else break;
                                        }
                                    }
                                }
                            }
                            return;
                        }
                    }
                    catch (Exception)
                    {
                        // ignore exceptions
                    }
                }
            }

            Log(Level.Error, "ERROR: Artifact '{0}::{1}' with version '{2}' not found!.", group, name, version);
        }
    }
}

Wednesday, January 20, 2010

Subversion authentication problem in Hudson

I tried to move a TeamCity .NET project to Hudson.
Build script is written in NAnt so I only needed to change the TeamCity environment properties to the Hudson equivalents.

But I run into the same authentication problem as with TeamCity described in earlier post:

ERROR: Failed to update https://<removed>/svn/MyProject/trunk org.tmatesoft.svn.core.SVNCancelException: svn: authentication cancelled

Hudson uses the SvnKit as well so I was pretty sure what the cause was, the Java NTLM implementation.

After adding the property -Dsvnkit.http.ntlm=jna to the Hudson configuration file, hudson.xml, and restarting the Hudson Windows service, everything worked perfectly!

Wednesday, October 21, 2009

Using secure Subversion from TeamCity

I have installed TeamCity (5.0 EAP version) on a Windows 2003 server. Both the Tomcat web server and the build agent are started as windows services.

The Subversion server is installed on a Linux server with Apache Tomcat web server. The Subversion server is protected with HTTPS.

When I tried to connect to the SVN-server through TeamCity, I always received the authentication error:

svn: Authentication required for '<https://<server name>:443>'

And the strange thing is that it worked fine if I used the SvnKit command tool, with the same user. So it was no certificate problem.

Searching the Internet for solutions always ended up with suggestions to change the svnkit.http.methods parameter.
But it had no effect on My problem. I was sure that NTLM authentication should be used, and from version 4.0.2 of TeamCity, the NTLM protocol is used by default.

Finally, I found out that SvnKit includes two NTLM implementations, the default is pure Java. But its also possible to use the native NTLM through the JNA library.

I added the svnkit.http.ntlm=jna parameter and suddenly the SVN connection was successful!!!
So much pain for this small window :-)

image 

JNA is included in the TeamCity Windows build agent package, so its not even necessary to install it on the server.

The SvnKit parameter must be defined in two places, for the build agent and for the web server:

1. The build agent properties file, i.e. <install path>TeamCity\buildAgent\launcher\conf\wrapper.conf:

# TeamCity agent JVM parameters
wrapper.app.parameter.2=-ea
wrapper.app.parameter.3=-Xmx512m
# The next line can be removed (and the rest of the lines renumbered) to prevent memory dumps on OutOfMemoryErrors
wrapper.app.parameter.4=-XX:+HeapDumpOnOutOfMemoryError
# Preventing process exiting on user log off
wrapper.app.parameter.5=-Xrs
# Uncomment the next line (insert the number instead of "N" and renumber the rest of the lines) to improve JVM performance
# wrapper.app.parameter.N=-server
wrapper.app.parameter.6=-Dlog4j.configuration=file:../conf/teamcity-agent-log4j.xml
wrapper.app.parameter.7=-Dsvnkit.http.ntlm=jna
wrapper.app.parameter.8=-Dteamcity_logs=../logs/
wrapper.app.parameter.9=jetbrains.buildServer.agent.AgentMain
# TeamCity agent parameters
wrapper.app.parameter.10=-file
wrapper.app.parameter.11=../conf/buildAgent.properties

2. Configure the Tomcat web server.

Open the configuration window with <install path>TeamCity\bin\tomcat6w.exe //ES//TeamCity.

Add the SvnKit parameter in the Java tab – Java Options:

image

Restart both services to get the new parameter initiated.

Monday, March 9, 2009

Continuous Integration Server Configuration

There are a few simple steps to set up a build server, but I always forget how and where I found the answer on the few problems that always pops up.

Here are the steps for building .NET 3.5 web-projects on a Windows 2008 server (using NAnt and NUnit):

Install:
  1. NAnt
  2. NUnit
  3. NCover
  4. NCoverExplorer (for fancy unit test coverage reports)
  5. .NET 3.5 Framework SDK
  6. CruiseControl.NET (or TeamCity)
Some special handling after installation:
  1. Open NAnt.exe.config and change the sdkInstallRoot value to a correct path.
    <readregistry
        property="sdkInstallRoot"
        key="SOFTWARE\Microsoft\Microsoft SDKs\Windows\v6.1\WinSDKNetFxTools\InstallationFolder"
        hive="LocalMachine"
        failonerror="false" />
  2. Create the C:\Program Files\MSBuild\Microsoft\VisualStudio\v9.0\WebApplications folder and copy the Microsoft.WebApplication.targets file from a computer with Visual Studio installed. (Used by Web Application Projects)