Showing posts with label TeamCity. Show all posts
Showing posts with label TeamCity. Show all posts

Friday, October 8, 2010

Simple NAnt dependency manager for the TeamCity repository

I did not found any suitable tool that can be used to load dependencies when building .NET project in a TeamCity environment (using NAnt scripts).
Until now all dependencies where stored in VCS together with the source, not so sophisticated.

All our TeamCity projects contains one SDK-configuration that is a zipped file with all output assemblies and executables.

The TeamCity offers URL patterns to access build artifacts. For example:

http://teamcity.mycompany.com/guestAuth/repository/download/AGroup::AProduct/1.1.124/SDK/AProduct_SDK-1.1.124.zip

These two facts trigged me to build a custom NAnt task.
It simply downloads SDK-artifacts from the TeamCity repository, and unpacks the assemblies in the current build environment before the build is started.

I added a “pom”-file to all TeamCity configuration projects that defines the dependencies:

<project>
 <dependencyManagement>
   <repositories>
     <repository>http://teamcity.mycompany.com/guestAuth/repository/download/</repository>
   </repositories>
   <dependencies>
     <dependency>
       <groupId>AGroup</groupId>
       <artifactId>AProduct</artifactId>
       <version>1.1.124.0</version>
     </dependency>
     <dependency>
       <groupId>AGroup</groupId>
       <artifactId>BProduct</artifactId>
       <version>5.4.42.0</version>
     </dependency>
     <dependency>
       <groupId>BGroup</groupId>
       <artifactId>CProduct</artifactId>
       <version>2.5.24.0</version>
     </dependency>
   </dependencies>
 </dependencyManagement>
</project>

The NAnt task is executed in the build file:

<target name="loadDependencies">
  <loaddependencies filename="${Build.Base}\dependencies.xml" target="${Build.Base}\Dependencies"/>
</target>
The two parameters defines the name of the dependency file and where I want the assemblies to be copied.

And here is the source if someone is interested:

using System;
using System.Collections.Generic;

namespace MyNAnt.Build.Tasks
{
    using System.IO;
    using System.Net;
    using System.Xml;

    using global::NAnt.Core;
    using global::NAnt.Core.Attributes;
    using ICSharpCode.SharpZipLib.Zip;

    [TaskName("loaddependencies")]
    public class LoadDependenciesTask : Task
    {
        // NAnt parameters
        [TaskAttribute("filename", Required = true)]
        [StringValidator(AllowEmpty = false)]
        public string DependencyFile { get; set; }

        [TaskAttribute("target", Required = true)]
        [StringValidator(AllowEmpty = false)]
        public string Target { get; set; }

        // Lokal parameters
        private List<Dependency> Dependencies { get; set; }
        private List<string> Repositories { get; set; }

        /// <summary>
        /// Executes the NAnt task
        /// </summary>
        protected override void ExecuteTask()
        {
            // load all dependency information
            LoadDependencies();

            // download dependencies from TeamCity
            foreach (var dependency in Dependencies)
            {
                DownloadArtifact(Repositories, dependency.Group, dependency.Name, dependency.Version, Target);
            }
        }

        /// <summary>
        /// Reads the dependency list from configuration file
        /// </summary>
        private void LoadDependencies()
        {
            var doc = new XmlDocument();

            Log(Level.Info, "Loading dependencies from '{0}'.", DependencyFile);
            doc.Load(DependencyFile);

            // load all repositories to search for assemblies
            var repList = new List<string>();
            var repElemList = doc.GetElementsByTagName("repository");
            foreach (XmlNode repository in repElemList)
            {
                repList.Add(repository.InnerText);
            }
            Repositories = repList;

            // load all dependency assemblies
            var depList = new List<Dependency>();
            var depElemList = doc.GetElementsByTagName("dependency");
            foreach (XmlNode dependency in depElemList)
            {
                var item = new Dependency();
                if (dependency != null)
                {
                    item.Group = dependency["groupId"].InnerText;
                    item.Name = dependency["artifactId"].InnerText;
                    item.Version = dependency["version"].InnerText;
                }
                depList.Add(item);
            }
            Dependencies = depList;
        }

        /// <summary>
        /// Downloads and unzip artifact from TeamCity repository
        /// </summary>
        /// <param name="repositoryList"></param>
        /// <param name="group"></param>
        /// <param name="name"></param>
        /// <param name="version"></param>
        /// <param name="destination"></param>
        private void DownloadArtifact(List<string> repositoryList, string group, string name, string version, string destination)
        {
            Log(Level.Info, "Destination folder: '{0}'.", destination);

            // create destination folder if it not exist
            Directory.CreateDirectory(destination);

            foreach (var repository in repositoryList)
            {
                // URL-example
                // http://teamcity.mycompany.com/guestAuth/repository/download/AGroup::AProduct/1.1.124/SDK/AProduct_SDK-1.1.124.zip
                var address = repository + group + "::" + name + "/" + version + "/SDK/" + name + "_SDK-" + version + ".zip";

                using (var wc = new WebClient())
                {
                    try
                    {
                        using (var streamRemote = wc.OpenRead(new Uri(address)))
                        {
                            Log(Level.Info, "Found artifact '{0}'.", address);

                            var zis = new ZipInputStream(streamRemote);
                            ZipEntry ze;
                            while ((ze = zis.GetNextEntry()) != null)
                            {
                                if (ze.IsDirectory)
                                {
                                    Directory.CreateDirectory(ze.Name);
                                }
                                else
                                {
                                    var buffer = new byte[2048];
                                    var fileName = Path.GetFileName(ze.Name);

                                    Log(Level.Info, "Unzipping '{0}'.", fileName);

                                    using (
                                        Stream outstream = new FileStream(
                                            destination + "\\" + fileName, FileMode.Create))
                                    {
                                        while (true)
                                        {
                                            var bytes = zis.Read(buffer, 0, 2048);
                                            if (bytes > 0) outstream.Write(buffer, 0, bytes);
                                            else break;
                                        }
                                    }
                                }
                            }
                            return;
                        }
                    }
                    catch (Exception)
                    {
                        // ignore exceptions
                    }
                }
            }

            Log(Level.Error, "ERROR: Artifact '{0}::{1}' with version '{2}' not found!.", group, name, version);
        }
    }
}

Wednesday, October 21, 2009

Using secure Subversion from TeamCity

I have installed TeamCity (5.0 EAP version) on a Windows 2003 server. Both the Tomcat web server and the build agent are started as windows services.

The Subversion server is installed on a Linux server with Apache Tomcat web server. The Subversion server is protected with HTTPS.

When I tried to connect to the SVN-server through TeamCity, I always received the authentication error:

svn: Authentication required for '<https://<server name>:443>'

And the strange thing is that it worked fine if I used the SvnKit command tool, with the same user. So it was no certificate problem.

Searching the Internet for solutions always ended up with suggestions to change the svnkit.http.methods parameter.
But it had no effect on My problem. I was sure that NTLM authentication should be used, and from version 4.0.2 of TeamCity, the NTLM protocol is used by default.

Finally, I found out that SvnKit includes two NTLM implementations, the default is pure Java. But its also possible to use the native NTLM through the JNA library.

I added the svnkit.http.ntlm=jna parameter and suddenly the SVN connection was successful!!!
So much pain for this small window :-)

image 

JNA is included in the TeamCity Windows build agent package, so its not even necessary to install it on the server.

The SvnKit parameter must be defined in two places, for the build agent and for the web server:

1. The build agent properties file, i.e. <install path>TeamCity\buildAgent\launcher\conf\wrapper.conf:

# TeamCity agent JVM parameters
wrapper.app.parameter.2=-ea
wrapper.app.parameter.3=-Xmx512m
# The next line can be removed (and the rest of the lines renumbered) to prevent memory dumps on OutOfMemoryErrors
wrapper.app.parameter.4=-XX:+HeapDumpOnOutOfMemoryError
# Preventing process exiting on user log off
wrapper.app.parameter.5=-Xrs
# Uncomment the next line (insert the number instead of "N" and renumber the rest of the lines) to improve JVM performance
# wrapper.app.parameter.N=-server
wrapper.app.parameter.6=-Dlog4j.configuration=file:../conf/teamcity-agent-log4j.xml
wrapper.app.parameter.7=-Dsvnkit.http.ntlm=jna
wrapper.app.parameter.8=-Dteamcity_logs=../logs/
wrapper.app.parameter.9=jetbrains.buildServer.agent.AgentMain
# TeamCity agent parameters
wrapper.app.parameter.10=-file
wrapper.app.parameter.11=../conf/buildAgent.properties

2. Configure the Tomcat web server.

Open the configuration window with <install path>TeamCity\bin\tomcat6w.exe //ES//TeamCity.

Add the SvnKit parameter in the Java tab – Java Options:

image

Restart both services to get the new parameter initiated.