maven: Target server failed to respond

Was getting an exception with the maven (3.2.2) wagon plugin version 2.6.

Looking deeper, this plugin uses wagon-provider 2.6 which depends on httpclient 4.3.  This can be seen here: http://repo1.maven.org/maven2/org/apache/maven/wagon/wagon-providers/2.6/wagon-providers-2.6.pom
and looks like this:

 ...  
 <dependencyManagement>  
 <dependencies>  
 <dependency>  
 <groupId>org.apache.httpcomponents</groupId>  
 <artifactId>httpclient</artifactId>  
 <version>4.3.1</version>  
 </dependency> ...  

The exception seen when running a maven build looks similar to this:

Caused by: org.apache.maven.wagon.providers.http.httpclient.NoHttpResponseException: The target server failed to respond

This bug: https://issues.apache.org/jira/browse/HTTPCLIENT-1531

indicates that there is a bug with httpclient in the versions 4.3 >= 4.3.4 & 4.4 Alpha1 when using a proxy (like apache httpd) between the client and server, without client authentication enabled.  The bug causes the no response exception later in the wagon plugin called via the WagonRepositoryConnector class.

Specifically, the bug occurs when authentication is disabled because as of 4.3.x the MainClientExec.java has a function to create tunnel to target, (createTunnelToTarget) that when authentication is disabled, the for loop does not exit properly and the request is never completed.

http://svn.apache.org/repos/asf/httpcomponents/httpclient/branches/4.3.x/httpclient/src/main/java/org/apache/http/impl/execchain/MainClientExec.java


In our case, just upgrading to 3.2.5 worked.   Another option is to include the lightweight http provider with wagon configuration.  this uses the Java HTTP libraries instead of apache's implementation.

Fetching Artifactory Maven Dependencies via Python

This clever python script will fetch your dependencies identified in a YAML file.  Don't forget that python uses white space for scope.  So make sure if you're going to copy and paste, to get it right.  The script uses artifactory's GAVC API.  This is the same API used by maven plugin for artifactory.  A nice feature is that you can run it multiple times, and by comparing md5 hashes, it will only download JAR files that have changed.  Also make note that javadoc, sources and POMs are omitted in the condition on line 67.

Because Python has a great API for Docker as well.  I will be using this code to implement something like fig to automatically deploy containers in my environment which can pull latest dependencies from Artifactory for installation.

First, an example of the YAML:

artifacts:
- artifact:
     artifactid:     accumulo-core
     groupid:     org.apache.accumulo
     version:     1.5.1

- artifact:
     artifactid:     accumulo-fate
     groupid:     org.apache.accumulo
     version:     1.5.1

- artifact:
     artifactid:     accumulo-trace
     groupid:     org.apache.accumulo
     version:     1.5.1



The Script:

#!/usr/bin/env python

import yaml

import hashlib

import os

import sys

import httplib

import json

import urllib2

from urlparse import urlparse

__author__ = 'champion'

artifactory_url = "art.mydomain.com:8081"

#local download folder

local_folder = "./deps"

conn = httplib.HTTPConnection(artifactory_url)

def download(filename, remote):

    print "\nDownloading: " + remote

    req = urllib2.urlopen(remote)

    blocksize = 16 * 1024

    with open(local_folder + "/" + filename, 'wb') as fp:

        while True:

            chunk = req.read(blocksize)

            if not chunk:

                break

            fp.write(chunk)

        fp.close()

def main():

    if not os.path.exists(local_folder):

        os.mkdir(local_folder)

    # Take last arg as filename

    filename = sys.argv[-1]

    if os.path.isfile(filename):

        stream = open(filename, 'r')

        yaml_instance = yaml.safe_load(stream)

        stream.close()

        artifacts = yaml_instance["artifacts"]

        print "\nFetching Artifacts in '" + filename + "' from Artifactory... "

        #for each element in YAML...

        for artifact in artifacts:

            entry = artifact["artifact"]

            artifact_version = str(entry["version"])

            artifact_groupid = str(entry["groupid"])

            artifact_artifactid = str(entry["artifactid"])

            # Create API call

            api_call = "/artifactory/api/search/gavc?g=" + artifact_groupid + "&a=" + artifact_artifactid + "&v=" + artifact_version

            # GET the results

            conn.request("GET", api_call)

            r1 = conn.getresponse()

            # If GET was Successful

            if r1.status == 200 and r1.reason == "OK":

                uris = json.loads(r1.read())["results"]

                # Omit Javadoc, Sources, POMs...

                for uri in uris:

                    link = uri["uri"]

                    if not link.endswith("pom") and not link.endswith("sources.jar") and not link.endswith("javadoc.jar"):

                        #Request the Artifact information

                        conn.request("GET", link)

                        artifact_json = conn.getresponse().read()

                        artifact_props = json.loads(artifact_json)

                        downloaduri = artifact_props["downloadUri"]

                        md5 = artifact_props["checksums"]["md5"]

                        fname = urlparse(downloaduri).path.split('/')[-1]

                        #Always Download Dep, unless conditions change.

                        omit_dl = False

                        if os.path.exists(local_folder + "/" + fname):

                            print "\nLocal Copy of '" + fname + "' Exists, checking md5..."

                            print "Remote MD5: " + md5

                            curr_md5 = hashlib.md5(open(local_folder + "/" + fname).read()).hexdigest()

                            print " Local MD5: " + curr_md5

                            if curr_md5 == md5:

                                omit_dl = True  # conditions changed

                        if not omit_dl:

                            download(fname, downloaduri)

                        else:

                            print "Hashes match, omitting download..."

                    else:

                        #artifact is not the binary jar

                        continue

            else:

                print "Artifact was not found in Artifactory."

        conn.close()

        print "Done."

    else:

        print "YAML file: '" + sys.argv[-1] + "' not found."

main()